Training an ensemble DNNs in a for loop

username: tanweer-mahdi
cooked: <p>Hello everyone,</p>
<p>I have been using PyTorch for last couple of days. I have faced two encounters where I noticed some unusual loss pattern while training DNNs in a for loop. I am presenting one of those here.</p>
<p>I am trying to train an ensemble of 5 DNNs here.</p>
<pre><code class="lang-auto">for e in range(ensemble):
    training = 'training_set_' + str(e) + '.npy'
    labels = 'labels_' + str(e) + '.npy'
    nomadata = NOMAdata(training, labels)
    for layer in model.children():
        if hasattr(layer, 'reset_parameters'):
            layer.reset_parameters()
    # model = DeepNOMA(structure)


    for fold, (train_idx, test_idx) in enumerate(cv.split(nomadata)):

        # creating the sampler
        train_sampler = torch.utils.data.SubsetRandomSampler(train_idx)
        test_sampler = torch.utils.data.SubsetRandomSampler(test_idx)

        trainloader = DataLoader(nomadata, batch_size = 250, sampler  = train_sampler)
        testloader = DataLoader(nomadata, batch_size= 250, sampler = test_sampler)

        # resetting the parameters
        # model = DeepNOMA(structure)

        running_loss = 0
        if fold == 0:
            for epoch in range(5):
                for i, val in enumerate(trainloader):
                    inputs, targets = val
                    # clear the gradients
                    optimizer.zero_grad()
                    # model output
                    yhat = model(inputs)
                    # calculate loss
                    loss = lossfun(yhat, targets)
                    # backprop
                    loss.backward()
                    # update model parameter
                    optimizer.step()
                    # loggin training performance
                    running_loss += loss.item()

                    if i % 20 == 19:
                        # # calculating validation loss
                        model.eval()
                        validation_loss = 0
                        for j, batch in enumerate(testloader):
                            test, labels = batch
                            ypred = model(test)
                            runval = lossfun(ypred, labels)
                            validation_loss += runval.item()

                        writer.add_scalars('Training/Validation Loss', {'Training loss': running_loss/20, 'Validation Loss': validation_loss/j}, epoch*len(trainloader) + i)

                        model.train()
                        print(running_loss/20, validation_loss/j, fold, e)
                        running_loss = 0







    # Saving the entire model
    save_path = 'model' + str(e) + '.pth'
    torch.save(model.state_dict(), save_path)
</code></pre>
<p>As you can see, each DNN is being trained with different datasets. Below is the value of loss functions printed in (training loss, validation loss, fold, ensemble index) format.</p>
<pre><code class="lang-auto">0.6160063549876214 0.5588574174678687 0 0
0.35422628968954084 0.34084207271084643 0 0
0.25784978866577146 0.2524514478264433 0 0
0.22559681087732314 0.22244494445998259 0 0
.
.
.
.
0.015405059373006225 0.005930508576295894 0 0
0.016571250976994634 0.005989199118557001 0 0
0.02103429418057203 0.0058264776904399344 0 1
0.02045736024156213 0.0058095609872705406 0 1
0.021323334984481336 0.0058715936229235 0 1
0.020745716243982314 0.005786245124358119 0 1

</code></pre>
<p>As you can see, although the loop proceeds into training a new model (the ensemble index changed from 0 to 1), the loss function is surprisingly small! (if you compare with the value at the top of the snippet). It seems like the parameter reset function was not working at all.</p>
<p>All these my experiences are suggesting that there must have been a better practice for training model within a for loop which is not known to me. Can anyone elaborate on this issue? Thanks in advance!</p>