How to reset model weights to effectively implement crossvalidation?

username: DaftJamrock
cooked: <p>I am trying to implement cross validation by running an instance of my LSTM model on different crossvalidation fold datasets. The issue I’m having is that the model is remembering the weights of the parameters with each subsequent run of the cross validation. What is the easiest way to reset the weights of the model so that each cross validation fold starts from some random initial state and is not learning from the previous folds?</p>
<p>Here is my model as currently defined:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    '''
    LSTM Model parameters
    '''
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTMModel, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.output_dim = output_dim
        self.num_layers = num_layers
        self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)
        self.forecast = nn.Linear(self.hidden_dim, self.output_dim)

    def forward(self, x):

        batch_size = 1
        h0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.num_layers, batch_size, self.hidden_dim).requires_grad_()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        out = self.forecast(out[:, -1, :])
        out = out.unsqueeze_(-1)
        return out

def train_model(datafile, date_col, target_col, hidden_dim,
                num_layers, output_dim, num_epochs, learning_rate):
    dataset = CreateDataset(datafile, date_col, target_col)
    input_dim = dataset.num_features

    X_train_sets = dataset.X_train_sets
    y_train_sets = dataset.y_train_sets
    X_test_sets = dataset.X_test_sets
    y_test_sets = dataset.y_test_sets

    model = LSTMModel(input_dim, hidden_dim, num_layers, output_dim)
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    hist = np.zeros(num_epochs)
    test_error = []

    for X_train, y_train, X_test, y_test in zip(
            X_train_sets, y_train_sets, X_test_sets, y_test_sets):
        for epoch in range(num_epochs):
            optimizer.zero_grad()
            output = model.forward(X_train)
            loss = criterion(output, y_train)

            if epoch % 100 == 0:
                print('Epoch ', epoch, 'Loss: ', loss.item())
            hist[epoch] = loss.item()
            loss.backward()
            optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>You could call <code>.reset_parameters()</code> on all child modules:</p>
<pre><code class="lang-python">model = LSTMModel(1, 1, 1, 1)
for name, module in model.named_children():
    print('resetting ', name)
    module.reset_parameters()
</code></pre>

username: DaftJamrock
cooked: <p>Thank you very much!</p>

username: i-cant-code
cooked: <p>would it also work to just do:</p>
<p>model = LSTMModel(input_dim, hidden_dim, num_layers, output_dim)</p>
<p>under the first loop block every time?</p>

username: ptrblck
cooked: <p>This could work and depends a bit on your work flow.<br>
Creating a new module would also force you to recreate the optimizer, since the old optimizer will contain the references to the already deleted model.</p>

username: i-cant-code
cooked: <p>Well, for my problem I was doing a 5 fold cross validation using Unet, and what I would do is create a new instance of the model every time and I would create a new instance of the optimizer as well. I assumed it was working because the F1 score would start at 0 each fold when I keep track of the F1 score each epoch.</p>

username: ptrblck
cooked: <p>Yes, as explained it might fit your use case, if you are re-initializing the optimizers as well. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: i-cant-code
cooked: <p>When you say It might work, is there a possibility that it is not working as intended? I just want to be absolutely sure. I am using the UNet in this repo:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/jvanvugt/pytorch-unet/blob/master/unet.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/jvanvugt/pytorch-unet/blob/master/unet.py" target="_blank" rel="nofollow noopener">jvanvugt/pytorch-unet/blob/master/unet.py</a></h4>
<pre><code class="lang-py"># Adapted from https://discuss.pytorch.org/t/unet-implementation/426

import torch
from torch import nn
import torch.nn.functional as F


class UNet(nn.Module):
    def __init__(
        self,
        in_channels=1,
        n_classes=2,
        depth=5,
        wf=6,
        padding=False,
        batch_norm=False,
        up_mode='upconv',
    ):
        """
        Implementation of
</code></pre>

  This file has been truncated. <a href="https://github.com/jvanvugt/pytorch-unet/blob/master/unet.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and my fold loop is like such:</p>
<pre><code class="lang-auto">for fold in range 5:
   model = UNet().to(device)
   optimizer = torch.optim.adam(model.parameters(), learning rate)
   criterion = ...
   #select data
   #train/validate
</code></pre>

username: ptrblck
cooked: <p>Not necessarily, but I can’t claim it’ll work without seeing code or reproducing it myself.<br>
That being said, the approach is right, and especially since you are seeing the desired behavior, you can assume it’s working.</p>

username: i-cant-code
cooked: <p>Is there any way to absolutely guarantee that the weights are reset? I just want to be precise as possible because it is for a paper I am working on.</p>

username: i-cant-code
cooked: <p>alternatively I could send you the code privately if you would be willing to take a look because it is not in a public repository, I’ll just need to verify that it is ok to do so.</p>

username: ptrblck
cooked: <p>No, I wouldn’t recommend to send private repositories to people outside your lab (or other collaborators).</p>
<p>A new model instance will have newly initialized parameters. You could double check it by creating a <code>deepcopy</code> of the already trained model, create the new model and optimizer, and compare the <code>state_dict</code>s of both.</p>

username: i-cant-code
cooked: <p>Ok, one follow up question. If I am to pretrain the model on one data set, save the weights and optimizer for continuing to train later, once I load in these weights to train on the new data set, should I reset the optimizer when I start to train on the new dataset? or should I load in the optimizer dictionary information? I was looking at a pytorch documentation about saving models and it said to save the optimizer when you want to continue training, but I am not sure if that applies to pretraining and then training on a new dataset.</p>

username: ptrblck
cooked: <p>If your optimizer contains internal states (running estimates of the gradients etc.), it would probably make sense to restore it. Otherwise you might see a peak in your loss when you continue the training on the new dataset. I don’t think the loading of the optimizer is dataset-dependent, but let us know, if you see a different behavior for your use case (which would be interesting to know).</p>

username: i-cant-code
cooked: <p>Ok yeah, I’ll try it both ways and update you on the results! Thanks!</p>