Resume Training Does not Work

skiddles: <p>I am new to PyTorch and I am trying to create word embeddings to learn PyTorch.  Unfortunately, I am running on an old laptop and only get limited training runs before having to shutdown.  So I am trying to set the model up to resume training, only it does not appear to resume training.  If interested, the full code is <a href="https://github.com/skiddles/PyTorch.WordEmbeddings/blob/master/PyTorch_WordEmbedding_BeigeBook.py" rel="nofollow noopener">here</a>.  I come to this conclusion as when I train for multiple epochs, the loss consistently declines, in this case from about 11,000 down to about 5,000.  From epoch n to n+1 the loss is relatively stable. When I try to resume from a saved checkpoint, the loss starts out at 11,000 again.</p>
<p>Maybe I am misinterpreting what is happening.  Here are the relevant snippits of code.</p>
<p>At the end of each epoch I save the model:</p>
<pre><code class="lang-auto">def save_checkpoint(state, filename):
    torch.save(state, filename)

save_checkpoint({
            'epoch': epoch + 1,
            'state_dict': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }, parms.save_dir)
</code></pre>
<p>Then on resume, I load the model with this code:</p>
<pre><code class="lang-auto">checkpoint = torch.load(parms.save_dir)
parms.start_epoch = checkpoint['epoch']
model.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])
</code></pre>
<p>Training resumes at the next epoch, but the loss values make me think the weights are not saved or reapplied.</p>
<p>What am I missing?</p>

ptrblck: <p>Did you use some kind of <code>lr_scheduler</code>?<br>
Your saveing code looks alright so far.<br>
I’ve created a small <a href="https://gist.github.com/ptrblck/a496b81ec705d4a8fe951becb2aa79c6">gist</a> a while ago, since another user reported a similar issue.<br>
Could you compare your code to mine and check, if there are any logical differences?</p>

skiddles: <p>Hi ptrblck,</p>
<p>Thanks for looking over my problem.  I can’t see any logical differences between my save/resume code and yours. Not saying that there’s no difference, just that I cannot see any difference.  Did you happen to see my full code?  One difference I did notice between you gist and my code is that I don’t have a train function.  Since I am new, I have been thinking this is okay because I have the code in the <strong>main</strong> section.  But I am questioning everything at this point.</p>

skiddles: <p>One other thing that I was looking at is the way I tried to implement mini-batch processing.  TBH, I don’t see how it would be causing the resume to work incorrectly.</p>
<p>Here is my line for the DataLoader:</p>
<pre><code class="lang-auto">    dataloader = DataLoader(text_data,
                            batch_size=parms.MINI_BATCH,
                            shuffle=True,
                            num_workers=3)
</code></pre>
<p>Later, in the training I use slices to pass in a MINI_BATCH of records at a time.</p>
<pre><code class="lang-auto">...
    for epoch in EPOCH_RANGE:

        current_start = 0 #This keeps track of the current starting row in the mini
        total_loss = torch.Tensor([0])

        keep_going = True
        while keep_going:
            if current_start + parms.MINI_BATCH &lt; len(text_data.target_ids):
                minibatchids = slice(current_start, current_start + parms.MINI_BATCH -1)
            else:
                minibatchids = slice(current_start, len(text_data.target_ids))
                keep_going = False

            model.zero_grad()
            log_probs = model(torch.Tensor(text_data.context_ids[minibatchids]).long()) # This passes in a slice of instances from the data.
            loss = loss_function(log_probs,
                                 torch.autograd.Variable(
                                     torch.squeeze(
                                        torch.Tensor(text_data.target_ids[minibatchids]).long()
                                        )
                                     )
                                 ) * parms.MINI_BATCH
            loss.backward()
            optimizer.step()
...

</code></pre>
<p>This SEEMs to work and I get a ten times improvement in throughput, but maybe this is part of the problem?  It shouldn’t be as its not the weights.  Maybe there is a better way to implement this in a more PyTorch fashion.</p>
<p>Since I didn’t document it earlier, I am running:</p>
<ul>
<li>Linux 16.04 LTS</li>
<li>Python 3.6</li>
<li>PyTorch 0.40 (CPU bound as no eligible GPU)</li>
</ul>
<p>Any help would be greatly appreciated</p>

ptrblck: <p>Could you help me out with some tensor shapes to get the code running without the dataset?</p>
<ul>
<li><code>len(text_data.vocab)</code></li>
<li>shape of <code>text_data.context_ids[minibatchids]).long()</code>
</li>
</ul>

skiddles: <p>len(text_data.vocab) = 15195</p>
<p>text_data.context_ids[minibatchids]).long()  = torch.Size([999, 2])</p>
<p>Thanks</p>

ptrblck: <p>I cloned your repo, since it was easier, but thanks for the numbers. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Using the default params, I get an error in calculating the loss.<br>
While <code>log_probs</code> has a shape of <code>[1, 15195]</code>, <code>text_data.target_ids[minibatchids]</code> has a shape of <code>[1, 1]</code>.<br>
Do you have the same issue?</p>

skiddles: <p><code>text_data.target_ids[minibatchids]</code> should be [999,1]</p>
<p><code>log_probs`` shape is [999, 15195]</code></p>
<p>Let me check the default params on github. I am new to github so maybe the defaults are not what I expected.</p>

skiddles: <p>I did actually update a few of the default parameters.</p>

ptrblck: <p>Thanks! I pulled the current code and still there is an error regarding the loss function.<br>
<code>len(text_data.target_ids)</code> is just <code>1</code>. Could it be the <code>Corpus.pkl</code> is wrong?</p>

skiddles: <p>I assume the pickle file (clean_corpus.pkl) is ok. I have downloaded it from github and it seems to work for me.</p>

skiddles: <p>Hi ptrblck,</p>
<p>I really appreciate that you have been helping me with this.  I have been trying to confirm as much as I can, although the holiday has been eating my time thankfully. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I used the following code to try and prove that the weights are being reloaded on a restart.</p>
<pre><code class="lang-auto">        f = open('./data/weights.txt', 'w')
        for x in model.parameters():
            f.write(str(x.data))
        f.close()
</code></pre>
<p>I run this code at the end of each epoch and after loading the <code>model</code>.</p>
<p>I have managed to confirm that the model is setting the weights on load.  That said, the loss still goes back to basically the starting values.  I am struggling to interpret this behavior. I am left with the following questions:</p>
<ol>
<li>Is there a reasonable explanation for this behavior?</li>
<li>Is this a function of the loss measure?</li>
</ol>
<p>Given that this is an unsupervised process, I cannot tell if the model is trained enough.  I would assume that I would run it until the loss flatlines, but I cannot run this on my laptop that long, which is why I am focus on the restart function.</p>
<p>Any advice would be appreciated.</p>

ptrblck: <p>Could you train you model for a while, save the optimizer, reload it, and continue the training in the same session?<br>
The issue seems strange and I think we have to debug it step by step.</p>

skiddles: <p>Hello again,</p>
<p>I did save the optimizer and reloaded it in session and the loss continued to decline from the previous epochs.  When I did this in a subsequent session, the loss would bounce back up to the approximate loss from the first epoch.</p>
<p>Is the loss function a static construct?  Should I be saving the loss function and reloading it next session?</p>
<p>Thanks ptrblck</p>

ptrblck: <p>What kind of loss function are you using?<br>
Your code just specifies <code>loss_function</code>.<br>
Is it stateful, i.e. does it contain any parameters?</p>

skiddles: <p><code>loss_function = nn.NLLLoss()</code> no parameters.</p>
<p>Should it be stateful?  How do I make it stateful?</p>
<p>Sounds like it should be stateful</p>

ptrblck: <p>No, it’s fine. You don’t have to save and load it, since it’s just a function without any stored parameters.</p>

skiddles: <p>Okay.  I am at a loss (pun intended)</p>
<p>What to do next?</p>

ptrblck: <p>Your current code in the repo was updated, right?<br>
At least that’s not the local copy I have.<br>
Currently you are accumulating the gradients in the <a href="https://github.com/skiddles/PyTorch.WordEmbeddings/blob/master/PyTorch_WordEmbedding_BeigeBook.py#L62">train function</a>.</p>
<p>I’ll pull again and try to run your code. The last time I still had size mismatch errors.</p>

skiddles: <p>There are minor differences between the repo and the test I ran saving the optimizer and reloading it.</p>
<p>Let me know if it does not work for you.</p>