State_dict contains erroneous parameters

visionscaper: <p>Hello,</p>
<p>I have a strange issue when loading checkpoints from a long running training process. Before I create an issue on Github about this I wanted to know if anyone has encountered something like this, or that it maybe is a known issue?</p>
<p>I have a training process that has been running for about 22 days now (Pytorch 1.6, DistributedDataParallel using 4 GPUs, Pytorch native mixed precision training). I checked a saved checkpoint after about 8 days and after loading it, it worked well (inference results made sense). Now after 22 days, I loaded the most recent checkpoint and got inference results that made no sense.</p>
<p>So I started comparing the parameters in both checkpoints. What I found was that in the ‘faulty’ checkpoint the first two values of each parameter tensor are close to zero, while this is not the case in the good checkpoint. Please see the attached images.</p>
<p>Is any issue like this known to you?</p>
<p>Good checkpoint:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc" title="good-checkpoint-22112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc_2_687x500.png" alt="good-checkpoint-22112020" data-base62-sha1="5urdq3OWrEYVELe8mNlg2K5yjiQ" width="687" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc_2_687x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc_2_1030x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/267bb8c32dfe96a5d47e0e01aef27d54ffd5d4dc_2_1374x1000.png 2x" data-dominant-color="444649"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">good-checkpoint-22112020</span><span class="informations">2688×1954 892 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Faulty checkpoint:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66cb8770b651cc816087e33a4db430b7903669e8.png" data-download-href="https://discuss.pytorch.org/uploads/default/66cb8770b651cc816087e33a4db430b7903669e8" title="faulty-checkpoint-22112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66cb8770b651cc816087e33a4db430b7903669e8_2_689x499.png" alt="faulty-checkpoint-22112020" data-base62-sha1="eFmLbg3slPOQjWvVWIw1GNBvYYU" width="689" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66cb8770b651cc816087e33a4db430b7903669e8_2_689x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66cb8770b651cc816087e33a4db430b7903669e8_2_1033x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66cb8770b651cc816087e33a4db430b7903669e8_2_1378x998.png 2x" data-dominant-color="46474A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">faulty-checkpoint-22112020</span><span class="informations">2702×1958 939 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note: just to be clear, the faulty checkpoint was from much later in the training process and should have a much lower loss (as was calculated during training).</p>

ptrblck: <p>What was the training and validation accuracy of the “fauly” checkpoint before storing it?<br>
If I understand your issue correctly, you are seeing worse results when reloading the checkpoint than during training?</p>

visionscaper: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for responding. This is not an overfitting issue for sure, but to provide you some insight in to the model performance I added some of the tensorboard graphs at the end. Yes, the issue occurs when I load the checkpoint.</p>
<p>To provide some more context, this is an open-domain chatbot model. I trained these kinds of models using Pytorch many times before and I know the difference between when such a model is not trained well in some way, and when there is really something technically wrong. When it is not trained well (under-fitted or over-fitted), you can still make a conversation with it, but it makes not much sense, and/or you might see glitches in the text generation. In the case of the ‘faulty’ checkpoint the model simply outputs garbage, without any structure at all.</p>
<p>Further, it is very strange that all parameter Tensors, over all layers(!), start with the same kind of values (zero, or very close to zero). This can’t be right.</p>
<p>Some more details; <code>chat_gru</code> and <code>conversation_gru</code> are actually not GRUs, but Simple Recurring Units (SRUs), also see <a href="https://github.com/asappresearch/sru" class="inline-onebox" rel="noopener nofollow ugc">GitHub - asappresearch/sru: Training RNNs as Fast as CNNs (https://arxiv.org/abs/1709.02755)</a>.</p>
<p>Because I started my experiment using the SRUs for the first time, I thought maybe the issue is related to using the SRUs. But if that is the case, why would other layers (e.g. feed-forward layer <code>out</code>) also have the first two parameters close to zero? Could it be that on the C level, memory gets overwritten? (SRU has a C/CUDA implementation)</p>
<p>When I started the experiment, I also started using Pytorch 1.6.0, so maybe it is related to that, I can’t really tell.</p>
<p>Below are some screenshots of my Tensorboard for this experiment:</p>
<p>Moving average of cross entropy loss for training and validation set at earlier “good”/“functioning” checkpoint:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1893eecbfcdb65e877ecc863f3517a92570949c.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1893eecbfcdb65e877ecc863f3517a92570949c" title="good-checkpoint-performance-24112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1893eecbfcdb65e877ecc863f3517a92570949c_2_690x321.png" alt="good-checkpoint-performance-24112020" data-base62-sha1="n30V4rWCukpCJvuySeNO4BNQONC" width="690" height="321" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1893eecbfcdb65e877ecc863f3517a92570949c_2_690x321.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1893eecbfcdb65e877ecc863f3517a92570949c_2_1035x481.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1893eecbfcdb65e877ecc863f3517a92570949c_2_1380x642.png 2x" data-dominant-color="EBE9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">good-checkpoint-performance-24112020</span><span class="informations">2888×1346 525 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Moving average of cross entropy loss for training and validation set at later “faulty”/“garbage-out” checkpoint:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba" title="faulty-checkpoint-performance-24112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba_2_690x338.png" alt="faulty-checkpoint-performance-24112020" data-base62-sha1="cRMC70iWkoeoTg3KfeHLfjOobdE" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba_2_690x338.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba_2_1035x507.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2e81fb9b56248c103a24a588d77b6d8ba8d3ba_2_1380x676.png 2x" data-dominant-color="EBEAE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">faulty-checkpoint-performance-24112020</span><span class="informations">2932×1438 550 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Cross entropy over whole training set, calculated once per epoch:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0080a00957e81eb08cbbb763386e093675b97e3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/0080a00957e81eb08cbbb763386e093675b97e3e" title="epoch-training-cross-entropy-before-faulty-checkpoint-24112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0080a00957e81eb08cbbb763386e093675b97e3e_2_689x393.png" alt="epoch-training-cross-entropy-before-faulty-checkpoint-24112020" data-base62-sha1="4rzQcHwHpBwfxFyq37H5vyIhT8" width="689" height="393" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0080a00957e81eb08cbbb763386e093675b97e3e_2_689x393.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0080a00957e81eb08cbbb763386e093675b97e3e_2_1033x589.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/0080a00957e81eb08cbbb763386e093675b97e3e.png 2x" data-dominant-color="DADADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">epoch-training-cross-entropy-before-faulty-checkpoint-24112020</span><span class="informations">1206×688 44.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Cross entropy over whole validation set, calculated once per epoch:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9acac9e1c67803f5866f6189bc3647d82eb6e77c.png" data-download-href="https://discuss.pytorch.org/uploads/default/9acac9e1c67803f5866f6189bc3647d82eb6e77c" title="epoch-validation-cross-entropy-before-faulty-checkpoint-24112020"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9acac9e1c67803f5866f6189bc3647d82eb6e77c_2_690x389.png" alt="epoch-validation-cross-entropy-before-faulty-checkpoint-24112020" data-base62-sha1="m5m0bCD2ANiZvNHbU80fXNvY9ty" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9acac9e1c67803f5866f6189bc3647d82eb6e77c_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9acac9e1c67803f5866f6189bc3647d82eb6e77c_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9acac9e1c67803f5866f6189bc3647d82eb6e77c.png 2x" data-dominant-color="D6D6D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">epoch-validation-cross-entropy-before-faulty-checkpoint-24112020</span><span class="informations">1160×654 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Last note : I use a dropout of 0.2, this is why the training loss is higher then the validation loss.</p>

ptrblck: <p>Did you check the values in the good run and saw that they are not close to zero?<br>
I would start with a comparison between the “good” model after training vs. the model after reloading.<br>
Also, could you use the latest nightly binary, as I’ve isolated a checkpoint issue a few weeks ago (which should be fixed by now), which corrupted the loading of <code>state_dict</code>s if CUDATensors were stored.</p>

visionscaper: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<blockquote>
<p>Did you check the values in the good run and saw that they are not close to zero?</p>
</blockquote>
<p>Yeah, that’s is exactly what I did, as explained in my initial post. In the attached images of my first post you can see parameters of the ‘good’ checkpoint and the ‘faulty’ checkpoint. As you can see the parameter tensors in the good checkpoint do not start with values close to zero.</p>
<blockquote>
<p>Also, could you use the latest nightly binary, as I’ve isolated a checkpoint issue a few weeks ago (which should be fixed by now), which corrupted the loading of <code>state_dict</code> s if CUDATensors were stored.</p>
</blockquote>
<p>Will do! Is there a merge request and/or a related issue I can read about this issue?</p>

ptrblck: <p>It’s this one here: <a href="https://github.com/pytorch/pytorch/issues/46020">https://github.com/pytorch/pytorch/issues/46020</a></p>

visionscaper: <p>I see that this fix was merged in to <code>pytorch:release/1.7</code> on the 12th of October, the 1.7.0 release was on the 23rd of October. So, I can assume that your fix landed in Pytorch 1.7.0, correct?</p>

ptrblck: <p>Yes, that should be the case.</p>