Problems about training accuracy and model reproducibility

urania880519_Li: <p>I have been working on a classifying problem using a model similar to DETR. Recently I found my model unstable and random seed has been set. During different times of training with the same hyperparameters and same epochs, the model behaved so differently that I think there must be problems in my code. The major problems are as follows:</p>
<ol>
<li>
<p>bad reproducibility (totally different trend regarding the accuracy just like the picture below, the y axis represents the times of training epochs and the x axis represents accuracy.)<br>
orange line: training accuracy during the 1st time of training for 100 epochs<br>
dark blue: validation accuracy during the 1st time of training for 100 epochs<br>
light blue: training accuracy during the 2nd time of training for 100 epochs<br>
pink: validation accuracy during the 2nd time of training for 100 epochs<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd_2_690x399.png" alt="Untitled" data-base62-sha1="4wpREu1q4JxOMi6dzQ2lnVxmst7" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd_2_690x399.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd_2_1035x598.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb2cb317bc95cbc2172dce59e8ef19cb2d84cfd.png 2x" data-dominant-color="C1C0C1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">1199×694 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>high accuracy at the second or more time of training (using the same parameters, and I always restart the kernel before starting a new round of training)</p>
</li>
</ol>
<p>Does anyone have similar experiences or know what kind of problems might be the cause?? I’m quite new to machine learning and deep learning, so I think I might have missed out some important tips when trying to build a model using Pytorch. Thnaks!!</p>

urania880519_Li: <p>The pic about the second problem(A new member in this forum can only upload one attachment to the post and I have to put it here):<br>
yellow line: training accuracy during the 1st time of training for 50 epochs<br>
purple: validation accuracy during the 1st time of training for 50 epochs<br>
green: training accuracy during the 2nd time of training for 100 epochs<br>
orange: validation accuracy during the 2nd time of training for 100 epochs<br>
The green and orange line can be found weird with the high accuracy since the first epoch and the performance somehow make me question about whether the trained weights of the first time training has been remembered…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14c23f94af05d4e2dd405a96c05d6a0fa248c017.png" data-download-href="https://discuss.pytorch.org/uploads/default/14c23f94af05d4e2dd405a96c05d6a0fa248c017" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14c23f94af05d4e2dd405a96c05d6a0fa248c017_2_689x383.png" alt="image" data-base62-sha1="2XDIVnWZFsAYhEizdNpCjQXSkDl" width="689" height="383" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14c23f94af05d4e2dd405a96c05d6a0fa248c017_2_689x383.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14c23f94af05d4e2dd405a96c05d6a0fa248c017_2_1033x574.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/14c23f94af05d4e2dd405a96c05d6a0fa248c017.png 2x" data-dominant-color="C0BDBB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1209×672 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>