Pytorch cuda : training not working at all with some random seed

Jean-Eric_Campagne: <p>Hello!<br>
Here is my package version<br>
Python 3.6.5<br>
Torch 1.3.0</p>
<p>Here is a simple schema of my code focusing on seed settings</p>
<pre><code>random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.manual_seed_all(args.seed)
</code></pre>
<p>Loop on epochs<br>
train<br>
Test</p>
<p>The problem is that with seed 1 the result are Ok and consistent if I repeat it many times<br>
for seed=2 the results are consistent from one run to another but are absolutly NOT OK at all (it is not just a matter of percent difference)</p>
<ol>
<li>have you experienced this kind of behavior</li>
<li>does the seed should respect some convention/range/values ? when I was starting research 30yrs ago with FORTRAN code the random seeds usuable were special, and if one did respect the rules than the results were completky crazy. Is it the case here?</li>
</ol>
<p>Best.</p>

albanD: <p>Hi,</p>
<p>No the random seeds can be anything in pytorch. It will just change where the RNG starts.</p>
<p>It is possible that some network training is not very stable and will work for one random seed but not the other.<br>
Also you might want to check the notes on reproducibility in the doc <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">here</a> to see if runs with the same seed will give the same result.</p>

Jean-Eric_Campagne: <p>Thanks <span class="mention">@alabanD</span> I have read the doc you mention before posting. It is correct that fixing all the random seed and determinist variables, give the same result. But my point is that switching seed 1 to 2, destroy completly the generalisation power. So, you point that some network training are unstable. Do you have any reference on this subject? thanks.</p>

KFrank: <p>Hi Jean-Eric!</p>
<aside class="quote no-group" data-username="Jean-Eric_Campagne" data-post="3" data-topic="59827" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jean-eric_campagne/48/12291_2.png" class="avatar"> Jean-Eric_Campagne:</div>
<blockquote>
<p>But my point is that switching seed 1 to 2, destroy completly the generalisation power. So, you point that some network training are unstable.</p>
</blockquote>
</aside>
<p>You might also try seeding the random-number generators with<br>
3, 4, and 5 (and <em>just in case</em> there is something funny about how<br>
the seeds work – which I strongly doubt – with 314158, 314159,<br>
and 314160, as well).</p>
<p>Can you characterize the goodness of your results with a single<br>
statistic?  It would be interesting to see that statistic for a number<br>
of different seeds.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Jean-Eric_Campagne: <p>Hi! Let me elaborate a liitle bit more my “goodness” probe:<br>
As input of my network I have images in tensor dimension C,H,W with for instance C=5, H=W=64;  the output is a vector of nbins length which reprensent the p.d.f of some variable ‘y’: p(y). I have train &amp; test set with y_true which is a float vector in [0,1]. My favorite variables are statistical summary of the distribution dy = (y_predic-y_true)/(1+y_true)   with y_predict is the weighted mean using the probability p(y) and the location of bins center value. So, my statistical summary is the sigma ~ 1 and the fraction of outliers (eta) defined as |y_predic - y_true| &gt; 5.</p>
<p>Now comparing different seed run: example<br>
seed=0 sigma=11, eta=1;    &lt;== OK<br>
seed=1 ~ seed=0<br>
seed=2 sigma=15, eta = 11    &lt;===== Not OK</p>
<p>Now, I have used other seeds and some are ok, other no.<br>
So I was wandering if there are “good” seeds as it generrate a good random number generation unbiased, and some other seeds which produce biased random series.</p>
<p>best.</p>

ptrblck: <p>There are no “good” and “bad” seeds.<br>
As explained by others before, your model or whole training procedure might not be very stable, so that some random numbers yield to good results, while others make the model collapse.</p>
<p>You can mostly avoid these effects in supervised learning by using appropriate parameter initialization methods.<br>
If you are working on a new method and would like to write the results in a paper, I would recommend to use a few random seeds and report the number or successful runs as well as crashes.<br>
On the other hand if you are explicitly using a “good” seed, I would consider this as a hack, which makes it hard to draw any useful conclusions from your method.</p>

Jean-Eric_Campagne: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks a lot, but do not be affraid I am more concerned by systematics and I will not pretend to get better model than others. Now, all of you point that it is a sign of instability. Good, as I was tracking a bug <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Let me more elaborate the use case: I’m using an Inception like model and try to train it with Adam with eps=0.1. I’m using torch but in tensorflow doc for this minimizer it is explicitely stated that for Inception on should use eps=0.1 or 1 and not using the 10^-8 default value. By the way I have experimented with eps=0.01 and the results were completly instable.</p>
<p>May be you have nice advise: eg. an other minimizer (eg. SGD)…? notice that I have used also AdamW and concerning the seed behavior I have not got more stable seed free results.</p>

ptrblck: <p>Thanks for the information.<br>
I’ve found this in the TF docs:</p>
<blockquote>
<p>For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since AdamOptimizer uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the “epsilon” referred to here is “epsilon hat” in the paper.</p>
</blockquote>
<p>I didn’t have a chance to look into the formulas closely and I’m not sure, if the TF docs are applicable to the PyTorch optimizer.</p>
<p>On a general question: were you able to use the default Adam optimizer and get “stable” results?</p>

Jean-Eric_Campagne: <p>Do you mean with an other model?</p>

ptrblck: <p>No, I meant the same model.<br>
If you are using an inception-like model, did you use the same or similar init methods?</p>
<p>Also, you you using the auxiliary loss?</p>

Jean-Eric_Campagne: <p>Well to answer your question I realize that I certainly do not master every subtilities of ML <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Concerning the model Inception-like I mean it is a network with 1 conv layer followed by 5 inception cells followed by 3 fully connected layers. Each Inception cells gather 1x1, 3x3 and 5x5 eventually kernel convolutions processed in paralell. This is in fact the mode used by some colleagues: <a href="https://arxiv.org/pdf/1806.06607.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1806.06607.pdf</a>.<br>
So, I find some Adam settings that is ok cf. the train and test loss derease epoch after epoch and I a certain point they diverge ie, overfitting is arround, so I stop the learning phase. As I save intermediate state it is quite easy. Now, I do not know what is the auxiliairy loss you are refeering too.<br>
An other infomation I got today: with SGD I get the same behaviour.</p>