Reproducibility

username: Dicko
cooked: <p>Hi folks, I am struggling with getting repeatable result for a DETR model I am running with pytorch.<br>
I have defined the following function:<br>
import os<br>
from numpy import random<br>
import numpy as np<br>
def seed_torch(seed=42):<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed) # if you are using multi-GPU.<br>
torch.backends.cudnn.benchmark = True<br>
torch.backends.cudnn.deterministic = True<br>
Please note that I have also tried this function with cudnn_benchmark set to false.<br>
This is the folder structure for DETR and the py files contained within it.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/262808d39384388a2d622a8b526b1886081a48c5.png" alt="image" data-base62-sha1="5rxUSixcZyvugKfdBleX1X0gpHn" width="602" height="153"></p>
<p>I have tried using the seed function in each file separately to see if anyone of them would make my results reproducible and this did not work.<br>
I then tried using the seed function in all files in each folder separately and this did not work.<br>
I tried using the seed function in the first three folders and the first epoch was repeatable but after the first epoch the numbers were different.<br>
The numbers I am talking about look like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/101bc8a83917314617a1f14d45d9a23faa09ed99.png" alt="image" data-base62-sha1="2ivaipAxMmjdtpn1h8IWOZwwG8x" width="602" height="93"></p>
<p>But I am comparing losses for now.<br>
In the end I copied the seed function to every py file and then used the function in every class and definition in the py files and the results are still not reproducible.<br>
Does anybody have any idea how to resolve this?<br>
I am running my code in the Anaconda terminal.</p>

username: googlebot
cooked: <p>try enabling <a href="https://pytorch.org/docs/master/generated/torch.set_deterministic.html#torch.set_deterministic" class="inline-onebox" rel="noopener nofollow ugc">torch.set_deterministic — PyTorch master documentation</a>, and read its doc</p>
<p>it may well fall with runtime errors, as some operations don’t have built-in deterministic implementations (on gpu); sometimes only ops used in backward() diverge, then only the first batch has reproducible outputs. if whole first epoch is reproducible, maybe problem is with dataloader’s workers.</p>

username: Dicko
cooked: <p>Thank you for your reply but I have went through each py file and underneath every line of code I have used the seed function . I still do not get reproducible results. All files have been seeded, every line seeded and still not reproducible.</p>

username: googlebot
cooked: <p>your problem is likely not related to random number generation, so setting seeds makes no sense. more likely culplit is:</p>
<aside class="quote no-group" data-username="googlebot" data-post="2" data-topic="108080">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>some operations don’t have built-in deterministic implementations (on gpu)</p>
</blockquote>
</aside>