Can't achive reproducability / determinism in pytorch training

shorty: <p>I am implementing a active machine learning object detection pipeline with pytorch inside a jupyter notebook. I am using fasterRCNN, COCO annotations, SGD optimizer and GPU training.</p>
<p>To ensure determinism i try to run one epoch of training two times and receive different losses by the end of both. The loss after the first step is always the same, so initialization is not the problem.</p>
<p>What i already tried:</p>
<ul>
<li>i made sure the order of images fed are in the same order</li>
<li>jupyter kernel restartet between training runs</li>
<li>batch_size = 1, num_workers = 1, disabled augmentation</li>
<li>CPU training is deterministic(!)</li>
<li>the following seeds are set:<br>
– seed_number = 2<br>
– torch.backends.cudnn.deterministic = True<br>
– torch.backends.cudnn.benchmark = False<br>
– random.seed(seed_number)<br>
– torch.manual_seed(seed_number)<br>
– torch.cuda.manual_seed(seed_number)<br>
– np.random.seed(seed_number)<br>
– os.environ[‘PYTHONHASHSEED’]=str(seed_number)</li>
</ul>
<p>Here is a link to the primary code for the training:<br>
<a href="https://colab.research.google.com/drive/1fPvTBt_fXE7-vXKumh1tOURT8x_RDgak" rel="noopener nofollow ugc">my code inside a colab</a><br>
(its not functional, since i just copied it out of my local jupyter but shows what i am trying to do)</p>
<p>The training log for two identical runs look like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/008cb9551fd9d81d617519b7ce6026c0eb292f10.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/008cb9551fd9d81d617519b7ce6026c0eb292f10" title="training_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/008cb9551fd9d81d617519b7ce6026c0eb292f10_2_690x101.jpeg" alt="training_loss" data-base62-sha1="4RuYMxMeiFKpXUYb9xKqgmp9fO" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/008cb9551fd9d81d617519b7ce6026c0eb292f10_2_690x101.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/008cb9551fd9d81d617519b7ce6026c0eb292f10_2_1035x151.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/008cb9551fd9d81d617519b7ce6026c0eb292f10_2_1380x202.jpeg 2x" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training_loss</span><span class="informations">1578×231 73.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Please let me know if there is any additional information needed <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

albanD: <p>Hi,</p>
<p>Did you had a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility notes</a> ?</p>

shorty: <p>Yes i did all the measurements suggested in the notes.<br>
Although i have heard that “the seeds do not behave globally”.<br>
Do have any information on this or other ideas i could try to achieve determinism?</p>

albanD: <p>In my experience, it is very very hard.<br>
Also won’t be reproducible as soon as you update any library/hardware/your code. So usually not very useful.</p>
<blockquote>
<p>Although i have heard that “the seeds do not behave globally”.</p>
</blockquote>
<p>Not sure what that means. If you use a single process it will work as expected.</p>

shorty: <p>Hmm okay.<br>
So in your experience it is impossible to achieve true determinism on pytorch gpu training?</p>

albanD: <p>Across hardware and library versions no.<br>
Unfortunately, floating point operations are not associative. So if any library changes the order of a single OP, then the whole thing breaks.<br>
Or if you GPU has a different number of processing units and so split the work differently.<br>
etc</p>
<p>That being said, for a fixed hardware and version, we try to be deterministic.<br>
I’m just saying this so that you don’t spend several days to get things reproducible on your machine but then realize it doesn’t work when you switch machine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Knowing that, if you still want to get reproducibility for that hardware and version, I would track down the operations that are listed as non-deterministic in the reproducibility note (maxpool I’m looking at you).</p>

shorty: <p>That sounds better already <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I am developing on my server a pipeline for my master thesis so the setup will stay the very same until i finish my experiments for the thesis.<br>
Therefore i am ready to invest into determinism a bit.</p>

Ecem_sogancioglu: <p>Hi,</p>
<p>I am having the same issue. I cannot get reproducible results training the FasterRCNN model in PyTorch. I have followed everything in <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">REPRODUCIBILITY doc</a>.</p>
<p>I set the seed at the beginning of my code as follows:</p>
<pre><code class="lang-auto">g = torch.Generator()
g.manual_seed(10)
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)
def set_seed(seed):
    os.environ['PYTHONHASHSEED'] = str(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.enabled = False
    #torch.use_deterministic_algorithms(True)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
set_seed(10)
</code></pre>
<p>And I have disabled the augmentation and set the number of workers as 0 in data loader.</p>
<pre><code class="lang-auto">data_loader = torch.utils.data.DataLoader(
            dataset, batch_size=2, shuffle=True, num_workers=0,
            collate_fn=utils.collate_fn, worker_init_fn=seed_worker, generator=g)
</code></pre>
<p>Model is created as follows:</p>
<pre><code class="lang-auto">self.model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
num_classes = 2  # 1 class (nodule) + background
in_features = self.model.roi_heads.box_predictor.cls_score.in_features
self.model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
self.model.to(self.device)
</code></pre>
<p>I use cuda 10.2 with torch version 1.10.0, so I have also set the environmental variable CUBLAS_WORKSPACE_CONFIG as mentioned in the tutorial. Do you have an idea why the results are not reproducible - and any suggestions of what could I try?</p>
<p>Also, if i set torch.use_deterministic_algorithms(True) after setting the CUBLAS_WORKSPACE_CONFIG, i get the following error:</p>
<p>RuntimeError: linearIndex.numel()<em>sliceSize</em>nElemBefore == value.numel()INTERNAL ASSERT FAILED at “/pytorch/aten/src/ATen/native/cuda/Indexing.cu”:250, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor10231</p>
<p>Many thanks, Ecem</p>

Ecem_sogancioglu: <p>I think the issue I mentioned was fixed recently in pytorch, I have installed the latest PyTorch version from the nightly build: ‘1.11.0.dev20211130+cu102’, and I do not get the following error anymore:</p>
<p>RuntimeError: linearIndex.numel()<em>sliceSize</em> nElemBefore == value.numel()INTERNAL ASSERT FAILED at “/pytorch/aten/src/ATen/native/cuda/Indexing.cu”:250, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor10231</p>
<p>Now I can set torch.use_deterministic_algorithms(True) in my code, but the results are still not reproducible.</p>
<p>Thanks!</p>