Determinism in inference

username: ck92
cooked: <p>I have the following code snippet. When I run the script consecutively, I get different results in prediction. Is there a setting I’m missing?</p>
<pre><code class="lang-auto">import torch
import numpy as np
import random

# Set the seed for all random number generators
seed = 42
torch.manual_seed(seed)
random.seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

# Ensure deterministic behavior
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

# Set CUBLAS workspace config to ensure deterministic cuBLAS
import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

torch.use_deterministic_algorithms(True)
torch.set_num_threads(1)

# Load the model
model = torch.jit.load('test.pt')
model = model.to(torch.device('cuda:0'))
model = model.half()

# Create a random array
torch.manual_seed(0)
random_arr = torch.randn(1,1,224, 224, 224, dtype = torch.half).to(torch.device('cuda:0'))
pred = model(random_arr).to(torch.device('cpu')).detach().numpy()
</code></pre>
<p>The <code>pred</code> variable is different in each run, max. difference is about 3e-2. Any help would be appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

username: ck92
cooked: <p>Thanks for your response. Here are two scripts, please save them as script_1.py and script_2.py in the working directory. For this experiment, I’m using a model from huggingface, but the results are similar with my own segmentation model (nnU-Net).</p>
<pre><code class="lang-auto"># Save this as script_1.py

import torch
import numpy as np
import random
import requests
import pickle
import argparse

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument("--result_filename", help="location to save the result prediction")

    args = parser.parse_args()
    
    # Set the seed for all random number generators
    seed = 42
    torch.manual_seed(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    # Ensure deterministic behavior
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

    # Set CUBLAS workspace config to ensure deterministic cuBLAS
    import os
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

    torch.use_deterministic_algorithms(True)
    torch.set_num_threads(1)

    # Download the file
    url = 'https://huggingface.co/spaces/Xajimel/Practica3/resolve/main/unet.pth'
    response = requests.get(url)
    with open('model.pt', 'wb') as f:
        f.write(response.content)

    model = torch.jit.load('model.pt')
    model = model.to(torch.device('cuda:0'))
    model = model.half()

    # Create a random array
    torch.manual_seed(0)
    random_arr = torch.randn(1, 3, 224, 224, dtype = torch.half).to(torch.device('cuda:0'))
    pred = model(random_arr).to(torch.device('cpu')).detach().numpy()

    with open(args.result_filename, 'wb') as f:
        pickle.dump(pred, f)
</code></pre>
<pre><code class="lang-auto"># Save this as script_2.py

import pickle
import numpy as np

with open('pred1.pkl', 'rb') as f:
    a = pickle.load(f)

with open('pred2.pkl', 'rb') as f:
    b = pickle.load(f)

print('Length of a: ', len(a))
print('Length of b: ', len(b))

print('Shape of first element in a: ', a[0].shape)
print('Shape of first element in b: ', b[0].shape)

diff = a[0] - b[0]

print('Min diff: ', np.min(diff))
print('Max diff: ', np.max(diff))
</code></pre>
<p>Run this with the following commands.</p>
<pre><code class="lang-auto">python script_1.py --result_filename pred1.pkl
python script_1.py --result_filename pred2.pkl
python script_2.py
</code></pre>
<p>Something I found interesting was that if I change the <code>url</code> in <code>script_1.py</code> to a different model (say <a href="https://huggingface.co/spaces/jannatulferdaws/amrin_gradio_grapevine/resolve/main/models/mobilenet.pt" rel="noopener nofollow ugc">this one</a>), I see no differences in the prediction.</p>
<p>I wonder if the issue then is non-determinism in some layers in the nnU-Net model or the way I’m saving the model. This is how I save the model.</p>
<pre><code class="lang-auto">self.network = self.network.to(self.device)
self.network.eval()
torch.use_deterministic_algorithms(True)
traced_model = torch.jit.trace(self.network, torch.randn(1, 1, 224, 224, 224).to(self.device))
torch.jit.save(traced_model, 'model.pt')
</code></pre>
<p>Let me know what you think or if you need some more info. Thanks!</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and see:</p>
<pre data-code-wrap="python"><code class="lang-python">Length of a:  1
Length of b:  1
Shape of first element in a:  (5, 224, 224)
Shape of first element in b:  (5, 224, 224)
Min diff:  0.0
Max diff:  0.0
</code></pre>

username: ck92
cooked: <p>This is what I see. I’m on <code>torch 2.3.0+cu121</code> if that helps.</p>
<pre><code class="lang-auto">Length of a:  1
Length of b:  1
Shape of first element in a:  (5, 224, 224)
Shape of first element in b:  (5, 224, 224)
Min diff:  -0.01563
Max diff:  0.01172
</code></pre>

username: ck92
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> On the machine that I was initially testing on, I see the issue on torch <code>2.3.0+cu121</code> and torch <code>2.4.0+cu121</code>. On another machine, I don’t see the issue in either versions. These machines have different GPUs (earlier: A40, now: RTX 3050 Ti).</p>
<p>Could the underlying GPU make a difference?</p>

username: chris.rapson
cooked: <p>The problem with reproducibility for nnunet has been observed before</p><aside class="onebox githubissue" data-onebox-src="https://github.com/MIC-DKFZ/nnUNet/issues/1016">
  <header class="source">

      <a href="https://github.com/MIC-DKFZ/nnUNet/issues/1016" target="_blank" rel="noopener nofollow ugc">github.com/MIC-DKFZ/nnUNet</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue" data-github-private-repo="false">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/MIC-DKFZ/nnUNet/issues/1016" target="_blank" rel="noopener nofollow ugc">Reproducibility of prediction result</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-04-08" data-time="02:24:02" data-timezone="UTC">02:24AM - 08 Apr 22 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2023-09-22" data-time="08:19:56" data-timezone="UTC">08:19AM - 22 Sep 23 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/kevinkwshin" target="_blank" rel="noopener nofollow ugc">
          <img alt="" src="https://avatars.githubusercontent.com/u/38489569?v=4" class="onebox-avatar-inline" width="20" height="20">
          kevinkwshin
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          stale
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hello, Febian. Thanks for your sharing your great code.

I had an issue on rep<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">roducibility of prediction result.
I tried 3d_fullres inference with I use same weight without ensemble, however, the results were slightly different.
In my opinion, TTA may cause this problem. Is there something I can do? (off the do_tta or fix seed?)

Regards,
Keewon Shin.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>At this stage, I am not aware of a solution.</p>