Pytorch Model Results

username: Andre_Amaral_IST
cooked: <p>Hey,<br>
How can I make the model train be the same if the hyperparameters are the same?<br>
What is the seeds code that I need to put?</p>
<p>Regards,<br>
André</p>

username: Andrei_Cristea
cooked: <p>Take a look <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">here</a>.</p>

username: Andre_Amaral_IST
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea4ca91ae013e534ece200dfbcdd658810fd10d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea4ca91ae013e534ece200dfbcdd658810fd10d6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea4ca91ae013e534ece200dfbcdd658810fd10d6.png" alt="image" data-base62-sha1="xqHYiDER1ODnckxMIYtqN64ch94" width="690" height="371" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">706×380 7.08 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here it sets the seed for the data loader using torch.Generator(), does it make the batches equal from run to run?</p>
<p>The seed_worker keeps the reproducibility of what?</p>

username: Andrei_Cristea
cooked: <p>Yes, per the docs, you need both the <em>generator</em> and <code>worker_init_fn=seed_worker</code> to make the behavior of the DataLoader the same from run to run. <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading" rel="noopener nofollow ugc">This</a> suggests that the generator is PyTorch’s RNG, and the code inside the <code>seed_worker</code> function is used to fetch this seed and pass it to other libraries (like numpy and random) to ensure consistency.</p>

username: Andre_Amaral_IST
cooked: <p>To make my code give the same result run to run i am doing the following:</p>
<pre><code class="lang-auto"># Seed
seed = 42
torch.manual_seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

def seed_worker(worker_id):
    np.random.seed(42)
</code></pre>
<p>And in the DataLoader I am doing the following:</p>
<pre><code class="lang-auto">dl = DataLoader(ds, batch_size=BatchSize, shuffle=False, num_workers= 8, pin_memory=True, drop_last=True, generator=g, worker_init_fn= seed_worker)
</code></pre>
<p>Despite this, my code when I run it again with the same parameters it gives different results?</p>
<p><strong>Note:</strong> I initialize the LSTM cell and hidden states only in the <em>init</em> method of my LSTM class. So when I say I make a new run it implies creating the model object model = LSTM(args).</p>

username: Andre_Amaral_IST
cooked: <p>Another question that naturally comes is: What is the best seed number?</p>

username: Andrei_Cristea
cooked: <p>Try this as well:</p>
<pre><code class="lang-auto">torch.use_deterministic_algorithms(True)
</code></pre>
<p>Where are you running all those commands? Presumably in the main body of the program, not inside the LSTM class initialization, right?</p>
<p>Can you please include all the stuff inside <code>seed_worker</code> per the tutorial? I don’t know if <code>np.random.seed</code> is enough because maybe internally some of the libraries call <code>random</code> rather than <code>numpy.random</code> so you may need the <code>random.seed(worker_seed)</code> part as well.</p>
<blockquote>
<p>Another question that naturally comes is: What is the best seed number?</p>
</blockquote>
<p>This question is closely related to “what is the best lottery number to play?” In both cases, the answer is that it ought to be random and unpredictable.</p>

username: Andre_Amaral_IST
cooked: <blockquote>
<p>Where are you running all those commands? Presumably in the main body of the program, not inside the LSTM class initialization, right?</p>
</blockquote>
<p>The seeds I showed there I am running in the program body, not inside the LSTM class.</p>

username: Andre_Amaral_IST
cooked: <aside class="quote no-group" data-username="Andrei_Cristea" data-post="7" data-topic="151454">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p><code>torch.use_deterministic_algorithms(True)</code></p>
</blockquote>
</aside>
<p>When i put the “torch.use_deterministic_algorithms(True)” i got the following error:</p>
<p><code>RuntimeError: Deterministic behavior was enabled with either </code>torch.use_deterministic_algorithms(True)<code>or</code>at::Context::setDeterministicAlgorithms(true)<code>, but this operation is not deterministic because it uses CuBLAS and you have CUDA &gt;= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility</code></p>

username: Andrei_Cristea
cooked: <p>No solutions here but I did Google that error and found several threads on it on the forum, those might be helpful.</p>

username: Andre_Amaral_IST
cooked: <pre><code class="lang-auto">os.environ["CUBLAS_WORKSPACE_CONFIG"]=":4096:8"
torch.use_deterministic_algorithms(True)
</code></pre>
<p>I am doing this at the beginning of the code. The solution i found out <a href="https://stackoverflow.com/questions/66682127/problem-with-determinism-set-a-buffer-size-in-the-cublas-workspace-config-envi" rel="noopener nofollow ugc">here</a>. I will tell if it works or not.</p>
<p><strong>Edit:</strong> It works and solves the problem!</p>

username: Andre_Amaral_IST
cooked: <p>It solves the error, however the code still not reproducible do you have more suggestions <a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> ?</p>