Torch deterministic algorithms error

username: Nexkira
cooked: <p>Hi, when I am using torch.use_deterministic_algorithms(True) and I run this code, I got this error.<br>
sum_img are images coverted  into a torch tensor of dimension: batch_size x channel x img_size (128x128)</p>
<p>If I move sum_img to cpu instead of cuda, I have got no error</p>
<p>The code gives me error even if I put a number instead a list of number<br>
If I run this code with deterministic algorithms false run without a problem, how can I fix it? thank you in advice<br>
<br>
</p>
<pre><code class="lang-auto">list_n = [0.85,0.5,0.3] 
for i in range(3):
		channel = sum_img[:,i,:,:]
		channel[channel&gt;=5] = list_n[i]   




RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/native/cuda/Indexing.cu":253, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor2684801</code></pre>

username: ptrblck
cooked: <p>Could you post the output of <code>python -m torch.utils.collect_env</code> as well as the shapes for all tensors, please?<br>
Based on the error message it could be a valid error, which we would have to debug and fix (if it’s still observed in the current nightly release).</p>

username: Nexkira
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="125200">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>python -m torch.utils.collect_env</code></p>
</blockquote>
</aside>
<p>thank you for answering the output is:<br>
<br>
</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.9.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0
Libc version: glibc-2.26

Python version: 3.7 (64-bit runtime)
Python platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: GPU 0: Tesla P100-PCIE-16GB
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.9.0+cu102
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.10.0
[pip3] torchvision==0.10.0+cu102
[conda] Could not collect

</code></pre>
<p>the shape are:<br>
<br>
</p>
<pre><code class="lang-auto">sum_img = torch.Size([64, 3, 128, 128])
channel = torch.Size([64, 128, 128])</code></pre>

username: Nexkira
cooked: <p>Do you need more information?</p>

username: ssykiotis
cooked: <p>Hi, I am assuming that you want to make the output of your model deterministic. I had exactly the similar issue in my project and solved it using the following solution:</p>
<p>torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark     = False<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)</p>
<p>Avoid use_deterministic_algorithms(True). The above code solved the determinism issue for me. If you run your code on colab, a couple of additional lines are needed, let me know if that is the case so that I can provide it to you.</p>

username: Nexkira
cooked: <p>Yes, thank you, this fix my problem. I think torch.cuda.manual_seed and seed_all no need it cause torch.manual_seed should enable both cpu and gpu</p>

username: ptrblck
cooked: <p>It’s not sufficient to only seed the code and disable non-deterministic algorithms in cudnn as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a>, so the suggestion is not a solution.<br>
Do not avoid using <code>torch.use_deterministic_algorithms(True)</code>, as it’s the proper way to get a full deterministic code.</p>
<p>In any case, thanks for the additional information. We are tracking the issue <a href="https://github.com/pytorch/pytorch/issues/61032">here</a> and will fix it.</p>

username: Nexkira
cooked: <p>oh ok thank you for making clear, I’ll wait when the fix is ready</p>

username: Chuyang
cooked: <p>Thanks! Solved my issue too</p>