Possible solution to the reproducibility issues using scatter_add operation

username: amosella
cooked: <p>As it is well known to all of us some pytorch operations are undeterministic such as scatter_add operation and some times it can be annoying have different results using the same hyperparameters in different runs. Recently I found this <a href="https://people.eecs.berkeley.edu/~hdnguyen/public/papers/ARITH21_Fast_Sum.pdf" rel="nofollow noopener">article</a> that describes a methodology to obtain a deterministic behaviour in Floating-Point Summation. I’m wondering if it can be useful implement this in pytorch. What do you think?</p>

username: tom
cooked: <p>I don’t think it’s the algorithms as much as the lack of people doing it.<br>
For example, for many operations (Upscaling comes up frequently) it would be relatively easy to make the cuda backward deterministic by having each <code>grad_in</code> location treated by one thread.<br>
For others, using a <code>index_add</code>/<code>scatter_add</code>-style operation, the <code>Embedding</code> backward may serve as a blueprint.</p>
<p>Best regards</p>
<p>Thomas</p>

username: amosella
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> thanks for your answer.</p>
<blockquote>
<p>I don’t think it’s the algorithms as much as the lack of people doing it.<br>
For example, for many operations (Upscaling comes up frequently) it would be relatively easy to make the cuda backward deterministic by having each  <code>grad_in</code>  location treated by one thread.</p>
</blockquote>
<p>I see what you mean. It is not a problem of knowledge it is more a human resources problem, right? However, in my opinion, should be a priority to have a deterministic mode for all the current algorithms. I know that there is a general idea that non-deterministic algorithms do not have a lot of influence on the results, but, in some cases, there is a huge impact. For instance, I started this thread because in pytorch_geometric some of the operations are based on non-deterministic implementations of pytorch. As you can see in this github <a href="https://github.com/rusty1s/pytorch_geometric/issues/446" rel="noopener nofollow ugc">issue</a> this behavior is impacting the results. At the end this non-deterministic behavior has an impact in other libraries like pytorch geometric, causing the lack of reproducibility.</p>
<blockquote>
<p>For others, using a  <code>index_add</code> / <code>scatter_add</code> -style operation, the  <code>Embedding</code>  backward may serve as a blueprint.</p>
</blockquote>
<p>Can you give me more details about this? I am not fluent in cuda but if I have a blueprint I can try to collaborate to have a deterministic mode in statter_add/index_add. I think the deterministic mode is a must to have.</p>

username: tom
cooked: <p>In <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Embedding.cu" rel="nofollow noopener"><code>aten/src/ATen/native/embedding/Embedding.cu</code></a>, <code>embedding_dense_backward_cuda</code> essentially computes and <code>index_add</code>  – the gradient for embedding matrix row i is the sum of all input gradient elements where i was embedded in the forward. (Disregard the <code>scale_grad_by_weight</code> and only consider the <code>false</code> case.)</p>
<p>Two variants are implemented</p>
<ul>
<li>For small batches (&lt; embedded786 elements in total) a direct variant is implemented in <code>embedding_backward_feature_kernel</code>. That is somewhat elaborate cuda with explicit communication amongst the threads to ensure determinism. If you didn’t do much cuda programming, this will be an opportunity to learn a lot, but at &lt;100 lines of code with ample comments it should be OK.</li>
<li>For large batches one falls back to sorting the indices (this is still done in <code>embedding_dense_backward_cuda</code>) and then hand off to <code>embedding_backward_cuda_kernel</code> in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/EmbeddingBackwardKernel.cu" rel="nofollow noopener"><code>EmbeddingBackwardKernel.cu</code></a>. Here again, you can assume <code>offset2bag</code> to be undefined. It’ll do quite a few steps to partition the work and then execute it, but it’s implemented as thrust calls and a few fairly simple kernels.</li>
</ul>
<p>So it takes a quite a bit of work to do this properly, but the ingredients are all there.</p>
<p>Best regards</p>
<p>Thomas</p>

username: yf225
cooked: <p><a class="mention" href="/u/amosella">@amosella</a> Would you like to post this on <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>, since this is not only related to libtorch? We would be able to prioritize general feature requests there based on community needs and available resources.</p>

username: amosella
cooked: <p><a class="mention" href="/u/yf225">@yf225</a> Thanks for your suggestion. I created this <a href="https://github.com/pytorch/pytorch/issues/23151" rel="nofollow noopener">issue</a>.</p>