Different behaviour when min/max reduce over all vs dim

tuero: <p>I’m trying to figure out what the correct behaviour should be when computing the gradient over a min/max all-reduce vs reducing over a dim. This example is taken from this issue, which appears to have been closed: <a href="https://github.com/pytorch/pytorch/issues/35699" class="inline-onebox" rel="noopener nofollow ugc">backward for tensor.min() and tensor.min(dim=0) behaves differently · Issue #35699 · pytorch/pytorch · GitHub</a></p>
<p>The below example in theory does the same thing, but gives different gradients.</p>
<ol>
<li>If multiple elements in the input have the minimal value, should gradients flow back through both or a single one</li>
<li>If gradients should only flow through one of the min elements, how do we determine which? (EDIT: The docs here say that the first element will be selected: <a href="https://pytorch.org/docs/stable/generated/torch.min.html#torch.min" class="inline-onebox" rel="noopener nofollow ugc">torch.min — PyTorch 2.6 documentation</a>)</li>
</ol>
<p>In this example, the gradient is split between both min elements in an all-reduce, but only one (non-determinstically?) is selected when reducing over a dim.</p>
<pre><code class="lang-auto">a = torch.tensor([0.1, 0.3, 0.1], dtype=torch.float32, requires_grad = True)
a_cp  = torch.tensor([0.1, 0.3, 0.1], dtype=torch.float32, requires_grad = True)
b = a.min()
b.backward()
print(a.grad)  # Output is tensor([0.5000, 0.0000, 0.5000])
c, d = a_cp.min(dim=0)
c.backward()
print(a_cp.grad)  # Output is tensor([1., 0., 0.])

</code></pre>
<p>Thanks!</p>

Avis682Holmes: <p>hello!<br>
This is quite a common issue, please suggest me if you find any solution for it.</p>
<p>Best Regards,<br>
Avis<br>
<a href="https://www.choice-advantages.com" rel="noopener nofollow ugc">choiceADVANTAGES com</a></p>

KFrank: <p>Hi Jake!</p>
<aside class="quote no-group" data-username="tuero" data-post="1" data-topic="216335" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tuero/48/45045_2.png" class="avatar"> tuero:</div>
<blockquote>
<p>This example is taken from this issue, which appears to have been closed: <a href="https://github.com/pytorch/pytorch/issues/35699" class="inline-onebox" rel="noopener nofollow ugc">backward for tensor.min() and tensor.min(dim=0) behaves differently · Issue #35699 · pytorch/pytorch · GitHub</a></p>
</blockquote>
</aside>
<p>I see the same behavior as you on pytorch version 2.6.0.  I do see this as a (minor)<br>
bug.</p>
<p>I read the github issue you linked to as follows</p>
<ol>
<li>
<p>This is a legitimate issue.</p>
</li>
<li>
<p>The <code>torch.min (dim = 0)</code> behavior is “correct.”</p>
</li>
<li>
<p>The issue was closed as “completed,” which presumably means fixed.</p>
</li>
</ol>
<p>I have no idea whether is was actually fixed, but if it was, the current behavior is a<br>
regression.  (Note, I haven’t tried to verify whether the current, “correct” <code>min (dim = 0)</code><br>
behavior is deterministic or not.)</p>
<aside class="quote no-group">
<blockquote>
<p>(EDIT: The docs here say that the first element will be selected: <a href="https://pytorch.org/docs/stable/generated/torch.min.html#torch.min" class="inline-onebox" rel="noopener nofollow ugc">torch.min — PyTorch 2.6 documentation</a>)</p>
</blockquote>
</aside>
<p>The <a href="https://pytorch.org/docs/2.6/generated/torch.min.html" rel="noopener nofollow ugc">pytorch 2.6 documention</a> seems to be garbled on this point.  Quoting:</p>
<blockquote>
<p>This function produces deterministic (sub)gradients unlike <code>min(dim=0)</code></p>
</blockquote>
<p>This agrees neither with the actual behavior  – subgradients are not produced for<br>
<code>min()</code> – nor with the desired behavior as indicated by the github issue – subgradients<br>
<em>are</em> produced for <code>min (dim = 0)</code>.</p>
<p>I agree with you that this is a real, if minor, bug and ideally it should be fixed.  At a<br>
minimum, the documentation should be brought into agreement with reality.</p>
<p>Best.</p>
<p>K. Frank</p>

tuero: <p>Thanks for your detailed reply! I haven’t actually come across a usecase where this behavior has caused me problems. I’m writing my own “torch lite” library and only noticed this when writing my own implementation, and what should be the intended behaviour was thus confusing.</p>
<p>As mentioned above, there is a github issue tracking this behaviour for those curious to follow!</p>

ngimel: <p>There is no regression and the issue was fixed correctly. For non-differentiable functions we return lowest norm subgradient, which, in case of <code>min</code> with no dimension, would mean sending part of the gradient to all the min elements. <code>min</code> with a dimension returns both an index of (one of the) minimum elements and its value, thus, due to semantics of this function, the gradient will have to go to the index returned by the operation. If you want the same behavior as <code>min</code> with no dimension, you should use <code>amin(dim=0)</code> which propagates gradient to all the minimum value elements.</p>

tuero: <p>This is a bit out of depth of understanding, but are you able to shed some light on why gradients would propoxate to single elements vs split to all matching? Is the split gradient the proper default when working with these reduction operations, but because semantically <code>.min</code> also ties along with a specific index, that restricts to where the gradient should flow back to?</p>

ngimel: <p>Correct, split gradient is minimum norm subgradient, which is what we generally want, but for <code>min</code> operation with index we should propagate gradient only to this single index.</p>

tuero: <p>Thanks for the reply!</p>

KFrank: <p>Hi <a class="mention" href="/u/ngimel">@ngimel</a>!</p>
<p>Thanks for the explanation.</p>
<aside class="quote no-group" data-username="ngimel" data-post="5" data-topic="216335" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/48/14766_2.png" class="avatar"> ngimel:</div>
<blockquote>
<p>There is no regression and the issue was fixed correctly.</p>
</blockquote>
</aside>
<p>This question is not to imply that a correct fix hasn’t been put into place.</p>
<p>Am I missing some other (more relevant) github issue?  My reading of the discussion<br>
in  <a href="https://github.com/pytorch/pytorch/issues/35699" rel="noopener nofollow ugc">github issue 35699</a> (which Jake cited) is pretty clearly that the <code>min (dim)</code> behavior<br>
(namely, that only one element gets the gradient) is correct and that <code>min()</code> would be<br>
made consistent with <code>min (dim)</code>.  I haven’t looked at the PRs, but the issue is then<br>
“closed as completed,” so pytorch’s current behavior is out of sync with that particular<br>
github discussion.</p>
<p>As for the documentation, I find it to be so terse as to be ambiguous.  I suppose it<br>
can be read as being consistent with pytorch’s current behavior, but that’s not<br>
very helpful.</p>
<aside class="quote no-group">
<blockquote>
<p>For non-differentiable functions we return lowest norm subgradient, which, in case of <code>min</code> with no dimension, would mean sending part of the gradient to all the min elements. <code>min</code> with a dimension returns both an index of (one of the) minimum elements and its value, thus, due to semantics of this function, the gradient will have to go to the index returned by the operation.</p>
</blockquote>
</aside>
<p>I believe I understand the logic – there are two competing concerns.  Because<br>
<code>min (dim)</code> returns an index, you want to populate just the gradient for that index.<br>
On the other hand, because <code>min()</code> does not return an index you prefer to spread<br>
the gradient across multiple elements that have the minimum value in order to get<br>
the norm-wise smallest subgradient (but at the cost of being inconsistent with<br>
<code>min (dim)</code>).</p>
<p>To me, consistency seems to be the more important consideration.  Can you give<br>
a real-world use case where I would want, or be surprised by not getting, the<br>
norm-wise minimum subgradient?  (As it stands, I am surprised by not getting the<br>
behavior of <code>min (dim)</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

ngimel: <p>In <span class="hashtag-raw">#35699</span> prior to the fix the gradient for all minimum position was 1, so it was not a subgradient and that was clearly wrong. It was fixed to return the minimum norm subgradient.<br>
As for consistency concern, the inconsistency is in the naming of 2 conceptually different functions - one returning indices and another not returning indices - as <code>min</code>. Unfortunately, we cannot change this due to bc concerns, and it would require a lot of churn in the users code for no practical gain. The real counterpart of <code>min()</code> with no dimension is <code>amin(dim=...)</code>, and the gradient behavior of these functions is consistent.<br>
For a discussion on the choice of minimum norm subgradient see <a href="https://github.com/pytorch/pytorch/issues/41779" class="inline-onebox" rel="noopener nofollow ugc">Wrong gradient for torch.norm(x, p=float('inf')) when input tensor has non-unique max values · Issue #41779 · pytorch/pytorch · GitHub</a></p>

tuero: <p>One last question. I noticed that in a <code>max_pool2d</code>, the gradient follows <code>max(dim=0)</code> where the entire gradient flows back to a single matching element. From the logic above, I would have assumed that the gradient should be <em>split</em> between all elements. From reading your linked discussion, it seems like the decision was made to follow the behaviour of the pooling ops, but why did the pooling ops decide on not splitting the gradient?</p>
<p>Consider the following:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.ones((1, 1, 2, 2), requires_grad=True)
out = torch.nn.functional.max_pool2d(x, kernel_size=2,stride=1, padding=0)
grad_out = torch.ones_like(out)
out.backward(grad_out)
print(x.grad)
</code></pre>
<p>The given result is</p>
<pre><code class="lang-auto">tensor([[[[1., 0.],
          [0., 0.]]]])
</code></pre>
<p>But I would have expected the gradient to be</p>
<pre><code class="lang-auto">tensor([[[[0.25, 0.25],
          [0.25, 0.25]]]])
</code></pre>

KFrank: <p>Hi <a class="mention" href="/u/ngimel">@ngimel</a>!</p>
<aside class="quote no-group" data-username="ngimel" data-post="10" data-topic="216335" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/48/14766_2.png" class="avatar"> ngimel:</div>
<blockquote>
<p>In <span class="hashtag-raw">#35699</span> prior to the fix the gradient for all minimum position was 1, so it was not a subgradient and that was clearly wrong.</p>
</blockquote>
</aside>
<p>Thank you for this explanation – this makes a lot more sense now.</p>
<p>There was a real bug (real in the sense of being mathematically wrong, above and<br>
beyond any consistency concerns).  This bug was fixed and the github issue was<br>
closed.</p>
<p>That the bug was fixed with the minimum-norm-subgradient approach rather than the<br>
consistency-with-<code>min (dim)</code> approach is a separate issue and the minimum-norm<br>
approach is certainly mathematically legitimate.</p>
<p>Best.</p>
<p>K. Frank</p>

ngimel: <p>Yeah max_pool behavior deviates from minimum norm subgradient, so it’s a bug, but tbh I don’t think there’s a lot of appetite for fixing it. It returns a subgradient, just not the minimum norm subgradient.</p>

tuero: <p>Thanks for confirming. I agree that its not a high priority issue but I wanted to make sure I understand what should the desirable behaviour be with respect to subgradients.</p>