Potential bug in ADAM for complex parameters?

borisb: <p>In the documentation/code of ADAM <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html#Adam" rel="noopener nofollow ugc">here</a> I noticed something that seems like an error to me. Even though <code>torch.view_as_real</code> is called on <code>grad</code>, <code>grad.conj()</code> is called later for the computation of <code>exp_avg_sq</code>. Unless I misunderstood the code - this seems wrong to me. We want to compute the absolute value of the gradient here right?</p>
<p>Can anyone clarify if this is a bug? Here is the code in question:</p>
<pre><code class="lang-python">        if torch.is_complex(param):
            grad = torch.view_as_real(grad)
            exp_avg = torch.view_as_real(exp_avg)
            exp_avg_sq = torch.view_as_real(exp_avg_sq)
            if amsgrad:
                max_exp_avg_sqs[i] = torch.view_as_real(max_exp_avg_sqs[i])
            param = torch.view_as_real(param)

        # Decay the first and second moment running average coefficient
        exp_avg.lerp_(grad, 1 - beta1)
        exp_avg_sq.mul_(beta2).addcmul_(grad, grad.conj(), value=1 - beta2)
</code></pre>

KFrank: <p>Hi Boris!</p>
<aside class="quote no-group" data-username="borisb" data-post="1" data-topic="189912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/3bc359/48.png" class="avatar"> borisb:</div>
<blockquote>
<p>Even though <code>torch.view_as_real</code> is called on <code>grad</code>, <code>grad.conj()</code> is called later</p>
</blockquote>
</aside>
<p>I don’t really understand what is going on here, but here’s what I <em>think:</em></p>
<p>(Note, I’ve never used <code>Adam</code> with complex parameters.)</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        if torch.is_complex(param):
            grad = torch.view_as_real(grad)
</code></pre>
</blockquote>
</aside>
<p>Okay, if <code>param</code> is complex, we will <code>view_as_real()</code> its <code>grad</code>.  That seems<br>
reasonable, given what the rest of the code does.</p>
<p>However, if <code>param</code> is <em>not</em> complex, I don’t believe that <code>param.grad</code> can<br>
ever be complex.  In general, I don’t believe that pytorch permits this, and<br>
pytorch doesn’t permit calling <code>.backward()</code> on a complex loss, so you<br>
couldn’t use this as a scheme to generate a complex <code>grad</code> for a real <code>param</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        exp_avg_sq.mul_(beta2).addcmul_(grad, grad.conj(), value=1 - beta2)
</code></pre>
</blockquote>
</aside>
<p>Therefore, at this point, <em>whether or not the <code>is_complex (param)</code> <code>if</code> block were<br>
executed,</em> <code>grad</code> will be real, <code>grad.conj()</code> is a no-op, and you’re just performing<br>
a weighted averaging of <code>grad**2</code> into <code>exp_avg_sq</code>.</p>
<p>So it looks a bit odd, but I guess it’s correct.</p>
<p>Why might it be coded this way?  I suspect that it’s historical.  Complex autograd<br>
was designed, to some extent, on the fly.  The definition of a complex derivative<br>
wasn’t initially consistent across the various pieces of the implementation, and<br>
the question of differentiating a complex loss hadn’t been sorted out.  I suspect<br>
that the <code>grad.conj()</code> was an attempt to do the right thing in the presence of<br>
a complex <code>grad</code> (that I suspect could occur – rightly or wrongly – in some<br>
previous versions of complex autograd), and when the issue was addressed<br>
by other means, nobody noticed (or maybe cared) that <code>grad.conj()</code> was<br>
now a no-op, so it stayed in the code.</p>
<p>(But I could be wrong about all of this.  You might want to do the experiment<br>
of taking out the <code>.conj()</code> and see if anything breaks …)</p>
<p>Best.</p>
<p>K. Frank</p>

borisb: <p>Hi KFrank,</p>
<p>thank you for the detailed explanation. This would indeed explain things and it would make it a no-op. Was this documented? Because the alternative way of thinking about Adam for complex weights would be to not regard <code>real</code> and <code>imag</code> as disjoint parameters but as one parameter. In this case, one could have to estimate the variance via <code>grad * grad.conj()</code> as implemented but this is invalidated by casting it to complex.</p>
<p>In fact, I found this <a href="https://github.com/pytorch/pytorch/issues/59998" rel="noopener nofollow ugc">bug discussion here</a> which would indeed point to this being a temporary way of disabling it?</p>
<p>Best,<br>
Boris</p>

KFrank: <p>Hi Boris!</p>
<aside class="quote no-group" data-username="borisb" data-post="3" data-topic="189912" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/3bc359/48.png" class="avatar"> borisb:</div>
<blockquote>
<p>I found this <a href="https://github.com/pytorch/pytorch/issues/59998" rel="noopener nofollow ugc">bug discussion here</a></p>
</blockquote>
</aside>
<p>Good find!</p>
<p>Digging through some of the github links, it appears that <a href="https://github.com/pytorch/pytorch/pull/62946" rel="noopener nofollow ugc">pull request 62946</a><br>
initially implemented the fix with <code>grad.conj()</code> and then <a href="https://github.com/pytorch/pytorch/pull/80279" rel="noopener nofollow ugc">pull request 80279</a><br>
reimplemented the fix with <code>torch.view_as_real(grad)</code>, but left in the<br>
<code>grad.conj()</code>.</p>
<p>I’m still of the opinion that the <code>grad.conj()</code> is unnecessary (but harmless),<br>
so I guess that the <code>grad.conj()</code> is indeed a historical artifact.</p>
<p>Best.</p>
<p>K. Frank</p>

borisb: <p>Hi K. Frank,</p>
<p>while I agree that the <code>conj</code> call is unnecessary if the parameters are cast to real, what I wonder is, why is this done? Treating them as separate seems to affect the convergence and lead to subpar performance.</p>
<p>This post here illustrates the issue of treating complex parameters as 2d real ones: <a href="https://github.com/keras-team/tf-keras/issues/30" class="inline-onebox" rel="noopener nofollow ugc">Adam (and other) optimizers miscalculate the momentum update for complex variables · Issue #30 · keras-team/tf-keras · GitHub</a></p>
<p>Thanks and best,<br>
Boris</p>