Auto gradient computation

username: Kevinzhu
cooked: <p>I have a fundamental question here. Consider the objective function I have is <strong>F[G(b)]</strong>, where <strong>F</strong> is function of matrix <strong>G</strong>, that <strong>F[G] = trace(inv(G))</strong>. <strong>G(b)</strong> is a function that use vector <strong>b</strong> as input. Now I’m using autograd function to compute numerical gradient of <strong>F</strong> over <strong>b</strong>, which is  <strong>\Delta F / <br>
Delta b</strong>. In my configuration, vector <strong>b</strong> is double-type and objective function <strong>F</strong> generate result of a complex number, but the image part is extremely small that can be ignore (image part number is usually around 1e-20, which I think is computational error). Under this setting, when I get the numerical gradient of vector <strong>b</strong>, the gradient is a complex number. I would like to know if this is a possible situation or this is caused by some settings about autograd function.</p>
<p>Kevin</p>

username: KFrank
cooked: <p>Hi Kevin!</p>
<aside class="quote no-group" data-username="Kevinzhu" data-post="1" data-topic="144481" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/67e7ee/48.png" class="avatar"> Kevinzhu:</div>
<blockquote>
<p>In my configuration, vector <strong>b</strong> is double-type and objective function <strong>F</strong> generate result of a complex number, but the image part is extremely small that can be ignore</p>
</blockquote>
</aside>
<p>Could you post a short, self-contained, runnable script that illustrates<br>
your issue?</p>
<p>Where in the process does the real <strong>b</strong> get turned into a complex <strong>F</strong>?<br>
Is <strong>G</strong> already complex?</p>
<p>You mention that <strong>F</strong> is essentially real (very small imaginary part).<br>
Mathematically speaking, should <strong>F</strong> be purely real?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Kevinzhu
cooked: <p>The script I’m running is a MATLAB script that calls python function so it is a bit complicated to post it all. But the python function I’m running is here:</p>
<pre><code class="lang-auto">def F_Grad(Ch, CH, CW, F, b):
    Ch_Ten = torch.from_numpy(Ch).to(torch.complex128)
    b_Ten = torch.from_numpy(b).to(torch.complex128)
    b_Ten.requires_grad_()
    B_Ten = torch.diagflat(b_Ten).to(torch.complex128)
    F_Ten = torch.from_numpy(F)
    F_H_Ten = torch.transpose(torch.conj(F_Ten), 0, 1)
    A_Ten = torch.matmul(B_Ten, F_Ten)
    A_H_Ten = torch.transpose(torch.conj(A_Ten), 0, 1)
    CH_Ten = torch.from_numpy(CH).to(torch.complex128)
    CW_Ten = torch.from_numpy(CW).to(torch.complex128)
    Ch_Tilde = torch.diagflat(torch.diagonal(torch.matmul(A_H_Ten, torch.matmul(CH_Ten, A_Ten)), 0))
    G_Temp1 = torch.matmul(A_Ten, torch.matmul(Ch_Tilde, A_H_Ten))
    G_Temp2 = torch.inverse(torch.add(G_Temp1, CW_Ten))
    G = torch.matmul(Ch_Tilde, torch.matmul(A_H_Ten, G_Temp2))
    GA = torch.matmul(G, A_Ten)

    F_OBJ1_Temp1 = torch.sub(torch.matmul(F_Ten, GA), F_Ten, alpha=1)
    F_OBJ1_H_Temp1 = torch.transpose(torch.conj(F_OBJ1_Temp1), 0, 1)
    F_OBJ1_Final = torch.trace(torch.matmul(F_OBJ1_Temp1, torch.matmul(Ch_Ten, F_OBJ1_H_Temp1)))

    F_OBJ2_Temp1 = torch.matmul(F_Ten, G)
    F_OBJ2_H_Temp1 = torch.transpose(torch.conj(F_OBJ2_Temp1), 0, 1)
    F_OBJ2_Final = torch.trace(torch.matmul(F_OBJ2_Temp1, torch.matmul(CW_Ten, F_OBJ2_H_Temp1)))
    F_OBJ_tensor = F_OBJ1_Final + F_OBJ2_Final
    F_OBJ_tensor.backward()
    b_grad_tensor = b_Ten.grad
    b_grad = b_grad_tensor.numpy()
    F_OBJ = F_OBJ_tensor.detach().numpy()
    return [F_OBJ, b_grad]
</code></pre>
<p>There are five input variables. <strong>Ch</strong> is a diagonal real matrix, <strong>CH</strong> is a symmetric complex matrix, <strong>CW</strong> is a diagonal real matrix, <strong>F</strong> is DFT matrix, and <strong>b</strong> is a real vector.</p>
<p>Mathematically objective function should be purely real because the physical meaning of it is the mean square error.</p>

username: KFrank
cooked: <p>Hi Kevin!</p>
<aside class="quote no-group quote-modified" data-username="Kevinzhu" data-post="3" data-topic="144481" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/67e7ee/48.png" class="avatar"> Kevinzhu:</div>
<blockquote>
<pre><code class="lang-auto">def F_Grad(Ch, CH, CW, F, b):
    Ch_Ten = torch.from_numpy(Ch).to(torch.complex128)
    b_Ten = torch.from_numpy(b).to(torch.complex128)
    b_Ten.requires_grad_()
    ...
    b_grad = b_grad_tensor.numpy()
    F_OBJ = F_OBJ_tensor.detach().numpy()
    return [F_OBJ, b_grad]
</code></pre>
</blockquote>
</aside>
<p>Okay, that was opaque …</p>
<aside class="quote no-group">
<blockquote>
<p><strong>CH</strong> is a symmetric complex matrix,<br>
…<br>
<strong>F</strong> is DFT matrix,</p>
</blockquote>
</aside>
<p>Does “DFT” mean “discrete Fourier transform?”  Is <strong>F</strong> also complex?</p>
<aside class="quote no-group">
<blockquote>
<p>Mathematically objective function should be purely real because the physical meaning of it is the mean square error.</p>
</blockquote>
</aside>
<p>If the objective function is mathematically purely real, I would try to<br>
reorganize the calculation so that all the intermediate results are purely<br>
real, as well.  Doing so would prevent “mathematically zero” imaginary parts<br>
from creeping in through round-off error.</p>
<p>If you can do this, your gradient will remain real.</p>
<p>To do so, you might have to express complex objects such as <strong>CH</strong><br>
explicitly in terms of their real and imaginary parts (e.g., <strong>CH_Real</strong><br>
and <strong>CH_Imag</strong>) so that everything would be <code>torch.float64</code> and<br>
nothing would be <code>torch.complex128</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Kevinzhu
cooked: <p>Yes, <strong>F</strong> is discrete fourier transform matrix. Thanks for your suggestion by the way. I would also like to know from the top-level concept, is it true that as long as the objective function result is strictly real, the gradient of the function over a real vector should be real as well?</p>
<p>Best</p>
<p>Kevin</p>