How to implement a custom loss function which include frobenius norm?

username: Yuchen_Mu
cooked: <p>Hi,</p>
<p>I am using Pytorch to try to build and train a multilayer perceptron which is aimed at mapping a 450 dimension input to a 120 dimension output. Actually the data I have is in complex matrix, so I vectorize it and concatenate its real&amp;imaginary part together (and this is why my input and output are of this high dimension). I am now hoping to use a customized loss function which includes the matrix frobenius norm between the predicted results and the target.</p>
<p>I searched a lot on the forum but still not sure about how to implement this, since I am not sure whether pytorch will do the auto-gradient for it and whether I should also do something with the back-propagation of the customized loss function etc. At the same time, when dealing with this kind of complex matrix data, is there any better way to handle it? Or apart from MSE, is there any good loss fuction to deal with this kind of regression problem?</p>
<p>Thanks in advance!</p>

username: Yuchen_Mu
cooked: <p>Should I figure out how the derivative will affect each element and implement the backpropagation by hand?</p>

username: KFrank
cooked: <p>Hello Yuchen!</p>
<aside class="quote no-group" data-username="Yuchen_Mu" data-post="1" data-topic="104547" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ea5d25/48.png" class="avatar"> Yuchen_Mu:</div>
<blockquote>
<p>Actually the data I have is in complex matrix, so I vectorize it and concatenate its real&amp;imaginary part together</p>
</blockquote>
</aside>
<p>Complex tensors are still a work in progress in pytorch, with more<br>
functionality (and more functionality that is actually correct) being<br>
added in successive versions.</p>
<p>Note that as of version 1.6.0, I don’t believe that the pytorch optimizers<br>
accept complex <code>Parameter</code>s, so to use pytorch’s complex machinery,<br>
you will have to either use real <code>Parameter</code>s that you combine into<br>
complex tensors or write your own complex-aware optimizer.</p>
<p>All in all, depending on what you are doing, it might be safest to<br>
represent the real and imaginary parts of your complex tensors as<br>
separate real tensors and carry out the complex arithmetic “by hand.”</p>
<aside class="quote no-group">
<blockquote>
<p>I am now hoping to use a customized loss function which includes the matrix frobenius norm between the predicted results and the target.</p>
</blockquote>
</aside>
<p>The Frobenius norm of a (complex) matrix is simply the square root<br>
of the sum of the squares of the (absolute values of the) individual<br>
matrix elements.  Pythorch’s tensor operations can do this* reasonably<br>
straightforwardly.</p>
<p>*)  With the proviso that complex tensors are a work in progress.</p>
<p>Note that as of version 1.6.0, <code>torch.norm()</code> is incorrect for complex<br>
tensors – it uses the squares, rather than the squared absolute values,<br>
of the matrix elements.</p>
<p>Here is a script that illustrates calculating and backpropagating the<br>
Frobenius norm:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

_ = torch.random.manual_seed (2020)

x = torch.randn ([2, 3])
print ('x = ...\n', x)
print ('torch.norm (x) =', torch.norm (x))   # okay
z = torch.randn ([2, 3], dtype = torch.cfloat)
print ('z = ...\n', z)
print ('torch.norm (z) =', torch.norm (z))   # oops, should be positive real
z.requires_grad = True
znorm = torch.sqrt ((z * z.conj()).sum())
print ('znorm =', znorm)
znorm.backward()
print ('z.grad =', z.grad)
z.grad.zero_()
znormb = torch.sqrt ((torch.real (z)**2).sum() + (torch.imag (z)**2).sum())
print ('znormb =', znormb)
znormb.backward()
print ('z.grad =', z.grad)
</code></pre>
<p>And here is its (version 1.6.0) output:</p>
<pre><code class="lang-plaintext">x = ...
 tensor([[ 1.2372, -0.9604,  1.5415],
        [-0.4079,  0.8806,  0.0529]])
torch.norm (x) = tensor(2.4029)
z = ...
 tensor([[ 0.0531+0.3378j, -0.4779-1.5195j, -0.8105-0.1923j],
        [ 0.7118-0.0294j, -0.9088-0.3499j, -0.9167-0.8840j]])
torch.norm (z) = tensor(1.3644+1.4714j)
znorm = tensor(2.5350+0.j, grad_fn=&lt;SqrtBackward&gt;)
z.grad = tensor([[ 0.0210-0.1332j, -0.1885+0.5994j, -0.3197+0.0759j],
        [ 0.2808+0.0116j, -0.3585+0.1380j, -0.3616+0.3487j]])
znormb = tensor(2.5350, grad_fn=&lt;SqrtBackward&gt;)
z.grad = tensor([[ 0.0210-0.1332j, -0.1885+0.5994j, -0.3197+0.0759j],
        [ 0.2808+0.0116j, -0.3585+0.1380j, -0.3616+0.3487j]])
</code></pre>
<p>Be careful, however, with what you do with a complex gradient.  You<br>
will have to take the complex conjugate of the gradient to use it with<br>
gradient-descent optimization.</p>
<p>This script illustrates this behavior by minimizing the Frobenius norm<br>
with gradient descent:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

_ = torch.random.manual_seed (2020)

za = torch.randn ([2, 3], dtype = torch.cfloat)
zb = za.clone()

lr = 0.001

# gradient descent A
za.requires_grad = True
print ('za =', za)
for  i in range (10001):
    if  not za.grad == None:  _ = za.grad.zero_()
    znorm = torch.sqrt ((za * za.conj()).sum())
    znorm.backward()
    with torch.no_grad():
        _ = za.copy_ (za - lr * za.grad)   # doesn't converge
    if  i % 1000 == 0:  print ('znorm =', znorm)

print ('za =', za)

# gradient descent B
zb.requires_grad = True
print ('zb =', zb)
for  i in range (10001):
    if  not zb.grad == None:  _ = zb.grad.zero_()
    znorm = torch.sqrt ((zb * zb.conj()).sum())
    znorm.backward()
    with torch.no_grad():
        _ = zb.copy_ (zb - lr * zb.grad.conj())   # use conjugate of gradient to get convergence
    if  i % 1000 == 0:  print ('znorm =', znorm)

print ('zb =', zb)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">za = tensor([[ 0.8749-0.6791j,  1.0900-0.2884j,  0.6227+0.0374j],
        [ 0.0531+0.3378j, -0.4779-1.5195j, -0.8105-0.1923j]],
       requires_grad=True)
znorm = tensor(2.4970+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(2.8249+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(3.6008+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(4.5226+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(5.4901+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(6.4745+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(7.4661+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(8.4612+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(9.4581+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(10.4563+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(11.4551+0.j, grad_fn=&lt;SqrtBackward&gt;)
za = tensor([[ 0.1324-4.4859j,  0.1649-1.9052j,  0.0942+0.2472j],
        [ 0.0080+2.2312j, -0.0723-10.0381j, -0.1226-1.2704j]],
       requires_grad=True)
zb = tensor([[ 0.8749-0.6791j,  1.0900-0.2884j,  0.6227+0.0374j],
        [ 0.0531+0.3378j, -0.4779-1.5195j, -0.8105-0.1923j]],
       requires_grad=True)
znorm = tensor(2.4970+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(1.4970+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.4970+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
znorm = tensor(0.0010+0.j, grad_fn=&lt;SqrtBackward&gt;)
zb = tensor([[ 7.4477e-06-5.7811e-06j,  9.2789e-06-2.4552e-06j,
          5.3007e-06+3.1863e-07j],
        [ 4.5219e-07+2.8753e-06j, -4.0686e-06-1.2936e-05j,
         -6.9001e-06-1.6372e-06j]], requires_grad=True)
</code></pre>
<p>You can see that without taking its complex conjugate, the gradient<br>
pushes the imaginary parts of the tensor away from zero so that the<br>
Frobenius norm grows.</p>
<aside class="quote no-group">
<blockquote>
<p>I am not sure whether pytorch will do the auto-gradient for it and whether I should also do something with the back-propagation of the customized loss function etc. At the same time, when dealing with this kind of complex matrix data, is there any better way to handle it?</p>
</blockquote>
</aside>
<p>As long as you write your custom loss function using pytorch tensor<br>
operations, you will get autograd and backpropagation (but not<br>
complex optimization) “for free.”  You won’t have to write an explicit<br>
<code>.backward()</code> function for your loss function.</p>
<p>This will certainly be true if your represent the real and imaginary parts<br>
of your complex tensors as explicit real tensors.</p>
<p>However, I think its worth trying pytorch’s complex tensors, but if you<br>
decide to go this route, you should use the latest version of pytorch<br>
that otherwise works for you, and test your complex manipulations<br>
carefully, especially the various functions you use and backpropagation.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Yuchen_Mu
cooked: <p>Hi Frank,</p>
<p>Thank you very much for taking time to write this detailed reply!</p>
<p>Yes, I split the real and imaginary part of the original complex matrix then vectorize it. At the output side, I build a real matrix (which is the real part of the complex matrix) and a imaginart matrix (which is the imaginary part.) .</p>
<p>Specifically, I try to calculate the square of frobenius norm between the conjugate transpose of output and the target, here is what I did:</p>
<pre><code class="lang-auto">        # First matrix V_1_Hermitian(2*5)
        real_part_1 = output_from_network[i,0:5].float()
        real_part_2 = output_from_network[i,5:10].float()
        img_part_1 = -output_from_network[i,10:15].float()
        img_part_2 = -output_from_network[i,15:20].float()
        # A
        real_part_final = torch.stack((real_part_1,real_part_2),0)
        # B
        img_part_final = torch.stack((img_part_1,img_part_2),0)

        # Second matrix V_2(5*2)
        real_part_1_ = torch.reshape(target[i,0:5].float(), [5,1])
        real_part_2_ = torch.reshape(target[i,5:10].float(), [5,1])
        img_part_1_ = torch.reshape(target[i,10:15].float(), [5,1])
        img_part_2_ = torch.reshape(target[i,15:20].float(), [5,1])
        # C
        real_part_final_ = torch.cat((real_part_1_,real_part_2_),1)
        # D
        img_part_final_ = torch.cat((img_part_1_,img_part_2_),1)

        # M = V_1_Hemitian * V_2
        # M = Re(M) + Im(M)*j
        # Re(M) = A*C-B*D, Im(M) = A*D+B*C
        # frobenius norm square of M = frobenius norm square of Re(M) + frobenius norm square of Im(M)
        Re_M = torch.matmul(real_part_final, real_part_final_) - torch.matmul(img_part_final, img_part_final_)
        Im_M = torch.matmul(real_part_final, img_part_final_) + torch.matmul(img_part_final, real_part_final_)
        Re_M_norm_square = torch.pow(torch.norm(Re_M,'fro'),2)
        Im_M_norm_square = torch.pow(torch.norm(Im_M,'fro'),2)
        Final_M_norm_square = Re_M_norm_square + Im_M_norm_square
</code></pre>
<p>Sorry about the mess. the output and the target is a vector of size 20 (corresponding to a 5*2 matix), where first 10 elements are the real part and remaning 10 elements are the imaginary part. Since before I am not sure whether Pytorch can do the auto-gradient ‘for free’, I. to some extent, write the process of calculating this frobenius norm ‘by hand’ (as shown in the code, really messy <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> ), will this code work?</p>
<p>Actually I verified that the final output, which is <code>Final_M_norm_square</code>, is the value that I want (so the manipulation I did give the correct result as I want it to be - The frobenius norm square between two complex matrix), my concern is whether the gradients will be calculated precisely during the backpropagation, since instead of building the complex matrix from the real part and imaginary part and calculate the frobenius norm as you shown me in your script example, I just directly calculate it from the real part and imaginary part, respectively.</p>
<p>Again, thank you very much for taking time and look at my question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hello Yuchen!</p>
<aside class="quote no-group" data-username="Yuchen_Mu" data-post="4" data-topic="104547" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/ea5d25/48.png" class="avatar"> Yuchen_Mu:</div>
<blockquote>
<p>will this code work?</p>
<p>Actually I verified that the final output, which is <code>Final_M_norm_square</code>, is the value that I want</p>
</blockquote>
</aside>
<p>I haven’t looked at your code in detail.</p>
<p>But from what you say, my understanding is that:</p>
<ol>
<li>
<p>You are not using any complex tensors, so you don’t have<br>
to worry about any pytorch complex issues.</p>
</li>
<li>
<p>Your calculation only uses pytorch tensor operations (i.e.,<br>
you don’t switch over to something like numpy for part of<br>
the calculation).</p>
</li>
<li>
<p>Your function gives you the correct “forward-pass” result.</p>
</li>
</ol>
<p>So, yes, your code should work, it should work “for free” with<br>
autograd, and backpropagation should work properly.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Yuchen_Mu
cooked: <p>Hi Frank,</p>
<p>Thank you very much! Really appreciate your help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Regards,<br>
Yuchen</p>