How to transfer MatConvNet to Pytorch

username: yuhaoooo
cooked: <p>Hi guys, I encounter a problem. I have a function in MatLab, but I need to realize this in PyTorch. Here is the function.</p>
<pre><code class="lang-auto">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Join the gradients (from the
% discriminator and the generator)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function d = join_gradients(dgdz_d, dgdz_f, kappa)
    % check kappa's range
    if kappa&lt;0 || kappa&gt;1
        error('kappa has to be in the [0,1] range!');
    end
    % join gradients
    d = dgdz_d;
    for d_i = 1:size(dgdz_d,1)
        d1 = dgdz_d(d_i,:,:,:);
        d2 = dgdz_f(d_i,:,:,:);
        if norm(d1(:))&gt;norm(d2(:))
            d1 = d1*norm(d2(:))/norm(d1(:));
        else
            d2 = d2*norm(d1(:))/norm(d2(:));
        end
        d(d_i,:,:,:) = kappa*d1 + (1-kappa)*d2;
    end
end
</code></pre>
<p>I explain the meaning of this function. The <code>dgdz_d</code> is the gradient calculated from the loss, named as <code>loss_1</code>, and the <code>dgdz_f</code> is the gradient calculated from another loss, named as <code>loss_2</code>. The <code>kappa</code> is an coefficient to balance the two gradients.<br>
Could I just use</p>
<pre><code class="lang-auto">loss = kappa * loss_1 + (1.0 - kappa) * loss_2
loss.backward()
</code></pre>
<p>to substitute this function in PyTorch?<br>
If not, could you give me advice about how to realize this in PyTorch? Thanks a lot.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Doing the weighting on the loss will definitely match the behavior of <code>d(d_i,:,:,:) = kappa*d1 + (1-kappa)*d2;</code> yes.</p>
<p>But there seem to be some renormalization happening here before the gradients are combined? I’m not sure I understand what it’s trying to do though.</p>

username: yuhaoooo
cooked: <p>Yes, in MatLab code, before joining the gradients, they normalize the two gradients. I think the aim of using normalization is to scale the two gradients to the same magnitude.</p>
<pre><code class="lang-auto">if norm(d1(:))&gt;norm(d2(:))
    d1 = d1*norm(d2(:))/norm(d1(:));
else
    d2 = d2*norm(d1(:))/norm(d2(:));
end
</code></pre>
<p>If the norm of <code>d1</code> is bigger than <code>d2</code>, it will scale <code>d1</code> to the same magnitude of <code>d1</code>. Because <code>d1</code> and <code>d2</code> have the same magnitude, the <code>kappa</code> will work better.<br>
So, How does PyTorch perform the same with this MatLab code?</p>

username: albanD
cooked: <p>Well if you want to do that. You cannot use a single loss.<br>
You will have to backward each loss independently and save the gradients (with .clone()).<br>
And then you can do the same code that check the norm of each gradient and compute the final gradient.</p>

username: yuhaoooo
cooked: <p>Thanks a lot.<br>
According your advice, I write the following code in PyTorch.</p>
<pre><code class="lang-auto"># calculating the gradients of two losses
grads_loss_1 = autograd.grad(loss_1, model.parameters(), retain_graph=True)
grads_loss_2 = autograd.grad(loss_2, model.parameters(), retain_graph=False)
# update generator parameters gradients
for idx, p in enumerate(gen.parameters()):
    grad_1 = grads_loss_1[idx]
    grad_2 = grads_loss_2[idx]
    if torch.norm(grad_1, p=2) &gt; torch.norm(grad_2, p=2):
        grad_1 = grad_1 * torch.norm(grad_2, p=2) / torch.norm(grad_1, p=2)
    else:
        grad_2 = grad_2 * torch.norm(grad_1, p=2) / torch.norm(grad_2, p=2)
    p.grad = (kappa * grad_1 + kappa * grad_2).clone()
</code></pre>
<p>It works well in my work. Please point it out, if you find any errors.<br>
Thanks again for your suggestions!</p>

username: albanD
cooked: <p>Hi,</p>
<p>No error that I can see. it looks quite good!</p>