How to use Grad in AutoGrad pytorch

username: Sen_Trankim
cooked: <p>I have a long fomular:<br>
tensor a = […]<br>
tensor b = […]<br>
cosin_a = cosin(a)<br>
cosin_b = cosin(b)<br>
…<br>
output = cosin_a + consin_b)<br>
Final = loss(output, target)<br>
grad_ = torch.autograd.grad(output, (tensor a , tensor b), create_graph=True, retain_graph=True)<br>
After that, I got “One of the differentiated Tensors appears to not have been used in the graph”<br>
But If I change tensor a and tensor b by other parameter directly to out such a cosin_a or consin(b) and I got grad_ dimension of output different with tensor a and b and I don’t know how to update variable a and b to optimize loss,<br>
Thank you for considering my issue!</p>

username: tom
cooked: <p>This means that there is no autograd connection (the <code>.grad_fn</code>s of the tensor) between the input and the output.<br>
This can happen for example if you at some point only conditionally use one or the other.<br>
So this error can be suppressed by passing <code>allow_unused=True</code> to <code>torch.autograd.grad</code>. However, I always recommend that to find out why exactly this situation happens and add a comment to the invocation detailing that reason to make sure that it is not a bug in your code. In my personal experience, articulating the reasons of why certain default behaviour is not applicable in my particular case helps me a lot with sorting out whether it is an error on my side or a case where the behaviour should legitimately deviate from the default.</p>
<p>You could also try to change your computation to do this differently (e.g. use <code>torch.where</code> instead of an <code>if ... else</code> clause (this can also be computationally more efficient), but it would really depend on the details of your situation whether this is a good course of action.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Sen_Trankim
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/352beebfee754c228f36472ebdcae6ff11954567.png" data-download-href="https://discuss.pytorch.org/uploads/default/352beebfee754c228f36472ebdcae6ff11954567" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/352beebfee754c228f36472ebdcae6ff11954567.png" alt="image" data-base62-sha1="7AnqJgixbk5Rh7gnP4W82BpZZ3N" width="690" height="443" data-dominant-color="F4F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">989×636 28.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In my code, I want to minimize loss depend on self.m variable at list index “label”. And I got “One of the differentiated Tensors appears to not have been used in the graph” after that I set allow_unused=True then I got the output of grad is None. I don’t know to fix it, And I think all of steps use variable self.m. and variable easy_margin default False. Thank you for your support!</p>

username: tom
cooked: <ol>
<li>Autograd only works with torch.sin / torch.cos, not math.cos. But then you don’t need to go via enumeration but can just use the pointwise function on the tensor.</li>
<li>You probably want to differentiate w.r.t. self.m itself and then take the index (unless self.m is prohibitively large, in which case you should assign <code>self.m[label]</code> to a Python variable at the very top of your function and then re-use that).</li>
</ol>
<p>This should get you a bit further.</p>
<p>P.S.: With triple backticks (```) before and after your code and the code as text, it is easier to copy-paste things for rearranging.</p>

username: Sen_Trankim
cooked: <pre><code class="lang-auto">class ArcMarginProduct(nn.Module):
    r"""Implement of large margin arc distance: :
        Args:
            in_features: size of each input sample
            out_features: size of each output sample
            s: norm of input feature
            m: margin
            cos(theta + m)
        """
    def __init__(self, in_features, out_features, s=30.0, m=0.50, easy_margin=False, ls_eps=0.0):
        super(ArcMarginProduct, self).__init__()
        self.in_features = torch.tensor(in_features)
        self.out_features = torch.tensor(out_features)
        self.s = torch.tensor(s)
        self.m = torch.repeat_interleave(torch.tensor([m],requires_grad=True),CFG.num_classes)
        self.ls_eps = torch.tensor(ls_eps)  # label smoothing
        self.weight = Parameter(torch.FloatTensor(out_features, in_features))
        nn.init.xavier_uniform_(self.weight)
        self.easy_margin = easy_margin
        self.torch_pi = torch.acos(torch.zeros(1))*2
#         self.critition = critition
    def forward(self, input, label, critition=None):
        
        m = self.m[label]
        cos_m = torch.tensor([torch.cos(value) for value in m]).to(CFG.device).reshape(-1,1)
        sin_m = torch.tensor([torch.sin(value) for value in m]).to(CFG.device).reshape(-1,1)
        th = torch.tensor([torch.cos(torch.sub(self.torch_pi, value)) for value in m]).to(CFG.device).reshape(-1,1)
        mm = torch.tensor([torch.mul(torch.sin(torch.sub(self.torch_pi, value)), value) for value in m]).to(CFG.device).reshape(-1,1)
        
        # --------------------------- cos(theta) &amp; phi(theta) ---------------------------
        cosine = F.linear(F.normalize(input), F.normalize(self.weight))
        sine = torch.sqrt(torch.sub(1.0,torch.pow(cosine, 2)))
        phi = torch.sub(torch.mul(cosine,cos_m),torch.mul(sine,sin_m))
        
        if self.easy_margin:
            phi = torch.where(cosine &gt; 0, phi, cosine)
        else:
            phi = torch.where(cosine &gt; th, phi, torch.sub(cosine,mm))
        # --------------------------- convert label to one-hot ---------------------------
        # one_hot = torch.zeros(cosine.size(), requires_grad=True, device='cuda')
        one_hot = torch.zeros(cosine.size(), device='cuda')
        one_hot.scatter_(1, label.view(-1, 1).long(), 1)
        if self.ls_eps &gt; 0:
            one_hot = torch.add(torch.mul(torch.sub(1, self.ls_eps), one_hot), torch.div(self.ls_eps, self.out_features))
        # -------------torch.where(out_i = {x_i if condition_i else y_i) -------------
        output = torch.add(torch.mul(one_hot, phi), torch.mul(torch.sub(1.0,one_hot),cosine))
        output = torch.mul(output, self.s)
        
        ### update m
        if critition != None:
#             training_gradient = (training_loss)
            loss = critition(output, label)
            out = torch.autograd.grad(loss, m , create_graph=True, retain_graph=True,allow_unused=True)
            print(out) # output none
#             self.m[label] = self.m[label] - phi_b[0]
            return output, loss
        return output
</code></pre>
<p>I replace all operator *,-,/,+ to torch pointwise, and assing m = self.m[index] and resuse it. And I got the same error, output of torch grad equal “None”. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> . Thank you for your work.</p>

username: Sen_Trankim
cooked: <p>In the end, my issue is solved by fix some initial variables. Thank for your work!</p>

username: tom
cooked: <p>Congrats, I am glad you figured it out!</p>