DataParallel/LOSS element 0 of tensors does not require grad and does not have a grad_fn

idanpd1: <p>Hi,<br>
Im trying to create new weights in order to calculate projection between the featuremaps and the new weights.</p>
<p>I created a class called ArcFace that some of you may know in the initialized model.<br>
self.arc= ArcFace(s = 10, num_classes=num_classes, margin= 2 )<br>
in ArcFace class I initiate a new self.weight =  torch.nn.Parameter(torch.normal(0, 0.01, (num_classes,self.feature_dim)))<br>
before running nn.DataParallel(model).cuda() I had the same weights saved under the:<br>
model.arc.weight variable - like it should<br>
After the model is being set there is the DataParallel that destroys it all:<br>
the variable changes to : model.module.arc.weight variable<br>
both with required_grad = True<br>
_backward_hooks:<br>
None<br>
_grad_fn:<br>
None<br>
_grad:<br>
None</p>
<pre><code>if cfg.use_gpu:
    model = nn.DataParallel(model).cuda()
</code></pre>
<p>then the optimizer is being set and after the scheduler.</p>
<p>After all that when i am training the model</p>
<pre><code>    if self.loss == 'arcloss':
        
        logits  = self.model(imgs) - output the embeddings of the net for each img in a 64 batch size.
</code></pre>
<p>then: we send the logits to the Forward function of ArcFace class:<br>
in this class we normalize the same initialized weights (self.weight that we initialized in the first time)</p>
<p>requires_grad has chaned to False !!!<br>
and grad_fn still None<br>
Although, the weights are just normalized and multiplied it with the same embeddings from the output Net.</p>
<pre><code>        outputs = self.model.module.arc(embeddings,pids)  (here, inside the self.model.module.arc there is the self.weight that was initialized in the ArcFace and multiplied with the embeddings)
</code></pre>
<p>class ArcFace(torch.nn.Module):<br>
“”" ArcFace (<a href="https://arxiv.org/pdf/1801.07698v1.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1801.07698v1.pdf</a>):<br>
“”"<br>
def <strong>init</strong>(self, s=30.0 , num_classes =10, margin=0.5):<br>
super(ArcFace, self).<strong>init</strong>()<br>
self.classes = num_classes<br>
self.out_features = 512<br>
self.scale = s<br>
self.cos_m = math.cos(margin)<br>
self.sin_m = math.sin(margin)<br>
self.theta = math.cos(math.pi - margin)<br>
self.sinmm = math.sin(math.pi - margin) * margin<br>
self.easy_margin = False</p>
<pre><code>    self.weight = torch.nn.Parameter(torch.randn( self.classes, self.out_features ),requires_grad=True)
    #self.weight.requires_grad = True
    nn.init.xavier_uniform_(self.weight)
    #self.classifier = nn.Linear(self.out_features, num_classes).cuda()
    #parametrize.register_parametrization(self.classifier, "weight", Norm())
    #print(self.classifier.weight)
    #self.classifier.requires_grad_ = True

def forward(self, logits: torch.Tensor,  labels: torch.Tensor):
    #print(self.classifier.weight)
    #cosine = F.linear(F.normalize(logits), self.head.weight)
    #print(self.weight)
    #
    cosine = F.linear(F.normalize(logits), F.normalize(self.weight))
    cosine = cosine.clamp(-1,1)
    sine = torch.sqrt(1.0 - torch.pow(cosine, 2))
    phi = cosine * self.cos_m - sine * self.sin_m
    if self.easy_margin:
        phi = torch.where(cosine &gt; 0, phi, cosine)
    else:
        phi = torch.where(cosine &gt; self.theta, phi, cosine - self.sinmm)
    # --------------------------- convert label to one-hot ---------------------------
    one_hot = torch.zeros(cosine.size(), device='cuda')
    #one_hot = torch.zeros(cosine.size()).cuda()
    one_hot.scatter_(1, labels.view(-1, 1).long(), 1)
    # -------------torch.where(out_i = {x_i if condition_i else y_i) -------------
    output = (one_hot * phi) + ((1.0 - one_hot) * cosine)  # you can use torch.where if your torch.__version__ is 0.4
    output *= self.scale
    return output
</code></pre>
<p>and with the returning variable “output” I calculate softmaxloss with the model<br>
self.criterion = CrossEntropyLoss(<br>
num_classes=self.datamanager.num_train_pids,<br>
use_gpu=self.use_gpu,<br>
label_smooth=label_smooth<br>
)</p>
<p>loss = self.compute_loss(self.criterion, outputs, pids)<br>
and when running<br>
self.optimizer.zero_grad()<br>
loss.backward() - Here i got the error !<br>
element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>I dont understand,if i multiply my Normalized(self.weight) with the embeddings ,  If I did a multiplication operation, why doesn’t grad_fn recognize the operation and thus it knows by what to deviate. but it is not like that - I actually get this error :</p>
<p>Exception has occurred: RuntimeError</p>
<p>element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>I tried to read for couple of days and understand the problem. but with no luck.</p>
<p>I just want to calculate the projection of embeeding that comes from  the net with initialized weights (that initialized in the ArcFace module).</p>
<p>I dont understand why loss and the same weights has no grad_fn and no requierd_grad.</p>
<p>Please!!! Help meeeeeeee!!!</p>
<pre><code>   self.num_classes = len(pids)
    ### Need to output the y = classifier(v) , that is the connection for the acc
    #outputs  = self.model(imgs)
    
    if self.loss == 'arcloss':
        
        logits  = self.model(imgs)
        outputs = self.model.module.head(logits,pids)
    else:
        outputs  = self.model(imgs)  
           
            
    loss = self.compute_loss(self.criterion, outputs, pids)    ##metric_fc
    
    self.optimizer.zero_grad()
    loss.backward()     -Error !!!!! I tried .double and all other stufff
    self.optimizer.step()
    
    #acc = np.mean((outputs == pids).astype(float))
    loss_summary = {
        'loss': loss.item(),
</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing the issue, please?<br>
You can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>