RuntimeError: element 0 of tensors does not require grad and does not have a

username: How_Range
cooked: <p>I connected a classification network consisting of two linear layers in front of a standard transformer. I wanted to freeze the parameters of the standard transformer and only train the preceding classification layer. So, I traversed the parameters of the entire network, set their require_grad to False, and set the require_grad of the classification layer to True. However, this caused the error as shown in the title.<br>
Interestingly, when I set require_grad to True for either the encoder or decoder, everything returns to normal. And the entire network also works fine when none of the layers are frozen.</p>
<p>below is the classifier network and the whole network</p>
<pre><code class="lang-auto">class binaryClassifier(BaseFairseqModel):
    def __init__(self, in_dim, hidden_size=32, out_dim=1, ):
        super().__init__()
        self.layer1 = nn.Linear(in_dim, hidden_size)
        self.layer2 = nn.Linear(hidden_size, out_dim)
        self.activate = nn.Sigmoid()

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return self.activate(x)


class binFormerModel(TransformerModel):
    def __init__(self, args, encoder, decoder, in_dim=2, hidden_size=32, out_dim=1):

        super().__init__(args, encoder, decoder)
        self.binaryClassifier = binaryClassifier(in_dim, hidden_size, out_dim)
        for param in self.parameters():
            param.requires_grad = False
        for param in self.binaryClassifier.parameters():
            param.requires_grad = True
</code></pre>
<p>below is the forward function of binformer<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/027c93f3f0682fb2eed5dcbc16ce488c850807ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/027c93f3f0682fb2eed5dcbc16ce488c850807ac" title="545b6eb5faf79ed4b74b4231fc3be99"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027c93f3f0682fb2eed5dcbc16ce488c850807ac_2_690x402.png" alt="545b6eb5faf79ed4b74b4231fc3be99" data-base62-sha1="lZRrkrY9WbZ9KxDBScmbn2TBMM" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027c93f3f0682fb2eed5dcbc16ce488c850807ac_2_690x402.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027c93f3f0682fb2eed5dcbc16ce488c850807ac_2_1035x603.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027c93f3f0682fb2eed5dcbc16ce488c850807ac_2_1380x804.png 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">545b6eb5faf79ed4b74b4231fc3be99</span><span class="informations">1425×831 59.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: thecho7
cooked: <p><code>real_src</code> and <code>real_length</code> are tensors with no gradients.<br>
I’m not sure that <code>pure_source</code> and <code>src_tokens</code> have the same dimension but if yes,</p>
<pre><code class="lang-auto">real_src = torch.where(src_tokens, pure_source, pred &gt;= 0.5)
</code></pre>
<p>(It could return error since I’m not available to test the code)</p>

username: How_Range
cooked: <p>thanks for your reply</p>
<p>pred is a tensor （batch_size,1）<br>
source_tokens  （batch_size,sequence_length1）<br>
pure_source (batch_size, sequence_length2)<br>
usually sequence_length1 is not equall to length_2</p>
<p>so I have no idea with how to write torch.where</p>

username: thecho7
cooked: <p>It is weird.<br>
Could <code>real_src[i] = src_tokens[i]</code> work?<br>
It may return error since they have the different dimensions</p>

username: How_Range
cooked: <p>you help me find out another bug. Since pred is very low at the beginning so the branch you mentioned has never worked indeed.<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>