Finetuning two SentenceTransformers (twin BERT like)

username: belhal
cooked: <p>Hello all,</p>
<p>Similar issues than <a href="https://discuss.pytorch.org/t/runtimeerror-element-0-of-variables-does-not-require-grad-and-does-not-have-a-grad-fn/11074" class="inline-onebox">RuntimeError: element 0 of variables does not require grad and does not have a grad_fn</a></p>
<p>I am training a double sentence transformer models (training a cosine similarity function between two embeddings) using the code below.</p>
<p>My issue: the loss does not change, as I suspect that the loss does not propagate (gradients of the optimizer returns None). Something must be frozen here but I really don’t see what.</p>
<pre><code class="lang-auto">class CosineLoss(nn.Module):
    def __init__(self):
        super(CosineLoss, self).__init__()
        self.loss = nn.MSELoss(reduction='sum')

    def forward(self, output1, output2, label):
        cos_sim = F.cosine_similarity(output1, output2)
        final_loss = self.loss(cos_sim, label)
        return final_loss
      
class TestModel(nn.Module):
  def __init__(self):
    super().__init__()
    self.modelA = SentenceTransformer('clip-ViT-B-32')
    self.modelB = SentenceTransformer('clip-ViT-B-32-multilingual-v1')

  def embed_A(self, inputA):
    x = self.modelA.encode([Image.open(BytesIO(requests.get(filepath).content)).convert('RGB')for filepath in inputA], convert_to_tensor=True)
    return x
  
  def embed_B(self, inputB):
    x = self.modelB.encode(inputB, convert_to_tensor=True)
    return x

  def forward(self, inputA, inputB):
    x1 = self.embed_A(inputA)
    x2 = self.embed_B(inputB)
    return x1, x2


mymodel = TestModel()
criterion = CosineLoss()
optimizer = optim.Adam(mymodel.parameters(), lr=lr,betas=betas,eps=eps,weight_decay=wd) 

for param in mymodel.parameters():
    param.requires_grad = True

for epoch in range(maxepochs):
  print('Epoch:', epoch)
  mymodel.train()
  for batch in dm.train_dataloader():
    mymodel.zero_grad()
 
    list_A,list_B,ground_truth = batch 
   
    outputA, outputB = mymodel(list_A,list_B)
    ground_truth = ground_truth.to(device)

    optimizer.zero_grad()
    loss = criterion(outputA,outputB,ground_truth)

    loss.requires_grad = True
    loss.backward()
    optimizer.step()
    print(optimizer.param_groups[0]['params'][0].grad)

print("Saving model for epoch:", epoch)
print("Total Loss for Epoch number {} is {}".format(epoch, loss))
</code></pre>
<p>The prints of that code are</p>
<pre><code class="lang-auto">Epoch: 0
None
None
None
None
None
Saving model for epoch: 0
Total Loss for Epoch number 0 is 0.4687237625608282
Epoch: 1
None
None
None
None
None
Saving model for epoch: 1
Total Loss for Epoch number 1 is 0.4687237625608282
Epoch: 2
None
None
None
None
None
Saving model for epoch: 2
Total Loss for Epoch number 2 is 0.4687237625608282
</code></pre>
<p>I believe that the line  loss.requires_grad = True is the issue but getting rid of it just keeps returning the error message "element 0 of tensors does not require grad and does not have a grad_fn’.</p>
<p>I would love to get your help on this.</p>
<p>Thank you very much.<br>
Belhal</p>

username: belhal
cooked: <p>Update: I was able to get rid of the line loss.requires_grad = True by adding .requires_grad_() to my input embeddings as in</p>
<pre><code class="lang-auto">def embed_A(self, inputA):
    x = self.modelA.encode([Image.open(BytesIO(requests.get(filepath).content)).convert('RGB')for filepath in inputA], convert_to_tensor=True).requires_grad_()
    return x
  
  def embed_B(self, inputB):
    x = self.modelB.encode(inputB, convert_to_tensor=True).requires_grad_()
    return x
</code></pre>
<p>Now print(loss.grad_fn) return an object, yet print(optimizer.param_groups[0][‘params’][1].grad) retuirns None and the loss still stays constant.<br>
Seems like while my lioss is now differentiable, the optimizer does not consider my mymodel.params() as variable.</p>
<p>Any ideas on how to fix it?<br>
Thank you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="belhal" data-post="2" data-topic="153219">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/belhal/48/22879_2.png" class="avatar"> belhal:</div>
<blockquote>
<p>I was able to get rid of the line loss.requires_grad = True by adding .requires_grad_() to my input embeddings as in</p>
</blockquote>
</aside>
<p>This also doesn’t sound like a proper solution. Calling <code>.requires_grad_()</code> on the output of <code>self.modelB.encode</code> will not stitch the computation graphs together, but will start a new one.<br>
The operations in <code>self.modelB</code> (and before that) are still detached and the parameters used in these operations will not get a valid gradient (using the current output).</p>
<p>If the return value of <code>self.modelB.encode</code> is already detached (check its <code>.grad</code> attribute and see if it’s <code>None</code>), then you would need to dig into the model definition and see if gradient calculation is disabled or which operation detaches the graph.</p>

username: belhal
cooked: <p>Thank you for that precision.<br>
You are totally right.<br>
Basically those 2 simples lines returns None</p>
<pre><code class="lang-auto">modelB = SentenceTransformer('clip-ViT-B-32-multilingual-v1')
out = modelB.encode('a sentence', convert_to_tensor=True)
print(out.grad)
&gt;&gt;&gt; None
</code></pre>
<p>So indeed those output embeddings (from modelA and modelB) are detached and no grad can be computed.</p>
<p>I wonder if it has to do with the Hugging Face SentenceTransformer model that by default is like that (not fine tuning friendly by default which would be understandable).</p>
<p>Any thoughts?</p>
<p>Thanks again, you were spot on right <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>I’m not sure which repository you are using, but assume it’s <a href="https://github.com/UKPLab/sentence-transformers"><code>UKPLab/sentence-transformers</code></a>. If so, then note that the <code>.encode</code> method explicitly disables gradient calculation <a href="https://github.com/UKPLab/sentence-transformers/blob/bf9f26b09df53b52bf263ede29fee7488870bdcb/sentence_transformers/SentenceTransformer.py#L163">here</a> so it seems to be by design.<br>
<a href="https://github.com/UKPLab/sentence-transformers/issues/274">This issue</a> seems to be related and the answer was:</p>
<blockquote>
<p>You can use the forward method of SentenceTransformers. This allow to include SentenceTransformer into a larger model.<br>
However, the forward method expects a correctly formatted batch as input.</p>
</blockquote>

username: belhal
cooked: <p>yes I am using this repo</p>
<pre><code class="lang-auto">from sentence_transformers import SentenceTransformer
</code></pre>
<p>Ok I will dig into the batch formatting.<br>
The github ticket does not say much unfortunately and i would expect that a simple input text such as a string (as in '‘a sentence’) would work.</p>
<p>Thank you for your help for narrowing it down <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: belhal
cooked: <p>For future reference, from Nils Reimers</p>
<p>This should work:</p>
<pre><code class="lang-auto">encoded_input = model.tokenize(your_input)
out = model(encoded_input)
</code></pre>
<p>Out will contain a dict with different features. The <code>sentence_embedding</code> feature is the final output of the model</p>
<p>This is the solution to backprop using SentenceTransformer API.</p>