One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 768]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead

username: ammaarahmad1999
cooked: <p>I am getting this error.</p>
<p>Here is my code</p>
<p>class Adversarial_Multitask(nn.Module):</p>
<pre><code>def __init__(self, bert):
  
  super(Adversarial_Multitask, self).__init__()

  self.bert1 = bert
  self.bert2 = bert
  self.bert3 = bert
  self.drop = nn.Dropout(0.1)
  self.fc1 = nn.Linear(1536, 2)       #Main Task
  self.fc2 = nn.Linear(1536, length)  #Section Identifcation
  self.act = nn.ReLU()

def diff_loss(self, output1, output2):

  print(output1.shape)
  output1 -= torch.mean(output1, 0)
  output2 -= torch.mean(output2, 0)

  print(output1.shape)

  output1 = torch.norm(output1, dim=1)
  output2 = torch.norm(output2, dim=1)

  print(output1.shape)

  loss_dif = torch.matmul(output1, output2)
  return loss_dif

#define the forward pass
def forward(self, batch, x):

  #pass the inputs to the model  
  output2 = self.bert2(batch[0], attention_mask=batch[1])

  if (x == 1):
    output1 = self.bert1(batch[0], attention_mask=batch[1])
    pooled_output = torch.cat((output1[1], output2[1]), dim=1)
    output = self.fc1(pooled_output)
    loss_dif = 0 #
    self.diff_loss(output1[1], output2[1])
    
  else:
    output3 = self.bert3(batch[0], attention_mask=batch[1])
    pooled_output = torch.cat((output3[1], output2[1]), dim=1)
    output = self.fc2(pooled_output)
    loss_dif = 0 #
    self.diff_loss(output3[1], output2[1])   
  return output, loss_dif
</code></pre>
<p>I am trying to use orthogonality constraint loss for multi task learning as used in this paper: <a href="https://www.aclweb.org/anthology/P17-1001.pdf" rel="noopener nofollow ugc">https://www.aclweb.org/anthology/P17-1001.pdf</a></p>
<p>I am multiplying 2 vectors to get the loss_dif but it is creating this issue. I need that loss for better performance. How can this be solved?</p>

username: ptrblck
cooked: <p>You could remove the inplace operations and replace them with their out of place equivalent, e.g.:</p>
<pre><code class="lang-python">  output1 -= torch.mean(output1, 0)
  output2 -= torch.mean(output2, 0)
</code></pre>
<p>and check, if this would solve the issue.</p>

username: ammaarahmad1999
cooked: <p>Sorry for late reply. This issue was solved. I don’t know how. Maybe there was bug somewhere else</p>

username: ammaarahmad1999
cooked: <p>Can you answer to this question.  I need little urgent</p><aside class="quote quote-modified" data-post="1" data-topic="125258">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b77776/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-a-shuffle-dataloader-from-2-datasets-and-then-shuffling-the-batches/125258">How to create a shuffle dataloader from 2 datasets and then shuffling the batches</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    I want to create a dataloader with batches containing data from 1 dataset. I have 2 datasets 
train_data = TensorDataset(train_sn, train_sm, train_tt, train_y) 
scaffold_data = TensorDataset(sca_txt, sca_att, sca_y) 
I want to create a single dataloader for these 2 dataset such that a batch contains data from either 1 dataset but not both. Also I want to shuffle the batches among themselves. That is it shouldn’t be like all batches from 1 dataset followed by batches from other dataset while trai…
  </blockquote>
</aside>