RuntimeError with DistributedDataParallel 
Author: ghpu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Ubuntu 18.04
- Python version: 3.7.7.final.0
- PyTorch version (GPU?): 1.6.0 gpu
- Tensorflow version (GPU?): X
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed set-up
- Conda : 4.8.3 
- cuda =11.0

### Who can help
 @LysandreJik  @sgugger

## Information

Model I am using (Bert, XLNet ...): bert mulitilingual cased

The problem arises when using:
* [x] my own modified scripts : Triplet Loss applied to sentence similarity, to practice writing efficient distributed learning code.


The tasks I am working on is:
* [x] my own task or dataset: Triplet Loss applied to sentence similarity, to practice writing efficient distributed learning code.


## To reproduce

I have not been able to generte a minimal version yet.

Stack trace

```
Traceback (most recent call last):
[...]
  File "my_program.py", line 246, in epoch
    loss.backward()
  File "conda/lib/python3.7/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 200]] is at version 9; expected version 7 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```

My model : 

```python

def __init__(self, trainfile, devfile, outdir, rank=0):
         # default config
          self.max_sent_len = 200
          self.batchsize = 10
          self.lr = 2e-5
          self.accumulated_batches = 6
          self.max_epochs = 10
          self.seed = 42
          self.embs = "bert-base-multilingual-cased"
          self.tokenizer, self.model = self.transformers(self.embs)
          self.tripletloss = lossless_triplet_loss.LosslessTripletLoss()
  
          print("use %s gpus" % torch.cuda.device_count())
          self.model = self.model.to(rank)
          self.model = torch.nn.parallel.DistributedDataParallel(self.model, device_ids=[rank], output_device=rank, find_unused_para  meters=True)
          self.rank = rank
  
          half = False
          self.model = self.model.to(self.rank) 
  
          self.optimizer = optim.Adam(filter(lambda param: param.requires_grad, self.model.parameters()), lr=self.lr)
  
          self.traindataset = data.TripleDataset2(trainfile, self.tokenizer, self.max_sent_len)
          self.devdataset = data.TripleDataset2(devfile, self.tokenizer, self.max_sent_len)
          self.traindataloader = DataLoader(self.traindataset,
                                            collate_fn=self.collate_fn,
                                            batch_size = self.batchsize,
                                            num_workers = 0)
          self.devdataloader = DataLoader(self.devdataset,
                                          collate_fn=self.collate_fn,
                                          batch_size = self.batchsize,
                                         num_workers = 0)

 def train(self, epoch):
          self.model.train()
  
          iterations = int(math.ceil(len(self.traindataset)/self.batchsize))
          iterator = tqdm.tqdm(enumerate(self.traindataloader),
                               ncols=100,
                               ascii=True,
                               desc="epoch %d" % (epoch),
                               mininterval=1.0,
                               total = iterations)
          lastloss = None
          for batch_idx, anchor_pos_neg in iterator:
              (anchor, amask), (pos, pmask), (neg, nmask) = anchor_pos_neg
  
              anchor = anchor.to(self.rank)
              pos = pos.to(self.rank)
              neg = neg.to(self.rank)
  
              amask = amask.to(self.rank)
              pmask = pmask.to(self.rank)
              nmask = nmask.to(self.rank)
  
              anchor_vec = self.tokens2vec(anchor, attention_mask=amask)
              pos_vec = self.tokens2vec(pos, attention_mask=pmask)
              neg_vec = self.tokens2vec(neg, attention_mask=nmask)
              loss, distp, distn = self.calc_triplet_loss(anchor_vec, pos_vec, neg_vec)
  
              loss.backward()
              if (batch_idx+1) % self.accumulated_batches == 0:
                  self.optimizer.step()
                  self.optimizer.zero_grad()

      def tokens2vec(self, tokens_tensor, attention_mask=None):
          hidden_states = self.model(tokens_tensor, attention_mask=attention_mask)[0]
          token_vecs = torch.squeeze(hidden_states, dim=0)
  
          a = torch.sum(torch.mul(token_vecs, torch.unsqueeze(attention_mask, 2)), dim=1)
          m = torch.unsqueeze(torch.sum(attention_mask, dim=1), 1)
          sentence_embedding = torch.sigmoid(torch.div(a, m))
          return sentence_embedding
  
  
      def transformers(self, name):
          if name.startswith("bert"):
              from transformers import BertTokenizer, BertModel
              tokenizer = BertTokenizer.from_pretrained(name, do_lower_case=False)
              model = BertModel.from_pretrained(name)
          else:
              tokenizer = None
              model = None
              print("ERROR: invalid embeddings name <%s>" % name, file=sys.stderr)
          return tokenizer, model

```

The error happens with batch_idx = 0.

200 is my maximum sequence length. I only use Longtensor as input to BertModel (for anchor, pos and neg). 
What I find strange is that I am sending samples in batches of size 10, so I am wondering if my problem is really with the embedding layer, or rather caused by my custom collate_fn  : 

```python
  def collate_fn(self, batch):
          anchor = []
          pos = []
          neg = []
          anchor_mask = []
          pos_mask = []
          neg_mask = []
          for a,p,n in batch:
              anchor.append(a[0])
              pos.append(p[0])
              neg.append(n[0])
              anchor_mask.append(a[1])
              pos_mask.append(p[1])
              neg_mask.append(n[1])
          return (torch.stack(anchor, dim=0),torch.stack(anchor_mask, dim=0)), \
               (torch.stack(pos, dim=0),torch.stack(pos_mask, dim=0)), \
               (torch.stack(neg, dim=0),torch.stack(neg_mask, dim=0))
```

My model is working well with only one GPU, or when using DataParallel. My issue arises only with DistributedDataParallel.


## Expected behavior
A running training.

Comments:
The `BertModel` runs absolutely fine in `DistributedDataParallel` (just retried) so the issue probably comes from the rest of your code and not the library. I can't see the code for the `LosslessTripletLoss`, maybe the issues comes from there.

In any case, to debug I would follow the instruction given in the error message:
```
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 200]] is at version 9; expected version 7 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```

Well, thanks for your tips. 

But when using set_detect_anomaly, my problem seems to be located in EmbeddingBackward :

```
[W python_anomaly_mode.cpp:60] Warning: Error detected in EmbeddingBackward. Traceback of forward call that caused the error:
  File "<string>", line 1, in <module>
  File "/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/lib/python3.7/multiprocessing/spawn.py", line 118, in _main
    return self._bootstrap()
  File "/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "my_program.py", line 525, in demo_basic
    f4n.train()
  File "my_program.py", line 213, in train
    self.epoch(epoch)
  File "my_program.py", line 249, in epoch
    pos_vec = self.tokens2vec(pos, attention_mask=pmask)
  File "my_program.py", line 444, in tokens2vec
    hidden_states = self.model(tokens_tensor, attention_mask=attention_mask)[0]
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/transformers/modeling_bert.py", line 831, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/transformers/modeling_bert.py", line 198, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
```



The problem occurs when calling BertModel the second time in the epoch, when calculating "pos".

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, have you solved this problem? I also met this issue.

No, sorry, I finally gave up.

Same Error. Any one has solved the problem?

I've solved this problem. Mine was because when I build the input sequences as follows:
`decoder_inputs = labels.narrow(1, 0, seqlen - 1).clone()`
`decoder_masks = label_masks.narrow(1, 0, seqlen - 1).clone()`
`decoder_labels = labels.narrow(1, 1, seqlen-1).clone()`
`decoder_label_masks = label_masks.narrow(1, 1, seqlen - 1).clone()`
`decoder_labels[~decoder_label_masks.bool()] = -100`

I forgot to add `.clone()` at the beginning. So when I change some of the `decoder_labels` to be -100, some of the elements in `decoder_inputs` were also changed, which caused the error.

Hope this information could help you a bit!

> I've solved this problem. Mine was because when I build the input sequences as follows:
> `decoder_inputs = labels.narrow(1, 0, seqlen - 1).clone()`
> `decoder_masks = label_masks.narrow(1, 0, seqlen - 1).clone()`
> `decoder_labels = labels.narrow(1, 1, seqlen-1).clone()`
> `decoder_label_masks = label_masks.narrow(1, 1, seqlen - 1).clone()`
> `decoder_labels[~decoder_label_masks.bool()] = -100`
> 
> I forgot to add `.clone()` at the beginning. So when I change some of the `decoder_labels` to be -100, some of the elements in `decoder_inputs` were also changed, which caused the error.
> 
> Hope this information could help you a bit!

Thanks a lot!

I met exact same issue with triplet loss using transformer 4.5.1 in NCCL distributed training setting.
I found this problem may occur to using multiple forward Bert output and compute loss at once.
I solved this by concat inputs ahead and chunk it to calculate loss:
```python
input_ids = torch.cat([anchor_input_ids, pos_input_ids, neg_input_ids], dim=0)
attention_mask = torch.cat([anchor_attention_mask, pos_attention_mask, neg_attention_mask], dim=0)
token_type_ids = torch.cat([anchor_token_type_ids, pos_token_type_ids, neg_token_type_ids], dim=0)

emb = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
anchor_emb, pos_emb, neg_emb = emb.pooled_output.chunk(3, dim=0)
loss = criterion(anchor_emb, anchor_emb, anchor_emb)
```


My problem was solved after adding `broadcast_buffers=False` to `torch.nn.parallel.DistributedDataParallel`
Following https://github.com/ashkamath/mdetr/issues/16#issuecomment-878388469