position_ids version changed during training
Author: Gforky
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: linux
- Python version: 3.6.9
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?): no
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [* ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [* ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. deployed BertModel in modeling_bert.py to train text classification model.
2. submitted task using `torch.distributed.launch` with 2 P40 GPUs.
3. left `position_ids` empty, and initialized it in `BertEmbeddings` while doing `forward` computation.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected training to be successfully finished, but encountered issue below:
```
RuntimeError: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 128]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [1, 128]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
```
Maybe the initialization of position_ids should be suffixed with clone() in line 195 under `transformers.models.bert.modeling_bert`? I resolved the issue by adding clone to `self.position_ids`.

Comments:
Hello! Do you have a reproducible code example? Did you use the text-classification examples given in this repo?

> Hello! Do you have a reproducible code example? Did you use the text-classification examples given in this repo?

Hi, thanks for your reply. Since it's a private project, I cannot provide you the entire code. But I can share with you the code that triggers the issue:
```
def get_loss(args, model, train_batch, unsup_batch, global_step, total_steps):
    # batch
    input_ids, attention_mask, token_type_ids, input_len, labels = train_batch
    if unsup_batch:
        ori_input_ids, ori_attention_mask, ori_token_type_ids, ori_input_len, ori_labels, \
        aug_input_ids, aug_attention_mask, aug_token_type_ids, aug_input_len  = unsup_batch

        input_ids = torch.cat((input_ids, aug_input_ids), dim=0)
        attention_mask = torch.cat((attention_mask, aug_attention_mask), dim=0)
        token_type_ids = torch.cat((token_type_ids, aug_token_type_ids), dim=0)
    # torch_device_one used by loss computation
    torch_device_one = torch.tensor(1., device=args.device)
    # logits
    outputs = model(input_ids, attention_mask, token_type_ids)
    logits = outputs[0]
    # loss fct
    sup_loss_fct = CrossEntropyLoss(reduction='none')
    unsup_loss_fct = KLDivLoss(reduction='none')
    # sup loss
    sup_size = labels.shape[0]
    sup_loss = sup_loss_fct(logits[:sup_size], labels)  # shape : train_batch_size
    if unsup_batch and args.do_tsa:
        tsa_thresh = get_tsa_thresh(args.tsa_type,
                                    global_step,
                                    total_steps,
                                    start=args.tsa_start,
                                    end=1,
                                    scale=args.tsa_scale)
        larger_than_threshold = torch.exp(-sup_loss) > tsa_thresh
        loss_mask = torch.ones_like(labels, dtype=torch.float32) * (1 - larger_than_threshold.type(torch.float32))
        sup_loss = torch.sum(sup_loss * loss_mask, dim=-1) / torch.max(torch.sum(loss_mask, dim=-1), torch_device_one)
    else:
        sup_loss = torch.mean(sup_loss)
    # unsup loss
    if unsup_batch:
        uda_softmax_temp = args.uda_softmax_temp if args.uda_softmax_temp > 0 else 1.
        with torch.no_grad():
            ori_outputs = model(ori_input_ids, ori_attention_mask, ori_token_type_ids)
            ori_logits = ori_outputs[0]
            ori_prob = F.softmax(ori_logits, dim=-1)  # KLdiv target
            # confidence-based masking
            if args.uda_confidence_thresh != -1:
                unsup_loss_mask = torch.max(ori_prob, dim=-1)[0] > args.uda_confidence_thresh
                unsup_loss_mask = unsup_loss_mask.type(torch.float32)
            else:
                unsup_loss_mask = torch.ones(len(logits) - sup_size, dtype=torch.float32, device=args.device)
            # softmax temperature controlling
            ori_logits = ori_logits / uda_softmax_temp
            ori_prob = F.softmax(ori_logits, dim=-1)
        aug_log_prob = F.log_softmax(logits[sup_size:], dim=-1)
        unsup_loss = torch.sum(unsup_loss_fct(aug_log_prob, ori_prob), dim=-1)
        unsup_loss = torch.sum(unsup_loss * unsup_loss_mask, dim=-1) / torch.max(torch.sum(unsup_loss_mask, dim=-1),
                                                                                 torch_device_one)
        final_loss = sup_loss + args.uda_unsup_coeff * unsup_loss
        return final_loss, sup_loss, unsup_loss
    return sup_loss, None, None
```

Finally we've noticed the issue is we call the forward computation twice, and we change the inner parameters within a no_grad operation. So we move the no_grad code block to the position before the first forward computation, it also resolves the issue.

But back to my question, is that possible to just add a clone() to the position_ids initialization should avoid this issue under other scenarios? Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.