RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 4096, 1]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead.
Author: Sepelbaum
Pull request: no

When I run trainer to fine-tune pertained long former for sequence classification I get the following error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 4096, 1]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead.

I'm not sure how to debug this as the error points me to internal processes handled by the trainer:

Traceback (most recent call last):
  File "finetune_longformer_3.py", line 126, in <module>
    trainer.train()
  File "/......./conda/envs/diss/lib/python3.8/site-packages/transformers/trainer.py", line 1269, in train
    tr_loss += self.training_step(model, inputs)
  File "/....../conda/envs/diss/lib/python3.8/site-packages/transformers/trainer.py", line 1772, in training_step
    self.scaler.scale(loss).backward()
  File "/......../conda/envs/diss/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/........./conda/envs/diss/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(


any help would be much appreciated!

Comments:
Hello! Could you provide the information required by the template, please? Especially the code that you used, as it's hard to help without it. Thanks

I have a similar problem during Finetuning LED for Summarization Task in Colab, with the following error message:
-----------------------------
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [12, 4096, 1]], which is output 0 of ViewBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
------------------------------
The Settings for the Training are as follow:

Training Set: 17 Samples, each with less than 4000 tokens.
As for environment, I ran !pip install -r requirements.txt, where requirements come from the latest master branch of longformer.
----------------------
transformers @ git+http://github.com/ibeltagy/transformers.git@longformer_encoder_decoder#egg=transformers
pytorch-lightning @ git+http://github.com/ibeltagy/pytorch-lightning.git@v0.8.5_fixes#egg=pytorch-lightning
torch>=1.6.0
tensorboardX
test-tube==0.7.5
nlp
rouge_score
-----------------------------------

CUDA for the colab session was: 
Sun Jul 18 03:58:07 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.42.01    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   44C    P0    30W / 250W |      0MiB / 16280MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+                          
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Other Training Configurations are as follow:

loaded from pretrained is the "allenai/led-base-16384" with HuggingFace.

max_input_length = 4096
min_output_length = 256
max_output_length = 512
batch_size = 2

# set generate hyperparameters
led.config.encoder_layers=6
led.config.decoder_layers=6
led.config.attention_window=128 # left and right so total 256
led.config.num_beams = 2
led.config.length_penalty = 2.0
led.config.early_stopping = True
led.config.no_repeat_ngram_size = 3

# adjust output length according to training and val datasets
led.config.max_length = max_output_length # now at 256
led.config.min_length = min_output_length # now at 512

# enable fp16 apex training
training_args = Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy="epoch",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    fp16=True,
    output_dir=path_models,
    logging_steps=5,
    eval_steps=10,
    save_steps=10,
    save_total_limit=4,
    load_best_model_at_end=True,
    gradient_accumulation_steps=4, 
    num_train_epochs=6,
)

trainer = Seq2SeqTrainer(
    model=led,
    tokenizer=tokenizer,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)


Enabling  "torch.autograd.set_detect_anomaly(True)", point to the following:

/led/modeling_led.py", line 589, in _compute_attn_output_with_global_indices
    attn_probs_only_global.transpose(1, 2), value_vectors_only_global.transpose(1, 2)

It seems that the global attention calculation made changes to torch and somehow created some conflicts with gradient computation in terms tracking steps. 

I had successfully trained larger samples (600+ samples) with up to 8192 input tokens, with generate length between 256 and 512 , attention window size = 512 (1024 total from both side), using the led-base checkpoint. So seeing this error message is a bit frustrating. Any help is highly appreciated. Let me know if you need more information. Thank you.
-----------------------------
***** Running training *****
  Num examples = 17
  Num Epochs = 6
  Instantaneous batch size per device = 2
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 4
  Total optimization steps = 12
 [ 3/12 00:06 < 00:57, 0.16 it/s, Epoch 0.89/6]
Epoch	Training Loss	Validation Loss
**/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py:149: UserWarning: Error detected in BmmBackward0. Traceback of forward call that caused the error:**
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/function.py", line 87, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/checkpoint.py", line 122, in backward
    outputs = ctx.run_function(*detached_inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 1816, in custom_forward
    return module(*inputs, is_global_attn, output_attentions)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 915, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 726, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 282, in forward
    is_local_index_global_attn_nonzero=is_local_index_global_attn_nonzero,
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 589, in _compute_attn_output_with_global_indices
    attn_probs_only_global.transpose(1, 2), value_vectors_only_global.transpose(1, 2)
 (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
  allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py:149: UserWarning: 

Previous calculation was induced by CheckpointFunctionBackward. Traceback of forward call that induced the previous calculation:
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/usr/local/lib/python3.7/dist-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelapp.py", line 499, in start
    self.io_loop.start()
  File "/usr/local/lib/python3.7/dist-packages/tornado/platform/asyncio.py", line 132, in start
    self.asyncio_loop.run_forever()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
    self._run_once()
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
    handle._run()
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.7/dist-packages/tornado/platform/asyncio.py", line 122, in _handle_events
    handler_func(fileobj, events)
  File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 451, in _handle_events
    self._handle_recv()
  File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 434, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/usr/local/lib/python3.7/dist-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2828, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-74-3b02fb48d903>", line 1, in <module>
    trainer.train()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1269, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1762, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 2362, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 2206, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/led/modeling_led.py", line 1826, in forward
    is_index_global_attn,
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/checkpoint.py", line 211, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
 (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:109.)
  allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-74-3b02fb48d903> in <module>()
----> 1 trainer.train()
      2 #resume_from_checkpoint=True

6 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    147     Variable._execution_engine.run_backward(
    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    150 
    151 

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [12, 4096, 1]], which is output 0 of ViewBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

> /led/modeling_led.py", line 589, in _compute_attn_output_with_global_indices
> attn_probs_only_global.transpose(1, 2), value_vectors_only_global.transpose(1, 2)

I seem to fix the problem by changing the following, by detaching the torch before transpose operation:
from

/led/modeling_led.py", line 589, in _compute_attn_output_with_global_indices
attn_probs_only_global.transpose(1, 2), value_vectors_only_global.transpose(1, 2)

to 
/led/modeling_led.py", line 589, in _compute_attn_output_with_global_indices
attn_probs_only_global.detach().transpose(1, 2), value_vectors_only_global.detach().transpose(1, 2)

I'm getting exactly the same issue and it works fine if i don't specify a global attention mask, which leads me to believe its in the merge function in forward.

@Herais Detach would remove the tensors from the computation graph, wouldn't it be preferable to use .clone() instead? 

> @Herais Detach would remove the tensors from the computation graph, wouldn't it be preferable to use .clone() instead?

I think you are right, I was wondering about what detach does to the computational map, especially with the gradient accumulation set to True.  Using clone() also solves the versioning problem,  I would like to see how it does to predictions, will update. Thank you=)

I was testing global attention at the beginning of the document and the global attention at the beginning of each paragraph..

Hi, I also encountered this exact same bug when using the longformer for sequence classification. I had successfully trained this model previously before oversampling as well as a LED for summarization so I was  thrown off at first when I got it. I realized that the model kept throwing an error at the last batch  and when comparing the length of my data to my total batch size (batch_size=2 and gradient_accumulation=4) I realized that my last batch was a batch size of 1. I dropped a single row and then I was able to train the model successfully. I recently turned on gradient_checkpointing and ran it again (batch_size=7 and gradient_accumulation=4) and the error was triggered again when my last batch was 22/28 if you count gradient accumulation, so once again the batch size of 1 created the error.

Hi - is there a preferred fix for this?  I'm blocked on it right now.  I can just clone the offending tensor but want to make sure that's the preferred behavior.

Sorry I'm a bit lost on this issue. Could someone add a **minimum** reproducible code snippet that allows us to reproduce the error?

I think most people here are running into issues on the backward pass of the Longformer E-D.

I will share my code in a bit but I'm curious if the provided colab works.  If I were to reproduce my bug, it would be similar to the colab.

https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v

I tried cloning the offending tensor but it didn't seem to resolve it . Here's my stack trace

`(fresh) griadams@ip-172-31-19-18:~/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer$ pythons main.py -debug
Using GPUS --> 4...
Num GPUs --> 1
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Using native 16bit precision.
Starting training...
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
wandb: W&B syncing is set to `offline` in this directory.  Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.

  | Name  | Type                        | Params
------------------------------------------------------
0 | model | LEDForConditionalGeneration | 161 M 
------------------------------------------------------
161 M     Trainable params
0         Non-trainable params
161 M     Total params
647.378   Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:102: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:102: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Epoch 0:   0%|                                                                                                                                         | 0/16512 [00:00<?, ?it/s]/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/__init__.py:147: UserWarning: Error detected in BmmBackward0. Traceback of forward call that caused the error:
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/function.py", line 87, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore[attr-defined]
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 122, in backward
    outputs = ctx.run_function(*detached_inputs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 1816, in custom_forward
    return module(*inputs, is_global_attn, output_attentions)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 908, in forward
    attn_outputs = self.self_attn(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 719, in forward
    self_outputs = self.longformer_self_attn(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 277, in forward
    attn_output = self._compute_attn_output_with_global_indices(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 588, in _compute_attn_output_with_global_indices
    attn_output_only_global = torch.matmul(
 (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
  Variable._execution_engine.run_backward(
/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/__init__.py:147: UserWarning: 

Previous calculation was induced by CheckpointFunctionBackward. Traceback of forward call that induced the previous calculation:
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 137, in <module>
    run(args)
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 101, in run
    trainer.fit(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 325, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 93, in pre_optimizer_step
    result = lambda_closure()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 823, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 290, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/model.py", line 36, in training_step
    output = self.model(**batch, use_cache=False)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 2346, in forward
    outputs = self.led(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 2198, in forward
    encoder_outputs = self.encoder(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 1820, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 211, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
 (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:109.)
  Variable._execution_engine.run_backward(
[W python_anomaly_mode.cpp:104] Warning: Error detected in CheckpointFunctionBackward. Traceback of forward call that caused the error:
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 137, in <module>
    run(args)
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 101, in run
    trainer.fit(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 325, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 93, in pre_optimizer_step
    result = lambda_closure()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 823, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 290, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/model.py", line 36, in training_step
    output = self.model(**batch, use_cache=False)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 2346, in forward
    outputs = self.led(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 2198, in forward
    encoder_outputs = self.encoder(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/transformers/models/led/modeling_led.py", line 1820, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 211, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
 (function _print_stack)
Traceback (most recent call last):
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 137, in <module>
    run(args)
  File "/home/griadams/CompMedDsumEval/src/comp_med_dsum_eval/baselines/longformer/main.py", line 101, in run
    trainer.fit(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 499, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 738, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 434, in optimizer_step
    model_ref.optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 325, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 93, in pre_optimizer_step
    result = lambda_closure()
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 732, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 836, in training_step_and_backward
    self.backward(result, optimizer, opt_idx)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 869, in backward
    result.closure_loss = self.trainer.accelerator.backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 308, in backward
    output = self.precision_plugin.backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 62, in backward
    closure_loss = super().backward(model, closure_loss, optimizer, opt_idx, should_accumulate, *args, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 79, in backward
    model.backward(closure_loss, optimizer, opt_idx)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1275, in backward
    loss.backward(*args, **kwargs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/function.py", line 87, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore[attr-defined]
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/utils/checkpoint.py", line 138, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/home/griadams/miniconda3/envs/fresh/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 6144, 1]], which is output 0 of ViewBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

wandb: Waiting for W&B process to finish, PID 125448
wandb: Program failed with code 1. 
wandb: Find user logs for this run at: /efs/griadams/weights/default/wandb/offline-run-20210809_103548-2aq43v1n/logs/debug.log
wandb: Find internal logs for this run at: /efs/griadams/weights/default/wandb/offline-run-20210809_103548-2aq43v1n/logs/debug-internal.log
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /efs/griadams/weights/default/wandb/offline-run-20210809_103548-2aq43v1n`

First time I see an error message from PyTorch that says "Good luck!" haha. This will be complex then I guess

Okey, but I still don't have a code example that let's me reproduce this error I'm afraid :D 

The official colab here: https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing seems to work just fine

I'm getting this error as well using Longformer. This seems to be happening at the very end of my training. I'm assuming that it might be happening because there is a batch that has fewer number of examples than batch size. Maybe that could be something that should be tried? I'm currently investigating this issue on my end and I'll share more information if I find something.

Similar problem here. It happens at the end of the first epoch in my case,  when the batch size is smaller.

`File "/home/user/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1269, in train
    tr_loss += self.training_step(model, inputs)

  File "/home/user/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1780, in training_step
    loss.backward()

  File "/home/user/.conda/envs/transformers/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

  File "/home/user/.conda/envs/transformers/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [12, 4096, 1]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).`

This has to do with is_global_attn=True, else there is no problem.

EDIT : downgrading to torch 1.7 works for me

@patrickvonplaten @ibeltagy could you please advise?

Thanks,
Alessandro

Hi all,

The very same issue `RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation` occurred for me during a continued pre-training, i.e., warm-start a Lonformer model from the miniLMv2 checkpoint and contiue training the model with an MLM objective. I use the standard HF script, i.e., `run_mlm.py` provided in the examples. I have an ugly temporary solution down the lines, so please read, if interested.

I personally altered the tokenization pre-processing to provide custom global attention masks in a every separator token `
`, which I aim to use as a paragraph separator:

```python
def tokenize_function(examples):
            # Remove empty lines
            examples[text_column_name] = [
                line for line in examples[text_column_name] if len(line) > 0 and not line.isspace()
            ]
            batch = tokenizer(
                examples[text_column_name],
                padding=padding,
                truncation=True,
                max_length=max_seq_length,
                # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it
                # receives the `special_tokens_mask`.
                return_special_tokens_mask=True,
            )
            # provide custom global attention mask
            batch.data['global_attention_mask'] = [[1 if token_id in [tokenizer.cls_token_id, tokenizer.sep_token_id]
                                                    else 0 for token_id in seq] for seq in batch.data['input_ids']]
            return batch
```

After 1186 training steps, the aforementioned error occurred...

# Solution

In order to be able to train the model -until there is a proper solution- I "hacked" the `Trainer` class in the `train` function, wrapping this part of the code in a try-except block:

https://github.com/huggingface/transformers/blob/010965dcde8ce9526f6a7e6e2c3f36276c153708/src/transformers/trainer.py#L1277-L1286

I copy-pasted the `trainer.py` in a new personal file `mytrainer.py` and did the following minor update, which moves to the next mini batch (step), while it also zero-out the gradients:

```python
try:
    if (
        ((step + 1) % args.gradient_accumulation_steps != 0)
        and args.local_rank != -1
        and args._no_sync_in_gradient_accumulation
    ):
        # Avoid unnecessary DDP synchronization since there will be no backward pass on this example.
        with model.no_sync():
            tr_loss += self.training_step(model, inputs)
    else:
        tr_loss += self.training_step(model, inputs)
except:
    tr_loss += 0
    logger.warning(f'Issue at training step {step} !!! Training continues...')
    model.zero_grad()
    continue
```

I re-run the code, which started from the latest checkpoint `checkpoint-1100` and passed the tricky part successfully:

```
09/11/2021 20:03:34 - WARNING - mytrainer - Issue at training step 1187 !!! Training continues...
```

So far there is not further issue and the training loss is keep decreasing 😄 

```
{'loss': 4.12, 'learning_rate': 9.724264705882353e-06, 'epoch': 2.19}
{'loss': 4.0383, 'learning_rate': 9.632352941176471e-06, 'epoch': 2.36}
{'loss': 3.8487, 'learning_rate': 9.448529411764707e-06, 'epoch': 2.7}
{'eval_loss': 3.653672456741333, 'eval_runtime': 61.6433, 'eval_samples_per_second': 8.111, 'eval_steps_per_second': 1.022, 'epoch': 3.0}
```


@iliaschalkidis thanks for the update. Even thought this goes around the issue, it looks like there is something fundamentally wrong with the current implementation? I hope that @patrickvonplaten or @ibeltagy could comment on this 🙏

@aleSuglia that's absolutely true and that's why I describe my solution as a "dirty" hack trying to avoid seg faults by skipping a few param updates when this weird error occur.

Let's hope for a real solution in the underlying issue.

@iliaschalkidis actually, now that you have a try/except in place for that issue, why don't you serialise the faulty batch and share it in a Colab so that @patrickvonplaten or @ibeltagy can play around with it? I think that would be terribly useful to debug!

The problem comes from LongformerSelfAttention for longformer. If this happens for another model, its probably from its SelfAttention module too.

@iliaschalkidis any chances to get the faulty batch out of your training?

Not yet, sorry. I'm currently (pre-)training the models. I'll try to add a save functionality in the `except` handling and save a tricky batch later this week. 

FWIW I agree with @benderama3 ; I also have a feeling that this inconsistency is a by-product of the really complicated attention code, i.e., there are multiple `reshape` and `gather` -like computations with dynamically inferred shapes :P 

Some other edge cases that I've spotted:

```
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 1024, 46]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
    Variable._execution_engine.run_backward(
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 1024, 37]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 1024, 43]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 1536, 73]], which is output 0 of ViewBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
```

@patrickvonplaten , 

Here's the Colab that I got this problem.  Finally got a chance o strip down the notebook code. The error comes up  5 to 10 minutes into training.

[https://colab.research.google.com/drive/1ZoYJaJZmhygKBEAb5gPm2MaySdFOqgbo?usp=sharing](url)

Error message  was:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [12, 16384, 16]], which is output 0 of ViewBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

@Herais thanks for sharing your notebook. I've simplified it to make it easier for people to reproduce the bug and dissect the actual model code: https://colab.research.google.com/drive/13rKxs6Ype0kDEBlnywsGynE2zpzv2CR-#scrollTo=h7k8m9OV8xIR

cool, thank you.

@patrickvonplaten @ibeltagy I'm happy to send a PR with the fix. There are some in-place operations that require `clone` to work. Let me know if you're interested!

@aleSuglia and @Herais thanks for diving into this issue! We would happily welcome a PR to see the code changes and what needs to be fixed. 

Thank you!

@LysandreJik just sent it. I hope it can be merged asap :)