RuntimeError: leaf variable has been moved into the graph interior
Author: bing0037
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.1
- Platform: GPU
- Python version: python 3.6.9
- PyTorch version (GPU): torch==1.4.0

### Who can help
@TobiasLee  @julien-c

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

- Model that I am using: Bert
- Code Modification: 
Actually I didn't change any code, I just used the newest code in master branch.
I am trying to use run_bertology.py to carry out head pruning by using the following script:
- The script that I used:
```
export GLUE_DIR=glue_data
export TASK_NAME=RTE
CUDA_VISIBLE_DEVICES=3 python run_bertology.py \
  --model_name_or_path bert-base-uncased \
  --try_masking \
  --task_name $TASK_NAME \
  --data_dir $GLUE_DIR/$TASK_NAME/ \
  --max_seq_length 128 \
  --output_dir output_headpruning_bert/${TASK_NAME} \
  --overwrite_output_dir  
```
But I got the error:
```
Traceback (most recent call last):
  File "run_bertology.py", line 449, in <module>
    main()
  File "run_bertology.py", line 445, in main
    prune_heads(args, model, eval_dataloader, head_mask)
  File "run_bertology.py", line 213, in prune_heads
    args, model, eval_dataloader, compute_entropy=False, compute_importance=False, head_mask=head_mask
  File "run_bertology.py", line 103, in compute_heads_importance
    loss.backward()  # Backpropagate to populate the gradients in the head mask
  File "/home/bil19003/anaconda3/envs/pytorch_huggingface/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/bil19003/anaconda3/envs/pytorch_huggingface/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: leaf variable has been moved into the graph interior
```

I got similar results to #3895, seemingly the problem still exists. Could you please help?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@bing0037 I'm running into the same problem. Did you find a fix for it?