failure to use conda-forge apex  with torch1.6 and --amp_backend='apex' + --fp16_opt_level O1 
Author: XiangLi1999
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.2.0
- Platform: Linux-4.4.0-1111-aws-x86_64-with-debian-stretch-sid
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 -->

 @stas00 

## Information

Model I am using: BART

The problem arises when using:

torch 1.6 + conda-forge apex w/ --fp16 --amp_backend='apex' + --fp16_opt_level O1 to run finetune.py

```
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 429, in fit
    self.accelerator_backend.setup(model)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 53, in setup
    model = self.trainer.precision_connector.connect(model)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/precision_connector.py", line 78, in connect
    model, optimizers = self.backend.connect(model, self.trainer.optimizers)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/plugins/apex.py", line 38, in connect
    self.trainer.reinit_scheduler_properties(optimizers, self.trainer.lr_schedulers)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/pytorch_lightning/trainer/optimizers.py", line 143, in reinit_scheduler_properties
    scheduler.__class__.__mro__[idx].__init__(scheduler, optimizer)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 74, in __init__
    self.optimizer.step = with_counter(self.optimizer.step)
  File "/home/ubuntu/anaconda3/envs/nightly/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 56, in with_counter
    instance_ref = weakref.ref(method.__self__)
AttributeError: 'function' object has no attribute '__self__'
```


The tasks I am working on is:
* [ ] Finetune BART on XSUM 

## To reproduce

Steps to reproduce the behavior:
It is the same issue mentioned in https://github.com/huggingface/transformers/issues/8403#issuecomment-724787083
If you control find "\_\_self\_\_"


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
@XiangLi1999, thank you for making a separate issue for this.

(note: I edited your post to format the error traceback and also edited the link to point to the relevant comment (https://github.com/huggingface/transformers/issues/8403#issuecomment-724787083) - if you click in the right upper corner of the comment - you will see an option to copy a link to that comment and not just thread.)

Yes, I saw that error but didn't have a chance to try to understand the cause at that time - I had a closer look now and this seems to be a `pytorch-lightning` bug - so you might have to ask via their issue tracker.

I can suggest two possible solutions:

1. Download pytorch-nightly which has the leak fixed so you can use native amp no problem. 

`pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html -U`

2. There is also a relatively new `finetune_trainer.py` in the same directory, which uses HF trainer. Have a look and perhaps it'd work better for you.

Please let me know if one of the proposed solutions addresses your needs.

And feel free to file a bug report with pytorch-lightning if you'd like to follow that use case through.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.