Remove .data usages in optimizations.py
Author: alanwaketan
Pull request: yes

# What does this PR do?

.data usages is deprecated in recent releases of PyTorch. See https://github.com/pytorch/pytorch/issues/91093#issuecomment-1397317273

This change replace all .data usages in optimizations.py with modern alternatives.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@connor-henderson @stas00 

Comments:
@muellerzr  the usage of the `.data` will greatly slow down pytorch/xla on nightly, we were hoping we can fix this issue before the next release.

_The documentation is not available anymore as the PR was closed or merged._

This is a very old and deprecated implementation since it doesn't even follow the AdamW algorithm exactly. One should use `torch.optim.AdamW` instead, which also has a fused version since pt-2.0.0 which is almost as fast as apex's fused AdamW. So really you shouldn't be using this version anyway.

The only reason it was kept is for BC for those who rely on exact results remaining exact after new `transformers` versions are released, otherwise we would have just replaced it with `torch.optim.AdamW` in the first place.

p.s. no objections though to making it better...

@stas00 Thanks for the reply. How about the adafactor then?

oh, sorry, I didn't see it was Adafactor too. It's hard to see from the diff as it doesn't show the class names.

This Adafactor is being used for sure, but its implementation is super old as well. So certainly it'd be a blessing to bring it up to more modern code standard.

@stas00 Do you mind give this pr a review? Thanks.

Thanks for your PR. Just to be sure though, is this all going to work with PyTorch 1.8+? 1.8 is the minimum version we offically support at the moment (for a couple more weeks at least, then 1.9 starting mid-June).

I'm almost 100% sure it is the case. the whole direct `.data` usage deprecation is a few years old at least.

Let me quickly test it with pt-1.8

```
$ pytest tests/optimization/test_optimization.py -k test_adafactor
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.8, pytest-7.3.1, pluggy-0.13.1
rootdir: /mnt/nvme0/code/huggingface/transformers-master
configfile: setup.cfg
plugins: timeout-1.4.2, typeguard-2.12.1, flakefinder-1.0.0, forked-1.3.0, monitor-1.6.0, hypothesis-6.47.0, instafail-0.4.2, xdist-2.2.1
collected 3 items / 2 deselected / 1 selected

tests/optimization/test_optimization.py .                                                                                          [100%]

================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED tests/optimization/test_optimization.py::OptimizationTest::test_adafactor
============================================== 1 passed, 2 deselected, 4 warnings in 0.16s ===============================================

$ pt-ver
pt=1.8.2, cuda=11.1, nccl=2708
```

At least the Adafactor test that we have is passing.

Thanks @sgugger and @stas00 for reviewing the changes.