Use Accelerate in `from_pretrained` for big model inference
Author: sgugger
Pull request: yes

# What does this PR do?

This PR is a first draft for using the newly released big model inference APIs from Accelerate inside `from_pretrained`. For now it does this with the option `low_cpu_mem_usage=True` and:
- instantiates the model inside the context manager to initialize empty weights (faster and less memory-intensive)
- has the same behavior as before if no `device_map` is passed
- otherwise will put each model weight on the specified device as the loading is done and properly sets the hook so that the model can still be used normally. As with Accelerate, `device_map="auto"` will auto-infer a proper device map with the available GPU(s) RAM and CPU RAM.

This PR is just a first step, there is a bit more cleanup to do, namely:
- put the utils flagged as belonging in Accelerate there and once a new release of Accelerate is done, use them
- clean up some old code (like move_model_to_meta_device)

Example of use:

```py
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", revision="sharded", device_map="auto")

tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
inputs = tokenizer("Task: copy but say the opposite. PSG won its match against Barca.", return_tensors="pt")
inputs = inputs.to(0)

output = model.generate(inputs["input_ids"])
tokenizer.decode(output[0].tolist())
```

Still missing:
- [ ] integration test
- [x] doc
- [ ] add the "block" attribute to more model classes

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Not necessarily linked to this PR, but in general the following code fails:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", low_cpu_mem_usage=True)

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
inputs = tokenizer("Task: copy but say the opposite. PSG won its match against Barca.", return_tensors="pt")
#inputs = inputs.to(0)

output = model(inputs["input_ids"])
```

with: 
```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, meta and cpu!
```

Should we maybe throw a nice warning in `from_pretrained(...)` that certain parameters are on meta and need to be manually initialized?

> Should we maybe throw a nice warning in from_pretrained(...) that certain parameters are on meta and need to be manually initialized?

Warning, no, but assert yes - it's abnormal if a model returned with weights that are on meta. The whole meta device things is a behind the scenes hack and it shouldn't bleed out to user-land, IMHO.

Thanks a lot for your reviews @patrickvonplaten and @stas00 !
Here are a few answers to your general comments.

> Should we maybe throw a nice warning in from_pretrained(...) that certain parameters are on meta and need to be manually initialized?

The model should be fully initialized outside of the meta device. I haven't checked yet models with randomly initialized heads (as the primary goal is inference) but will make sure this is fixed before merging.

> Also please checkout out a related interesting new development at NVIDIA with GPUDIRECT https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html which would allow allocating tensors on disc.
> 
> Tunji is working on this feature in Deepspeed, this would allow `tensor.to(nvme)` and then use it as a normal tensor.

Once it's landed I'd be very interested in using it when `DeepSpeed` is available. Do you also know if they have plans to make their API to prefetch weights offloaded on the CPU/disk somewhat abailable?

> Additionally Tunji and I are working on a universal checkpoint for huge models which doesn't contain any topology data and can shrink/expand on the fly. This is based on my earlier proposal for a checkpoing format where each tensor is a separate file.
> 
> The problem with all other current approaches is that they require TBs of CPU memory for models like 176B if you have to manipulate optim_states, etc.

Note that in this instance passing a `device_map` only works for model inference (not training). The best way to train large models is still to use DeepSpeed directly.



>> Tunji is working on this feature in Deepspeed, this would allow tensor.to(nvme) and then use it as a normal tensor.
>Once it's landed I'd be very interested in using it when DeepSpeed is available. Do you also know if they have plans to make their API to prefetch weights offloaded on the CPU/disk somewhat abailable?

@tjruwase, just a heads up - as you work on these new features - could you please consider making the offload/prefetch API public so that the HF Trainers and the core could make a direct use of those? Thank you!

Though I understand that it's deeply tied into the tracing mechanism, which is currently inseparable from the pre-fetch mechanism - the tracing mechanism figures out which params to prefetch and when. But perhaps we can discuss with Sylvain how he envisions using it.