Add `accelerate` support for LongT5 models
Author: pszemraj
Pull request: yes

Signed-off-by: peter szemraj <peterszemraj@gmail.com>

# What does this PR do?

This PR adds `accelerate` support for the longT5 models (i.e., make it possible to use `device_map="auto"`), so these models can be loaded in 8bit using load_in_8bit=True.

This helps enable inference with trained/fine-tuned SoTA long summarization models using limited memory :relaxed:  

Took inspiration from reviewing similar PRs for other models: #19912 and #19927 

cc @sgugger 

## test results

I made [a Colab notebook](https://colab.research.google.com/gist/pszemraj/6ea0a3046452fc51061f4bde2df0aa77/testing-accelerate-long-t5-tglobal-base-16384-book-summary.ipynb) that clones the branch from my fork to demo the `load_in_8bit=True` working. Everything else is the same for comparison purposes (_except the function that says the model size_) as [the fp32/standard notebook](https://colab.research.google.com/gist/pszemraj/d9a0495861776168fd5cdcd7731bc4ee/example-long-t5-tglobal-base-16384-book-summary.ipynb) listed on [my fine-tuned model card](https://huggingface.co/pszemraj/long-t5-tglobal-base-16384-book-summary).

I also ran the tests for `longT5` locally: 

```bash
$ python -m pytest -n auto --dist=loadfile -s -v tests/models/longt5/test_modeling_longt5.py 

( ... many things here ...)

=================================================== 196 passed, 58 skipped, 118 warnings in 30.49s ===================================================
```


Comments:
cc @KMFODA for inputs on tests & more :crossed_fingers: 

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the feedback & good catch on the Colab! I've updated the notebook - will run and resolve the slow tests/accelerate items later today/tomorrow and revert back 👌

Hey @pszemraj ! 
How is the integration going 💪 ? Let me know if I can help at some point to debug / make the tests pass ;) !

Hi @pszemraj !
Is it ok if I try to take over the PR? this addition could be very nice to the lib! Let me know what do you think :) 

Hey! let me give it a stab today (I was sick for a week) if you don't see anything by tomorrow, feel free to take it home!

Email | ***@***.*** 
On 12/6/2022 8:54:39 AM, Younes Belkada ***@***.***> wrote:
Hi @pszemraj [https://github.com/pszemraj] !
Is it ok if I try to take over the PR? this addition could be very nice to the lib! Let me know what do you think :)
—
Reply to this email directly, view it on GitHub [https://github.com/huggingface/transformers/pull/20341#issuecomment-1338921676], or unsubscribe [https://github.com/notifications/unsubscribe-auth/AR3GSMFN4MP444ZC72B4EN3WL3WL7ANCNFSM6AAAAAASGEAOLE].
You are receiving this because you were mentioned.Message ID: ***@***.***>
[31e14b4b-28c3-4714-8081-803278962750]

@younesbelkada hey - was trying to get the tests to pass and evaluate further but unfortunately the machine I _do_ have access to a GPU on and can work this was running into some install issues with the `dev` dependencies for `pytest` etc

If you're willing to finish this, that would probably be easiest 😅  I'll add the line for accelerate as you suggested and rebase as per the contrib guidelines, feel free to take whatever you find useful :) 

Thanks a lot @pszemraj for your great efforts, will have a look ASAP ;) this is definitely in my TODO list

thanks so much! I see you pushed so I will leave you to it (but feel free to let me know if questions or you need me to change anything on my end)

then we can get [this bad boi](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary) usable on free Colab runtimes :) 

Thanks for taking it home @younesbelkada! and thanks for the review @sgugger. Happy to help :)