How to parallelize large model(like t5-11b) at transformer version 3.0.2
Author: ZeyiLiao
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-189-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.13
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <Yes>
- Using distributed or parallel set-up in script?: <Try to>

### Who can help?

@stas00 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

git clone https://github.com/GXimingLu/neurologic_decoding.git

I add file at ``./neurologic_decoding/seq2seq/decode.py `` a bit and try to parallelize large-model
```
    model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)

    if model_name in ["t5-3b","t5-11b"]:
        print(f'{model_name} is parallizaing')
        model.parallelize()
```


But error said that:
```
torch.nn.modules.module.ModuleAttributeError: 'T5ForConditionalGeneration' object has no attribute 'parallelize'
```

### Expected behavior

Parallelize T5-3b/11b at transformer 3.0.2. I know the `parallelize` function may not work in the 3.0.2 version, or maybe I should use deep-speeds (If so, can u recommend some tutorials)?

Comments:
the naive `parallelize` was never supported by t5. just gpt2 and bart. and it'll be removed soon altogether as there are better solutions.

edit: that was a wrong statement - only gpt2 and t5 have every been supported.

you can do 2 things:

1. `accelerate` will automatically do naive parallelization for you https://github.com/huggingface/accelerate
2. and of course deepspeed-zero is likely to perform faster as it'll utilize the gpus more efficiently https://huggingface.co/docs/transformers/main/main_classes/deepspeed#deepspeed-trainer-integration

if you are just doing only inference also check out: deepspeed-inference

As this is not a bug I'm closing this Issue, but please don't hesitate to ask questions.

@stas00 , yeah, thanks a lot!

And I wanna clarify that I want to use Version 3.0.2 transformers (since the code is using some specific function at V 3.0.2 and it would need a large amount of change if adapted to the latest version) and just check that `src/transformers `don't have deepspeed.py.

wait, I got it wrong. it is gpt2 and t5 that used to support `parallelize` - my apologies - I implemented bart support long time ago but it was never merged as we decided not to continue with this approach. So t5 should work just fine.

https://github.com/huggingface/transformers/blob/a26114777ee1c2802e91bd9cb26a3b39974d52ba/src/transformers/models/t5/modeling_t5.py#L209-L216

I think perhaps you're trying to use a really old transformers version that haven't yet had t5 support for naive `parallelize` added.

As you can see this is really old and it indeed doesn't have `parallelize`

https://github.com/huggingface/transformers/blob/v3.0.2/src/transformers/modeling_t5.py

Perhaps you can move the function that you need from that old version to the modern code? 

You can of course code your own integration with Deepspeed if you really have to. The Deepspeed site has lots of examples on how to do that.

Any modern solutions like accelerate will require the current `transformers` versions.

@stas00 Okay! 

So do you have some recommendations for how to find the corresponding version of the function at older version efficiently?
Like for function```self._use_cache``` at transformers==3.0.2

In this case I think this is just `use_cache=True` here:

https://github.com/huggingface/transformers/blob/4c2e983f44ce4d3b9c8502d42cc568e45897bd15/src/transformers/generation_utils.py#L892

from the original:
https://github.com/huggingface/transformers/blob/v3.0.2/src/transformers/generation_utils.py#L39

Please let me know if that's what you're after. and if not please show me which specific code has been moved or changed.