Cannot run Deepspeed inference of GPT-Neo with low_cpu_mem_usage enabled
Author: Jiyeon1230
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: Ubuntu 18.04.6 LTS
- Python version: Python 3.6.9
- PyTorch version (GPU?): 1.10.0+cu113
- Tensorflow version (GPU?): n/a
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Using Deepspeed 0.5.0

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library: 

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 --> @stas00

## Information

Model I am using (Bert, XLNet ...): EleutherAI/gpt-neo-1.3B

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. write the following code which is referred from https://www.deepspeed.ai/tutorials/inference-tutorial/#end-to-end-gpt-neo-27b-inference
```ruby
import os
import deepspeed
import torch
from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))

model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B", low_cpu_mem_usage=True)
#model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", low_cpu_mem_usage=True)

tokenizer_i = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
#tokenizer_i = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")

generator = pipeline('text-generation', model=model, device=local_rank, tokenizer=tokenizer_i)

generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.float,
                                           replace_method='auto')

string = generator("DeepSpeed is", do_sample=True, min_length=50)
if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(string)

```
2. execute the code using Deepspeed as the following command
```shell
deepspeed --num_gpus 1 test.py
```
3. failed to execute
```log
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    replace_method='auto')
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/__init__.py", line 285, in init_inference
    quantization_setting)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/inference/engine.py", line 70, in __init__
    self._apply_injection_policy()
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/inference/engine.py", line 148, in _apply_injection_policy
    self.quantize_groups))
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 308, in replace_transformer_layer
    _replace_policy=policy)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 404, in replace_module
    replaced_module, _ = _replace_module(model, policy)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 429, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 429, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 425, in _replace_module
    layer_id))
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 301, in replace_fn
    layer_id=layer_id)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 224, in replace_with_policy
    dense_w = transpose(dense_w)
  File "/home/ubuntu/env/lib/python3.6/site-packages/deepspeed/module_inject/replace_module.py", line 218, in transpose
    data.view(-1).copy_(data.transpose(-1, -2).contiguous().view(-1))
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Run deepspeed inference successfully without any failure
<!-- A clear and concise description of what you would expect to happen. -->



## Comment
Hi all,

I'm trying to run GPT-Neo inference using Deepspeed. Because of my system environment, I need to reduce the peak RAM usage, so added the argument, low_cpu_mem_usage as True to from_pretrained. But it gets failed as I described.
I'm filing this case to HF because removing low_cpu_mem_usage or changing model to gpt-j-6B, it succeed to run.
Could you advise for this problem? If low_cpu_mem_usage feature doesn't support GPT-Neo, it would be appreciated if you say so.

Thanks,

Comments:
Deepspeed Inference is not a completed product AFAIK, and it's not yet integrated into Transformers because of that. 

As you can see from the trace the `transformers` library is not being used. So please re-file this issue with Deepspeed and tag @RezaYazdaniAminabadi.

Any reason why you are not using Deepspeed ZeRO Inference? https://huggingface.co/transformers/master/main_classes/deepspeed.html#deepspeed-zero-inference


Deepspeed Inference and Deepspeed ZeRO Inference are 2 completely different things. The former uses Tensor parallelism - the 2nd is ZeRO sharding.

@stas00 
Thanks for your comment. I filed the case to transformers because the issue is not reproduced with low_cpu_mem_usage disabled. Do you think it needs to be handled by Deepspeed even so?

I'm eventually trying to load a bigger GPT-Neo like model which doesn't fit to one GPU. That's why Deepspeed Inference is used. Appreciate your advise though. 

@stas00 
I have no idea which diff fixes the issue, but it fixed after I update Deepspeed Inference code to the latest. I really appreciate your advise since I did only focus on transformers. Thanks!!!

It's an actively developed new product, so someone must have reported this issue recently and it got fixed.

I'm glad this is now working for you, @Jiyeon1230 

> I'm eventually trying to load a bigger GPT-Neo like model which doesn't fit to one GPU. That's why Deepspeed Inference is used. 

And I repeat again that Deepspeed ZeRO is already well tested scalability solution that you can use today to use models larger than one GPU and it's fully integrated into Transformers. It has additional features like CPU Offload, which scales better, and which I don't think Deepspeed Inference supports at the moment. See the doc link in my last comment.

But, of course, it's up to you what you use.

@stas00 
Oh, sure! I may have misunderstood about Deepspeed ZeRO. I'll definitely look into it. Thanks for your advise.