run_generation.py GPT2 model only using 1st GPU, OOM error
Author: randywreed
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

GPT2-xl (pre-trained)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
Run in jupyter notebok script below, GCP T4x2

## To reproduce

Steps to reproduce the behavior:
```
prompt_text="If God is defined as something that is all powerful and all knowing, a strong artificial intelligence might be an actual God. If this happens the implications for religion are"
num_of_responses=15
!python transformers/examples/text-generation/run_generation.py --model_type GPT2 --model_name_or_path /spell/GPT2Model/GPT2Model/ --length 1000 --stop_token '
' --temperature .7 --repetition_penalty 85 --prompt "$prompt_text" --num_return_sequences $num_of_responses
```
Nvidia-smi during run only reports gpu1 using memory  gpu2 sits at 11m and never moves
Ultimately program errors out with OOM error:

Traceback (most recent call last):
  File "transformers/examples/text-generation/run_generation.py", line 268, in <module>
    main()
  File "transformers/examples/text-generation/run_generation.py", line 237, in main
    num_return_sequences=args.num_return_sequences,
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1148, in generate
    model_specific_kwargs=model_specific_kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py", line 1190, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 615, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 499, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 236, in forward
    use_cache=use_cache,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_gpt2.py", line 191, in forward
    present = torch.stack((key.transpose(-2, -1), value))  # transpose to have same shapes for stacking
RuntimeError: CUDA out of memory. Tried to allocate 84.00 MiB (GPU 0; 14.76 GiB total capacity; 13.69 GiB already allocated; 37.44 MiB free; 13.93 GiB reserved in total by PyTorch)
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- `transformers` version: 2.9.0
- Platform: Linux-5.3.0-1018-gcp-x86_64-with-glibc2.10
- Python version: 3.8.2
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: 
- Using distributed or parallel set-up in script?: 

Comments:
I don't think we currently support multi-gpu in this script.

16 GB VRAM might not be enough to allow batching of generated texts.

Actually, if you install apex and are able to cast the model as FP16 using `model.half()`, you can get a batch size of up to 30 for the 1.5B. (nvidia-smi is reporting 14GB VRAM used on a single T4)

My tool is slightly different than the script, but I'll be sure to include bulk 1.5B generation as a documentation example.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.