Memory issue with OPT models when given long input sequences
Author: hxiaoyang
Pull request: no

### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-4.18.0-305.45.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@gante @patrickvonplaten @Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When the input sequence is long, `opt-30b` and `opt-66b` gave the following memory error:
```
Traceback (most recent call last):
  File "/home/x/script.py", line 189, in <module>
    main()
  File "/home/x/script.py", line 186, in main
    compute_logprobs(int(sys.argv[3]), int(sys.argv[4]), sys.argv[2], model_name=sys.argv[1])
  File "/home/x/script.py", line 170, in compute_logprobs
    logits = model(input_ids).logits
  File "/sw/pkgs/arc/python/3.9.12/pytorch/1.12.1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 929, in forward
    outputs = self.model.decoder(
  File "/sw/pkgs/arc/python/3.9.12/pytorch/1.12.1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 693, in forward
    layer_outputs = decoder_layer(
  File "/sw/pkgs/arc/python/3.9.12/pytorch/1.12.1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 346, in forward
    hidden_states = self.fc1(hidden_states)
  File "/sw/pkgs/arc/python/3.9.12/pytorch/1.12.1/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/accelerate/hooks.py", line 148, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/x/hf/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 256, in forward
    out = bnb.matmul(x, self.weight, state=self.state)
  File "/home/x/hf/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 410, in matmul
    return MatMul8bitLt.apply(A, B, out, state)
  File "/home/x/hf/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 328, in forward
    out32, Sout32 = F.igemmlt(C32A, state.CxB, SA, state.SB)
  File "/home/nickhu/hf/lib/python3.9/site-packages/bitsandbytes/functional.py", line 1332, in igemmlt
    out, Sout = get_transform_buffer(
  File "/home/x/hf/lib/python3.9/site-packages/bitsandbytes/functional.py", line 294, in get_transform_buffer
    return init_func((rows, cols), dtype=dtype, device=device), state
RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 44.37 GiB total capacity; 42.44 GiB already allocated; 40.50 MiB free; 43.23 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

To reproduce, simply call `model(input)` on some long input sequence.

Additional information:
- My total GPU memory: `2*42 GB` for `opt-30b` and `4*42 GB` for `opt-66b`.
- Also, the same inputs did not cause errors in the smaller opt models.

### Expected behavior

No `CUDA out of memory` error.

How can I fix this? Any help is appreciated!

Comments:
```
    compute_logprobs(int(sys.argv[3]), int(sys.argv[4]), sys.argv[2], model_name=sys.argv[1])
```
    
What are those values, and what do they correspond to ?

Hi @xiaoyangnickhu 👋 This seems business as usual for large models and a forum question for the PyTorch experts.

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗

My bad... Thanks!