RWKV's "RNN mode" results in unchecked memory growth, causing OOM
Author: Vectorrent
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-6.5.3-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

@sgugger @ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run this code, and your GPU will go OOM very quickly:
```
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "RWKV/rwkv-4-430m-pile"

string = "Lorem ipsum dolor sit amet, consectetur adipiscing elit."


tokenizer = AutoTokenizer.from_pretrained(
    model_name,
)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    output_hidden_states=True,
    device_map="auto",
)

state = None

while True:
    inputs = tokenizer(string, return_tensors="pt")
    outputs = model(inputs["input_ids"][:, :2].to(model.device.type), state=state)
    state = outputs.state
    time.sleep(1)
```

### Expected behavior

According to @BlinkDL (the creator of RWKV), using RNN mode should not cause unchecked memory growth like it does here. Yet when I accumulate the model's state into itself over and over again, VRAM grows indefinitely, until CUDA finally crashes.

I can't tell if I'm doing something wrong, but my code is nearly identical to @sgugger's documentation: https://huggingface.co/docs/transformers/model_doc/rwkv

Thanks in advance for any help you can provide!

Comments:
Hey! Thanks for opening a detailed issue! If you are training it would make sense to have an increasing memory usage! Coud you try with `model.eval()` and `with torch.no_grad()`? 


I'm not training, but `torch.no_grad()` does seem to work! Y'all HF guys are always so quick to respond; I really appreciate the tip! I'm going to give this a day or two (because memory has not _technically_ stopped increasing, though it has slowed-down immensely). If all goes well, I'll close the issue.

Thanks again!

You need to apply `detach()` to the state like for a traditional RNN (`state = outputs.state.detach()` in your code) otherwise you compute gradients over all your steps which indeed takes a lot of memory 😅 

Well, I couldn't really figure how to how to use `detach()`, since every attempt ended in some variant of this error:
```
vtx-bot-1   | Traceback (most recent call last):
vtx-bot-1   |   File "/src/main.py", line 7, in <module>
vtx-bot-1   |     import machine
vtx-bot-1   |   File "/src/machine.py", line 9, in <module>
vtx-bot-1   |     from lab import scratch
vtx-bot-1   |   File "/src/lab/scratch.py", line 25, in <module>
vtx-bot-1   |     state = outputs.state.detach()
vtx-bot-1   | AttributeError: 'list' object has no attribute 'detach'
```
Nevertheless, the `torch.no_grad()` method does work. Sadly though, the model degenerates so quickly, I'm not really sure how to make effective use of RNN mode at this point.

I appreciate the advice though. Issue resolved!

> Nevertheless, the `torch.no_grad()` method does work. Sadly though, the model degenerates so quickly, I'm not really sure how to make use of RNN mode at this point.

Looks like more bugs. All inference of RWKV are in RNN mode and works perfectly well.

Please try official rwkv pip package: https://pypi.org/project/rwkv/