issue when training Falcon 7b with deepspeed due to cache
Author: Epliz
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Linux-4.18.0-477.13.1.el8_8.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: 4x nvidia A10
- Using distributed or parallel set-up in script?: DeepSpeed


### Who can help?

@ArthurZucker , @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine-tune Falcon 7b with deepspeed Zero 3, cpu offloading and gradient checkpointing.
I get the following error:
```
  File "<venv>/lib64/python3.8/site-packages/transformers/models/falcon/modeling_falcon.py", line 1279, in forward                                                                 
    transformer_outputs = self.transformer(                                                                                                                                                               
  File "<venv>/lib64/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl                                                                                 
    result = forward_call(*args, **kwargs)                                                                                                                                                                
  File "<venv>/lib64/python3.8/site-packages/transformers/models/falcon/modeling_falcon.py", line 1189, in forward                                                                 
    presents = self._convert_cache_to_standard_format(presents, batch_size) 
  File "<venv>/lib64/python3.8/site-packages/transformers/models/falcon/modeling_falcon.py", line 954, in _convert_cache_to_standard_format
    batch_size_times_num_heads, kv_length, head_dim = past_key_value[0][0].shape                                                                                                                          
IndexError: tuple index out of range
```
However, I see that use_cache should have been set to False according to 
```
use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
```

But the Falcon modeling code at https://github.com/huggingface/transformers/blob/244a53e0f6a8d95d429559cfc49a07a4e85cc680/src/transformers/models/falcon/modeling_falcon.py#L1193 is written in a way that it will go in the body of the if even though use_case was disabled, due to the fact the use_cache is set to False when using checkpointing AFTER that `presents` is set to `()`.

Solution is probably to set use_cache to False earlier if needed, e.g. after L1069 directly.

### Expected behavior

It should work.

Comments:
never mind; looks like it has been fixed in main