Correct Falcon code in github does not match Falcon's checkpoint
Author: afcruzs
Pull request: no

### System Info

Transformers version: 4.31.0


### Who can help?

 @ArthurZucker @younesbelkada @Rocketknight1

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


I see there's a falcon implementation added by the HF team in the transformer github repo. What's the intent for this code? what model in the hub does actually uses it? 

The official model as far as I can tell in the hub (https://huggingface.co/tiiuae/falcon-7b) uses different (outdated?) code that is included in the checkpoint itself (`modeling_RW.py`). I've found `modeling_RW.py` has a number of problems to key/value caching (either bad model outputs, or key/value caching might not get used at all) that has been fixed already in the current code in github, this has been observed by others in the [model discussions](https://huggingface.co/tiiuae/falcon-7b/discussions/17), unfortunately without an official response from the Falcon team.

That said, I am not fully sure the falcon model in the hub is compatible with the code in github, I do get warnings if I try to use it (`You are using a model of type RefinedWebModel to instantiate a model of type falcon. This is not supported for all configurations of models and can yield errors.`).

**Concrete question**: what is the intended usage of the current falcon code (`src/transformers/models/falcon`) in the transformers repo? Is it compatible with the official falcon models?

Steps to reproduce - loading with FalconForCausalLM
```
# Load model directly
from transformers import FalconForCausalLM
DEVICE = 'cuda'
# This gives me: You are using a model of type RefinedWebModel to instantiate a model of type falcon. This is not supported for all configurations of models and can yield errors.
model = FalconForCausalLM.from_pretrained(<path of the tiiuae/falcon-7b in the hub downloaded locally>).to(DEVICE)
```

Steps to reproduce - loading with Auto
```
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-7b", trust_remote_code=True).to('cuda')
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b")
input_text_tokens = tokenizer("Hello world, this is the story of Bob, a", return_tensors="pt").input_ids.to('cuda')

# This actually *does not* use KV caching, due to a name bug from "past_key_values" to "past" in `prepare_inputs_for_generation`
# If one attempts to fix this, shape errors might occur. If those are fixed, the output is gibberish due to position ids not correctly 
# into the RoPE embeddings. See this for more details: https://huggingface.co/tiiuae/falcon-7b/discussions/17
with torch.no_grad():
    model.eval()
    generate_fn_output = model.generate(input_text_tokens, max_length=64, num_beams=1, do_sample=False)
    print("###".join(tokenizer.batch_decode(generate_fn_output, skip_special_symbols=True)))
```

### Expected behavior

Loading the falcon model should get the weights correctly read and calling the `generate` method should perform and correct efficient inference with KV caching.

Comments:
The Falcon model inside Transformers is not ready to be used yet and is not compatible with the online checkpoint. To make the online checkpoints compatible with Transformer, we need to do some changes in the model repo that will break its integration with text-generation-inference. We are waiting for the new version of test-generation-inference to be deployed to be able to do those changes, and once this is done, the model will work with the code in Transformers.

So TL;DR: be patient and use `trust_remote_code=True` for the time being.

Thanks @sgugger - would appreciate if this issue gets tagged once those changes are in the repo :) 

Sure thing! Pinging @Rocketknight1 for when he does the migration.

Hi @afcruzs - you're correct on all counts here. Falcon-7B uses a different model architecture to Falcon-40B. When we ported Falcon to `transformers`, I added some config variables to handle the different code paths taken by the two models. The main variable is `config.new_decoder_architecture` - you can see it [in the repo code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/falcon/modeling_falcon.py#L211).

Unfortunately, because I added these config variables and standardized the names of some others, the `config.json` in the current Falcon checkpoints is not compatible with our library code right now. This is the cause of the `RefinedWebModel` errors you saw. We intend to update the Falcon checkpoints to move them from custom code to library code very soon, which should resolve these errors, as well as fixing the issues with the generation cache. However, we're waiting to give users and other libraries a chance to prepare, since the change will affect the existing custom code checkpoints!

Ohh I didn't catch the differences between 40b and 7b before, good to know; thanks @Rocketknight1 

I've just played around with an alleged fix, and it seems to be working well:

https://huggingface.co/tiiuae/falcon-40b/blob/refs%2Fpr%2F85/modelling_RW.py

The author explains his reasoning: https://huggingface.co/tiiuae/falcon-40b/discussions/85

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any update @Rocketknight1 ?

Hi @afcruzs, you're just in time! There's a release today (release notes still WIP) that resolves this issue and moves Falcon checkpoints to the in-library code. If you `pip install --upgrade transformers` you can start using it right away. There is also a conversion script included in the release to help you convert any fine-tuned of Falcon models to the new in-library code, which should resolve these issues.

I'm going to close this issue as completed at this point, but feel free to reply here and ping me if you encounter any issues!