RuntimeError: result type Float can't be cast to the desired output type Byte
Author: KaifAhmad1
Pull request: no

### System Info

OS: Windows 11 x64
Cuda: 12.1 
transformers: 4.37.2
sentence_transformers: 2.3.1
bitsandbytes: 0.42.0
pip: 23.1.2
Python 3.10.10

### Who can help?

Hey, @ArthurZucker , @younesbelkada Please guide for this issue. 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Here is the code snippet 
I am using  RWKV architecture model `RWKV/HF_v5-Eagle-7B` for my KG-enhanced RAG use cases on domain-specific datasets. 


``` Python 
from torch import cuda, bfloat16
import transformers
model_id = 'RWKV/HF_v5-Eagle-7B'
device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'
```

``` Python 
# begin initializing HF items, you need an access token
model_config = transformers.AutoConfig.from_pretrained(
    model_id,
    use_auth_token=hf_auth,
    trust_remote_code=True
)
```

```
# BnB Configuration
bnb_config = transformers.BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=bfloat16
)
```

``` Python 
model = transformers.AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    config=model_config,
    device_map='auto',
    use_auth_token=hf_auth,
    quantization_config=bnb_config,
    low_cpu_mem_usage=True
)
```

``` Python 
tokenizer = transformers.AutoTokenizer.from_pretrained(
    model_id,
    use_auth_token=hf_auth
)
```

#### Stopping Criteria 

``` Python 
# List of strings representing stop signals or markers
stop_list = ['\nHuman:', '\n```\n']
stop_token_ids = [tokenizer(x)['input_ids'] for x in stop_list]
stop_token_ids
```

``` Python 
# Convert token IDs to LongTensor objects
import torch
stop_token_ids = [torch.LongTensor(x).to(device) for x in stop_token_ids]
stop_token_ids
```

``` Python 
from transformers import StoppingCriteria, StoppingCriteriaList

# Define a custom stopping criteria class
class StopOnTokens(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        for stop_ids in stop_token_ids:
            if torch.equal(input_ids[0][-len(stop_ids):], stop_ids):
                return True
        return False

stopping_criteria = StoppingCriteriaList([StopOnTokens()])

```


``` Python 
# Set up text generation pipeline
generate_text = transformers.pipeline(
    model=model,
    tokenizer=tokenizer,
    return_full_text=True,
    task='text-generation',
    stopping_criteria=stopping_criteria,
    temperature=0.3,
    max_new_tokens=512,
    repetition_penalty=1.1
)
```

``` Python 
result = generate_text("What are the primary mechanisms underlying antibiotic resistance, and how can we develop strategies to combat it?")
print(result)
```

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-26-cab67dc592cd>](https://localhost:8080/#) in <cell line: 1>()
----> 1 result = generate_text("What are the primary mechanisms underlying antibiotic resistance, and how can we develop strategies to combat it?")
      2 print(result)

16 frames
[~/.cache/huggingface/modules/transformers_modules/RWKV/HF_v5-Eagle-7B/d777ebf28bd74abf9ada4a07f58c38c6688f5365/modeling_rwkv5.py](https://localhost:8080/#) in _rescale_layers(self)
    748                         block.feed_forward.value.weight.mul_(2 ** int(block_id // self.config.rescale_every))
    749                     else:
--> 750                         block.attention.output.weight.div_(2 ** int(block_id // self.config.rescale_every))
    751                         block.feed_forward.value.weight.div_(2 ** int(block_id // self.config.rescale_every))
    752 

RuntimeError: result type Float can't be cast to the desired output type Byte

### Expected behavior

It will return an answer to the given query without raising any exceptions. 

Comments:
Hey ! You are using custom code, we probably won't have the bandwidth to debug it for you! Best recommendation is to put a debug breakpoint and see what is happening! 🤗 

@KaifAhmad1 thanks for the issue! The issue seems to be related to the custom code that is on the Hub ! 
I recommend opening an issue in https://huggingface.co/RWKV/HF_v5-Eagle-7B/blob/main/modeling_rwkv5.py 
Make sure that the trust remote code model has correctly implemented layer re-scaling for quantized layers: https://github.com/huggingface/transformers/blob/d90acc16437e8c9e45e068fa1cc1a263b9a7208f/src/transformers/models/rwkv/modeling_rwkv.py#L711

@KaifAhmad1 I attempted to fix it here: https://huggingface.co/RWKV/HF_v5-Eagle-7B/discussions/5 can you try to pass `revision="refs/pr/5"` in `transformers.AutoModelForCausalLM.from_pretrained`?

 Hey, @younesbelkada still not getting right results

``` Python 
model = transformers.AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    config=model_config,
    revision="refs/pr/5",
    device_map='auto',
    use_auth_token=hf_auth,
    quantization_config=bnb_config,
    low_cpu_mem_usage=True
)
```

```
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-6-59c99693831c>](https://localhost:8080/#) in <cell line: 1>()
----> 1 model = transformers.AutoModelForCausalLM.from_pretrained(
      2     model_id,
      3     trust_remote_code=True,
      4     config=model_config,
      5     revision="refs/pr/5",

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py](https://localhost:8080/#) in register(cls, config_class, model_class, exist_ok)
    584         """
    585         if hasattr(model_class, "config_class") and model_class.config_class != config_class:
--> 586             raise ValueError(
    587                 "The model class you are passing has a `config_class` attribute that is not consistent with the "
    588                 f"config class you passed (model has {model_class.config_class} and you passed {config_class}. Fix "

ValueError: The model class you are passing has a `config_class` attribute that is not consistent with the config class you passed (model has <class 'transformers_modules.RWKV.HF_v5-Eagle-7B.5d64a05fb9748b26b55162cb56162c9da83135c4.configuration_rwkv5.Rwkv5Config'> and you passed <class 'transformers_modules.RWKV.HF_v5-Eagle-7B.d777ebf28bd74abf9ada4a07f58c38c6688f5365.configuration_rwkv5.Rwkv5Config'>. Fix one of those so they match!
```

@KaifAhmad1 you need to pass that argument to both automodelxxx and autoconfig

 @younesbelkada @ArthurZucker  Now getting attribute error. 
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-12-cab67dc592cd>](https://localhost:8080/#) in <cell line: 1>()
----> 1 result = generate_text("What are the primary mechanisms underlying antibiotic resistance, and how can we develop strategies to combat it?")
      2 print(result)

17 frames
[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in __getattr__(self, name)
   1693             if name in modules:
   1694                 return modules[name]
-> 1695         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1696 
   1697     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'Rwkv5Model' object has no attribute '_bnb_4bit_dequantize_and_rescale'
```

Here is the colab link for your reference: 
https://colab.research.google.com/drive/1glmO6e7Qro3mQtCwnmoBGi537ej7aTHl?usp=sharing

@KaifAhmad1 thanks for testing out! 
I invite you to continue from this branch: https://huggingface.co/RWKV/HF_v5-Eagle-7B/tree/refs%2Fpr%2F5 as it uses code on the Hub feature, and modify the modeling file to your needs. Make sure to have the same quant / dequant logic as in https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/modeling_rwkv.py . Let me know if you need any help !

 @younesbelkada PR is waiting for merge.
https://huggingface.co/RWKV/HF_v5-Eagle-7B/discussions/7#65cdc51d0cdcf1ddd539b712