noamaanMulla-03/transformers-issues huggingface dataset:
sequence_bias feature is not working for Whisper ASR model.
Author: vchagari
Pull request: no

### System Info

Hi @sanchit-gandhi and @gante 
Sequence_bias feature is not working, I tried with similar example shown in the SequenceBiasLogitsProcessor class (https://github.com/huggingface/transformers/blob/7b2bd1fbbd50e57cf28013e2d0737912ecc0f2eb/src/transformers/generation/logits_process.py#L942) with fine-tuned Whisper ASR HF model, 

I recorded an audio and gave it to the Whisper ASR model with biasing terms like mentioned below, unfortunately i  didn't see any effect in the output.

**More details:**
Transformers Commit: 1c7e5e236823cd38faac8115f96205a82c17fff9

Test-Case:  Steps how to reproduce the issue. 
Audio contents: "The full name of Donald is Donald J. Trump Jr"
sequence_bias = {get_tokens_as_tuple("Donald Duck"): 10.0}

model = WhisperForConditionalGeneration.from_pretrained(model_dir).to("cuda") 
feature_extractor = WhisperFeatureExtractor.from_pretrained(model_dir)
processor = WhisperProcessor.from_pretrained(model_dir)
input_features = feature_extractor(audio, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features.to("cuda"), sequence_bias=sequence_bias, num_beams=4)
text = [processor.decode(predicted_id, skip_special_tokens=True) for predicted_id in predicted_ids]
transcript = text[0]

The output is still came as "The full name of Donald is Donald J Trump Jr"




### Who can help?
@sanchit-gandhi and @gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Test-Case:  Steps how to reproduce the issue. 
Audio contents: "The full name of Donald is Donald J. Trump Jr"
sequence_bias = {get_tokens_as_tuple("Donald Duck"): 10.0}

model = WhisperForConditionalGeneration.from_pretrained(model_dir).to("cuda") 
feature_extractor = WhisperFeatureExtractor.from_pretrained(model_dir)
processor = WhisperProcessor.from_pretrained(model_dir)
input_features = feature_extractor(audio, sampling_rate=16000, return_tensors="pt").input_features
predicted_ids = model.generate(input_features.to("cuda"), sequence_bias=sequence_bias, num_beams=4)
text = [processor.decode(predicted_id, skip_special_tokens=True) for predicted_id in predicted_ids]
transcript = text[0]


### Expected behavior

The full name of Donald is Donald Duck.

Comments:
Hi @vchagari 👋 

The [same comment](https://github.com/huggingface/transformers/issues/28822) applies here :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`erfinv_` and `clamp_` ops do not exist in float16+cpu
Author: VictorSanh
Pull request: yes

On cpu, some ops are not defined. This messes up a lot of init weights operations for siglip.
I know fp16 + CPU is weird and probably never happening in practise. As such, feel free to ignore this PR.

Reproduction case:
```python
from transformers import AutoModel, AutoConfig
import torch
config = AutoConfig.from_pretrained("google/siglip-so400m-patch14-384")
model = AutoModel.from_config(config, torch_dtype=torch.float16)
print(sum([m.sum().item() for m in model.parameters()])) # sanity check
```

I am using torch==2.0.1 (+ cu118).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28800). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BART-base save_pretrained triggers a warning about GenerationConfig
Author: Kripner
Pull request: no

### System Info

- `transformers` version: 4.37.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker @younesbelkada

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-base")
model.save_pretrained("model")
```

### Expected behavior

Expected behavior: The model is saved without warnings.

Actual behavior: Following warning is triggered before saving the model:

> Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
> Non-default generation parameters: {'early_stopping': True, 'num_beams': 4, 'no_repeat_ngram_size': 3, 'forced_bos_token_id': 0, 'forced_eos_token_id': 2}

Comments:
Hey that is expected as it is a pretty old model! You can safely ignore this for now, otherwise they should be removed. Feel free to open an issue on the repository's discussion tab ! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Stop Sequence stopping criteria
Author: mina58
Pull request: no
Labels: name: Feature request

### Feature request

someway to stop the llm generation if a sequence of word is generated

### Motivation

I am working with a verilog code llm and I want it to stop generation if the word/tokens "endmodule" occur, to save the compute of the extra tokens that will be truncated.

### Your contribution

I will try to implement it for now, should this issue be accepted, I will make a pr.

Comments:
Hey! #28932 should fix this

Thanks a lot.
[New Model] Add GemMoE: Mixture of Experts model based on Gemma
Author: Crystalcareai
Pull request: yes

## Introduction

This pull request introduces GemMoE, a Mixture of Experts (MoE) model based on the Gemma architecture. GemMoE is comprised of 8 separately fine-tuned Gemma models, with 2 experts per token. It incorporates bug fixes for Gemma, allowing for seamless training and inference like any other model in the Transformers library.

## Motivation

The development of GemMoE aimed to create a powerful and optimized MoE model based on Gemma. Despite the challenges faced during the process, including the lack of experience and the need to create a custom model class and MoE architecture tailored for Gemma, the resulting model showcases significant improvements and potential.

## Implementation Details

- Created a new model class `GemMoEModel` within the Transformers library.
- Implemented a custom MoE architecture tailored for Gemma.
- Fine-tuned 8 different versions of Gemma and combined them using a hidden gate with MergeKit (modifications were made to support Gemma).
- Incorporated bug fixes specific to GemMoE's architecture.
- Fine-tuned the combined model on the H4's deita-10k-v0-sft dataset for 2.5 epochs and the MoD dataset for 1 epoch.

## Usage

GemMoE can be used for training and inference just like any other model in the Transformers library. No extra work is required from the user's end.

## Current Limitations and Future Work

1. GemMoE currently does not support distributed training via Axolotl (and potentially base Accelerate). Some parameters are not calculated towards the loss, leading to an error in PyTorch. We are actively investigating this issue and will provide an update via a future pull request once resolved.
2. There may be additional issues that have not been identified yet. We welcome the community's feedback and contributions to further improve GemMoE.

## Acknowledgments

We would like to express our gratitude to Hugging Face for providing a generous compute grant that allowed for the optimization and refinement of GemMoE's architecture and pipeline before committing resources to the full training. We also extend our thanks to Daniel Han from Unsloth for identifying and implementing bug fixes for Gemma within Unsloth, which greatly facilitated the development of GemMoE.

## Checklist

- [x] Added new model class `GemMoEModel`
- [x] Implemented custom MoE architecture for Gemma
- [x] Incorporated bug fixes specific to GemMoE
- [x] Updated documentation and usage instructions
- [ ] Resolved distributed training issue (work in progress)

We appreciate your consideration of this pull request and look forward to any feedback or suggestions from the community.

Comments:
Add gpt_neox in fx supported models?
Author: TXacs
Pull request: no

### System Info

change        
`qkv = qkv.view(*new_qkv_shape)`
to
` qkv = qkv.view(new_qkv_shape)`
I've made a modification, and now it can be traced by `hftracer`.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

I wonder if this could be incorporated into future versions.

Comments:
feel free to open a PR and ping @fxmarty ! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Missed a guard on hf_quantizer.check_quantized_param
Author: pharaouk
Pull request: no

### System Info

```
    model = AutoModelForCausalLM.from_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 562, in from_pretrained
    return model_class.from_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3504, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3732, in _load_pretrained_model
    ) or not hf_quantizer.check_quantized_param(
AttributeError: 'NoneType' object has no attribute 'check_quantized_param'
```

only happens w multi gpu for some reason


missed a guard on hf_quantizers

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Private code

### Expected behavior

Code not breaking

Comments:
cc @younesbelkada 

Yes, should be fixed by #28804

@pharaouk could you try on transformers main?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AWQ models loaded with AutoModelForCausalLM.from_pretrained not as fast as AutoAWQForCausalLM.from_quantized
Author: nilichen
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.11.7
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
    model = AutoModelForCausalLM.from_pretrained(
        "TheBloke/Mistral-7B-Instruct-v0.2-AWQ",
        device_map="auto",
    )

    model = AutoAWQForCausalLM.from_quantized(
        "TheBloke/Mistral-7B-Instruct-v0.2-AWQ",
        fuse_layers=False,
        quant_filename="model.safetensors",
        safetensors=True,
    )
```

I tested both on long inputs (~5k tokens, data of privacy so won't be able to share) for generation, in both cases, I didn't use fused modules. `AutoModelForCausalLM.from_pretrained` took ~24s each while `AutoAWQForCausalLM.from_quantized` took ~18s each. The output token count didn't seem to differ much.

I ran the comparison at least 3 times and got consistent results.

### Expected behavior

The latency should be close.

Comments:
Hey! Thanks for reporting. This is expected, the operations are not fused by default in transformers, [as per the documentation](https://huggingface.co/docs/transformers/quantization#fused-modules). 

Hi. I did have `fused_layers = False` when loading with `AutoAWQForCausalLM.from_quantized` so in theory they should share similar latency?

cc @younesbelkada 

@nilichen I think the autoawq use by default fused layer norm maybe that explains the whole difference - make sure also to run greedy generation to avoid any potential bottleneck, @casper-hansen is the speedup over the transformers implementation expected?

There should not be any difference between the two implementations when fused layers are off

That's what I thought. Let me see if I can reproduce on a public dataset and report back.

ok that would be great - thanks @nilichen !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
SDPA attention causes graph break while compiling model
Author: huzama
Pull request: no

### Feature request

Static graph support in SDPA attention 

### Motivation

The use of SPDA attention significantly enhances transformers' performance and memory utilization. However, when attempting to compile it with the XLA backend on TPUs, there are graph breaks that require continuous recompilation. This makes the use of SPDA attention on TPUs impractical. 

### Your contribution

I did some debugging and found out that this [line](https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/src/transformers/modeling_attn_mask_utils.py#L371)  of code uses torch.all on attention_mask which changes for each iteration causing dynamic control flow, which causes recompilation in XLA-TPU.


Output of torch._dynamo.explain for Llama-7b with SDPA attention

 graph_count=6, graph_break_count=5, break_reasons=[GraphCompileReason(reason='Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow. For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#cond-operands', user_stack=[<FrameSummary file /home/ubuntu/miniforge3/envs/transformers/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py, line 1035 in forward>, <FrameSummary file /home/ubuntu/miniforge3/envs/transformers/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py, line 372 in _prepare_4d_causal_attention_mask_for_sdpa>], graph_break=True), GraphCompileReason(reason='generic_jump TensorVariable()', user_stack=[<FrameSummary file /home/ubuntu/miniforge3/envs/transformers/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py, line 372 in _prepare_4d_causal_attention_mask_for_sdpa>], graph_break=True), GraphCompileReason(reason='dynamic shape operator: aten.nonzero.default', user_stack=[<FrameSummary file /home/ubuntu/miniforge3/envs/transformers/lib/python3.10/site-packages/transformers/modeling_attn_mask_utils.py, line 243 in _unmask_unattended>], graph_break=True)], op_count=977,

Comments:
Hey! would recommend you to try #27931 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
codellama/CodeLlama-34b Fine-Tune Evaluation 
Author: sanipanwala
Pull request: no

Hello,

I have done training and saved the model, and adapter config file on the local disk.

When I load the model from the local disk again to generate the output I get the below error.

Anyone can help me with this issue?

`File "PythonV2.py", line 11, in <module> model = AutoPeftModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/python3.12/site-packages/peft/auto.py", line 127, in from_pretrained return cls._target_peft_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/python3.12/site-packages/peft/peft_model.py", line 354, in from_pretrained model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs) File "/python3.12/site-packages/peft/peft_model.py", line 695, in load_adapter adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/python3.12/site-packages/peft/utils/save_and_load.py", line 313, in load_peft_weights adapters_weights = safe_load_file(filename, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/python3.12/site-packages/safetensors/torch.py", line 308, in load_file with safe_open(filename, framework="pt", device=device) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization `

Thanks.

Comments:
cc @younesbelkada safetensors + peft

hi @sanipanwala 
do you face the same issue when you update `peft` and `safetensors` ? `pip install -U peft safetensors`

@younesbelkada ,

Yes, I have created a fresh conda environment and installed all the latest packages.

Thanks,

Have you used torch.compile by any chance? It seems to be a duplicate of https://github.com/huggingface/transformers/issues/27397

Hi @younesbelkada ,

I'm not using torch.compile but if I'm using torch.compile before trainer.train() It gives me the below exception.

```
    raise RuntimeError("Dynamo is not supported on Python 3.12+")
RuntimeError: Dynamo is not supported on Python 3.12+
```

Thanks.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
KeyError: 'cache_params'. Unsafe key access in generate method _update_model_kwargs_for_generation
Author: nbarlowportfolio
Pull request: no

### System Info

Solution: Changing the code to ignore fixed it. I am unsure as to if there is a more proper fix further back in the pipeline.

def _update_model_kwargs_for_generation(
        self, outputs: ModelOutput, model_kwargs: Dict[str, Any], **kwargs
    ) -> Dict[str, Any]:

        if "cache_params" in outputs:
            model_kwargs["cache_params"] = outputs["cache_params"]
        return model_kwargs

Here is the full code I ran to get the error and test the fix: https://github.com/nbarlowportfolio/Mamba-Hello-World/blob/master/mamba_official_with_fixes.ipynb

Thank you.


### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the full notebook here: https://github.com/nbarlowportfolio/Mamba-Hello-World/blob/master/mamba_official_with_fixes.ipynb


### Expected behavior

Expected: Generating without error.

Comments:
Thanks, I am adding the fix in #29472
[`Mamba doc`] Post merge updates 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Official checkpoints are now available on the official `state-spaces` org. Also fixes  #29501

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29472). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @ArthurZucker! Great work integrating Mamba in HF :)
BTW, is this PR gonna be merged soon? I think it is the last one pending to use Mamba completely. Right? Or there are more PRs needed?
Also, which commit would you recommend to install while Mamba it is in main branch instead of a release?
Thank you so much!
FIX [`quantization` / `ESM`] Fix ESM 8bit / 4bit with bitsandbytes
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/29323

Currently on main, simply running: 
```python
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("facebook/esm2_t36_3B_UR50D", load_in_4bit=True)
```
Fails with an error

```bash
  File "/home/younes_huggingface_co/code/transformers/src/transformers/modeling_utils.py", line 802, in _load_state_dict_into_meta_model
    or (not hf_quantizer.check_quantized_param(model, param, param_name, state_dict))
  File "/home/younes_huggingface_co/code/transformers/src/transformers/quantizers/quantizer_bnb_4bit.py", line 124, in check_quantized_param
    if isinstance(module._parameters[tensor_name], bnb.nn.Params4bit):
KeyError: 'inv_freq'
```
This is because the model pushed in  `"facebook/esm2_t36_3B_UR50D"` do not contain the `inv_freq`. Maybe during the HfQuantizer refactor we did not properly dealt with that specific scenario, leading to this bug for transformers > 4.37

cc @SunMarc 

I ran the quantization tests and they seem to all pass on my end

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29329). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the quick fix, everyone!