noamaanMulla-03/transformers-issues huggingface dataset:
`convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py` does not work due to weight norm key
Author: gau-nernst
Pull request: no
Labels: name: Audio

### System Info

NA

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Fairseq uses `nn.utils.weight_norm()`, which produces `weight_g` and `weight_v` keys

https://github.com/facebookresearch/fairseq/blob/d9a627082fd03ec72a27a31a4e56289bfcb2e4e4/fairseq/models/wav2vec/wav2vec2.py#L939

The weight conversion script assumes this old-style weight norm

https://github.com/huggingface/transformers/blob/2daf2c3eaa2992018b8d1ceb3421f5baa649bad5/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py#L118-L121

Thus, the weight conversion script will fail since `transformers` has switched to `nn.utils.parametrizations.weight_norm` for quite some time, which produces `parametrizations.weight.original0` and `parametrizations.weight.original1` keys instead.

https://github.com/huggingface/transformers/blob/1218e439b5fe05423693407653a1fb064263fef4/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L392-L394

Thus the weight conversion script should be updated accordingly.

### Expected behavior

Weight conversion script works.

Comments:
cc @sanchit-gandhi 

I had something working for me. f you don't have time to look into it, I can open a PR. I can provide a snippet to test it with some original fairseq checkpoints too. I

@gau-nernst Yes please, that would be great cc @kamilakesbi too :)
Freely Long-Thinking Transformer (FraiLT)
Author: akbayt
Pull request: no
Labels: name: New model

### Model description

Hi! 
I am the author of the following study:

https://arxiv.org/abs/2401.11626

I want to add this model to 🤗 transformers. 

Implementation is in progress... 👨‍💻

*Abstract:*
Freely Long-Thinking Transformer (FraiLT) is an improved transformer model designed to enhance processing capabilities without scaling up size. It utilizes a recursive approach, iterating over a subset of layers multiple times, and introduces iteration encodings to maintain awareness across these cycles. Iteration encoding allows FraiLT to achieve the interpretive depth of larger models in a compact form. When evaluated on a synthetic story dataset, FraiLT outperformed larger models, showcasing its ability to deliver high-quality performance while reducing memory demands. This model represents a step forward towards more efficient and accessible language models.

### Open source status
- [X] The model explained in the paper

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2401.11626, https://www.academia.edu/113629981

Comments:
Hey @akbayt. Thanks a lot for opening this new model request and contributing to the HF ecosystem! 🤗
We have recently been trying to push for `model on the hub` and have as much support as we can there. It will also be easier to integrate it. Here is a [tutorial](https://huggingface.co/docs/transformers/custom_models) if that sound good to you!

Hi @amyeroberts. Thanks for your quick response and guidance. As far as I can see, sharing the models on hub will make it easier for me and will add a slight work when someone wants to use it. This is still acceptable. However, the models I have trained so far are very modest in scale (you can see the details in the paper), and I am actively seeking support and resources to train larger models. I believe that having my model integrated into the 🤗 transformers library, rather than sharing it on the hub, could increase its visibility within the community. In turn, it might help me find some support easier. Thanks again.

The best way to help make your model visible is by making sure the model is easy to find (be on the hub), easy to use (has a detailed model card with code snippet users showing how to get started) and a space demo showing the model's capabilities. Many models have become popular that were just on the hub. For example, falcon and phi-2 started on the hub.
Fix Phi3Config attention_bias issue while running on tgi server
Author: manikawnth
Pull request: yes

# What does this PR do?

This PR fixes the PhiConfig's missing `attention_bias` attribute error while running on tgi server.
Similar to LlamaConfig, the attribute is an optional one and should default to False.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/31020

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


@ArthurZucker 
@younesbelkada



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Models with Phi3Config fail due to missing attention_bias
Author: manikawnth
Pull request: no

### System Info

- `transformers` version: 4.41.1
- Platform: Linux-5.15.0-1055-aws-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running the model using huggingface text-generation-inference server will cause the issue

```
AttributeError: 'Phi3Config' object has no attribute 'attention_bias' 
```

Reference issue:
https://github.com/aws/sagemaker-huggingface-inference-toolkit/issues/123 


### Expected behavior

tgi server should successfully run the Phi3 model

Comments:
cc @ArthurZucker @younesbelkada 

Hi @manikawnth !
See my comment on the PR, IMO the fix should go on TGI directly, what do you think?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add token classification support for Cohere model
Author: hadikhamoud
Pull request: yes

## What does this PR do?

This PR adds token classification for Cohere:

- CohereForTokenClassification

## Before submitting

- [x]  Did you read the contributor guideline, Pull Request section?
- [ ]  Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
- [x]  Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
- [x]  Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
@LysandreJik Added  '# Copied from' in CohereForTokenClassification and test_Cohere_token_classification_model

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AttributeError: 'LlamaForCausalLM' object has no attribute '_setup_cache'
Author: mobicham
Pull request: no

### System Info

- `transformers` version: 4.41.1
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.23.2
- Accelerate version: 0.30.1
- PyTorch version (GPU?): 2.4.0.dev20240527+cu121 (True)

### Who can help?

@ArthurZucker  @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Seems like `_setup_cache()` was removed in the newer version of transformers. Is there an alternative to this?
It's necessary to do that call in order to properly compile the model for faster generation.
Thanks!



### Expected behavior

`model._setup_cache()` should setup the cache.

Comments:
Static Cache was moved to be a standalone object in #30476. You have to init StaticCache outside the model and pass in every forward call, similar to following:

```python
past_key_values = StaticCache(model.config, bs, max_cache_length, model.device, model.dtype)
for i in range(max_new_tokens):
    out = model(next_token, past_key_values=past_key_values, return_dict=True, **model_kwargs)
    past_key_values = out.past_key_values
    next_token = sample_next_token(out)
    model_kwargs = update_model_kwargs(model_kwargs)
```

Also you can simply pass-in `cache_implementation="static"` to `generate()` (see [docs](https://huggingface.co/docs/transformers/v4.41.2/en/llm_optims#static-kv-cache-and-torchcompile)):
```python
model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
model.generation_config.max_new_tokens = 20
out = model.generate(input_ids, cache_implementation="static)
```

> Static Cache was moved to be a standalone object in #30476. You have to init StaticCache outside the model and pass in every forward call, similar to following:

Thanks, will try that!

> ```python
> model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
> model.generation_config.max_new_tokens = 20
> out = model.generate(input_ids, cache_implementation="static)
> ```

That's not the same thing. This will compile the whole forward pass, which will force re-compilation every time the prompt length changes. I only want to compile the decoding part and leave the prefill phase uncompiled.  



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Off-by-one error in strided perplexity calculation
Author: bauwenst
Pull request: no

The computation of perplexity [in the docs](https://huggingface.co/docs/transformers/perplexity) seems to be incorrect. When passing `input_ids` to a causal LM for predicting them, what happens internally (as suggested in the code snippet) is that the model shifts the input IDs to the left by 1 so that the label of the current token becomes the ID of the next token. The last input is hence cut off because there is no label for it.

That means that when you take a window of tokens to predict, you do not do a prediction for the last token and you should start your next window on that token. Yet, in the docs, you start the next window on the token *after* the previous window:

https://github.com/huggingface/transformers/blob/a25f7d3c12975fe21eab437dda7363e9024de7c0/docs/source/en/perplexity.md?plain=1#L129

It should be that `-1`. The implementation [here](https://github.com/bauwenst/LaMoTO/blob/b7a7bc0f02e6b0cff6fa0bdb5dc64787acd52c0e/src/lamoto/measuring/ppl.py#L49-L86) is clearer to read and incorporates this bugfix.

Comments:
cc @ArthurZucker 

do you want to open a PR to clarify the doc? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TextGenerationPipeline output differs from ConversationalPipeline for BlenderBot
Author: Rocketknight1
Pull request: no
Labels: name: Core: Pipeline

name: bug

`TextGenerationPipeline` loads `BlenderBotForCausalLM` by default but `ConversationalPipeline` loads `BlenderBotForConditionalGeneration`. These pipelines are supposed to give identical output for chat input, so this is a bug! It went unnoticed because most models don't have both `CausalLM` and `ConditionalGeneration` classes.

Self-assigning this one, but just making an issue to keep track of it.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Let's maybe make sure to fix this before removing the `ConversaitonalPipeline` altogether

I'm not really sure what the solution is, unfortunately - thankfully the issue is mostly limited to `BlenderBot`, but the pipeline can't know if it'll be used for chat or standard CLM completion before it receives its first input!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
DirectML storage compatibility issues
Author: sqxccdy
Pull request: no
Labels: name: Feature request

### Feature request

I'm trying to run transformers models using DirectML, and it works fine in most cases. However, since Microsoft's DirectML does not support storage, this causes an error whenever an untyped_storage method is involved.


Package versions installed:

torch~=2.3.1
torch_directml=0.2.2.dev240614
transformers=4.42.3


### Motivation

compatibility issues

### Your contribution

I hope there is a way to provide a custom device context method without modifying the transformers source code. In my current use case, I can write compatible code for storage and skip this part.

Comments:
Hi @sqxccdy, thanks for opening this feature request. 

Could you provide some more details? Without a reproducible code snippet, explanation of the observed and expected behaviour, and all the relevant information about the error e.g. full error traceback there's not much we can do.
Add siglip flashattention support?
Author: lucasjinreal
Pull request: no

Add siglip flashattention support?

I noticed that the navit-flashatten-siglip version already added flash attention,  would transformers add it support inside ?

Comments:
cc @molbap or @qubvel maybe :)

these two guys looked but didn't response.

@lucasjinreal thanks for the feature request, I will take a look at it!

thanks, since more and more MLLM used siglip as vision encoder, if flash attention support would be very reduce training cost time
Avoid failure `TFBlipModelTest::test_pipeline_image_to_text`
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently,

> FAILED tests/models/blip/test_modeling_tf_blip.py::TFBlipModelTest::test_pipeline_image_to_text - NameError: name 'torch' is not defined

is failing on `main` after (#31342)

See https://app.circleci.com/pipelines/github/huggingface/transformers/97187/workflows/81b63940-fd42-4a6a-aafe-404301fafc1d/jobs/1285954

There is a slight difference between `Pipeline.__init__` and `pipeline`: the later one has 

>         if isinstance(torch_dtype, str) and hasattr(torch, torch_dtype):
>            torch_dtype = getattr(torch, torch_dtype)

which we should have a check on if `torch` is available. I will open another PR for that.

----------------------------------------------
Also for 2 `Failed: Timeout >120.0s`: the fix for them is the same as their **non** `_fp16` versions.

FAILED tests/models/vits/test_modeling_vits.py::VitsModelTest::test_pipeline_feature_extraction_fp16 - Failed: Timeout >120.0s

FAILED tests/models/efficientnet/test_modeling_efficientnet.py::EfficientNetModelTest::test_pipeline_image_feature_extraction_fp16 - Failed: Timeout >120.0s

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31827). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix incorrect accelerator device handling for MPS in `TrainingArguments`
Author: andstor
Pull request: yes

This PR fixes incorrect accelerator device handling for MPS in `TrainingArguments`. 

Fixes #31811


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @SunMarc

Comments:
Incorrect accelerator device handling for MPS in TrainingArguments
Author: andstor
Pull request: no

### System Info

- `transformers` version: 4.43.0.dev0
- Platform: macOS-14.4.1-arm64-arm-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.3.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@muellerzr @SunMarc

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Execute the following code using accelerate with MPS accelerator.

```python
from transformers import TrainingArguments

training_args = TrainingArguments(output_dir = "tmp_trainer")
print(training_args.device)
```

### Expected behavior

I expect to see `mps` will be displayed. However, I see `cpu`.

Comments:
transformers.fx.symbolic_trace supports inputs_embeds
Author: fxmarty
Pull request: yes

We should prompt users to use `torch.export.export` instead.

Fixes https://github.com/huggingface/transformers/issues/31414
Fixes https://github.com/huggingface/transformers/issues/31200

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31574). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @fxmarty 

Could you take a look on the following (see https://app.circleci.com/pipelines/github/huggingface/transformers/97223/workflows/f7cf9ddb-909b-4544-994a-18aaadbb77ac/jobs/1286547)?

FAILED tests/models/blenderbot_small/test_modeling_blenderbot_small.py::BlenderbotSmallModelTest::test_torch_fx - ValueError: You have to specify either input_ids or inputs_embeds
FAILED tests/models/blenderbot_small/test_modeling_blenderbot_small.py::BlenderbotSmallModelTest::test_torch_fx_output_loss - ValueError: You have to specify either input_ids or inputs_embeds
FAILED tests/models/pegasus/test_modeling_pegasus.py::PegasusModelTest::test_torch_fx - ValueError: You have to specify either input_ids or inputs_embeds
FAILED tests/models/pegasus/test_modeling_pegasus.py::PegasusModelTest::test_torch_fx_output_loss - ValueError: You have to specify either input_ids or inputs_embeds
FAILED tests/models/m2m_100/test_modeling_m2m_100.py::M2M100ModelTest::test_torch_fx - ValueError: You have to specify either input_ids or inputs_embeds
FAILED tests/models/m2m_100/test_modeling_m2m_100.py::M2M100ModelTest::test_torch_fx_output_loss - ValueError: You have to specify either input_ids or inputs_embeds