noamaanMulla-03/transformers-issues huggingface dataset:
ValueError: InternVLChatModel does not support Flash Attention 2.0 yet.
Author: dszpr
Pull request: no
Labels: name: Feature request

### System Info

transformers version: 4.37.2
flash-attn version: 2.3.1post1/2.6.1(latest version)

### Who can help?

@ArthurZucker @muellerzr @SunMarc

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi! I am finetuning InternVL2-26B based on LLaVA training code. And I modified the model loading code as below:
            model = transformers.AutoModel.from_pretrained(
                '/workspace/vlm_dataset/InternVL/InternVL2-26B',
                cache_dir=training_args.cache_dir,
                attn_implementation=attn_implementation,
                torch_dtype=(torch.bfloat16 if training_args.bf16 else None),
                trust_remote_code=True,
                **bnb_model_from_pretrained_args
            )
And an error occurred:
ValueError: InternVLChatModel does not support Flash Attention 2.0 yet. Please request to add support where the model is hosted, on its model hub page: https://huggingface.co//workspace/vlm_dataset/InternVL/InternVL2-26B/discussions/new or in the Transformers GitHub repo: https://github.com/huggingface/transformers/issues/new
I also tried the latest version of transformers and flash-attention, and this error still occurs. 
Is this error concerned with transformers or flash-attention or InternVL2?

### Expected behavior

In transformers, InternVLChatModel support Flash Attention 2.0

Comments:
Hi @dszpr, thanks for raising an issue! 

It seems you're working from a local model, but I'm assuming it's a download of [OpenGVLab/InternVL2-26B](https://huggingface.co/OpenGVLab/InternVL2-26B/tree/main). As the modeling code is hosted on the hub, you should open a request to support flash attention on the [model's discussion page](https://huggingface.co/OpenGVLab/InternVL2-26B/discussions)

Got it. Thanks a lot!
Add language to word timestamps for Whisper
Author: algo-robin
Pull request: yes
Labels: name: Audio

# Add language to word timestamps for Whisper

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

Fixes [29520](https://github.com/huggingface/transformers/issues/29520)

This fix enables whisper to return word-level timestamps and the predicted language at the same time. Before, enabling word-level timestamps would discard the language prediction.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@kamilakesbi 
@sanchit-gandhi

Comments:
Hi @robinderat, thanks for working on this! 

Could you please add a test in `test_tokenization_whisper.py` to verify that we can indeed return both word-level timestamps and the predicted language at the same time? 



LGTM! thanks for iterating on this :) 

cc @amyeroberts for final review! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31572). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts gentle reminder 😄
Track for flaky batching tests
Author: zucchini-nlp
Pull request: no
Labels: name: WIP

name: bug

name: Vision

### System Info

We added a test for comparing model outputs when the input is batched or not batched in [this PR](https://github.com/huggingface/transformers/pull/29297). The assumption is that batching should not have impact on the results, disregarding numerical precision errros.

Currently, all the models pass the test, but there are 3 flaky tests. This issue is to keep track of them. The models that are flaky: MobileNetV2, DPT hybrid, VIT hybrid. I cannot give any more information except that it is flaky, so it needs more exploration.

### Who can help?

@amyeroberts for tracking

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

NA

### Expected behavior

NA

Comments:
unpin `numpy<2.0`
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed internally, let's allow the community to run `transformers` with `numpy=2.x`

Fix  #31802

Comments:
Note, our CI is still running with `numpy < 2.0` as the dependency would resolve to a version < 2 at the end. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32018). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Chameleon: minor fixes after shipping
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #32022  by making the merge process an out-of-place op

Additionally, changes AutoMapping of the model making it a `ConditionalModel`. I believe we can do such changes of model naming for Chameleon, because it was merged just yesterday and won;t be breaking. I can ofc leave naming as Causal if you think it will break stuff for users

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32037). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Chameleon model failed after receiving two times the same inputs
Author: francescortu
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.0.dev0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no 
- Using GPU in script?: yes
- GPU type: NVIDIA A100-SXM4-40GB

### Who can help?

@zucchini-nlp 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

After importing the model and processor
```python
from transformers import ChameleonProcessor, ChameleonForCausalLM
processor = ChameleonProcessor.from_pretrained("facebook/chameleon-7b", torch_dtype=torch.float16)
model = ChameleonForCausalLM.from_pretrained("facebook/chameleon-7b", torch_dtype=torch.float16, device_map="cuda:0")

from PIL import Image
image1 = Image.open("<path-to-image>")
image2=Image.open("<path-to-image>")
prompt = "<image>"

inputs = processor(text=[prompt,prompt], images=[image, image2], return_tensors="pt").to("cuda:0")
```
 Execute
```python
output = model(**inputs)
output = model(**inputs)
```
The second time I run the code, it raises an error. After some investigation, I found that the problem is with the forward pass, which substitutes the <image> token in input_ids with the correct image token from the autoencoder. However, when run the second time, it attempts to perform the substitution again since pixel_values are still present, but the <image> token is no longer there (as it has been replaced with the real image tokens).

### Expected behavior

I expect the forward pass not to modify the input tensors.

Comments:
cc @zucchini-nlp 

@francescortu thanks for noticing, made a PR to fix!
pin numpy < 2
Author: ydshieh
Pull request: yes

# What does this PR do?

Let's pin numpy < 2.0.0 for now. It doesn't work well. See [internal thread](https://huggingface.slack.com/archives/C01NE71C4F7/p1718639652402109): not so bad with torch only env., but not good with tf/flax.

Also see #31569

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `Seq2seqTrainerTester::test_bad_generation_config_fail_early`
Author: ydshieh
Pull request: yes

Similar to #28575, this test is currently failing due to the global states being modified.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31866). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
We Need Compile Support For Mamba!
Author: zhenglongjiepheonix
Pull request: no
Labels: name: Feature request

name: Compilation

### Feature request

This feature adds `torch.compile` support for mamba archtecture

### Motivation

The motivation is that by supporting compile on mamba, we can get faster inference speed and better throughput even if we don't have high performance specified mamba kernels installed!

### Your contribution

I will be implementing this feature

Comments:
cc @ArthurZucker
fix: Removed `duplicate entries` in a dictionary
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Just a small fix, the following key-value pairs are repeated in the dictionary.
https://github.com/huggingface/transformers/blob/1c37e8c1a6274e6e87b45c6319eb190757214c2a/src/transformers/image_utils.py#L61
https://github.com/huggingface/transformers/blob/1c37e8c1a6274e6e87b45c6319eb190757214c2a/src/transformers/image_utils.py#L67

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
docs: Fixed 2 links in the docs along with some minor fixes
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Here, the link to **Python 3.8** might keep changing due to changes in `setup.py`. So, I changed it to a `permalink`.
https://github.com/huggingface/transformers/blob/4c040aba02b0283619a06bdc40ecf868508b9e52/CONTRIBUTING.md?plain=1#L135
Here, the link to `testing_utils.py` is not working in docs, as the docs are linked to [this file](https://github.com/huggingface/transformers/blob/main/docs/source/en/contributing.md). So, I just changed the link to the file's github link
https://github.com/huggingface/transformers/blob/4c040aba02b0283619a06bdc40ecf868508b9e52/CONTRIBUTING.md?plain=1#L349

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
Avoid deepcopy in ProgressCallback
Author: fozziethebeat
Pull request: yes

# What does this PR do?

When ProgressCallback runs after the WandbCallback, in some scenarios it currently tries to deepcopy an object that can't be pickled.  This makes a shallow copy of the log values to avoid any pickling and safely log everything.

<!-- Remove if not applicable -->

Fixes #32064


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

I manually tested via integration with Axolotl and TRL Training.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ amyeroberts 

- trainer: @muellerzr and @SunMarc 


Comments:
Fix `GenerationMixin.generate` compatibility with pytorch profiler
Author: fxmarty
Pull request: yes

As per title and discussed internally, more details at https://github.com/pytorch/pytorch/issues/130622

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31935). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llava: add default chat templates
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR adds default chat templates for Llava models, so that user-defined models on the hub do not fail when using `apply_chat_template`. Users will see a warning in any case that the default template is used

Chat templates were added in all llava-hf models on the hub, can be verified by the below script

```python
messages = [{
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s the content of this image?"},
                {"type": "image"},
                ],
        },
        {
            "role": "assistant",
            "content": [{"type": "text", "text": "This picture shows a red stop sign."},]
        }
    ]
processor = AutoProcessor.from_pretrained("llava-hf/vip-llava-7b-hf")

convo = processor.apply_chat_template(messages)
>>> ###Human: <image>\nWhat’s the content of this image?###Assistant: This picture shows a red stop sign.
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31691). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The tests will be failing until we add the templates on the hub. I will ping when we decide how to tackle the breaking change and it's is ready for review

Removed default chat templates (#31733 already removed all defaults) and added a hack to load chat templates from a separate json file called "chat_template.json". We can leave it as is for 2-3 major versions and then remove, because I guess everyone will have a version with `chat_template` in `init` by then. 

Ready for review @amyeroberts :)

@amyeroberts I deleted a comment that it's a temporary hack, as we decoded to leave templates in their own json files. Ready for the final review

I encountered something today on how we handle chat templates in some existing models (llava-next-video).

Llava-next-video was merged exactly at the moment when we merged chat-template PR and thus has it template in `processor_config`. Merging this PR will not break anything anywhere, but imo we need to be consistent on where the template is stored in the hub.

I'll upload a `chat_template.json` file on the hub for llava-next-video today. Now we're raising a warning if we find `processor_kwargs["chat_template"]` in any of the models, asking users to keep it in its own file.

Update: Also we needed to change the way processors are saved and save templates in CHAT_TEMPLATE_NAME file.
fix: wandb artifact throws error if user sets wandb runtime name with "/"
Author: YeonwooSung
Pull request: yes

# What does this PR do?

Wandb artifact throws error when the user names the wandb runtime with string that contains "/".

For example, if the user sets the wandb runtime name with the "mistralai/Mistral-Nemo-Instruct-2407", which is the exact name of the model in the huggingface hub, the wandb artifact throws exceptions.

Since most of the models in the huggingface-hub contain "/" in its name, might be better to replace the "/" by "-" automatically.


Comments:
Hi @YeonwooSung, thanks for opening this PR! 

I'm slightly wary of this auto-renaming as it can create surprising behaviour for users. I'd be more of a fan if at the start of the `setup` method, we just raise an error if "/" is present in `model_name`. Then the user can decide how they wish to modify. 

> Hi @YeonwooSung, thanks for opening this PR!
> 
> I'm slightly wary of this auto-renaming as it can create surprising behaviour for users. I'd be more of a fan if at the start of the `setup` method, we just raise an error if "/" is present in `model_name`. Then the user can decide how they wish to modify.

Hi @amyeroberts, thanks for the reply.
That makes sense. Will close this PR then.
Support generating with fallback for short form audio in Whisper
Author: Kimahriman
Pull request: no
Labels: name: Feature request

name: Good Difficult Issue

name: Audio

### Feature request

Generating with temperature fallback based on certain criteria was added to Whisper as part of the long-form generation. We should be able to apply the same fallback criteria to short-form audio. See the discussion [here](https://github.com/huggingface/transformers/pull/27658#issuecomment-1976370427).

### Motivation

The upstream OpenAI implementation does fallback for all audio. In fact there is no distinguishing between "short" and "long" audio, everything is essentially treated as "long audio", and if there's only one segment to transcribe, that's all.

See https://github.com/openai/whisper/blob/main/whisper/transcribe.py#L178

### Your contribution

I probably cannot address this myself.

Comments:
cc @sanchit-gandhi @ylacombe 

cc @sanchit-gandhi, seems that there's a few requests for making long-form audio features compatible with short form audio, do you have time to look into this ?

This is a very valid request and we should indeed refactor `generation_whisper.py` to make no distinction between short and long-form generation (e.g. as per the original codebase). Would you like to have a go at this @kamilakesbi? Happy to help with reviews and questions!
Chameleon: not supported with fast load
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

I added a flag indicating fast load in not supported, due to the composite nature of the model. Also, changed `dtype` in docs to `bf16` to match the implementation in project repo.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32091). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The ProgressCallback triggers a `cannot pickle '_thread.lock' object` failure
Author: fozziethebeat
Pull request: no
Labels: name: bug

### System Info

Transformers version[v4.40.0](https://github.com/huggingface/transformers/releases/tag/v4.40.0) onward

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

With Transformers version[v4.40.0](https://github.com/huggingface/transformers/releases/tag/v4.40.0) onwards, [this commit ](https://github.com/huggingface/transformers/commit/766810153b3a701e99229229c3899e370be7b7a1) introduces a deep copy of logs before reporting progress.

When this deep copy happens *after* the [WandbCallback](https://github.com/huggingface/transformers/blob/b31d59504003c8140adf66a4077b1c50799fbe89/src/transformers/integrations/integration_utils.py#L728) runs, some wandb objects can't be picked.  Specifically, when TRL logs a [WandB table](https://github.com/huggingface/trl/blob/4e85bd75a9dfca0074eef3a90130054c283eed39/trl/trainer/dpo_trainer.py#L1558) , the `WandbCallback` mutates the table such that pickling breaks and ultimately the `ProgressCallback` fails with a stack trace documented [here](https://github.com/huggingface/trl/issues/1849).

I think the fix is to update [this commit](https://github.com/huggingface/transformers/commit/766810153b3a701e99229229c3899e370be7b7a1) to avoid doing any pickling.

### Expected behavior

WandB tables shouldn't be pickled when doing callbacks.

Comments:
Hi @fozziethebeat, thanks for reporting! 

Would you like to open a PR with the proposed fix? This way you get the github contribution for finding the solution 

Sure, I see there's no unittesting (as far as I can tell) for this code path so let me try this out and manually test with an end to end trl run.

Done!

Thanks! 

>Sure, I see there's no unittesting (as far as I can tell) for this code path so let me try this out and manually test with an end to end trl run.

Yes, the transformers team don't officially maintain the integrations with libraries like wandb, so we don't have a CI or tests for those. Thanks for adding the fix and manually checking! 

Glad to fix this.  Had to wander through a lot of layers to figure out where this was being triggered.