noamaanMulla-03/transformers-issues huggingface dataset:
`Qwen2_5_VLVisionAttention` with flash attention has no `is_causal` attribute
Author: allentsouhuang
Pull request: no
Labels: name: bug

### System Info

transformers v4.53.0

### Who can help?

@amyeroberts, @qubvel 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

New in v4.53.0, I see that Qwen2_5_VLVisionAttention can now use flash attention. 

https://github.com/huggingface/transformers/blob/v4.53.0/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L252

However, in this line https://github.com/huggingface/transformers/blob/v4.53.0/src/transformers/integrations/flash_attention.py#L71

we assume that Qwen2_5_VLVisionAttention has an is_causal attribute but by inspection it does not. This crashes inference!

### Expected behavior

I would expect it not to crash.

Comments:
What is one supposed to do to over come this? I need to use a recent version of transformers due to other dependencies. And I also want to use flash attention. What is the recommended way to solve this? 

Wait for solution. 

Will be fixed in https://github.com/huggingface/transformers/pull/39121 and we'll add it in patch release

> Will be fixed in [#39121](https://github.com/huggingface/transformers/pull/39121) and we'll add it in patch release


Thank you， when will it be online?


so how to fix it?

For now it works with `transformers==4.52`

> For now it works with `transformers==4.52`

Thanks @svaraborut, below worked for me

```
pip uninstall transformers
pip install transformers==4.52
```

It'd be great if it's fixed, as I have another dependency that requires transformers >=4.53. 🤦🏼 Tku

Would be great if the patch is published relatively soon as well !

It should be done today!
fix default value of config to match checkpionts in LLaVa-OV models
Author: ved1beta
Pull request: yes

# What does this PR do?
fix default value of config to match checkpionts in LLaVa-OV models

Fixes #39089
## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@zucchini-nlp 


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llava_onevision

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39163). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Blip2 fixes
Author: remi-or
Pull request: yes

This PR fixes some tests for the `blip_2` model:
1. there was a device mismatch issue when `getattr(self.config, "image_token_id", None) is not None` was `True`
2. the  `Blip2EncoderLayer` could be split, which led to faulty device maps
3. although the `qformer` attribute does not support FA2, some models that inherited from `Blip2Model` did not change the support flag, which led to error, eg:
```
tests/models/blip_2/test_modeling_blip_2.py::Blip2ForConditionalGenerationDecoderOnlyTest::test_flash_attn_2_fp32_ln
ValueError: Blip2QFormerModel does not support Flash Attention 2.0 yet. Please request to add support where the model is hosted, on its model hub page: https://huggingface.co//discussions/new or in the Transformers GitHub repo: https://github.com/huggingface/transformers/issues/new
```
4. added some expectations for AMD

About (3): not sure if we want to mark some models as not supporting FA2 when it's only a single of their sub-modules that does not support it, but there way the test is written makes me think it's the appropriate fix, so comments are welcome on that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39080). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blip_2

run-slow: blip_2
[docs] ViTPose
Author: stevhliu
Pull request: yes

Updates model card for ViTPose and fixes #36773 

Comments:
Sorry for the delay @NielsRogge, but I think it should be resolved here. Just trying to figure out why the `build_pr_documentation` test is failing :(

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38630). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let me know if there any other changes @NielsRogge , otherwise lets merge it! 😄
Move get_mask_sizes from Cache to masking_utils and remove use of get_seq_length.
Author: manueldeprada
Pull request: yes

This PR depends on #39106

Look at the last commit, f09e0cd1c0d61fca0098487e9dcd9dfe409b364b:

I think having the get_mask_sizes out of cache makes much more sense. There is only one extra change:
https://github.com/huggingface/transformers/blob/6b6314d5f8304a1f547b04a00a4bd06a11f51681/src/transformers/masking_utils.py#L643
It substitutes `past_seen_tokens=past_key_values.get_seq_length()` (which depends on cache info that might be hard to cumpute, i.e., QuantizedCaches). What we would like to compute is
```py
past_seen_tokens = cache_position[-1]
```
but that is not compatible with `torch.export`.

The new solution is `torch.export` friendly and works both when `cache_position = torch.tensor([ 0,  1,  2,  3,  4,  5,  6])` (prefill phase) and when `cache_position = torch.tensor([16])`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39142). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Reduce Glm4v model test size significantly
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. The model test was about 180M params, making the CI quite slow. cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39173). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v

Thanks!

Yes, less than 1M params now, so somewhere between 180x and 200x smaller 🤗

wowww!
Modular m4t speecht5 sew
Author: nikosanto13
Pull request: yes

# What does this PR do?

Completes the effort of #35902, by adding modular files for ~~SEW~~, SEW-D, SeamlessM4T, SeamlessM4Tv2, SpeechT5 ([as proposed](https://github.com/huggingface/transformers/pull/35902#pullrequestreview-2705453204) by @eustlb)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker  @eustlb 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @eustlb and maybe @vasqu

Again, thanks so much for the good work @nikosanto13 , reviewing ASAP (coming days) 🤗

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: seamless_m4t, seamless_m4t_v2, sew, sew_d, speecht5, unispeech, unispeech_sat, wav2vec2_bert
[generate] document non-canonical beam search default behavior
Author: gante
Pull request: yes

# What does this PR do?

Related to #38778: this PR adds more context behind some legacy decisions that shouldn't be broken for BC purposes.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39000). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[serve] Model name or path should be required
Author: LysandreJik
Pull request: yes


Comments:
Need to handle the case where no URL is passed which spawns a server

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39178). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Random serve fixes
Author: pcuenca
Pull request: yes

* Index out of bounds repro
  - Start server
  - Open chat session using model `meta-llama/Llama-3.2-3b-Instruct`, enter `hello` in the chat
  - Open another session to the same model, enter the same message `hello`
* The chat cli specifies the revision, while other clients (Jan) don't. This may result in model reloading (and memory doubling) when it's not necessary. We compare the full names now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39176). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Pinging @gante as well for the `is_continuation`

haha, the same fix is in [@gante's PR](https://github.com/huggingface/transformers/pull/39133) lol

(updating branch to rerun CI, the rerun button is not working 🤔 )
Update expected values (after switching to A10) - part 2
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline, will merge to move fast as it's only expected outputs updated for A10

Comments:
run-slow: grounding_dino, mask2former, maskformer, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, rt_detr, rt_detr_v2, segformer

This comment contains run-slow, running the specified jobs: 

models: ['models/grounding_dino', 'models/mask2former', 'models/maskformer', 'models/mobilenet_v1', 'models/mobilenet_v2', 'models/mobilevit', 'models/mobilevitv2', 'models/rt_detr', 'models/rt_detr_v2', 'models/segformer']
quantizations: [] ...

run-slow: grounding_dino, mask2former, maskformer, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, rt_detr, rt_detr_v2, segformer

@remi-or This will fail some AMD tests. As discussed on slack, we need to change the behavior of `Expectations`.

This comment contains run-slow, running the specified jobs: 

models: ['models/grounding_dino', 'models/mask2former', 'models/maskformer', 'models/mobilenet_v1', 'models/mobilenet_v2', 'models/mobilevit', 'models/mobilevitv2', 'models/rt_detr', 'models/rt_detr_v2', 'models/segformer']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39165). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: grounding_dino, mask2former, maskformer, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, rt_detr, rt_detr_v2, segformer

run-slow: grounding_dino, mask2former, maskformer, mobilenet_v2, rt_detr, rt_detr_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/grounding_dino', 'models/mask2former', 'models/maskformer', 'models/mobilenet_v2', 'models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...

run-slow: grounding_dino, mask2former, maskformer, mobilenet_v2, rt_detr, rt_detr_v2

This comment contains run-slow, running the specified jobs: 

models: ['models/grounding_dino', 'models/mask2former', 'models/maskformer', 'models/mobilenet_v2', 'models/rt_detr', 'models/rt_detr_v2']
quantizations: [] ...
Test fixes for Aria (and some Expectation for llava_next_video)
Author: remi-or
Pull request: yes

This PR aims to fix some issues with the `aria` model tests:
- the image used in the test is fetched from an url that is no longer up: I replaced the url with the image used in `llava-vl` which seems to be the same
- some test have an outdated image token `<image>` which I updated to the new `
` token
- there are some device / dtype issues that were fixed with `.to`
- added Expectations for AMD. Also did that for `llava_next_video` and it's in this PR because it's a little to small for a standalone PR imo.

All in all, this brings the number of failing tests from 21 to 7 in `aria`. The ones leftover have to do with a larger issue of FA2 in the `AriaTextModel` which seems a little out of scope here since it's modeling code.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39131). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, llava_next_video
scale loss per token/local sequence for discrete system representation
Author: wesboyt
Pull request: no
Labels: name: Feature request

### Feature request

I have implemented a scaling loss for my discrete game usecase.
I would like to be able to hand my model a batch with a map of sequences and their associated randomness(or importance) and then scale the loss relative to the next tokens predictability.
My implementation works but is very slow relative to: 
https://github.com/huggingface/transformers/blob/v4.52.3/src/transformers/trainer.py#L3795

### Motivation

My system has highly variable regions that are truly random and prevent my models from converging without manual training which is incredibly inefficient.

### Your contribution

My implementation:
```
    for val in train:
        val = torch.tensor(val['input_ids'], dtype=torch.long).to(device)
        logits = model(val).logits
        ttl_loss = torch.zeros(1,dtype=torch.float32).to(device)
        for i in full_range:
            loss = loss_function(logits[:, i - 1], val[:, i])
            if i > 26 or i < 6:
                losses.append(loss.item())
            else:
                loss = loss * 0.1
            ttl_loss += loss
        ttl_loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        lr_scheduler.step()
```
I improved accuracy for these desireable and rarer sequences from 88% to 96% in only 20000 sentences.

Comments:
Hi @wesboyt, scaling classification loss with the sequence entropy is a cool idea and could definitely be relevant in some domains, including DNA. However, it's quite advanced and extremely domain-specific, so I don't think we should add it to the `Trainer`! cc @sunmarc to confirm.

Was just comparing the performance to trainer loop (which is roughly 15x faster).
Would not intend to tell your team how or where to implement it.


I could imagine a impl where the user just hands another field with the batch, loss_modifier or something. with the expectation being they iterated over the inputs and calculated which tokens deserved which loss modifier.
batch{
    inputs:[[1,2,3]],
    loss_mod:[[0.05,0.1,1.0]],
    attention:[[1,1,1]]
}
[glm4v] fix video inference
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

I broke video inference in a recent PR by copying from Qwen2-VL. Comes out GLM4V doesn't have a video token and instead uses image token id repeated many times per frame

No need for patching, only the main branch is broken

Comments:
run-slow: glm4v



This comment contains run-slow, running the specified jobs: 

models: ['models/glm4v']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39174). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm4v
[BugFix] QA pipeline edge case: `align_to_words=True` in `QuestionAnsweringPipeline` can lead to duplicate answers
Author: Vixel2006
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #26286

Solves the duplicate answers problem when `align_to_words=True` which is default in the `question_answering` pipeline  by keeping track of the answers history and if the answer is already in the answers list we just merge the two answers by summing their probabilities.
```py
from transformers import pipeline

answers = pipeline("question-answering", model="deepset/tinyroberta-squad2")(
    question="Who is the chancellor of Germany?",
    context="Angela Merkel was the chancellor of Germany.",
    top_k=10,
)

for answer in answers:
    print(answer)
```
This solution makes sure that there is no duplicate answers in the output but we don't know how many answers are returned.

## Who can review?
[@Rocketknight1](https://github.com/Rocketknight1) [@ArthurZucker ](https://github.com/ArthurZucker)


Comments:
Hi @yushi2006, PR seems good, but I have a couple of suggestions:

1) We don't need a `handle_duplicate_answers` argument. We can just do that by default!
2) We don't want to skip duplicate answers. Instead, we should probably merge their scores somehow. Maybe taking the highest score, or summing their probabilities?

[@Rocketknight1](https://github.com/Rocketknight1) Thanks for the feedback! I've removed the `handle_duplicate_answers` argument and now merge duplicate answers by summing their probabilities, as suggested. Let me know if you'd prefer a different merge strategy or if there's anything else to tweak.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38761). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Rocketknight1! I just updated the branch and it reset the doc build job — could you please re-trigger the “Build PR documentation” check so I can merge? All other checks are passing again

@yushi2006 yes, sorry - I thought I'd set auto-merge last time. Merging now, and thanks again for the PR!
Suggest jobs to use in `run-slow`
Author: ydshieh
Pull request: yes

# What does this PR do?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39100). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Suggested jobs to run:\n

Suggested jobs to run:\nrun-slow: bark bert autoround fbgemm_fp8

run-slow: bark, bert, autoround, fbgemm_fp8

run-slow: bark, bert, autoround, fbgemm_fp8

This comment contains run-slow, running the specified jobs: 

models: ['models/bark', 'models/bert']
quantizations: ['quantization/autoround', 'quantization/fbgemm_fp8'] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark, bert, autoround, fbgemm_fp8

> which will run them by default?

nono, just suggestion. I don't want to trigger GPU runs on each PR commit. 

And for security reason, we can't checkout to PR branch without team member explicitly leaving the comment.

Hi @ydshieh , This workflow is adding the comment after every commit and even on a draft PR. It would be great if we can fix this as it's super annoying - probably only add the comment if the PR is approved so kind of suggesting the maintainers that run-slow before merging if required as the PR is approved.

Hi @yaswanth19 

The previous comment is deleted in the workflow, but one has to refresh page, don't know why it (page content refresh) could not be done automatically.

>  only add the comment if the PR is approv

I tried to `trigger when an approval review submitted` but that could not be used (security reason).

I will try to add the comment (and of course delete the previous one) only if the previous one is 30 minutes ago or longer.



#39170 should make it better 🤗 

Great feedback @yaswanth19 ! Completely 🤗
Decouple device_map='auto' and tp_plan='auto' 
Author: SunMarc
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/38771. 
This PR decouples device_map='auto' and tp_plan='auto'. Before, launching a distributed run when the `device_map` was set to "auto" triggered TP. We removed that and the user should instead pass tp_plan="auto" or tp_size to tigger TP. 
Also, we warn/info that this might lead to an unexpected behavior with someone tries to run a distributed run with device_map = "auto". 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38942). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

is it possible to land this? people are still hitting DTensor + DDP error and filing issues in pytorch

Merged !
Make run_object_detection compatible with `datasets` 4.0
Author: lhoestq
Pull request: yes

run_object_detection isn't compatible with `datasets` 4.0, here is this error:

```
examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_object_detection - AttributeError: 'dict' object has no attribute 'feature'
```

This change makes run_object_detection compatible with `datasets` 4.0 (out next week). Existing dataset will still work as soon as `datasets` 4.0 is used.

Keeping this as a draft and we can merge once `datasets` 4.0 is out

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39060). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

That would be great but it requires another PR to add torchcodec, I'll try to to make it later 
today


No stress, but let me know if I could trigger a run this weekend. It's ok if we do it next week when you are ready :-)

I opened https://github.com/huggingface/transformers/pull/39156 which adds torchcodec so you can run the test suite to see if everything is fine with `datasets` on `main` (4.0.0.dev0)

Let me know if you need anything else

closing this one in favor of https://github.com/huggingface/transformers/pull/39156 which includes it