noamaanMulla-03/transformers-issues huggingface dataset:
add a new flax example for Bert model inference
Author: louie-tsai
Pull request: yes

# What does this PR do?

Add a new Flax inference example and show case how to optimize performance with Bfloat16 data types.
Most of the Flax examples are for Training/Fine Tuning, so it should be a good reference for inference



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



@ArthurZucker
@sanchit-gandhi
@stevhliu
@agramesh1
@mdfaijul

Comments:
Thanks! Can you run `make fixup` or `make style` to fix the failing CI test (see step 5 [here](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request) for more details)?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34794). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks! Can you run `make fixup` or `make style` to fix the failing CI test (see step 5 [here](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request) for more details)?
@stevhliu  ran and fixed the issue accordingly. thanks

 @ArthurZucker   Could you help to review the PR? much appreciated

> Thanks! I think adding an example with Llama would be a tad bit better as more up to date, but thanks for the contrib!

that indeed could be next examples or new feature based on the PR. we will look into that. thanks.
Add support for GOT-OCR2.0
Author: VladOS95-cyber
Pull request: no
Labels: name: New model

### Model description

As an OCR-2.0 model, GOT can handle all artificial optical signals (e.g., plain texts, math/molecular formulas, tables, charts, sheet music, and even geometric shapes) under various OCR tasks. On the input side, the model supports commonly used scene- and document-style images in slice and whole-page styles. On the output side, GOT can generate plain or formatted results (markdown/tikz/smiles/kern) via an easy prompt. Besides, the model enjoys interactive OCR features, i.e., region-level recognition guided by coordinates or colors. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Implementation: https://github.com/Ucas-HaoranWei/GOT-OCR2.0/
Paper: https://arxiv.org/abs/2409.01704

Comments:
Hello, If someone from the core-team is not already working on this, or there is interest it in, I would really love to contribute this model to transformers with some help!

hi @VladOS95-cyber If you don't mind can I help u with this issue, if you are working on it ?

Hi @GargDivanshu, I don't mind at all, let's wait for decision from @qubvel @LysandreJik 


Hey @VladOS95-cyber @GargDivanshu !
I'm planning to start working on it very soon, I'll tag this issue once I've opened a PR for it, if you want to have a look then!

cool 🙌 

+1

Any movement on this? Looking forward to trying it out

Been testing stepfun's demo code along with the model -- would love to see this in transformers!

Hey all!
Implementation is well underway, and I'll open a PR in a couple of days for it (the entire Hugging Face team is currently at an off-site). Most likely, only inference will be available initially, and support for fine-tuning will be added if there is strong demand for it.

@yonigozlan if you remember, please paste the PR link in this thread, would love to subscribe

Hi again!
The GOT-OCR PR is live [here](https://github.com/huggingface/transformers/pull/34721) if you want to follow the progress :)

GOT-OCR is now merged in the main branch of Transformers 🤗
BatchEncoding.to throws away columns silently, thus no way to pass non-tensor columns such as String in Trainer metric computation
Author: fzyzcjy
Pull request: no
Labels: name: bug

### System Info

unrelated

### Who can help?

@muellerzr @SunMarc
(original tags, no longer valid)

@ArthurZucker 
(re-tag because want to discuss patch release)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi thanks for the library! Consider this simple line:

```
x = transformers.tokenization_utils_base.BatchEncoding({'a': ['x','y']})
x.to('cpu') # or cuda or whatever
```

The column `a` is then silently removed :(

This is annoying in the following scenario: For each of my training/eval sample, I have a string column that serves as a tag for it, and want to utilize it when computing metrics and losses.

Then it does not work. After some debugging, the root reason is that it gets silently removed in the `to` mentioned above.

It seems torch does not support a tensor of dtype `str`, thus it seems impossible to have data pass through.

### Expected behavior

(see above)

Comments:
Hmm looks like https://github.com/huggingface/transformers/pull/33932 already fixed in on master? Then I would appreciate it if it could be released as a patch!

Seems I need to tag @ArthurZucker instead (given that https://github.com/huggingface/transformers/commits/v4.46.3 patch released are done by him). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

With the latest release, is the issue fixed @fzyzcjy ? 

@SunMarc Have not tried yet, will reply when upgraded!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
feat: pyutils variable for pytorch 2.5
Author: ruidazeng
Pull request: yes

Pytorch 2.5 released in October 17, 2024 features a new CuDNN backend for SDPA, improvements to TorchDynamo, regional compilation of torch.compile, and more.

https://pytorch.org/blog/pytorch2-5/

This also causes some backward compatibility issue, where some old models could not run on the newer Pytorch stable release version 2.5, so they will need this variable to assists (this variable needs to be `FALSE` in that case).

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
modernbert logits do not have gradient
Author: andersonbcdefg
Pull request: no
Labels: name: bug

### System Info

latest transformers version (from source), python 3.10

### Who can help?

@Arthurz

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
model_id = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id).to("cuda")
# Create a simple input
inputs = {
"input_ids": torch.randint(0, 1000, (1, 10)).cuda(),
"attention_mask": torch.ones(1, 10).cuda()
}

# Set to train mode and check all parameters
model.train()
for name, param in model.named_parameters():
    print(f"{name}: requires_grad = {param.requires_grad}")

# Do forward pass
outputs = model(**inputs)
print("\nOutput logits requires_grad:", outputs.logits.requires_grad)
print("Output logits grad_fn:", outputs.logits.grad_fn)
```

### Expected behavior

When I do this, the output is:
```
Output logits requires_grad: False
Output logits grad_fn: None
```

Despite explicitly setting all the parameters to requires_grad = True! And when printing all the params, they all are correctly set to requires_grad = True.

Just to sanity check, I ran the same code but set model_id = "bert-base-uncased", and got:
```
Output logits requires_grad: True
Output logits grad_fn: <ViewBackward0 object at 0x7f0ca6abf370>
```

So it's def a ModernBERT specific problem!

Comments:
This is a bug, and I have a fix for it in #35404.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing this was as it was fixed thanks to the PR above.
[generation] automatic compilation fixes: don't log when `output_attentions` is `True`
Author: gante
Pull request: yes

# What does this PR do?

WIP

[⚠️ pattern change] Skip the log regarding eager attention fallback at compilation time. Instead, when compiling, silently fallback to eager attention. This was preventing calls like `model.generate(model_inputs, cache_implementation="static", output_attentions=True, return_dict_in_generate=True)` on most models

This fixes a bunch of failing tests and unlocks #33212 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35989). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixing the pattern and testing its effect is exposing new bugs 🤔 

Opening separate PRs to avoid bloating this one.

Closing -- changing this pattern might not be the best idea. Updating tests instead.
Support batching for UsefulSensors Moonshine
Author: njeffrie
Pull request: yes

Add attention masking to the moonshine model.

Tested on Open ASR Leaderboard with batch_size=256.
- WER (tiny): 12.53
- WER (base): 9.89
- RTFx (tiny): 2062
- RTFx (base): 1634

Unblocks [this OpenASR Leaderboard PR](https://github.com/huggingface/open_asr_leaderboard/pull/48)
@eustlb 

Comments:
[generation] fix type hint
Author: gante
Pull request: yes

# What does this PR do?

Fixes #35723 (code crashes because of bad type hint)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35725). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry, my bad, thanks for fixing.

However I'm wondering why this wasn't caught by the tests? I thought they ran on 3.9?

Nightly might yeah but CI is on 3.10 I think

Yes CI is on 3.10. On CircleCI, we might be able to switch to 3.9 easily, but not on daily CI. Let me check.
Add support for Janus model from DeepSeek AI
Author: ighoshsubho
Pull request: no
Labels: name: New model

### Model description

Janus is an autoregressive framework that unifies multimodal understanding and generation. Unlike previous approaches that use a single visual encoder for both tasks, Janus decouples visual encoding into separate pathways while utilizing a unified transformer architecture for processing. This decoupling addresses the conflict between visual encoder roles in understanding and generation, enhancing flexibility and performance.

Key features:

- Unified framework for multimodal understanding and generation
- Decoupled visual encoding pathways
- Single, unified transformer architecture for processing
- Improved performance in multimodal understanding tasks
- Flexibility to select optimal encoding methods for each component

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The Janus model is developed by DeepSeek AI. Here are the relevant links for implementation:

Paper: [Janus: Bridging the Gap Between Multimodal Understanding and Generation](https://arxiv.org/pdf/2410.13848)
GitHub repository: [deepseek-ai/Janus](https://github.com/deepseek-ai/Janus)

Comments:
Would be super nice to have indeed! 🤗 
cc @yonigozlan @zucchini-nlp @qubvel 🤗 

I think there are people working on it (https://github.com/huggingface/transformers/issues/35928)

Closing as a duplicate!
[generate] handle attention mask (encoder + cross attention) for encoder decoder models
Author: eustlb
Pull request: no
Labels: name: Feature request

### Feature request

For encoder-decoder autoregressive models attention masks, we follow the convention (cf [here](https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/generation/utils.py#L1958C67-L1959C119)):
`attention_mask` →  applies to the encoder
`decoder_attention_mask` →  applies to the decoder

Moreover, we also go with the following convention concerning modeling code:
1. a `CoolModelNameModel` class that handles the forward pass, inferring the encoder if `encoder_outputs` is not provided and inferring the decoder
2. a usage-specific class that will inherit from the amazing `GenerationMixin` and hold a `CoolModelNameModel` instance. 

For reference, the original code for `SpeechEncoderDecoderModel` forward calls (same goes for other encoder-decoder models like [Bart](https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/models/bart/modeling_bart.py#L1643-L1659) or [Moonshine](https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/models/moonshine/modeling_moonshine.py#L1589-L1603)):
https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py#L545-L558
where `encoder_attention_mask` is inferred from the passed `attention_mask`.

🐛 **Such model inputs are prepared in the `prepare_inputs_for_generation`, nevertheless `attention_mask` is lost in:**
https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/generation/utils.py#L409-L411

### Motivation

When doing batch inference and therefore requiring most of the time padding on the inputs values + attention_mask, it is required for the forward of the model to be passed the `attention_mask` in order to compute the correct encoder hidden state and mask the cross attentions.

### Your contribution

I've implemented a naïve fix in #36018 🤗

Comments:
cc @gante @zucchini-nlp 

My bad, looks like it is not a feature request (obviously we did handled attention masks for cross attentions) but rather a bug introduced in https://github.com/huggingface/transformers/pull/35852

Oh right! The fix was needed for Whisper models which needs position ids unlike some other text models. I am wondering, aren't we supposed to prepare encoder-hidden-states before the generation starts, so there should be no need for encoder attention anymore?

Unfortunately Whisper is not a good reference for that! They do not mask the encoder hidden states but rather rely on the fact that the model learned that padding zeros on the input values before passing to the encoder represent silence (so no need to mask). Nevertheless most encoder-decoder models will still need an attention mask for the cross attentions (see Bart or Moonshine for example).
Update Granite Vision Model Path / Tests
Author: alex-jw-brooks
Pull request: yes
Labels: name: run-slow

# What does this PR do?

This is a follow-up on https://github.com/huggingface/transformers/pull/35579, which added support for running IBM's granite vision models as an instance of Llava Next models in `transformers`. Our [2B model preview](https://huggingface.co/ibm-granite/granite-vision-3.1-2b-preview) was released today; this enables the integration test from the other PR, and updates the example in the granite vision docs to use the right model name!

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


@ArthurZucker @zucchini-nlp 



Comments:
[run-slow] llava_next

run-slow: llava-next

run-slow: llava_next

This comment contains run-slow, running the specified jobs: ['models/llava_next'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35998). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the quick reviews!
cannot custom `warmup_min_lr` of deepspeed lr scheduler
Author: SeunghyunSEO
Pull request: no
Labels: name: DeepSpeed

name: bug

### System Info

4.45 version

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

[link](https://github.com/huggingface/transformers/blob/4eb17b26e77611d4fbcdcbbc20c7bf275eb015c9/src/transformers/integrations/deepspeed.py#L171)

idk why it's hardcoded

### Expected behavior

warmup_min_lr value is not working

Comments:
I have created the pull request. Could someone please review it for further improvements?



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Qwen2-VL: fix rope delta calculation
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36009. We need to get new rope deltas every time a new input comes in. When users want to do several `forward` calls we don't reset prev cached rope deltas and try to rely on `cache position`, but users usually don't use `cache_position` at all

This PR adds a condition based on cache being `None` or empty to check if we are in decoding phase or not

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36013). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
apply_chat_template: consistent behaviour for return_assistant_tokens_mask=True return_tensors=True
Author: mrsndmn
Pull request: yes

# What does this PR do?

This PR fixes the simultaneous use of flags `return_assistant_tokens_mask`, `return_tensors`, in `apply_chat_template` method.

Is related to https://github.com/huggingface/transformers/issues/28950 and corresponding PR https://github.com/huggingface/transformers/pull/30650


**Before fix**
```python
output = tokenizer.apply_chat_template(
    conversations,
    tokenize=True,
    padding=True,
    return_dict=True,
    return_assistant_tokens_mask=True,
    return_tensors='pt',
)

assert isinstance(output['attention_mask'], torch.Tensor)  # ok
assert isinstance(output['assistant_masks'], torch.Tensor) # not ok: type of `output['assistant_masks']` is `List`
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yonigottesman @amyeroberts @Rocketknight1 


Comments:
Hi! Take a look, please

Hi @mrsndmn, I'm sorry we missed this PR! @yonigottesman can you take a look? It's okay if you don't have time - just let me know and I can take it instead

Hey @yonigottesman, just checking in — did you have a chance to look at this PR? No rush, but let me know if you need anything from me!