noamaanMulla-03/transformers-issues huggingface dataset:
fix: condition bos_token_id and space as token
Author: desaxce
Pull request: yes

Fixes # (https://github.com/huggingface/transformers/issues/36210). 

### Changes:
In the `heal_tokens` function, we now: 
- allow tokenizers which don't have a `bos_token_id`
- skip replacement of space when tokenizer doesn't specify space character as token

Just making the code not raise exceptions, there may be something more clever to do. 

### To test:
Use token healing on https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct:
```
from transformers import AutoTokenizer, Qwen2ForCausalLM, Qwen2Tokenizer

pipe = Qwen2ForCausalLM.from_pretrained("./")
tokenizer = Qwen2Tokenizer.from_pretrained("./")

prompt = f'Complete the following Lean 4 code:\n\n```lean4\nimport '
inputs = tokenizer(prompt, return_tensors="pt")

generate_ids = pipe.generate(inputs.input_ids, tokenizer=tokenizer, max_new_tokens=1, token_healing=True)
tokenizer.batch_decode(generate_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False)[0]
```

Comments:
cc @arthurzucker @itazap for tokenizers!

gentle ping @arthurzucker @itazap

Thanks for the catch and fix! 🤗 I took a look at the original motivation for 'token healing' (https://github.com/huggingface/transformers/pull/30081) and found that there might be a different root cause to the error. For models like `Qwen`, the space token is actually `"Ġ"` instead of `" "`, and could be handled with:
```python
space_tok = tokenizer.convert_ids_to_tokens(tokenizer.convert_tokens_to_ids(tokenizer.tokenize(" ")))[0]
```

vs. the current:

```python
space_tok = tokenizer.convert_ids_to_tokens(tokenizer.convert_tokens_to_ids(" "))[0]
```

cc @ArthurZucker lmk what you think 😊 

cc @gante for generation

+1 to Ita's comment -- tokenizing the space to get the space token seems more intelligent and robust

The other change, regarding `bos_token_id`, looks good to me :)

Great! Last thing would be great to add a test for this, making sure that we get some output instead of an error, let me know if you'd like to add it or I can help 🤗  @desaxce 

> Great! Last thing would be great to add a test for this, making sure that we get some output instead of an error, let me know if you'd like to add it or I can help 🤗 @desaxce

Will take care of it before end of week.

Appreciate it! Thank you! 😊 

@itazap Sorry for the wrong timing, had a difficult end of week and I kept busy over the weekend. 
Let's plan for a Wed. finish line so we start May ready to merge :)

@desaxce no worries, thanks for the update!
Define warmup allocator for torchao quantization
Author: MekkCyber
Pull request: yes

# What does this PR do?

Since when computing the total number of bytes of a torchao quantized model using : `param_byte_count = param.numel() * param.element_size()` the `param.element_size()` is not correct, we need to account for that and have a factor of 4 for `int8` models and a factor of `8` for `int4` models

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37764). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix duplicated weights in fp8 quantization
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/35926 moved the fp8 params from buffers to parameters (which makes sense), but the quantizer itself was not updated, so the parameters would still be added as buffers leading to duplicated weights (they would live as both buffers and parameters).

Also, note that we SHOULD NEVER use `set_module_tensor_to_device` as it clears the cuda cache at each call which is really inefficient, and defeats all the purpose of cuda warmup we do in `from_pretrained`

cc @SunMarc @MekkCyber 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
[RT-DETR] Improve docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes a "todo" in the docs of RT-DETR. cc @SangbumChoi 

Fixes #37800

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
link is a 404 page
Author: co63oc
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.0.dev0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.0
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.5.2

### Who can help?


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/main/src/transformers/models/rt_detr/configuration_rt_detr.py

![Image](https://github.com/user-attachments/assets/b2369375-06e8-4faf-83fa-3e14301dd43b) The link is a 404 page.

![Image](https://github.com/user-attachments/assets/3e4d4514-d0d2-4d27-b0b8-0d0a734f1476)

### Expected behavior

none

Comments:
:red_circle: Update CLIP vision attention to new attention interface
Author: molbap
Pull request: yes

# What does this PR do?

As per title. Removes the old definitions of `ATTENTION_FUNCTIONS` used before and aligns with the current standard, similar to recent siglip changes by @qubvel . I mostly did it because it'll unbloat the `modular` version of molmo in #33962 . 

I ported the changes to all models copying the CLIP backbone, fixed some outdated copied from and typos.

To note: this also will correctly set the attention weights returned to `None` if `output_attentions` is passed as `False`.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37498). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @Cyrilvallez  for core maintainer review! :saluting_face: 

run-slow clip

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/clip']
quantizations: [] ...

MLCD Vision model was added in https://github.com/huggingface/transformers/issues/36182 and inherits from CLIP- I regenerated the modeling from modular, reran the tests and it seems to work fine! merging this and keeping an eye out :hugs:
Model debugger upgrades
Author: molbap
Pull request: yes

# What does this PR do?

A continuation of #36798 , now:

- The debugger will only output the first and last layer of a sequence of layers.
- mean/stds are added as well, and a `..._SUMMARY.json` file will contain only statistics, not full tensors.
- General printing improvements.


Comments:
Can we make sure hooks are removed after we exit the context manager? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37391). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @eustlb @zucchini-nlp @qubvel @yonigozlan , my fellow model adders, could be helpful! Check out the doc (and @Cyrilvallez but I think you've seen/used it already maybe)

Hey @molbap, I just had a random thought while reviewing a model and was thinking about your super nice util, so dropping it here right now as I'm seeing the tag (this is NOT intended as a follow-up to your util hahaha, just to see if it could be helpful for everyone/others have ideas about that):

Another super cool helper IMO would be some kind of library scanner to find close models for modular. E.g., I want to find a MLP with only 2 Linear layers and Dropout -> helper find the different related models
As far as I know we cannot easily do it with usual IDE search tools as most of the time we don't know (and don't really care, as we have a weight converter most of the time anyway) the actual names of those layers



Following @CyrilVallez's idea, it might indeed be very helpful to run such a tool even across the library to find identical modules that may have differently named attributes but are identical actually. Unfortunately, we cannot rename attributes because it would break the weights loading (or we would need a hook for renaming), but at least we can choose one as a standard and leave comments for the rest, indicating that e.g. LlamaMLP is identical to this one and is preferred to be used for modular purposes

Ah that's true, I'll add a configuration option now to output all the layers and add to the doc. 

For the `modular scanner`: indeed, it'd be a very nice util for model adders as well, agree that we can't easily modularize existing code (we can only if we don't change the names), but I'm pretty sure we could add a name mapping util to make sure to preserve naming. E.g. "this module is identical to that one, but rename `self.attn` to `self.self_attn`".  

Added:
- configurable `do_prune_layers` and associated test @qubvel (+ torch guarding the whole thing, why not)
- more documentation
- docstrings because I'm feeling nice today

![image](https://github.com/user-attachments/assets/4a3d2fc1-a1d3-4379-8c8c-51794be07cd2)

merging!

Very nice!
Add Optional to remaining types
Author: cyyever
Pull request: yes

# What does this PR do?

While there is debate about enabling the related RUFF rules, the changes can be applied first.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37808). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
change XLA deprecated api
Author: SunMarc
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/37301

In torch-xla version 2.7, `xm.xrt_world_size()` is replaced by `xr.world_size() `and `xm.get_ordinal()` is replaced by `xr.global_ordinal()`

Should be safe to switch since it was added in the torch_xla 2.1 release https://github.com/pytorch/xla/pull/5011 and we enforce users to use torch 2.1 minimum 

Not sure what to do with XLA_USE_BF16 and XLA_DOWNCAST_BF16 starting from 2.7. We rely on these for `get_parameter_dtype`. 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM! Not sure I see the direct link with the env variables you mentioned though?

Yeah my bad, here you go ! https://github.com/pytorch/xla/issues/8915#issuecomment-2770324804

Ha nice thanks! Looks like for now it's still safe to use as they will only be gone in 2.8, but something to keep in mind for next release for sure!

LGTM thanks!
[config] revert #37603
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

#37603 introduced a regression by deleting a key from config which should not be deleted. This PR reverts the change

cc @BenjaminBossan to verify it works for PEFT


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37821). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This resolves the original issue, thanks.
Fix: Correct tensor shape comment in Mamba modeling
Author: ShadyPi
Pull request: yes

# What does this PR do?

Correct a wrong tensor shape comment in Mamba modeling.

Fixes # (None, as no issue was explicitly mentioned)

The comment on line 307 of `src/transformers/models/mamba/modeling_mamba.py` incorrectly states the shape of the tensor `scan_output`. It should be `[batch, intermediade_size, seq_len]` instead of `[batch, seq_len, intermediade_size]`.

`scan_outputs` is a list of `seq_len` `[batch, intermediade_size]` tensors. When stack them on dimension `-1`, the correct shape should be `[batch, intermediade_size, seq_len]`.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
    Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
    to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
    [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
    [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? (Not applicable for a comment correction)


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @alxndrTL 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AutoTokenizer: Phi-3 drops spaces when decodes a token at a time
Author: Andrei-Aksionov
Pull request: no
Labels: name: Core: Tokenization

### System Info

- `transformers` version: 4.41.2
- Platform: macOS-14.5-x86_64-i386-64bit
- Python version: 3.11.6
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

phi_2_tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
phi_3_tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

for name, tokenizer in (("phi-2", phi_2_tokenizer), ("phi-3", phi_3_tokenizer)):
    print(f"Tokenizer: {name}")
    tokens = tokenizer.encode("This is a test string")
    print(f"{tokens=}")
    print(tokenizer.decode(tokens))
    print("".join([tokenizer.decode(token) for token in tokens]))
    print("-" * 50)
```

```text
Tokenizer: phi-2
tokens=[1212, 318, 257, 1332, 4731]
This is a test string
This is a test string
--------------------------------------------------
Tokenizer: phi-3
tokens=[1, 910, 338, 263, 1243, 1347]

 This is a test string

Thisisateststring
--------------------------------------------------
```

### Expected behavior

I expect that, even if I decode a single token at a time, the resulting string should contain spaces between tokens.
As one can see, with Phi-2 model there are no problems, but for some reason Phi-3 does produce such a concatenated string.

Comments:
cc @itazap 

Hey @Andrei-Aksionov , thanks for the reproducer! It has to do with Phi-3 being based on the LlamaTokenizerFast and Phi-2 on CodeGen. LlamaTokenizerFast strips leading whitespace in order to manually add a prefix space on `add_prefix_space`. I'm looking into a fix now that handles this better! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Is there currently a workaround for this behavior?

Hey @mattf1n 

Yes, I've implemented a workaround [here](https://github.com/Lightning-AI/litgpt/blob/main/litgpt/tokenizer.py#L146-L152).
Don't know what are edge cases, but seems that it works.
SAM-HQ implementation in transformers
Author: IskanderNakipov
Pull request: no
Labels: name: New model

name: Vision

### Model description

HQ-SAM is an upgrade SAM for high-quality zero-shot segmentation. It's design mostly repeats the design of SAM, but additionally early VisionEncoder features are used in enhanced MaskDecoder for fine-grained segmentation

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Code is available here https://github.com/SysCV/sam-hq
Paper https://arxiv.org/abs/2306.01567

Authors @lkeab et al.

Comments:
cc @amyeroberts @qubvel @molbap 

Linking PR https://github.com/huggingface/transformers/pull/31150

@qubvel @amyeroberts Is anyone working on this PR ? 
I would like to contribute on this issue the above pr has been closed actually .
if its not done will start working on pr


Hi @sushmanthreddy , AFAIK there is no one currently working on this PR, @IskanderNakipov has done some work on this model on above PR that you can use. Feel free to start and ping me for a first review :) 

Since SamHQ is based off Sam, we should use `modular` transformers for the addition. `modular` allows using inheritance in transformers, I recommend checking https://huggingface.co/docs/transformers/en/modular_transformers as well as https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma2/modular_gemma2.py, https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm/modular_glm.py . That will allow you to reuse components of Sam implicitly, and write explicitly only what's new.

@molbap thanks for the directions ,will start  working on it
Fast image processor for VitMatte added and bug in slow version fixed
Author: henrikm11
Pull request: yes

-) Added a fast image processor for VitMatte
-) fixed a bug in the slow image processor that processed images incorrectly for input format ChannelDimension.FIRST in which case the trimaps were not added in the correct dimension, this bug was also reflected in the tests through incorrectly shaped trimaps being passed to the preprocess function.
-) adjusted old tests to also cover the fast image processor
-) expanded tests





Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @yonigozlan

@yonigozlan thanks a lot for the detailed review, seems like i got sidetracked by the bug a bit and lost sight of other things..also, I hope I pushed forcefully correctly following the contribution guide.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37616). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Gemma3 is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

### Feature request

Enable Gemma3 model for ExecuTorch

### Motivation

See details in https://github.com/huggingface/transformers/issues/32253

### Your contribution

Enablement

Comments:
Done. E2E enablement can be found in Optimum-ExecuTorch: https://github.com/huggingface/optimum-executorch/pull/58
Qwen3 is ExecuTorch compatible
Author: guangy10
Pull request: no
Labels: name: Feature request

### Feature request

Enable Qwen3 model for ExecuTorch

### Motivation

See details in https://github.com/huggingface/transformers/issues/32253

### Your contribution


Enablement

Comments:
E2E support in Optimum-ExecuTorch: https://github.com/huggingface/optimum-executorch/pull/59

Merged
Add "EAT: Self-Supervised Pre-Training with Efficient Audio Transformer"
Author: roskoN
Pull request: no
Labels: name: New model

name: Feature request

### Model description

The original authors of the model write: 
> EAT is an audio self-supervised learning model with high effectiveness and efficiency during self-supervised pre-training. You can find details in the paper [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://arxiv.org/abs/2401.03497).

A self-supervised learning model can benefit the community greatly, since it requires no labelled data, and can be trained on any dataset. Especially since, the strength of this approach is that it can be applied to variable-length audio. With enough resources (for example, compute, and, data), it could have a similar reach as BERT.  

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

- GitHub Repo: https://github.com/cwx-worst-one/EAT
- Links for model checkpoints:
    - [EAT-base_epoch30](https://drive.google.com/file/d/19hfzLgHCkyqTOYmHt8dqVa9nm-weBq4f/view?usp=sharing) (pre-training)
    - [EAT-base_epoch30](https://drive.google.com/file/d/1aCYiQmoZv_Gh1FxnR-CCWpNAp6DIJzn6/view?usp=sharing) (fine-tuning on AS-2M)
    - [EAT-large_epoch20](https://drive.google.com/file/d/1PEgriRvHsqrtLzlA478VemX7Q0ZGl889/view?usp=sharing) (pre-training)
    - [EAT-large_epoch20](https://drive.google.com/file/d/1b_f_nQAdjM1B6u72OFUtFiUu-4yM2shd/view?usp=sharing) (fine-tuning on AS-2M)
- Paper: https://www.ijcai.org/proceedings/2024/421

Comments:
@roskoN 
@LysandreJik 

Are there any updates?