noamaanMulla-03/transformers-issues huggingface dataset:
Split `transformers chat` and `transformers serve` 
Author: LysandreJik
Pull request: yes

This PR splits the `transformers chat` frontend from its backend (now moved to `transformers serve`).

We take this opportunity to have `transformers serve` act as a server with an OpenAI-compatible API, via http, using SSE to stream tokens. 

You can take it for a spin either by spawning a server:

```
transformers serve
```

and then a chat on top of it in another terminal:

```
transformers chat meta-llama/Llama-3.2-3b-Instruct  
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38443). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry maybe I am not understanding this well, but isnt "`transformer serve`" like ollama? 

@qaiwiz it is similar, yes, but `transformers serve` uses `transformers` as a backend :) `ollama` uses `llama.cpp` (see [here](https://github.com/ollama/ollama?tab=readme-ov-file#supported-backends))

With `transformers serve`, everyone can immediately test anything that's `transformers`-compatible in their apps 🤗
Update expected values (after switching to A10) - part 3
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline, will merge to move fast as it's only expected outputs updated for A10

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: dpt, oneformer, poolformer, pvt, pvt_v2, regnet, resnet, seggpt, swin2sr, switch_transformers

run-slow: dpt, oneformer, poolformer, pvt, pvt_v2, regnet, resnet, seggpt, swin2sr, switch_transformers

This comment contains run-slow, running the specified jobs: 

models: ['models/dpt', 'models/oneformer', 'models/poolformer', 'models/pvt', 'models/pvt_v2', 'models/regnet', 'models/resnet', 'models/seggpt', 'models/swin2sr', 'models/switch_transformers']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39179). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Gaudi] the seamless_m4t cannot work on Gaudi. No need to fix. Workaround PR is merged.
Author: yuanwu2017
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.0.dev0
- Platform: Linux-5.15.0-140-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.33.0
- Safetensors version: 0.5.3
- Accelerate version: 1.7.0
- Accelerate config:    not found
- DeepSpeed version: 0.16.1+hpu.synapse.v1.21.0
- PyTorch version (accelerator?): 2.6.0+hpu_1.21.0-555.gitabf798b (HPU)
- Tensorflow version (GPU?): 2.15.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using distributed or parallel set-up in script?: <fill in>
- Using HPU in script?: <fill in>
- HPU type: GAUDI2



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

It is a known issue of Gaudi. The root cause is that Gaudi torch.gather cannot support input of int64 or long . It seems slow to fix this bug from the Gaudi pytorch. After they fix, I will remove the workaround [PR38363](https://github.com/huggingface/transformers/pull/38363).
```
git clone https://github.com/yuanwu2017/llm-dbg.git
PT_HPU_LAZY_MODE=1 run_text-to-speech.py
```

![Image](https://github.com/user-attachments/assets/f23664c1-20cd-48b7-9c71-fac158d29e16)

### Expected behavior

No crash.

Comments:
@IlyasMoutawwakil  @ydshieh @yao-matrix 

Thanks @yuanwu2017 

What I means in [that comment](https://github.com/huggingface/transformers/pull/38363#issuecomment-3005297700) is an issue page that lists all workaround patch (like the one in your PR) done (in several different PRs) that we would remove in the future. 

Not a detailed issue for a particular one. It's most for us, but you can keep this page open if you prefer.





i will take care of creating an workarounds tracker this evening 😊
Better return typehints for `from_pretrained`
Author: qubvel
Pull request: yes
Labels: name: typing

# What does this PR do?

Better typing while using from_pretrained for a specific config/processor/feature_extractor. This allows for better availability of class docs, attributes, and function signatures.

### On `main` (before)
![from-pretrained-typehints-before](https://github.com/user-attachments/assets/c1a9370f-300c-4927-971b-d7937fa08833)

### On branch (after)
![from-pretrained-typehints-after](https://github.com/user-attachments/assets/ac7c4e16-12e1-4060-a46f-cdc00e8de45a)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39184). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is a kind of placeholder for `cls` type, meaning that any class that inherits from `Pretrained*` can be passed, and the instance of that class would be returned.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: clvp, falcon, qwen2_5_omni, t5gemma
Naming incosistencies of `PreTrained*` classes.
Author: vitormbesen
Pull request: no
Labels: name: Feature request

### Feature request

Add an alias `PreTrainedConfig` for the class `PretrainedConfig`, to follow PascalCase convention more closely, and to mirror other `PreTrained*` classes.

### Motivation

`PreTrainedModel`, `PreTrainedTokenizer` follow the PascalCase naming convention. However, `PretrainedConfig` does not. This inconsistency, though minor, is inconvenient. 

### Your contribution

Add the line `PreTrainedConfig = PretrainedConfig` and submit a PR.

Comments:
`transformers`' dependency on `sentencepiece` blocks use on windows in python 3.13
Author: leondz
Pull request: no
Labels: name: Usage

### System Info

Due to 
* changes in Python 3.13,
* an incompatibility in `sentencepiece`,
* `transformers` dependency on `sentencepiece`,

`transformers` cannot be easily installed under windows + py3.13, and does not work as a dependency of other packages in this environment

There are multiple issues and a merged PR on sentencepiece (https://github.com/google/sentencepiece/pull/1084) from Feb 26 2025 but no release has been forthcoming



### Who can help?

* people currently using `sentencepiece` in `transformers` code they own
* people determining what the scope of `transformers`' OS & python support is
* `sentencepiece` pypi maintainers


### Reproduction

1. Be on windows
2. Be on python 3.13
3. Try to install current `transformers` from pypi
4. If you get this far, use any function importing `sentencepiece`, e.g. loading an `xlm_roberta` model

### Expected behavior

Code doesn't raise exception

Comments:
I'm sorry, `sentencepiece` is quite an important dependency for a lot of tokenizers! We probably won't be able to remove it. If `sentencepiece` completely stalls then we might have to figure out a solution or migrate away from it, but most likely there will be a release soon.

Yeah, sentencepiece for sure has its place, it's tough to remove it. I didn't get its value add over bpe when it came out but it was here to stay. 

I know python 3.13 is the current, normal, release version and has been since October 2024. v 3.14 is slated for release in just over three months. I haven't tried it myself but I hear a non-trivial proportion of people, especially newcomers to AI , use Windows for development. And people new to things tend to install the current version. 

There have been many issues, PRs, and comments - with high and increasing frequency - on the sentencepiece repo since 3.13 release last year, with no impact so far. It has 117K projects depending on it, iirc.

How do you quantify "soon"? 

Hey @leondz , `sentencepiece` is not a hard dependency, it's a soft dependency. 
`XlmRoberta` has a `fast` tokenizer, which is backed by `tokenizers`, which is available and tested on windows with python 3.13. 

This begs the question as to why use `sentencepiece`, and what led you to not use:
```python 
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("FacebookAI/xlm-roberta-base") 
tokenizer._tokenizer
```


Great! Please also send your recommendations for
 
* `M2M100Tokenizer`
* `MarianTokenizer`

Or should I take it from the fast close that these classes are now deprecated in `transformers`?

Yep slow ones are basically deprecated
Update expected values (after switching to A10) - part 5
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline, will merge to move fast as it's only expected outputs updated for A10

We are close!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: emu3, encodec, falcon_h1, gemma3, glm, helium, paligemma, qwen2, qwen2_vl, starcoder2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39205). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update expected values (after switching to A10) - part 6
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline, will merge to move fast as it's only expected outputs updated for A10

🚀 🚀 🚀 Almost!

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, gemma

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39207). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove device to host sync triggered in _flash_attention_forward
Author: piyifan123
Pull request: no
Labels: name: Feature request

### Feature request

# Problem
In https://github.com/huggingface/transformers/blob/037755ed54208eefa77673b0af2a0b13e51f2fb1/src/transformers/modeling_flash_attention_utils.py#L521, the condition check `(torch.diff(position_ids, dim=-1) >= 0).all())` would cause the result from the device tensor `position_ids` to be synced to the host side.

During inference/training, it can cause serious performance degradation due to CPU blocking. see the following for an example:

<img width="2476" alt="Image" src="https://github.com/user-attachments/assets/e5c1adf1-940e-407a-b78d-9875d6ce9513" />

# Proposal
Precompute the result (torch.diff(position_ids, dim=-1) >= 0).all()) and store it in the `FlashAttentionKwargs` so that we don't have to perform this device to host sync in every attention call in every layer.

The only question is whether there exists a model that cannot precompute this, i.e., the position_ids seq changes during the forward process for the same batch? Based on the fact that we are caching cu_seqlens in FlashAttentionKwargs anyway (which is equivalent to positions_ids), we can assume that?

### Motivation

As stated above, this could severely degrade the out of box performance of transformers and it's usually hard for normal user to notice. And the fix would follow an existing mechanism, i.e., `FlashAttentionKwargs` approach to avoid recomputation for FA needed kwargs.

### Your contribution

Can prepare a PR if the team thinks the proposed approach is OK.

Comments:
Update expected values (after switching to A10) - part 7
Author: ydshieh
Pull request: yes

# What does this PR do?

The final one 🏅 

As discussed offline, will merge to move fast as it's only expected outputs updated for A10

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere2, data2vec, depth_pro, olmo2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39218). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update expected values (after switching to A10) - part 8 - Final
Author: ydshieh
Pull request: yes

# What does this PR do?

2 remaining failed tests that were missed in the previous PRs.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: moonshine, regnet

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39220). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Possible Typo in "Mask2FormerLoss"
Author: NiccoloCavagnero
Pull request: no
Labels: name: bug

### System Info

None

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

Hi everyone, 

I think there might be a typo within the **Mask2FormerLoss** class in **modeling_mask2former.py**, line **506**.
In particular, when the matcher is defined, the **class_weight** is not passed to the matcher, where there is an hardcoded **1.0**.

Current definition:
```
self.matcher = Mask2FormerHungarianMatcher(
            cost_class=1.0,
            cost_dice=config.dice_weight,
            cost_mask=config.mask_weight,
            num_points=self.num_points,
        )
```

It should be changed to:
```
self.matcher = Mask2FormerHungarianMatcher(
            cost_class=config.class_weight,
            cost_dice=config.dice_weight,
            cost_mask=config.mask_weight,
            num_points=self.num_points,
        )
```

Best, 
Niccolò Cavagnero

Comments:
@NielsRogge you reviewed the original PR, can you take a look?

Hi @NiccoloCavagnero thanks for spotting this, feel free to submit a PR!

@NielsRogge @NiccoloCavagnero , I can fix that in EoMT PR and we can then directly import the `Mask2FormerLoss`

Thank you so much @yaswanth19.

Indeed, the same error was present in EoMT PR as my colleague Tommie noticed.

Best,
Nick

Well, technically, it was hardcoded to the correct value, so only when someone would want to use a different class weight is when it would become an issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This was fixed by #37610.
Add support for older versions in is_torchdynamo_compiling().
Author: Pqlet
Pull request: yes

Support is_torchdynamo_compiling() for older versions.

# What does this PR do?

Fixes error in `is_torchdynamo_compiling()`, which is thrown for older versions of torch, e.g. torch==2.2.2. The support for `torch._dynamo.external_utils.is_compiling()`  was deprecated and torch=2.2.2 does not manage torch.compile() on Llama 3.2 model.

@ArthurZucker 

Comments:
Tokenizer returns float32 tensor for empty string input instead of long dtype
Author: Goer17
Pull request: no
Labels: name: bug

Hi team,

I found an unexpected behavior when using HuggingFace Transformers' tokenizer with an empty string as input. When I run the following code:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B", trust_remote_code=True)
input_ids = tokenizer("", return_tensors="pt")['input_ids']
print(input_ids, input_ids.dtype)
# output:
# tensor([], size=(1, 0)) torch.float32
```

**Potential impact**:
For example, if this float32 tensor is concatenated with other torch.long tensors using torch.cat, the result will be promoted to float32, causing all token IDs to become floats. This can break downstream code that expects integer token IDs.

Could you please take a look? Thank you!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B", trust_remote_code=True)

input_ids = tokenizer("", return_tensors="pt")['input_ids']
print(input_ids, input_ids.dtype, sep="\n")

# output:
# tensor([], size=(1, 0))
# torch.float32
```

### Expected behavior

The tokenizer should always return torch.long tensors for input_ids, regardless of the input content.

Comments:
Hi! I'd like to work on this issue as my first contribution.

I've read through the codebase and identified the likely cause. Planning to submit a fix along with a simple test to ensure `input_ids` return as `torch.long` even for empty input.

Please let me know if it's okay for me to take this on 

Hi! I've submitted a PR to fix this issue: [#38421](https://github.com/huggingface/transformers/pull/38421).
It ensures that the `input_ids` return a `torch.long` tensor even for empty string inputs.
Looking forward to your feedback. Thanks! ❤️

Thank you for your prompt response and for providing a fix.
♥️♥️♥️

Hi @Goer17 and @Flink-ddd, thanks for the fix, but we actually have a PR for this already that fixes it for all tokenizers! See #36555

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Specifying multiple metrics in TrainingArguments.metric_for_best_model
Author: gumran
Pull request: no
Labels: name: Feature request

### Feature request

Currently, `TrainingArguments.metric_for_best_model` only accepts a `str`, such as "loss" or "accuracy". I suggest we allow specifying multiple metrics as an iterable so that the training loop tracks each of them and saves to separate directories. This means we also need to allow `output_dir` to be an iterable whose size must be the same. Besides, we would need to adjust the logic of `greater_is_better`, `load_best_model_at_end`, and several other arguments.

### Motivation

The rationale for this change is because it is very often that we are interested in saving checkpoints based on multiple metrics. As an example, when I was recently training an LLM via `trl.DPOConfig`, which inherits from `TrainingArguments`, I wanted to save checkpoints that achieve the highest validation accuracy and validation margin, which do not necessarily coincide. Instead, I had to pick one and lose the other or save a checkpoint at every evaluation, which is problematic in terms of disk space, among other issues.

### Your contribution

If you should approve feature, then I might submit a PR.

Comments:
This is my first ever response to an issue on a Github repo.

I am trying to learn more about the transformers library on Python.
Incorrect API call
Author: shashank-shivam
Pull request: no
Labels: name: bug

### System Info

torch.get_default_device() is not an official PyTorch API (even in 2.2), but it’s being called inside modeling_utils.py in version 4.52.1.

get_torch_context_manager_or_global_device() function is calling the following API
Error - Exception :  module 'torch' has no attribute 'get_default_device'

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

HuggingFaceEmbeddings(model_name=self._model_name)

Initialization fails with Exception :  module 'torch' has no attribute 'get_default_device'

### Expected behavior

Can be replaced with 
default_device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


Comments:
@shashank-shivam  Can you please add some relevant details

- which platform
- python version
- pytorch version


Hi @shashank-shivam, `torch.get_default_device` is part of the PyTorch API. Have you tried updating your version of PyTorch?

Standard PyPI repository doesn't have new releases of torch 
"ERROR: Could not find a version that satisfies the requirement torch==2.7.0 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2)"

For more context:
Platform macOS 15.4.1
Python 3.10
pytorch 2.2.2

Hi @shashank-shivam, is it possible you have an x86 mac? Newer versions of Torch are probably not getting released there and Apply is dropping support?

Yes that's correct. Torch doesn't seem to release newer version after 2.2.2 for mac x86intel. But I was wondering that the code can overcome the hardware dependency of libraries specially if it doesn't have a core functionality dependency. But, that's just my thought. Feel free to close the bug if that doesn't make sense

Hi @shashank-shivam yes, good point. I think the latest version on `main` doesn't use `get_default_device` anymore so `torch 2.2.2` should continue working for another while

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
quantizer_hqq should not require a gpu/cuda device to run
Author: learning-chip
Pull request: no

`quantizer_hqq.py` requires cuda device:

https://github.com/huggingface/transformers/blob/badc71b9f604ca910bb87a43979c795eaf6e7d64/src/transformers/quantizers/quantizer_hqq.py#L74-L75

However the original HQQ library also runs on CPU, by falling back to default aten operators: https://github.com/mobiusml/hqq?tab=readme-ov-file#usage-with-models

Comments:
cc @MekkCyber 

cc @mobicham, do you think we can remove the `torch.cuda` check ?

Yes I think so, it should work on cpu.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Question] The logic of data sampler in data parallel.
Author: kxzxvbk
Pull request: no

Hi, thanks for your attention.

When reading the source code of transformers, I cannot understand the implementation of `_get_train_sampler` in `trainer.py`. Why the default data sampler is `RandomSampler` rather than `DistributedSampler`? How does the trainer handle the sampler for data parallel?

reference code: https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L975

Comments:
And I notice that in very earlier versions. There are some if cases to check whether the world size > 1 and use a different sampler. Why this can be removed in the newer version? 
This change is introduced in this commit: https://github.com/huggingface/transformers/commit/ebd94b0f6f215f6bc0f70e61eba075eb9196f9ef

Hi @kxzxvbk, this isn't really a bug, but we have a community support space for asking questions like these and getting answers from other users or Hugging Face engineers! https://huggingface.co/spaces/transformers-community/support 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.