noamaanMulla-03/transformers-issues huggingface dataset:
Can't use Trainer on mps device
Author: smartliuhw
Pull request: no
Labels: name: bug

### System Info

version information:
``transformers -- 4.48.1``
``torch -- 2.2.2``
``accelerate -- 1.3.0``
``system -- mac os 15.2 (24C101)``

### Who can help?

@muellerzr @SunMarc 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm using c3 dataset to finetune Qwen2.5-3B-Instruct for classification task. The code runs successfully on a cuda device, while I try to run the same code on the mps device, got error messages below
<img width="1003" alt="Image" src="https://github.com/user-attachments/assets/e3432191-4be1-489f-bfaa-ebc75178a737" />
The following are the most important part of my code:
```python
## Train the model

exp_num=0

training_args = TrainingArguments(
    output_dir=f"./output/{lora_rank}_{exp_num}",
    learning_rate=5e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=2,
    num_train_epochs=5,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,
    bf16=False
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=dev_dataset,
    tokenizer=tokenizer,
    data_collator=datacollator,
    compute_metrics=compute_metrics
)
logger.info("Start training")
trainer.train()
```


### Expected behavior

The code runs sucessfully

Comments:
This is indeed not supported yet. Follow the this PR which should solve your issue. https://github.com/huggingface/accelerate/pull/3373

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Tokenizer does not split text according to newly added input tokens
Author: jiongjiongli
Pull request: no
Labels: name: Core: Tokenization

name: bug

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu121 (False)
- Tensorflow version (GPU?): 2.17.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.5 (cpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33

### Who can help?

@ArthurZucker and @itazap 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Repro Steps:

1. Initialize a tokenizer with use_fast=False
2. Add new tokens "red" and "e" to the tokenizer vocabulary using add_tokens()
3. Try to tokenize the word "read"

- Expected: The tokenizer should split "read" into ['r', 'e', 'ad'] since "e" is now a token
- Actual: The tokenizer keeps "read" as a single token, ignoring the newly added vocabulary

Repro Code:

```Python
from transformers import AutoTokenizer


def test(text, input_tokens):
    tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
    tokenizer.add_tokens(input_tokens)
    output_tokens = tokenizer.tokenize(text)
    print(f"Output tokens: {output_tokens}")


model_id = "google-bert/bert-base-cased"
test("read", ["red", "e"])
```

- Expected Output: Output tokens: ['r', 'e', 'ad']
- Actual Output: Output tokens: ['read']


### Expected behavior

The tokenizer should split "read" into ['r', 'e', 'ad'] since "e" is now a token.





Comments:
Answered on the PR! TLDR can't reproduce

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Only cast logits to float when computing loss
Author: ringohoffman
Pull request: yes



# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

A few updates that were missed in #31292 and #33902.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc: @Cyrilvallez @gante @ArthurZucker 

Comments:
Llama3 tokenizer decode is incorrect for ' ...' with leading space
Author: Naqu6
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-5.4.0-187-generic-x86_64-with-glibc2.31
- Python version: 3.12.9
- Huggingface_hub version: 0.29.1
- Safetensors version: 0.5.3
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA L40S


### Who can help?

Hi @ArthurZucker @itazap (tagging you per instructions), when I use the Llama3 tokenizers and encode the string ' ...', and then decode the resulting token, I get the string '...' back instead of the string ' ...' (leading space is missing).

I believe that decode should be the inverse of encode in this case, and it's unclear to me why it isn't.

Sorry if I'm misunderstanding something! Thanks for your time :) 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

tokenizer.decode(tokenizer.encode(" ...")[1:])
```

This outputs `'...' ` (no leading space).

### Expected behavior

I believe that `decode` should be the inverse of `encode`.

E.g

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

tokenizer.decode(tokenizer.encode(" Hello world")[1:]) #[1:] to remove beginning of sequence token
```

outputs " Hello world", as expected.


Comments:
See https://github.com/huggingface/transformers/issues/35938#issuecomment-2654022379
Very slow to load deep seekv3 int4 model and device_map="auto" "sequential" bug
Author: wenhuach21
Pull request: no
Labels: name: bug

### System Info

transforms 4.47.0

### Who can help?

_No response_


### Reproduction

please refer the code in model card https://huggingface.co/OPEA/DeepSeek-V3-int4-sym-gptq-inc

### Expected behavior

1 **Loading is very slow.** Loading the model (https://huggingface.co/OPEA/DeepSeek-V3-int4-sym-gptq-inc) on a DGX system with 2TB of memory and 7x80GB A100 GPUs is very slow, taking 30 minutes to 1 hour. 

**2 device_map bug.** Additionally, on a 7x80GB A100 GPUs, using device_map='auto' results in an OOM error, while switching to device_map='sequential' still causes an OOM error on card 0, even with max_memory configured.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @wenhuach21, the slow loading issue should be resolved on `main`. I'm not sure about the device_map bug, though!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Memory Access out of bounds in mra/cuda_kernel.cu::index_max_cuda_kernel()
Author: dingfen
Pull request: no
Labels: name: bug

### System Info

* OS: Linux ubuntu 22.04 LTS
* Device: A100-80GB
* docker: nvidia/pytorch:24.04-py3
* transformers: latest, 4.47.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Reproduction
1. pip install the latest transformers
2. prepare the UT test enviroments by `pip install -e .[testing]`
3. `pytest tests/models/mra/test_modeling_mra.py`

## Analysis
There might be some memory access out-of-bound behaviours in CUDA kernel `index_max_cuda_kernel()`
https://github.com/huggingface/transformers/blob/main/src/transformers/kernels/mra/cuda_kernel.cu#L6C1-L58C2

Note that `max_buffer` in this kernel is `extern __shared__ float` type, which means `max_buffer` would be stored in shared memory.
According to https://github.com/huggingface/transformers/blob/main/src/transformers/kernels/mra/cuda_launch.cu#L24-L35, CUDA would launch this kernel with 
* gird size: `batch_size`
* block size: 256
* shared memory size: `A_num_block * 32 * sizeof(float)`

In case that `A_num_block` < 4, the for statement below might accidentally locate the memory out of `A_num_block * 32`, since num_thread here is 256, and threadIdx.x is [0, 255].
```
for (int idx_start = 0; idx_start < 32 * num_block; idx_start = idx_start + num_thread) {
```

Therefore, when threadblocks of threads try to access `max_buffer`, it would be wiser and more careful to always add `if` statements before to avoid memory access out of bounds.

So We suggest to add `if` statements in two places:
![捕获](https://github.com/user-attachments/assets/172271ec-31d8-4e4d-8827-0cc66fbdd9be)




### Expected behavior

UT tests should all pass!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

These kernels were added in #24513, and haven't really been touched since then - @novice03 wrote the original PR. If you're still active @novice03, what's your take?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`HybridCache`] disable automatic compilation
Author: gante
Pull request: yes

# What does this PR do?

Recently I've expanded auto compilation in `generate` from `StaticCache` only to all compilable caches, through a new `is_compileable` attribute.

This PR sets `HybridCache.is_compileable` to `False`, which stops automatic compilation. There have been reports of speed loss on models that use this cache -- disabling until the root cause is found.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36620). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[deepspeed] any plans for deepspeed-domino?
Author: SoundProvider
Pull request: no
Labels: name: Feature request

### Feature request

## deepspeed-domino
[sample code](https://github.com/deepspeedai/DeepSpeedExamples/blob/master/training/DeepSpeed-Domino/pretrain_llama.py)

### Motivation

deepspeed released a plog post about deepspeed-domino, which is designed to significantly reduce communication burdens btw nodes[(link)](https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepspeed-domino/README.md)


is there any plans for this feature add-ons?


### Your contribution

I'm planning to integrate it on Trainer but I want to know if it's planned beforehand

Thank you a lot 

Comments:
cc @muellerzr
Fix auto-assign reviewers
Author: Rocketknight1
Pull request: yes

The `fnmatch` function didn't catch all of the cases I thought it did, so I had to write my own regex instead. I tested various cases that fail before and they work now!

A couple of lines in the codeowners file were also incorrect, so those have been rewritten.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36631). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ValueError: The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_ids,attention_mask,pixel_values,aspect_ratio_ids,aspect_ratio_mask,cross_attention_mask
Author: youky860423
Pull request: no
Labels: name: bug

### System Info

ValueError: The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_ids,attention_mask,pixel_values,aspect_ratio_ids,aspect_ratio_mask,cross_attention_mask

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Call trainer.train();
2. system return error for no logits

### Expected behavior

successfully running forward and compute loss

Comments:
Hey, you need to provide `labels` to get the loss calculated

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Check your dataset label's data type. It should be class not Value of string.
[docs] Serving LLMs
Author: stevhliu
Pull request: yes

Adds some docs about serving the Transformers model implementation from TGI and vLLM.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36522). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix AriaForConditionalGeneration flex attn test
Author: ivarflakstad
Pull request: yes

We had some tests failing related to flex_attention. I don't think there is a real bug present related to flex_attention, but rather that we simply haven't marked classes correctly as not supporting flex_attention.

Here are the observations/assumptions I've made:
Aria supports flex_attention.
Idefics3VisionTransformer does not.
AriaForConditionalGeneration depends on Idefics3VisionTransformer.
AriaForConditionalGeneration does not support flex_attention.

If that seems correct, as well as my solution of just slapping a `_supports_flex_attn = False` on AriaForConditionalGeneration, then we're good to go 👍 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[preview] configs as dataclasses
Author: gante
Pull request: yes

# What does this PR do?

DO NOT MERGE

This PR is for future reference only: Config objects work if we add `@dataclass` to them

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36396). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tests: fetch generation tests
Author: gante
Pull request: yes

WIP -- now that generation tests are being run, we can notice a few broken ones 👀 

Generation tests that were not part of the model mixin tests were not being fetched. This PR fixes it, by allowing each job to apply multiple separate filters to then run the **union** of all filtered tests. 

This PR:
- Enables non-mixin generation tests
- Fixes failing generation tests

_____________________________________________

👉 New files being fetched:
`test_preparation/tests_generate_test_list.txt`
![Screenshot 2024-11-19 at 14 35 41](https://github.com/user-attachments/assets/89f23542-e3df-48aa-a873-89b6b88155f4)


👉 dummy modification in a generation test triggering a test run: 
(early commit before tests were fixed)
![Screenshot 2024-11-19 at 14 38 12](https://github.com/user-attachments/assets/eec91f10-9063-4547-b806-975c259122f2)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34807). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
experiment: use `seen_tokens` instead of `cache_positions`
Author: gante
Pull request: yes

**FOR REFERENCE, NOT FOR MERGING**

Tests the old strategy: keep track of the number of seen tokens (= cache write index) as an internal integer in `StaticCache`, as opposed to using the external `cache_positions` Tensor 

⚠️ the executorch+static cache test fails with this change, I think because it doesn't reset the cache

Test commands: confirm that eager == static compiled, and that the cache is correctly reset between runs:
```py
import torch
from transformers import LlamaForCausalLM, AutoTokenizer, GenerationConfig

model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", device_map="auto", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")

prompts = ["Simply put, the theory of relativity states that "]
model_inputs = tokenizer(prompts, return_tensors="pt").to(model.device)

print("\neager\n")
generated_tokens = model.generate(**model_inputs, do_sample=False, max_new_tokens=20)
generated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
print("\n", generated_text, "\n")

print("\nstatic\n")
generated_tokens = model.generate(**model_inputs, do_sample=False, max_new_tokens=20, cache_implementation="static")
generated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
print("\n", generated_text, "\n")

print("\nstatic compiled\n")
model.forward = torch.compile(model.forward, fullgraph=True, mode="reduce-overhead")
generated_tokens = model.generate(**model_inputs, do_sample=False, max_new_tokens=20, cache_implementation="static")
generated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
print("\n", generated_text, "\n")

print("\nstatic compiled (rerun to confirm that the cache is reset)\n")
model.forward = torch.compile(model.forward, fullgraph=True, mode="reduce-overhead")
generated_tokens = model.generate(**model_inputs, do_sample=False, max_new_tokens=20, cache_implementation="static")
generated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
print("\n", generated_text, "\n")
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35299). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Hybrid models
Author: johnnynunez
Pull request: no
Labels: name: Feature request

### Feature request

HymbaForCausalLM

### Motivation

Hybrid models not supported:
Support:
[Hymba](https://developer.nvidia.com/blog/hymba-hybrid-head-architecture-boosts-small-language-model-performance/)
A hybrid attention mechanism combining local sliding window attention and global attention.
Grouped-query attention (GQA).
A mix of global and local rotary embeddings.


### Your contribution

No

Comments:
Extract embeddings of many seqs using ESM2
Author: DM0815
Pull request: no

I used esm2 conducting 
"from transformers import AutoTokenizer, EsmModel.
tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t36_3B_UR50D")
model = EsmModel.from_pretrained("facebook/esm2_t36_3B_UR50D")

inputs = tokenizer(AA_seq.tolist()[:10000], return_tensors="pt", padding=True, truncation=True, max_length=340)
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
last_hidden_states.shape "

when I extract embedding of one seq , it's ok.  when I input many seqs (such as 130k), It will kill my kernel. But when I pred seq one by one, I find that shape of each embedding are different.

What should I do? I hope all seqs have a same shape using ESM2.

Thanks.

Comments:
Hi @DM0815, these questions are usually better suited to the forums or the Discord because they're not bugs in `transformers`! I like ESM though, so you get a free answer: The reason for the difference in shape is probably the output embedding has one vector per amino acid, so if your input sequences have different lengths then your output sequences will as well. The reason they become the same length when you run them as a batch is because each batch is always padded to the length of the longest sequence in the batch with padding tokens, but the embeddings for the padding tokens may not be meaningful.

If you want to pad all batches to the same length, you can use `padding="max_length"` in the tokenizer, which will pad all sequences to the max length of 340 that you set, even if the longest sequence in the batch is smaller than that.