noamaanMulla-03/transformers-issues huggingface dataset:
fix: loss computation after embeddings resize - mllama
Author: Ssukriti
Pull request: yes

# What does this PR do?

background: https://github.com/huggingface/transformers/pull/36591

Fixes loss computation when vocab is resized by resize_embeddings. Only vocab size of parent conditionalgenerationclass is modified, hence loss has to be calculated there. Chosen approach until bigger refactor. Solution discussed with @zucchini-nlp in above PR 

Fixes # (issue)
https://github.com/huggingface/transformers/issues/36590

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@zucchini-nlp as discussed solution

Comments:
@zucchini-nlp all tests have passed and comments addressed . Thank you for the review.

Thank you @zucchini-nlp . Can the PR be merged soon as well ? It is actually blocking a use case we have 

Yep, merging, sorry
Fix: bamba error handling kwargs with forward pass 
Author: Ssukriti
Pull request: yes

This PR adds a small change to handle additional kwargs passed to the forward function of `BambaModel` architecture. 

Fixes a behavior when tuning Bamba models - when HF Trainer would pass additional LossArgs to `BambaForCausalLM.forward()` , which are  passed to `BambaModel.forward` , and need to be ignored. 
This would fix the error - `num_items_in_batch` is an unexpected arg , while tuning the model.

Additional Context:
 [LlamaForCausalLM](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L784) class passes all kwargs to self.model.forward , hence they need to be handled. 
In the future, we would add support for flashAttentionKwargs in BambaModel.forward() . 

cc: @fabianlim 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
@ArthurZucker @molbap this is just a temporary minor fix to make the model work with `trainer.Trainer`. We have not properly handled the `FlashAttentionKwargs` yet because we plan to get to that when we support padding free. But that work is involved, and for now we want to make sure the model is able to tune properly

cc: @ani300 @raghukiran1224
Remove old code for  PyTorch,  Accelerator and tokenizers
Author: cyyever
Pull request: yes

# What does this PR do?
Remove outdated conditions and comments for PyTorch and Accelerator.
 
Specifically, some code for PyTorch < 2.1 has been found and removed. As a result, the functions ``is_torch_bf16_cpu_available``,`is_torch_fx_available`, `is_torchdynamo_available` and `is_torch_compile_available` are now equivalent to `is_torch_available`. Some tests are further simplified using this fact.

There is one change regarding to old Accelerator code.
And old tokenizers version check is removed.
 
     
 
    
 




   
 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Added a warning message about unsupported PT versions.

Hi @cyyever, sorry about this, my fault! We actually have a separate PR to bump the minimum torch version to 2.1 here: https://github.com/huggingface/transformers/pull/37277

As a result, you can skip the check - sorry for suggesting it!

@Rocketknight1 I will rebase after it's merged to see whether there are more to delete..

@Rocketknight1 Lots of new commits since last review.

@ydshieh Follow-up of your clean-up.

Thanks! Will check. Could you refine the PR title and description 🙏 ? Seems this PR focus on `is_torch_fx_available`? A more detailed information regarding what a PR is doing is much appreciated.

@ydshieh Ignore the branch name...

Almost ready 🚀 to go 👍 

@ydshieh done

Merge! Thank you for the contribution and iteration 🤗
enhance require_deterministic_for_xpu
Author: yao-matrix
Pull request: yes

While revisiting this PR https://github.com/huggingface/transformers/pull/30774,  we are for "still try to set it to deterministic for XPU (in the decorator body) and use it (despite you might get some failures at some points as I mentioned). You might get lucky" option from @ydshieh. XPU is seeking the same level of usability and coverage as CUDA. So, we need enable these numerical cases w/ deterministic enabled, if some still fail, we Intel guys need investigate.

I enhance the require_deterministic_for_xpu decorator, so if it's XPU, while enter the body, we will save the deterministic state and set it to True; while leaving, we will restore the prior deterministic state. So, we don't impact the global setting, just make it case specific.

@ydshieh , pls help review, thx.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@faaany , FYI

> Nice.
> 
> Just an information: Some of these `require_xxx` now get a bit confused. Most of them work with both test class and method, but some only for test method, like `require_read_token` and now `require_deterministic_for_xpu` (I think).
> 
> See [this comment](https://github.com/huggingface/transformers/pull/37412#issuecomment-2792635298) if you want. No need to change anything extra is this PR, just at some point I might rework these decorators :-)

thx, will keep following latest status.
TypeError: object of type 'IterableDataset' has no len()
Author: mxjmtxrm
Pull request: no
Labels: name: bug

### System Info

When I tried a LLM using run_clm.py in example, I met the following error:
```
File "/workspace/train.py", line 611, in main
    data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
TypeError: object of type 'IterableDataset' has no len()
```
`len(train_dataset)` is not compatible with `IterableDataset`.

### Who can help?

@ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using the command in the [README](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/README.md).

### Expected behavior

fix this error

Comments:
@mxjmtxrm This happens because IterableDataset objects in PyTorch (and by extension, Hugging Face's datasets library) are designed to be iterated over rather than having a fixed length like a standard Dataset.  That's why len() is not applicable in IterableDataset.

hi @mxjmtxrm, what dataset are you using for this?

I use fineweb-edu and set streaming is True while loading the dataset. As the dataset is super large, I have to use IterableDataset. I think it should be fixed in this case.

Hi @mxjmtxrm, we can't really "fix" this because the code is designed to run for `max_train_samples`, and if that isn't set then it uses `len(dataset)`. When neither are set, the code really has no idea how long it should train for!

The only solution I can think of is to raise an error in `run_clm.py` if both are unset, telling the user that they should manually set `max_train_samples`. We'd accept a PR if anyone wants to add that!

`max_train_samples` cannot be used with  `IterableDataset`. I set `max_steps` as the max training step. I think there is a way to calculate `max_train_samples` with `max_steps`.

Hi @mxjmtxrm, what happens if you pass `--max_train_samples` as an argument to `run_clm.py` with an `IterableDataset`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fixes: Corrects file path for CUDA kernels
Author: DonggeunYu
Pull request: yes

# What does this PR do?
Corrects the file path used to locate the CUDA kernels for the Deformable Attention module. This ensures that the kernels are loaded correctly, resolving potential errors during module initialization and usage.

## Who can review?

- vision models: @amyeroberts, @qubvel

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Error: TypeError: argument 'ids': 'float' object cannot be interpreted as an integer
Author: joyce0105-ops
Pull request: no

My transformers version is V4.49.0
When running unsloth-GRPO, the error occurs 
```
File “/usr/local/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py”, line 668, in _decode
 text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument 'ids': 'float' object cannot be interpreted as an integer
```

Comments:
Hi @joyce0105-ops , I am facing the same issue with transformers version  V4.50.0, how did you resolve the issue?
Update-kernel-pin
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Make sur backward passes are fine

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37448). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add moe kernels
Author: ArthurZucker
Pull request: yes
Labels: name: Kernel Fusion

name: kernels

# What does this PR do?
Add MoE kernels

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37376). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks all!
Fix the test fetcher
Author: LysandreJik
Pull request: yes

The list of tests to fetch became an infinitely growing list with the new init approach; this instead converts it to a dict, and completes the list of importable objects of each key in the dict.

If the key wasn't here previously, it adds it with the list of objects; otherwise, it checks whether the list is incomplete, and completes it.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37452). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: Add gradient testing for Flash Attention 2
Author: crStiv
Pull request: yes

This commit adds gradient testing for Flash Attention 2 (FA2) implementation.
The test ensures that gradients computed with FA2 match those computed with
the eager implementation within specified tolerance thresholds.

Key changes:
- Added gradient testing in train mode
- Compare gradients between eager and SDPA implementations
- Use same tolerance thresholds as forward pass comparison
- Properly cleanup gradients and restore eval mode

This addresses the TODO comment in test_modeling_common.py and improves
test coverage for FA2 implementation.

Comments:
Can you rebase / make sure cis are green now? 🤗 

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Add print_stack() in isin_mps_friendly() of pytorch_utils.py(#37423)
Author: f2janyway
Pull request: yes

@manueldeprada 



Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
[processor] clean up mulitmodal tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, remove overriden parts that handle special multimodal tokens. Also fixes Aria and Idefics image processors by adding `do_rescale` in configurable init params, thus removing overridden tests

cc @ydshieh for tests and @qubvel for main processor code

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

I think the conflict comes from #37282 that is merged. Need some work to resolve them. See one of my comment in the review.

Other than that, the changes either remove tests (maybe we no longer overwrite them, or just don't need to test them), or add `prepare_processor_dict` to make the processor matching what we need (like `chat_template`).

So LGTM on my side, just need to resolve the conflicts carefully.

Thank you!



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37362). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Keep getting an OOM when doing a evaluation
Author: davebulaval
Pull request: no

## Environment info

  - `transformers` version: 4.8.2
  - Platform: Linux-5.8.0-55-generic-x86_64-with-debian-bullseye-sid
  - Python version: 3.7.9
  - PyTorch version (GPU?): 1.9.0+cu111 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Flax version (CPU?/GPU?/TPU?): not installed (NA)
  - Jax version: not installed
  - JaxLib version: not installed
  - Using GPU in script?: Yes cuda:0 only one GPU
  - Using distributed or parallel set-up in script?: No

### Who can help
No idea.

Models:
- albert, bert

Library:
- trainer: @sgugger

## Information
The problem arises when using evaluating a model, I train using a batch size of 64 and 80% of my data but when I do an evaluation (600 000 data points) it ALWAYS give a OOM at 94% even with a batch size of one and the memory allocation IS ALWAYS 3.75 GB no matter the size of the batch. I've deleted the .cache many times and still got the problem.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: which is a sequence classification of postal address (max 30 tokens).

## To reproduce
My script is base on the [token classification with trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py).


Comments:
Found [this](https://discuss.huggingface.co/t/cuda-out-of-memory-during-evaluation-but-training-is-fine/1783/2) explication that fixed my problem.

i.e. Huggingface keeps all the prediction in the GPU.

I ran into a very similar issue while working with `transformers==4.50.3` on a ~500k example evaluation run using `bert-base-cased`. The crash occurred consistently at around 94–95%, with GPU memory pegging at ~3.75 GB regardless of batch size or cache clearing.

I ended up writing up the full environment, reproduction steps, and mitigation that worked for me in a separate issue [here](https://github.com/knotreeltest/no-repo/issues/1)

Hope this helps anyone running into the same failure!
Trainer skipping to end of epoch
Author: harryjulian
Pull request: no
Labels: name: bug

### System Info

Hi there,

I’m using transformers==4.49. I’m using the trainer with a webdataset, where I’m setting an approximate length to the webdataset, for compatibility with the trainer.

Regardless of what I do, and even if I use max steps rather than 

![Image](https://github.com/user-attachments/assets/49b07331-99f0-4c53-bd39-35fae18f2ccb)

num_train_epochs, my wandb dashboard is consistently showing the epoch value shooting up to the ‘end of the epoch’ (obviously an ‘epoch’ doesn’t really exist in wds) very prematurely.

Is this something to be worried about? I’m concerned a load of my data is being skipped for no reason and the dataloader is being reinitalized, starting from the beginning.

I may be able to share some example code if the answer isn’t obvious.

### Who can help?

@zach-huggingface @SunMarc 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Initialise a training run with a webdataset.

### Expected behavior

Epochs are skipped.

Comments:
I think this is something to do with how shards usually have a single dataloader worker assigned to them, and that the trainer defaults to a single dataloader worker. Would this be the case when using `datasets` to load the webdataset shards?

Hey @harryjulian, if you can provide a reproducible script, that would help figuring out where the issue is. 

Hey @SunMarc -- I've fixed it on my end. The reason was the above, as there was only one dataloader worker and therefore only 1/N shards were being sampled -- leading to the trainer thinking the epoch was finished when it had made it's way through a single shard. Think it's just a sharp edge of using `webdataset` with the Trainer. 

Nice, glad that you fixed it !
Disable kernels for quantization
Author: MekkCyber
Pull request: yes

# What does this PR do?
Disabling kernels for quantization tests using the flag from this pr https://github.com/huggingface/kernels/pull/70, since the linears are different so we need different kernels in that case


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37446). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLama4 RuntimeError
Author: yinzhangyue
Pull request: no
Labels: name: bug

### System Info

    attn_weights = attn_weights + causal_mask
                   ~~~~~~~~~~~~~^~~~~~~~~~~~~
RuntimeError: The size of tensor a (8192) must match the size of tensor b (177) at non-singleton dimension 3


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

def generate_with_llama4(model, tokenizer, prompt, system_prompt=None, temperature=0.6):
    """Generate a response using Llama-4 model"""
    # Create the message format
    if system_prompt:
        messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": prompt}]
    else:
        messages = [{"role": "user", "content": prompt}]

    # Apply chat template without padding/truncation
    model_inputs = tokenizer.apply_chat_template(
        messages, 
        return_tensors="pt", 
        add_generation_prompt=True, 
        return_dict=True
    )
    
    prompt_token_count = len(model_inputs["input_ids"][0])

    # Move inputs to the same device as the model
    model_inputs = {k: v.to(model.device) for k, v in model_inputs.items() if isinstance(v, torch.Tensor)}

    # Generate
    generated_ids = model.generate(
        **model_inputs, 
        max_new_tokens=100, 
        temperature=temperature, 
        do_sample=True
    )
    
    # Extract just the newly generated tokens
    new_tokens = generated_ids[:, prompt_token_count:]
    completion_token_count = new_tokens.shape[1]

    # Decode to text
    response = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)[0]

    return response, prompt_token_count, completion_token_count


### Expected behavior

Generate tokens

Comments:
Hi @yinzhangyue, the reproducer doesn't include everything we need to run that code. Can you give us a complete code block or jupyter notebook, including all imports, model names, etc. that we can run to see this error?

@Rocketknight1 I got the exact same issue with every example scripts on this page: https://huggingface.co/blog/llama4-release

No issue with 4.51.0, seems to have happened with 4.51.1 and the current main branch

Same issues

> 4.51.0

Resolved it by !pip install transformers==4.51.0

Getting the same issue as well with `transformers==4.51.1`.  Reverting to `4.51.0` solved it for me too.

Hi @kingabzpro @pikruse @yinzhangyue we did another bugfix release today - can you confirm if the issue is fixed on `4.51.2`?

Fixed on `4.51.2` on my end! Thanks.

Thanks!