noamaanMulla-03/transformers-issues huggingface dataset:
Enabling Access to Currently Training Model in Callback Handler
Author: jp1924
Pull request: no
Labels: name: Feature request

### Feature request

Update `self.callback_handler.model` to reference the actual training model instance (`model`) instead of the pre-prepared model (`self.model`) in `Trainer._inner_training_loop`.

### Motivation

- When debugging model training, developers often need to monitor model weights and other parameters at each step, similar to `wandb.watch`
- Currently, callbacks receive the pre-prepared model instance (`self.model`) rather than the actual training model (`model`)
- This is evident from different memory IDs:
  ```python
  id(self.callback_handler.model): 140049287962320
  id(model): 140048547892912  # Model memory id in training_step
  ```
- The code flow suggests this might be unintended:
  ```python
  model = self._wrap_model(self.model_wrapped)
  self.callback_handler.model = self.model  # Current implementation
  ```
[trainer_code](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2303C9-L2410)

### Your contribution

The proposed change is to update the model reference in the callback handler to point to the actual training model:
```python
# Current implementation
self.callback_handler.model = self.model

# Proposed change
self.callback_handler.model = model
```

This would allow callbacks to access and monitor the actual model being trained, enabling proper debugging and monitoring capabilities like examining weights during training steps.

This change would ensure that when users implement custom callbacks for debugging or monitoring purposes, they have access to the correct model instance that's being actively trained, rather than the pre-prepared version.

Comments:
Trainer: Use second last checkpoint if last checkpoint loading fails
Author: SilverSoldier
Pull request: no
Labels: name: Feature request

### Feature request

Currently, the checkpoint is saved in the [`_save_checkpoint()`](https://github.com/huggingface/transformers/blob/b2f2977533445c4f62bf58e10b1360e6856e78ce/src/transformers/trainer.py#L3197) method which saves the model, optimizer (optionally) and finally the Trainer state.

The [`resume_from_checkpoint()`](https://github.com/huggingface/transformers/blob/b2f2977533445c4f62bf58e10b1360e6856e78ce/src/transformers/trainer.py#L2070) function gets the checkpoint directory from the `get_last_checkpoint` function. Then, the model etc. are loaded using the `self._load_from_checkpoint()` function and trainer state is loaded using the `TrainerState.load_from_json` call. 

If training program ended abruptly in the middle of checkpointing, the directory is created but some of the files are missing. For ex. if the trainer state was not yet written, this throws an error during the `TrainerState.load_from_json` call and training is not able to resume at all.

Proposal: If either loading model or trainer state fails (in `resume_from_checkpoint()`) since the last directory is incomplete, then use the second last checkpoint folder for resuming.


### Motivation

Motivation: Currently, our job can be killed in the middle of checkpointing and is not able to resume since last checkpoint is incomplete. We need to manually delete the folder to resume from previous checkpoint. 

Some info (not sure if relevant): I use accelerate launch to launch the training. 

### Your contribution

I am willing to submit a PR for this if this feature seems acceptable.

~~We would need to wrap the 2 loads in a try/catch block and if either of them fail, re-run using the second last directory.~~
Looks like there are a lot of places where `load_from_checkpoint` is done, with exceptions for FSDP etc. which would complicate things. Maybe we can have a list of expected files and check for existence else load second last dir.

Comments:
Zamba new attention standard
Author: pglorio
Pull request: yes

Incorporates new attention standard #35235 in the Zamba architecture

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @Cyrilvallez 

Comments:
ci: mark model_parallel tests as cuda specific
Author: dvrogozh
Pull request: yes

`parallelize()` API is deprecated in favor of accelerate's `device_map="auto"` and therefore is not accepting new features. At the same time `parallelize()` implementation is currently CUDA-specific. This commit marks respective ci tests with `@require_torch_gpu`.

Fixes: #35252
CC: @ArthurZucker @SunMarc


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35269). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How to run the model on another machine and send the answer to another machine.
Author: ixn3rd3mxn
Pull request: no
Labels: name: bug

### System Info

transformers 4.31.0 , window os , python 3.10.12

### Who can help?

vision models: @amyeroberts, @qubvel

I have tried using this model on my machine myself, and it works normally, but the processing is very slow because the GPU on my machine is not that powerful. However, I have a server with a strong GPU. If I install this model on the server and run the code on my machine, when it reaches the video processing stage, it sends the task to the server, and the server sends back the result. Then my machine will print the answer and display the result. Is this possible? If so, how can I do it?


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

I expect it to work in a hybrid way between my computer and the server to achieve faster results.

Comments:
You want a model/inference server. Perhaps start with [gradio](https://www.gradio.app/) as it's relatively simple. NVIDIA Triton is another example if you're looking for something "industrial strength)

Hi @ixn3rd3mxn, we are trying to keep issues on github for bug reports, feature requests, and issues related to the library code. For your question, it's better to use the forum https://discuss.huggingface.co/. But in general, I agree with @david-waterworth, it's possible and gradio or any framework like Flask or FastAPI might be useful for your purpose.
Add support for Allegro
Author: VladOS95-cyber
Pull request: no
Labels: name: New model

### Model description

Allegro is a powerful text-to-video model that generates high-quality videos up to 6 seconds at 15 FPS and 720p resolution from simple text input.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Model code: https://github.com/rhymes-ai/Allegro

Comments:
I could try to contribute to model implementation if nobody is working on it

Hi @VladOS95-cyber I see that the diffusers team are already working on adding this model https://github.com/huggingface/diffusers/pull/9736 in the diffusers library.


Closing as it was added to diffusers!
Check whether rescale is requested before checking is_scaled_image
Author: squidpickles
Pull request: yes

# What does this PR do?

This change checks whether `do_rescale` is enabled in image processors before checking `is_scaled_image()`.

`is_scaled_image()` needs to scan every element to get the min and max values, to determine whether they are between 0 and 1. If `do_rescale` is `False`, this is an unnecessary check.

This change swaps the order of the check, so it first checks `do_rescale` and then `is_scaled_image()`. Just a minor performance improvement on some image processing.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts, @qubvel

Comments:
added interpolation for owlvit & owlv2.
Author: kshitij0-0
Pull request: yes

# What does this PR do?

Fixes # (issue)

Towards: Community contribution: enable dynamic resolution input for more vision models. #30579
Added interpolation for dynamic resolution inputs.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts do review this, the only task that needed a check in the box :)



Comments:
cc @qubvel @molbap @yonigozlan 

Hi @molbap , currently i've pushed changed only with respect to OWLViT , if it looks good , will replicate it on the v2 as well.


Hey @kshitij0-0 the issue has been resolved in 
 - https://github.com/huggingface/transformers/pull/34764
 
 Thanks for taking the time to contribute and sorry that your PR was not merged
Deepseek v2
Author: VladOS95-cyber
Pull request: no
Labels: name: New model

### Model description

There is stale PR https://github.com/huggingface/transformers/pull/31976. Is anybody working on this model? 

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @ArthurZucker @SunMarc @LysandreJik @qubvel. If nobody in HF team or contributors is working on this model, I could try to take care of it. I did not find implementation in the repo or any other PR's.

cc @itazap 

You can take it over! The cheer size of the model makes it hard to iterate fast on it but a great addition for sure! 🤗 

sorry for the late reply!


Hi @ArthurZucker! Thank you for your answer! So, now I am working on D-FINE model for Transformers, once I finish that I can take DeepSeekV2 if nobody is willing to work on it.
Fix Qwen2RotaryEmbedding Device Matching Bug
Author: developer0hye
Pull request: yes

# What does this PR do?

Fixes #35505


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel


Comments:
@qubvel 

Sorry, I missed #35417 and the comment of @zucchini-nlp  before creating the PR. I've followed your suggestion for now, but #35417 should be merged first if these works are really needed.
[Qwen2Audio] handle input ids expansion during processing
Author: eustlb
Pull request: yes

# What does this PR do?

This PR adds input_id expansion to the processor. 

Before: 
1. embedding of the input_ids`"


Generate the caption in English:"` →  `inputs_embeds` tensor
2. creation of a new embedding tensor [here](https://github.com/huggingface/transformers/blob/a821b9c7ab6b0ce4a4557be770cba5946df5c322/src/transformers/models/qwen2_audio/modeling_qwen2_audio.py#L1051C8-L1053C10), taking into account the number of audio tokens to be merged with inputs_embeds
3. replacement of audio embedding vectors 

Now:
1. expansion of input ids at the processing stage: `"


Generate the caption in English:"` →  `"

...

Generate the caption in English:"` with as many` 
` tokens as the number of audio embedded vectors after encoding and projection
2. creation of `inputs_embeds`, directly with the correct shape
3. replacement of the audio embedding vectors

This approach allows to remove the unnecessary step of creating a new embedding tensor in the model forward. Moreover, it simplifies the code as correct padding is directly handled at the processing stage. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35534). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Release GPU memory after Optuna trial
Author: JanetVictorious
Pull request: yes

# What does this PR do?

This PR aims at releasing GPU memory after an Optuna trial when running `hyperparameter_search`.

Fixes [#13019](https://github.com/huggingface/transformers/issues/13019)

Solution has been tested locally.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @SunMarc 

Comments:
@ArthurZucker do you have any comments on this change?
Add AIMv2 to Transformers
Author: AlanPonnachan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #35351

This PR adds [AIMv2](https://arxiv.org/abs/2411.14402) support in Transformers. AIMv2 showed better performance than SigLIP.


## TODO
- [x] Completed almost every steps mentioned in this [guide](https://huggingface.co/docs/transformers/main/en/add_new_model)



## Who can review?

@qubvel
@Rocketknight1

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
[Whisper] fix docstrings typo
Author: eustlb
Pull request: yes

# What does this PR do?

Fix mini typo in rendering links to GenerationMixin's generate :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35338). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Correctly list the chat template file in the Tokenizer saved files list
Author: Rocketknight1
Pull request: yes

The tokenizer has a list of the files it saves, which wasn't correctly being updated with the new standalone chat template file. This PR fixes that issue!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34974). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Test added!
Remove sortish_sampler from Seq2SeqTrainingArgument and the docs
Author: eyalmazuz
Pull request: no
Labels: name: bug

### System Info

Not Needed

### Who can help?

@muellerzr 
@SunMarc 
@stevhliu 



### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In  [Seq2SeqTrainingArguments](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments) There's a parameter called sortish_sampler

* sortish_sampler (bool, optional, defaults to False) — Whether to use a sortish sampler or not. Only possible if the underlying datasets are Seq2SeqDataset for now but will become generally available in the near future.
It sorts the inputs according to lengths in order to minimize the padding size, with a bit of randomness for the training set.

This argument exists in the Seq2SeqTrainer 

https://github.com/huggingface/transformers/blob/9d0ad5074eb4214efa8063de21ac5253470f4e08/src/transformers/training_args_seq2seq.py#L39 

But it does nothing aside from taking space
In the Trainer class it points to nothing, never referenced, never used, and in the trainer_utils there isn't even a class called SortishSampler

Personally I didn't knew it, I used this argument for a long while now thinking it does something in my training scripts
but only upon entering the source code and seeing it doesn't exists, I started a deep dive into old issues and pull requests.

Then I found #9574 which is a (almost) 4 years old PR
in which it was renamed and technically got deprecated.

I suggest to remove the argument completely from the documentation and trainer code to remove confusion

As personally I though that ``group_by_length`` is meant mostly for auto-regressive models which only takes a single sequence as their input, while ``sortish_sampler`` is used for seq2seq models and takes into account both sequences

So I decided to use the latter rather than the former for the past 3 weeks thinking I'll get some memory savings in my scripts.

In the end, I think this argument should officially get deprecated and removed from both the code and the docs and it may cause confusion for others beside me.

### Expected behavior

Less confusion, better model training :)

Comments:
cc @muellerzr @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@muellerzr 
@SunMarc 



Thanks for the report @eyalmazuz ! Feel free to deprecate completely the arg as it is already doing nothing at the moment as you said. We do have some legacy code related to Seq2seq Trainer (examples/lecacy/seq2seq) where this arg is still used but since it is legacy, we won't touch the code there.
Deepspeed integration: support batch sizes that are less than the number of gpus/ranks
Author: IMbackK
Pull request: no
Labels: name: Feature request

name: DeepSpeed

### Feature request

Currently when training using trainer and the deepspeed zero integration the minimum total effective batch size is one per gpu/rank, thus on a 8 gpu machine the minimum effective batch size is 8.

As far as i know there is no fundamental requirement with zero style shading for there to be one batch per GPU.

### Motivation

Having one batch per gpu may be undesirable, for instance when there is insufficient memory.

Comments:
cc @muellerzr @pacman100 

Hello, this isn't possible. DeepSpeed is a Data Parallel paradigm, meaning each process/rank should get at least one sample. 

> As far as i know there is no fundamental requirement with zero style shading for there to be one batch per GPU.

Data Parallel by definition requires at least one sample per process/GPU. Could you state where the above claim is 
mentioned?