noamaanMulla-03/transformers-issues huggingface dataset:
Allow for nightly packages of `compressed_tensors`
Author: kylesayrs
Pull request: yes

# What does this PR do?
This PR follows up on #31704 by allowing the `compressed-tensors-nightly` package to be used to load compressed tensors modules. This is done by relaxing installation requirements to only require the `compressed-tensors` module spec, but not the `compressed-tensors` package.

Compressed Tensors models cannot be loaded with nightly package
```python3
from transformers import AutoModel
model = AutoModel.from_pretrained("nm-testing/TinyLlama-1.1B-Chat-v1.0-W8A8_tensor_weight_static_per_tensor_act-e2e")
# ImportError: Using `compressed_tensors` quantized models requires the compressed-tensors library: `pip install compressed-tensors`
```

Compressed Tensors is reported as unavailable, despite the module being available through the nightly package
```python3
from transformers.utils.import_utils import is_compressed_tensors_available
is_compressed_tensors_available()
# False
```

After the changes, models can be imported with the nightly package and compressed tensors is reported as being available

## Who can review?
@SunMarc @ArthurZucker
@dsikka @dhuangnm @andy-neuma
Anyone in the community is free to review the PR once the tests have passed.


Comments:
Fix kwargs passed by AutoQuantizationConfig.from_pretrained
Author: kylesayrs
Pull request: yes

# What does this PR do?
Fixes wrongly passed kwargs in `AutoQuantizationConfig.from_pretrained`
```python3
from transformers.quantizers import AutoQuantizationConfig
q_config = AutoQuantizationConfig.from_pretrained("nm-testing/tinyllama-w4a16-compressed-hf-quantizer")
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ksayers/kylesayrs_transformers/src/transformers/quantizers/auto.py", line 111, in from_pretrained
    quantization_config.update(kwargs)
TypeError: QuantizationConfigMixin.update() takes 1 positional argument but 2 were given
```

## Who can review?
@SunMarc @ArthurZucker
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33798). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CI: run `make fix-copies`
Author: gante
Pull request: yes

# What does this PR do?

Makes our CI happy

(our repo consistency job is failing until this is merged)

Comments:
https://github.com/huggingface/transformers/pull/33878 -- this merged PR already has the fix 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33895). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Additional options for include_num_input_tokens_seen in Trainer
Author: iarbel84
Pull request: no
Labels: name: trainer

name: Feature request

### Feature request

Track only the training, avoiding the count of padding tokens

### Motivation

It appears that this metric also includes padding tokens. If one would use example packing, then it really tracks the “correct” number of tokens seen by the model.

However, I can think of two cases where this will not be accurate:
1. In cases where packing is not used, training examples are padded to the longest sequence in the batch
2. For SFT training on completions only

For the first case, a more accurate calculation would be to sum the attention mask.
For the second case, I'm not sure how this should be regarded. However, we can consider counting only label tokens != `-100`

### Your contribution

Replace lines 2248-2258 in trainer.py (v4.43.4) with the following:
```python
self.state.num_input_tokens_seen += (
    torch.sum(
        self.accelerator.gather(
            torch.tensor(
                inputs['attention_mask'].sum(), device=self.args.device, dtype=torch.int64
            )
        )
    )
    .cpu()
    .item()
)
```

Comments:
Hi @iarbel84, thanks for opening this feature request! 

Makes sense to me - would you like to open a PR with the suggested change? 

cc @SunMarc @muellerzr 

Yes, I'll be happy to. A few points for discussion:
1. When it come to training on completions - what should be the behavior? Are we interested in counting only the tokens that count towards the loss, or all the tokens?
2. Currently, the count uses the model's "main_input_name" property, and defaults to "input_ids". Is it acceptable to simply use "attention_mask" in the new feature?
3. Are we looking to change the behavior of the current feature, or add additional new options?

Hi @iarbel84, apologies for the delay in response. 

Let's get the input from @SunMarc or @muellerzr here. 

In general, for 
1. I would keep the behaviour which is most consistent with what's currently happening (which I believe is count all the tokens). 
2. I think you'll have to do some checks that `"attention_mask"` is provided, although I think it should be possible to use. You'll probably need to account for whether it's in the 2d format or not. 
3. I would add a new option -- otherwise people's metrics will change under their feet. It might be that adding this introduces more complexity into trainer and we'll need to think about either abstracting out to make it easy to swap; or leaving the PR as a feature branch people can checkout and use without it being merged into main.
Config: allow saving with generative parameters if untouched
Author: gante
Pull request: yes
Labels: name: WIP

# What does this PR do?

⚠️ ⚠️ ⚠️ don't review yet, sorting all tests and use cases is taking more time than I thought 👀 

______________________

In a recent PR (#32659), we started raising exceptions if we were to store a model config with generative parameters. If we were saving a model, which saves the config, the generative parameters were moved to `GenerationConfig` before saving the `PreTrainedConfig`, working around the issue when saving a model.

However, a common use case was uncovered. If a user loads a `PreTrainedConfig` with generative parameters and tries to save it, the user has done nothing wrong and yet they see an exception. This PR lowers the exception to a warning in that situation, preventing the code from crashing :)

(Thank you @regisss for flagging this issue and @ydshieh for the testing suggestion)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33886). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This approach is opening a can of worms 😠 

Going to fix this issue another way
fix llava next incorrect padding size
Author: zyddnys
Pull request: yes

# What does this PR do?
Fix incorrect llava next image padding, if i have an input image size 950x1256 and using siglip as image encoder (shortest_edge=384) for llava next, the original code will create padded image with size 768x767 not 768x768.
This PR will fix the issue by directly calculating the right pad size


Comments:
Thanks for the fix @zyddnys ! two things: 
- this fix should also be included into `llava_onevision` if relevant there, because this processing part was copied from llava next. 
- can we add a test that'd break for the situation you described in the PR? 

- I have also updated the code for `llava_onevision`
- Sure, tests are also welcome

cc @zucchini-nlp since you worked on llava next!

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).
Update an keyerror on _save_check_point prevent confusion of missing …
Author: fadingNA
Pull request: yes

# What does this PR do?

### This PR improves the error handling in the `_save_checkpoint` 
- function by updating the `KeyError` message when the `metric_for_best_model` argument is not found in the evaluation metrics. The new error message provides clearer guidance, including suggestions on checking the `compute_metrics` function and updating `TrainingArguments` if necessary.


Fixes issues #33193

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@muellerzr and @SunMarc



Comments:
- issue #33193 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33832). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc  I have updated into the _evaluate function and commit your suggestion case error.
Add gguf for granite
Author: g-prz
Pull request: yes

# What does this PR do?

This PR adds gguf support for granite models


Contributes to #33260 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@SunMarc 


Comments:
Batch is empty when fine-tuning flan-t5 using LoRA
Author: MorenoLaQuatra
Pull request: no
Labels: name: trainer

name: lora

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@muellerzr @SunMarc

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The issue is reported here: https://discuss.huggingface.co/t/valueerror-the-batch-received-was-empty-your-model-wont-be-able-to-train-on-it-double-check-that-your-training-dataset-contains-keys-expected-by-the-model-args-kwargs-label-ids-label/20200


### Expected behavior

Correctly working fine-tuning with LoRA. Anyway, I thing the bug can be solved adding:

```python
self._signature_columns += list(set(["labels", "input_ids"]))
```

To the function `_set_signature_columns_if_needed` - [here](https://github.com/huggingface/transformers/blob/a4e1a1d02894b8d801f5d0182e1979b55daaeaa4/src/transformers/trainer.py#L794).

May not be the best way to go tho.

Comments:
Gentle ping @muellerzr @SunMarc 

Hello @MorenoLaQuatra,
Sorry for the delayed response, as you suspected, the error is related to the `_set_signature_columns_if_needed` function. This function removes all dataset columns that don't match the necessary arguments for the model's forward function, along with the columns "label" and "label_ids." While this approach works well for transformer models, where the forward function explicitly lists the required arguments, it doesn't behave as expected for the `LowRankAdapterModel`. In this case, the forward signature only includes `args` and `kwargs`, so all dataset columns get removed, leaving us with an empty batch, which causes the error.
To avoid the error, you can set `remove_unused_columns=False` (it's `True` by default) in the `TrainingArguments`, and manually clean your dataset.


Is it wrong to assume LoRA models will actually use the same columns as other models (e.g., input_ids and labels)? I got that there maybe workarounds but having it supported by default maybe a good choice, isn't it?

It's not wrong, but I don't think it's the cleanest approach. Setting `remove_unused_columns` to `False` seems like a cleaner solution for now, while in future versions we can think about allowing users to override the signature columns themselves if needed


Got it! Thanks @MekkCyber :+1:
VLM Generate: tag `test_static_cache_matches_dynamic` as flaky
Author: gante
Pull request: yes

# What does this PR do?

`test_static_cache_matches_dynamic` is VERY flaky on some VLMs. Tags it as flaky to avoid breaking CI while it is being investigated (cc @zucchini-nlp)

This test was enabled recently on VLMs on #33533 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33630). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(cc @LysandreJik -- can we fix this band-aid for the time being? It is a common failure in our CI :) )
Migrate the CI runners to the new clusters
Author: glegendre01
Pull request: yes

# What does this PR do?

Migrate the CI runners to the new clusters.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33849). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix module initialization for root module under Zero3
Author: Ben-Schneider-code
Pull request: yes

# What does this PR do?
Fixes #28808

Behaviour:  
```loaded_keys = {k.replace(f"{module_name}.", "") for k in state_dict_keys if k.startswith(f"{module_name}.")}``` in ```Trainer.py``` misses the case where the module_name is the empty string (the root module), as its looking for state_dict_keys that start with ```"."```. This causes the root module (CLIPModel, AlignModel, etc.) to always have ```model.apply(model._initialize_weights)``` called on it. Which leads to initialization of an empty parameter tensor when running zero3 (that doesn't need to be initialized as it will be replaced later) in ```modeling_align.py```. The reason this causes a crash in ```modeling_align.py``` and not other models like CLIP is because ALIGN uses ```xavier_uniform_``` which asserts the tensor must be 2d whereas CLIP uses ```nn.init.normal_``` which just returns another empty tensor.

Fix:  
Use the whole state_dict as the root module's set of load_keys.

Please let me know if there is anything I should revise / this should be fixed in another way.😄 
@ArthurZucker @muellerzr

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33632). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> This LGTM but can you please add a comment about why we are doing this? 🤗

Done! :+1:

Thanks 🫡
Add `SplinterTokenizer` unit test
Author: ariepratama
Pull request: yes
Labels: name: Core: Tokenization

# What does this PR do?
* Add `SplinterTokenizer` unit test
* Pass `question_token` in `SplinterTokenizer.save_pretrained`, to ensure the saved tokenizer_config will have the same parameter.

Fixes #16627


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 







Comments:
> Looks alright, but in general we try to copy a lot from other tests, this is missing `Copied from` in a few places

thanks! I believe I've applied the comments 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32652). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

THanks for your contribution!
Fix update stateful callbacks state to save checkpoint
Author: lucasrearte
Pull request: yes

* fix: avoid appending to the same list in every callback state update, when there are multiple callbacks of the same class

* fix: make _tune_save_checkpoint method updates callbacks state too

* test_stateful_duplicate_callbacks: add modifiable field to test changes in state while training

# What does this PR do?

This PR fixes a problem when there are multiple stateful callbacks of the same class to be saved. Their states are updated, but they are appended to the same list (of all callbacks of that same class) every time a checkpoint is saved, causing it to grow without limit. In these cases, as a consequence, when training is resumed from a checkpoint, only the first callback states (of the same class) are loaded to the callback handler, possibly ignoring further changes in their states (the latest elements in the list). To solve this problem, a new list is created with the updated callback states, instead of appending them to the existing list.

Regarding the update of the callback states, this PR also makes method _tune_save_checkpoint(...) do the same thing as method _save_checkpoint(...).

## Who can review?

@muellerzr @amyeroberts @pedrobrs 


Comments:
cc @SunMarc
Generate tests: modality-agnostic input preparation
Author: gante
Pull request: yes

# What does this PR do?

Requirement for #33212
Follow-up to #33663  

This PR rewrites the LLM-centric `_get_input_ids_and_config()`, the function that creates random model inputs for tests, into a modality-agnostic `prepare_config_and_inputs_for_generate()`

The rest of the diff consists in propagating the change. Highlights:
1. most `_get_input_ids_and_config()` overwrites were deleted as a result of the changes 🔪 
2. most test generate calls receive a dictionary of input, as opposed to `input_ids`
3. `_check_outputs` now receives the model's main input, as opposed to `input_ids`
4. because of the changes above, a few test overwrites that needed to be updated could be deleted instead 🙏 

___________________________

In a follow-up PR: hunt tests that no longer need to be skipped/overwritten as a result of these changes 🎯 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33685). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker that is a cool idea, model-agnostic inputs, potentially dependent on modality (or perhaps simply looking at the signature of the fwd pass?)

I took note of it to explore after the current round of reactors, to avoid adding more parallel threads :D