noamaanMulla-03/transformers-issues huggingface dataset:
Trainer: To keep unused columns for `compute_metrics`
Author: sadra-barikbin
Pull request: no
Labels: name: trainer

Hi there!

Currently, columns not used by the model are removed in `self.get_*_dataloader()` upon data loader creation, but one might want to have them in `compute_metrics` (when `include_inputs_for_metrics=True`). 

My case is fine-tuning on prompt-completion's and I use tokenizer's `token_type_ids` as a mask to compute accuracy only on the completion tokens.

To this end, the best way I've come up with is to keep that column in dataset & data loader using `remove_unused_columns=False` and then remove it in `self._prepare_inputs()` by overriding it.

Is there a better way to achieve this? Generally, isn't better to move removing unused columns logic to `self._prepare_inputs` if the logic serves only as the gatekeeper for `model(**inputs)`?

Comments:
cc @muellerzr @SunMarc 

@sadra-barikbin 
for the trainer API you should specify `remove_unused_columns=False` in the `TrainingArguments`
checkout https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.remove_unused_columns for more info


`model(**inputs)` rejects those additional columns and raises error by only using `remove_unused_columns=False`.

Minimal reproduction:

```python
from typing import Dict, List
from transformers import Trainer, TrainingArguments, AutoTokenizer, AutoModelForCausalLM
from transformers import LineByLineTextDataset, DataCollatorForLanguageModeling
from datasets import Dataset

model = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-14m")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-14m")
tokenizer.pad_token = tokenizer.eos_token

data = Dataset.from_dict({'prompt':['A', 'B'], 'completion': ['a', 'b']})

def tokenize(example: Dict[str, str]) -> Dict[str, List[int]]:
  return tokenizer(example['prompt'], f"{example['completion']}.", return_token_type_ids=True)

dataset = data.map(tokenize, remove_columns=['prompt', 'completion'])

args = TrainingArguments(
    output_dir="test",
    report_to='none',
    remove_unused_columns=False,
    max_steps=3,
)

trainer = Trainer(
    model=model,
    tokenizer=tokenizer,
    args=args,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
    train_dataset=dataset,
)

trainer.train()
```

```
TypeError: GPTNeoXForCausalLM.forward() got an unexpected keyword argument 'token_type_ids'
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Might be something related to the requested feature, I am adding `predict_with_generate` in Trainer which will let us compute metrics on the generated text, also can compute only on generated part (wo prompt that was fed to the model). Track in https://github.com/huggingface/transformers/pull/32346
RoPE: Add numerical tests ✨ 
Author: gante
Pull request: yes

# What does this PR do?

As promised in #32135: add numerical tests and sanity-checks for all rope strategies.

Walking through the tests, it should be very clear what each rope strategy does!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32380). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate] only require an attention mask for mps with torch<2.4
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

torch>=2.4 provides the `.isin` operation for mps, enabling us to relax a constraint on passing the attention mask if a valid torch version is installed.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32367). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sanchit-gandhi can we also update `__call__` in `EosTokenCriteria`, to throw a torch 2.4 warning? (that way we probably won't forget to update this special path when we stop supporting 2.3)
When max_steps < save_steps with deepspeed zero3 stage
Author: macheng6
Pull request: no
Labels: name: trainer

name: DeepSpeed

### System Info

transformers  4.41.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I set **max_steps=100** and **save_steps=200**, I found that the trainer could not save the trained weights and was blocked somewhere, causing the program to be unable to stop.
the code is:
` if training_args.do_train:
        model.gradient_checkpointing_enable()
        model.enable_input_require_grads()
        trainer.train(args.resume_from_checkpoint)
        # For convenience, we also re-save the tokenizer to the same directory,
        # so that you can share your model easily on huggingface.co/models =)
        if trainer.is_world_process_zero():
            trainer.save_model()
            tokenizer.save_pretrained(training_args.output_dir)
`
   

### Expected behavior

I hope that :
- either an error message will be displayed, prompting the user that **max_steps<save_steps**, 
- or the last trained weights can be saved.

Comments:
cc @muellerzr @SunMarc 

When I removed `trainer.is_world_process_zero()`,  the code ran normally,  but if it is not removed, the code blocks here `state_dict = self.accelerator.get_state_dict(self.deepspeed)` of the function `trainer.save_model()`

Hi @macheng6, are you sure that this is an issue with max_steps and save_steps ? If you set max_steps > save_steps, do you get the issue and is the the code below` trainer.is_world_process_zero() `running fine ? If you have a minimal reproducer, that would be great ! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix perceiver latent initialization modeling_idefics2.py
Author: ncoop57
Pull request: yes

# What does this PR do?

The perceiver resampler in idefics2 initializes latents as ones, but they should be initialized as random numbers with a gaussian distribution (see this implementation: https://github.com/lucidrains/perceiver-pytorch/blob/main/perceiver_pytorch/perceiver_io.py#L153). This should fix instability issues for training


Models:

=- vision models: @amyeroberts

Comments:
looks ok to me too and makes sense!
we had `torch.randn` as init before the `torch.ones` in our internal codebase, but I can't find in my notes why we made that switch...

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any update @ncoop57 ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Checkpoint validation as an option
Author: ChenchaoZhao
Pull request: no
Labels: name: Feature request

### Feature request

Added `validate_checkpoint_key: bool` in `from_pretrained`

### Motivation

When pytorch loads state dict there is an option `strict=True` but there is no counterpart in `from_pretrained`. If people saved the pretrained model incorrectly, the incorrect keys can silently pass through.

### Your contribution

There is already an validation step in the `from_pretrained` one just need to raise the error if `validate_checkpoint_key=True`

Comments:
Hi @ChenchaoZhao, thanks for opening a feature request! 

We do validation already within `from_pretrained` when we load in checkpoints, and raise errors if there's missing or unexpected keys in the state dict versus the model architecture. 

Perhaps you could expand a bit more on the kind of validation you'd like to see? 

> If people saved the pretrained model incorrectly, the incorrect keys can silently pass through.

Could you expand on this a bit, specifically how this might happen? Maybe a code snippet would help, as there could be a bug in the `save_pretrained` method 

Hi @amyeroberts 
For example, my friend saved a model from distributed training without unwrapping. The state dict keys got a prefix that did not match the unwrapped model state dict keys. When loading the model `AutoModelForMaskedLM.from_pretrained` there was a warning saying every key was not matched. I hope the warning can be turned into a raise by a configurable kwarg in the `from_pretrained`

@ChenchaoZhao OK, just to make sure I've correctly understood, what you're proposing is a flag which enforces strict loading such that if any weights in the model are not present in the state_dict then an exception is raised? 

> @ChenchaoZhao OK, just to make sure I've correctly understood, what you're proposing is a flag which enforces strict loading such that if any weights in the model are not present in the state_dict then an exception is raised? 

That's correct. Thanks!

I'm not opposed to it in principle, although I'm not sure how impactful it would be WDYT @LysandreJik @ArthurZucker? Is there a reason we did have this feature in the first `from_pretrained` designs? 

Yeah sounds good. In general people load base model inside specifc §`ForZZZ` classes, which is why we don't have this by default. Not sure how much usage this will get but feel free to open a PR, we can make sure this is used in every single one of our "conversion scripts" to ensure we don't make mistakes! (happened to me!)
Sequence Length Invariant Text Models
Author: chirag4798
Pull request: yes

### What does this PR do?
This PR introduces changes to the `huggingface/transformers` repository aimed at making models like CLIP and similar ones invariant to sequence length. The primary modification involves applying Bi-Linear interpolation to the embeddings generated by the `position_embedding` layer. This allows the embeddings to be interpolated to the required sequence length, enabling the models to handle sequences of varying lengths effectively.

#### Motivation and Context
The motivation behind this change is to improve the flexibility and robustness of models when processing inputs of different lengths. The `max_position_embedding` for pre-trained models like CLIP is very limiting. By ensuring sequence length invariance, the models can maintain performance and consistency regardless of input sequence variations. Also to enable the use of these models with StableDiffusion Pipelines in `huggingface/diffusers` for long prompt generation, enhancing their applicability and integration in more complex workflows.

#### Fixes
Fixes: https://github.com/huggingface/diffusers/issues/2136

#### Dependencies
There are no additional dependencies required for this change.

#### Checklist
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


@ArthurZucker, @amyeroberts, @qubvel  and @younesbelkada



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
No special tokens for LLaMA 1 in conversion script
Author: viktor-shcherb
Pull request: no
Labels: name: Good First Issue

name: bug

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.43.3
- Platform: Linux-5.15.0-116-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.3
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:NO
- Using GPU in script?: NO
- GPU type: Tesla V100-SXM2-32GB

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download original weights to `llama-tokenizer/7B`
2. Run:
```
✗ python ../venv/lib/python3.12/site-packages/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir llama-tokenizer/7B --output_dir llama7b --model_size 7B
Traceback (most recent call last):
  File "/mloscratch/homes/shcherba/landmark-attention/llama/../venv/lib/python3.12/site-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 478, in <module>
    main()
  File "/mloscratch/homes/shcherba/landmark-attention/llama/../venv/lib/python3.12/site-packages/transformers/models/llama/convert_llama_weights_to_hf.py", line 452, in main
    args.special_tokens = DEFAULT_LLAMA_SPECIAL_TOKENS[str(args.llama_version)]
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '1'
```

### Expected behavior

No errors, completed conversion, as described in https://huggingface.co/docs/transformers/en/model_doc/llama

Comments:
Hey! Indeed, we can set the default to `[]`. As this is only used for Llama3. 
Would you like to open a PR for the fix? 🤗 

Sure, one moment
SPLIT PR: add_prefix_space fix
Author: itazap
Pull request: yes

Fixes portion of https://github.com/huggingface/transformers/issues/30824

Adds support for add_prefix_space. 

TASKS:
#### convert_slow_tokenizer.py
- [x] `add_prefix_space` is set based on original_tokenizer, otherwise based on proto's noramlizer. 
#### llama_tokenizer.py
- [x] `add_prefix_space` is updated based on normalizer if unset.
#### llama_tokenizer_fast.py
- [x] **remove** forcing `from_slow` conversion if `add_prefix_space` is not set (see tokenization_utils_fast.py updates on how this is handled without conversion). This allows passing this field **wthout sentencepiece installed.**
#### tokenization_seamless_m4t.py
- [x] updated comment 'Copies' - ruff complained
#### tokenization_t5.py
- [x] do not set prefix to True, set based on  normalizer
#### tokenization_t5_fast.py
- [x] set prefix space, do not force from slow if set
#### tokenization_utils.fast.py
- [x] set prefix space
- [x] allow using `add_prefix_space` without converting from slow, thus allowing to use without sentencepiece installed
- [x] update rust pre_tokenizer and normalizer based on `add_prefix_space` and underlying normalizer. legacy logic copied from `convert_from_slow.py`

TESTS:
- [x] test legacy tokenizer in llama
- [x] test legacy tokenizer in t5
- [x] test tokenizer without sentencepiece installation (mock)

Reviewer:
@ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31315). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

#1590 supports :
- indexing
- updating with None
- updating the indexed element
fix wav2vec2 with torch.compile
Author: zrr1999
Pull request: yes
Labels: name: Audio

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #101160


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sanchit-gandhi
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


 -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: SeamlessM4TFeatureExtractor stride remainder
Author: TechInterMezzo
Pull request: yes

# What does this PR do?

When the padded input length of SeamlessM4TFeatureExtractor is not a multiple of stride then the feature extraction fails.
This can happen when pad_to_multiple_of is not a multiple of stride and num_frames % stride is not zero.
The solution is to subtract the remainder from num_frame and adjust the size of some tensors.

Fixes #31916


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ylacombe

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32088). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ylacombe Do I need to do anything else?