noamaanMulla-03/transformers-issues huggingface dataset:
validation of the eval dataset should be done in advance
Author: jackyjinjing
Pull request: no
Labels: name: bug

### System Info

I found that the validation of the eval dataset should be done in advance. It only prompted that the validation set was empty after I had trained for half an hour and completed one epoch.The code is as follows.I set                         eval_strategy="epoch" in train_args,  but I don't set eval dataset in Trainer init.


`from transformers import TrainingArguments

train_args = TrainingArguments(output_dir="./checkpoints",     
                               per_device_train_batch_size=2, 
                               per_device_eval_batch_size=1,  
                               logging_steps=1,                
                               **eval_strategy="epoch",**    
                               save_strategy="epoch",          
                               save_total_limit=3,            
                               learning_rate=2e-5,           
                               weight_decay=0.01,
                               num_train_epochs=30,
                               dataloader_drop_last=True,
                               metric_for_best_model="f1",   
                               load_best_model_at_end=True
                               )  

from transformers import DataCollatorWithPadding, Trainer

trainer = Trainer(model=model, 
                  args=train_args, 
                  train_dataset=process_dataset["train"], 
                  **eval_dataset=None,** 
                  data_collator=collate_fn)

`

transformers version:4.45.0
python version:3.10
paltform:ubuntu

@muellerzr @Sunm

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. set eval_strategy="epoch" in train_args
2. set eval dataset=None in Trainer init
3. run Trainer.run()

### Expected behavior

validation of the eval dataset should be done in advance

Comments:
Image pipelines spec compliance
Author: Rocketknight1
Pull request: yes

This PR checks for Hub spec compliance for a lot of image pipelines that have very similar inputs. In most cases, no changes are required except deprecating the `timeout` argument, and renaming the `images` input to `inputs`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually thinking more about it and inline with having less and less warnings in `transformers`, I wonder if we want to have a deprecation warning for `images` vs `outputs`, or if we just want to enable both without warning

@LysandreJik yes, I'm fine with that! Want me to just drop all the deprecation warnings? I can maybe throw an error if users specify **both** arguments, or something.

If no deprecation warning is raised (i.e. accept both `inputs` and `images` on the long run), I would make sure the method signature is in line with the recommendation (i.e. passing `inputs`). In that sense, I don't think that having `inputs: ... = None` on the long run is desirable as it can be misleading to users. If it was only for a few versions before settling for the definitive signature (i.e. no `images` at all), it would have been fine but that's not the case, right?

@Wauplin Unfortunately, if we don't set `inputs = None`, then users have to pass `inputs`, and we can't accept `images` as an alternative.

I think it's okay, though! There are a lot of classes in `transformers` that have `None` defaults for mandatory inputs - it just means that we handle the error when the argument is missing somewhere else.

I'll let you decide the best course of action in terms of signature/arguments @Rocketknight1, but dropping the deprecation warnings seems to me like the right thing to do here

Done! Warnings are removed, but I'll accept both keywords for now - I don't think the main input having a default of `None` should be too confusing, especially since the docstrings are clear.
BatchFeature.to() supports non-tensor keys
Author: Rocketknight1
Pull request: yes
Labels: name: run-slow

This PR fixes a bug in the preprocessing for several pipelines. The pipelines were calling `.to()` on a `BatchFeature` which had a string feature, which caused an error. This update was also requested internally in Slack!

cc @ydshieh 

Comments:
Thank you @Rocketknight1 .

2 questions:

- Is this the only one to change? I see 3 failed tests (in torch pipeline CI reports).
- Do you think it makes sense to have this implemented directly in `BatchFeature` instead of outside it?


@ydshieh good spot, I fixed it for all three! We could consider rewriting `BatchFeature.to()` like you suggested though, let me ask people

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33918). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ydshieh I changed the PR to update `BatchFeature.to()` instead, which means the individual pipeline fixes aren't needed anymore

cc @ArthurZucker for core maintainer review! This PR is simple: It patches `BatchFeature.to()` to check that its keys are tensors before it calls `.to()` on them. This means that `BatchFeature.to()` works even with string keys now, which used to throw an error.

Are there other areas that should be updated? Arthur tells me about Pixtral as an example

@LysandreJik I checked the codebase for methods overriding `to()`. Mostly this was in modeling code to create objects like `NestedTensor` - these are safe because their contents should always be `torch.Tensor`.

The exceptions were:
- Pixtral
- `BatchEncoding` (the tokenizer output class)

In both cases, I updated their methods to make sure they only call `to()` on `torch.Tensor`!
Add barlow twins
Author: dame-cell
Pull request: no
Labels: name: New model

name: Vision

### Model description


I'm seeking guidance on potentially contributing a pretrained ResNet50 model to the Hugging Face model hub. but the problem is that they did not provided the linear weights which they evaluated the model on the imagenet1k images 
would contributing this model still seems viable

Model details:
- Architecture: ResNet50 
- Training method: Self-supervised learning 
- Weights: Pretrained backbone weights available, but no linear classification layer

in their repo they provided the weights for both the renset50 and full checkpoints but both do not contain linear weights 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

model implementation -  [barlow-twins](https://github.com/facebookresearch/barlowtwins)
model weights - [weights](https://github.com/facebookresearch/barlowtwins)
paper - [paper](https://arxiv.org/pdf/2103.03230)



Comments:
@amyeroberts @

cc @qubvel

Hi @dame-cell, thanks for your interest in contribution! We already have [resnet50](https://github.com/huggingface/transformers/tree/main/src/transformers/models/resnet) model in transformers, if you are interested in this particular weights, you can convert them to a compatible format and put it on the [Hub](https://huggingface.co/models) so that anyone can use it!

@qubvel  I see thank you so much for  replying I got it
Fixup DeepSpeed things
Author: muellerzr
Pull request: yes
Labels: name: DeepSpeed

# What does this PR do?

Fixes a (soon to be failing) DS test with accelerate 1.0. We need to manually override the DS plugin when doing hp sweeps since we now don't allow for arbitrary passing in of a ds plugin to an already initialized accelerator

~~Also notes an actual deprecation point for when we want to force users to use `integrations.deepspeed` rather than `.deepspeed` to 5.0. @ArthurZucker @LysandreJik let me know if that's too soon, but it's something that was irking me :)~~

Seems we already got rid of it! 🔥 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @LysandreJik 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34007). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typing issue
Author: SunMarc
Pull request: yes

# What does this PR do ?
This PR fixes a typing issue that this [PR](https://github.com/huggingface/transformers/pull/33940) introduced. 
TLDR: we can't do dict[str,dict] with python 3.8 cc @ydshieh 

Traceback received: 

```
Traceback (most recent call last):
  File "/home/marc/transformers/src/transformers/utils/import_utils.py", line 1780, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/marc/transformers/src/transformers/models/__init__.py", line 15, in <module>
    from . import (
  File "/home/marc/transformers/src/transformers/models/mt5/__init__.py", line 36, in <module>
    from ..t5.tokenization_t5_fast import T5TokenizerFast
  File "/home/marc/transformers/src/transformers/models/t5/tokenization_t5_fast.py", line 23, in <module>
    from ...tokenization_utils_fast import PreTrainedTokenizerFast
  File "/home/marc/transformers/src/transformers/tokenization_utils_fast.py", line 34, in <module>
    from .modeling_gguf_pytorch_utils import load_gguf_checkpoint
  File "/home/marc/transformers/src/transformers/modeling_gguf_pytorch_utils.py", line 245, in <module>
    weights: np.ndarray, parsed_parameters: dict[str, dict], name: str, tensor_key_mapping: dict
TypeError: 'type' object is not subscriptable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test_gemma_save.py", line 1, in <module>
    from transformers import AutoModelForCausalLM
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/home/marc/transformers/src/transformers/utils/import_utils.py", line 1770, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/marc/transformers/src/transformers/utils/import_utils.py", line 1782, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback):
'type' object is not subscriptable
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34012). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix type bug for python version < 3.9.0
Author: real-zhangzhe
Pull request: yes

# What does this PR do?

Fix type bug for python version < 3.9.0
`dict[str, dict]` just work for python version >= 3.9.0, for other python version, "'type' object is not subscriptable"

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- pipelines: @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- text models: @ArthurZucker

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)

- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
https://github.com/huggingface/transformers/pull/34012 fixed!
so close this PR

Thanks anyways for the intention! 🤗
fix awq tests due to ipex backend
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes a couple of issues (mainly versioning) from the ipex backend that was added to awq quantizer. We were importing the kernel but it is only available with awq from source. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34011). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, that why ;)
Remove `decoder_config=None`
Author: SunMarc
Pull request: yes

# What does this PR do?
I've merged this [PR](https://github.com/huggingface/transformers/pull/33934) a little to fast, we need to remove a line as stated [here](https://github.com/huggingface/transformers/pull/33934#discussion_r1790157995). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I think it is decoder_config that needs to be removed, default_config is actually used.

Indeed thanks !
feat(tokeniser): add lazy token conversion to ids
Author: winstxnhdw
Pull request: yes
Labels: name: Feature request

## Summary

Add fast tokenisation method to lazily convert tokens to ids. This doesn't introduce any breaking changes.

Comments:
I have my own decode method that can lazily evaluate the tokens so that I can have an end-to-end lazy pipeline from my model all the way to client. 

@winstxnhdw are you experiencing memory limitations with the existing`convert_tokens_to_ids`?

I am running this in a really light VPS with little memory and this function is forcing my program to accumulate all the tokens into memory.

Sure, if not for the feature, could you at least get rid of this redundant if-condition.

https://github.com/huggingface/transformers/pull/33644/files#diff-a8b32d72cf9a75d81e800354cf32933ddcf7c8c3e3d3d12bb7450fee98bb9595L352-L353

This would fail `[self._convert_token_to_id_with_added_voc(token) for token in tokens]` for k in None 

> This would fail `[self._convert_token_to_id_with_added_voc(token) for token in tokens]` for k in None

It would fail the type checker first.

It will fail type checking but does not prevent anyone from calling that function! 

If the user passes in a variable of the wrong type, that’s a user error and not the library’s responsibility. What’s the point of type hints if you are going to check for `None` unnecessarily? Why not just do this check for every single parameter of all your publicly exposed functions/methods since the user could potentially pass in `None`?

I agree with you! 🤗 Feel free to just remove this check then! 🤗 

I have opened a new PR to remove this check at #34030. I will be closing this PR.
Fix `trainer_seq2seq.py`'s `__init__` type annotations
Author: benglewis
Pull request: yes

Fix type annotations for `trainer_seq2seq.py`'s `__init__` call.

@ylacombe @eustlb
@muellerzr @SunMarc

Apologies if I tagged irrelevant people, this is my first PR. I also see some other type annotation issues, e.g. `overloads` that are missing. If this kind of PR will be considered, I would be happy to contribute other fixes too!

Comments:
Could you also run the code quality tool to ensure that the code quality passes? You can install them with the following, from the root of your clone:
```
pip install -e ".[quality]"
```
And then run them with:
```
make fixup
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34021). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik I ran `make fixup` following that install command and installing a few other pip dependencies that seemed to actually be necessary to run it and everything is fine following the fix I applied after @muellerzr 's comment.
[Tests] Diverse Whisper fixes
Author: ylacombe
Pull request: yes
Labels: name: run-slow

# What does this PR do?

There's a lot of pending failing tests with Whisper. This PR addresses some issues:

1. #31683 and #31770 mentioned a out-of-range word level timestamps. This happens because `decoder_inputs_ids` were once `forced_input_ids`. This had an impact on the `beam_indices`. 

`beam_indices` has a length of `decoder_input_ids + potentially_generated_ids` but doesn't take into account `decoder_input_ids` when keeping track of the indices. In other words `beam_indices[0]` is really the beam indice of the first generated token, instead of `decoder_input_ids[0]`.

2. The Flash-Attention 2 attention mask was causing an issue

3. The remaining work is done on the modeling tests. Note that some of these tests were failing because of straightforward reasons - e.g the output was a dict -  and are actually still failing, but their reasons for failing are not straightforward anymore. Debugging will be easier though.


**Note:** With #33450 and this, we're down from 29 failing tests to 17
  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33665). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LMK when you want me to review @ylacombe (please fix the tests and the conflicts first :blush:)

Hey @LysandreJik, could you review now ? It seems that the failing check is unrelated to Whisper !

cc @ArthurZucker  for a quick look !

cc @ydshieh , there's still this error when doing slow tests: 
 ```
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
Bus error (core dumped)
```



hi @ylacombe let me merge #33849 and try to rebase this PR on top of main and let's see.

Could you rebase and trigger again with `[run-slow] whisper`?

It's working thanks !

The list of failing tests are as expected, merging !

Great work! I noticed there's a small issue in this PR, especially when `prompt_ids` is specified—the model always returns a timestamp of 0.0.

To fix this quickly, you can update https://github.com/huggingface/transformers/blob/38f9f10dd9240619ea17fb6c7acb51b3bc592232/src/transformers/models/whisper/generation_whisper.py#L191 to:
```python
cross_attentions.append(torch.cat([x[i] for x in generate_outputs.cross_attentions], dim=2)[:, :, num_input_ids-3:, :])
```

Hey @leng-yue, thanks for your message!
While there's maybe something wrong with how cross-attentions weights are computed, I'm not sure that your proposed solution works. 
Could you provide a code snippet and maybe an explanation of the proposed solution if you have time?
Thanks!

Basically `num_input_ids` is `prefix_tokens` (timestamps, language, etc) + `prompt_ids`, therefore, we want to strip the`prompt_ids`, which is `num_input_ids` - 3.