noamaanMulla-03/transformers-issues huggingface dataset:
Log a warning when best model is not loaded
Author: akwako
Pull request: yes

Specifying "load_best_model_at_end=True" in TrainingArguments should ensure that the best model is always loaded at end of training. However, this is not always the case: When best_model_checkpoint is None, the best model is not loaded, and the user may be unaware of this behavior.

Add a warning to the log to let the user know when the best model is not loaded at the end of training. Suggest that the user check the "save_strategy" TrainingArguments, as failing to to do so is one possible reason why the best model failed to load.

@muellerzr 
@pacman100 

Comments:
Hi @amyeroberts, 

Thank you for your reply and feedback!

As far as I can tell, the main issue is when `max_steps` < `args.save_steps`, and only when `save_strategy="steps"`. Since `max_steps` is computed in the `_inner_training_loop`, I have added a check there to make sure training fails fast, as you suggested. However, in case this is happening in other ways, it might be good to keep the warning in the log, too, as per the original PR. What do you think? 

Possibly related to #27332 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28500). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @amyeroberts, 

Thanks for your suggestions! 

I'm looking into [#27332](https://github.com/huggingface/transformers/issues/27332). You may be right that these are related issues. When I run `trainer` with `EarlyStoppingCallback`, I do get the (newly added) message: `ValueError: args.save_steps must be less than max_steps...` I will look further into `EarlyStoppingCallback` to confirm.

Update: In order to determine if the issue [#27332](https://github.com/huggingface/transformers/issues/27332) is due to max_steps < args.save_steps, I think we need to know what the other training arguments are, and how large the training sample is. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Table Transformer] Convert more checkpoints
Author: NielsRogge
Pull request: yes

# What does this PR do?

Microsoft released some new Table Transformer (TATR) checkpoints, I've converted them and pushed them to the hub:

* https://huggingface.co/microsoft/table-transformer-structure-recognition-v1.1-fin
* https://huggingface.co/microsoft/table-transformer-structure-recognition-v1.1-pub
* https://huggingface.co/microsoft/table-transformer-structure-recognition-v1.1-all

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27723). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add Fusion-In-Decoder(FiD) Models
Author: oh-gnues-iohc
Pull request: yes
Labels: name: New model

add Fusion-In-Decoder(FiD) Models
It was simply implemented based on t5
@ArthurZucker 

Comments:
> Thanks a lot for contributing! My best recommendation would be to first share the model on the hub using[ this ](https://huggingface.co/docs/transformers/v4.38.0/custom_models)tutorial, and then if the community shows interest we can start working on the addition to `transformers` 🤗 Sharing the model on the hub is the easiest way to reach your users!

Here, I have prepared everything I can share about this model. [model](https://huggingface.co/ohgnues/FiDT5), [space](https://huggingface.co/spaces/ohgnues/FiD)

Is this model merged only when it gains popularity among many people?
@ArthurZucker
Fix missing translation in README_ru
Author: strikoder
Pull request: yes

# What does this PR do?
This PR fixes the Russian translation of the README file by translating one line from English to Russian that was forgotten to be translated.

## Before submitting
- [x] This PR improves the docs.

## Fixes #26208 

 @stevhliu 
@MKhalusova 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29054). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thank you for spotting the missing paragraph, and adding translation. I left a couple of suggestions.

Okay, thank you!
[Gemma] Fix eager attention
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes the Gemma "eager" attention implementation, which is the default for torch versions <= 2.1. This issue was reported on the [Hub discussions](https://huggingface.co/google/gemma-7b-it/discussions/13) and by @osanseviero from the model cards/blog post. 

The PR also includes a set of slow tests to ensure:
1. Logits equivalence between {eager, sdpa, flash attention 2}
2. Expected generation results for {eager, sdpa, flash attention 2}

=> these tests confirm all attention implementations work and have equivalence between back-ends

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29187). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the catch. A real pity our 280 prior test did not catch this! 🤗 

I'll look at integrating 2 tests in `test_modeling_common.py` that can automatically test for eager vs sdpa and eager vs fa2. Currently, this is done on an ad-hoc basis for each model (e.g. Whisper [here](https://github.com/huggingface/transformers/blob/2a9b1f80c45cab19b542bc7cc004937d39d6f6fb/tests/models/whisper/test_modeling_whisper.py#L891)).

It's also that it tests dummy models, but here the head_dim != hidden / head! So a case no studied

@sanchit-gandhi Not true, see https://github.com/huggingface/transformers/blob/2cc8cf6ce7ae0416561acbb639df4bbc5f409b6f/tests/test_modeling_common.py#L3431. It is just a slow test. For FA2 vs eager it's possible there is not global test though, did not check that

eager vs sdpa passed 😉
EsmTokenizer does not return correct length token type ids
Author: lhallee
Pull request: no

### System Info

transformers 4.37
python 3.10.11

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

EsmTokenizer ignores special tokens when generating

```
tokenizer = EsmTokenizer.from_pretrained('facebook/esm2_t6_8M_UR50D')
seq = 'PROTEIN'
len(seq) # 7

tokens = tokenizer(seq, seq, return_token_type_ids=True)

len(tokens.input_ids), len(tokens.token_type_ids) # 17, 14 
```

Is the same for return_special_tokens=True or False

I understand Esm does not natively use token_type_ids but some personal versions and upcoming contributions to the field do and could benefit from having EsmTokenizer with correct token_type_ids.

### Expected behavior

```
tokenizer = EsmTokenizer.from_pretrained('facebook/esm2_t6_8M_UR50D')
seq = 'PROTEIN'
len(seq) # 7

tokens = tokenizer(seq, seq, return_token_type_ids=True)

len(tokens.input_ids), len(tokens.token_type_ids) # 17, 17 
```

Comments:
Suggested changes that fix this in Pull request [#28657](https://github.com/huggingface/transformers/pull/28657)

cc @Rocketknight1 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Let's close this as not planned 😉
Fast tokenizer's time complexity is not linear
Author: getao
Pull request: no

### System Info

torch 2.1.2
transformers 4.36.2

### Who can help?

@ArthurZucker 

I'm using the streaming dataloader to train on a large dataset. However, I find some process is usually stuck and finally lead to NCCL timeout After I carefully checked, I found the problem may come from the tokenizer. 

When a batch (I think it is 1000 by default) contains too many tokens (e.g., the batch has a document that is a book and is very long), the tokenization process will be extremely slow. I test the tokenization efficiency for sequences with different lengths and find the time cost is not linear to the sequence length but looks like quadratic. 

Tokenizing a 50k-word sequence costs 0.5s but tokenizing a 500k-word sequence costs 70s (about 140x slower).

I don't know if it is a bug. If it is by design, how should I prevent the tokenizer being stuck by some batches with too many tokens. I think one way is to reduce the batch size (default 1000). Is there any other way?

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
import time

def test_time(test_str):
    a = time.time()
    tokens = tokenizer(test_str)
    b = time.time()
    return b-a

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
test_str = "I love NLP. " * 100000  # we can change the number to try different lengths
time_cost = test_time(test_str)
print(time_cost)
```

### Expected behavior

Linear time cost increase as the sequence length

Comments:
Feel free to check this issue (duplicate) #25873

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Part of the issue with Llama tokenizer were debugged see here! https://github.com/huggingface/tokenizers/releases/tag/v0.15.2
No, you cannot set a `token` to an `id`. It is the same as `tokenzier.pad_token_id = 0` if `tokenizer.eos_token_id` is `0`
Author: rafa852
Pull request: no

              No, you cannot set a `token` to an `id`. It is the same as `tokenzier.pad_token_id = 0` if `tokenizer.eos_token_id` is `0`

_Originally posted by @ArthurZucker in https://github.com/huggingface/transformers/issues/26072#issuecomment-1859852130_
            

Comments:
Hi @rafa852 - what is the issue being raised here?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as there is no answer and no explanation!
Qwen2 weights are not there/deleted?
Author: aliencaocao
Pull request: no

### System Info

https://huggingface.co/Qwen2/Qwen2-7B-Chat-beta gives a 404, despite the tutorial in https://huggingface.co/docs/transformers/main/model_doc/qwen2 quoting it.
https://huggingface.co/Qwen only has Qwen-1 models.
@ArthurZucker @younesbelkada @stevhliu 

By the way, the docs is inconsistent in the model path. Most uses https://huggingface.co/Qwen2/Qwen2-7B-beta but in docs for the [config class](https://huggingface.co/docs/transformers/main/model_doc/qwen2#transformers.Qwen2Config), it uses https://huggingface.co/Qwen/Qwen2-7B-beta which also does not exist.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Visit https://huggingface.co/Qwen2/Qwen2-7B-Chat-beta

### Expected behavior

Model is there and accessible

Comments:
cc @JustinLin610 the author of the pull request! Are the weights not publicly available yet? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Weights still not up

Models are [here](https://huggingface.co/collections/Qwen/qwen15-65c0a2f577b1ecb76d786524) just not called `Qwen2` ! 

Shouldnt be closed as docs is still linking the wrong one.

Opening a PR now!
[`Doc`] update model doc qwen2
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #28641 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29238). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Models with remote code are not loaded correctly when there's `.` in their name.
Author: BlackSamorez
Pull request: no

### System Info

```
- `transformers` version: 4.37.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

quantized_model = AutoModelForCausalLM.from_pretrained(
    "BlackSamorez/Mixtral-8x7B-Instruct-v0.1-AQLM-2Bit-1x16-hf",
    trust_remote_code=True, torch_dtype="auto", device_map="cuda"
)
```
produces:
```
ModuleNotFoundError: No module named 'transformers_modules.BlackSamorez.Mixtral-8x7B-Instruct-v0'
```
It's clear that `.` in the name break custom code imports

### Expected behavior

Expected that model is loaded correctly.

Comments:
cc @Rocketknight1 

Hi @BlackSamorez, this issue has been reported already at #28919. We're working on a fix right now! I'm going to close this issue as a duplicate.

This has now been fixed in #29175
Use torch 2.2 for daily CI (model tests)
Author: ydshieh
Pull request: yes

# What does this PR do?

`torch 2.2` is released 3 weeks ago, and our CircleCI already use it, not not daily CI so far.

This PR update the docker file to use torch 2.2 for daily CI, but only for the modeling tests.

(p.s. We are in the process of moving other tests to separate workflow files)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29208). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add SiglipForImageClassification and CLIPForImageClassification
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds 2 new classes to the library, `SiglipForImageClassification` and `CLIPForImageClassification`.

This makes it easier to fine-tune SigLIP/CLIP for image classification, as otherwise people had to manually write a class based on SiglipVisionModel/CLIPVisionModel with a head on top. Given that SigLIP and CLIP are among the best vision encoders out there, it makes sense to add them.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28952). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I assume the "build PR documentation" check is failing but unrelated to this PR, so I can merge?

@NielsRogge Yes, if you look on main you can see that the build is failing for lots of (unrelated) PRs at the moment. I'm happy for you to merge as all the other CIs are passing. 

Hi @NielsRogge and @amyeroberts,

Thanks for your contribution to the community @NielsRogge! I am wondering if `"do_rescale": true` is necessary for `SiglipImageProcessor`? Let me know if I miss anything!

Also, did you encounter the same problem as in https://github.com/huggingface/transformers/issues/28968?

Thanks!

Best,

@zhjohnchan `do_rescale=True` is necessary if the input images are in the range `[0, 255]`. If they've already been rescaled to `[0, 1]` you can set it to `False`. 

@amyeroberts Thank you so much! Just realized it should be rescaled for all Transformers' preprocessors.

@zhjohnchan I haven't encountered the same problem, made a fine-tuning notebook for SigLIP here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SigLIP/Fine_tuning_SigLIP_and_friends_for_multi_label_image_classification.ipynb.

@NielsRogge Got it! Thank you so much!

Hi, I am the one who proposed to add a CLIPForImageClassification. Now I am back to contributing with it. 
I see "Build documentation" failure. How can it be fixed?

HI @Andron00e, thanks for suggesting it, it's now available here: https://huggingface.co/docs/transformers/en/model_doc/siglip#transformers.SiglipForImageClassification

Thank you! I'll try to fix minor bugs with CLIPForImageClassification and place its code into SigLIP's directory.
Download reconfiguration
Author: LOVE-YOURSELF-1
Pull request: no

### Feature request

Download reconfiguration

### Motivation

Separate out the download of the pretrained function for model、configuration and tokenizer.


### Your contribution

Do you think it is necessary？

Comments:
Hi @LOVE-YOURSELF-1, thanks for opening this feature request! 

If you want to download a specific file from the hub, you can use `hf_hub_download` from `huggingface_hub`. 

If you are using the auto classes, they will only download what's necessary. For example: 

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
```

will only download the files for the tokenizer, not the model weights. 

If we have multiple download sources，do you think it is necessary to bulit a new class named downloader?

@LOVE-YOURSELF-1 Could you explain what you mean in more detail? What do you mean by 'multiple download sources'? 

https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L3258
Can we rewrite the code about download（config，tokenizer and model）?
rebuilt a class named downloader, which can accomplish download, (from local and hugging hub)

@LOVE-YOURSELF-1  There's a plan to harmonise how e.g. loading and downloading shards is handled across libraries which will also account for e.g. local versus downloading from the hub. 

I don't see the reason for having a downloader class which handles downloading these all files - it's bundling together responsibilities making the scope of the class ill-defined. What problem would this be solving? 

As you said, i want to integrate downloading and loading. It can be rewritten in downloader.py.
Can you give me some suggestion or idea?

@LOVE-YOURSELF-1 Any contributor is welcome to open a PR to demonstrate and new feature or refactor. As this isn't something we or the community are requesting, it's up to you to define the feature you'd like to add. Once there's a clear diff or feature description, then we'll be able to help.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
LLM fine-tuning with deepspeed
Author: vallabh001
Pull request: no

I was trying to fine-tune llama2 by referring this [blog](https://www.philschmid.de/instruction-tune-llama-2l), but the training time is very high (in days). So, I was thinking of using deepspeed optimization for the training process. However, there is no proper documentation for fine-tuning llm's using deepspeed. 

I executed the command below to start training, but encountered an error. I have single A100 40gb GPU.
`torchrun --num_gpus=1 --nnodes 1 --nproc_per_node 1 llm_training.py --deepspeed "ds_zero2_no_offload.json" --ddp_find_unused_parameters False
`

Error

`RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 447 with name base_model.model.model.layers.31.mlp.down_proj.lora_B.default.weight has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.`

Comments:
Hi @vallabh001, thanks for raising an issue! 

So that we can best help you, please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide:
* The full error traceback
* A minimal code snippet we can use to reproduce the error. In this case, we can't help without knowing what's in `llm_training.py`
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output

There's the Alignment Handbook which uses Deepspeed to train any decoder LLM like Llama or Mistral on multiple GPUs: https://github.com/huggingface/alignment-handbook. It includes scripts for both supervised fine-tuning (SFT) and human preference fine-tuning using DPO (direct preference optimization): https://github.com/huggingface/alignment-handbook/tree/main/scripts.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.