noamaanMulla-03/transformers-issues huggingface dataset:
Add DINOv2 with registers (using diff)
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a follow-up of #31832 but uses the diff instead of rewriting from scratch.

Had 2 issues with the diff:

- [ ] the `drop_path` function was causing an issue. Applying the suggestion at [#31832](https://github.com/huggingface/transformers/pull/31832#issuecomment-2298687531) fixed it.
- [ ] the diff doesn't properly add `super().__init__()` to the Backbone class. I manually added it in the modeling file as otherwise there's an error.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32905). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker thanks for the review, addressed your comments. Note that there were still some issues with the diff converter (see my first message), but other than that this PR seems ready

Support for dinov2 with registers would be very much appreciated!  

#33248 will resolve all your issues 🤗 

I'll ask you to wait a tiny little bit as it will conflict with this work! 

> @ArthurZucker thanks for the review, addressed your comments. Note that there were still some issues with the diff converter (see my first message), but other than that this PR seems ready

Looks like Modular transformers: modularity and inheritance for new model additions https://github.com/huggingface/transformers/pull/33248 was merged a few hours ago!


Yep! Make sure to rebase! 


I think I did the wrong @ initially. @NielsRogge do you have time to rebase and update?  If not, we can give it a go. 

> > @ArthurZucker thanks for the review, addressed your comments. Note that there were still some issues with the diff converter (see my first message), but other than that this PR seems ready
> 
> Looks like Modular transformers: modularity and inheritance for new model additions #33248 was merged a few hours ago!



Hi, feel free to take over, it's probably easy as only one method needs to be overwritten.

Closing this one, will try out the new modular Transformers when I have time
add setter for trainer processor
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Following #32385, adding a setter to not break TRL!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33911). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unnecessary flash attention warnings
Author: alex-hh
Pull request: no
Labels: name: bug

name: Flash Attention

### System Info

latest version

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

init a model with attn_implementation=flash_attn_2 by passing a config to PretrainedModel.\__init__

torch.get_default_dtype will be passed to _autoset_attn_implementation, triggering the warning.

I know that the warning can be disabled by changing torch's default dtype - but it this the recommended way to cast models to bf16/fp16? if not then the warning shouldn't be raised.

this was previously discussed in #28052 but the warning still occurs

### Expected behavior

this shouldn't raise a warning when the dtype of the model is unknown

Comments:
Add support for `weights_only` flag when loading state_dict
Author: jerryzh168
Pull request: yes

Summary:
This is to enable loading a state_dict with wrapper tensor subclasses (used in torchao to for quantized weights)

Test Plan:
tested locally with torchao weights, also need https://github.com/huggingface/transformers/pull/32306:
```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import TorchAoConfig
from torchao.utils import benchmark_model
import torchao

DEVICE_TYPE = "cuda"

def init_model_and_benchmark(model_id, torch_dtype=torch.bfloat16, quantization_config=None):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    if quantization_config is not None:
        model = AutoModelForCausalLM.from_pretrained(model_id, device_map=DEVICE_TYPE, torch_dtype=torch.bfloat16, quantization_config=quantization_config)
    else:
        model = AutoModelForCausalLM.from_pretrained(model_id, device_map=DEVICE_TYPE, torch_dtype=torch.bfloat16, weights_only=False)

    # sanity check: run the model
    input_text = "What are we having for dinner?"
    input_ids = tokenizer(input_text, return_tensors="pt").to(DEVICE_TYPE)
    output = model.generate(**input_ids, max_new_tokens=1000)
    print(tokenizer.decode(output[0], skip_special_tokens=True))

    NUM_WARMUP = 1
    NUM_RUNS = 5

    if quantization_config is not None:
        torchao.quantization.utils.recommended_inductor_config_setter()

    model = torch.compile(model, mode="max-autotune")

    benchmark_model(model.generate, NUM_WARMUP, kwargs=input_ids, device_type=DEVICE_TYPE)
    print("running benchmark")
    results = benchmark_model(model.generate, NUM_RUNS, kwargs=input_ids, device_type=DEVICE_TYPE)
    return model, results

model_id = "jerryzh168/test-model"
torchao.quantization.utils.recommended_inductor_config_setter()
bf16_model, bf16_time = init_model_and_benchmark(model_id)
print(f"bf16: {bf16_time}")
```

Reviewers:

Subscribers:

Tasks:

Tags:


Comments:
I learned from @mikaylagawarecki that `weights_only` is going to be set to True by default in the future. but this flag could still be helpful if people are using older versions of pytorch and want to use torchao I think

Hi @ArthurZucker can you take a look at this PR?

please let me know if we want to add a test, but load test is a bit harder to write because it needs to upload some models like: https://github.com/huggingface/transformers/blob/abbffc4525566a48a9733639797c812301218b83/tests/test_modeling_utils.py#L930

Can you rebase on main and resolve the conflicts? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32481). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks for the review @ArthurZucker and @amyeroberts I just updated the PR, please feel free to merge when the CI is green

@jerryzh168 Could you run `make fix-copies` and push the changes? This should resolve the failing quality checks

thanks, updated, please take a look again @amyeroberts 

Thanks once more @jerryzh168
[feat] add possibility to give kwargs handler to accelerator in trainer
Author: laurentd-lunit
Pull request: yes

# What does this PR do?

At one point I wanted to change the `timeout` of `Accelerator`'s `init_process_group` and I couldn't find an easy way to access this property when using the `Trainer` since it handles the `Accelerator` instantiation internally but gives limited access to what arguments we can modify.

In this PR I added some fields to the `AcceleratorConfig` to allow passing arguments to construct the different `kwargs_handlers` available for `Accelerator` and also updated the `create_accelerator_and_postprocess` method to create and  inject the `kwargs_handlers` when instantiating the `Accelerator`.
I left out the `DistributedDataParallelKwargs` because it seems it's already handled internally in `Trainer`. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr and @SunMarc as this relates to `Trainer`.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Config: lower `save_pretrained` exception to warning
Author: gante
Pull request: yes

# What does this PR do?

Round 2, after #33886 -- TL;DR stops raising exceptions when saving a model config with generative parameters, if the user is not responsible for those parameters being there.

(copied from 33886)
```
In a recent PR (https://github.com/huggingface/transformers/pull/32659), we started raising exceptions if we were to store a model config with generative parameters. If we were saving a model, which saves the config, the generative parameters were moved to GenerationConfig before saving the PreTrainedConfig, working around the issue when saving a model.

However, a common use case was uncovered. If a user loads a PreTrainedConfig with generative parameters and tries to save it, the user has done nothing wrong and yet they see an exception. This PR lowers the exception to a warning in that situation, preventing the code from crashing :)
```

33886 would be my intended solution to the problem, but I ran into multiple implementation issues which I'll iron out in separate PRs 👀 This PR is a poor man's version of the solution, to unblock others: simply lower the exception to a warning. 

(cc @regisss )



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix Joao!
The examples in the examples directory are mostly for fine-tuning pre-trained models？how to trian from  scratch
Author: zc-Chao
Pull request: no
Labels: name: New model

### Model description

no 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
The examples for causal LM and masked LM can also be used for training from scratch!

+1
"Automatic Speech Recognition with Transformer - Keras"
is from scratch and works well.
Where is a huggingface example?
Mixtral: Reduce and Increase Expert Models
Author: ryan-minato
Pull request: no
Labels: name: Feature request

### Feature request

Add methods to MixtralSparseMoeBlock, MixtralDecoderLayer, and MixtralModel for reducing (and enlarging) the number of expert models.

Implement a mechanism to decrease the number of expert models by removing corresponding rows in gate weights. This should enable the removal expert models by id.

https://github.com/huggingface/transformers/blob/1a585c1222a56bcaecc070966d558d4a9d862e83/src/transformers/models/mixtral/modeling_mixtral.py#L688-L710

### Motivation

This will allow scaling down or up the model size from a pre-trained model while preserving existing weights, eliminating the need to retrain from scratch.

### Your contribution

I am willing to contribute a pr, but need a few time. 

I would like to know if such a PR is likely to be accepted, before I start working on it.

Comments:
cc @younesbelkada @ArthurZucker 

Hi @minato-ellie 
This could makes sense yes, do you know if it has been empirically proven that decreasing the number of experts do not lead in performance degradation? I also wonder if this does not make the HF mixtral implementation "too modulable" which goes against transformers philosophy

@younesbelkada 

I created #28092 .

I tried it and it seems that I can pass it forward using the new model, but I can't output the results. 
I didn't try to finetune it because I couldn't use GPU over the weekend.

Also, someone have created mixtral by merging 4 mistral models as experts and it seems to work well.
https://huggingface.co/chargoddard/mixtralnt-4x7b-test

I'll try to finetune it when I can use gpu, and let you know the results.

Ok perfect, let us know how it goes!
Clarification on Classification Token.
Author: Saeed11b95
Pull request: no

https://github.com/huggingface/transformers/blob/0a7af19f4dc868bafc82f35eb7e8d13bac87a594/src/transformers/models/layoutlmv3/modeling_layoutlmv3.py#L1349

In the above line the first token from the output of transformer is used to do classification logit computation. I am confused because in the source code the learnable classification token is not at the zeroth index. The cls_token is concatenated at the start of image patch tokens 
visual_tokens = torch.cat([cls_token, visual_path_embeddings])
and to create transformer input the text+bbox inuputs and image tokens are concatenated as follows,
transformer_inp = torch.cat([text_embeddings, visual_embedding])
this means that my classification is at index 512 considering that we put limit of 512 tokens on text inputs.
This is just for clarification, using first token for classification also does a fine job.
Thanks

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

I think you can ping @NielsRogge as he contributed this model! 
Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
latest 44.4.2 doesn't support falcon_mamba
Author: thistleknot
Pull request: no
Labels: name: bug

### System Info

python 3.10
rocky linux 9

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pip install transformers==4.44.2

```
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 993, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 695, in __getitem__
    raise KeyError(key)
KeyError: 'falcon_mamba'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/text-generation-webui/modules/ui_model_menu.py", line 231, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
  File "/home/user/text-generation-webui/modules/models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
  File "/home/user/text-generation-webui/modules/models.py", line 155, in huggingface_loader
    config = AutoConfig.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code)
  File "/home/user/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 995, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `falcon_mamba` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
```

### Expected behavior

mamba falcon loads

Comments:
Afair it hasn't been included in any version yet. It most likely will be released with the next major version 4.45 (so in max 1-2 weeks).

For now you could install from source. Should be something like:
`pip install git+https://github.com/huggingface/transformers.git`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

v4.45 was released, so I'm closing this (thanks, as always, @vasqu!)
Fix distil whisper segment computation
Author: ylacombe
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Distilled versions of Whisper couldn't return segments because of how they computed past key values. This introduces a test that failed before introducing the fix.

cc @eustlb @LysandreJik 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33920). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

They also tried with the latest transformers version,and used the ASR pipeline
Ignore keys on `validate_rope`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33401. Adds ignore_keys argument when checking received keys and adds `mrope_section` in qwen2-vl config file as a key we should ignore

I'll open PRs on the hub to change rope type from mrope to default, but until ten decided to leave the comment. For curious users who might open up more issues trying to find why we do manual modifications on rope type :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33753). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).
[`PR run-slow`] 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update workflow to run on latest torch, not torch docker installed 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33939). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add a section on writing tool templates to the chat template docs
Author: Rocketknight1
Pull request: yes

The chat template docs have an extensive section on using tool templates, but they're missing good guidance to template authors. I added a section to help them, and also so there's a point of reference for the API we expect tool templates to follow.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33924). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'm happy with this now - merging without core maintainer review since it's a pure documentation change
`resize_token_embeddings` in NLLB leading to empty outputs
Author: bhavitvyamalik
Pull request: no
Labels: name: Good Second Issue

name: Good Difficult Issue

name: bug

### System Info

- `transformers` version: 4.42.3
- Platform: Linux-4.18.0-513.11.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: yes

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")
tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", additional_special_tokens=[f"code_{i}" for i in range(18)], use_fast=True)
model.resize_token_embeddings(len(tokenizer))
```

After resizing, generation using an official example:
```
article = "Şeful ONU spune că nu există o soluţie militară în Siria"
inputs = tokenizer(article, return_tensors="pt")

translated_tokens = model.generate(
    **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("deu_Latn"), max_length=30
)
tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
```
Output is: `'t t t t t t t t t t'`

### Expected behavior

Generation should work without any errors. One interesting thing to note here is if I add just 2 new tokens, it works fine.

Comments:
Hey! 🤗 This is not a bug and is expected. It's a duplicate of #29899 (there are many others). https://nlp.stanford.edu/~johnhew/vocab-expansion.html is the solution you are looking for.

cc @LysandreJik I think it would make sense to do this by default no? 

Yes if this error often we should find a better way to handle this. I'll write this as a good second issue in case a community member wants to take an initial stab at it.