noamaanMulla-03/transformers-issues huggingface dataset:
Fix optimum.quanto quantization call in cache_utils
Author: w3rew
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
Fixes call to optimum.quanto.quantized_weight in QuantoQuantizedCache, which currently lacks `scale` and `shift` parameters and thus fails. This was introduced by https://github.com/huggingface/transformers/commit/cac4a4876b5c263e51b1c0e8887f35cfb6f266b1 when migrating to optimum.quanto I think.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@SunMarc



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I've updated the PR [here](https://github.com/huggingface/transformers/pull/34750) so we fix it quickly ! I've added you as co-author @w3rew !
Update llava.md
Author: LysandreJik
Pull request: yes

Closes https://github.com/huggingface/transformers/issues/34680

Comments:
cc @zucchini-nlp 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34749). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

No worries, thanks for the review!
Add functionality for deleting adapter layers in PEFT integration
Author: itsskofficial
Pull request: no
Labels: name: Feature request

### Feature request

This request aims to introduce functionality to delete specific adapter layers integrated with PEFT (Parameter-Efficient Fine-Tuning) within the Hugging Face Transformers library. This would enable users to manage memory and computational resources more efficiently by unloading adapters that are no longer needed during model inference or fine-tuning.

### Motivation

This feature request addresses scenarios where users load multiple adapters during fine-tuning or inference but need the ability to selectively unload adapters without reloading the entire model. This enhancement is crucial for optimizing performance in memory-constrained environments and enhancing the flexibility of adapter management within Transformer models.

### Your contribution

Yes, I have submitted a PR #34650 

Comments:
cc @benjaminbossan, WDYT?

Yes, makes sense, I reviewed the PR. See also [this previous comment](https://github.com/huggingface/transformers/issues/32933#issuecomment-2379522352).
Remove FSDP wrapping from sub-models.
Author: eljandoubi
Pull request: yes

# What does this PR do?

Fixes #34113 

## Who can review?

Library:

- trainer: @muellerzr and @SunMarc


Comments:
@SunMarc @muellerzr Did you get a different result than I did?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34452). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc I migrated to `unwrap_model`.

Please rebase this PR on main in order to pass the CI @eljandoubi ! 

@SunMarc @LysandreJik @muellerzr Is there any update on the pull request?

We were on a company wide offsite! merging as they all approved 🤗
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP is not working with the Trainer
Author: eljandoubi
Pull request: no
Labels: name: trainer

name: bug

name: PyTorch FSDP

### System Info

<img width="896" alt="image" src="https://github.com/user-attachments/assets/67c71759-adae-46d9-aa41-3fecf8d28684">
acc_cfg.yml:

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: true
fsdp_config:
  fsdp_activation_checkpointing: true
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch: NO_PREFETCH
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: true
  fsdp_offload_params: true
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_process_ip: 0.0.0.0
main_process_port: 0
main_training_function: main
mixed_precision: bf16
num_machines: 3
num_processes: 24
rdzv_backend: etcd-v2
same_network: false
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

accelerate launch --config_file acc_cfg.yml train.py $TRAINING_ARGS
the train.py is any training script that train using transformers.Trainer
$TRAINING_ARGS are the TrainingArguments plus some path to data
<img width="907" alt="fdsp_trans" src="https://github.com/user-attachments/assets/f9228d6c-3bfa-44b7-8c79-20f4793cfb5f">

### Expected behavior

Train Paligemma model with FSDP and have PaliGemmaMultiModalProjector wrapped.

Comments:
cc @muellerzr and @SunMarc 

Hi! Can you please show your entire script you are running?

Same problem in case of peft LORA with NO_WRAP.  @muellerzr  @SunMarc 
<img width="890" alt="image" src="https://github.com/user-attachments/assets/7f390209-7872-494e-ba47-5e2e4955fc4c">


@muellerzr When `auto_find_batch_size=True`, the `find_executable_batch_size` function runs ` _inner_training_loop` multiple times. Since `self.accelerator.prepare` wraps (apply auto policy that is undetectable via `self._wrap_model`) `self.model` before the CUDA out-of-memory error is thrown, the wrapped `self.model` in the next iteration causes the FSDP auto-wrapping error.
🧼 remove v4.44 deprecations
Author: gante
Pull request: yes

# What does this PR do?

See title :)

@zucchini-nlp -> video llava changes
@SunMarc -> all other changes

Comments:
> since the changes will invalidate many models on the hub that are not owned by us.

We should then support it for as long as possible :) If it is critical to any popular model, I would not even set a deprecation version target at all. Leave the exception without target, and set a target when it blocks you from doing something OR if it is causing many issues. An example of this is supporting `model.config` modifications to parameterize `generate` -- it is super inconvenient for maintenance, but many examples/models rely on it. 

If it is only used in some niche models, and applying the deprecation (raising exception) is a significant win for transformers, then I would set at least 6 minor versions.

LMK which one is it, and I will update the warning.

I hope you meant `warning`, not `exception`. I don't think we should be raising exception unless at least the official checkpoint is updated accordingly. 

I would say this particular one is a niche model, given download stats from the hub. And most older llava and BLIP models may be outdated fast as we get newer and better releases each month. But I think we still can't enforce users to do something by elevating this to an Exception. My preference is to keep it for 6 minor versions as a warning, and try to update official models on the hub until that. We have write access to all of them afaik except for Video-LLaVA so updating will be straightforward

> I hope you meant `warning`, not `exception`

@zucchini-nlp haha yes, `warning`! In essence, allowing the behavior.

@zucchini-nlp @SunMarc PR comments addressed, LMK if they look good to you :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34245). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp done ✅ 

To pass the CI, you need to rebase the PR @gante  ;)
The support of `Mllama` in AutoModel
Author: HanGyeol-Yoo
Pull request: no
Labels: name: Feature request

### Feature request

The `AutoModel.from_config` does not work with Mllama (MllamaConfig, MllamaVisionConfig). I would like to request the ability to use Mllama through `AutoModel`.

### Motivation

There are many codes written to dynamically load models using `AutoModel`. It would be great if `AutoModel` could support Mllama to accommodate these codes.

### Your contribution

not yet.

Comments:
Hi, I could contribute to this feature if there will be approval from core team

cc @qubvel 

Hi @HanGyeol-Yoo! Thanks for the issue, for mllama (and other VLMs) we have `AutoModelForVision2Seq` auto class, so you can load different VLMs as easily as:

```python
from transformers import AutoModelForVision2Seq

checkpoint = "meta-llama/Llama-3.2-11B-Vision-Instruct"   # or "llava-hf/llava-v1.6-mistral-7b-hf"
model = AutoModelForVision2Seq.from_pretrained(checkpoint)
```

please see the full list of supported architectures here

https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/auto/modeling_auto.py#L739-L762
add `norm_topk_prob` argument on MixtralConfig
Author: bzantium
Pull request: no
Labels: name: Feature request

### Feature request

add `norm_topk_prob` config which is to decide whether to normalize router weights or not.
ref: [qwen2_moe](https://github.com/huggingface/transformers/blob/eb811449a2389e48930c45f84c88fd041735cf92/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py#L681), [deepseek_moe](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/521d2bc4fb69a3f3ae565310fcc3b65f97af2580/modeling_deepseek.py#L315)

### Motivation

I want to use Mixtral architecture but without normalizing router weights after topK.

### Your contribution

add config for norm_topk_prob to make possible to turn off.

Comments:
Beam search and greedy search set inconsistent eos_token
Author: user799595
Pull request: no
Labels: name: bug

name: Generation

### System Info

transformers: 4.44.1
Platform: Linux-5.15.0
Python: 3.10.6
PyTorch: 2.4.0+cu121

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import transformers
import torch

num_beams = 2
model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit"
model = transformers.AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="cuda")
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you? Answer briefly."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False)
tokenized_prompt = tokenizer(prompt, return_tensors="pt").to("cuda")
generation = model.generate(**tokenized_prompt, num_beams=num_beams, max_new_tokens=100)
results = tokenizer.batch_decode(generation, skip_special_tokens=False)[0]
print(results)
```

### Expected behavior

I would expect the eos_tokens used to be consistent.

With `num_beams = 2` (beam search)

> Arrr, me hearty! I be Captain Chat, the scurviest pirate chatbot to ever sail the Seven Seas o' the Interwebs!


With `num_beams = 1` (greedy search)

> Arrr, me hearty! I be Captain Chat, the scurviest chatbot to ever sail the seven seas... er, chat the digital seas!




Comments:
I think I found the culprit, according to https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit/blob/main/generation_config.json#L4 , since there are 2 eos tokens the model can either generate token 128009 or 128001.
Weird thing is that according to https://huggingface.co/unsloth/llama-3-8b-Instruct-bnb-4bit/blob/main/special_tokens_map.json#L10 , the eos token should be `
` and not both of them so i'm not sure what to make out of this, i'll continue looking at this at my spare time, but I wanted to report this first.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
_prepare_4d_causal_attention_mask mask inversion should work boolean masks
Author: JackCai1206
Pull request: no
Labels: name: Feature request

### Feature request

https://github.com/huggingface/transformers/blob/0fdea8607d7e01eb0e38a1ebeb7feee30a22f0cf/src/transformers/modeling_attn_mask_utils.py#L332

Here we can just assume the user provides a tensor of 0s and 1s right? In that case, the inversion can be done like `inverted_mask = ~(attention_mask.to(torch.bool))`

### Motivation

There are two advantages I found with doing the conversion assuming the input is bool
* Its slightly faster 
* When using a cached huggingface dataset to generate the mask, it is way more space-efficient to store the masks as bool type. 

### Your contribution

`inverted_mask = ~(attention_mask.to(torch.bool))` 

Comments:
cc @ArthurZucker 

I would need to check where this is used, but it's not used everywhere (especially llama like) cc @zucchini-nlp we saw a recent comment about that! 
WOuld you like to open a PR let's see if this works? 

Yeah, I think soon we will not be using that anymore and rely on `update_causal_mask` method similar to llama. The comment we saw was about accepting attn as boolean type for `forward()` call, and that will definitely break things in VLMs (maybe not LLMs). I guess this issue is quite different but we can see how it will fit :)

I want to +1 this since it caught me off guard. I was passing a boolean mask which worked on CPU since it used `scaled_dot_product_attention` which supports both boolean and inverted masks. Switching to GPU caused me to switch attention implementations and lose that behavior silently.

There used to be an error for this case, but it was removed:
https://github.com/huggingface/transformers/pull/32227/files#diff-0ebada593d7651a4bd67df9568471123d78edee35a2f0b8bb44751a192e5a4deL777-L781

Either adding it back, or adding support for boolean masks would be great.
[Multiple errors] Maintainablity video-text-to-text in transformers?
Author: SoyGema
Pull request: no
Labels: name: bug

name: Vision

name: Video

### System Info

transformers==4.45.2

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### Context
Hello there! Im going through the [video-text-to-text content](https://huggingface.co/docs/transformers/tasks/video_text_to_text) and I am experiencing a series of issues with broken links, unexistent content and errors. As they all come from the task page initially, decided to list them here as a hole possible maintenance issue, but please feel free to decompose if you find that this works best for you

#### Listed issues in Video-text-to-text page


1. [Link broken] Unaccesible Codelab . When trying to access to both the colab  (both on pytorch and tensorflow ) and amazon SageMaker Studio Lab, seems to point to an existing path , probably you changed it , and currently the notebook is not under your examples (Images above)
<img width="1386" alt="Captura de pantalla 2024-10-10 a las 10 31 57 1" src="https://github.com/user-attachments/assets/426de62f-3e95-4c61-825a-b46b858b0a8f">


![Uploading Captura de pantalla 2024-10-10 a las 11.02.17.png…]()


3. [Error in code] Sent a quick PR for this, missing an import #34058 

4. [Link broken] link at the end of the page , in the text `To learn more about chat templates and token streaming for video-text-to-text models, refer to the [image-text-to-text]`(https://huggingface.co/docs/transformers/image_text_to_text) task guide because these models work similarly. , when clicking , it goes to 2 links resulting in a 404 error. 
<img width="1147" alt="Captura de pantalla 2024-10-10 a las 10 32 11" src="https://github.com/user-attachments/assets/f9445411-d9ab-4d5e-a35d-bc7a5a9d9302">

<img width="1155" alt="Captura de pantalla 2024-10-10 a las 10 32 22" src="https://github.com/user-attachments/assets/3c4afb8e-66a7-4539-8d0e-675714f1e9c0">

Is this somehow similar to what happened in #24497 ??

Thanks for having a look to this ! 
Keep up the good work and have a nice day!


### Expected behavior

Links working
Code working


Comments:
cc @merveenoyan maybe? :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add support for OLMo November release
Author: 2015aroras
Pull request: no
Labels: name: New model

### Model description

An updated OLMo model will be released in November. The new model has a few small architecture changes compared to the existing model in transformers:

- RMSNorm is used instead of standard layer norm.
- Norm is applied to attention queries and keys.
- Norm is applied after attention/feedforward rather than before.

It would be great to have it integrated in transformers from day 1.

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

OLMo code repo: https://github.com/allenai/OLMo
Authors include @2015aroras 
Weights are not yet available, will be made available at release.

Comments:
Hi @2015aroras , 

I would like to contribute to this new model. Can I be assigned to this issue.
Thanks.

Regards

Thanks @huzaifa-jawad367 for the interest! Unfortunately I've already done most of the implementation for releasing this model on transformers, so it might be redundant to have us both work on it.

If you're interested in contributing to this model's release and don't mind contributing to another library, then implementations into llama.cpp is currently available I think (let me know!). If you want to work on an OLMo-related model in transformers and don't mind working on multimodal, the Molmo 1B seems to be up for grabs (other sizes are in progress here https://github.com/huggingface/transformers/pull/33962). Otherwise, if you want to do a text model in transformers then there might be something in the issues list but nothing OLMo-related to my knowledge.
fix Qwen2-VL usage in image-to-text pipeline
Author: Spycsh
Pull request: yes

# What does this PR do?

Fixes # [34769](https://github.com/huggingface/transformers/issues/34769)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Hello @zucchini-nlp . Could you please have a review?


Comments:
Thanks @zucchini-nlp  , update to `image-text-to-text` works:)
Fix broken link
Author: ofek
Pull request: yes

Page: https://huggingface.co/docs/transformers/main/en/tasks/image_classification
Current link: https://huggingface.co/docs/transformers/main/en/tasks/model_doc/vit
Correct link: https://huggingface.co/docs/transformers/main/en/model_doc/vit

Comments:
fix a typo bug where 'id2label' was incorrectly written as 'i2label' when reading config
Author: ZuoChenFttS
Pull request: yes

fix a typo bug in examples/pytorch/image-classification/run_image_classification_no_trainer.py where 'id2label' was incorrectly written as 'i2label' when reading pretrained config

# What does this PR do?
Fix a  typo bug in examples/pytorch/image-classification/run_image_classification_no_trainer.py
Where 'id2label' was incorrectly written as 'i2label' when reading pretrained config; 
This bug will cause the saved config to not properly reflect the relationship between tags and ids.

## Before submitting
- [ √] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:
- vision models: @amyeroberts, @qubvel

Maintained examples (not research project or legacy):
- PyTorch: See Models above and tag the person corresponding to the modality of the example.


Comments:
save `training_args` in another file format 
Author: not-lain
Pull request: no
Labels: name: Feature request

### Feature request

switch saving training arguments from `bin` format to another file extension. 

### Motivation

for security reasons `bin` format is deemed unsafe as marked by huggingface's new security system.
![image](https://github.com/user-attachments/assets/03d9c511-716e-4d15-8a2d-38d9a108e289)

### Your contribution

I'll try submitting a pr, but if anyone wishes to submit their own feel free to do so

Comments:
Hey @not-lain! That sounds like a good idea indeed. Pinging @muellerzr; Zach, Lain has offered to do a PR to help update this.

Any format in particular you think would be good here?

In a sense, `.safetensors` are basically json files, so maybe we can use them, else we can use the more simplified way and store the parameters in `.json` files directly.
whatever works best