noamaanMulla-03/transformers-issues huggingface dataset:
Fix method name which changes in tutorial
Author: andimarafioti
Pull request: yes

The method `model_download_tool` was called `model_download_counter` earlier in the tutorial, this raises an error when following the code.

# What does this PR do?

This PR fixes a small typo in the documentation. I was following the code here and realized that the method had different names during the tutorial but always refers to the same thing.


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @


Models:

- text models: @ArthurZucker

Documentation: @stevhliu

 -->


Comments:
LLaMA RoPE precision with bf16 model
Author: marviss
Pull request: no

### System Info

- transformers == 4.38.1
- python == 3.10.13
- torch == 2.2.0

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using the `bf16` argument for the trainer, the resulting `freqs` tensor is also in `bf16` precision, although the operands are explicitly cast to `float32`:

From https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L126:
```
inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1)
position_ids_expanded = position_ids[:, None, :].float()
freqs = (inv_freq_expanded @ position_ids_expanded).transpose(1, 2)  # ---> bf16
```

### Expected behavior

This produces a different output compared to [v4.37.2](https://github.com/huggingface/transformers/releases/tag/v4.37.2), where the resulting `freqs` tensor is `float32`, obtaining different position embeddings for a given sequence length.

From https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/models/llama/modeling_llama.py#L142:

```
t = torch.arange(self.max_seq_len_cached, device=device, dtype=self.inv_freq.dtype)
freqs = torch.outer(t, self.inv_freq)  # ---> float32
```

Is using `bf16` precision for `freqs` with `bf16` models the expected behaviour?

Thank you!

Comments:
Hi @marviss 👋 

We are aware of this issue -- this is a duplicate of #29285, let's keep all discussion there :)
Fix UDOP dtype issue
Author: Rocketknight1
Pull request: yes

The UDOP processor uses the LayoutLMV3 image processor, but this results in dtype errors in the CI. Trying to fix it in this PR!

Comments:
cc @NielsRogge UDOP seems to be failing in the CI because the image type is `uint8`, which causes an overflow error when it's scaled by a negative number during normalization. I'm guessing this means we need to set the dtype to be `float32` there, but I'm not sure exactly where to make the change.

Figured it out, the cause is:

- `do_resize` and `do_rescale` both `True`
- Specific `PIL` version

Because `PIL` is used for the resize, the output of the resize can sometimes by `np.uint8`, the standard image format, rather than `np.float32`, which is necessary once data is being rescaled.

cc @NielsRogge @LysandreJik for review!

cc @qubvel or @molbap for CV review as well

I'd appreciate a quick review on this one, since it's blocking other PRs, and it's only one line!

Hey @Rocketknight1, sorry about that, quick review to unblock: I'd be more pro-having the modification within the definition of `self.resize`, in order to keep the pattern consistent with the other classes. 
You could even catch the PIL version causing issues, parse it with `packaging.version`, and if it's above/below the version causing issues do the dtype switch in that case, right? + commenting on why this is needed!

@molbap I think it's simpler to just do `.astype(np.float32)`, firstly because I can't figure out exactly which version in the CI triggers this issue, and secondly because `.astype(np.float32)` on a `float32` array [does not copy or modify the array](https://numpy.org/devdocs/reference/generated/numpy.ndarray.astype.html) if we use `copy=False`, so there's no performance penalty to just calling it anyway.

Still, I'll move it inside `self.resize()`!

Investigated further, and the issue is not where I thought! The cause is actually this PR to Numpy, which changes type promotion rules: https://github.com/numpy/numpy/issues/25639

In fact, `uint8` is the expected dtype and this hasn't changed. What **has** changed is that this line:

`rescaled_image = image * scale`

fails in cases where `image` is `uint8` and `scale` is a negative integer, which occurs in the UDOP tests. Previously, this would silently promote the dtype to `np.int16`, but this is no longer the case. The reason we don't see more problems is that multiplying a `uint8` array by a positive/negative **float** still silently promotes the array, and this is much more common. Only UDOP uses a negative **integer** scale in its tests.

This means the fix actually needs to go in `image_transforms.py`, one sec!

Pushed the fix - now we always manually upcast to `np.float64`. This was what happens silently 99% of the time anyway, but now we make it explicit, which also fixes the cases where promotion doesn't silently happen anymore.

Thanks @Rocketknight1 ! cc @yonigozlan here, on the negative scale x uint8 issue I mentioned last time (just FYI)

Thanks for flagging this! I'll also change the ProcessorMixin tests to make the type promotion more explicit in case they change it again :)
Correct check for SDPA in Vision Language Models
Author: zucchini-nlp
Pull request: no
Labels: name: Should Fix

name: WIP

name: Vision

### System Info

In current implementation of VLMs, the "_supports_sdpa" attribute checks and activates SDPA attention only for the language model. For example in [Llava](https://github.com/huggingface/transformers/blob/ae49b218c3d718df90d8e4a109016450fb8f0632/src/transformers/models/llava/modeling_llava.py#L159-L164)

It should also check and if available use SDPA attention for vision tower.
- CLIP SDPA has an open PR: https://github.com/huggingface/transformers/pull/30390
- SigLip SDPA is merged: https://github.com/huggingface/transformers/pull/31499 

We can raise a warning for composite models if only one part support sdpa, but other does not, and activate SDPA for the supported part. That waythe user knows what is happening in the background. 

### Verified models

- [ ] BLIP-2
- [ ] InstructBLIP
- [ ] InstructBLIPVideo
- [ ] KOSMOS-2
- [ ] LLaVa
- [ ] LLaVa-NeXT
- [ ] LLaVa-NeXT-Video
- [ ] VipLLaVa
- [ ] Video-LLaVa
- [ ] Idefics
- [ ] Idefics2
- [ ] PaliGemma

Comments:
Edited your issue to include a list of models to check ;) feel free to expand

Please consider add for specific Vsion Encoders, such as Siglip CLIP InternViT etc.
VLM: add more modularity
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As mentioned in https://github.com/huggingface/transformers/issues/33948, this PR simply refactors code a bit to make it more modular, Specifically we now will have special public methods for obtaining image/video features that users can easily overwrite if they want to modify the process. In any way this makes less code in forward and more standardization in API

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34175). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
T5 compile compatibilty
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Same as https://github.com/huggingface/transformers/pull/33754, I accidentally force pushed to wrong branch and the PR got closed 🙃 

Comments:
Will merge this later today if we are all okay. The comment for tokenization is added within code with a TODO for us, in case it is not a bug we can remove the comment later
The model's address is https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx。I don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you
Author: pengpengtao
Pull request: no
Labels: name: Feature request

### Feature request

hello，The model's address is [https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx。I](https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx%E3%80%82I) don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you

### Motivation

hello，The model's address is [https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx。I](https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx%E3%80%82I) don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you

### Your contribution

hello，The model's address is [https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx。I](https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx%E3%80%82I) don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you

Comments:
cc @xenova :)

Have you made any progress and how to reason
Is it possible to infer the model separately through encoder.onnx and decoder.onnx
Author: pengpengtao
Pull request: no
Labels: name: Feature request

### Feature request

Is it possible to infer the model separately through encoder.onnx and decoder.onnx

### Motivation

Is it possible to infer the model separately through encoder.onnx and decoder.onnx

### Your contribution

Is it possible to infer the model separately through encoder.onnx and decoder.onnx

Comments:
I hope I can defer to our ONNX specialist @xenova :smile: 

Hi @pengpengtao 👋 Which model are you trying to run? In general, yes, you can infer the model separately through encoder.onnx and decoder.onnx, with outputs from the encoder being used as inputs to the decoder (usually).

> Hi @pengpengtao 👋 Which model are you trying to run? In general, yes, you can infer the model separately through encoder.onnx and decoder.onnx, with outputs from the encoder being used as inputs to the decoder (usually).

hello，The model's address is https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx。I don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you

@xenova ,hello,Can you help me? The model's address is [https://huggingface.co/Xenova/nllb-200-distilled-600M/tree/main/onnx]。I don't know how to load encode.onnx and decoder.onnx, and successfully translate a sentence into another language. Can you help me write an inference code to achieve the translation effect through the encoder and decoder? thank you

Have you made any progress and how to reason
GenerationConfig parameters are deleted from the website on 4.52.2 and main
Author: eyalmazuz
Pull request: no
Labels: name: Documentation

name: bug

### System Info

opened previously on:

https://github.com/huggingface/hub-docs/issues/1460

### Who can help?

@stevhliu  

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When opening the documentation for [v4.52.2](https://huggingface.co/docs/transformers/v4.45.2/en/main_classes/text_generation) or [main](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation) about GenerationConfig, the first thing it shows it the ``from_pretrained`` method and not the various kwargs you can set for generation.


Image for example:
![image](https://github.com/user-attachments/assets/af542f9e-d56b-4312-94fb-ecac57be8d1e)


### Expected behavior

**Describe the expected behaviour**
When checking [v4.42.2](https://huggingface.co/docs/transformers/v4.44.2/en/main_classes/text_generation)
The first thing it shows is the "Parameters that control the length of the output"

Image for example:

![image](https://github.com/user-attachments/assets/f9eb5060-5f7b-4ce2-a344-18bc9e1f9391)

Comments:
cc @gante as well
bitnet support 
Author: Darshan2104
Pull request: no
Labels: name: Feature request

name: Quantization

### Feature request

ValueError: Unknown quantization type, got bitnet - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'eetq', 'hqq', 'fbgemm_fp8']


### Motivation

I was trying to use "HF1BitLLM/Llama3-8B-1.58-100B-tokens" but i encounter above error.

#### Sample code snipt
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("HF1BitLLM/Llama3-8B-1.58-100B-tokens", device_map="cuda", torch_dtype=torch.bfloat16)    
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

### Your contribution

I can help.

Comments:
cc @MekkCyber :)


Hey @Darshan2104 are you installing transformers from source ? 
If not try doing that, just clone the repo and inside run `pip install -e .`

Hey @MekkCyber I've not installed it from source. I was using it in colab :)

you can install the latest version in colab then using : `pip install git+https://github.com/huggingface/transformers.git`

@MekkCyber It worked in colab Thanks, 
Question is why it is taking longer time to generate results ? It should ideally take shorter time compared to Base Llama model.

Hey @Darshan2104, For now I think it takes roughly the same time, It should ideally take a shorter time that's true, for now it's only using `@torch.compile` to unpack the weights, and do the matmul, but in the future we need to integrate some new kernels for int8xint2 like `Bitblas` or triton kernels. For `Bitblas` i was waiting for a bug to be resolved, I will look into that

@MekkCyber Thanks for update.
Sync video classification pipeline with huggingface_hub spec
Author: Rocketknight1
Pull request: yes


Comments:
cc @LysandreJik @Wauplin - this was a simple one, no Hub changes needed!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34288). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update PR templates
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR changes the PR templates. @MekkCyber will help with quantization maintenance from now ! 

Can you also add @MekkCyber to https://github.com/orgs/huggingface/teams/transformers team @ArthurZucker ? 

Comments:
I can't cc @LysandreJik 👀 


Done!
How to implement weight decay towards the pre-trained model?
Author: sedol1339
Pull request: no
Labels: name: Usage

name: Feature request

Hello, let me one question.

If using HF Trainer for supervised fune-tuning, how do I implement penalizing the distance between starting and current weights? This was shown to be effective in https://arxiv.org/abs/1706.03610

Comments:
There is no out-of-the-box option to specify weight decay against pretrained weights that I am aware of, but you could achieve it easily by adding the loss yourself.

For example, [here](https://huggingface.co/docs/transformers/en/training#training-loop) is an example torch training loop. You could patch it to include the differences between the initial parameters and current ones, square it, and sum to the total loss with a weight term.

> There is no out-of-the-box option to specify weight decay against pretrained weights that I am aware of, but you could achieve it easily by adding the loss yourself.
> 
> For example, [here](https://huggingface.co/docs/transformers/en/training#training-loop) is an example torch training loop. You could patch it to include the differences between the initial parameters and current ones, square it, and sum to the total loss with a weight term.

However, this is not `transformers.Trainer`, but instead a custom loop. So, with Trainer this is not achievable?

Looking at the available args for trainer it doesn't seem to be supported, but I am not 100% sure

This seems like a custom usage, I don't think we support this out of the box!

I think you can subclass and override the `compute_loss` method of `Trainer`, like [this example](https://huggingface.co/docs/transformers/v4.45.2/en/trainer#customize-the-trainer).

We also updated the API to support updating the `self.loss_function` of each models!
Mamba slow implementation datatype mismatch
Author: Adibvafa
Pull request: no
Labels: name: bug

### System Info

transformers==4.44.0

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Setup a PyTorch Lightning training with float16 precision. 
2. Setup `MambaConfig` with `use_mambapy=False`
3. Train `MambaForCausalLM` model

### Expected behavior

The expected behavior is training proceeds with no issues. However, this is the error I got:

Code: cache_utils.py, MambaCache class, update_conv_state method
```
conv_state[:, :, cache_position] = new_conv_state.to(conv_state.device)
RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and Half for the source.
```


I was able to fix the issue by changing `conv_state[:, :, cache_position] = new_conv_state.to(conv_state.device)` to `conv_state[:, :, cache_position] = new_conv_state.to(device=conv_state.device, dtype=conv_state.dtype)`

I will open a PR with the fix.


Comments:
cc @molbap too as I've seen you across some of the Mamba issues and PRs :) 

BTW when you train you should not be using the `cache_position` so not super sure I understand what's going on! Unless you are in eval model, which would mean the init dtyp was wrong 

> BTW when you train you should not be using the `cache_position` so not super sure I understand what's going on! Unless you are in eval model, which would mean the init dtyp was wrong

I was training using PyTorch Lightning, I didn't manually specify it to use `cache_position`. Is it unexpected that that chunk of code was ran in this setting?

Yeah, `use_cache` is probably set to `True`: https://github.com/huggingface/transformers/blob/21beb57558f90131439bf03e1da672d117777901/src/transformers/models/mamba/modeling_mamba.py#L603 is where we init the cache for mamba, and here is where you have the if else that calls https://github.com/huggingface/transformers/blob/21beb57558f90131439bf03e1da672d117777901/src/transformers/models/mamba/modeling_mamba.py#L248

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thank you @ArthurZucker @amyeroberts!
Fix: tensor of examples of the same length triggers invalid stacking
Author: pbelcak
Pull request: yes

Fix: tensor of examples of the same length triggers invalid stacking

# What does this PR do?

This PR fixes the issue that occurs when `DataCollatorForLanguageModelling` is used on whole tensors (e.g. `input_ids` being a long tensor of shape `(4, 2048)`). When this happens, `torch.stack` fails because it is expecting `examples` argument to be a tuple of tensors and not a tensor on its own.
Note that if the preprocessing step applied to the dataset (e.g. batched `dataset.map`) produces tensors of varying length, this occurs only in the very rare (and difficult to track down) special case when the collator happens to come across two tensors of exactly the same length. This is what occurred to me on the GSM8k dataset in roughly one out of every three runs.

@ArthurZucker

Comments:
Add post_process_depth_estimation to image processors
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

name: Vision

### Feature request

The image processors of depth estimation models could benefit from a `post_process_depth_estimation` method, similar to the `post_process_object_detection`,  `post_process_semantic_segmentation` methods.

This includes:

- [ ] DPT
- [ ] Depth Anything (and v2)
- [ ] ZoeDepth (soon: #30136) => this one requires some special postprocessing logic as shown in [this notebook](https://colab.research.google.com/drive/1lrH4IN3ipR3jstpmV-nAzsxoRRFM91mh#scrollTo=_orJsZFU7HG3)

### Motivation

People now need to manually interpolate the predicted depth to the original size of the image as seen [here](https://huggingface.co/docs/transformers/model_doc/dpt#transformers.DPTForDepthEstimation.forward.example).

The interpolation could be replaced by a simple `image_processor.post_process_depth_estimation` method which postprocesses the predicted depth and interpolates it to the original image size.

### Your contribution

I could work on this, but would be great if someone else can take this up.

Comments:
@NielsRogge, I can take this up!

Hi, if no one is working on this issue, I can take this! 

@ms42402 there's no active PRs, so feel free to tackle this if you'd like! 

hey anybody working on this? i would like to try and solve this issue

@anirudh2781998 As above, if there isn't an active PR open, then anyone is free to work on any issue/feature request
Qwen2.5 is ExecuTorch Compatible
Author: guangy10
Pull request: yes

# What does this PR do?

Qwen2.5 is compatible with `ExecuTorch`.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #33833
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker


Comments:
@ArthurZucker do you mind reviewing this PR?
Olmo is ExecuTorch Compatible
Author: guangy10
Pull request: yes

# What does this PR do?

`OLMo-1B-hf` is compatible with `ExecuTorch`.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #33840
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker


Comments: