noamaanMulla-03/transformers-issues huggingface dataset:
bnb_8bit, gptq generation tests updated
Author: poedator
Pull request: yes

## What does this PR do?
Updating tests to (presumably) adjust for minor packages/models changes elsewhere. 
See Issue #26533  for details.
 
## Who can review?
- generate: @gante
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

## Testing
I tested this PR locally with RUN_SLOW=1 and two GPUs.  Only ran quantization tests. 
The generation tests now pass, nothing is broken (unless broken before, see the issue for the [remaining problem in model class mismatch](https://github.com/huggingface/transformers/issues/26533#issuecomment-1742764417)).

Comments:
in issue discussion https://github.com/huggingface/transformers/issues/26533 @younesbelkada suggested that this could be caused by Ampere vs Turing architecture differences. 
While this fix is not really pressing, it may make life easier for developers with Ampere cards, so that they see fewer failed tests and focus on important issues.

@poedator thanks! I agree this would make life easier for developpers
Do you know if there is a way to check with any torch utility method (or whatever) the architecture of the GPU? If that's the case you could maybe convert this PR to a simple checks that keeps the current expected generations for turing GPUs and yours for ampere. What do you think?

> @younesbelkada wrote: 
> Do you know if there is a way to check with any torch utility method (or whatever) the architecture of the GPU?

there is `torch.cuda.get_device_capability()` which returns `8.x` for Ampere and `7.x` for Turing. [More details here at nVidia site.](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities). But separating tests by compute level may be an overkill. How likely is it to have an error in generation code ran on Ampere that gives some output expected from Turing? I see that this package already allows [multiple correct answers in GPTQ tests.](https://github.com/huggingface/transformers/blob/6824461f2a35546a3d781fe60576e00f6db7bedf/tests/quantization/gptq/test_gptq.py#L87-L93). Besides, there may be more factors than just hardware, like torch version. With too few results seen bu myself in the generation tests, I am hesitant to claim that mine are the only  correct ones for Ampere cards. Could you still consider this PR as is?  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix Gemma2 sliding window attention
Author: pcuenca
Pull request: yes

Fixes https://huggingface.co/google/paligemma2-10b-pt-896/discussions/4

Not visible on the 448 model because the input sequence is << than the sliding window.

This was lost in the [big attention refactor](https://github.com/huggingface/transformers/pull/35235). I'm not sure if we're supposed to deal with it differently.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35691). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @pcuenca! Thanks for looking into this! I noticed the issue yesterday and opened https://github.com/huggingface/transformers/pull/35681 already (there is also a logic issue in the masking and slicing!)

Yes, saw it but was not sure if it fixed the same issue as the title mentioned Flash Attention and I saw no sliding_window arg being passed, IIRC (I see it's there now). Closing then!

Yes, I added it today! Thanks for double-checking!!
Support LLMs With No Image Placeholder Embedding in LLava-based Models
Author: alex-jw-brooks
Pull request: no
Labels: name: Feature request

name: Multimodal

name: VLM

### Feature request

Currently, llava-based models, e.g., llava-next, will throw `IndexError: index out of range in self` from the LLM embedding if the LLM does not contain the image embedding. However, at inference time, the embedding value isn't actually used, because the indices corresponding to the image token will be overwritten by the image features.

Other inference engines, e.g., vLLM, separately mask out the text and multimodal embeddings and merge them together (e.g., [here](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/utils.py#L377)). This prevents such indexing errors if the image token is only part of the tokenizer vocabulary, and not part of the encapsulated language model's embedding vocab.

This can be fixed on the model side by resizing the token embeddings to add the image token to the LLM, but it would be nice to take a similar approach in transformers to allow use of models that don't have the image token in the LLM embedding vocabulary.

### Motivation

Fixing this will allow the use of llava-based models that don't have an embedding for the placeholder image token 😄 

### Your contribution

I am happy to submit a PR for this if the team is open to it!

Comments:
Exactly. Taking llava-next as an example, this feature was introduced in v4.41.0:
https://github.com/huggingface/transformers/blob/4c6c45ba138202f42582b5cea98126af87195a95/src/transformers/models/llava_next/modeling_llava_next.py#L750-L754
However, it was removed starting from v4.45.0.

Totally agreed that it would be a useful feature to have, a PR is welcome 🤗 

Great! I will work on this and open a PR in the near future 😄
Enhanced Installation Section in README.md
Author: egojoseph
Pull request: yes

This PR improves the Installation section in the README.md file by:
- Adding GPU setup instructions for TensorFlow and PyTorch.
- Including troubleshooting tips for Python version issues and missing dependencies.
- Separating OS-specific virtual environment activation commands for clarity.
- Providing links to relevant installation guides for backend frameworks.

These enhancements aim to improve the clarity and usability of the installation instructions for new users.



Comments:
cc @stevhliu!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35094). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @Rocketknight1 and @stevhliu, I noticed that this pull request requires workflow approval and a review to proceed. All checks have passed so far and I have updated the branch to sync with main. Could you kindly review and approve the workflows when convenient? Let me know if further changes are required. Thanks!"

We want the README.md to stay quite minimal, but those changes could be applied in the installation section in the docs!

Hi @LysandreJik, @Rocketknight1, @stevhliu and team,

Thank you for the earlier feedback. Based on your suggestion, I have migrated the detailed installation instructions from the README.md file to the installation.md section under docs/en.

**Summary of Changes: **
Virtual Environment Setup Instructions:

Steps for creating and activating virtual environments across macOS, Linux and Windows, with additional guidance for potential Windows-specific issues.

**GPU Setup Section: **
Optional instructions for GPU acceleration, including compatibility resources for PyTorch and TensorFlow, along with a command to verify GPU availability.

**Troubleshooting Guide: **
Addressing common installation issues like unsupported Python versions and missing dependencies.
I believe these updates provide clear and practical guidance for users while adhering to the minimalistic approach in the README.md file as suggested. Looking forward to your feedback and approval.

Best regards,
@egojoseph

Hi @stevhliu,
I have gone through all the comments and addressed each one. Thanks for your detailed feedback, it really helped improve the documentation! Let me know if there’s anything else to adjust.
 @LysandreJik, @Rocketknight1 and team,
Enhance DataCollatorForLanguageModeling with Configurable Token Replacement Probabilities
Author: mahdibaghbanzadeh
Pull request: yes

This pull request introduces enhancements to the `DataCollatorForLanguageModeling` class, providing greater flexibility for token replacement during masked language modeling (MLM). The key changes include:

1. **Configurable Replacement Probabilities**:
   - **`mask_replace_prob`**: Specifies the probability of replacing masked tokens with the `[MASK]` token (default: 80%).
   - **`random_replace_prob`**: Specifies the probability of replacing masked tokens with random tokens from the vocabulary (default: 10%).
   - The remaining masked tokens are left unchanged (default: 10%).

2. **Edge Case Handling**:
   - Properly scales `random_replace_prob` to the remaining probability after applying `mask_replace_prob`.
   - Includes validation to ensure the sum of `mask_replace_prob` and `random_replace_prob` does not exceed 1.

3. **Backward Compatibility**:
   - Default behavior mimics the traditional 80-10-10 rule for MLM token replacement.

### Examples of New Functionality
- **Default Behavior**:
  Replace 80% of masked tokens with `[MASK]`, 10% with random tokens, and leave 10% unchanged.
- **Custom Configurations**:
  - Replace all masked tokens with `[MASK]`:
    ```python
    mask_replace_prob=1.0, random_replace_prob=0.0
    ```
  - Replace all masked tokens with random tokens:
    ```python
    mask_replace_prob=0.0, random_replace_prob=1.0
    ```
  - Balanced replacement:
    ```python
    mask_replace_prob=0.5, random_replace_prob=0.4
    ```

### Additional Notes
- Updated docstrings to reflect the new configuration options.
- Added validations for probability values and enhanced edge case handling for robust training workflows.

This enhancement gives users greater control over MLM training configurations, catering to various pretraining and fine-tuning use cases.

Comments:
Thanks for the feedback! 
I updated the docstring and added the following tests:
1. **`test_probability_sum_error`**: Ensures an error is raised if `mask_replace_prob + random_replace_prob` is not within `[0, 1]`.  
2. **`test_all_mask_replacement`**: Verifies functionality when `mask_replace_prob=1`, ensuring all tokens are either the original token or `[MASK]`.

@mahdibaghbanzadeh this looks good now! Let me know whenever you're ready for final review and I'll ping a core maintainer

@Rocketknight1 Thanks, Please let them know to do the final review. 


cc @arthurzucker for core maintainer review!

hi @mahdibaghbanzadeh, and sorry for the Christmas delay! The core maintainers are pretty overworked at the moment, but I just did a final review and I'm happy with this. Let me know if there's anything you want to change before we merge!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35251). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hi @Rocketknight1 and happy New Year!
Thank you for letting me know. At this time, this looks fine and final from my end.
Fix typo in /docs/source/ja/model_doc/decision_transformer.md URL
Author: hiroaki222
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

This PR fixes a typo in the URL provided in the `decision_transformer.md` documentation. The incorrect URL (`decion-transformer`) has been corrected to `decision-transformer`.

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
RuntimeError: shape '[1, 3098, 6, 5, 128]' is invalid for input of size 12689408
Author: LuchenZhou
Pull request: no
Labels: name: bug

### System Info

conda create -yn duo python=3.10
conda activate duo

conda install -y git
conda install -y nvidia/label/cuda-12.4.0::cuda-toolkit
conda install -y nvidia::cuda-cudart-dev
conda install -y pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

pip install transformers==4.45.2 accelerate sentencepiece datasets wandb zstandard matplotlib huggingface_hub==0.25.2
pip install tensor_parallel==2.0.0

pip install ninja packaging
pip install flash-attn==2.6.3 --no-build-isolation

# LongBench evaluation
pip install seaborn rouge_score einops pandas

pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

We encountered a shape mismatch error while trying to reproduce Duo Attention. We tested versions 4.37 to 4.47, and the issue shifted from a RuntimeError: Boolean value of Tensor with more than one value is ambiguous to a RuntimeError: shape '[1, 3098, 6, 5, 128]' is invalid for input of size 12689408. We couldn't resolve the issue by changing the versions.

We also tried different models with the following commands:

huggingface-cli download togethercomputer/Llama-2-7B-32K-Instruct --local-dir Llama-2-7B-32K-Instruct
huggingface-cli download gradientai/Llama-3-8B-Instruct-Gradient-1048k --local-dir Llama-3-8B-Instruct-Gradient-1048k
huggingface-cli download gradientai/Llama-3-8B-Instruct-Gradient-4194k --local-dir Llama-3-8B-Instruct-Gradient-4194k
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.2 --local-dir Mistral-7B-Instruct-v0.2
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3 --local-dir Mistral-7B-Instruct-v0.3

However, none of these models worked. There was a previous issue suggesting that updating the transformer version could solve the problem, but we are still getting shape mismatch errors.

Could there be other packages that need to be updated as well?

### Expected behavior

A solution of RuntimeError: shape '[1, 3098, 6, 5, 128]' is invalid for input of size 12689408

Comments:
Hi @LuchenZhou, I can't actually find the code you ran to get this error, only the dependencies you installed. Can you make a simple reproducer script so we can validate it?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Clean-up composite configs
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Based on https://github.com/huggingface/transformers/pull/34410, I realized that BLIP models were the only ones failing a certain common test and the reason is that we manually set the `tie_word_embedding` from `text_config`. I think some config attributes, including `vocab_size` (successfully deprecated in VLMs) or `tie_word_embeddings` should be only attributes of text config. We should access them via `config.get_text_config()` whenever needed

Having an option to access through general composite config and through text config creates two sources of truth, and we might easily end up with situations like in the `check_config_arguments_init`. If one sets the attribute through `setattr`, it will not be different from the attribute in text config. 

So we expect users to set the attr in the respective config if they want it to be used correctly with transformers code. We could restrict setting these common attr in correct sub-config by `attribute_map` prob, but I thought it was too much 

Plus, clean up some `attn_implementation=config._attn_implementation` that were left, because it should be passed already as part of respective config

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34603). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Tests are passing, the `run-all` last time worked as well with a few new VLMs that needed a fix. Merging when CI turns green again
Add future import for Py < 3.10
Author: Rocketknight1
Pull request: yes

One of our files uses `X | None` style types, which requires `from __future__ import annotations` on versions of Python < 3.10. This PR adds the import

Fixes #35639

Comments:
Scanned the repo and found the same issue in `convert_olmo2_weights_to_hf.py`, so I made the fix there as well

cc @arthurzucker for review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35666). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix : Nemotron Processor in GGUF conversion
Author: MekkCyber
Pull request: yes

# What does this PR do?
Fixes an issue when converting models from gguf format for Nemotron models. In RMSNorm we should substract 1 from the weights, see https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py#L4666
The test `test_nemotron_weights_conversion_fp16` covers this and it passes with the fix.
## Who can review 
@SunMarc @Isotr0py 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35708). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typo in /docs/source/ja/model_doc/decision_transformer.md URL
Author: hiroaki222
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

This PR fixes a typo in the URL provided in the `decision_transformer.md` documentation. The incorrect URL (`decion-transformer`) has been corrected to `decision-transformer`.

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
cc @stevhliu
Adding warnings or errors when provided sequence length is bigger than config.max_position_embeddings
Author: Taha1506
Pull request: no
Labels: name: Feature request

### Feature request

Add a new ValueError when you pass input_ids larger than config.max_position_embeddings somwhere in the code.

### Motivation

In the [CLIPTextEmbeddings class](https://github.com/huggingface/transformers/blob/a1844a3209eb7e75582684809203bc189931a90c/src/transformers/models/clip/modeling_clip.py#L200) When you pass input_ids larger than config.max_position_embeddings you don't get errors until the last stage where you add input_embeds and position_embeddings where you get dimension incompatibility error, which is quite vague at that stage. 

### Your contribution

Not sure where is the best place to put the ValueError. I can make a PR If you specify this.

Comments:
Hi @Taha1506, thanks for opening this feature request! 

I think this makes sense, although we'd need to be careful to not add this to cases when interpolation or scaling of position embeddings would occur like RoPE. 

I'd suggest putting this in the forward method of embeddings layer cc @ArthurZucker for your thoughts 

sounds good to me, we are indeed wasting compute otherwise

This is fixed in PR #32156 .