noamaanMulla-03/transformers-issues huggingface dataset:
Remove remote code warning
Author: Rocketknight1
Pull request: yes

When `trust_remote_code=True` is passed to a non-Auto class, the class raises a warning that the argument has no effect, but continues.

This warning is getting thrown because some of our pipelines are passing the `trust_remote_code` kwarg through to the model. Users can't avoid this because `trust_remote_code` is required for the pipeline. I can dig into that in future, to make sure unnecessary args aren't going to `model.from_pretrained()` but for now I think we can just remove this warning because it's kind of pointless and our code needs fewer warnings anyway :sparkles: 

Fixes #36273

Comments:
cc @arthurzucker @cyrilvallez for review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36285). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This change is also nice for reducing warnings in automated tests

gentle ping @arthurzucker @cyrilvallez
Loading a pipeline with `trust_remote_code=True` raises warning
Author: tcapelle
Pull request: no
Labels: name: bug

### System Info

Doing 
```
from transformers import pipeline

pipe = pipeline(
    task="custom-task",
    model=local_model_path,
    trust_remote_code=True,
)
```
raises a warning:
```
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
```
but if we remove it, ad the pipeline actually needs to load a custom pipeline or model, you get an error.
```
pipe = pipeline(
    task="custom-task",
    model=local_model_path,
    # trust_remote_code=True,
)

ValueError: Loading this pipeline requires you to execute the code in the pipeline file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Above

### Expected behavior

Should not raise a warning.

Comments:
Thanks for the report! Opened a PR to fix it at #36285
Warning related to torch.tensor() usage in transformers.models.encodec.modeling_encodec.py (Version 4.47.0)
Author: GiorgiaAuroraAdorni
Pull request: no
Labels: name: bug

### System Info

When using the transformers library (version 4.47.0), I am encountering a warning regarding the use of torch.tensor() in the modeling_encodec.py file. The warning suggests that it is recommended to use sourceTensor.clone().detach() instead of torch.tensor() to avoid potential issues. This issue appears to be triggered by the following line in the code:

`self.register_buffer("padding_total", torch.tensor(kernel_size - stride, dtype=torch.int64), persistent=False)`

I am currently using the following versions of relevant libraries in my environment:
nvidia/cuda: 12.3.1-devel-ubuntu22.04
Python: 3.10
torch: 2.4.0+cu124
transformers: 4.47.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Install the dependencies from the requirements file.
Use the transformers library to generate music or perform any task.
Observe the warning in the console related to torch.tensor() usage.

### Expected behavior

The code should not produce a warning regarding the use of torch.tensor() and should follow the recommended usage of clone().detach() for tensor creation.


Comments:
Thanks for flagging this! Fix is open at #36538
Remove research projects
Author: Rocketknight1
Pull request: yes

This PR removes the research projects from `transformers` and adds a link to the repo where they live now: https://github.com/huggingface/transformers-research-projects/

Comments:
@arthurzucker I fixed `release.py`! I also redirected all references to `research_projects` in `transformers` to the new repo

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36645). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Stop warnings from unnecessary torch.tensor() overuse
Author: Rocketknight1
Pull request: yes

Users are getting errors caused by calling `torch.tensor()` on an input that is already a tensor. This is unnecessary because `kernel_size - stride` will already be a new tensor object, and so it doesn't need to be cloned/detached/converted with `torch.tensor()`.

Fixes #36533

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

cc @arthurzucker @cyrilvallez for core maintainer review

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix gguf docs
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR updates a bit the gguf docs to include a pip install mention + small fixes

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36601). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Update docs dependency
Author: stevhliu
Pull request: yes

Removes the `pip install ".[docs]"` dependency (which doesn't exist anymore) from the contribution guide (partially addresses #36541).

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36635). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Downloading models in distributed training
Author: nikonikolov
Pull request: no

When I run distributed training, if the model is not already downloaded locally on disk, different ranks start fighting for the download and they crash. 

I am looking for a fix such that:
1. If the model is not yet downloaded on disk, only one rank downloads it. The rest of the ranks are waiting until the file is downloaded
2. If the model is already on disk, all ranks load it simultaneously, no waiting for each other
3. The solution is universal. In other worlds, I still instantiate the model via `AutoModel` instead of with some wrapper function and I don't write a bunch of if-else statements every time I need to create a model

I wasn't able to find something that can achieve this right now. I guess a very simple solution could be adding lock files when downloading a model such that other ranks wait until the completion of the download and then use the downloaded files directly

Comments:
Hi @nikonikolov, can you give us a simple reproducer script to demonstrate this?

import random
lower = int(input("Enter the lower bound: "))
upper = int(input("\nEnter the upper bound: "))
if lower >= upper:
print('\nUpper bound must be greater than lower bound')
exit()
num = random.randint(lower, upper)
chances = 3
print(f"\nYou have {chances} chances to guess the number!\n")
count = 0
guessed = False
while count < chances:
count += 1
guess = int(input("Guess a number: "))
if num == guess:
print("Congratulations you did it in ", count, " try")
guessed = True
break
elif num > guess:
print("You guessed too small.")
elif num < guess:
print("You Guessed too high.")
if not guessed:
print("\nThe number is %d" % num)
print("\nBetter Luck Next time!")

Fix this





Fix thisssss

Hmm, I used to get this problem, but now I can't reproduce it. Maybe it was fixed in a recent release. Closing for now
Attention mechanisms elaboration
Author: vitormbesen
Pull request: yes

# What does this PR do?

The [Attention](https://huggingface.co/docs/transformers/main/en/attention) section on conceptual guides in the transformers library was lacking some important mechanisms. In this PR, I elaborate mainly on the multi-head attention mechanism of the Transformer architecture. This includes scaled description of mechanisms such as dot-product attention, self-attention, causal self-attention and cross-attention.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
 @stevhliu


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36597). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @stevhliu!

Thanks for your PR, we're currently discussing whether to include content like this in the docs or in the course. Feel free to check out https://github.com/huggingface/course/issues/839 for more context!
The number of safetensors files is different when using CPU and CUDA.
Author: makcedward
Pull request: no
Labels: name: bug

### System Info

transformers version: 4.49.0

### Who can help?

@muellerzr @SunMarc

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using CPU to save models will cause a double number of safetensors when compared to CUDA. So far, there have been no problems with usage, but just a noticeable difference in behavior.

For example, using CUDA to save `meta-llama/Llama-3.1-8B-Instruct` model will result in 4 safetensors, while using CPU to save the same model will result in 8 safetensor files.

### Expected behavior

Same number of safetensor files

Comments:
Hi @makcedward, can you paste the code you're running here? This seems like quite an unusual bug.

A reproducer will help a lot ! Does this also happen in previous version of transformers @makcedward ?

It is caused by loading incorrect model and causing this issue. Closed ticket.
Fix bnb regression due to empty state dict
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes a bug that was introduced since we are loading empty state dict now. For bnb, we need to pass a full state dict as we extract some params from it. 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36663). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen2 MoE manual `head_dim`
Author: yunju63
Pull request: no
Labels: name: Feature request

### Feature request

https://github.com/huggingface/transformers/blob/81aa9b2e07b359cd3555c118010fd9f26c601e54/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py#L317

For qwen2 moe, `head_dim` is now forced to be `hidden_size // num_heads`. 

### Motivation

manual `head_dim` setting support in llama, mistal, mixtral modeling

### Your contribution

PR

Comments:
hi @yunju63, won't the reshape in the attention layer fail if we set `self.head_dim` to any value other than `self.hidden_size // self.num_heads`?
GPTQ quantization on Jetson Orin Nano
Author: lukazso
Pull request: no
Labels: name: bug

I am trying to use GPTQ quantization support on my Jetson Orin Nano Super. The [docs](https://huggingface.co/docs/transformers/main/quantization/gptq) say that I can use either `auto-gptq` or `gptqmodel` for this. However, when only installing gptqmodel - since auto-gptq is supposed to be the deprecated version - I get the error:

```
ImportError: Loading a GPTQ quantized model requires the auto-gptq library (`pip install auto-gptq`)
```

Am I doing something wrong or are the docs not up to date?

Thanks!
Lukas

-------------------


**System Info:**
Machine: aarch64
System: Linux
Distribution: Ubuntu 22.04 Jammy Jellyfish
Release: 5.15.148-tegra
Python: 3.10.12
CUDA: 12.6.68


### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Minimum example to reproduce:
```python3
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_id = "facebook/opt-125m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda", quantization_config=gptq_config)
```

Here the complete traceback:
```
ENV: Auto setting PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' for memory saving.
ENV: Auto setting CUDA_DEVICE_ORDER=PCI_BUS_ID for compatibililty.
Traceback (most recent call last):
  File "/home/lukas/Development/db-agent/test.py", line 39, in <module>
    quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="cuda", quantization_config=gptq_config)
  File "/home/lukas/Development/db-agent/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/lukas/Development/db-agent/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3620, in from_pretrained
    hf_quantizer.validate_environment(
  File "/home/lukas/Development/db-agent/venv/lib/python3.10/site-packages/transformers/quantizers/quantizer_gptq.py", line 59, in validate_environment
    raise ImportError(
ImportError: Loading a GPTQ quantized model requires the auto-gptq library (`pip install auto-gptq`)
```

### Expected behavior

No error

Comments:
cc @SunMarc @MekkCyber

Hi @lukazso, I think you just need to update your transformers version, because this change is very recent

It seems like I already have the latest version (I meanwhile switched to python3.12):

```
transformers       4.48.3
gptqmodel          1.9.0
optimum            1.24.0
```

Yes you are right @lukazso, sorry it's not part of the release yet, It will be in the next release 4.49 in the coming days, for now you can just install the main branch, it's stable
[Feature Request]: refactor _update_causal_mask to  a public utility
Author: Irvingwangjr
Pull request: no
Labels: name: Feature request

### Feature request

 refactor _update_causal_mask to  a public utility

### Motivation

After this pr https://github.com/huggingface/transformers/pull/35235/files#diff-06392bad3b9e97be9ade60d4ac46f73b6809388f4d507c2ba1384ab872711c51
all the attention implement already refactor to use ALL_ATTENTION_FUNCTIONS and people can register their own implement very easy.

I notice that there are still another function: _update_causal_mask  is copy-and-paste everywhere and related to attention modules

if people register an attention impl, this _update_causal_mask will add attention_mask if it is not flash_attention_2. so hope this function can be refactor too

### Your contribution

I can do some testing and submitting a PR, we can add ulysess implementation as a third party example

Comments:
cc @cyrilvallez

Hey @Irvingwangjr! It is actually on the roadmap, and we will start working on it very soon! 🤗 I did not yet opened a PR, but mask creation will be refactored for sure 😉
cc @zucchini-nlp for viz as well, as we talked about it recently and we'll collaborate on this one!
Added handling for length <2 of suppress_tokens for whisper
Author: andreystarenky
Pull request: yes

Added handling for <2 length of suppress_tokens for whisper

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Encountered the following error when implementing a fine-tuned whisper model:
```
site-packages/transformers/models/whisper/generation_whisper.py", line 1783, in _prepare_decoder_input_ids
    prev_start_of_text = suppress_tokens[-2]
                         ~~~~~~~~~~~~~~~^^^^
IndexError: index -2 is out of bounds for dimension 0 with size 0
```
The current code assumes that if `suppress_tokens` is not None it has at least 2 elements. This throws `IndexError` when `suppress_tokens` is empty but not None (or length 1) as below:
```
prev_start_of_text = suppress_tokens[-2] if suppress_tokens is not None else None
```

I believe the initial code may have intended to check for empty list (`if suppress_tokens` would have worked) although this would still miss the case of length 1.

My change checks that the list is not None and has a length greater than or equal to 2 (needed to reference second last elem as per original code)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

My apologies if I am tagging the wrong individuals, using list from comments
Models:
- speech models: @ylacombe, @eustlb 
Library:
- generate: @gante 


Comments:
FIx base model loading
Author: SunMarc
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36579
This PR fixes loading into a base model when using a checkpoint from a non base model. 
The solution is a bit hacky for now but it will be cleaned after @Cyrilvallez refactor which deals with all these prefix logic.

The following fails: 
```python 
from transformers import AutoModel
model = AutoModel.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", device_map="auto")
```


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36581). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Verified on my issue.

Just checked, and it was fixed by https://github.com/huggingface/transformers/pull/36033!

thanks @Cyrilvallez !
Don't accidentally mutate the base_model_tp_plan
Author: Rocketknight1
Pull request: yes

All credit goes to @gante and @arthurzucker for figuring this one out and I'm just swooping in and stealing credit because I want this to be merged quickly!

If no `tp_plan` is provided, our code uses `self.config.base_model_tp_plan` as the default. The problem is that this is a mutable, instance-level dict, and we do in fact mutate it, which causes the instance dict to get very large and weird over time. We resolve the issue by correctly copying the base dict as an instance attribute instead of mutating it in-place.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Lots of failing checks, and I think these are either Hub timeouts or disguised Hub timeouts

cc @sunmarc @muellerzr `test_gradient_accumulation_loss_alignment_with_model_loss` is failing on `main`. Marking it as slow (it takes ~30s on my machine) just so I can start clearing the CI, but we should address that one at some point!

> cc @SunMarc @muellerzr test_gradient_accumulation_loss_alignment_with_model_loss is failing on main. Marking it as slow (it takes ~30s on my machine) just so I can start clearing the CI, but we should address that one at some point!

@IlyasMoutawwakil is looking at it !
Fix Failing GPTQ tests
Author: MekkCyber
Pull request: yes

# What does this PR do?
Fix GPTQ tests, according to this discussion https://github.com/ModelCloud/GPTQModel/issues/1354. 
For now when using gptqmodel, exllama kernels are not used because they don't support packing in the main branch, the default kernels used in the CI are triton kernels

## Who can review ? 
@SunMarc 

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36666). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.