noamaanMulla-03/transformers-issues huggingface dataset:
add `modules_in_block_to_quantize` arg in GPTQconfig
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR adds the `modules_in_block_to_quantize ` quantization arg for gptq. This is necessary for converting specific layers to quantized layers. With this PR, we should be able to run the gptq mixtral model. See related [PR](https://github.com/huggingface/optimum/pull/1585) in optimum.

```python 
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_name = "TheBloke/Mixtral-8x7B-v0.1-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map={"":0})
print(model)

inputs = tokenizer.encode("Hello, how are you today ?", return_tensors="pt").to(0)
outputs = model.generate(inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27956). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add models from cpmant to derformable_detr
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #27885

[](https://github.com/huggingface/transformers/issues/27885)#27885

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova




Comments:
@stevhliu, I built docs locally, it should pass now.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27884). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Did the final change as well, as we discussed on the mail, can you open a separate issue for a model doc files with listing of the `model_docs` files that are yet to be translated?
An error occurred when saving the model
Author: Decem-Y
Pull request: no

https://github.com/huggingface/transformers/blob/14666775a296a76c88e1aa686a9547f393d322e2/src/transformers/trainer.py#L2349
When I update transformers == 4.36.0 for multi-GPU training to save the model, an error occurs prompting to save to "tmp-checkpoint-xx" instead of "checkpoint-xx".

Comments:
I find it https://github.com/huggingface/transformers/issues/27925#issuecomment-1851779674

Please fix the bug caused by this merged PR https://github.com/huggingface/transformers/pull/27820.

cc @pacman100 @muellerzr
Issue with Whisper model
Author: lem21h
Pull request: no

### System Info

After upgrading from version 4.30.2 to 4.35.0 the Whisper model stopped working.

The issue is with the following piece of the code:
```  
if tokenizer is None:
        tokenizer = get_tokenizer(model.is_multilingual)
if tokenizer.language is None or tokenizer.language_token not in tokenizer.sot_sequence:
        raise ValueError(f"This model doesn't have language tokens so it can't perform lang id")
```

For version 4.30.2 the `tokenizer.language` returns 50259
for version 4.35.0 the `tokenizer.language` returns 50349

### Who can help?

@ArthurZucker

### Reproduction

```
model = whisper.load_model("medium")
tokenizer = get_tokenizer(model.is_multilingual)
print(tokenizer.language_token)
print(tokenizer.language_token not in tokenizer.sot_sequence)
```

For 4.30.2 the returned values are:
50259
False

For 4.35.0 the returned values are:
50349
True


### Expected behavior

I would expect that this will work the same.
Also why the same model out of sudden changes the language token value ??

Comments:
After a bit of digging the Whisper AI builds a map of special tokens:
```
additional_tokens = dict(
            zip(
                tokenizer.tokenizer.additional_special_tokens,
                tokenizer.tokenizer.additional_special_tokens_ids,
            )
        )
```
     
on version 4.30.2 the EN gets ID  50259
and on version 4.35.0 the EN gets ID  50349

Hey! I am not sure I follow, where is the snippet taken from? What you are showing seems to use `openai-whisper` which now uses `tiktoken` rather `tokenizers` as  a backend so not sure we can do much here. 


hi, thank you for the reply.

Well whisper.ai in version 1.0.0.1 (from dec 2022) still using GPT2 tokenizer.
The issue is that when the model is loaded, then tokenizer should return the ID of the language (in my case english). It is done by checking those `additional_special_tokens` and matched position against `additional_special_tokens_ids`

In version  4.30.2 of tokenizers, the value for english is returned  50259 and in version  4.35.0 it is out of sudden  50349.

I'm not saying that what they did in Whisper AI is ok, but just curious how ID of EN tokenizer has different ID between those two versions for the same model.
That is the whole issue about :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
cannot import name 'LlamaConfig' from 'transformers
Author: Ahmed-Roushdy
Pull request: no

### System Info


- `adapter-transformers` version: 3.2.1
- Platform: Linux-5.8.0-59-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@SunMarc  @younesbelkada


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

>>> from transformers import LlamaConfig

``` Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'LlamaConfig' from 'transformers' (/home/aelkordy/.local/lib/python3.8/site-packages/transformers/__init__.py)```

### Expected behavior

to be able to load LlamaConfig file

Comments:
Hi @Ahmed-Roushdy !
I see from the details above that:

> adapter-transformers version: 3.2.1

Can you try to install the latest `transformers` package instead? `pip install -U transformers`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bart vs T5 inference time
Author: hadifar
Pull request: no

### System Info

```
- `transformers` version: 4.35.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.4 (cpu)
- Jax version: 0.4.16
- JaxLib version: 0.4.16
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@ArthurZucker @younesbelkada @nar

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is the colab:
https://colab.research.google.com/drive/1h_Ejm1YiyT7JOYMS-HrMkx6uNmiaiVbY?usp=sharing

### Expected behavior

I expect similar inference time for Bart-base & t5-base. However, the t5-base is 3x times faster. As far as know, the Bart has larger context window but overall it is a smaller model (almost 2/3 of parameters of t5).

Comments:
@hadifar can i work on this issue ? i think  we can use quantization to reduce the size of the BART-base model and potentially improve its inference speed.

Hey 🤗 You are running a summurization task, which depends a lot on how well the model performs because the generation time is bounded by how much text is generated. Longer does not necessarily means better performances. You should compare the forward of the model to get better estimates of the latency you can get. 

@ArthurZucker @hadifar Absolutely, , Comparing the forward pass time for both models would indeed provide a more accurate measure of their inference speed. Additionally, consider the hardware on which the models are running, as different hardware configurations can impact performance.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Code Llama HF tokenizer length is 32004 whereas vocab_size is 32000
Author: dineshkh
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.2
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-34b-hf")
model = AutoModelForCausalLM.from_pretrained("codellama/CodeLlama-34b-hf")
embedding_size = model.get_input_embeddings().weight.shape[0]
print("Length of tokenizer:  {} : ".format(len(tokenizer)))
print("vocab_size: {} : ".format(model.config.vocab_size))
print("embedding_size: {} : ".format(embedding_size))
```

```
Length of tokenizer: 32004
vocab_size: 32000 
embedding_size: 32000 
```


### Expected behavior

```
Length of tokenizer: 32004
vocab_size: 32004 
embedding_size: 32004 
```




Comments:
cc @ArthurZucker 

Hey, that's because the tokenizer was not updated. I opened [this](https://huggingface.co/codellama/CodeLlama-34b-hf/discussions/15/files) but will open anotherone to make sure they merge is and we have the latest format. 
Thanks for reporting 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Duplicated issue: #27053 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
SeamlessM4T: `test_retain_grad_hidden_states_attentions` is flaky
Author: gante
Pull request: yes

# What does this PR do?

Adds the `@is_flaky()` decorator to `test_retain_grad_hidden_states_attentions` in SeamlessM4T, as it is a flaky test with a ~11% failure rate.

As discussed internally on Slack.

Comments:
Generate: assisted decoding now uses `generate` for the assistant
Author: gante
Pull request: yes

# What does this PR do?

Subset of the original changes in #27979

"Reworks assisted candidate generation to call .generate(), instead of having its own custom generation loop. For most models this is nothing more than a nice abstraction. However, for models with a custom generate() function, this means the assistant model will now make use of it! (🤔 does this mean that DistilWhisper gets better numbers with this refactor?)"

The following tests were run locally and are passing:
1. `RUN_SLOW=1 py.test tests/models/whisper/ -k speculative`
2. `py.test tests/ -k test_assisted`


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28030). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts the failing test is also failing in the daily CI (i.e. unrelated to this PR, as it doesn't depend on assisted generation), and I can't reproduce it on my end 🤔 

@amyeroberts I can't merge due to the failing test (which is also failing on `main` 👀 ). Would you be able to merge?

(the test is flaky 👉  https://github.com/huggingface/transformers/pull/28035)

@gante In this case we can merge :) 

edit: note this was discussed offline as the reason for failing tests was identified and confirmed as independent from this PR
Fix languages covered by M4Tv2
Author: ylacombe
Pull request: yes

# What does this PR do?

Currently, M4Tv2 into-text tasks (ASR, S2TT, T2TT) do not work for languages outside of the 36 for which audio is supported. This is linked to a test at the beginning of the model generate. The model previously verified if the `tgt_lang` was in a bunch of dictionaries, independently from the output modality. This PR aims to fix that.

I've added a test to make sure it works.

cc @amyeroberts 

Comments:
Thanks for the suggestion @amyeroberts ! I've integrated them and will merge when the CI is green!
Generate: Mistral/Mixtral FA2 cache fix when going beyond the context window
Author: gante
Pull request: yes

# What does this PR do?

The FA2 code path was indexing the `Cache` object incorrectly. This PR fixes it.

Fixes #27985 

_____________________________________________________________

NOTE: `tests/models/mistral/test_modeling_mistral.py::MistralIntegrationTest::test_model_7b_long_prompt` (slow test) was failing on `main`, but it was not popping up in our daily slow CI 🤔 because of that, this issue flew under the radar. It is passing now.

Edit: the test was not run because we are skipping FA2 tests (`@require_flash_attn`). @ydshieh is on it :)

Comments:
This indeed seems to address what was described [here](https://github.com/huggingface/transformers/issues/27985#issuecomment-1855524359), well done!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28037). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As @younesbelkada mentioined and in the official site

> Ampere, Ada, or Hopper GPUs (e.g., A100, RTX 3090, RTX 4090, H100). Support for Turing GPUs (T4, RTX 2080) is coming soon, please use FlashAttention 1.x for Turing GPUs for now.

nothing we can't do unless we run on  a different machine
[Seamless] Fix links in docs
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Relative links were broken, updated to absolute URL ones.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27905). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support PEFT models when saving the model using trainer
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently if one calls `save_model` using the Trainer or `push_to_hub` with a PEFT model, it will push the base model instead of the adapters

to reproduce (after `pip install trl peft transformers`):
```python
from datasets import load_dataset
from trl import SFTTrainer
from peft import LoraConfig

dataset = load_dataset("imdb", split="train")

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

trainer = SFTTrainer(
    "EleutherAI/gpt-neo-125m",
    train_dataset=dataset,
    dataset_text_field="text",
    peft_config=peft_config
)
trainer.save_model("test-sft")
```


cc @pacman100 @sgugger @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

How do we make sure this is the behavior when checkpointing the model during training?

Thanks!

hi @zachschillaci27 ,
you can check the folders that are produced by trainer during training and make sure they contain `adapter_model.safetensors` or `adapter_model.bin` files

> hi @zachschillaci27 , you can check the folders that are produced by trainer during training and make sure they contain `adapter_model.safetensors` or `adapter_model.bin` files

Thanks for the fast reply!
Replace build() with build_in_name_scope() for some TF tests
Author: Rocketknight1
Pull request: yes

Should have included this in the TF `build()` PR but I missed it until now - some of the TF tests should use `build_in_name_scope()` to ensure layer names aren't changed by that PR!

This fix is just for our tests - users shouldn't be affected by the `build()` PR unless they're manually calling `build()` on models and then trying to crossload weights into them.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28046). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts It's hard to say - it would depend exactly on how they were loading the weights. What has changed with this PR is that if you initialize a model via `from_config` **and then explicitly call its build() method**, you'll get slightly different weight names for it than you would have before. For every other case (e.g. initialize with `from_pretrained()` or implicitly call `build()` via calling the model), there should be no difference.

I think for users to notice an issue they would need to:
1) Load a model with `from_config`
2) Explicitly build it with `build()`
3) Have their own weight-loading script that they use to load weights by name into their model after 2)

I think it's pretty unlikely that anyone is doing that, and even if so, they can fix it by using `build_in_name_scope()` instead.
Remove warning when Annotion enum is created
Author: amyeroberts
Pull request: yes

# What does this PR do?

The Annotion enum was deprecated in #26941. 

I asked for there to be a deprecation warning to let users know if they chose to use the enum. This was overly defensive and in light of recent [complaints of objects being removed/moved from the library](https://github.com/huggingface/transformers/issues/25948#issuecomment-1758537251) (even if they were never meant to be used directly).

However, complete oversight on my part is that the __init__ of the enum will be created any time any object from the `image_utils` module file is imported - resulting in verbose error messages unrelated to what the user was trying to do - my bad. 

This PR removed the warning from the enum itself and adds it into a validation check that happens on annotations. 

It ultimately means that we might break things when we remove `Annotion` - but this is likely to be very unlikely and with a simple, quick resolution. 

Partially resolves #28042

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28048). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.