noamaanMulla-03/transformers-issues huggingface dataset:
Add Qwen2MoE
Author: bozheng-hit
Pull request: yes

# Adding Qwen2MoE
This PR adds the support of codes for the coming Qwen2MoE models. For information about Qwen, please visit https://github.com/QwenLM/Qwen. @ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29377). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey, I guess our update of the testing solves the issues mentioned. Take another look?

@ArthurZucker Hi, I have rebased our branch and solved all the conflicts. I think the codes are ready to be merged now.

Thanks for your efforts! merging 🥳 

Hi @bozheng-hit 

Thank you for adding this model 🚀 

I see `Qwen/Qwen1.5-MoE-A2.7B` is used in the tests, but I could not find it on the Hub. See 

https://huggingface.co/models?search=Qwen1.5-MoE

Could you check this, open a PR to make some necessary updates, and make sure the integration tests pass by running

> RUN_SLOW=1 TF_FORCE_GPU_ALLOW_GROWTH=yes python3 -m pytest -v tests/models/qwen2_moe

Thank you in advance



Didn't saw `Qwen/Qwen1.5-MoE-A2.7B` anywhere, time estimation on new model release?
[`GptNeox`] don't gather on pkv when using the trainer
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #29376 by adding `"keys_to_ignore_at_inference"` to the config

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Flashatten2 avaiable should handle if hardware support or not?
Author: lucasjinreal
Pull request: no

For some docker images the flashattn installed, but v100 not support.
It will return True and raise error in runtime.

Also, since torch already support flash2 inside, add torch2.2 check and using inside version?

Comments:
Hi @lucasjinreal, thanks for opening an issue! 

Please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide: 
* A minimal code reproducer
* All relevant error information including the full error traceback
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove misleading model disclaimers in docs for gpt2 and gpt neo QA.
Author: Whenning42
Pull request: yes

The removed disclaimers suggest the real checkpoints aren't correct and that they need to be replaced by themselves.

GPT 2 disclaimer:
"This example uses a random model as the real ones are all very big. To get proper results, you should use gpt2 instead of gpt2. If you get out-of-memory when loading that checkpoint, you can try adding device_map="auto" in the from_pretrained call."

and

GPT Neo disclaimer:
"This example uses a random model as the real ones are all very big. To get proper results, you should use EleutherAI/gpt-neo-1.3B instead of EleutherAI/gpt-neo-1.3B. If you get out-of-memory when loading that checkpoint, you can try adding device_map="auto" in the from_pretrained call."


# What does this PR do?
Remove misleading model disclaimers in docs for gpt2 and gpt neo QA.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu or @MKhalusova for documentation review.

Comments:
Hi @Whenning42, thanks for opening this PR! 

Could you share what the effect is of removing the `real_checkpoint` argument? Does the disclaimer disappear?  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add W2V2 example to CTC training readme
Author: ylacombe
Pull request: yes

# What does this PR do?

This PR adds a W2V2-Bert training example config to the CTC folder. This might be a bit light, I can add another training config example on TIMIT or turkish CV tomorrow if needed.

cc @sanchit-gandhi 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`pipeline`]. Zero shot add doc warning
Author: ArthurZucker
Pull request: yes

# What does this PR do?
fixes #27783 to make sure `hypothesis_template` is documented

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29845). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`ZeroShotAudioClassificationPipeline` documentation includes example to duplicate hypothesis template
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker (original implementer) @Vaibhavs10 @sanchit-gandhi (audio team)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Visit [documentation](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.ZeroShotAudioClassificationPipeline)
2. Example is:
```py
from transformers import pipeline
from datasets import load_dataset

dataset = load_dataset("ashraq/esc50")
audio = next(iter(dataset["train"]["audio"]))["array"]
classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")
classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
```
3. However, this is duplicated due to [this](https://github.com/huggingface/transformers/blob/29f1aee3b6c560182415fcb9e2238125e2f5b29c/src/transformers/pipelines/zero_shot_audio_classification.py#L118-L122) portion of code:
https://github.com/huggingface/transformers/blob/29f1aee3b6c560182415fcb9e2238125e2f5b29c/src/transformers/pipelines/zero_shot_audio_classification.py#L118-L122

4. Sentences become: `['This is a sound of Sound of a dog.', 'This is a sound of Sound of vaccum cleaner.']`

### Expected behavior

Sentences should be either:
- `['Sound of a dog.', 'Sound of vaccum cleaner.']`
- `['This is a sound of a dog.', 'This is a sound of vaccum cleaner.']`

Comments:
FWIW, the output is near-identical, but this might cause unexpected issues later down the line.

Yeah, not super clear what we should do, maybe regex the hypothesis template? Updating the doc is a good start 


I'll open PRs for this and Whisper 

Oops opening this today! 🤗 

Happy to take care of this Arthur or want myself or @ylacombe to take a look?

sure, completely forgot about it!
Tests: replace `torch.testing.assert_allclose` by `torch.testing.assert_close`
Author: gante
Pull request: yes

# What does this PR do?

Replaces `torch.testing.assert_allclose` by `torch.testing.assert_close`. The former is [deprecated](https://pytorch.org/docs/stable/testing.html#torch.testing.assert_allclose) since torch 1.12 (and will be removed in a future version), with the later being its replacement

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29915). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Transformers 4.39.0 has breaking change on torch compatibility.
Author: xkszltl
Pull request: no

### System Info

torch 1.13 + transformers 4.39.0

### Description

`torch.optim.LRScheduler` was only exposed in torch 2:
- https://github.com/pytorch/pytorch/pull/88503

Previously it was called `_LRScheduler` in torch 1.

This recent PR included in 4.39.0 imports it and lead to a hard requirement on torch 2:
- https://github.com/huggingface/transformers/pull/29588

FYI @younesbelkada 

This issue is captured by our CI where we have dependency on torch 1.

### Expected behavior

Suggest to check on version or API availability before use.
Given it's only a visibility change, a transparent fallback/re-route would be ideal.

Comments:
This is a duplicate of #29847, I'll push something asap and do a patch release. FYI @LysandreJik and @amyeroberts
Safe import of LRScheduler
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes #29895

cc @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29919). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Trainer: Functions to inspect model and optimizer status
Author: yqy2001
Pull request: no
Labels: name: Good First Issue

name: trainer

name: Feature request

### Feature request

In huggingface Trainer, are there any functions to inspect model and optimizer status? such as, how many parameters require grad, learning rate of each parameter, which optimizer group each parameter belong...

I didn't find any related function in Trainer, and I know implementing it by myself is easy, but I just want to know whether such functions already exist.

### Motivation

Such inspection is useful for correcting training.

### Your contribution

I propose a question.

Comments:
cc @muellerzr @pacman100 

Hi, can I take over the issue?

@CKeibel Sure! No need to claim on an issue, we prioritise based on PRs open, as we find this helps prevent issues from going stale without being addressed. Once you have something opened, feel free to ping me and @muellerzr for review 🤗 

Hey, thanks for the reply. Maybe I should have read the contribution guidelines first. I'm used to asking beforehand in other open source projects. But then I will start contributing diligently. :)
Extra linear scaling in `LlamaRotaryEmbedding` classes
Author: gtebbutt
Pull request: no
Labels: name: solved

I've been doing some work around NTK and YaRN scaling, and I noticed that PR https://github.com/huggingface/transformers/pull/29198 adds linear scaling `t = t / self.scaling_factor` to the constructor for the `LlamaRotaryEmbedding` base class, possibly pulled in from `GPTNeoXLinearScalingRotaryEmbedding` (which uses a different style of `forward` function to `LlamaLinearScalingRotaryEmbedding`) - if I'm reading this correctly, it's a behavioural change that means the Llama RoPE subclasses are now being scaled twice, once linearly by the base class constructor and then again by their respective `forward` functions. 

Should be a simple fix to just take out this one line, assuming I haven't missed anything: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L108

Comments:
cc @ArthurZucker @gante 

Hi @gtebbutt 👋 

It is not a bug, but a backward compatibility-intended duplication. If you look at the code, the line `t = t / self.scaling_factor` is used exclusively to build the backward compatible `sin_cache` and `cos_cache`. These are currently unused in the model forward pass. As such, there is no duplication :)

In any case, it is a fair doubt -- I had to double-check the code, despite being familiar with it! To avoid regressions, I'm opening a PR with a test to confirm the correctness of RoPE scaling, where a duplicated scaling factor would cause the test to fail.

Thanks @gante - appreciate you taking the time to check, I'd missed that subtlety around the caching!
RoPE models: add numerical sanity-check test for RoPE scaling
Author: gante
Pull request: yes

# What does this PR do?

Fixes #29765 

#29765 asks a pertinent question, where I had to look at the code to confirm the answer. This question should be checked automatically in a test instead -- confirm that RoPE scaling is working as intended.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29808). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Rather than generations, could we check the embedding values to see whether they've been rescaled as expected instead?

Much better 👀 and doesn't need to be a slow test. Going to rework the PR

@amyeroberts deleted the previous slow test, and added a numerical test to the embeddings instead as you suggested -- much much faster to test, and a more precise sanity check.

I've added a test on all RoPE-scaling compatible models. The test can't be abstracted into the mixin (for now), as there are a few variations while we are working on torch.compile/caching.
[`Mamba`] from pretrained issue with `self.embeddings`
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #29631. There was a typo when porting the models, `self.embeddings` vs `self.embedding` which was not caught because of a `from_pretrained` issue. 

```python
>>> from transformers import AutoModel
>>> model = AutoModel.from_pretrained("state-spaces/mamba-130m-hf")
```
no longer gives a warning

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #29452. Before the `prefix_space` added by `add_dummy_prefix_space` was eaten by the `decode` function. This is silent, not expected and thus should not be there. 

`input_ids = tokenizer.encode("hello", add_special_tokens=True)` would produce `["
", "▁hello"]` but decoded as `"
hello"`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29453). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`BC`] Fix BC for other libraries
Author: ArthurZucker
Pull request: yes

# What does this PR do?

Fixes #29783 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29934). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

For sure. accessing the past_key_values should maybe be done via a `getter` or rely on `generation_config.cache_implementation`!
Deberta-v2 position_biased_input error in documentation
Author: NilsLusch
Pull request: no
Labels: name: Documentation

name: Good First Documentation Issue

### System Info

The documentation states that the default value for position_biased_input in deberta-v2 is False but in the source above shows that the default value is True. As far as I can tell in the code, it defaults to False. But cleaning it up in the documentation would be good

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The issue is with the documentation not with the code

### Expected behavior

The source should show that the value defaults to False
![issue](https://github.com/huggingface/transformers/assets/96436846/38d9ec1e-0b03-454b-83e3-cbd5baaba231)
 

Comments:
Hi @NilsLusch, thanks for raising this! 

Would you like to open a PR to fix this? This way you get the github contribution for the solution.

Hi @amyeroberts!

Can I pick and solve this issue?
[`LlamaSlowConverter`] Slow to Fast better support
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Makes sur that the eos token, bos token and unk token are correctly taken when initializing from slow. 

Fixes #29774, fixes #29836


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
LLaVA image_token_index is not 64000 but 64002
Author: XuweiyiChen
Pull request: no
Labels: name: Should Fix

name: bug

name: Multimodal

### System Info

LLaVA image_token_index is not 64000 but 64002 in the latest version of the code. (main)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. pip install the latest transformer from main branch
2. run llava 34b model.

### Expected behavior

You can do inference freely but now an error will pop out saying there is 0 image token.

Comments:
Hi,

Thanks for reporting, this is also reported here: https://huggingface.co/llava-hf/llava-v1.6-34b-hf/discussions/2 and will be resolved by https://github.com/huggingface/transformers/pull/29797
Fix timeout propagation for world_process_zero things
Author: muellerzr
Pull request: yes

# What does this PR do?

For `torch.distributed` to catch timeout issues, we need to actually spawn a point to have the timeout be called/synchronized. For things that matter (like large model checkpoint uploads), we can do so by adding a `wait_for_everyone` call at the end. 

Pretty sure this fixes https://github.com/huggingface/trl/issues/1403 and https://github.com/huggingface/accelerate/issues/2236

Leaving as a draft until user can verify


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29523). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix doc issue #29758 in DebertaV2Config class
Author: vinayakkgarg
Pull request: yes

# What does this PR do?
In the code, the value for position_biased_input defaults to True. The documentation states that value of position_biased_input parameter defaults to False which is incorrect.

Fixes #29758 (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@amyeroberts 

Comments:
Update installs in image classification doc
Author: MariaHei
Pull request: yes

# What does this PR do?

Trainer with PyTorch now requires accelerate to be installed. Therefore, accelerate is added to the list of libraries to be installed.

Fixes #29174 (partly)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29947). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.