noamaanMulla-03/transformers-issues huggingface dataset:
Why the transcription of the audio by Whisper pipeline and processor are different?
Author: MartaSamoilenkoPn
Pull request: no

I am using two approaches: processor or pipeline because the pipeline returns timestamps but no confidence levels and the processor returns confidence levels. Anyway, the problem is that pipeline transcription sometimes is worse than processor transcription. Is it a problem in the Transformers lib?

processor:
" I got six, and seven, and eight, and nine, and ten, and eleven, and twelve, and thirteen, and twenty, and fifteen, and sixteen.",

pipeline:
 " I got six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen, sixteen.",

The audio for both approaches is the same.

Comments:
Hey! When you say that you are using the processor, are you using the `generate` api? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐 [i18n-KO] Translated `chat_templating.md` to Korean
Author: enchantee00
Pull request: yes

# What does this PR do?

Translated the `chat_templating.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
@win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32362). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Encounter error when loading checkpoint generated by latest accelerate>=0.34.0
Author: jiangjiadi
Pull request: no
Labels: name: Core: Modeling

name: bug

name: PyTorch FSDP

name: Accelerate

### System Info

- `transformers` version: 4.43.4
- Platform: Linux-4.9.151-015.ali3000.alios7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.34.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@muellerzr

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Train the model (e.g. Qwen1.5-0.5B) using FSDP with accelerate>=0.34.0. When you get the checkpoint, then load the model using AutoModelForCausalLM.from_pretrained. Then you will get the following errror.
![image](https://github.com/user-attachments/assets/77fd400c-443f-4695-89ff-3303897d05c9)


### Expected behavior

When I downgraded the accelerate version to 0.33.0, the issue was resolved. Further investigation revealed that it was these two lines of code in the `get_state_dict` function of the `Accelerator` in `accelerate.py` that caused the problem.

![image](https://github.com/user-attachments/assets/4146760e-5b11-4aa1-a497-1bde7d4e0cca)

Transformers may need to adapt to this change in accelerate.

Comments:
Thanks @jiangjiadi! cc @muellerzr indeed, cc @SunMarc as well

Thanks for the report @jiangjiadi ! Linked to this issue https://github.com/huggingface/accelerate/issues/3061. Maybe we should revert + patch this @muellerz as this break FSDP basic usage ? 

Reverted ! I think we can close the issue @jiangjiadi ? 

@SunMarc Sure, thanks a lot.
XLMRobertaTokenizer false description for build_inputs_with_special_tokens function
Author: TianmengChen
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.41.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.9
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer

model_name_or_path = "BAAI/bge-reranker-base"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

query = ["what is openvino" , "text"]
query_passage_pairs = "what is openvino 

 text"

input_tensors_pairs = tokenizer(
    query_passage_pairs, padding=True, truncation=True, return_tensors="pt"
)
input_tensors_query  = tokenizer(
    query , padding=True, truncation=True, return_tensors="pt"
)

```
### Expected behavior

different result between input_tensors_pairs and input_tensors_query

Comments:
Hey! Not sure why you would get different results, when a pair is passed, you do: `
 A 

 B 
`, and for a single item you will do: `
 X 
`, so if X= `A 

 B`, then you get `
 A 

 B 
`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AttributeError: 'T5Stack' object has no attribute 'layer' when using t5_model.prune_heads(heads_to_prune)
Author: anaghasid
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.4


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
heads_to_prune = {0: [3, 4, 5, 6, 7], 1: [0, 1, 2, 3, 4, 6, 7], 2: [3, 4, 5, 6, 7], 3: [1, 3, 4, 5], 4: [4], 5: [0, 1, 2, 5]}
model.prune_heads(heads_to_prune)
```

```bash
[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in __getattr__(self, name)
   1707             if name in modules:
   1708                 return modules[name]
-> 1709         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1710 
   1711     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'T5Stack' object has no attribute 'layer'
```

### Expected behavior

Expected model to change with the given heads pruned

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Flaky test - tests/models/mobilenet_v1/test_modeling_mobilenet_v1.py::MobileNetV1ModelTest::test_batching_equivalence
Author: amyeroberts
Pull request: no
Labels: name: Tests

name: Vision

### System Info

`tests/models/mobilenet_v1/test_modeling_mobilenet_v1.py::MobileNetV1ModelTest::test_batching_equivalence` periodically fails causing red on CI runs

Example CI run: https://app.circleci.com/pipelines/github/huggingface/transformers/94887/workflows/6766b7cd-c38c-4334-ab25-2afcb8e0d842/jobs/1246458

### Who can help?

@zucchini-nlp as you added the batching equivalence tests :) 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run enough times to hit an error 

### Expected behavior

Doesn't occasionally fail

Comments:
Yeah, probably it was flaky at the time of adding because the MobileNetV2 is also flaky as marked #29516 

As discussed, I am not CV expert but maybe I can give it another look later next week

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Refactor module imports and dependency handling
Author: swalehmwadime
Pull request: yes

### Summary
This pull request refactors the module imports and dependency handling in the codebase. The main changes include:

- **Updated Import Structure**: Refined the import structure to conditionally include modules based on the availability of PyTorch. This ensures that optional dependencies are only loaded if they are available.
  
- **Lazy Loading**: Introduced lazy loading for certain modules to optimize performance and reduce the overhead of importing unused modules when PyTorch is not present.

- **Type Checking**: Added type hints and utilized `TYPE_CHECKING` from the `typing` module to facilitate development-time type checking without affecting runtime behavior.

### Details
- The `_import_structure` dictionary now conditionally includes modules based on whether PyTorch is available.
- Lazy loading is implemented using `_LazyModule` to handle module imports dynamically.
- Type hints are provided for better code clarity and development-time checks.

### Testing
- Verified that the script correctly handles the absence of PyTorch by not including optional modules.
- Confirmed that lazy loading functions as expected when PyTorch is not available.
- Ensured type hints do not interfere with runtime functionality.

Please review the changes and provide feedback. Thank you!


Comments:
Enable non-safetensor ser/deser for TorchAoConfig quantized model 🔴 
Author: jerryzh168
Pull request: yes

Summary:
After https://github.com/huggingface/huggingface_hub/pull/2440 we added non-safetensor serialization and deserialization in huggingface, with this we can now add the support in transformers

Note that we don't plan to add safetensor serialization due to different goals of wrapper tensor subclass and safetensor see README for more details

Test Plan:
tested locally
https://gist.github.com/jerryzh168/965ccdbd595c9210d49cfbe31dc6705f

Reviewers:

Subscribers:

Tasks:

Tags:



Comments:
cc @SunMarc @Wauplin can you take a look?

@SunMarc thanks for your thoughtful reviews! I have addressed all the comments I think, please take a look again, also not sure if the CI failure is relevant or not

btw, current pytorch nightly has a perf regression: https://github.com/pytorch/ao/issues/898 and we hope to fix this before 2.5 cherry-pick deadline

To fix the failing test, can you rebase on main ? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33456). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker can you take a look at the PR?

I've merged recently PR adding a new quantizer @jerryzh168. Sorry for that but could you rebase on main and update the `is_serializable` method ? 

@SunMarc @ArthurZucker updated, please take a look again

Thanks a lot @jerryzh168 🤗 great contributions and I love that we can upload serialized quantized weights to the hub now!
Fix typo in documentation 
Author: qgallouedec
Pull request: yes

# What does this PR do?

Fix a typo in the documentation.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu @amyeroberts 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Misleading warning
Author: krisztian-gajdar
Pull request: no
Labels: name: bug

https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1618C1-L1619C1

The removal of this warning did not happen in v4.45

cc: @ArthurZucker and @itazap

Comments:
patched! https://github.com/huggingface/transformers/releases/tag/v4.45.1 Thanks for flagging!

@itazap I'm still getting the same warning after upgrading to v.4.45.1

should be fixed now, sorry about that! there were 2 warnings to remove there

This should be it, thanks for the effort @itazap !
[`modular`] fixes! 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update the modular converter to make sure all node definition are re-used if done in the modulare file 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Idefics 2 fixed bug shown up again
Author: hukkai
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.1
- Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed
- Flax version (CPU?/GPU?/TPU?): not installed 
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

@amyeroberts @qubvel


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just run the official code from https://huggingface.co/HuggingFaceM4/idefics2-8b for idefics2-8b can reproduce the bug.

### Expected behavior

File ~/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py:1296, in Idefics2Model.inputs_merger(self, input_ids, inputs_embeds, image_hidden_states)
   1294 new_inputs_embeds = inputs_embeds.clone()
   1295 reshaped_image_hidden_states = image_hidden_states.view(-1, vision_hidden_size)
-> 1296 new_inputs_embeds[special_image_token_mask] = reshaped_image_hidden_states
   1297 return new_inputs_embeds

RuntimeError: shape mismatch: value tensor of shape [640, 4096] cannot be broadcast to indexing result of shape [0, 4096]


The bug was first reported in https://github.com/huggingface/transformers/issues/31380. I can confirm it is fixed in `transformers` version: 4.45.0-dev. However, as I upgraded to 4.45.1, it shows up again.

Comments:
Hi @hukkai, thanks for reporting! 

I've opened #33776 which should resolve this issue. For reference, same issue reported in #33752

Hey @amyeroberts I have created a PR before you created and got a review approved
Is this conflict okay?

Thanks

[#33766](https://github.com/huggingface/transformers/pull/33766) This is the PR
May i merge this 

Hi @aroun-coumar, apologies for the duplicated work. I saw #33752 first, and so opened a PR for that issue. 

Your PR was opened first, so let's merge that one in.
[IDEFICS2] Fix past_seen_tokens unset bug
Author: amyeroberts
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Generation in Idefics2 was broken by #33541, as `past_seen_tokens` wouldn't be set if `past_key_values` was a `Cache` instance

Fixes #33763 #33752


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33776). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
minor typo fix
Author: a-r-r-o-w
Pull request: yes

# What does this PR do?

dytpe -> dtype

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33784). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama3 Insturct Tokenizers.Encoding.offsets is wrong
Author: efsotr
Pull request: no
Labels: name: Core: Tokenization

name: Fast Tokenizers

name: bug

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-5.4.0-193-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
print(t("are you ok?", add_special_tokens=False)[0].offsets)
```

output:
```
[(0, 0), (3, 3), (7, 7), (10, 10)]
```

### Expected behavior

```
[(0, 3), (3, 7), (7, 10), (10, 11)]
```

Comments:
```python
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
print(t("今天天气好", add_special_tokens=False)[0].offsets)
```

```
[(0, 2), (2, 3), (3, 4), (4, 5)]
```

If it encodes Chinese characters, it's output is correct.

Hey @efsotr 🤗 It appears that the issue is resolved on later `transformers` versions, please let me know if it persists!

`transformer==4.44.2`:
<img width="796" alt="Screenshot 2024-09-26 at 10 01 07" src="https://github.com/user-attachments/assets/7f2743f2-038e-4e4f-bf5f-f0f9049a9798">

@itazap 
In my server, run script: 
```python
from transformers import __version__, AutoTokenizer
t = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
print(__version__)
print(t("are you ok?", add_special_tokens=False)[0].offsets)
```

and the output is
```
4.44.2
[(0, 0), (3, 3), (7, 7), (10, 10)]
```

hmm can you please share your `tokenizers` version as well? 

```python
import transformers
import tokenizers
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
print(transformers.__version__, tokenizers.__version__)
print(t("are you ok?", add_special_tokens=False)[0].offsets)
```

```
4.45.0 0.20.0
[(0, 0), (3, 3), (7, 7), (10, 10)]
```

Thank you! @ArthurZucker looks like a regression in `tokenizers==0.20`, I'm able to reproduce on several versions of  `transformers` (incl 4.45) and only workes with `tokenizers==0.19`?

It works with tokenizers==0.19.0
It doesn't with tokenizers==0.19.1

Mmm that's weird, https://github.com/huggingface/tokenizers/releases/tag/v0.19.1 only had the `ignore_merges` serialization, which is set to `True` for llama3 (try setting it to False!) 

we'll see if we can do a patch