noamaanMulla-03/transformers-issues huggingface dataset:
Fix TorchAoConfig not JSON serializable
Author: andrewor14
Pull request: yes

**Summary:** TorchAoConfig optionally contains a
`torchao.dtypes.Layout` object which is a dataclass and not JSON serializable, and so the following fails:

```
import json
from torchao.dtypes import TensorCoreTiledLayout
from transformers import TorchAoConfig

config = TorchAoConfig("int4_weight_only", layout=TensorCoreTiledLayout())

config.to_json_string()

json.dumps(config.to_dict())
```

This also causes `quantized_model.save_pretrained(...)` to fail because the first step of this call is to JSON serialize the config. Fixes https://github.com/pytorch/ao/issues/1704.

**Test Plan:**
python tests/quantization/torchao_integration/test_torchao.py -k test_json_serializable

Comments:
@SunMarc @ArthurZucker @jerryzh168 Please take a look, thanks!

Yes, it solves my issue. Thanks!

LGTM @andrewor14 thanks for the fix !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36206). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove flakiness in VLMs 
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title. Only one test was flaky for VLMs - `test_promp_loopup_decoding`. Unfortunately it can't be fixed by suppressing image/video tokens because the Prompt Lookup doesn't apply any logits processor. This can be fixed with some additional effort, but not worth it. VLMs don't use these type of assisted decoding usually, haven't seen any issues from the community


   

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36242). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you for fixing @zucchini-nlp 🤗
Deepseek v2 support
Author: itazap
Pull request: yes

Model: https://huggingface.co/deepseek-ai/DeepSeek-V2


**Key Features**
- 400GB of weights, slow
- Yarn support
- `topk_method` is group limited greedy
- Attention: down-projection and up-projection of Q, KV

**Key Changes to Authors' code:**

General:
- [x] removed `ep_size` logic

MoE:
- [x]  removed separate inference logic in `MoEGate` --> More like Mixtral/Llama/ etc.
- [x] upcast to float32 in hidden state calculations

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31976). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey is there any update on this pr?

@itazap is off for a while feel free to take over!

https://github.com/huggingface/transformers/pull/35926

Can we use this instead? (change v3 -> v2)
Au revoir flaky `test_fast_is_faster_than_slow`
Author: ydshieh
Pull request: yes

# What does this PR do?

Time to say goodbye ~~~~~~~~ 

For `tests/models/depth_pro/test_image_processing_depth_pro.py::DepthProImageProcessingTest::test_fast_is_faster_than_slow`

(running 1000 times)
- main: 3.7% failure
- pr: 0%

For `tests/models/rt_detr/test_image_processing_rt_detr.py::RtDetrImageProcessingTest::test_fast_is_faster_than_slow`

(running 1000 times)
- main: 2.4% failure
- pr: 0.6 % failure (so we use `is_flaky`)

(haven't run for all processors yet, but from above check, this PR does its job in genral)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36240). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks a lot for this!
autocast() got an unexpected keyword argument 'cache_enabled when use trainer.torch_jit_model_eval
Author: Wanguy
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: Linux-4.18.0-147.el8_1.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.10
- Huggingface_hub version: 0.26.5
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@muellerzr 
@SunMarc

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using the `torch_jit_model_eval()` method in trainer, it prompts 

> "failed to use PyTorch jit mode due to: autocast() got an unexpected keyword argument 'cache_enabled'." 

Looking at the details, it was found that the error was caused by the `self.accelerator.autocast(cache_enabled=False)` method. Its method definition is `def autocast(self, autocast_handler: AutocastKwargs = None)`, and there is no `cache_enabled` method.

Is this because the code here has not been updated, or because I ignored some settings?

Is there a solution now?

### Expected behavior

Work normally.

Comments:
The last `accelerator.autocast` method that supports the cache_enabled parameter is in version v0.34.2.

Thanks for the report @Wanguy ! This is indeed deprecated as you can see [here](https://github.com/huggingface/accelerate/pull/3098#:~:text=Accelerator.autocast%20no%20longer%20accepts%20a%20cache_enabled%20arg%2C%20users%20must%20make%20an%20AutocastKwargs%20instance%20and%20pass%20it%20in%20as%20a%20kwarg%20handler). Would you like to submit a PR to fix that ? Thanks ! 

@SunMarc Of course, I will submit a fix PR ASAP to fix this bug. :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
load once first before memory measure for `test_from_pretrained_low_cpu_mem_usage_equal`
Author: ydshieh
Pull request: yes

# What does this PR do?

I don't have any valid theory, but this tests fails (almost every time) if there is no local cache.
Once we have a local cache, it seems passing all the time.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added Support for Custom Quantization
Author: keetrap
Pull request: yes

This PR adds a new feature to support custom quantization in the Transformers library.

Closes #35814
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
- [x] Was this discussed/approved via a Github issue 

@SunMarc @MekkCyber


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35915). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @keetrap, Thanks for PR ! It looks great and it's a very handy feature for the community ! I just left some very small nits

@SunMarc @MekkCyber @keetrap  Sorry I'm just seeing this now - I've been on Chinese New Year vacation.

I ran into a problem with the QuantizationMethod enum not being extensible, so I couldn't come up with a good solution.

I think this PR is impressive and aligns well with my expectations. LGTM!

Thanks your your input @ice-tong ! @keetrap, i've merged a PR that requires you to do some updates on your code. Could you try to fix the conflits ? Thanks ! 

@SunMarc @MekkCyber 
I have resolved the conflict and I do not think any change is required to logic, correct me if I am wrong. It is working locally.
But tests are failing.
Can we merge this ?

Sorry for the delay @keetrap, the CI is green will merge now
[Feature Request] Support register customize quantization method out-of-tree
Author: ice-tong
Pull request: no
Labels: name: Feature request

### Feature request

Support register customize quantization method out-of-tree.

The usage would be as follows:

```python
from transformers.quantizers import HfQuantizer
from transformers.quantizers import regsiter_quantization_config, register_quantizer
from transformers.utils.quantization_config import QuantizationConfigMixin


@regsiter_quantization_config("custom")
class CustomFakeQuantizationConfig(QuantizationConfigMixin):
    """The custom fake quantization config."""


@register_quantizer("custom")
class CustomFakeQuantizer(HfQuantizer):
    """The custom fake quantizer."""
```
### Motivation

We would greatly appreciate it if HuggingFace could support registering custom quantization schemes externally. This would allow us to integrate the schemes of any LLM quantization tools and evaluate fake quantization models using the powerful combination of `lm_eval` + `huggingface`. Thank you for considering this!

Similar features have already been supported by vLLM, see:
- https://github.com/vllm-project/vllm/issues/11926
- https://github.com/vllm-project/vllm/pull/11969

### Your contribution

If this feature request is considered, I'd happily submit a PR to implement it.

Comments:
cc @sunmarc @MekkCyber 

Looks like a nice feature ! Feel free to submit a PR for that !  cc @mht-sharma, I think you wanted to have something like that too
Recent Qwen2VL merge request (#35837) break compatibility with DeepSpeed
Author: ArdalanM
Pull request: no

The recent merge request (#35837) works with accelerate but breaks with DeepSpeed (w/ and w/o deepspeed config) 

- distributed_type: MULTI_GPU (work)
- distributed_type: DEEPSPEED (no longer works)

To be more precise the issue lies in this section: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L200 
```
    emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
    cos = emb.cos().float()
    sin = emb.sin().float()
else:
    cos, sin = position_embeddings
q, k = apply_rotary_pos_emb_flashatt(q.unsqueeze(0), k.unsqueeze(0), cos, sin)
```

`cos, sin = position_embeddings` these are not casted to float and are subject to various dtypes depending on the DeepSpeed and mixed_precision config.

This accelerate config works:
```
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity:  #false
main_training_function: main
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
mixed_precision: bf16
```

This accelerate config  no longer works:
```
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: DEEPSPEED
deepspeed_config:
  zero_stage: 3
downcast_bf16: 'no'
enable_cpu_affinity: false
main_training_function: main
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```








Comments:
same issue

Nice catch!

Resolved on `main` now
Torchao `int4_weight_only` save error when passing layout
Author: jiqing-feng
Pull request: no
Labels: name: bug

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.49.0.dev0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.4.5
- Accelerate version: 1.4.0.dev0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 5,6
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.6.0+cu126 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100 80GB PCIe


### Who can help?

@SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

More details refer to [torchao](https://github.com/pytorch/ao/issues/1704)


### Expected behavior

Hi @SunMarc . Do you think we can handle this in transformers?

Comments:
Padding error when using Universal Assisted Generation with ASR pipeline
Author: danielkorat
Pull request: no
Labels: name: Core: Pipeline

name: bug

name: Audio

### System Info

Can not solve by supplying padding arg to pipeline (not accepted).

@gante 

transformers version: https://github.com/huggingface/transformers.git@refs/pull/34504/merge
Ubuntu
Python 3.10.15

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

![image](https://github.com/user-attachments/assets/5818757b-25a6-471c-81a9-20679b43f73d)


### Expected behavior

Should complete pipeline execution as normal

Comments:
cc @ylacombe @eustlb

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Keeping this issue live

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

gentle ping @gante 

@danielkorat The two models expect two different `input_features`, and `generate` doesn't support passing custom inputs for the assistant model. I also don't want to include this possibility, as it would greatly increase the complexity of the codebase for limited gains.

The solution for this specific pair of models would be to implement custom code on the user side :)

@gante makes sense
I wonder if we can find a pipeline other than TextGeneration to test UAG.
This was the reasoning behind this

Uhmm in theory, we should be able to test speech recognition with a LLM assistant 👀 (we might have bugs, but they should be simple to fix -- the LLM shares the input_ids with the audio model)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add BidirectionalWordpieceTokenizer for improved Korean/Japanese tokenization
Author: springkim
Pull request: yes

This PR adds a new BidirectionalWordpieceTokenizer class that supports bidirectional tokenization while maintaining compatibility with the existing WordpieceTokenizer interface. This implementation is based on the technique introduced in KR-BERT: A Small-Scale Korean-Specific Language Model (https://arxiv.org/abs/2008.03979), which has shown improved performance particularly for Korean and Japanese text processing.
Key changes:

Add BidirectionalWordpieceTokenizer class
Implement bidirectional tokenization with forward and backward vocabularies
Choose tokenization based on token length
Add test cases to verify functionality

The implementation has been tested and passes all core tokenization tests.
The bidirectional approach has demonstrated better performance specifically for Korean and Japanese languages, making it a valuable addition to our tokenization capabilities.

Comments:
[tests] remove `pt_tf` equivalence tests
Author: gante
Pull request: yes

# What does this PR do?

See title.

Follows the same logic as in #36235: equivalence tests slow down development on the PT side.

In this particular case, this PR unblocks #36238: some models will start returning a `Cache` instance by default, which breaks the equivalence tests on those models.

cc @LysandreJik @ArthurZucker as we discussed on Slack; @Rocketknight1 fyi

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36253). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Happy to break the equivalence tests at this point!

@ydshieh 

> Are we going to remove pt_flax tests too?

Yes, right after this PR gets merged 😎 (to avoid making a massive PR, which tend to lower the quality of the review process)
TP initialization module-by-module
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title! At loading time, the parallelization is now applied module-by-module, so that no memory overhead is required compared to what the final weight distribution will be!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35996). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add base tensor parallel support to Phi
Author: tanuj-rai
Pull request: yes

# What does this PR do?
This PR adds Tensor Parallel to the Phi model.

Fixes #34789

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ArthurZucker and anyone from the community



Comments:
Sorry for the timeline, I think it was merged on main in #35748 

No worries! Thanks for the clarification @ArthurZucker.
not support flash-attention
Author: LiuYinfeng01
Pull request: no

```
Traceback (most recent call last):
  File "/home/yinfeliu/test_bge-reranker-v2-m3.py", line 10, in <module>
    model = AutoModelForSequenceClassification.from_pretrained(model_dir,torch_dtype=torch.float16, attn_implementation="flash_attention_2").cuda()
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4124, in from_pretrained
    config = cls._autoset_attn_implementation(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1586, in _autoset_attn_implementation
    cls._check_and_enable_flash_attn_2(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1707, in _check_and_enable_flash_attn_2
    raise ValueError(
ValueError: XLMRobertaForSequenceClassification does not support Flash Attention 2.0 yet. Please request to add support where the model is hosted, on its model hub page: https://huggingface.co/BAAI/bge-reranker-v2-m3/discussions/new or in the Transformers GitHub repo: https://github.com/huggingface/transformers/issues/new
```

my transformers version is 4.47.1 and I install flash-attention the version is 2.7.4.post1. And I check print(is_flash_attn_2_available()), the value is true, but when I use 
`model = AutoModelForSequenceClassification.from_pretrained(model_dir,torch_dtype=torch.float16, attn_implementation="flash_attention_2").cuda()` , raise value error, how can I use flash-attn correctly?


Comments:
Hi @amd-liuyinfeng, there's no bug here! The problem is that `XLMRoberta` is an older model that does not support flash attention, so you should not set `attn_implementation="flash_attention_2"`.

If someone from the community wants to make a PR to add it, though, we'd be happy to accept that!
[tests] deflake dither test
Author: gante
Pull request: yes

# What does this PR do?

- Deflakes `test_dither`, added in #34638. On a `--flake-finder --flake-runs 1000` runs we now get 0 failures.
- Upgrades the `sampling_rate` warning message to specify what `"this function"` means (it was unclear from the warning message).

Comments:
I would love to leave this PR for @eustlb 🤗 but feel free to ping me if anything gets blocked for too long

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36284). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(@eustlb perhaps I didn't convey my message correctly -- I agree with the warning 🤗 I wanted to clarify that this PR's diff in the lines where we call the processor exists so that we don't throw a warning at test time, which ultimately makes the `flake-finder` less seamless)
[tests] remove flax-pt equivalence and cross tests
Author: gante
Pull request: yes

# What does this PR do?

Follow-up to #36253 -- removes FLAX-PT equivalence and cross-tests.

After this PR, I believe PT development should be mostly free from other frameworks :)

cc @LysandreJik @ArthurZucker as we discussed

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh sorry, the same happened again 😅 

the `if foo(): pass` pattern was removed (or at least I can't find more)

(@ydshieh not clear from your previous message -- if you're happy with the PR, I'll need an approval to merge 🤗 )

ohhhhh, I forgot the ✅ 

boom done :D