noamaanMulla-03/transformers-issues huggingface dataset:
Add openpangu_moe model
Author: yzy1996
Pull request: yes

# What does this PR do?

This PR introduces inference support for the openpangu_moe model. By integrating the inference implementation directly into the Transformers library, we aim to ensure seamless compatibility and enable continuous updates alongside future open-source releases.

I welcome any feedback on potential improvements or additional changes that would enhance this implementation. Please feel free to share your suggestions!

Our open-source model is available at: https://ai.gitcode.com/ascend-tribe/openPangu-Ultra-MoE-718B-V1.1

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

- text models: @ArthurZucker @Cyrilvallez

 -->


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, openpangu_moe
Fix bnb for the weights refactor
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR fixes bnb support (8bit + 4bit) in the new weight loading logic. 

## Testing 

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_name = "meta-llama/Llama-3.2-3B-Instruct"
quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")

#model_name = "unsloth/Llama-3.2-3B-Instruct-bnb-4bit"
# don't pass quantization_config

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map=0
)

input_text = "Write me a poem about Machine Learning."
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids, do_sample=False, max_new_tokens=1024)
print(tokenizer.decode(outputs[0]))
```

- [ ] check why the memory is way too high when quantizing on the fly
- [x] bnb tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: bnb

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19232411170)

✅ No failing test specific to this PR 🎉 !


@bot /style

Style fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19437535599).

@bot /style 

Style bot fixed some files and pushed the changes.

@bot /style 

Style bot fixed some files and pushed the changes.

run-slow: bnb

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

> LGTM.Just tie twice is my nightmare but good otherwise

I've upstreamed some code from accelerate to fix tied weights. This should be easier this way and we can better tweak device map related code in the future. 

@bot /style

Style fix is beginning .... [View the workflow run here](https://github.com/huggingface/transformers/actions/runs/19472292632).

@bot /style 

Style bot fixed some files and pushed the changes.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19472237564)

✅ No failing test specific to this PR 🎉 !


run-slow: bnb

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/bnb"]

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19472622549)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bnb, finegrained_fp8
Fix looping in torch guard decorator
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

`sys.modules` can end-up being quite large, especially when using `pytest`. This avoids the loop while keeping the benefits

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42260). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix other quants
Author: SunMarc
Pull request: yes

# What does this PR do?

This adds supports for quants methods that needs to add Quantize class. From what I saw, hqq will be the hardest to deal with ... (not fixed at all for now). The other will only require to check if the tests are passing. 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: fbgemm_fp8, higgs, hqq
Fix the issue Arthur was talking about.
Author: Vaibhavs10
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Note sure what were the discussions, but just wanna pin-point that this is known on `main` and is not an issue as param loading happens after the call to `_initialize_missing_keys`, so even if we indeed overwrite some params (because we can only init full modules), they are then correctly loaded afterwards

TLDR `_initialize_missing_keys` is never called alone, always BEFORE the actual loading
fix crash when using chat to send 2+ request to gptoss
Author: sywangyi
Pull request: yes

without it， 2+ request in chat using gptoss will coredump，error like
Traceback (most recent call last):
  File "/mnt/disk0/wangyi/miniforge3/envs/transformers/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/mnt/disk0/wangyi/miniforge3/envs/transformers/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/wangyi/transformers/src/transformers/commands/serving.py", line 981, in generate_with_cache
    generate_output = model.generate(**kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/disk0/wangyi/miniforge3/envs/transformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangyi/transformers/src/transformers/generation/utils.py", line 2539, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/wangyi/transformers/src/transformers/generation/utils.py", line 2860, in _sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangyi/transformers/src/transformers/generation/utils.py", line 569, in prepare_inputs_for_generation
    inputs_embeds, input_ids = self._cache_dependant_input_preparation(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangyi/transformers/src/transformers/generation/utils.py", line 475, in _cache_dependant_input_preparation
    or (cache_position[-1] >= input_ids.shape[1])  # Exception 3
        ~~~~~~~~~~~~~~^^^^
IndexError: index -1 is out of bounds for dimension 0 with size 0


Comments:
@SunMarc ，please review it

you could use lmsys/gpt-oss-20b-bf16 to reproduce

the crash is caused by past_key_value size > input id size, when 2 request send

Nice I had the same issue [here](https://github.com/issues/created?issue=huggingface%7Ctransformers%7C40504) 

> Thanks, I'm not sure this is the right fix as I think this is an issue with the model. The changes here is the same as ignoring the kv-cache if there was an issue.

yes, I am not sure if other models have such issue. but it's safe to ignore the kv-cache if it's longer than the input_ids.

I'll let @gante comment on that ! I don't mind having this as a temporary fix but it would be better to have a real fix for this 

@gante , could you pls share your insight on this?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40536). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add xpu support in test_modeling_janus.py::JanusIntegrationTest::test…
Author: sywangyi
Pull request: yes

…_model_generate_images

add xpu support in tests/models/janus/test_modeling_janus.py::JanusIntegrationTest::test_model_generate_images


Comments:
@ydshieh please help review

The change in modeling is because:  xpu set allow_is_causal_skip to True for prefill in static cache case for performance optimization in sdpa, as a result the attention_mask may not be contained in model_inputs.

Intel XPU: @IlyasMoutawwakil

@sywangyi , pls fix CI issues, thx.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: janus
find cpu_amp is incorrect set, it's only set if self.sharded_ddp is n…
Author: sywangyi
Pull request: yes

…ot None...


- trainer: @sgugger


Comments:
amp does not work if self.sharded_ddp is None

_The documentation is not available anymore as the PR was closed or merged._

If Accelerate's support isn't handling mixed precision or if there is some issue with that, raise an issue with the reproducer. As of now, no changes are required to mixed precision handling.

> Now, mixed precision handling is moved to Accelerate. Only when Fairscale is being used mixed precision handling is supported in trainer as it isn't present in Accelerate.

Hi, I see in evaluation_loop and prediction. accelerate prepare is only called when  self.is_deepspeed_enabled and self.model_wrapped is self.model, so how about the non-deepspeed case?
         

> Hi, I see in evaluation_loop and prediction. accelerate prepare is only called when self.is_deepspeed_enabled and self.model_wrapped is self.model, so how about the non-deepspeed case?

That makes sense, thank you! I've raised a PR addressing this as linked above.

Hi, @pacman100 , I have another question. any idea of jit trace in evaluation in https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1384, it also need the amp info.
use accelerate autocast in jit eval path, since mix precision logic is…
Author: sywangyi
Pull request: yes

… in accelerator currently

Fixes # (issue)
mix precision logic is all moved to accelerator, so autocast_smart_context_manager does not take effect any longer. use accelerate autocast instead




## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


- trainer: @sgugger



Comments:
@yao-matrix

_The documentation is not available anymore as the PR was closed or merged._
if output is tuple like facebook/hf-seamless-m4t-medium, waveform is …
Author: sywangyi
Pull request: yes

…the first element


Comments:
test code
```
import torch
from transformers import pipeline, set_seed
import scipy


def main():
    set_seed(555)

    generator = pipeline(
        "text-to-speech",
        model="facebook/hf-seamless-m4t-medium",
        torch_dtype=torch.float32,
        device="cuda",
    )

    speech = generator("Hello, my dog is cute", forward_params={"tgt_lang": "rus"})
    print(f"speech = {speech}")
    scipy.io.wavfile.write("techno2.wav", rate=speech["sampling_rate"], data=speech["audio"][0])


if __name__ == "__main__":
    main()
```

```
Traceback (most recent call last):
  File "/mnt/disk1/wangyi/transformers/test.py", line 23, in <module>
    main()
  File "/mnt/disk1/wangyi/transformers/test.py", line 17, in main
    speech = generator("Hello, my dog is cute", forward_params={"tgt_lang": "rus"})
  File "/mnt/disk1/wangyi/transformers/src/transformers/pipelines/text_to_audio.py", line 182, in __call__
    return super().__call__(text_inputs, **forward_params)
  File "/mnt/disk1/wangyi/transformers/src/transformers/pipelines/base.py", line 1206, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/mnt/disk1/wangyi/transformers/src/transformers/pipelines/base.py", line 1214, in run_single
    outputs = self.postprocess(model_outputs, **postprocess_params)
  File "/mnt/disk1/wangyi/transformers/src/transformers/pipelines/text_to_audio.py", line 204, in postprocess
    output_dict["audio"] = waveform.cpu().float().numpy()
AttributeError: 'tuple' object has no attribute 'cpu
```

@amyeroberts please have a review.

Hmmm, this is funny, as it indicates either unexpected outputs from the model, it shouldn't be used with this pipeline or that it wasn't properly tested for the pipeline @ylacombe  could you please take a look into this?

could anyone help review the pr？

test added and also found some issue when test batch_size=2 @ylacombe 

I also add "return_intermediate_token_ids" in the testcase and fix the issue in postprocess.
fix patch_attention_mask incorrect setting which leads to the differe…
Author: sywangyi
Pull request: yes
Labels: name: run-slow

…nce in the generated text if batch > 1


Fixes # (issue)
found the evaluate accuracy differ between batch=1 and batch>1.
if batch>1, some image would have padding pixel. and patch_attention_mask incorrect setting makes the generated result different.
- vision models: @amyeroberts, @qubvel


Comments:
idefics2 enable_input_require_grads not aligned with disable_input_re…
Author: sywangyi
Pull request: yes

…quire_grads

make peft+idefics2 checkpoints disable fail

error like
AttributeError: 'GaudiIdefics2ForConditionalGeneration' object has no attribute '_require_grads_hook'

- vision models: @amyeroberts


Comments:
> Thanks for opening this PR and fixing!
> 
> For my own understanding of the changes - it seems there's two changes happening here: one which ensures proper disabling of hooks, the other which adds peft-related logic to the tests. Is the latter necessary to catch the former?

yes, the latter is used to catch the former.

@sywangyi Could you explain a bit why? In the case of the test, `_hf_peft_config_loaded=False` seems to be testing the previous logic (and I assume passing on main). The reason I ask is that it's good we find a way to catch possible issues but the relation to PEFT isn't obvious to me -- this indicates that either the original test isn't properly capturing what we thought or there's a different behaviour in the case of peft which e might need to address. 

In particular, I think that we probably want a peft-specific test to be separated out of the current test to make sure what's being tested and why is clear 

see logic in https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2380 and https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L2429 this path (_hf_peft_config_loaded=True) is not tested. so I set _hf_peft_config_loaded to False and True to test both path.

@sywangyi OK, I see. Hmmm -- we shouldn't really need to know about PEFT + trainer within the modeling utils but alas it's there. Could you split up the tests so that we have a separate one for the PEFT condition and resolve the failing tests? After than I think the PR will be good to go 

I check the failure case
FAILED examples/tensorflow/test_tensorflow_examples.py::ExamplesTests::test_run_image_classification - ValueError: Task image-classification is not compatible with this dataset! Available tasks: []
======== 1 failed, 5 passed, 3 skipped, 8 warnings in 300.61s (0:05:00) ========

this case has nothing to do with the PR
fix crash in tiiuae/falcon-11B-vlm image-to-text generation
Author: sywangyi
Pull request: yes
Labels: name: Vision

name: Multimodal

# What does this PR do?

fix crash when using tiiuae/falcon-11B-vlm for image-to-text generation task

fix two crashes
1. one is in inputs_embeds calculation, "image_token_index": 65024 while "vocab_size": 65024, so embedding operation out of bounds
2. language_modeling forward in llava_next. the parameters contains num_logits_to_keep, while forward of FalconForCausalLM does not contain the params. crash in forward API calling 



- vision models: @amyeroberts, @qubvel



Comments:
Hi @sywangyi, thanks for submitting a PR! Can you please provide your environment setup and minimum example, so we can reproduce the error on our side too? Adding a test also would be much appreciated! Thank you!

cc @zucchini-nlp for vlms

I use the latest transformers code 
and here's my example

```
from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from PIL import Image
import requests
import torch

processor = LlavaNextProcessor.from_pretrained("tiiuae/falcon-11B-vlm", tokenizer_class='PreTrainedTokenizerFast')
model = LlavaNextForConditionalGeneration.from_pretrained("tiiuae/falcon-11B-vlm", torch_dtype=torch.bfloat16)


url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
cats_image = Image.open(requests.get(url, stream=True).raw)

prompt = "User:<image>\nWhat is shown in this image?\nAssistant:"
inputs = processor(text=prompt, images=cats_image, return_tensors="pt").to('cuda:0')
model.to('cuda:0')
output = model.generate(**inputs, max_new_tokens=100, do_sample=False)


generated_captions = processor.decode(output[0], skip_special_tokens=True).strip()

print(generated_captions)
```




The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34728). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
from_pretrained should handle xpu case
Author: sywangyi
Pull request: yes


 @SunMarc @zach-huggingface

please help review. find the issue when inference meta-llama/Llama-4-Scout-17B-16E in TGI. while TGI use FLASH_TRANSFORMERS_BACKEND for llama4.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
fix speecht5 failure issue in test_peft_gradient_checkpointing_enable…
Author: sywangyi
Pull request: yes
Labels: name: run-slow

…_disable

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
https://github.com/huggingface/transformers/issues/33527


- speech models: @ylacombe, @eustlb


Comments:
@amyeroberts 

gentle ping @ylacombe @eustlb

@Rocketknight1  gentle ping.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34454). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
xpu optimization for generation case
Author: sywangyi
Pull request: yes


Comments:
@liangan1 @vasqu @LuFinch pls help to review this pr.

LGTM.
I have verified that 
1. SDPA can run into `enable_gqa=True` for XPU now.
2. SDPA can run into `is_causal=True` for XPU when using Static KVcache.


xpu sdpa kernel does not have the limitation as cuda that is_causal should be set False when seqlen q not equal kv

Could you also take a look into the CI failures, seem like there are some dynamo issues :eyes: 

> Could you also take a look into the CI failures, seem like there are some dynamo issues 👀


I have fixed the dynamo ci issue
update perf_train_cpu_many doc
Author: sywangyi
Pull request: yes

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._