noamaanMulla-03/transformers-issues huggingface dataset:
fix gpt-oss out shape
Author: jiqing-feng
Pull request: yes

Hi @SunMarc . This commit should be included in #40304 , but I forgot to push this commit. We need this PR; otherwise, the out value will be wrong. Please review and merge this PR. Thanks!

Comments:
run-slow: gpt_oss

run-slow: gpt_oss

run-slow: gpt_oss

When I tested I did not have any issues, do you mind sharing a reproducer?! 🤗 

This comment contains run-slow, running the specified jobs: 

models: ['models/gpt_oss']
quantizations: [] ...

```python 
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

torch.manual_seed(0)
model_id = "openai/gpt-oss-20b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype = "bfloat16"
).eval()

messages = ["Hello my good sir, how are you doing today?", "Sir, how are you doing today?"]

inputs = tokenizer(
    messages,
    return_tensors="pt",
    padding=True
)

# generated = model.generate_batch(**inputs, max_new_tokens=512, do_sample=True)
generated = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(
    f"rank {model.device} ------ {tokenizer.batch_decode(generated[inputs['input_ids'].shape[-1] :])}"
)

```
rank this without any issues!

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gpt_oss

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix pipeline dtype
Author: jiqing-feng
Pull request: yes
Labels: name: for patch

The dtype in the pipeline does not work if I don't pass device_map. To reproduce the error:

```python
from transformers import pipeline
import torch

image_to_text = pipeline(
        "image-feature-extraction",
        model="google/vit-base-patch16-224-in21k",
        dtype=torch.bfloat16,
    )
print(image_to_text.model.dtype)
```

output before this PR
```
torch.float32
```

output after this PR
```
torch.bfloat16
```

Hi @SunMarc . Please review this PR. Thanks!

Comments:
Hi @SunMarc , the failed CI seems not related to my changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40638). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
revert `_prepare_4d_causal_attention_mask_with_cache_position` for gpt2
Author: jiqing-feng
Pull request: yes

Hi @zucchini-nlp 

The PR #39754 deleted `_prepare_4d_causal_attention_mask_with_cache_position` on gpt2, which caused 40% performance regression on CPU. You can reproduce it by 
`numactl -C 0-7 --membind 0 python test.py`

```python
import time
import torch
from transformers import pipeline, set_seed, AutoTokenizer

set_seed(42)

model_id = "openai-community/gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.padding_side = 'left'
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id

pipe = pipeline("text-generation", model=model_id, tokenizer=tokenizer, torch_dtype=torch.float16, device_map="cpu")

generation_config = pipe.model.generation_config
generation_config.do_sample = False
generation_config.use_cache = True
generation_config.max_new_tokens = 128
generation_config.min_new_tokens = 128
generation_config.cache_implementation="static"
generation_config.temperature = 1.0
generation_config.top_p = 1.0
generation_config.num_beams = 1
pipe.model.config._attn_implementation="sdpa"

inputs = "It is done, and submitted. You can play 'Survival of the Tastiest' on Android, and on the web. Playing on the web works, but you have to simulate multiple touch for table moving and that can be a bit confusing. There is a lot I'd like to talk about. I will go through every topic, insted of making the typical what went right/wrong list. Concept Working over the theme was probably one of the hardest tasks which I had to face. Originally, I had an idea of what kind of game I wanted to develop, gameplay wise - something with a lot of enemies/actors"

for _ in range(5):
    set_seed(42)
    pipe(inputs, generation_config=generation_config)

for _ in range(5):
    set_seed(42)
    start = time.time()
    pipe(inputs, generation_config=generation_config)
    end = time.time()
    print(f"{pipe.model.dtype} time costs {(end-start)*1000} ms")
```

Revert `_prepare_4d_causal_attention_mask_with_cache_position` can fix the regression.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gpt2

run-slow: gpt2

OK, please let me know when the PR is ready. Thanks!
fix check inputs for text2text pipeline
Author: jiqing-feng
Pull request: yes

The pipeline should check generation_config before checking inputs.
```python
import torch
from transformers import pipeline, AutoTokenizer

model_id = "sshleifer/distilbart-cnn-12-6"
tokenizer = AutoTokenizer.from_pretrained(model_id)
generator = pipeline(
    "summarization",
    model=model_id,
    tokenizer=tokenizer,
    dtype=torch.float16,
)
generation_config = generator.model.generation_config
generation_config.max_new_tokens = 4
generation_config.min_new_tokens = 4
input_sentence = "This is an example inputs for summarization"
output = generator(
    input_sentence, generation_config=generation_config
)
```

output:
```
Traceback (most recent call last):
  File "/home/jiqing/HuggingFace/tests/workloads/test_summa.py", line 16, in <module>
    output = generator(
             ^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/text2text_generation.py", line 285, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/text2text_generation.py", line 182, in __call__
    result = super().__call__(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/base.py", line 1289, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/base.py", line 1296, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/base.py", line 1200, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/text2text_generation.py", line 198, in _forward
    self.check_inputs(
  File "/home/jiqing/transformers/src/transformers/pipelines/text2text_generation.py", line 291, in check_inputs
    if max_new_tokens < min_length:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'NoneType' and 'int'
```

This PR can fix it by getting the correct max_new_tokens in generation config.

Comments:
Hi @SunMarc . Could you please review this PR? Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41556). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SunMarc , the failed CI seems not brought by this PR, could you help retrigger CI, thx very much.
fix bnb model loading
Author: jiqing-feng
Pull request: yes

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4")
```

Before this PR:
```
Traceback (most recent call last):
  File "/home/jiqing/transformers/src/transformers/pipelines/base.py", line 262, in load_model
    model = model_class.from_pretrained(model, **fp32_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/modeling_utils.py", line 272, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/modeling_utils.py", line 4669, in from_pretrained
    hf_quantizer, config, dtype, device_map = get_hf_quantizer(
                                              ^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/quantizers/auto.py", line 301, in get_hf_quantizer
    config.quantization_config = AutoHfQuantizer.merge_quantization_configs(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/quantizers/auto.py", line 210, in merge_quantization_configs
    quantization_config = AutoQuantizationConfig.from_dict(quantization_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/quantizers/auto.py", line 130, in from_dict
    raise ValueError(
ValueError: Unknown quantization type, got bitsandbytes - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq
', 'aqlm', 'quanto', 'quark', 'fp_quant', 'eetq', 'higgs', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet', 'vptq', 'sp
qr', 'fp8', 'auto-round', 'mxfp4']
```

After this PR, it works.


Hi @SunMarc . Please review this PR. Thanks!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41499). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
update torchao doc
Author: jiqing-feng
Pull request: yes

Update torchao doc for CPU and XPU based on https://github.com/pytorch/ao/issues/3300. 

Hi @stevhliu  @SunMarc , would you please review the PR? Thanks!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42139). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enable XPU doc
Author: jiqing-feng
Pull request: yes

Hi @SunMarc . This PR enables XPU torchao example in the doc, and also fixed a minor bug on the doc. Please review it. Thanks!

Comments:
Convert this PR to draft mode for verifying FP8.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
update bnb ground truth
Author: jiqing-feng
Pull request: yes

The previous ground truth was generated by ipex backend, we should update it since bnb is going to use sycl/triton backend.

Comments:
Waiting for BNB SYCL merge.

Hi thanks!

LGTM, but on which hardware you obtain this new values? We are using A10.

@MekkCyber 

Don't forget you can use

run-slow: bnb

Oh yes 💯 ! thanks @ydshieh 

run-slow: bnb

This comment contains run-slow, running the specified jobs: 

models: []
quantizations: ['quantization/bnb'] ...

@MekkCyber you can use `ssh into runner workflow` to launch a A10 runner and update 

> tests/quantization/bnb/test_mixed_int8.py::MixedInt8TestPipeline::test_pipeline

see https://github.com/huggingface/transformers/actions/runs/15974913856/job/45054899427

> Hi thanks!
> 
> LGTM, but on which hardware you obtain this new values? We are using A10.

I ran this test on Intel XPU PVC node, but it's a common issue. You can update the ground truth by your testing device. Otherwise, it will fail on any device because of the sampling.

I also verified this PR on A100, it has the same outputs as XPU.

run-slow: bnb

This comment contains run-slow, running the specified jobs: 

models: []
quantizations: ['quantization/bnb'] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39117). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you 🎉
fix caching_allocator_warmup with tie weights
Author: jiqing-feng
Pull request: yes

### Motivation
The test `tests/quantization/finegrained_fp8/test_fp8.py::FP8QuantizerTest::test_quantized_model_multi_accelerator` failed with
```
>       self.assertTrue(set(quantized_model.hf_device_map.values()) == {0, 1})tests/quantization/finegrained_fp8/test_fp8.py:206:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.11/dist-packages/transformers/testing_utils.py:609: in wrapper
    return test_case(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
E   AssertionError: False is not true
```

### RootCause
The `caching_allocator_warmup` didn't consider tie weights, so it reserved more memory than the model actually needed.


Comments:
cc @ydshieh

cc @MekkCyber
enable xpu on kv-cache and hqq doc
Author: jiqing-feng
Pull request: yes

Enable xpu on kv-cache and hqq doc. The previous quantized kv-cache usage is out-of-date, I have updated it and verified it on both A100 and XPU.

Comments:
Hi @yao-matrix , please review this change. Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39246). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
set document_question_answering pipeline _load_tokenizer to True
Author: jiqing-feng
Pull request: yes

Hi @Rocketknight1 . The document-question-answering pipeline requires a tokenizer in the preprocessing step.

script: [impira/layoutlm-document-qa](https://huggingface.co/impira/layoutlm-document-qa)
```python
from transformers import pipeline

nlp = pipeline(
    "document-question-answering",
    model="impira/layoutlm-document-qa",
)

nlp(
    "https://templates.invoicehome.com/invoice-template-us-neat-750px.png",
    "What is the invoice number?"
)
```

error:
```
Traceback (most recent call last):
  File "/home/jiqing/transformers/test_dqa.py", line 8, in <module>
    nlp(
  File "/home/jiqing/transformers/src/transformers/pipelines/document_question_answering.py", line 308, in __call__
    return super().__call__(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/base.py", line 1450, in __call__
    return next(
           ^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/pt_utils.py", line 269, in __next__
    processed = self.infer(next(self.iterator), **self.params)
                           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/utils/data/dataloader.py", line 734, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/utils/data/dataloader.py", line 790, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/utils/data/_utils/fetch.py", line 33, in fetch
    data.append(next(self.dataset_iter))
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/pt_utils.py", line 186, in __next__
    processed = next(self.subiterator)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/pipelines/document_question_answering.py", line 324, in preprocess
    max_seq_len = self.tokenizer.model_max_length
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'model_max_length'
```

Please review this PR. Thanks!

Comments:
Hi @jiqing-feng, yes, thank you for catching this so quickly! We also saw the errors in our nightly CI. Hopefully this PR resolves things.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39411). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
enable triton backend on awq xpu
Author: jiqing-feng
Pull request: yes

IPEX is going to be deprecated. As autoawq already archived, we can only change transformers to enable triton kernel on XPU. Only XPU chanegd from ipex to triton as the AWQ triton kernel does not work on CPU.

Comments:
cc @IlyasMoutawwakil 

Hi @Rocketknight1 . This PR is ready to be reviewed. Thanks!

Sorry for the mistake. You were right! The awq xpu does not support bfloat16.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39443). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unskip and fix test_offline_mode by using HF_HUB_OFFLINE and hermetic cache warmup
Author: arvinder004
Pull request: no
Labels: name: Tests

`tests/utils/test_offline.py::OfflineTests::test_offline_mode` was skipped and relied on `TRANSFORMERS_OFFLINE`. Update to use `HF_HUB_OFFLINE=1` and make the test hermetic by warming the cache in a first subprocess run, then running offline with socket mocking to forbid network access.

### Affected area:
File: `tests/utils/test_offline.py`
Test: `OfflineTests::test_offline_mode`

### Problem:
Test was `@unittest.skip` with a TODO.
Behavior depended on environment set after import and on external cache state.

Comments:
cc @ydshieh

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This would be quite a helpful feature to fix/test for my use case. I am deploying in an environment that cannot directly access the Hub and the standard environment variable `HF_HUB_OFFLINE = 1` doesn't prevent Transformers from attempting to redownload models / check configs, even if I point to a fully resolved local model path. Please let me know if I'm commenting in the wrong place and I'll move this.

@fr1ll Please open an issue with a reproducible code snippet. We will look into it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Fix] Fix FA2 kernels ut
Author: YangKai0616
Pull request: yes

# What does this PR do?

This PR fixes the UTs where FA2 falls back to kernels loading.
For example `pytest -rA tests/models/sam3/test_modeling_sam3.py::Sam3ModelTest::test_flash_attn_2_can_dispatch_composite_models`.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
Hi @vasqu , please help review, thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42803). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fixes 2 failing tests from AMD CI
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?

Fixes 2 small failing tests
- [OPENAIGPTModelLanguageGenerationTest](https://github.com/huggingface/transformers/actions/runs/19983883282/job/57315088128#step:14:1067)
- [HeliumIntegrationTest](https://github.com/huggingface/transformers/actions/runs/19983883282/job/57315087965#step:14:1177)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: helium, openai

cc @ivarflakstad for AMD/ROCm

run-slow: helium, openai

Thanks everyone! Taking a look.

This comment contains `run-slow`, running the specified jobs: 

models: ["models/helium", "models/openai"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20227175481)

✅ No failing test specific to this PR 🎉 !


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42777). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Improve contribution guidelines for Quantization
Author: MekkCyber
Pull request: yes

# What does this PR do?

Adds a small explanation for the difference between `get_quantize_ops` and `get_conversion_weights` with small examples to hopefully help new contributors understand the workflow better

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42870). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Do not rely on config for inferring model dtype
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Back to what it was before https://github.com/huggingface/transformers/pull/42805. Relying on the config is an issue in some cases, see https://github.com/huggingface/transformers/pull/42805#issuecomment-3646172797

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42838&sha=fcb609

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42838). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks

Thanks for the fix, this also e.g. fixes `model.half()` saving as `"dtype": "float32"` via `model.save_pretrained(...)` in `5.0.0rc01`, which broke some tests in Sentence Transformers.