noamaanMulla-03/transformers-issues huggingface dataset:
🌐 [i18n-KO] Translated ` deepspeed.md` to Korean
Author: ahnjj
Pull request: yes

# What does this PR do?

Translated the ` deepspeed.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->
@4N3MONE, @Kim-Ju-won, @ahnjj, @FacerAin, @ssum21, @TaskerJang, @HyunZ118

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
 @stevhliu May you please review this PR? 

Comments:
> 전반적으로 깔끔한 번역입니다! 👍 긴 문장을 두 문장으로 분리하여 가독성 향상 제안드립니다! ㅎㅎ. 확인 부탁드립니다!

현상님 리뷰 감사 감사합니다!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/18570467050)

@stevhliu I've made final review and added toctree! Appreciate if you approve it!

@Kim-Ju-won 주원님 나이스 마지막 리뷰감사합니다! 

Nice, would you mind pushing an empty commit to trigger the CI tests please? Thanks so much for your patience 🤗
Docs(zh-hans): Refine wording for professionalism in README
Author: Ri-Nai
Pull request: yes

# What does this PR do?

This PR improves the localization quality of the Simplified Chinese README (`README_zh-hans.md`) by replacing a colloquial term with more professional vocabulary.

Specifically, the informal term `魔改` (mó gǎi), which is a slang for "heavy modification," has been replaced with the standard and formal term `修改` (xiū gǎi), meaning "to modify."

This subtle but important change enhances the document's professionalism and ensures the language is accessible and appropriate for a broad technical audience.

<!-- Remove if not applicable -->

<!-- This is a minor documentation improvement, so no associated issue is required. -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for the fix!
> 
> Feel free to add any other changes you may have to this PR instead of opening a new one :)

Great, thank you for the feedback!

In that case, I will also incorporate the changes I proposed in Issue #40946 into this PR. This includes reverting terms like `抱抱脸` back to `Hugging Face` and `流水线` back to `pipeline` to improve clarity and professional tone in the Chinese documentation.

I'll add the new commits to this PR shortly. Thanks again!


Gentle ping @stevhliu to make sure this doesn't get lost! I just saw a stale warning on the original issue
Add vision contribution guide
Author: molbap
Pull request: yes

# What does this PR do?

This updates the vision contribution guide, to help external contributors merge their work faster and reduce the load on maintainers with a checklist. It is a very minimal checklist that will need to be completed with specific model implementation guides following the new API.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41456). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AutoTokenizer: clear ImportError when loading Voxtral without mistral-common + unit test
Author: Elon7069
Pull request: yes

Summary: Fix confusing TypeError from SentencePiece when loading Voxtral tokenizer without mistral-common by raising a clear ImportError instead. Fixes #41553.
Rationale: Without mistral-common, loading via AutoTokenizer ends in a low-level TypeError inside sentencepiece. Users should see a direct, actionable message.
What’s changed
AutoTokenizer guard:
In AutoTokenizer.from_pretrained, after resolving config and before class selection, raise a clear error if config.model_type == "voxtral" and mistral-common is missing.
Message:
"The Voxtral tokenizer requires the 'mistral-common' package. Please install it using pip install mistral-common."
Tests:
Added tests/models/voxtral/test_tokenization_voxtral.py
Mocks is_mistral_common_available to False and get_tokenizer_config to avoid network, then asserts ImportError mentioning "mistral-common".
Why this approach
Keeps mapping logic intact; avoids unexpected fallbacks.
Ensures the user sees a clear, actionable message as early as possible in the loading path.
Testing
Targeted test: pytest tests/models/voxtral/test_tokenization_voxtral.py -q → passes.
No network calls thanks to test monkeypatching.
Backward compatibility
No behavior change when mistral-common is installed.
Only affects Voxtral when the dependency is missing.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, voxtral

Maintainer please check my PR lemme know what i have to improve more 

I think this is a duplicate of #41592!

I think both are different ki lemme check> I think this is a duplicate of #41592!
Swapped to_seq_len/from_seq_len in comment
Author: nikitakit
Pull request: no

I'm pretty sure this comment:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2c5d993ba48841575d9c58f0754bca00b288431c/modeling.py#L339-L343

should instead say:
```
# Sizes are [batch_size, 1, 1, to_seq_length] 
# So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length] 
```

When masking out tokens for attention, it doesn't matter what happens to attention *from* padding tokens, only that there is no attention *to* padding tokens.

I don't believe the code is doing what the comment currently suggests because that would be an implementation flaw.

Comments:
Yes! fixed the comment
Add pluggable safety hooks to text generation via optional safety_config
Author: Tejassveer08
Pull request: yes



# What does this PR do?

<!--
GenerationConfig: Added optional safety_config kwarg to carry a user-defined safety configuration or factory. Stopping criteria: Introduced SafetyCriteria to allow user-provided callbacks to stop generation based on input_ids/scores. GenerationMixin integration:
Construct and append safety processors from generation_config.safety_config.construct_processors(...) if available. Construct and append safety stopping criteria from generation_config.safety_config.construct_criteria() if available. Wrapped construction in try/except to preserve backward compatibility and avoid failing generation on safety construction errors.
-->

<!-- Remove if not applicable -->

Fixes #41740 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
added moderation to text generation models
Author: DeXtAr47-oss
Pull request: yes

What does this PR do?
Adds a minimal moderation subsystem and generation-time integration to support content filtering during text generation.
Introduces a small, opt-in API so callers can supply custom safety/classifier implementations that run during generation.

Fixes # (41740)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
[kernels] Add version to function mapping
Author: MekkCyber
Pull request: yes

# What does this PR do?

We simply add a version attribute to the function kernel mapping to avoid breaking changes for kernels in the future

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41685). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes added the test here : https://github.com/huggingface/transformers/pull/41765 with all the other tests, and kernels CI. I will merge this and rebase to make sure evtn passes
Fix graphormer model compilation with Cython 3.1.4
Author: alexmalyshev
Pull request: yes

Hitting this kind of error when running:

```
cython src/transformers/models/deprecated/graphormer/algos_graphormer.pyx
```

```
Error compiling Cython file:
------------------------------------------------------------
...
    (nrows, ncols) = path.shape
    assert nrows == ncols
    cdef unsigned int n = nrows
    cdef unsigned int max_dist_copy = max_dist

    path_copy = path.astype(long, order='C', casting='safe', copy=True)
                            ^
------------------------------------------------------------
src/transformers/models/deprecated/graphormer/algos_graphormer.pyx:88:28: undeclared name not builtin: long
```

This appears to have changed between cython==3.0 and cython==3.1.  AFAICT the correct type to use here would be `int`.  Switching to it makes the command succeed and generate an algos_graphormer.c file.

Comments:
I don't think we can diagnose `cython` bugs at all! In general there is very little point in running Transformers code with Cython because almost all of the performance-intensive work happens in `torch` anyway, which is already heavily optimized C++/CUDA code.

@Rocketknight1 Relevant to performance or not, it'd be unfortunate to leave this Cython file hanging around in a non-buildable state with the current Cython compiler.  I get that it's in the deprecated directory, but the fix seems simple enough.

From our end, we're not actually using this file but we do have an aggressive CI system that tries to build all Cython sources to validate they're in a good state.  I'm working around that internally but I figure it's worth fixing upstream.

Ah, I misread, I'm sorry! I didn't realize we actually had Cython files hiding in the `deprecated` folder of our codebase. My fault for skimming too quickly.

If it's causing you build trouble then a PR to replace `long` with `int` is welcome.

All this PR needs is an accept and a merge then :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41671). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update type hints in tokenization_utils.py to use | syntax
Author: faizan842
Pull request: yes

## What does this PR do?

This PR modernizes type hints in `modeling_rope_utils.py` to use Python 3.10+ `|` syntax instead of `Union` types where appropriate.

## Changes Made

- Replace `Union[float, dict[str, float]]` with `float | dict[str, float]`
- Remove unused `Union` import
- Maintain backward compatibility

## Testing

- ✅ All imports work correctly
- ✅ Rope utils functionality works
- ✅ No linting errors
- ✅ Maintains backward compatibility

## Related Issues

Similar to PRs #41675 and #41646 which also updated type hints to use `|` syntax.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41714). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM, thank you!

Waiting for your approval
Internvl3.5
Author: molbap
Pull request: yes

# What does this PR do?

WIP for the new iteration of InternVL3.5, see https://huggingface.co/OpenGVLab/InternVL3_5-241B-A28B/tree/main. Original authors are working on their own conversion to HF format (see https://huggingface.co/OpenGVLab/InternVL3_5-GPT-OSS-20B-A4B-Preview/discussions/2 ), pushing this for safekeeping and will likely delete! 

Current plan:
- [x] Conversation and generation for small, dense models work. 
- [x] Image processor not included, has dynamic preprocessing, looks similar to phi4 with an aspect ratio targeting. Needs to be added.
- [x] Generations for 20B/241B are TODO. Conversion (key mapping) works without errors. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40506). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Generations now work for gpt-oss based models, based on the updated conversion script.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: internvl, phi4_multimodal

Update, by using `Phi4MultimodalImageProcessorFast` we can directly convert and use the models with `InternVLForConditionalGeneration`.  

```python
from transformers import InternVLForConditionalGeneration, AutoProcessor

model_id = "/converted/model/path/"
image_path = "your/test/image"

model = InternVLForConditionalGeneration.from_pretrained(
    model_id, 
    dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_path},
            {"type": "text", "text": "Please describe the image shortly."}
        ],
    },
    ]

processed_chat = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(device=model.device, dtype=model.dtype)

response = model.generate(
    **processed_chat,
    do_sample=True,
    max_new_tokens=1024,
    )
dec = processor.batch_decode(response, skip_special_tokens=True)
print(dec)
```
Works for the 1B, 8B, MoE, etc.

Not needed as the checkpoints are now fully compatible with existing `InternVL` code!
Fixed incorrect model_type for qwen2vl and qwen2.5vl when config is saved and loaded again
Author: i3hz
Pull request: yes

# What does this PR do?
Fixes the issue where if you save the config and load it again it would return the incorrect model_type . 
Minor fix in __getattribute__ method of the config class for both models .


Fixes # 41746


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.  - https://github.com/huggingface/transformers/issues/41746
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@zucchini-nlp 
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
@ZX-ModelCloud  Please verify this. 

[ISSUE#41746](https://github.com/huggingface/transformers/issues/41746) Fixed in this PR.

Do I add it in the existing test_text_config function or make a new one? 

Let's add in existing, it's the same regression being tested

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2_5_vl, qwen2_vl

I've added the required tests

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41758). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.