noamaanMulla-03/transformers-issues huggingface dataset:
Fix TypeError when loading adapter models with _adapter_model_path
Author: jameslovespancakes
Pull request: yes

# What does this PR do?

This PR resolves a TypeError that occurs when loading models with LoRA adapters (like ibm-granite/granite-speech-3.3-2b).

## Issue
When calling `AutoModelForSpeechSeq2Seq.from_pretrained()` on models with LoRA adapters, the following error occurs:
TypeError: find_adapter_config_file() got an unexpected keyword argument '_adapter_model_path'

## Root Cause
In `auto_factory.py` line 302, `_adapter_model_path` is added to `adapter_kwargs`. Later, in `peft.py` line 221, these `adapter_kwargs` are passed to `find_adapter_config_file()` via `**adapter_kwargs`, but `find_adapter_config_file()` doesn't accept `_adapter_model_path` as a parameter.

## Solution
Remove `_adapter_model_path` from `adapter_kwargs` in `peft.py` before passing it to `find_adapter_config_file()`.

## Testing
Verified the fix resolves the TypeError when loading `ibm-granite/granite-speech-3.3-2b`.

Fixes #41760

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@BenjaminBossan @Cyrilvallez @ArthurZucker

Comments:
Thanks @jameslovespancakes but it's a duplicate of #41604.
🚀 Optimize MoE and Mamba performance with vectorized operations
Author: faizan842
Pull request: yes

## Performance Optimization

This PR optimizes two major performance bottlenecks in the transformers library:

### 🔍 Problems Identified

1. **Aria MoE Sequential GEMM**: The sequential_experts_gemm function used inefficient sequential processing for expert computations
2. **Jamba Mamba Slow Forward**: The slow_forward method had a sequential loop in the state space model computation

### ⚡ Solutions Implemented

#### **Aria MoE Optimization**
- **Vectorized Expert Selection**: Use advanced indexing instead of sequential loops
- **Batch Matrix Multiplication**: Use torch.bmm for efficient computation
- **Eliminated Sequential Processing**: Replace O(n) sequential operations with vectorized operations

#### **Jamba Mamba Optimization**  
- **Vectorized State Computation**: Pre-allocate output tensors and use vectorized operations
- **Optimized Matrix Operations**: Use torch.bmm for batch matrix multiplication
- **Reduced Memory Allocations**: Eliminate repeated tensor creation in loops

### 📊 Performance Improvements

- **Aria MoE**: Significant speedup for Mixture of Experts models with many experts
- **Jamba Mamba**: Faster state space model computation for long sequences
- **GPU Utilization**: Better utilization of parallel processing capabilities
- **Memory Efficiency**: Reduced memory allocations and better cache usage

### 🎯 Impact

- **All MoE models** benefit from the Aria optimization
- **All Mamba-based models** benefit from the Jamba optimization  
- **Long sequences** see the most improvement
- **Backward compatible** - no API changes
- **Production ready** - maintains exact same functionality

### 🧪 Testing

- ✅ All existing tests pass
- ✅ Results are identical to original implementation
- ✅ Vectorized operations tested and verified
- ✅ Both modeling and modular files updated

### 📁 Files Changed

- src/transformers/models/aria/modeling_aria.py - Optimized MoE GEMM
- src/transformers/models/aria/modular_aria.py - Optimized MoE GEMM  
- src/transformers/models/jamba/modeling_jamba.py - Optimized Mamba forward
- src/transformers/models/jamba/modular_jamba.py - Optimized Mamba forward

This optimization addresses critical performance bottlenecks in modern transformer architectures and will significantly improve the user experience for MoE and Mamba models.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, jamba
[device_map] Accelerate loading by computing device_map much faster
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

If a model has a lot of different parameters (which is the case for models with a lot of experts, such as Qwen3Next), the computation of the device_map itself (yes, just the mapping of modules to devices) is EXTREMELY slow when using `accelerate`.
This is because they traverse the model graph in a very non-optimized manner, resulting in quadratic complexity in the number of parameters and modules, when we could do it in linear time instead. For example, `Qwen/Qwen3-Next-80B-A3B-Instruct` has a total of `173529` parameters and intermediate modules, which is a a lot. So doing the following:

```python
from transformers import AutoConfig, Qwen3NextForCausalLM
from transformers.modeling_utils import _get_device_map
import torch
import time

model_name = "Qwen/Qwen3-Next-80B-A3B-Instruct"
config = AutoConfig.from_pretrained(model_name)

with torch.device("meta"):
    model = Qwen3NextForCausalLM(config)

t0 = time.time()
device_map = _get_device_map(model, device_map="auto", dtype=torch.bfloat16, max_memory=None, hf_quantizer=None, keep_in_fp32_regex=None)
dt = time.time() - t0
print(f"Time needed: {dt:.2f} s")

>>> Time needed: 60.02 s
```

takes un UNBEARABLE `60.02` s on my Macbook Pro M3. And this is only for the device_map computation, the loading has not even started yet...

This PR fixes the issue by rewriting the needed functions from `accelerate` with much better complexity and operations directly in Transformers, leading to linear time and the following:

```python
>>> Time needed: 1.33 s
```

So effectively going 60x faster from 1min to 1s

The device_map created are the exact same.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41548). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

FYI, we have this small improvement in accelerate https://github.com/huggingface/accelerate/pull/3776 but nice that you manage to optimize further
Simplify pipeline padding logic
Author: Rocketknight1
Pull request: yes

Follow-up to #41636 

When reviewing that PR I noticed the padding logic was cluttered and much more complex than it needed to be. We can eliminate a lot of code by just constructing shapes as lists and trusting in implicit slice dimensions

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41667). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use the process_group from device_mesh to all_reduce in GatherParallel
Author: y-sq
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

In`GatherParallel`, where the `device_mesh` is passed, but not used by the `all_reduce` call. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@3outeille @ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.




Comments:
Since we handle only 1D parallelism for now, the `all_reduce` will use by default the default process group. So we dont need to pass it explicitly

However, as we move on later to support ND parallelism, the `pg = device_mesh.get_group(mesh_dim=0)` is not correct because we don't know if `mesh_dim=0` means TP axis or DP axis for example. Plus, you added an `assert` to enforce 1D parallelism, so this is the same as letting torch uses the default process group. So I dont think the patch is necessary for now

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41679). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @arthurzucker for tensor parallel
Add safety infrastructure
Author: jameslovespancakes
Pull request: yes

# What does this PR do?

This PR adds a comprehensive safety checking infrastructure for text generation, enabling built-in content moderation similar to production LLMs (ChatGPT, Gemini, Claude).

## Issue

Open-source text generation models lack built-in safety moderation infrastructure. While production LLMs have safety systems, implementing effective moderation during inference is non-trivial for Transformers users. There's currently no native infrastructure to support this.

## Solution

This PR implements:
- **Abstract SafetyChecker base class** for pluggable safety implementations
- **KeywordSafetyChecker** reference implementation with keyword/pattern blocking
- **SafetyStoppingCriteria** to halt generation on safety violations
- **SafetyLogitsProcessor** to filter unsafe token continuations
- **GenerationConfig integration** with automatic processor/criteria instantiation
- **Comprehensive safety parameters** (check frequency, penalty values, violation handling)

## Key Features

- ✅ Automatic integration - just configure GenerationConfig, no manual setup
- ✅ String-based or instance-based safety checker specification
- ✅ Configurable modes (stop on violation, filter violations, or both)
- ✅ Performance tuning via check frequency
- ✅ Extensible architecture for custom safety checkers
- ✅ Batch processing support

## Example Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Automatic safety integration
config = GenerationConfig(
    safety_checker="keyword",
    safety_checker_kwargs={"blocked_keywords": ["unsafe", "explicit"]},
    safety_stop_on_violation=True,
    safety_filter_violations=True,
)

outputs = model.generate(**inputs, generation_config=config, tokenizer=tokenizer)
Testing
25 comprehensive tests covering all components
Integration tests verify automatic GenerationMixin integration
Tests cover keyword checking, custom checkers, stopping criteria, logits processing
Fixes #41740
Before submitting
 This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
 Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
 Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. - Yes, #41740
 Did you make sure to update the documentation with your changes?
 Did you write any new necessary tests? - Yes, 25 tests



Comments:
Fix qwen3_vl mix precision dtype
Author: theruiwang
Pull request: yes

`fast_pos_embed_interpolate` returns `pos_embeds` in the same dtype as the master weights.

Therefore, when the master weights are in FP32 but the forward pass runs in BF16, `hidden_states` will be upcast to FP32, causing dtype mismatches with other activations.

CC @yonigozlan @molbap @zucchini-nlp

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_omni_moe, qwen3_vl, qwen3_vl_moe
Add LightGlue fast image processor
Author: yonigozlan
Pull request: yes

# What does this PR do?

As the title says

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41670). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lightglue

Thanks for the review @molbap ! It should be good now :)
Fix ImportError: cannot import name 'GenerationMixin' from 'transformers.generation'
Author: sij411
Pull request: yes

Fixes #36010

Fixed codes to import GenerationMixin from .generation.utils

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/huggingface/transformers/pull/36011?shareId=fd459388-d4da-40d5-b022-c1328541d538).

Comments:
cc @gante 

Hi @sij411 👋 

I can't reproduce your issue in a fresh environment. The following script is working:

```py
from transformers import GenerationMixin
from transformers.generation import GenerationMixin
from transformers.generation.utils import GenerationMixin
```

It's possible there is some issue with your environment 🤗 

> Hi @sij411 👋 
> 
> 
> 
> I can't reproduce your issue in a fresh environment. The following script is working:
> 
> 
> 
> ```py
> 
> from transformers import GenerationMixin
> 
> from transformers.generation import GenerationMixin
> 
> from transformers.generation.utils import GenerationMixin
> 
> ```
> 
> 
> 
> It's possible there is some issue with your environment 🤗 

Thanks for answering. I'll look it up mine. Have a lovely day.

I haven't spent too much time as I have been using/installing this library successfully. But today I ended up getting this error. So just want to mention that there is a scenario where this issue is triggered.

how it is going? I could replicate this with uv
Fix bark after #41445
Author: ydshieh
Pull request: yes

# What does this PR do?

Some Bark model classes are not `GenerationMixin` and have their own `generation` method.

#41445 makes change to `GenerationMixin` and those changes are not applied to those Bark model classes, then causes the bark tests to fail. One example is

> tests/models/bark/test_modeling_bark.py::BarkModelIntegrationTests::test_generate_batching

This PR adds `GenerationMixin` to those models and the tests pass now.




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41645). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> bark model is not semantically a gernerate model

I agree, but it has a `generate` method which makes it even confused ...

(and it definitely try to use generation config )

As @gante wrote in the codebase,

```
    def can_generate(cls) -> bool:
        # Bark has a unique model structure, where the external class (`BarkModel`) doesn't need to inherit from
        # `GenerationMixin` (it has a non-standard generation method), but one of the internal models do
        # (`BarkSemanticModel`). This means that the base `can_generate()` will return `False`, but we need to
        # override it so as to do `GenerationConfig` handling in multiple parts of the codebase.
        return True
```

we see `BarkModel` is: 

  - has `generate` method (although previously not inherits from `GenerationMixin`)
  - set `can_generate = True` in order to handle `GenerationConfig`

So after the change #41445, the fix of the failing tests in this PR seems to be natural IMO.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark

Merged as approved offline on slack
Add safety checking infrastructure for text generation
Author: rice-e
Pull request: yes

**Draft PR:** Core implementation complete. Seeking feedback on design and approach before finalizing. Thank you!

# What does this PR do?

Adds safety checking infrastructure for text generation. Provides base classes, configuration, and processors that integrate with the generation pipeline. Users implement their own safety checkers for specific needs (harm, bias, PII, etc.).

Fixes #41740

# Motivation
As stated in the issue I opened, while production LLMs have built-in safety moderation systems, they are often insufficient and can lead to unexpected harmful behavior, especially over long conversations. As open-source text generation models become more capable and widely used, mitigating harm and ensuring user safety is a feature that should be built in. As far as I am aware, there is currently no built-in infrastructure to support this. The most effective approaches involve moderation during inference, which is a non-trivial feature for Transformers users to implement on their own. In addition, allowing for the configuration of safety with custom settings and classifiers can allow users to avoid harm in more specialized contexts than commercial LLMs currently address.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Currently being discussed in #41740
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
Safety Checking Infrastructure for Text Generation
Author: rice-e
Pull request: no
Labels: name: Feature request

### Feature request

Add moderation to text generation models, similar to the approaches used in production LLMs (e.g. ChatGPT, Gemini, Claude) so that open-source models can have built-in content filtering.

This could provide
- Abstract base class for implementing safety checkers/classifiers
- Integration hooks during generation (logits processing and stopping criteria)
- Pipeline support for safe generation
- Configuration for safety settings

### Motivation

While production LLMs have built-in safety moderation systems, they are often insufficient and can lead to unexpected harmful behavior, especially over long conversations. As open-source text generation models become more capable and widely used, mitigating harm and ensuring user safety is a feature that should be built in. As far as I am aware, there is currently no built-in infrastructure to support this. The most effective approaches involve moderation during inference, which is a non-trivial feature for Transformers users to implement on their own. In addition, allowing for the configuration of safety with custom settings and classifiers can allow users to avoid harm in more specialized contexts than commercial LLMs currently address.

### Your contribution

I would like to work on a PR for this feature if there is interest. It seems that Diffusers' `StableDiffusionSafetyChecker` provides precedent, and `LogitsProcessor` and `StoppingCriteria` would support the development of this feature.

Comments:
hi, I'd like to be assigned this issue if possible.


@Tejassveer08 thank you for your interest, but this isn't quite what I had in mind. If any maintainers have comments on whether the full feature (with a safety checker base class, processors, pipeline support, etc.) could be supported in Transformers, I would be happy to work on this and provide my own implementation soon.

hey i'd like to work on this issue

Hmn - cc @gante maybe, but let me know if you want me to refer it to someone else instead!

Thank you, @Rocketknight1 and @gante! I've opened my draft PR (#41776) with the implementation I've been developing for this feature. Looking forward to your feedback!
fix: Gemma 3 weights conversion vision and multimodal projector paths
Author: RyanMullins
Pull request: yes

# What does this PR do?

Fixes bugs in the Gemma 3 weights conversion script:

* Looking for multi-modal projector weights at the wrong location
* Had the wrong prefix on the vision model weights


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp


Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41767). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[quantization] fix compressed_tensors tests
Author: MekkCyber
Pull request: yes

# What does this PR do?

Fixing compressed_tensors related tests. Failling tests are passing locally

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: compressed_tensors_integration

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41780). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.