Repository: huggingface/transformers
Pull request: [Bug] qwen2_5_omni: cap generation length to be less than the max_position_embedding in DiT

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR resolved the following issue and added tests and provided a test script to validate the behavior before/after the fix to further validate the correctness. 
Issue: mel frames of noise_initialization is hardcoded to `30000` in current implmentation, it will cause issue of 'shape mismatch' if the hidden_states time len doesn't match that of condition_vector/code_embed tensors. i.e, We will get the below error when we run the script attached.
```
  File "/root/transformers/test_before.py", line 20, in <module>
    m.sample(cond, ref_mel, code, num_steps=2)
  File "/root/transformers/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3718, in sample
    solution_trajectory = ode_solver.integrate(time_embedding)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3550, in integrate
    delta_value, _ = self._compute_step(self.function, time_start, time_step, time_end, current_value)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3524, in _compute_step
    function_value_start = function(time_start, value_start)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3698, in ode_function
    model_output = self(
                   ^^^^^
  File "/root/transformers/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 3636, in forward
    hidden_states = self.input_embed(
                    ^^^^^^^^^^^^^^^^^
  File "/root/transformers/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/transformers/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py", line 2975, in forward
    hidden_states = self.proj(torch.cat((hidden_states, condition_vector, code_embed, speaker_embedding), dim=-1))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 30000 but got size 32000 for tensor number 2 in the list.
```
The bug is found when I used vllm-omni to run the exact qwen 2.5 omni mode and [a similar PR  ](https://github.com/vllm-project/vllm-omni/pull/543) has been merged to resolve the issue.
<!--
### To produce the bug: run this script on the main branch
[test_before.py](https://github.com/user-attachments/files/24402896/test_before.py)
result error logs: 
[before_fix.log](https://github.com/user-attachments/files/24402897/before_fix.log)

### To verify the issue has been resolved by this change: run this script on the current fix branch
[test_after.py](https://github.com/user-attachments/files/24402905/test_after.py)
result logs show it's been fixed:
[after_fix.log](https://github.com/user-attachments/files/24402901/after_fix.log)
-->

### To validate the issue exists before the fix ang get resolved after the fix. Run the following script:

[test_validate_max_position_embeddings_fix.py](https://github.com/user-attachments/files/24447776/test_validate_max_position_embeddings_fix.py)

All the re-production and validation is running on a B300 GPU.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Changed files:
filename: src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py
patch: @@ -3676,15 +3676,24 @@ def sample(
         guidance_scale=0.5,
         sway_coefficient=-1.0,
     ):
-        noise_initialization = torch.randn([1, 30000, self.mel_dim], dtype=reference_mel_spectrogram.dtype)
         maximum_duration = quantized_code.shape[1] * self.repeats
-        initial_state = noise_initialization[:, :maximum_duration].to(quantized_code.device)
         batch_size = reference_mel_spectrogram.shape[0]
-        conditioning_vector = conditioning_vector.unsqueeze(1).repeat(1, maximum_duration, 1)
-
         if batch_size != 1:
             raise ValueError("Only batch size = 1 is currently supported")
 
+        if maximum_duration > self.config.max_position_embeddings:
+            raise ValueError(
+                f"Requested mel length ({maximum_duration}) exceeds `dit_config.max_position_embeddings` "
+                f"({self.config.max_position_embeddings}). Provide shorter `quantized_code`."
+            )
+
+        initial_state = torch.randn(
+            [batch_size, maximum_duration, self.mel_dim],
+            dtype=reference_mel_spectrogram.dtype,
+            device=quantized_code.device,
+        )
+        conditioning_vector = conditioning_vector.unsqueeze(1).repeat(1, maximum_duration, 1)
+
         def ode_function(time_step, hidden_states):
             if guidance_scale < 1e-5:
                 prediction = self(
@@ -3695,6 +3704,7 @@ def ode_function(time_step, hidden_states):
                     time_step=time_step,
                     drop_audio_conditioning=False,
                     drop_code=False,
+                    apply_cfg=False,
                 )
                 return prediction
 

filename: src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py
patch: @@ -3835,15 +3835,24 @@ def sample(
         guidance_scale=0.5,
         sway_coefficient=-1.0,
     ):
-        noise_initialization = torch.randn([1, 30000, self.mel_dim], dtype=reference_mel_spectrogram.dtype)
         maximum_duration = quantized_code.shape[1] * self.repeats
-        initial_state = noise_initialization[:, :maximum_duration].to(quantized_code.device)
         batch_size = reference_mel_spectrogram.shape[0]
-        conditioning_vector = conditioning_vector.unsqueeze(1).repeat(1, maximum_duration, 1)
-
         if batch_size != 1:
             raise ValueError("Only batch size = 1 is currently supported")
 
+        if maximum_duration > self.config.max_position_embeddings:
+            raise ValueError(
+                f"Requested mel length ({maximum_duration}) exceeds `dit_config.max_position_embeddings` "
+                f"({self.config.max_position_embeddings}). Provide shorter `quantized_code`."
+            )
+
+        initial_state = torch.randn(
+            [batch_size, maximum_duration, self.mel_dim],
+            dtype=reference_mel_spectrogram.dtype,
+            device=quantized_code.device,
+        )
+        conditioning_vector = conditioning_vector.unsqueeze(1).repeat(1, maximum_duration, 1)
+
         def ode_function(time_step, hidden_states):
             if guidance_scale < 1e-5:
                 prediction = self(
@@ -3854,6 +3863,7 @@ def ode_function(time_step, hidden_states):
                     time_step=time_step,
                     drop_audio_conditioning=False,
                     drop_code=False,
+                    apply_cfg=False,
                 )
                 return prediction
 

filename: tests/models/qwen2_5_omni/test_modeling_qwen2_5_omni.py
patch: @@ -859,3 +859,100 @@ def test_small_model_integration_test_batch_flashatt2(self):
         decoded_texts = self.processor.batch_decode(output, skip_special_tokens=True)
         self.assertEqual(decoded_texts[0], EXPECTED_DECODED_TEXT)
         self.assertEqual(decoded_texts[1], EXPECTED_DECODED_TEXT)
+
+
+@require_torch
+class Qwen2_5OmniToken2WavMaxPositionEmbeddingsTest(unittest.TestCase):
+    """
+    Tests to verify that ValueError is raised when input length exceeds max_position_embeddings.
+    """
+
+    @classmethod
+    def setUpClass(cls):
+        """Create minimal DiT model config for testing - shared across all tests."""
+        from transformers.models.qwen2_5_omni.configuration_qwen2_5_omni import Qwen2_5OmniDiTConfig
+
+        # Use minimal dimensions to reduce memory usage
+        # Note: enc_channels needs at least 3 elements for the ECAPA-TDNN encoder architecture
+        cls.config = Qwen2_5OmniDiTConfig(
+            hidden_size=32,
+            num_hidden_layers=1,
+            num_attention_heads=2,
+            head_dim=16,
+            ff_mult=1,
+            emb_dim=16,
+            mel_dim=16,
+            enc_emb_dim=16,
+            enc_dim=16,
+            enc_channels=[16, 16, 16],
+            enc_kernel_sizes=[3, 3, 1],
+            enc_dilations=[1, 1, 1],
+            enc_attention_channels=8,
+            enc_res2net_scale=2,
+            enc_se_channels=8,
+            num_embeds=100,
+            look_ahead_layers=[],
+            look_backward_layers=[0],
+            max_position_embeddings=100,  # Small for testing
+            block_size=24,
+            repeats=2,
+        )
+
+    def setUp(self):
+        """Create model instance for each test."""
+        from transformers.models.qwen2_5_omni.modeling_qwen2_5_omni import Qwen2_5OmniToken2WavDiTModel
+
+        self.model = Qwen2_5OmniToken2WavDiTModel(self.config).to(torch_device)
+        self.model.eval()
+
+    def tearDown(self):
+        """Clean up model to free memory."""
+        del self.model
+        if torch.cuda.is_available():
+            torch.cuda.empty_cache()
+
+    def test_error_when_exceeding_max_position_embeddings(self):
+        """Verify ValueError is raised when maximum_duration > max_position_embeddings."""
+        batch_size = 1
+        # With repeats=2 and max_position_embeddings=100, we need > 50 tokens to exceed
+        num_speech_tokens = 60  # Will result in 120 mel frames, exceeds max_position_embeddings=100
+
+        conditioning_vector = torch.randn(batch_size, self.config.enc_emb_dim, device=torch_device)
+        reference_mel = torch.randn(batch_size, 200, self.config.mel_dim, device=torch_device)
+        quantized_code = torch.randint(0, self.config.num_embeds, (batch_size, num_speech_tokens), device=torch_device)
+
+        with self.assertRaises(ValueError) as context:
+            self.model.sample(
+                conditioning_vector=conditioning_vector,
+                reference_mel_spectrogram=reference_mel,
+                quantized_code=quantized_code,
+                num_steps=2,
+            )
+
+        self.assertIn("exceeds `dit_config.max_position_embeddings`", str(context.exception))
+        self.assertIn("120", str(context.exception))  # Requested mel length
+        self.assertIn("100", str(context.exception))  # max_position_embeddings
+
+    def test_no_error_when_within_limits(self):
+        """Verify no error when maximum_duration <= max_position_embeddings."""
+        batch_size = 1
+        # With repeats=2 and max_position_embeddings=100, 50 tokens = 100 mel frames (exactly at limit)
+        num_speech_tokens = 50
+
+        conditioning_vector = torch.randn(batch_size, self.config.enc_emb_dim, device=torch_device)
+        reference_mel = torch.randn(batch_size, 200, self.config.mel_dim, device=torch_device)
+        quantized_code = torch.randint(0, self.config.num_embeds, (batch_size, num_speech_tokens), device=torch_device)
+
+        # Should complete without error
+        output = self.model.sample(
+            conditioning_vector=conditioning_vector,
+            reference_mel_spectrogram=reference_mel,
+            quantized_code=quantized_code,
+            num_steps=2,
+        )
+
+        # Check output shape is valid
+        self.assertEqual(len(output.shape), 3)
+        self.assertEqual(output.shape[0], batch_size)
+        self.assertEqual(output.shape[1], self.config.mel_dim)
+        self.assertEqual(output.shape[2], 100)  # 50 tokens * 2 repeats