Repository: huggingface/transformers
Pull request: fix(testing): Fix MoonshineEncoder UnboundLocalError and Florence2VisionBackbone dtype mismatch

### What does this PR do?

The following failing tests were identified and fixed in this PR:

→ **Moonshine:** In [MoonshineEncoder.forward](https://github.com/huggingface/transformers/blob/main/src/transformers/models/moonshine/modular_moonshine.py#L469), the var `output_attention_mask` is only assigned inside the [if attention_mask is not None](https://github.com/huggingface/transformers/blob/main/src/transformers/models/moonshine/modular_moonshine.py#L498-L502) clause, but [create_bidirectional_mask](https://github.com/huggingface/transformers/blob/main/src/transformers/models/moonshine/modular_moonshine.py#L504-L509) called immediately after with `attention_mask=None` still returns a non-None tensor. Under `torch.compile(dynamic=True)`, `is_tracing()` returns `True` inside [_ignore_bidirectional_mask_sdpa](https://github.com/huggingface/transformers/blob/main/src/transformers/masking_utils.py#L304) → [not is_tracing(padding_mask)](https://github.com/huggingface/transformers/blob/main/src/transformers/masking_utils.py#L325) fails and [_ignore_bidirectional_mask_sdpa](https://github.com/huggingface/transformers/blob/main/src/transformers/masking_utils.py#L304) returns `False`, so the [allow_is_bidirectional_skip early-exit in sdpa_mask](https://github.com/huggingface/transformers/blob/main/src/transformers/masking_utils.py#L502) clause never fires and a full batched tensor is materialized (shown in the trace). Speaking to the return-time guard, [attention_mask is not None](https://github.com/huggingface/transformers/blob/main/src/transformers/models/moonshine/modular_moonshine.py#L527) becomes truthy even when no padding mask was provided, leaving `output_attention_mask` unset. The aforementioned case is hit by [test_sdpa_can_compile_dynamic](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L3650-L3710) since it [drops all attention masks](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L3702-L3703) before calling the model and causes an `UnboundLocalError`; this change should fix that.

Here are the trace outputs that led to arrive at the above explanation:

```
→ MoonshineEncoder.forward:   input attention_mask=None
→ MoonshineEncoder.forward:   attention_mask is None; output_attention_mask NOT assigned
→ MoonshineEncoder.forward:   calling create_bidirectional_mask with attention_mask=None
→ _preprocess_mask_arguments: early_exit=False, returning attention_mask=None, q_length=1, kv_length=1
→ create_bidirectional_mask:  mask_interface=sdpa_mask
→ create_bidirectional_mask:  calling mask_interface with allow_is_bidirectional_skip=True, attention_mask=None
→ create_bidirectional_mask:  mask_interface returned type=Tensor, shape=torch.Size([3, 1, 1, 1])
→ MoonshineEncoder.forward:   guard 'attention_mask is not None' = True
→ MoonshineEncoder.forward:   about to return, evaluating guard now
>> **UnboundLocalError**
```

→ **Florence2Vision:** [test_sdpa_can_compile_dynamic](https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L3650-L3710) loads the model in `bfloat16`, but [Florence2VisionBackbone.forward](https://github.com/huggingface/transformers/blob/main/src/transformers/models/florence2/modular_florence2.py#L1360) got `float32` activations from upstream processing but ([its conv weights self.convs[0].conv.weight](https://github.com/harshaljanjani/transformers/blob/571aba84c255bc11fdf3736c2d9d6b50ffd7300a/src/transformers/models/florence2/modular_florence2.py#L1363-L1364)) were `bfloat16`. Casting the dtype fixes this :)

**[CI Failures:](https://github.com/huggingface/transformers/actions/runs/22747324764)**

<img width="700" height="800" alt="3" src="https://github.com/user-attachments/assets/5e623dbd-f8f7-4486-9859-a69e6601ad57" />

**Before the fix (feel free to cross-check; these errors are reproducible):**

<img width="700" height="800" alt="1" src="https://github.com/user-attachments/assets/0863d326-d56c-4a1d-ac82-1c592f09a40b" />

**After the fix (feel free to cross-check):**

<img width="700" height="800" alt="2" src="https://github.com/user-attachments/assets/de92b7cd-5416-4b54-b499-62b2c10d3610" /><br>

cc: @Rocketknight1

### Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you fix any necessary existing tests?

Changed files:
filename: src/transformers/models/florence2/modeling_florence2.py
patch: @@ -558,6 +558,8 @@ def __init__(self, config: Florence2VisionConfig):
     def forward(
         self, hidden_states: torch.Tensor, **kwargs: Unpack[TransformersKwargs]
     ) -> tuple | BaseModelOutputWithPooling:
+        target_dtype = self.convs[0].conv.weight.dtype
+        hidden_states = hidden_states.to(dtype=target_dtype)
         for conv, block in zip(self.convs, self.blocks):
             hidden_states = conv(hidden_states)
             for layer in block:

filename: src/transformers/models/florence2/modular_florence2.py
patch: @@ -1360,6 +1360,8 @@ def __init__(self, config: Florence2VisionConfig):
     def forward(
         self, hidden_states: torch.Tensor, **kwargs: Unpack[TransformersKwargs]
     ) -> tuple | BaseModelOutputWithPooling:
+        target_dtype = self.convs[0].conv.weight.dtype
+        hidden_states = hidden_states.to(dtype=target_dtype)
         for conv, block in zip(self.convs, self.blocks):
             hidden_states = conv(hidden_states)
             for layer in block:

filename: src/transformers/models/moonshine/modeling_moonshine.py
patch: @@ -578,6 +578,7 @@ def forward(
         hidden_states = hidden_states.permute(0, 2, 1)
 
         # attention mask downsampling
+        output_attention_mask = None
         if attention_mask is not None:
             mask_len = self._get_feat_extract_output_lengths(attention_mask.shape[-1])
             downsample_stride = 64 * 3 * 2  # conv strides
@@ -607,7 +608,7 @@ def forward(
 
         return MoonshineEncoderModelOutput(
             last_hidden_state=hidden_states,
-            attention_mask=output_attention_mask.int() if attention_mask is not None else None,
+            attention_mask=output_attention_mask.int() if output_attention_mask is not None else None,
         )
 
 

filename: src/transformers/models/moonshine/modular_moonshine.py
patch: @@ -495,6 +495,7 @@ def forward(
         hidden_states = hidden_states.permute(0, 2, 1)
 
         # attention mask downsampling
+        output_attention_mask = None
         if attention_mask is not None:
             mask_len = self._get_feat_extract_output_lengths(attention_mask.shape[-1])
             downsample_stride = 64 * 3 * 2  # conv strides
@@ -524,7 +525,7 @@ def forward(
 
         return MoonshineEncoderModelOutput(
             last_hidden_state=hidden_states,
-            attention_mask=output_attention_mask.int() if attention_mask is not None else None,
+            attention_mask=output_attention_mask.int() if output_attention_mask is not None else None,
         )