Repository: huggingface/transformers
Pull request: [`fix`] Sam3Video: Avoid fpn_position_embedding; use pooler_output

Resolves #43474
Resolves #43475

# What does this PR do?

* Removes `fpn_position_embeddings`, should have been `fpn_position_encoding` all along. Affected 3 architectures, and was introduced in #42564 3 days ago.
* Uses `...get_text_features(..., return_dict=True).pooler_output` format, this is required since #42564

## Details

### Regarding `fpn_position_embeddings`
This should never have existed, it was a simply typo on my side, not caught due to the lack of tests for these models. After this change, `fpn_position_embeddings` exists nowhere in the entire repository.

### Regarding `'CLIPTextModelOutput' object has no attribute 'shape'`
I've been able to reproduce #43474 with this script:
```python
from transformers import Sam3VideoModel, Sam3VideoProcessor
from accelerate import Accelerator
import torch

device = Accelerator().device
model = Sam3VideoModel.from_pretrained("facebook/sam3").to(device, dtype=torch.bfloat16)
processor = Sam3VideoProcessor.from_pretrained("facebook/sam3")

# Load video frames
from transformers.video_utils import load_video
video_url = "https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/bedroom.mp4"
video_frames, _ = load_video(video_url)

# Initialize video inference session
inference_session = processor.init_video_session(
    video=video_frames,
    inference_device=device,
    processing_device="cpu",
    video_storage_device="cpu",
    dtype=torch.bfloat16,
)

# Add text prompt to detect and track objects
text = "person"
inference_session = processor.add_text_prompt(
    inference_session=inference_session,
    text=text,
)

# Process all frames in the video
outputs_per_frame = {}
for model_outputs in model.propagate_in_video_iterator(
    inference_session=inference_session, max_frame_num_to_track=50
):
    processed_outputs = processor.postprocess_outputs(inference_session, model_outputs)
    outputs_per_frame[model_outputs.frame_idx] = processed_outputs

print(f"Processed {len(outputs_per_frame)} frames")

# Access results for a specific frame
frame_0_outputs = outputs_per_frame[0]
print(f"Detected {len(frame_0_outputs['object_ids'])} objects")
print(f"Object IDs: {frame_0_outputs['object_ids'].tolist()}")
print(f"Scores: {frame_0_outputs['scores'].tolist()}")
print(f"Boxes shape (XYXY format, absolute coordinates): {frame_0_outputs['boxes'].shape}")
print(f"Masks shape: {frame_0_outputs['masks'].shape}")
```

And I can confirm that it works after this PR:
```
Processed 51 frames
Detected 2 objects
Object IDs: [0, 1]
Scores: [0.96875, 0.9765625]
Boxes shape (XYXY format, absolute coordinates): torch.Size([2, 4])
Masks shape: torch.Size([2, 540, 960])
```

This was a case of me missing a `get_text_features` use, likely because the architecture itself doesn't implement one of these methods, but only uses it from another architecture. There's a chance this exists in more architectures, sadly. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@zucchini-nlp @ArthurZucker 

- Tom Aarsen


Changed files:
filename: src/transformers/models/edgetam_video/modeling_edgetam_video.py
patch: @@ -2280,7 +2280,7 @@ def _prepare_vision_features(
             image_batch = inference_session.get_frame(frame_idx).unsqueeze(0)  # Add batch dimension
             image_outputs = self.get_image_features(image_batch, return_dict=True)
             vision_feats = image_outputs.fpn_hidden_states
-            vision_pos_embeds = image_outputs.fpn_position_embeddings
+            vision_pos_embeds = image_outputs.fpn_position_encoding
             # Cache features
             inference_session.cache.cache_vision_features(
                 frame_idx, {"vision_feats": vision_feats, "vision_pos_embeds": vision_pos_embeds}

filename: src/transformers/models/sam2_video/modeling_sam2_video.py
patch: @@ -1884,7 +1884,7 @@ def _prepare_vision_features(
             image_batch = inference_session.get_frame(frame_idx).unsqueeze(0)  # Add batch dimension
             image_outputs = self.get_image_features(image_batch, return_dict=True)
             vision_feats = image_outputs.fpn_hidden_states
-            vision_pos_embeds = image_outputs.fpn_position_embeddings
+            vision_pos_embeds = image_outputs.fpn_position_encoding
             # Cache features
             inference_session.cache.cache_vision_features(
                 frame_idx, {"vision_feats": vision_feats, "vision_pos_embeds": vision_pos_embeds}

filename: src/transformers/models/sam2_video/modular_sam2_video.py
patch: @@ -1536,7 +1536,7 @@ def _prepare_vision_features(
             image_batch = inference_session.get_frame(frame_idx).unsqueeze(0)  # Add batch dimension
             image_outputs = self.get_image_features(image_batch, return_dict=True)
             vision_feats = image_outputs.fpn_hidden_states
-            vision_pos_embeds = image_outputs.fpn_position_embeddings
+            vision_pos_embeds = image_outputs.fpn_position_encoding
             # Cache features
             inference_session.cache.cache_vision_features(
                 frame_idx, {"vision_feats": vision_feats, "vision_pos_embeds": vision_pos_embeds}

filename: src/transformers/models/sam3_tracker_video/modeling_sam3_tracker_video.py
patch: @@ -1909,7 +1909,7 @@ def _prepare_vision_features(
             image_batch = inference_session.get_frame(frame_idx).unsqueeze(0)  # Add batch dimension
             image_outputs = self.get_image_features(image_batch, return_dict=True)
             vision_feats = image_outputs.fpn_hidden_states
-            vision_pos_embeds = image_outputs.fpn_position_embeddings
+            vision_pos_embeds = image_outputs.fpn_position_encoding
             # Cache features
             inference_session.cache.cache_vision_features(
                 frame_idx, {"vision_feats": vision_feats, "vision_pos_embeds": vision_pos_embeds}

filename: src/transformers/models/sam3_video/modeling_sam3_video.py
patch: @@ -590,7 +590,8 @@ def run_detection(
                 text_embeds = self.detector_model.get_text_features(
                     input_ids=inference_session.prompt_input_ids[prompt_id],
                     attention_mask=inference_session.prompt_attention_masks[prompt_id],
-                )
+                    return_dict=True,
+                ).pooler_output
                 inference_session.prompt_embeddings[prompt_id] = text_embeds
             else:
                 text_embeds = inference_session.prompt_embeddings[prompt_id]