[SAM 3 Video] Sam3VisionEncoderOutput object has no attribute 'fpn_position_embeddings'
Author: vydpnguyen
Pull request: no
Labels: name: bug

### System Info

Version of `transformers`: main branch (`5.0.0.dev0`)
Platform: `Linux-6.6.105+-x86_64-with-glibc2.35`
Python version: `3.12.12`

### Who can help?

@yonigozlan @molbap

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
- Follow this section of the SAM 3 tutorial: https://huggingface.co/facebook/sam3#basic-video-tracking
- Specifically, the error happens at this step:
```
# Get masks for both objects on first frame (optional, you can also propagate the masks through the video directly)
outputs = model(
    inference_session=inference_session,
    frame_idx=ann_frame_idx,
)
```
Error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[/tmp/ipython-input-3287582112.py](https://localhost:8080/#) in <cell line: 0>()
      1 # Segment the object on the first frame (optional, you can also propagate the masks through the video directly)
----> 2 outputs = model(
      3     inference_session=inference_session,
      4     frame_idx=ann_frame_idx,
      5 )

5 frames
[/usr/local/lib/python3.12/dist-packages/transformers/models/sam3_tracker_video/modeling_sam3_tracker_video.py](https://localhost:8080/#) in _prepare_vision_features(self, inference_session, frame_idx, batch_size)
   1910             image_outputs = self.get_image_features(image_batch, return_dict=True)
   1911             vision_feats = image_outputs.fpn_hidden_states
-> 1912             vision_pos_embeds = image_outputs.fpn_position_embeddings
   1913             # Cache features
   1914             inference_session.cache.cache_vision_features(
AttributeError: 'Sam3VisionEncoderOutput' object has no attribute 'fpn_position_embeddings'
```

### Expected behavior

The model should successfully segment the objects and no error should be returned.

Comments:
checking this issue, I will be back with analysis and will update my thought process here.

I was able to reproduce the error locally using the tutorial code. The failure happens in  `Sam3TrackerVideoModel._prepare_vision_features` when accessing positional encodings from the vision encoder output.

What’s happening:

`get_image_features()` returns a `Sam3VisionEncoderOutput` where FPN positional encodings are exposed as `fpn_position_encoding`. However , `_prepare_vision_features()` (inherited from the SAM2 video base) expects the field `fpn_position_embeddings` This attribute mismatch causes:

```
AttributeError: 'Sam3VisionEncoderOutput' object has no attribute 'fpn_position_embeddings'
```

The issue is inherited from the SAM2 modular base implementation (sam2_video/modular_sam2_video.py), which defines `_prepare_vision_features`. `Sam3TrackerVideoModel` inherits this method and exposes the mismatch when used with SAM3 vision outputs.

Why this breaks SAM3:
SAM3’s vision encoder uses `fpn_position_encoding` as the canonical field name, while the base implementation still assumes the older `fpn_position_embeddings` name. This results in a broken internal contract between the vision encoder output and the tracking pipeline.

Planned fix:
I’ll update _prepare_vision_features in the SAM2 modular base to accept both field names:
	•	Prefer `fpn_position_embeddings` when present (for backward compatibility)
	•	Fall back to `fpn_position_encoding` (current SAM3 behavior)

so currently `_prepare_vision_features` in `modular_sam2_video.py` which generates the modeling file has the following.

```
image_outputs = self.get_image_features(image_batch, return_dict=True)
vision_feats = image_outputs.fpn_hidden_states
vision_pos_embeds = image_outputs.fpn_position_embeddings
```

which needs to be updated to 

```
image_outputs = self.get_image_features(image_batch, return_dict=True)
vision_feats = image_outputs.fpn_hidden_states
vision_pos_embeds = getattr(image_outputs, "fpn_position_embeddings", None)
if vision_pos_embeds is None:
    vision_pos_embeds = image_outputs.fpn_position_encoding
```

when I did that in my local its falling back to encoding if embedding is not present and that test is successful.

```
(transformer) vaibhavpandey@Vaibhavs-MacBook-Pro transformers % python repro.py                          
Loading weights: 100%|█| 845/845 [00:00<00:00, 10025.17it/s, Materializing param=vision_encoder.neck.fpn_layers.3.proj2.w
Sam3TrackerVideoSegmentationOutput(object_ids=[1], pred_masks=tensor([[[[-13.5596, -14.2192, -11.1233,  ..., -12.6796, -10.7800, -11.6731],
          [-11.8958, -16.6555, -12.1935,  ..., -16.4319, -12.4089, -13.5725],
          [-12.9352, -12.5809, -13.2425,  ..., -15.7400, -14.4574, -15.8720],
          ...,
          [-12.3227, -12.2283, -13.4806,  ..., -14.5942, -12.5399, -12.6515],
          [-12.9265, -13.5490, -13.2400,  ..., -16.5094, -16.4889, -18.1760],
          [-11.4869, -11.8413, -11.6470,  ..., -14.3616, -11.9702, -13.4727]]]],
       device='mps:0'), object_score_logits=tensor([[18.2911]], device='mps:0'), frame_idx=0)
(transformer) vaibhavpandey@Vaibhavs-MacBook-Pro transformers % 
```

updating modular file in `sam2_video` and the updating all the modeling file in `sam2_video` and `sam3_tracker_video` will get the fix inherited.

@yonigozlan @molbap looking for your suggestion here if my thought process is good to proceed with a PR, I can also add a regression test to verify this. 

awaiting your response.

thanks

Seems like it's caused by a recent refactor, @tomaarsen . Since it's same model, we'll fix it together with https://github.com/huggingface/transformers/issues/43474 in a single PR
 

I think `fpn_position_embeddings` is not used anywhere, I meant to use `fpn_position_encoding`; I'll remove it in a simple PR. Thank you @vaibhav-research for doing the research on this! The two issues seem potentially unrelated, looking into it @zucchini-nlp.

- Tom Aarsen