SAM 3: Segment Anything with Concepts
Authors: Nicolas Carion
Laura Gustafson
Yuan-Ting Hu
Shoubhik Debnath
Ronghang Hu
Didac Suris Coll-Vinent
Chaitanya Ryali
Kalyan Vasudev Alwala
Haitham Khedr
Andrew Huang
Jie Lei
Tengyu Ma
Baishan Guo
Arpit Kalla
Markus Marks
Joseph Greer
Meng Wang
Peize Sun
Roman Rädle
Triantafyllos Afouras
Effrosyni Mavroudi
Katherine Xu
Tsung-Han Wu
Yu Zhou
Liliane Momeni
RISHI HAZRA
Shuangrui Ding
Sagar Vaze
Francois Porcher
Feng Li
Siyuan Li
Aishwarya Kamath
Ho Kei Cheng
Piotr Dollar
Nikhila Ravi
Kate Saenko
Pengchuan Zhang
Christoph Feichtenhofer
Venue: ICLR 2026 Poster (2026)
Keywords: foundation models
open vocabulary segmentation
semantic instance segmentation
object tracking
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a strong model and challenging benchmark to advance open-vocabulary concept segmentation in images and videos.

Abstract: We present Segment Anything Model (SAM) 3, a unified model that detects,
segments, and tracks objects in images and videos based on concept prompts,
which we define as either short noun phrases (e.g., “yellow school bus”), image
exemplars, or a combination of both. Promptable Concept Segmentation (PCS)
takes such prompts and returns segmentation masks and unique identities for all
matching object instances. To advance PCS, we build a scalable data engine that
produces a high-quality dataset with 4M unique concept labels, including hard
negatives, across images and videos. Our model consists of an image-level detector
and a memory-based video tracker that share a single backbone. Recognition and
localization are decoupled with a presence head, which boosts detection accuracy.
SAM 3 doubles the accuracy of existing systems in both image and video PCS,
and improves previous SAM capabilities on visual segmentation tasks. We open
source SAM 3 along with our new Segment Anything with Concepts (SA-Co)
benchmark for promptable concept segmentation.

Published as a conference paper at ICLR 2026
SAM 3: SEGMENT ANYTHING WITH Concepts
Nicolas Carion", Laura Gustafson", Yuan-Ting Hu",
Shoubhik Debnath"
Ronghang Hu", Didae Suris",
Chaitanya Ryali",
Kalyan Vasudev Alwala"
Haitham Khedr", Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle,
Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu Tsung-Han Wu', Yu Zhou', Liliane Momeni', Ríshi Hazra², Shuangrui Ding', Sagar Vaze", Francois Porcher', Feng Li Siyuan Li", Nikhila Ravi', Kate Saenko', Pengchuan Zhangi, Christoph Feichtenhofer
Aishwarya Kamath', Ho Kei Cheng", Piotr Dollár),
Meta Superintelligence Labs
https://ai http://aaamaaaaaaaaaaeemm meta. com/sams
ABSTRACT
We present Segment Anything Model (SAM) 3, unified model that detects, segments, and tracks objects in images and videos based concept prompts, which we define as either short noun phrases (c.g., "yellow school bus"), image exemplars. combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, build scalable data engine that produces high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory- baased video tracker that share single backbone. Recogmition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.
INTRODUCTION
The ability to find and segment anything visual scene is foundational for multimodal AI, powering applications in robotics, content creation, augmented reality, data annotation, and broader sciences. The SAM series (Kirillov et al. 2023 Ravi et al.) 2024) introduced the promptable segmentation task for images and videos, focusing on Promptable Visual Segmentation (PVS) with points, boxes or masks to segment single object per prompt. While these methods achieved a breakthrough, they did not address the general task of finding and segmenting all instances of concept appearing anywhere in the input (c.g., all "cats" in a video).
core contributor, intern, ppooecct lead, order is random within groups
PROMPTABLE VISUAL SEGMENTATION
PROMPTABLE CONCEPT SEGMENTATION
striped
round cell
small window
IMAGE
kangaroo
VIDEO
Prompts: positive positive
nepaive
points
Prompts:
noun phrase
and/or positive
negative
image exemplar
Figure I: SAM improves SAM promptable visual segmentation with clicks (left) and introduces the new promptable concept segmentation capability (right). Users can segment instances a visual concept specified by a short noun phrase, image exemplars (positive or negative), combination of both.

---

Published as a conference paper at ICLR 2026
hand towel
toilet holder
white flower
young plant
Adirondack chair
cheesecloth
earring
rowboat
colander
SAM ORIGINAL
OWL2Z
Figure Examples of SAM improving segmentation of vpcabbocabulary concepts compared OWLv2 Miin derer et al. 2024), on the SA-Co benchmark. See $G.6. for additional SAM outputs.
To fill this gap, we present SAM 3, a model that achieves a step change in promptable segmentation in images and videos, improving PVS relative to SAM 2 and setting new standard for Promptable Concept Segmentation (PCS). We formalize the PCS task ( as taking text and/or image exemplars as input, and predicting instance and semantic masks for every single object matching the concept, while preserving object identities across video frames (see Fig. D To focus on recognizing atomic visual concepts,
constrain text to simple noun phrases (NPs) such as "red apple" or "striped cat". While SAM is not designed for long referring expressions or queries requiring reasoning, we show that it can be straightforwarddy combined with Multimodal Large Language Model (MLLM) to handle more complex language prompts. Consistent with previous SAM versions, SAM is fully interactive, allowing users resolve ambiguities by adding refinement prompts guide the model towards their intended output.
Our model ($3) consists of detector and tracker that share vision encoder (Bolya et al. 2025 The detector is a DETR-based (Carion et al. (2020) model conditioned on text, geometry, and image exemplars. To address the challenge of open-vocabulary concept detection, we introduce separate presence head to decouple recognition and localization, which is especially effective when training with challenging negative phrases. The tracker inherits the SAM transformer decoder architecture, supporting video segmentation and interactive refinement. The decoupled design for detection and tracking avoids task conflict, as the detector needs to be identity agnostic, while the tracker's main objective is to separate identities in the video.
To unlock major performance gains, we build a human- and -ooel-in-the-loop data engine ($4 that annotates large and diverse training dataset. We innovate upon prior data engines in three key ways: (i) media curation: we curate more diverse media domains than past approaches that rely on homogeneous web sources, (ii) label curation: we significantly increase label diversity and difficulty by leveraging an ontology and multimodal LLMs as "Al annotators" to generate noun phrases and hard negatives, (iii label verification: we double annotation throughput by fine-tuning MLLMs to be effective "AI verifiers" that achieve near-human accuracy.
Starting from noisy media-phrase-mee exhaustivity using both human and Al verifiers, filtering out correctly labeled examples and identifying challenging error cases. Human annotators then focus on fixing these errors by manually correcting masks. This enables us to annotate high-quality training data with 4M unigue phrases and 52M masks, and a synthetic dataset with 38M phrases and 1.4B masks. We additionally create the Segment Anything with Concepts (SA-Co) benchmark for PCS ($5) containing 207K unique concepts with exhaustive masks 120K images and 1.7K videos, > 50x more concepts than existing benchmarks.
pseudo-labels, our data engine checks mask quality and
Our experiments (56) show that SAM sets new ttate-of-the-ee in promptable segmentation, e.g.. reaching a zero-shot mask AP of 48.8 on LVIS vs. the current best of 38.5, surpassing baselines on our new SA-Co benchmark by at least 2 (see examples in Fig. 2). and improving upon SAM 2 on visual prompts. Ablations ($B) verify that the choice of backbone, novel presence head, and adding hard negatives all boost results, and establish scaling laws on the PCS task for both our high-quality and synthetic datasets. We source the SA-Co benchmark and release the SAM 3 checkpoints and inference code. On an H200 GPU, SAM 3 runs in 30 ms for a single image with 100+ detected objects. In video, the inference latency scales with the number of objects, sustaining near real-time time performance for 5 concurrent objects. We review related work in SA next, we dive into the task.
2

---

Published as a conference paper at ICLR 2026
Text: "a fish" and/or positive exemplar (green)
Masks masklets video) for each detected instance
T+ positive (green) or negative (red) exemplars
Refined masklltts) for each detected Instance
INITIAL PROMPT
OUTPUT
REFINEMENT PROMPTS
OUTPUT
Figure 3: Illustration of supported initial and optional interactive refinement prompts in the PCS task.
2 PROMPTABLE CONCEPT SEGMENTATION (PCS)
We define the Promptable Concept Segmentation task as follows: given an image or short video ($30 secs), detect, segment and track all instances of a visual concept specified by a short text phrase, image exemplars, or combination of both. We restrict concepts to those defined by simple noun phrases (NPs) consisting of a noun and optional modifiers. Noun-phrase prompts (when provided) global to all frames of the image/video, while image exemplars be provided individual frames as positive or negative bounding boxes to iteratively refine the target masks (see Fig. 3
All prompts must be consistent in their category definition, or the model's 's behavior is undefined; e.g.. "fish" cannot be refined with subsequent exemplar prompts of just the tail; instead the text prompt should be updated. Exemplar prompts particularly useful when the model initially misses some instances, or when the concept is rare.
Our vocabulary includes any simple noun phrase groundable in visual scene, which makes the task intrinsically ambiguous. There can be multiple interpretations of phrases arising from polysemy ("mouse" device vs. animal), subjective descriptors ("cozy", "large"). vague or context-dependent phrases that may not even be groundable ("brand identity"), boundary ambiguity (whether 'mirror' includes the frame) and factors such as occlusion and blur that obscure the extent of the object. While similar issues appear in large blosedvvccabulan are alleviated by carefully curating the vocabulary and setting a clear definition of the classes of interest. We address the ambiguity problem by collecting test annotations from three experts, adapting the evaluation protocol to allow multiple valid interpretations ($F.3), designing the data pipeline/guidelinss to minimize ambiguity in annotation, and ambiguity module ($D.2).
corpora (e.g.. LVIS (Gupta et al., 2019)). they
MODEL
SAM 3 is generalization of SAM 2, supporting the new PCS task ($) as well as the PVS task. takes concept prompts (simple noun phrases, image exemplars) or visual prompts (points, boxes, masks) to define the objects to be (individually) segmented and visual prompts be iteratively added on individual frames refine the target masks false positive and false negative objects can be removed or added respectively using image exemplars and an individual mask(let) can be refined using PVS in the style of SAM 2. Our architecture is broadly based on the SAM and (M)DETR (Carion et al. 2020 Kamath et al. 2021) series. Fig. shows the detector for image-level
tentorllmppr.lly.. Image exemplars
SAM 3 architecture, consisting of a dual encoder-decoder transformer capabilities- which is used in combination with tracker and memory for video. The detector and tracker ingest vision-language inputs from an aligned Perception Encoder (PE) backbone (Bolya et al. 2025). We present an overview below, see for details.
Detector Architecture. The architecture of the detector follows the general DETR paradigm. The image and text prompt are first encoded by PE and image exemplars, if present, are encoded by an exemplar encoder. We refer to the image exemplar tokens and text tokens jointly as "prompt tokens". The fusion encoder then accepts the unconditioned embeddings from the image encoder and conditions them by cross-attendin to the prompt tokens. The fusion is followed by a DETR- decoder, where learned object queries cross-attend the conditioned image embeddings from the fusion encoder.
Each decoder layer predicts classification logit for each object query (in case, binary label of whether the object corresponds to the prompt), and a delta from the bounding box predicted by the previous level, following Zhu al. (2020). We use eox-rreioonpositioon bias (Lin al. 2023)
3

---

Published as a conference paper at ICLR 2026
merge existing newly detected masks
Text Encodar
"a penguin'
Detector
masks detected framet
output for frame
exemplar
Encoder
Tracker
Memery Bank
from
from
masks propagated from frame t-1
mask, point or box
Figure 4: SAM architecture overview. See Fig. 10 for a more detailed diagram.
to help focalize the attention on each object, but unlike recent DETR models, we stick to vanilla attention. During training, we adopt dual supervision from DAC-DETR (Hu al.) 2023), and the Align loss (Cai et al. 2024). The mask head is adapted from MaskFormer (Cheng et al., 2021). In addition, we also have semantic segmentation head, which predicts a binary label for every pixel in the image, indicating whether or not it corresponds to the prompt. See $D for details.
Presence Token. It can be difficult for each of the proposal queries both recognize (what) and localize (where) an object in the image/frame. For the recognition component, contextual cues from the entire image are important. However, foreing proposal queries to understand the global context can be counterproductive, as it conflicts with the inherently local nature of the localization objective. We decouple the recognition and localization steps by introducing a learned global presence token. This token is solely responsible for predicting whether the target concept in the form of noun phrase (NP) is present in the image/frame, i.e. pNNP is present in input). Each proposal query only needs to solve the localization problem p(q, is match NP is present in input). The final score for each proposal query the product of its own score and the presence score.
Image Exemplars and Interactivity. SAM supports image exemplars, given pair a bounding box and an associated binary label (positive or negative)- which can be used in isolation or to supplement the text prompt. The model then detects all the instances that match the prompt. For example, given a positive bounding box on dog, the model will detect all dogs in the image. This different from the PVS task in SAM| 1 and where visual prompt yields only single object instance. Each image exemplar is encoded separately by the exemplar encoder using an embedding for the position, an embedding for the label, and ROI-pooled visual features, then concatenated and processed by small transformer. The resulting prompt is concatenated to the text prompt to comprise the prompt tokens. Image exemplars can be interactively provided based on errors current detections to refine the output.
Tracker and Video Architecture. Given a video and a prompt P. we use the detector and a tracker (see Fig. 4 to detect and track objects corresponding to the prompt throughout the video. On each frame, the detector finds new objects O₂ and the tracker propagates masklets M,-1 ispatial-temporaa masks) from frames at the previous time t 1 to their new locations M, on the current frame at time I. We use a matching function to associate propagated masklets M, with new object masks emerging in the current frame O
M2 = propagate (M--1), O2 = detect P), M₁ = match anduchaand_pdaatee (M,, O.).
Tracking an Object with SAM Style Propagation. A masklet is initialized for every object detected on the first frame. Then, on each subsequent frame, the tracker module predicts the new masklet locations M, of those already-tracked objects based on their previous locations M,-1 through single-frame propagation step similar to the video object segmentation task in SAM The tracker shares the same image/frame encoder (PE backbone) as the detector. After training the detector, we freeze PE and train the tracker as in SAM 2, including a prompt encoder, mask decoder, memory encoder, and a memory bank that encodes the object's appearance using features from the past frames and conditioning frames (frames where the object is first detected or user- sser-prompted) The memory encoder is transformer with self-attention across visual features on the current frame and cross- rros-attention from the visual features to the spatial memory features in the memory bank. We describe details of our video approach in D
4

---

Published as a conference paper at ICLR 2026
Media Pool
Mine
Propose
Propose mask
data
SAM 5
flowrr
EK
Al model
candidates
final anriotations
incomplete
1
human
Ontology
Al verifiers
label
Correct
fail-G
Human verifiers
train
pacs 2
Figure 5: Overview of the final SAM 3 data engine. See
F.I [or details of collected data.
During inference, we only retain frames where the object is confidently present in the memory bank. The mask decoder is a two-way transformer between the encoder hidden states and the output tokens. To handle ambiguity, we predict three output masks for every tracked object on each frame along with their confidence, and select the most confident output as the predicted mask on the current frame.
Matching and Updating Based on Detections. After obtaining the tracked masks Mr, we match them with the current frame detections O2 through simple IoU based matching function ($D.3) and add them to M, on the current frame. We further spawn new masklets for all newly detected objects that are not matched. The merging might suffer from ambiguities, especially in crowded scenes. We address this with two temporal disambiguation strategies outlined next.
First, we use temporal information in the form of a masklet detection score DD3 to measure how consistently a masklet is matched to a detection within temporal window (based on the number of past frames where it was matched to a detection). If masklet's detection score falls below a threshold, we suppress it. Second, we use the detector outputs to resolve specific failure modes of the tracker due to occlusions or distractors. We periodically re- re-prompt the tracker with high-confidence detection masks Or replacing the tracker's own predictions M,. This ensures that the memory bank has recent and reliable references (other than the tracker's own predictions).
Instance Refinement with Visual Prompts. After obtaining the initial set of masks (or masklets), SAM 3 allows refining individual masks(lets) using positive and negative clicks. Specifically, given the user clicks, we apply the prompt encoder to encode them, and feed the encoded prompt into the mask decoder predict an adjusted mask. In videos the mask then propagated across the entire video to obtain a refined masklet.
Training Stages. We train SAM in four stages that progressively add data and capabilities: 1) Perception Encoder (PE) pre-training. 2) detector pre-training, 3) detector fine-tuning, and 4) tracker training with frozen backbone. See $D... for details.
4
DATA ENGINE
Achieving a step change in PCS with SAM requires training on a large, diverse set of concepts and visual domains, beyond existing datasets (see Fig. [2). We build efficient data engine that iteratively generates annotated data via a feedback loop with SAM 3, human annotators, and A/ annotators, actively mining media-phrase pairs on which the current version of SAM 3 fails to produce high-quality training data to further improve the model. By delegating certain tasks to AI munotators-mode that match or surpass human accuracy-we more than double the throughput compared to human-only annotation pipeline. We develop the data engine in four phases, with each phase increasing the use of AI models to steer human effort to the most challenging failure cases, alongside expanding visual domain coverage. Phases 1-3 focus only on images, with Phase 4 expanding to videos. We describe the key steps here; details and metrics are in S
Data Engine Components (Fig, 5). Media inputs (image or video) are mined from large pool with the help of a curated ontology. An Al model proposes noun phrases (NPs) describing visual concepts, followed by another model (e.g., SAM 3) that generates candidate instance masks for each proposed NP. The proposed masks are verified by a two-step process: first, in Mask Verification (MV) annotators accept or reject masks based their quality and relevance to the NP. Second, in Exhaustivity Verification (EV) annotators check if all instances of the NP have been masked in the input. Any media-NP pairs that did not pass the exhaustivity check are sent manual correction
5

---

Published as a conference paper at ICLR 2026
true
the fabric
aheee, of comrgaaed metal
while sack
yel cie shirt
lo-flpp
tail light
plaid
the whibe Icanse plate
whitee checc shirt
gravel gold
nearly dome shaped roof the large, yet ow estate
us g sen eye the white metal frome
the red velvethhh
the trells
BR
cnuch, daybad, a sheep...
the small white building
bag
pomegranate
caroboard sign
the blue basket
astic basket
the persimmon
white baaket
bow!
blue carpet
black display
MacBook person's hand
white
cardboard
mango, bread small glare
avbrant orange 1975 Plymouth Barracuda
Figure 6: Example video (top) and images (bottom) from SA-Co with annotated phrases and instance masks/IDs.
stage, where humans add, remove or edit masks (using SAM in a browser based tool), or use "group" masks for small, hard to separate objects. Annotators may reject ungroundable or ambiguous phrases.
Phase 1: Human Verification. We first randomly sample images and NP proposal with a simple captioner and parser. The initial mask proposal model is SAM 2 prompted with the output of an off-the-shelf neee-vocabulary detector, and initial verifiers are human. In this phase, we collected 4.3M image-NP pairs as the initial SA-Co/HQ dataset. We train SAM 3 on this data and use the mask proposal model for the next phase.
Phase 2: Human + AI Verification. In this next phase, we use human accept/reject labels from the MV and EV tasks collected in Phase to fine-tune Llama 3.2 (Dubey et al. (024) to create AI verifiers that automatically perform the MV and EV tasks. These models receive image-phrase-mase triplets and output multiple-choie ratings of mask quality or exhaustivity. This new auto-verification process allows our human effort to be focused on the most challenging cases. We continue to re-train SAM 3 on newly collected data and update it 6 times. As SAM 3 and Al verifiers improve, a higher proportion of labels are auto-generated further accelerating data collection. The introduction of AI verifiers for MV and EV roughly doubles the data engine's throughput vs. human annotators. We refer to ş BB.4 for detailed analysis of how AI verifiers improve the data engine's s throughput. We further upgrade the NP proposal step a Llama-based pipeline that also proposes hard negative NPs adversarial to SAM 3. Phase 2 adds 122M image-NP pairs to SA-Co/HQ.
Phase 3: Scaling and Domain Expansion. In the third phase, challenging cases and broaden domain coverage in SA-Co/HQ to 15 datasets (Fig. 15). domain is unique distribution of text and visual data. In domains, the MV AI verifier performs well zero-shot, but the EV AI verifier needs to be improved with modest domain-specinf human supervision. We also expand concept coverage long-tail, fine-grained concepts by extracting NPs from the image alt-text where available and by mining concepts from a 22.4M node SA-Co ontology ($E.2) based on Wikidata (17 top-level categories, 72 sub- catecoegorisss.. We iterate SAM training times and AI verifiers 3 times, and add 19.5M image-NP pairs to SA-Co/HQ.
Al models to mine increasingly
This phase extends the data engine to video. We use a mature image SAM to collect targeted quality annotations that capture video-specific challenges. The data mining pipeline applies scene/motion filters, content balancing, ranking, and targeted searches. Video frames are sampled (randomly or by object density) and sent to the image annotation flow (from phase 3). Masklets (spatio-temporal masks) are produced with SAM 3 (now extended to video) and post-processed via deduplication and removal of trivial masks. Because video annotation is more difficult, we concentrate humans on likely failures by favoring clips with many crowded objects and tracking failures. The collected video data SA- Co-CCOVIDEO consists of 52.5K videos and 467K masklets. See ( for details.
Phase 4: Video Annotation.

---

Published as a conference paper at ICLR 2026
Instance Segmentation
Box Detection
Semantic Segmentation
IVIS
SA-Co
LVIS
COCO
SA-Co
ADE-847 PC-59 Cityscapes
Model
cg
AP
Gold
Silver Broaze
Bio
cgFi
AP
AP
APa
Gold
Silver
Bronze
Bio pmFi
mloU
mloU.
mioU
cgFi
cgFi
cgFi
pmFi
cgFi
cgFi
cgFi
Human
72.8
74.0
OWLy2
20.1
17.3
7.6
3.y
0.64
19.9
35.2
38.2
42.4
16.9
7.1
4.1
0.95
OWLv2*
29.3
43.4
24.6
11.5
0.04
30.2
45.5
46.1
23.9
24.5
11.0
12.0
0.08
gDino-T
14.7
3.3
2.7
7.0
0.34
15.1
20.5
45.7
35,3
3.4
2.5
7.6
0.35
LLMDetLL
35.1
36.3
6.5
7.1
12.5
0.15
39.3
42.0
55.6
49.8
6.8
6.7
14.0
0.17
APE-D"
53.01
16.4
7.3
12.4
0.00
59.61
58.31
17.3
7.7
14.3
0.00
9.21
58.51
44,21
DINO-X
38.51
21.3
52.41
56.0
22.5
Gemini 2.5
13.4
10.7
9.4
12.4
SAM 3
37.2
48.5
54.1
49.6
42.6
55.4
40.6
53.6
56.4
55.7
55.7
50.0
47.1
56.3
13.8
60.8
65.2
Table I: Evaluation on image concept segmentation with text. AP, corresponds to COCO-O accuracy, *: partially trained on LVIS, t: from original papers, & from DINO-X API. Gray numbers indicate usage of respective closed set training data (LVIS/COCO). See ff. more results and SF.4 for details of human performance.
SEGMENT ANYTHING WITH CONCEPTS (SA-Co) DATASET
Training Data. We collect three image datasets for the PCS task: (i) SA-Co/Q, the high-quality image data collected from the data engine in phases 1-4, (ii) SA-Co/SYN, a synthetic dataset of images labeled by a mature data engine (phase 3) without human involvement, and (iii) SA-Co/EXT, 15 external datasets that have instance mask annotations, enriched with hard negatives using ontology pipeline. Notably in the SA-Co/HQ dataset we annotate 5.2M images and 4M unique NPs, making it the largest high-quality open-vocab segmentation dataset. We also annotate video dataset, SA-CO/VIDEO, containing 52.5K videos and 24.8K unique NPs, forming 134K video-NP pairs. The videos on average have 84.1 frames at fps. See $F. for details including full statistics, comparison with existing datasets and the distribution of concepts.
SA-Co Benchmark. The SA-Co evaluation benchmark has 207K unique phrases, 121K images and videos, and over 3M media-phrase pairs with hard negative labels to test open-vocabulary recognition. It has 4 splits: SA-Co/Gold has seven domains and each image-NP pair is annotated by three different annotators (used to measure human performance): SA-Co/Silver has ten domains and only one human annotation per image-NP pair, SA-Co/Bronze and SA-Co/Bio are nine existing datasets either with existing mask annotations or masks generated by using boxes as prompts to SAM 2. The SA-Co/VEval benchmark has three domains and one annotator per video-NP pair. See Tab. 28 for dataset statistics and Fig. 6 for example annotations.
Metrics. We aim to measure the usefulness of the model in downstream applications. Detection metrics such average precision (AP) do not account for calibration, which means that models can be difficult to use in practice. To remedy this, we only evaluate predictions with confidence above 0.5, effectively introducing threshold that mimics downstream usages and enforces good calibration. The PCS task can be naturally split into two sub-tasks, localization and classification. We evaluate localization using positive micro F1 (pmF,) )) on positive media-phrase pairs with at least one ground-truth mask. Classification is measured with image-level Matthews Correlation Coefficient (IL. which ranges in [-1, 1] and evaluates binary prediction at the image level ("is the object present?") without regard for mask quality. Our main metric. classification gated F! (cgF,), combines these as follows: egl', 100 pmfF, MCC. Full definitions are in ST3
Handling Ambiguity. We collect annotations per NP on SA- SA-Co/Gold. We measure oracle accuracy comparing each prediction to all ground truths and selecting the best score. See F3
6 EXPERIMENTS
We evaluate SAM 3 across image and video segmentation, few-shot adaptation to detection and counting benchmarks, and segmentation with complex language queries with SAM 3 + MLLM. We also show subset of ablations, with more in $ References, more results and details are in $
Image PCS with Text. We evaluate instance segmentation, box detection, and semantic segmentation on external and our benchmarks. SAM 3 is prompted with single NP at time, and predicts instance masks, bounding boxes, or semantic masks. As baselines, we evaluate OWLv2, Grounding (gDino), and L.L box detection, and prompt SAM 1 with their boxes evaluate segmentation.
7

---

Published as a conference paper at ICLR 2026
ODiaW13 RF-100VL
COCO
LVIS
WD
Model
APo
API0
APo
AP10
AP
AP
APt
AP
AP
AP+
APt
AP+
AP
APT T
APT APt
Model
T
T
T+1
T
T
T+1
T+1
Gemini2.5-Pro
33.7
11.6
9.8
gDino-T T
49.7
15.7
33.7
T-Rex2
52.2
58.5
45.8
65.8
50.3
61.8
5-ioo1.5PPro
58.7
67.9
SAM3
56.4
58.8
76.8
78.1
52.4
54.7
76.0
78.4
61.1
63.1
82.2
81.8
SAM 3
61.0
71.8
15.2 36.5
Table 3: Prompting with 1 exemplar on COCO, LVIS and ODinW13. Eval- uation per prompt type: T (text-only). (image-only), and T+I (combined text and image). AP" is evaluated only on positive examples.
Table 2: Zero-shot and 10-shot transfer on in-the-wild datasets.
We also compare to APE, DINO-X, and Gemini 2.5 Flash, a generalist LLM. Tab. I shows that zero-shot, SAM 3 sets a new state-of of-the-art on dlosed-vocabull COCO, COCO-O and on LVIS boxes, and is significantly better on LVIS masks. On open- vocabulayy SA-Co/Gold SAM 3 achieves more than double the cgF, score of the strongest baseline OWLv2*, and 74% of the estimated human performance. The improvements are even higher on the other SA-Co splits. Open vocabulary semantic segmentation results on ADE-847, PascalConcept-59, and Cityscapes (val set) show that SAM 3 outperforms APE, strong specialist baseline. See $ for details.
Few-Shot Adaptation. We evaluate zero- and few-shot transfer of SAM 3 on ODinW13 and RF100- VL, with their original labels as prompts. We do not perform any prompt tuning. We fine-tune SAM 3 without mask loss, and report average bbox mAP in Tab. 2 SAM 3 achieves sfate-of-the-art 10-shot performance, surpassing in-context prompting in Gemini and object detection experts (gDino); more details in GG.3 RF-100VI. contains domains with specialized prompts that are out of SAM 3's current scope, but SAM 3 adapts through fine-tuning more efficiently than baselines.
PCS with Exemplar. We first evaluate image exemplars using single input box sampled random from the ground truth. This can be done only on "positive" data, where each prompted object appears in the image. We report the corresponding AP+ in Tab. Vaab.3jacma acrros three settings: text prompt (T), exemplar image (I), and both text and image (T+I); SAM outperforms prior state-of-the-art T-Rex2 by a healthy margin on COCO (+18.3),
0.30
0.75
0.70
0.65
0.50
SAM3 PCS
LVIS (+10.3), and ODin (+20.5). See G.2 for more details and results on SA-Co/Gold.
Perfec SAM PVS
0.55
Hotrid: 2 pooupe PCS this PVS
Hyhrid:4 priepte PCS thes PVS
PCS with K Exemplars. Next, we evaluate SAM 3 in an interactive setting. simulating collaboration with human an- notator. Starting with text prompt, we iteratively add one exemplar prompt at a time: missed ground truths are candidate positive prompts, false positive detections are candidate nega- tive prompts. Results (Fig. 7 are compared to a perfect PVS baseline, where we simulate the user manually fixing errors using ideal box-to-mask corrections. SAM 3's PCS improves cgF, more quickly, as it generalizes from exemplars (e.g.. detecting or suppressing similar objects), while PVS only corrects individual instances. After clicks, interactive PCS outperforms text-only by +21.6 cgFi points and PVS refinement by +2.0. Performance plateaus after 4 clicks, as exemplars cannot fix poor-quality masks. Simulating a hybrid switch to PVS at this point yields gains, showing complementary.
Numbera/PPumpt
Figure 7: cgFi vs. # of interactive box prompts for SAM 3 compared the ideal PVS baseline, averaged over SA- Co/Gold phrases.
CountBench
PisMo-Count
Object Counting. We evaluate on object counting benchmarks CountBench and PixMo-Count to com- pare with several MLLMs using Accuracy (%) and Mean Absolute Error (MAE) from previous techni- cal reports and our own evaluations. See Tab. for results and $G.4 for more evaluation details. Com- pared to MI.LMs, SAM not only achieves good object counting accuracy, but also provides object segmentation that most MLLMs cannot provide.
Model
MAE
Acc f
MAF
Acc
DINO-X
0.62
82.9
0.21
85.0
Quen2-VLL-22
0.28
86.7
0.61
63.7
Molmo-72R
0.27
92.4
0.17
88.8
Gemini 2.5 Pro
0.24
92.4
0.38
78.2
SAM
0.12
93.8
0.21
86.2
Table 4: Accuracy on counting benchmarks. Gray
indicates usage of training sets.
Video PCS with Text. We evaluate video segmentation with text prompts on both our SA-Co/VEval benchmark and existing public benchmarks. For SA-Co/VEval, we report egF, and pHOTA metrics (defined in GG.5) across its subsets (SA-V, YT-Temporal-IB SmartGlasses). For public benchmarks, we use their official metrics. Baselines include GLEE, an open- vocabulaay image and video seg- mentation model, "LLLMDet + SAM 3 Tracker" (replacing our detector with LLMDet),, and "SAM
8

---

Published as a conference paper at ICLR 2026
SA-Co'VEval benchmark test split
Public benchmarks
SA-V (2.0K NPs)
YT-Temporal-IB SmariGlasse
LVVIS (1.2 NPS)
BURST (482 NPs)
YTVISZI
OVIS
(L.7K NPS)
(2.4K NPs)
(40
(25 NPS)
Model
cgti PHOTA
egti PHOTA
egFi pHOTA
lest mAP
lest HOTA
val mAP
val mAP
Human
53.1
70.5
71.2
78.4
58.5
72.3
GLEE (all once)
8.7
4.7
20.8
28.4
38.7
GLEE (one NP time)
11.8
2.2
18.9
0.1
5.6
9.3
20.2
56.5
32.4
LLMDet SAM Tracker
2.3
30.1
8.0
37.9
0.3
18.6
15.2
33.3
31.3
20.4
SAM Detoctor + T-by-D
25.7
55.7
47.6
68.2
29.7
60.0
35,9
39.7
56.5
55.1
SAM
30.3
58.0
50.8
36.4
63.6
36.3
44.5
57.4
60.5
Table 5: Video PCS from text prompt (open-vocabulary video instance segmentation) on SA-Co/VEval and public henchmarks (see Tah, 39 for more results and analyses). SAM shows strong performance, especially on benchmarks with a large number of NPs. f: GLEE and L.MDeet do not perform well zero-shot on SA-Co/VEEval
3 Detector + "Tby-D" (replacing our tracker with an association module based the tracking-by by- detection paradigm). In Tab. 5S] SAM largely outperforms these baselines, especially on benchmarks with very large number of noun phrases. On SA-Co/VEval it reaches over 80% of human pHOTA. See GG5 for more details.
PVS. We evaluate SAM 3 range of visual prompting tasks, including Video Object Segmentation (VOS) and interactive image segmentation. Tab. comppress SAM 3 to recent state-of-the-art methods on the VOS task. SAM 3 achieves sigmificant improvements over SAM on most benchmarks, particularly on the challenging MOSEv2 dataset, where SAM 3 outperforms prior work by 6.5 points. For the interactive image segmentation task, we evaluate SAM on the 37 datasets benchmark introduced in Ravi et al. (2024). As shown in Tab. 7 SAM outperforms SAM 2 on average mIoU.
See also G and Fig. 21 for interactive video segmentation.
JKF
G
J&3
Avg. mloU
MOSEV1 DAVISS7IAOSSS
SA-V val
SA-V test
VTVOS19
Model
val
val
val
val
val
Model
1-elick 3-clicks 5-clicks
SAMURAI
72.6
89.9
84.2
79.8
80.0
88.3
51.1
SAM II
58.5
77.0
82.1
41.0
SAM21 Long
75.2
91.4
85.9
81.1
81.2
88.7
51.5
2.1
80.3
84.3
93.0
SeC
75.3
91.3
86.5
82.7
81.7
88.6
53.8
SAM
66.1
81.3
85.1
43.5
SAM 2.1L
77.9
90.7
79.6
77.9
78.4
89.3
47.9'
S
89.7
60.3
Table 7: Interactive image segmenta- tion on the SA-37 benchmark.
Table 6: SAM improves over SAM 2 in VOS. f: Zero-shot.
SAM 3 Agent. We experiment with MLLM that uses SAM tool to segment more complex text queries (see Fig. (25). The MLLM proposes noun phrase queries to prompt SAM and analyzes the returned masks, iterating until the masks are satisfactory. Tab. shows that this "SAM 3 Agent" evaluated zero-shot on ReasonSeg and OmniLabel surpasses prior work without training on any referring expression segmentation or reasoning segmentation data. SAM 3 Agent also outperforms previous zero-shot results on RefCOCO+ and RefCOCOg. SAM 3 can be combined with various MLLMs, with the same set of the system prompts for all those MLLMs, showing SAM 3's robustness. See for more details.
ReasonSeg (glot))
Omnilahel (AP)
val
test
val 2023
Model
MILMM
All
All
Shoet
Long
deser
descr-S
descr-M
descr-L
X-SAM SeyZero
Phi-3-3.8B
56.6
57.8
47.7
56.0
12.0*
17.1"
11.4*
8.8
Qwen2.5.
62.6
57.5
13.5*
20.79
12.4
9.1
RSVP
GPT-4o
64.7
55.4
61.9
60.3
Overall state of the art"
65.0
61.3
55.4
63.2
36.5
54.4
33.2
25.5
SAM.3 3 Agent
Qwen2.5.VI. VL. 7B
62.2
63.0
59.4
64.1
36.7
52.6
34.3
26.6
SAM. 3 Agent
Llama4 Maverick
68.5
67.1
66.8
67.2
32.8
43.7
30.9
27.5
SAM Agent
Qwen2.5 VL.
70.3
33.2
SAM Agent
Gemini 2.5 Pro
77.0
74.0
75.8
73.4
45.3
53.8
45.1
37.7
Table 8: SAM Agent results. Gray indicates fine-tuned results ReasonSeg (train), indicates reproduced for ReasonSeg; REAL for OmniLabel.
results, underline indicates the main metric. t:

---

Published as a conference paper at ICLR 2026
eghi
MCC
pmf-i
#/img
cgFi
MCC
pmf,i EXTSYN HQ
egFi
MCC
pmff
Model
egFi
MCC
pmffi
x 50.7
0.77
65.4
0
28.3
0.44
62.4
of
x
x
23.7
0.46
50.4
Human
72.8
0.94
77.0
63.4
39.4
0.62
62.9
0.57
56.9
SAM
0.82
65.9
15
41.8
0.67
62.4
V
45.5
0.71
64.0
EV AT
61.2
0.86
70.8
30
43.0
0.68
62.8
V
<
47.4
0.74
63.8
MV AT
62.3
0.87
71.1
(a) Presence head.
(b) Hard Negatives.
(c) Training data.
(d) SAM 3 + Al verifiers.
Table Selected model and data ablations SA-Co/Gold. Numbers across tables are not directly comparable.
Selected Ablations. In Tab. [0 we report a subset of the more extensive ablations from B Note that the ablated models are from different, shorter training runs than the model evaluated above. The presence head boosts cgFi by +1.5 (a). improving image-level recognition measured by ML. by +0.05. Tab. 96 shows that adding hard negatives significantly improves the model performance, most notably the image-level II. MCC from 0.44 to 0.68. Tab. 9c shows that synthetic (SYN) training data improves over the external (EXT) by +8.8 cgFi and our high-quality (HQ) annotations add + +14.6 cgFi on top of this baseline. We present detailed data scaling laws of both types of data in $B.2 showing their effectiveness on both in-domain and out-of-domain test sets. In Tab, gd we show how AI verifiers can improve pseudo-labels. Replacing the presence score from SAM 3 with that score from the exhaustivity verification (EV) AI verifier boosts cgFi by +7.2. Using the mask verification (MV) Al verifier to remove bad masks adds another 1.1 points. Overall, Al verifiers close half of the gap between SAM 3's and human performance.
Domaín adaptation ablation. With domain-specifie syn- thetic data generated by SAM + Al verifiers, we show that one can significantly improve performance on new domain without any human annotation. We hold out one of the SA-Co domains. "Food&drink" from training SAM 3 and AI verifiers. We then use three variants of training data for the novel "Food&drink" domain: high-quality Al+human annotations as in SA-Co/HQ (referred to SA- Co/HQ-Foodd), synthetic annotations as in SA-Co/SYN, using AI but no humans labels generated before the AI verification step, i.e. skip- ping both AI verifiers and humans (PL-Food).. Fig. 8 plots performance on the "Food&drink"" test set of the SA- Co/Gold benchmark as each type of training data is scaled up. We mix the domain specific data and high-quality gen- eral domain data 1:1 ratio. PL-Food provides some improvement compared to the bascline SAM 3 (zero-shot), but is far below the other variants due to its lower quality. HQ-Food and SYN-Food show similar scaling behavior, with SYN-Food slightly lower but eventually catching up, without incurring any human annotation cost. This points to scalable way to improve performance on new data distributions. More details are in {B.3
FLFFood
CWSYNN
SA
Teaches
ogf's (now domain)
50
Roseline
and pseudo-
1.5K
7.5K
30K
150K
750K
New
Figure 8: Domain adaptation via syn- thetic data. Synthetic (SYN) data generated by SAM + verifiers (leacher system) achieves similar scaling behavior as human- annotated (HQ) data.
7 CONCLUSION
We present Segment Anything with Concepts, enabling open-vocabulary text and image exemplars as prompts in interactive segmentation. Our principal contributions are: (i) introducing the PCS task and SA-Co benchmark, (ii) an architecture that decouples recognition, localization and tracking and extends SAM solve concept segmentation while retaining visual segmentation capabilities, (iii) high-quality, efficient data engine that leverages the complimentary strengths of human and AI annotators. SAM 3 achieves state-of-the-art results, doubling performance over prior systems for PCS on SA-Co in images and videos. That said, our model has several limitations. For example, it struggles to generalize to oft-o--domain terms, which could be mitigated by automatic domain expansion but requires extra training. We discuss this and other limitations of our model in $C We believe SAM 3 and the SA-Co benchmark will be important milestones and pave the way for future research and applications in computer vision.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank the following people for their contributions to the SAM 3 project: Alex He, Alexander Kirillov, Alyssa Neweomh, Ana Paula Kirsehner Mofarrej. Andrea Madotto, Andrew Westhury, Ashley Gabriel, Azita Shokpour, Ben Samples, Bermie Huang, Carleigh Wood, Ching-Feng Yeh, Christian Puhrsch, Claudette Ward, Daniel Bolya, Daniel Li. Facundo Figueroa, Fazila Vhora, George Orlin, Hanzi Mao, Helen Klein, Hu Xu, Ida Cheng. Jake Kinney, Jiale Zhi, Jo Sampaio, Joel Schlosser, Justin Johnson. Kai Brown, Karen Bergan, Karla Martucci, Kenny Lehmann, Maddie Mintz, Mallika Malhotra, Matt Ward, Michelle Chan, Michelle Restrepo, Miran Hco, Miranda Hartley, Muhammad Maaz, Nisha Deo, Peter Park, Phillip Thomas, Raghu Nayani, Rene Martinez Dochner, Robbic Adkins, Ross Girshik, Sasha Mitts, Shashank Jain, Spencer Whitchead, Ty Toledano, Valentin Gabeur, Vincent Cho, Vivian Lec, William Ngan, Xuchai Hc, Yacl Yungster, Ziqi Pang, Ziyi Dou, Zoc Quake. We also thank the IDEA team for granting DINO-X and T-Rex2 access to benchmark them on the SA-Co/Gold dataset.
APPENDIX
A
Related Work
12
B
Ablations
13
B.1
Model Ablations
13
B.2
Image Training Data Ablations
14
B.3
Automatic Domain Adaptation
B.4
Image Data Engine Annotation Speed
16
B.5
Video Data Engine Annotation Speed
17
B.6
Video Training Data Ablations
18
C
Limitations
18
D
Model Details
19
D.1
Model Architecture
19
D.2
Image Implementation Details
19
D.3
Video Implementation Details
21
D.4
Model Training
23
E
Data Engine Details
25
E.1
Media Pool
26
E.2
SA-Co Ontology
E.3
Phase 1: Human Verification
E.4
Phase 2: Human AI Verification
28
E.5
Phase 3: Scaling and Domain Expansion
32
E.6
Phase 4: Video Annotation]
32
F
SA-Co Dataset and Metric Details
33
F1
SA-Co Training Data
F22
SA-Co Evaluation Benchmark
35
F33
Metries
F.4
Human Performance on SA-Co
37
F.5
Additional Dataset Examples
38
G
Additional Experiments and Details
45
G.1
PCS with NPs on Images
45
G.2
Visual Exemplars and Interactivity
46
G.3
Few-Shot Fine-tuning
46
G.4
Object Counting
47
11

---

Published as a conference paper at ICLR 2026
G.5
Video PCS Details
48
G.6
PVS Details
50
H
SAM
3
Agent
H.1
Agent Design
55
H.2
Qualitative Analysis
56
H.3
Full Quantitative Results
60
Model and annotation cards
64
L1
Data annotation card
64
1.2
Model card
65
A RELATED WORK
Promptable and Interactive Visual Segmentation. SAM (Kiilllo et al. 2023) introduces "prompt- able" image segmentation with interactive refinement. While the original task definition included text prompts, they were not fully developed. SAM 2 (Ravi et al. 2024) extended the promptable visual segmentation task to video, allowing refinement points on any frame. SAM inherits geometry-bane -based segmentation while extending to include text and image exemplar prompts to segment all instances of a concept in images and videos.
Open- Vocabulary Detection and Segmentation in Images exhaustively labels every instance of an open-vocabulary object category with a coarse bounding box (detection) or a fine-grained pixel mask (segmentation). Recent open veccbvocayulary (OV) detection (Gu et al. 2021 Minderer et al. 2022) and
segmentation (Ding et al., 2022; Liang et al. 2023) methods leverage large-scale iision-language encoders such as CLIP (Radford et al. 1021) to handle categories described by arbitrary text, even those never seen during training. While DETR (Carion et al. 2020) is limited to a closed set of categories seen during training, MDETR (Kamath al.] 2021) evolves the approach to condition on raw text queries. Image exemplars used as prompts to specify the desired object category (e.g., DINOv (Li et al. 2023a), T-Rex2 (Jiang al. 2024)) present practical alternative text, but fall short in conveying the abstract concept of objects as effectively as text prompts. We introduce new benchmark for OV segmentation with 100x more unique concepts than prior work.
Visual Grounding localizes a language expression referring to region of the image with box or mask. (Plummer et al., 2020) introduces phrase detection as both deciding whether the phrase is relevant to an image and localizing it. GLIP (Li et al. 2022b) and GroundingDino (Liu et al. 2023) formulate object detection as phrase grounding, unifying both tasks during training. MQ-GLIP (Xu
al. 2023) adds image exemplars to text queries. Building on this trend toward models supporting multiple tasks and modalities, GLEE (Wu et al. 2024a) allows text phrases, referring expressions, and visual prompts for category and instance grounding in both images and vidcos. Unlike SAM 3. GLEE does not support exemplars or interactive refinement. LISA (Lai et al. 2024) allows segmentation that requires reasoning, while OMG-LLaVa (Zhang et al., 2024a) and GLaMM (Rasheed al. 2024) generate natural language responses interleaved with corresponding segmentation masks, with GLaMM accepting both textual and optional image prompts as input. Some general-p pprrpose MLLMS can output boxes and masks (Gemini2.5 (Comanici et al. (0255) ) or points (Molmo (Deitke et al.
2025
SAM can be used as a "vision tool" in combination with an MILM ($6).
Multi-Object Tracking and Segmentation methods identify object instances in video and track
them. associating each with a unique ID. backing-by-detectee methods, detection is performed independently on each frame to produce boxes and confidence scores, followed by association of
matching as SORT (Bewley et al. 2016 Wojke boxes using motion-based and appearance- Byte'Track (Zhang et al. 2022c), SAM2MOT (Jiang
et al.
2017) Tracktor (Bergmann et al. 2019).
ct al. 2025), or OC-SORT (Cao et al. 2023). An alternative is an end-to tol-tt--eaa end trainable architecture
that jointly detects and associates objects, e.g., TrackFormer (Meinhardt et al. 2022), TransTrack (Sun)
et al. 2020). or MOTR (Zeng et al. (022). TrackFormer uses a DETR-lik encoder-decoder that initializes tracks from static object queries and regrresssvlll follows existing tracks with prretttyppreseriigg track queries. A challenge with joint models is the conflict between detection and tracking (Feichtenhofer al. 2017
Yu al. 2023a), where needs to focus semantics while
12

---

Published as a conference paper at ICLR 2026
the other disentangling identities, even if their spatial locations overlap over time. SAM 3 is strong image detector tightly integrated into a tracker to segment concepts in videos.
B
ABLATIONS
B.1 MODEL ABLATIONS
We first ablate the impact of the presence token and the approach to its training. The presence token is included in the decoder (discussed further in $D.2). together with the object queries, and predicts a concept presence score. The presence score receives gradients only on the PCS task during joint training and always supervised with the presence (or absence) of the concept in the image using binary cross-entropy loss. Using a presence token to decouple presence and localization brings significant gains in performance, particularly on II. MCC, see Tab.
Presence Token.
When used with presence score, we found that it is better for the box/mask object scores to nor receive gradients when concept is an image-level negative, see Setting (a) in Tab. [0] Note that this is in contrast to the approach in typical DETR variants, where all individual object scores are supervised negatively to reflect the absence of the concept the image, see Setting (b) in Tab. TO
We find that (b) works worse than (a) when used with the presence score. When a concept is present in the image, individual object queries always receive classification supervision based on Hungarian matching. Setting (a) is consistent with recognition-locaa -ocaliizaoono decoupled design, where the presence score is responsible for recognition (existence in the image) and the object scores are responsible for localization (i.e., rank the best match the positive ground-truth highest among all the proposals).
During inference, use the product of the global presence score and the object score the total object score. In Setting (c), we explored directly supervising the total object scores (instead of the typical object scores) as positive or negative (as determined by matching); this setting can slightly improve the overall cgF1, but is less flexible as the presence and object scores are jointly calibrated, e.g. such a model less amenable to conditioning on a concept known be present in the image. Finally, Setting (d) in Tab. [0] investigaaes detaching the presence score from the computation graph while supervising the total scores, but this does not improve over (c).
Supervise mask scores only when concept present
Supervise total score
Sup. total score, detach presence
SA-Co/Gold
cgFi
[L_MCC
pmFF
a.
X
54.0
0.82
65.5
b.
X
52.2
0.81
64.2
c.
X
54.9
0.83
66.0
53.6
0.83
64.9
Table 10: Supervision strategy for object/mask scores for model with presence token. We find the best supervision strategy is to supervise mask scores only for positive concepts and to supervise the presence and mask scores separately, although their product is used as the total object score during inference.
Training with presence can be considered as form of post-training and occurs in Stage 3 (see DD4.1 of our training pipeline. By default, ablations do not undergo this stage unless otherwise mentioned.
Vision and Text Encoder. While SAM 2 uses an MAE (He et al., 2022) pre-trained Hiera (Ryali al. 2023) vision encoder for its strong localization capability and efficiency for the more geometric PVS task, SAM also needs strong semantic and linguistic understanding with broad coverage. We adapted PE (Bolya et 2025) for the vision and text encoders of SAM 3, so that large and diverse set of concepts is seen in Stage of training, while producing aligned image and text encoders. In Tab. TT, we compare performance with Hiera and DINOv2 (Oquab al., [002)); since these vision encoders lack an aligned text encoder, we use oosRil baase- (Sanh et al. 2019).. We find PE to be the best overall choice of vision backbone, and using its own aligned text encoder provides further gains over PE with an unaligned text baseline. Use of PE enables strong robustness in SAM 3 (here measured by AP on COCO-O, demonstrating good object detection across various domain shifts, e.g. "sketch", "cartoon". "painting". etc).
Implementation Details. The image resolution is set to 1008 px, 1008 px, 1152 px for PE, DINOv2, Hiera, respectively, ensuring the same number of tokens in the detector due their differences in
13

---

Published as a conference paper at ICLR 2026
Encoder (patch size)
SA-Co/Gold (egFi)
COCO-O (AP)
PE-L+ (14)
43.2
42.5
PE-L+ (14) w/ Distil DINOv2-L
38.1
39.6
35.3
31.9
Hicra-L (16)
32.8
22.0
Table 11:
Choice of encoders. As SAM 3 needs both semantic visual and linguistic understanding, we find PE's aligned image and text encoders work well.
patch size. All vision encoders used global attention in only a subset of the layers, using windowed (24 24 tokens) attention otherwise. Since Hiera is hierarchical multiscale encoder, we set the window size to 24 x 24 in stage 3 of the encoder, which has most of the FLOOPs. Since PE is capable of using relative positional information via RoPE (Su et al. 2021 Heo et al. 2024). we include relative positional embeddings in global layers for Hiera and DINOv2 following Bolya et al. (2024). All models are trained using SA-Co/HQ viewing 5 million samples over the course of training. Recipe is separately optimized for each choice of encoder. Tokens from the respective vision encoders are downsampled by 2 to 1296 tokens before being passed to the fusion encoder and detector.
B.2 IMAGE TRAINING DATA ABLATIONS
Setup. We adopt a simplifed, lighter model and training strategy for ablations in this section. Specifically, we use (i) a stride-28 (instead of 14) variant of SAM using 4 fewer tokens in the detector, (ii) limit to 45% of the entire SA-Co/SYN dataset and adopt, (iii) shorter training schedules and do not run "presence post-training" (see $ (iv) evaluations are on an internal version of SA-Co/ A.Coo,ool, which has slightly lower human performance than the public version (cgF1: internal 70.8 vs public 72.8). This allows running ablations more efficiently (but results in lower absolute accuracy vs. SAM 3). We observed similar trends when training at scale.
SAM3 3 Training Data. Tab. 9c analyzes the impact of various SA-Co training data subsets. Training with even with just SA-Co/XTT shows comparable performance with best external models on SA-Co/Gold (see OWLv2's and DINO-X's performance in Tab. [T. indicating a strong base model. Adding synthetic data SA-Co/S YN into the training mix results in significantly improved performance. The performance further increases after adding the hugh-quality SA-Co/HQ data due to its quality and distributional similarity with SA-Co/Gold. Although SA-Co/HQ is large-scale and in-domain with SA-Co/Gold, SA-Co/SYN shows further gains on SA-Co/Gold when added on top of SA-Co/HQ.
SA-Co/Gold (AII)
SA-Co/Gold-MetaCLIIP SA-Co/Gold Hol-Goldd-iki-eeee Wiki-Food&& &rink (in domain)
(in-domain)
(in-domain)
Training data
cgFi
II. MCC
pmF;
cgF1
II.
MCC
pmF
cgFt
IT. MCC
pmF1
SA-Cw/XT
23.7
0.46
50.4
21.5
0.45
47.7
20.5
0.45
45.4
1% SA-Co/HQ
34.0
0.57
59.6
30.8
0.56
54.6
33.4
0.55
60.7
4% SA-Co/HQ
37.3
0.62
59.6
35.4
0.65
54.7
39.2
0.66
58.9
10% SA-Co/HQ
40.0
0.65
60.9
37.5
0.67
55.7
46.6
0.71
65.5
20% SA-Co/IIQ
42.2
0.68
61.8
38.5
0.69
56.1
50.3
0.74
67.6
+ 100% SA-Co/HQ
45.5
0.71
64.0
40.3
0.71
57.1
53.3
0.77
68.9
Teacher (Human)
70.8
0.944
71.5
63.3
0.936
67.7
77.3
0,964
80.2
Table 12: SA-Co/HQ scaling, SA-Co/EXT data alone not enough to solve SA-Co/Gold, training on SA- Co/HQ scales well with increasing amount data. Human performance given as an estimated range where F.4 for details. Ablations use lighter model and training setting vs. SAM 3.
applicable, see
SA-Co/HQ Scaling Law. Tab. 12 investigates sealing behavior of the SA-Co/HQ training data. For this ablation, the data mix is sampled randomly from the entire SA-Co/HQ (collected from the three phases in 444 at a fixed percentage. We also report scaling behavior on two specific subsets of SA-Co/Gold: the MetaCLIP Xu et al. (2024b) subset annotated with generic caption-derived NPs, and Fiki-FFodd& Drink subset annotated with fine-grand grained NPs from SA-Co Ontology nodes. SA- both subsets expected, since they are from the same distribution
Co/HQ improves performance (in- domain). We also report the Teacher (Human) performance in the last row. Due to the simplified setting, the gap between SAM 3 and Human is larger than that of the best SAM 3 model.
14

---

Published as a conference paper at ICLR 2026
SA-Co/Gold (AII)
SA-CooGooddMMetaCIIP GiiCC-GGlld-Wii--ooooooo
(in-domain)
(out-of-domain
Training data
cgFi
IL_MCC pmffi
cglii
MCC
pmfi
cgli
IL_MMC
pmff
SA-Co/EXT
23.7
0.46
50.4
21.5
0.45
47.7
20.5
0.45
45,4
1% SA- SA-Co/SYN
30.1
0.52
57.3
31.0
0.58
53.7
26.2
0.44
59.1
4% SA-Cu/SYN
30.7
0,53
56.9
31.8
0.59
53.6
27.8
0.47
59.7
+ 15% SA-Co/SYN
32.1
0.56
56.6
33.3
0.62
53.4
29.5
0.50
59.5
+ 45% SA-Co/SYN
32.8
0.57
56,9
34,5
0.64
53,6
30.1
0.51
59,6
Teacher (SAM
verifiers)
55.4
0.84
48.3
58.5
59.0
0.87
68.1
Table 13: SA-Cw/YN scaling. SAM 3 benefits from increasing SA-Co/SYN data, both on MctaCLIP which is in-domain with the synthetic data, and on Wiki concepts which out-of-domain of the synthetic data. The teacher that generated the SA-Co/SYN data consists of older version of SAM 3, and Al verifiers from the SAM 3 data engine. Ablations use a lighter model and training setting vs. SAM 3.
SA-Co/SYN Scaling Law. Tab. 13 shows that SAM 3 scales well with SA-Co/SY data on SA- Co/Gold benchmark as it benefits from the large scale concepts captured from image captions generated by Llama4 and alt-text associated with the images, for both the in-domain MetaCLIP subset and the out-of-domain Wiki-Food&Drrnk subset within the SA-Co/Gold benchmark. The last row shows the Teacher performance (an older version of SAM 3 and AI verifiers) is much better than the student, and explains why SA-Co/SYN is useful. When comparing the SA-Co/S in Tab. 1 and SA-Co/HQ in Tab. T2 the lower in- in-ommii performance gap on MetaCLIP (42.5 vs. 49.0) comes from the relatively weaker annotation quality of IA-Co/SYN: due to lacking of the human correction step. The gap is larger on the out-of-domain
set (37.4 vs. 59.9), because SA-Co/SYN only covers the MetaCLIP images and noun phrases from a captioning model; see Table 26 We also show in Fig. 9 that with additional in-domain synthetic data, we can close the performance gap on SA-Co/Gold- -o-Co//old-iikiFFooddde Wiki-Food& Drink subset without any human involvement.
Hard Negatives. We ablate the number of hard negative noun phrases in SA-Co/HQ per image in Tab. 9b] We show that increasing the number of negatives improves SAM performance across all metrics, most notably MLC. Hard negatives phrases that are not present in the image but that (a previous generation of) SAM predicts masks for, i.c., they are adversarial to (a previous generation of) SAM 3. Training such difficult distractors helps improve the image-level classification performance captured by the IL_MCC metric.
SAM 3 and AI Verifiers. AI verifiers improve performance over the final SAM 3 model alone the
PCS task, as shown in Tab, Dd. with per-domain results in Tab, 14 We first replace the presence score from SAM with presence score from the Exhaustivity Verification (EV) AI verifier (given the image and noun phrase with no objects as input, the probability of not exhaustive, defined in Tab (2). This results in a +7.2 point gain in cgF1, from both MLC and pmF,, The reason why EV presence score can even improve pmF, is because it serves as a better calibration of object scores. Then we apply the Mask Verification (MV) AI verifier to each mask, and remove the rejected masks. This results in a further + +1.1 point gain in cgFi. The system closes nearly half the gap between SAM 3 and human performance, which indicates potential further improvements of SAM 3 by scaling up the SA-Co/SYN data and SAM 3 model size.
Natadl
SA-16
Crarded SEC
Sports
Altibiato LL SEC
WiAd-C
IL 1.C8
sal
MEC
IL MOC maF:
laf:
1.3
34.0
80.57
45.3
33.5
0.53
05.4
10.3
831
533
850
0.87
71.5
54.2
7101
43.9
0,70
R007
SASET WTV
52.2
2.56
70.6
51.3
0.5
640
56.
0.99
62.9
61.5
666
676
0.6
78.5
67.5
0.83
76.6
71.1
0.11
77.8
51.1
8.76
67.1
SASA ASNAMYY
82.
1357
11.1
56.5
038
64.3
360
0.90
662
009
206
65.0
036
78.9
672
0.89
16.
109
1092
774
52.5
8.20
86.5
Thana
72.6
17.0
641
6.4
66.3
0.9
66.6
60k
763
0.66
812
804
0.5
83.
86A
71.6
0.09
80.1
Table 14: Per-domain results of SAM 3 + AI verifiers in Tab, 9d
B.3 AUTOMATIC DOMAIN ADAPTATION
With domain-specific synthetic data generated by SAM 3 AI verifiers, we show that one can significantly improve performance on a new domain without any human annotation. We select "Food & drink" concepts with MetaCLIP images as the new domain. We generated three variants of synthetic training data on this "Food & drink" domain, while ensuring that no data from the new domain used in training the AI annotators (including SAM 3 and Al verifiers):
15

---

Published as a conference paper at ICLR 2026
PL-Food: We select "Food&drink" Wiki nodes and mine images from MetaCLIP (refer to Concept Selection, Offfine Concept Indexing and Online Mining steps in $E.4 for more details on data mining). For aseudo-annotatto fine-grained "Food&drink"" concepts, we use Wiki ontology to identify relevant coarse-grained concepts that SAM 3 works well on and prompt SAM with them generate masks. This data is similar to typical pseudo-labeled data used in prior work for detection self-training (c.g. Minderer et al. (2022))
SA-Co/SYN-Food PL-Food is cleaned by Al verifiers: MV AI verifier to remove bad masks, and EV Al verifier to verify cxhaustivity/neeaaiivy of (image, noun phrase) pairs, as the AI verification
step in Fig. 5
SA-Co/HQ-Food: PL-Food is cleaned by human verifiers for both MV and EV tasks. For non- exhaustive datapoints after EV, human annotators further manually correct them, as the "Correct"
step in Fig. 5
We study the data scaling law of these three variants by evaluating their performance on the Wiki- Food&Drink subset of the SA-Co/Gold benchmark.
Pl-Foood
So-oo
SA-CWHHQ-Foo
Teacher
Baseline
60
60
cgP2 (new domninn)
50
cgTy (new domnin)
30
20
New domain data
New domain data
(a) Data scaling mixing pre-training data 1:1 ratio. (b) Data scaling without mixing pre-training data.
Figure 9: Domain adaptation via synthetic data. (a) SAM Al verifiers (teacher system) annotate synthetic (SYN) data new domains (e.g., fine-grained food concepts) and achieve similar scaling hehavior as with human--omottree annotated (HQ) data. (h) Not mixing high-quality pre- pre-training data can limit performance gains when fine-tuning on new domains, particularly when using synthetic data.
We train the models in 2 steps to isolate the impact of the data from the new domain from other data as well as to amortize training costs. We first pre-train a base model using "SA-Co/H minus S SA-Co/HQ-Food same base model with the three data variants in two settings: with or without mixing the pre-training data.
to establish base capability and a common starting point. Next, we fine-tune the
Fig. Da shows the scaling law when mixing the synthetic data for the new domain with the pre-training data in 1:1 ratio. We observe some improvement with PL-Food compared to baseline, but there is large gap between the other variants due to its lower quality. SA-Co/HQ-Fooo and SA-Co/S YN-Food have similar data scaling behavior, with SA-Co/SYN-Fooch slightly lower but eventually catching up, without incurring any human annotation cost. The model trained on SA-Co/SYNo --oooodooo eventually surpassed the performance of its teacher system, thanks the high quality pre-training data mixed during the fine-tuning.
Fig. shows the scaling law when fine-tuned with only synthetic data for the new domain. All three data variants result in poorer performance than that in Fig. 9a| In this setting, there is a larger gap between SA-Co/HQ-Food Food and Ca--CC//Y--FAA (mainly lack of exhaustivity due to no human correction). Comparing Fig. 9a and [b] it is beneficial include high-quality general-domain data when fine-tuning SAM on new domains, particularly when using synthetic data.
reflecting the lower quality of SA-Co/SY Co/SYN-Food
B.4
IMAGE DATA ENGINE ANNOTATION SPEED
Tab. 15 measures the speedup in the SAM data engine from adding Al verifiers when collecting data domain with fine-grained concepts. We the setup Fig. 9 annotating
16

---

Published as a conference paper at ICLR 2026
Wiki-Food&Drrnk data generated with data engine where neither SAM 3 nor Al verifiers have been trained on Aiki-Foo&&Drinl data. We annotate the same set of image-NP pairs in four settings:
Human (NP Input), human annotator given a single image noun-phrase pair from SA-Co/HQ- Food, and is required manually annotate all instance masks. No mask proposals or Al-verifiers are used in the loop.
Human (Mask Input). The same annotation task as "NP input" but in this setting, the human annotators starts with PL-Food, i.e. image noun noun-phrase pairs with mask proposals generated by SAM 3.
Engine (All Human) Similar to Phase in the SAM 3 data engine, humans start with PL.FFod, and sequentially perform tasks: Mask Verification, Exhaustivity Verification and Correction, All three tasks are performed by humans.
Engine (Full) Similar to Phase in the SAM 3 data engine, Mask Verification and Exhaustivity Verification tasks are completed by Al verifiers, and Correction is done by humans i.e human annotators in the manual annotation task start with SA-Co/SYN-Food.
Task
Human from NP
Human from masks
Engine all human
Engine full
Time for clttamix (sec)
90
86
50
23
Time for positive NP (sec) Time for negative NP (sec)
236
205
207
152
71
70
30
6
Table 15:
Data engine efficiency Image. Al verifiers significantly increase throughput, allowing humans to focus on challenging cases and the manual correction task. Al verifiers allow for 5x speed up negative phrases and 36% speed up for positive phrases. The time for datamix calculated based on 88.5% negatives in SA-Co/HQ, see Tab. [4 for dataset composition. Timing is calculated based the Fiki-Food&Drrnn domain. Compared to captioner- based domains, fine-graing aeee-graied domains require more research by annotators to understand the concept, lcading to much higher annotation times for negative phrases and amortized time mask.
Tab. shows that a version of the SAM 3 model and AI verifiers that were never trained in this new domain double the throughput of the data engine. AI verifiers also allow verifying generated hard negative NPs at scale with close to no numan-annotaton involvement. As SAM 3 and Al verifiers are updated with the collected data and improve, human annotators need to manually correct fewer errors. This leads to increasingly higher throughput and the collection of more challenging data for given amount of human annotation time.
In Tab. 23 we show that Al verifiers achieve similar even better performance on the MV and EV tasks than human verifiers, so the quality of annotations from these four settings are similar.
B.5 VIDEO DATA ENGINE ANNOTATION SPEED
Using the same settings as described in Section B.4 evaluate annotation speed in the video data engine by comparing Human (NP Input) and Engine (All Human) on positive video-NP pairs from SA-Co/VEval SA-V. In contrast the image data engine, we observe that starting with PL increases annotation time, but also improves exhaustivity by providing annotators with more visual cues and candidate masklets.
Task
Human (NP Input)
Engine (All Human) human
Time for positive NP (sec)
2307
3221
Number masklets per vidco-NP pair Total masklets
2.52
2.76
1700
1860
Table 16: Data engine efficiency Video. While the Engine all human is 40% slower than Human (NP Input) annotation, it yields 9% more masklets. PT. helps annotators focus regions where masklets potentially exist, but the additional time required for human verification and conrection increases the overall annotation time. This experiment was run before the final cleaning of the SA-Co/VEval data, so number of masklets might differ from the released version.
17

---

Published as a conference paper at ICLR 2026
B.6 VIDEO TRAINING DATA ABLATIONS
We analyze how much the SAM 3 model benefits from the videos and annotations in SA-Co/VIDEO obtained through the video data engine, which are used in Stage 4 (video-level) training (described further $D.4.I). Specifically, we train the model with a varying amount of masklets from SA- Co/VIDEO as VOS training data, and evaluate the resulting checkpoints on SA-Co/VExal under the VOS task with the J&F metric. The results are shown in Tab. 17] where adding masklets collected with noun phrases through the video data engine (as additional Stage 4 training data) improves the J J&F performance on both SA- Vvvall and public benchmarks such as DAVIS17 (Pont-
al. 2017a) and SA-V (Ravi et al. 2024).
SA-Co/Evval YT-IB val J&F
SA-Co/VEval val J&F
DAVIS17 val J&F
SA-V val J&F
SA-V
Stage-4 training data
test &&F
using SAM video data only
80.7
84.9
91.6
77.0
77.1
25% SA-Co Train videos
80.9
85.3
91.3
75.2
76.9
50% SA-Co Train videos
81.2
85.3
91.5
76.7
77.0
75% SA-Co Train videos
81.4
85.9
91.7
76.5
78.5
+ 100% SA-Co Train vidcos
81.4
86.5
91.5
77.4
78.0
Table 17: Scaling analysis on SA-Co/VEval under Stage (video-level) training, evaluated multiple bench- marks through the Video Object Segmentation (VOS) task under the J&F metric. Note that "SA-Co/VEval YT-IB" and "SA-Co/VEval 5A-V" refer to the subset SA-Co/VEvvl built upon YT-Temporal-I1 videos and SA-V vidcos respectively, while "SA-V referred to the VOS evaluation dataset released in Ravi et al. (2024).
C LIMITATIONS
SAM 3 shows strong performance on the PCS task in images and videos but has limitations in many scenarios.
SAM 3 struggles to generalize to fine-grained out-of-domain concepts (e.g.. aircraft types, medical terms) in a zero-shot manner, especially in niche visual domains (e.g., thermal imagery). Concept generalization for PCS is inherently more challenging than the visual domains for the PVS task, with the latter being the key that enables SAM and SAM 2 to be successfully applied zero-shot in diverse settings. Our experiments show that SAM 3 is able to quickly adapt to new concepts and visual domains when fine-tuned on small quantities of human- -nnotated data (Tab. 2. Further, we show that can improve the performance in a new domain without any human involvement (Fig. 9.. using domain-specific synthetic data generated using our data engine.
generalization to new
From formulation of the PCS task, SAM 3 is constrained to simple noun phrase prompts and does not support multi-attribute queries beyond one or two attributes or longer phrases including referring expressions. We show that when combined with MLLM. SAM is able handle more complex phrases and H
In the video domain, SAM tracks every object with SAM style masklet, which means the cost of SAM 3 inference scales linearly with the number of objects being tracked. To support real-time inference (30 FPS) on videos in practical applications (e.g., web demo), we parallelize the inference over multiple GPUs: up to 10 objects on 2 H200s, up to 28 objects on 4 H200s, and up to 64 objects on 8 H200s. Further, under the current architecture, there no shared object-level contextual information to aid in resolving ambiguities in multi-object tracking scenarios. Future developments could address this through shared global memory across multiple objects, which would also improve inference efficiency.
Supporting concept-level interactivity for PCS, alongside instance-level interactivity for PVS, poses several challenges. To support instance-level modifications without affecting all other instances of the concept, we enforce a hard "mode-switch" within the model from concept to instance mode. Future work could include more seamlessly interleaving concept and instance prompts.
18

---

Published as a conference paper at ICLR 2026
D MODEL DETAILS
D.1 MODEL ARCHITECTURE
Our architecture is broadly based on the SAM series (Ravi et al. 2024 Kirillov et al.) 2023) and DETR (Carion et al. 2020) and uses a (dual) encoder-decoder transformer architecture, see Fig. I0 for overview. SAM is generalization of SAM 2, supporting the new Promptable Concept Segmentation (PCS) task as well as the Promptable Visual Segmentation (PVS) task (Ravi et al. 2024). The design supports multimodal prompts (e.g., text, boxes, points) and interactivity, in images and videos.
SAM 3 has ~550 parameters, distributed as follows: 450M and ~30M for the vision and text encoders (Bolya et al. 2025). and ~ 100M for the detector and tracker components. We next discuss the detector architecture for images followed by the tracker components built on top of it for video.
semantic mask
detector queries
Text
Semantic Seg Heead
preseence takan
initialize or remove
geometry
Detector
Masklet Matcher
Detector Decoder
Pixel Decoder
Image Encoder
Memory Attention
Tracker
Memory Bank
detector hidden states
Multimoda
mask, point or box
Image
per-object memory
Examplar Encoder
Components from Perception Encoder
Components SAM
visual features
features
mask
boxes scores
New components in
output
Figure 10: SAM architecture. New components are in yellow, SAM 2 (Ravi et al. 2024 blue and PE (Bolya 2025) in eyan.
D.2 IMAGE IMPLEMENTATION DETAILS
The image detector is an encoder-decoder transformer architecture. We describe its details this section.
Image and Text Encoders. The image and text encoders are Transformers (Vaswani et al. 2017 trained using constrastive vision language training using 5.4 billion image-text pairs following Perception Encoder (PE) (Bolya et al. 2025). see $D.4. 4...1 for training details. As in SAM 2, the
vision encoder uses windowed attention (Ryali al. 2023 Li et al. 2022d) and global attention
in only small subset of layers (4 out of 32), where an image of 1008 pixels is divided into xx3 non-overlapping windows of 336 pixels each. The vision encoder uses RoPE (Su et al.) 2021 Heo et al. 2024) in each layer and windowed absolute positional embeddings as in Bolya et al. (2024).
The text encoder is causal, with maximum context length 32.
As in Ravi et al. 2024). we use streaming approach, ingesting new frames as they become available. We run the PE backhone only once per frame for the entire interaction, which can span multiple forward/backwwrd propagation steps through video. The backbone provides unconditioned tokens (features/cmbeeddingg) representing each frame to the dual-encoder consisting the fusion encoder described below and memory attention for video.
Geometry and Exemplar Encoder. The geometry and exemplar encoder is primarily used to encode image exemplars (if present) for the PCS task. It is additionally used to encode visual prompts for the PVS task on images as an auxiliary functionality that primarily used to include pre- pre-training data for the PVS task in stages-2,-3 3 of training (see 1D4.11) enable more modular training approach.
19

---

Published as a conference paper at ICLR 2026
Each individual image exemplar is encoded using positional embedding, label embedding (positive or negative) and ROI-pooled visual features that are concatenated (comprising "exemplar tokens") and processed by small transformer. Visual prompts (points, boxes) for auxiliary training are encoded in similar manner, comprising "geometry tokens". It is possible for neither "geometry tokens" nor "exemplar tokens" to be present (e.g. when only text prompt is used). The geometry or exemplar tokens attend to each other via self-attention and also cross-attend to the trame-embeddings of the corresponding (unconditioned) frame from the image encoder.
Fusion Encoder. The text and geometry/exemplar tokens together constitute the prompi tokens. The fusion encoder accepts the unconditioned embedding and conditions on prompt tokens using a stack of 6 transformer blocks with self- and cross-attention (to prompt tokens) layers followed by an ML.. We use vanilla self-attention operations. The output of the fusion encoder are the conditioned frame-embeddings.
Decoder. The decoder architecture follows Carion al. (2200;; Kamath et al. (202I) as a starting
point and is a stack of 6 transformer blocks. Q learned object queries (not to be confused with prompts) self-attend to each other and cross attend to the prompts tokens (made up of text and geometry/exemplar tokens) and conditioned frame-embeddingg. followed by an MLP. We use box-to- pixel relative position bias (Lin et al. 2023) in the cross-attentis layers attending to the conditioned frame-embedee embeddinn.
Following standard practice in stronger DETR variants, we use iterative box refinement (Zhu et al. 2020), forward-twics (Zhang et al. 2022a) and hybrid matching (Jia al. 2022) and Divide- And-Conquer (DAC) DETR (Hu et al. (023). 3. By default, we use Q = 200 object queries. Bounding boxes and scores are predicted using dedicated MLPs and accept the object queries as input.
Presence Head. Classifying each object in isolation is often difficult, due to insufficient information, and may require contextual information from the rest of the image. Forcing each object query to acquire such global awareness is however detrimental, and can conflict with the localization objectives that are by nature very local. To address this, we propose decomposing the classification problem into two complementary components: global-level classification that determines object presence within the entire image, and local-Ievel localization that functions as foreground-background segmentation while preventing duplicate detections. Formally, we add the following structure: instead of predicting p(query, matches NP) directly, we break it down as
p(query, matches NP) p(query, matches NP| NP appears in image) p(NP appears in image).
To compute p(NP appears in image), then fed through an MLP classification head. Crucially, the presence score is shared by all object queries. The per-query classification loss is kept as usual, but to account for the decomposition, only compute it when the NP is present in the image (see IB. ablations on supervision strategy). The same decomposition is applied to the semantic segmentation head, where we reuse the same presence score, and train the binary mask head only on the positive examples.
a presence token, which is added to decoder and
Besides being more robust false positives, decomposing the prediction in this manner is also more flexible, e.g. in typical counting tasks, we already know the NP is present in the image and instead want know how many instances are present- in this case we can simply set p(NP is present in frame)- 1. The presence token is concatenated with the object queries in all operations, but is excluded from DAC.
We also learn 4 geometric queries. Their function is similar to the 4 geometric queries in SAM 1 and 2 (where they were called "output tokens") and are used to perform the PVS on individual image or video frames during the stags-2,-3 of training, see $D.4.1 The prompts are provided by the "geometry tokens" in the form of visual prompts. The presence score is set to when performing the PVS task a single frame, the target is known to be present in the frame.
Segmentation Head. The segmentation head is adapted from MaskFormer (Cheng et al. 1021 Semantic segmentation and instance segmentation share the same segmentation head. The conditioned features from the fusion encoder are used to produce semantic segmentation masks, while instance segmentation additionally uses the decoder's output object queries. "Multi-scale" features provided to the segmentation head using SimpleFPN (Li et al. 2022d), since the vision encoder is a (single-scale) VIT.
20

---

Published as a conference paper at ICLR 2026
Handling Ambiguity. Experimentally, if we train SAM 3 model without handling ambiguities as described in in any way, we observe that the model tends to predict several valid but conflicting interpretations of the phrase. This is expected; if in training dataset given phrase has two distinct interpretations, and roughly half the data annotated assuming the first one, while the other half follows the second one, then the solution that minimizes the training loss is to output both interpretations with 50% confidence, However, this behavior is undesirable for end-users, because it produces conflicting, sometimes overlapping masks.
To address this issue, we add an ambiguity head to our model. Similar to SAM and 2, this head is mixture of experts, where we train in parallel K experts, and only supervise the expert that gets the lowest loss train K > 3 experts due to mode collapse.
We find that K = 2 performs the best and that it is more difficult to
For mixture of K experts, each producing an output with loss Lk, the mixture loss is weighted average:
iLMo 0L; 08,
LOss:
M
Σ
p& Lk
Gradient:
Pj a8,
k=
In our winner-takes-all variant, only the expert with the lowest loss receives gradient:
Loss:
h-- arg min Lx:
LWTA Ck-
θC..
iLWIA
if =
Gradient:
00,-
00,
0,
otherwise.
Backpropagating the loss only through the expert which received the minimal loss allows each expert to specialize to one kind of interpretation. This behavior is illustrated Fig.
(a) Original image
(b) Prediction by Expert
(c) Prediction by Expert 2
Figure Two interpretations of the noun phrase "large circular shape" learned by two Experts (SA-IB image).
While this strategy allows experts to specialize, does not explicitly select which expert should be used inference time. To resolve this, train classification head that predicts the expert that has the highest probability of being correct. The classification head is trained in a supervised fashion with cross entropy loss, by predicting which expert obtained the minimal loss during training. The Ambiguity head adjusts only the classification logits, leaving masks, boxes, and presence scores unchanged. We train it on top of frozen SAM model.
Finally, to detect overlapping instances, we compute the Intersection- ceeeiioonovvee--eee ver-Minimum (IoM) between masks. loM is more effective than Intersection-ore -oveuunnnon Union (IoU) for identifying nested instances. With the ambiguity head, we obtain 15% reduction in overlapping instances.
D.3 VIDEO IMPLEMENTATION DETAILS
The tracker architecture follows Ravi et al. (2024). which we briefly describe for convenience followed by discussion of the disambiguation strategies introduce.
21

---

Published as a conference paper at ICLR 2026
Tracker. The image encoder is the shared PE (Bolya et al. 2025 with the detector and provides unconditioned tokens to the memory attention using a separate neck. The memory attention receives these unconditioned PE tokens stacks self- and cross- attention layers that condition the current frame's tokens on spatial memories and corresponding object pointers in the memory bank. Memories are encoded by fusing frame's mask prediction with the unconditioned PE tokens from the image encoder and placed in the memory bank.
As in Ravi et al. 2024). the decoder includes an occlusion head to indicate the likelihood of the object of interest being visible in the current frame. During inference, the occlusion score may also be used to select frames to place in the memory bank adaptively.
SAM introduced the ability to output multiple valid masks when faced with ambiguity about the object being segmented in an image. For example, when a person clicks on the tire of bike, the model can interpret this click as referring to only the tire or the entire bike and output multiple predictions. In videos, this ambiguity can extend across video frames. For example, if in one frame only the tire is visible, click on the tire might relate to jost the tire, or as more of the bike becomes visible in subsequent frames, this click could have been intended for the entire bike. To handle this ambiguity, SAM 2 predicts multiple masks at each step of the video. If further prompts do not resolve the ambiguity, the model selects the mask with the highest predicted IoU for the current frame for further propagation in the video although other strategies possible.
Disambiguation Strategy. As outlined in $3 tracking in videos can suffer from ambiguities in mask propagation, false predictions from the detector, or limitations of IoU-based matching in crowded scenes with highly overlapping objects. In this section, we present the details of the temporal disambiguation strategies used to address these challenges. We begin by introducing the notation used throughout this section.
Let D, and M, denote the set of detector outputs and the set of tracker's predicted masks on frame T respectively. We define frame-wise matching function (T( for masklet frame 7 as
+1, if E D, s.t. (d, iou_threshold
(.(T) - -1, otherwise,
(-,
where M is the predicted output mask of object i on frame T. In addition, we define a Masklet Detection Score (MDS) over an interval [t, t'] as (,t,,t) = Σ (T) This score measures how masklet is consistently matched to a detection within a temporal window. The first frame in which object appears is denoted iima
Track Confirmation Delay. To reduce spurious and duplicate masklets, we delay the output of the model slightly. Specifically, the output at frame 7 is shown only after observing frame t + T. This delay provides temporal context for validating candidate masklets before outputting their masks. By default, we use T = 15 which achieves good accuracy at a slight delay cost of around half a second for 30 frames per second videos. During the delay, we apply the following two criteria remove unconfirmed or duplicate masklets as follows.
Removal of Unconfirmed Masklets. A candidate masklet is considered unconfirmed within the confirmation window + 7] if their MDS is below a threshold, S, 2 T) < V, and the masklet first appears within the window t 2 t. If both conditions are satisfied within the confirmation delay, we remove the masklet from the tracker's 's state. We choose V = 0, requiring that the masklet has to be matched a detection for least half of the frames within the confirmation delay period to be confirmed. This strategy helps reject some false positive detections and not track them.
Removal of Duplicate Masklets. If the tracker temporarily fails to predict mask for an object in some frames, but the detector continues to detect the object during those frames, this can lead to the creation of a new masklet for the same object. As a result, two masklets may end up tracking the same object: the original (older) masklet, and new masklet that is initiated during the period when the tracker missed the object. To resolve this issue, during the confirmation delay period, if two masklets consistently overlap with the same detection, we remove the one that started later. Specificallyy two masklets i, j are considered duplicates on frame 7 there exists a detection E D, such that [oM(,, d) > iou threshold and LoU(M d) 2 iou threshold. If the two masklets i and j found to be duplicates for least [T/2] frames, we remove the one with the latest first.
22

---

Published as a conference paper at ICLR 2026
appearance finst only if first appeared within the confirmation window [t, + T). Empirically, find that using torrhhheholld = 0.1 gives the best results.
Masklet Suppression. For confirmed masklets that were not removed during the confirmation delay, we apply an additional suppression step: if a masklet's MDS over its entire lifetime falls below zero at any frame (i.e. S, (fins 7) 0), suppress its output by zeroing out its mask. However, retain the masklet in the tracker's s state, allowing for the possibility that the object may be confirmed in future frames. This strategy primarily addresses ambiguous detections, such objects entering the scene near the boundary. For example, if only person's 's hands are visible as they enter the frame, the detector may be unable determine whether the object matches the text prompt (e.g.. impossible to distinguish between man and a woman). In such cases, if the detector subsequently fails to detect the object after it fully enters the scene, the masklet suppression criterion ensures that these masklets are suppressed, unless they are consistently matched with new detections.
Periodic Re-Prompting. In challenging scenarios involving occlusions or visually similar distractor objects, the tracker may lose track of the target object. To address such tracking failures, we periodically re-prompt the tracker using the latest detection outputs. Specifically, on every N-th frame т, we compare each detection d D, with the tracker's current predictions My. If a detection has a high overlap with the tracker's 's prediction (i.e., [oU ) 2 0.8) and both the detection score and the masklet prediction score exceed a confidence threshold of 0.8, re-initialize the tracker for that object using the detection output mask. We observed that re-prompting is most effective on frames where objects are not occluded and are fully visible, which motivates our choice of high confidence thresholds. In our experiments, we set N 16 by default. This periodic re- re-prompting helps the tracker recover from temporary failures and maintain accurate object tracking throughout the video.
Detection-Guided Re-Prompting. In cases where the tracker's predictions may drift and its predicted masks become leaky, we employ the detectors' outputs. For each frame we compare every detection d E DD, with the tracker's s current predictions M,- If the highest-matching detection d has a low bounding box IoU (i.e., Loo (d, M < 0.85) with the corresponding tracker prediction M we recondition the tracker for that object using the latest detector output. This approach ensures that the tracker remains synchronized with reliable detection results.
The impact of these strategies is ablated in Tab. 39 and they show quantitative improvements across evaluation sets.
D.4 MODEL. TRAINING
D.4.1 TRAINING STAGES
SAM is trained in 4 stages, with each stage introducing new capabilities or refining existing capabilities. Tab. 18 lists the data used in each stage.
Stage 1. Perception Encoder (PE) pre-training (Bolya et al. 2025), which pre-trains the image and text encoders with 5.4 billion image-text pairs. In addition broad concept coverage, this stage is key for robustness (see $B.I.. Since the vision encoder has to support multiple tasks (while also not being too large) we opt for an "L+" size; The vision and text encoders are transformers with 450M and 300M parameters respectively. We largely follow Bolya et al. (2025), but do not use distillation and do not perform video fine-tuning in this stage.
This stage is for detector pre-training and trains the (image-level) detector as well as the vision and text encoders with large-scale image segmentation data (including video frames as images). This stage uses both stage is broad concept coverage (image, noun phrase, masks) tuples. At the end of this stage, the model is able to do open- vocabbbuyy object detection, instance and semantic segmentation across many domains fairly well.
Stage 2.
labellee and human--nnotated data, see Tab. [8] The main goal of this
An additional goal of this stage is prepare the base model for tasks in subsequent stages. To prepare for the PCS task, (image, noun phrase) pairs are randomly (p- = 0.2) converted into visual queries (i.e. noun phrase is dropped) augmented with input bounding boxes (p 0.2).
23

---

Published as a conference paper at ICLR 2026
Besides training for the PCS task, in this stage, the model is also pre- pre-trained on the visually prompted PVS task. This is done by adding 4 decoder queries specific to this task following the design of SAM 1 & 2. Training data includes images (e.g.. SA-1B) and videos frames (c.g, SA-V), see Tab. [8] the number of interactivity steps is restricted to 4 for efficiency. We largely follow the settings from
Ravi et al. (2224) but use the Align loss (Cai et al.] (024) in lieu of the loU prediction loss, co-opting the classification head for object queries for this task.
We train for 995k iterations with a batch size of 896 with 5k warm up and cooldown steps using AdamW (Loshchilov & Hutter 2019). We apply layer-wise learning rate decay (Clark el al. 2020) of 0.9 to the vision encoder. We use a reciprocal square-root schedule (Zhai al. 2022 and weight decay of 0.1. We use an initial learning rate of 5e-4, le-4 for vision and text encoder and 1e-3 for all other components. For boxes, we use I,1 and gloU losses with weights of 5 and 2. Classification loss uses weight of 100 and focal and dice losses use weights of 200 and 10 respectively. The encoder and decoder use a dropout of 0.1.
Stage 3. This stage further trains the model with the highest-quality human annotated image segmentation data, expands the interactivity capabilities and introduces post-training to improve detection performance.
Specifically, in terms of interactivity, (a) in the PVS task, the number of interactivity steps is increased to 7 and (b) interactivity is introduced into the PCS task, where positive or negative exemplars are provided based on model error, as described next. We iteratively sample box prompts to mimic the real user policy. Positive boxes sampled from false negative errors, and we prompt their corresponding ground-truie boxes. Negative boxes are sampled from high-confidence false positive predictions that do not have significant overlap with ground truths. At each iteration, the box inputs are added on top of the previous ones. If both a valid positive and negative box exist, we randomly select one; if no valid candidates are available, no additional prompt is given. The process is repeated for 5 iterations.
The expanded interactivity in the PCS and PVS in this stage significantly slows down training, but the extensive pretraining with limited interactivity for the PVS and no interactivity for PCS (but using image exemplars together with text prompts) prepares the model well ensure that a short stage 3 sufficient.
This stage retains only the highest quality, exhaustivity verified data (e.g.. SA-Co/SYN is dropped) and introduces a presence token (and presence loss) to better model presence of target segments and their location location greatly increasing the precision of the model. The presence loss is binary cross-entropy loss with weight of 20. All learning rates are lowered by factor of 0.025. We train for ~55k iterations with a batch size of 512, with other settings identical to stage 2.
Stage 4. For video, the tracker decoder is trained on top of the frozen backbone. Freezing the backbone at this stage is made possible by pre-trainmn pre-training on VOS data in previous stages at the video frame level. This stage retains the strong spatial grounding of the previous stage and focuses on spatial-temporal tracking without degrading other capabilities. We use a batch size of 512, train for ~ 1990 iterations using cosine schedule with peak learning rate of 5.0e-1 and a linear warmup of 1k iterations. We supervise the model's outputs using a weighted sum of losses: a linear combination of focal and dice losses for mask prediction, mean absolute error (MAE) loss for IoU prediction, and cross-entropy loss for object occlusion prediction, with respective weights of 20:1:1:1 For multi-mask predictions, we only apply supervision to the mask with the lowest segmentation loss. If frame's 's ground truth does not include a mask, we do not supervise any mask outputs for that frame; however, we always supervise the occlusion prediction head, which determines whether a mask should be present. As in Ravi et al. (2024), we further fine-tune the tracker with longer temporal context using 16-frame and 32-frame videos for 60k iterations, while scaling the learning rate by factor of 0.1.
D.4.2 ADDITIONAL. TRAINING SETTINGS
Data augmentation. For the PCS task, we apply the following transformations:
We use some cropping and resizing to vary the aspect ratios and help with small objects. The input resolution of our model is always a fixed square (usually 1008 x 1008). During evaluation, the images are resized to this size, without respecting their aspect ratio. During training,
Geometric:
24

---

Published as a conference paper at ICLR 2026
Dataset
Ingested As
Train
Test
Stage 1
Stage 2
Stage
Stage 4
Promptable Visual Segmentation- Images
SA-IB
Image
SA-Co/VIDEO
Frames
SA-Co/VIDEOEET
Frames
SA-37
Image
Promptable Visual Segmentation
Videos
SA-Co/VIDRO
Video
SA-Co/VIDEO-EXT
Vidco
SA-V val
Video
LVOSv2
Video
Promptable Concept Segmentation In Images
SA-Co/SYN
Image
SA-Co/HQ
Image
SA-Co/EXT
Image
SA-Co/VIDEO
Frames
SA-Co/Gold
Image
SA-Co/Silver
Image
SA-Co/Bronze, SA-Co/Bio
Image
Promptable Concept Segmentation- In Videos
SA-Co/VEval
Vidco
Table Dataset usage across different tasks and training stages.
we apply our augmentations, and pad if the resulting size is smaller than 1008 1008. We found it important to randomly distribute the padding on all sides, to avoid creating biases towards one particular region of the image. If the dataset does not contain notions of left and right, we also apply random horizontal flips.
Semantic: When training on datasets with a closed vocabulary, we leverage our mapping to wikidata to further enhance the training. There are three main ways we can leverage the ontology: (i) to sample synonyms, which expand the vocabulary of the model; (ii) to sample negatives (typically, the dataset is exhaustively annotated, we can sample any node the graph that corresponds to category and is not present in the image); and (iii) ensure the hierarchy closure of the concepts (for example, if we have some annotations for "canoe" and "boat" the same image, need to make sure that all the "canoe" objects also labeled "boat" since canoe is a type of boat).
To prevent the model from randomly making predictions for unsafe concepts, we randomly sample some of them at train time and add them as negatives. These concepts mainly include slurs of all kinds. We also try to prevent the model from making predictions for subjective and non-vise vonuvisual adjectives, especially when applied to person. This includes flattering ones (such as "a smart person") as well as derogatory ones (such as "a dull person")).
Safety:
further increase the complexity of the images by doing mo- saics (Bochkovskiy et al. 2020). The maximal grid size our mosaics is 3 x 3, and we sample any configuration that is most that, including irregular ones, long the constituents square. For example, in a 3 x regular grid, we can have a large image that effectively covers 2 area, and use 1 for the remaining 5 slots. Unifying different images can be tricky in an open vocabulary setting, since there is no guarantee that concepts are exhaustively annotated, For example, one image has car annotated, but the second does (neither as positive nor negative), then we do not know if the second image has a car or not, and thus could create some labeling noise. To avoid this, we only mosaic datasets that have low chance of such missing annotations (either closed vocabulary ones, or some created with specific mining patterns). To merge annotations, we again rely on the wikidata mapping available, otherwise rely on plain-text queries to merge appropriately.
Mosaics: On some datasets,
still
E DATA ENGINE DETAILS
The overview of the SAM data engine's components is shown in Fig. 5 this section we provide further details of how each component is implemented in the image (phases 1-3) and video (phase 4)
25

---

Published as a conference paper at ICLR 2026
versions of the engine. The datasets collected in each phase and the performance improvements in Tab. 19
E. MEDIA POOL
The media (image and video) pool consists of many sources with varying visual domains, from web- scraped data to datasets collected for specialized domains such as art, food, or driving. Tab. 26 lists the datasets used to mine media for each subset of the SA-Co training data. The web-scraped images and alt captions are sourced from MetaCIIP (Xu et al., 2024b), a curated version of CommonCrawl. We further expand coverage by mining media from a large pool with the help of a curated ontology. Compared previous works such as OWLv2 (Minderer et al. (2024)) which mainly rely uncurated web-scraped data, our target mining strategy resulted in coverage 12 media domains.
E.2 SA-Co ONTOLOGY
To track and improve the coverage and overall distribution of concepts in our data, we build a custom SA-Co ontology of visual concepts from Wikidata (Vrandečié & Krötzsch] 2014), which covers comprehensive set of entities and offers hierarchical information with its graph data structure. We manually select high-level Wikidata nodes (e.g., Human, Mammals) and recursively include all of their descendants. The resulting 22.4 million nodes are classified into 17 top-level categories (e.g. animal, furnishing & home) and 72 sub-categories (e.g.. birds, home appliance). The full list of categories and Wikidata node counts are shown Tab. 20 We further develop a mapping process that can map arbitrary NP to node in the SA-Co ontology by leveraging retrieval model (Sentence- BERT) to source candidate nodes and an Al annotator as judge (Llama 3.2) to select the closest match. This mapping used track the distribution of nodes in the dataset (see Fig. 12) as well as to create negative phrases (see below for details).
E.3 PHASE 1: HUMAN VERIFICATION
Data Mining, During this phase, we randomly sample images from MetaCLIP.
Proposing NPs. We generate image-level captions using the BLIP-2 captioner (Li et al. 2023b) followed by the spaCy parser (Honnibal et al. 2020) to parse the caption into NPs.
Proposing Masks. We prompt off-the-shelf ppcc-vocabulary detector, FIBER (Dou et al. 2022 or OWLv2 (Minderer al. (024) with the noun phrase and use the resulting boxes to prompt SAM to generate mask proposals.
Verification (Human). Verification of mask proposals consists of two tasks which can be performed by human or Al annotators: mask quality verification and mask exhaustivity verification. In Phase 1, verification is done by humans only. Each human verifier works exclusively on one task type.
Mask Verification (MV). Given triplet of an image, noun phrase and a of candidate masks for that phrase, the task is to accept reject each of the masks. A mask is accepted if it matches the given noun phrase and is high quality (no holes, coverage issues, etc.) If the mask is unrelated to the phrase, or low quality, it is rejected.
Exhaustivity Verification (EV). All accepted masks from the verification task are sent to an ex- haustivity check. Given an image, noun phrase, and any accepted masks that passed the previous mask verification for that phrase, the task is to decide whether or not the accepted masks (if any)
SA-Co/HQ
SA-Co/SN
SA-Co/EXT
SA-CO/VIDEO
SAM 3 performance
#unnotation domains
SA-Co/Gold SA-Co/Silver SA-Co/Esal
#img #img-NP
#img
#img-NP
fing
Aimg-NP #vid
#vid-NP
(egFi)
(test (HOTA)
Phase
1.2M
4.3M
0
0
Phase 2
2.4M
122.2M
5
0
0
0
21.4
18.9
Phase
1.6M
19.5M
15
39.4M
1.7B
9.3M
136.6M
54.4
50.5
Phase
52.5K
134.3K
54.5
50.1
63.9
Table 19: Data engine phases SAM 3 progress.
26

---

Published as a conference paper at ICLR 2026
animals
2.3M
electronics
10.2K
12. object parts
101.9K
insects & crustaceans
1.7M
electronics
6.9K
body parts
73.8K
molluses
188.4K
cameras
3.3K
other object parts
26.1K
other animals
166.5K
equipments
14.9K
13. other products
3,5
& other
85.7K
military equipments
other products
2.7K
birds
52.4K
sport equipments
2.0K
celebration supplies
384
mammals
38.3K
safety equipments
1.2K
animal-related products
359
reptiles
28, 2K
medical equipments
1.K
tobacco products
51
echinoderms
23.0K
agricultural machinery
458
14. patterns & material
896.6K
amphibians
14.2K
fashion & beauty
7.5K
885.6K
2. art, history & religion
3.1M
Fashion
3.9K
patterns & shapes
10.9K
artworks
beauty & healthcare
3.7K
plants fungi
1.5M
products
collectibles
10.2K
9. food & drinks
33.1K
plants
1.IM
religious objects
9.1K
disbes
12.9K
fungi
376.4K
flags
8.3K
other food
6.7K
16. tools & appliances
14.5K
musical instruments
4.9K
fruits
6.6K
other appliances
6.5K
gemstones
526
drinks
6.3K
3.0K
material
438
vegetables
621
tools
1.8K
3. buildings & locations
2.7M
10. furnishing home
2.7K
kitchenware
1.6K
places
2.4M
furnishing
1.3K
containers
945
goographical features
343.2K
home appliances
486
light sources
672
4. celestial
9.2K
stationery
472
17.
transportation
258.1K
meteomlogical phenomena
homsehold supplies
space reluted
human
11.3M
land vehicles
light reluted
734
humans
11.0M
aircraft
27 .K
5. documents &
201.3K
occupations
140.8K
other vehicles
6.4K
glyphs logos
173.4K
fictional characters
87.8K
transport infrastructures
3.7K
21.0K
gestures & expressions
158
construction machines
100
documents
6.0K
curds
435
infographics
324
GUI layout elements
135
maps
23
Table 20: SA-Co ontology top-level categories and sub-categories with corresponding node counts in Wikidata.
exhaustively cover all instances of the phrase in the image. If there are unmasked instances of the phrase, annotators decide whether or not at least one of the remaining instances is separable, if the remaining instances are too crowded together to separate. Phrases that are annotated non-exhaustive from this step are sent to the correction task. Phrases that are annotated as exhaustive are directly sent to final annotations.
Correction. Human annotators are given the same input as the exhaustivity task: an image, noun phrase, and any (0 or more) accepted masks from the mask verification task. Annotators manually add individual masks for the unmasked instances of the noun phrase by prompting SAM with clicks in browser based tool. If there are non-separable occurrences of the phrase, annotators use special group masks to indicate that the mask covers more than a single instance. The output of the task is the
complete set of instance and/or group masks covering all pixels in the image corresponding noun phrase. Noun phrases that are not present are submitted with no masks. If it not possible to reach complete set of masks due mask complexity, the annotator rejects the job.
In each task, annotators are given the ability to reject the image-NP pairing if they decide the phrase maskable set of objects (e.g "it", "blue") or if after research they still unsure if it is present (e.g., tine-grained species of animals). Filtering out vague phrases and allowing annotators to be unsure increases the consistency and agreement in the resulting annotations.
27

---

Published as a conference paper at ICLR 2026
E.4 PHASE 2: HUMAN AI VERIFICATION
Data Mining, We use a retrieval model (including Perception Encoder, DINOv2, and MetaCLIPv2) for mining concepts that are challenging and not prevalent in the caption NPs from Phase We leverage our SA-Co ontology to determine the list of candidate concepts, followed by offline concept indexing and online mining from MctaCLIP.
Concept Selection. We use a taxonomy-guided mining strategy to balance the overall ontological distribution, expand concept coverage and enhance performance on long-tail and fine-grained phrases. Two groups of concepts are selected from the SA-Co Ontology for targeted mining: Wiki-Common are nodes judged by LLM to be common concepts, Wiki-FG are all nodes from the "sports equipment" and "food and drink" sub-graphs, chosen to test the model's ability to generalize very ine-grained concepts like "kefir", "pastille", "kettlebell".
Offiine Concept Indexing. For every new concept, we collect reference images from Wikimedia and compute their K-dimensional embedding offline. We aggregate the embeddings from all reference images resulting in single embedding per concept. We repeat the process across all N concepts resulting in an N dimensional offline index.
Online Mining. Relevant images for each concept are retrieved using both image and text based mining. With image-based retrieval, we compute the embedding on every image, run KNN on the offline concept index followed by top-k sampling, and apply threshold before mapping to a specific concept. With text-based retrieval, we compute CLIP based similarity scores between the text embedding from input concepts and image embeddings from the corpus and apply a threshold before mapping the image to specific concept.
The following additional mining strategies are used to further refine the selection.
Image-Type Balancing. Web datasets are usually dominated by few types of images such as ads or product photos. To avoid over-representation 3.2) and prompt it zero-shot to classify an image into image types (such as ads, product photos, indoor and outdoor scenes, infographics), and sample based on a type-agnostic probability.
certain image types, we use a MILM (Llama
Proposing NPs. We improve this step generate higher-quality and more diverse noun phrases.
Ievel-Level Captioner and Parser. We use an image captioning model (Llama 3.2) to generates image-level captions and phrase parser (Llama 3.1) that proposes noun phrases given the caption. The Llama 3.2 captioning model improved concept recall compared to BLIP-2 from Phase 1. The phrase parser is fine-tuned for this task and significantly outperforms its zero-shot model variant and spaCy parser.
Removing GronGroundable Phrases. The parser can generate non-specific phrases such as "it", "them" or hard to segment phrases such as "middle". To address this, we use another Al verifier (MILLM that fine-tuned to classify such cases and remove them from the rest of the pipeline.
NP Balancing. We employ heuristics to avoid collecting too many frequent or easy objects. We remove NPs if the data engine has already annotated enough instances, if the SAM 3 has high accuracy when prompted with the NP, and based on fixed list (e.g. that occur frequently, are harmful). From Phase 3 we rely on AI verifiers to remove easy cases.
Cleaning NPs. We singularize noun phrases, deduplicate nearly-identical ones, and remove possessives.
Hard Negative Proposal. A hard negative phrase generator proposes image-level negative phrases, i.c. those that do not exist in the image and are adversarial to SAM 3. Given verified positive NPs (i.e. that exist in the image), negative NPs are proposed and then checked for adversariality.
Proposal. The proposal of hard negatives is done in two ways. The first approach maps every positive NP to a node in the SA-Co ontology, then navigates the ontology graph to find sibling, cousin, or uncle nodes corresponding to different but related concepts. For example, the noun phrase "gray Siamese cat" maps to the node "Siamese cat". which could result in negative candidates like "tabby cat" (sibling), "dog" (uncle), "Chihuahua" (cousin). The second approach relies on an MLLM (Llama 4), which proposes visually similar negatives for every positive NP.
28

---

Published as a conference paper at ICLR 2026
-Ceek for Adversariality Once the negative NPs are proposed, they are filtered to retain only those adversarial to the current SAM 3 version. For each negative NP candidate, predictions from SAM 3 are obtained. If the set of predictions is empty, the candidate is discarded. If the model predicts one or more objects, these predictions are compared to the original segmentation masks of the corresponding positive NP. If the overlap between the negative NP predictions and the positive NP annotations exceeds a certain threshold, the negative NP is retained as a hard negative. This final check is necessary because initial proposals may not be true negatives and instead may be only negatives relative to the existing positive NPs (i.e. the object could still be present somewhere else in the image).
Proposing Masks. We prompt SAM 3 with the set of positive and negative phrases to produce candidate instance and semantic masks for the image. For geeudo-annotaiin domains with fine- grained concepts that SAM 3 fails on (e.g., Zanclus cornutus), we identify the relevant coarse-grained concept that SAM 3 works well on frog), and use this as the prompts to generate masks, We deduplicate masks generated per NP based on a IoU metric. These noisy pseudo-labels undergo further cleaning by both human and Al annotators.
Verification (Human+AD. We train "AI verifiers" to perform the mask verification (MV) and cxhaustivity verification (EV) tasks. More specifically, we fine-tune Llama 3.2 Dubey et al. (2024) on human annotated data collected during Phase 1 of the data engine for both tasks.
Task Formulation. Tab. 21 provides example data point of the mask verification task: given an (image, phrase, mask) triplet, we render the mask on top of the image as the image prompt, provide the task guidance text prompt, and use the human annotation (1 out of 5 choices) as output. Each mask's s quality is evaluated independently from other masks for the same image-phrase pair, Rendering tricks are used better visualize small objects, and avoid color confusion from mask overlay. Tab. [2 provides an example data point of the exhaustivity verification task: given the (image, phrase, masks) triplet, we render the bounding boxes of the masks on top of the image and use this as the image prompt, provide the task guidance as the text prompt, and use the human annotation (1 out of 6 choices) as the output.
Evaluation. We construct test sets for "AI verifiers" from jobs that were reviewed by multiple human annotators for all SA-Co test sets. We leave one human annotation as human prediction, and use the majority vote of the remaining human annotations as ground truth. This allows us to compare human and Al verifiers' accuracy.
Training. The training data of each task comes from not only the task itself, but also from the Correction task. For example, each manually added mask is a good data point in the mask verification task. Each exhaustively finished job in the Correction task results in a good data point in exhaustivity verification task. We merge all training data for these two tasks together (over 200M image-text pairs) to pre-train a foundational AI verifier, and then only use high quality human annotated data from the task itself (around 10M scale) to fine-tune two Al verifiers, one for each task.
Result. Thanks to the simplicity of these two tasks (MCQ tasks on image-text pairs) and the large volume of training data from Phase 1, Al verifiers reach and even surpass human performance on these two tasks, as shown in Tab. 23 We also evaluate the system of SAM 3 and AI verifiers end-to-end on the PCS task, and the system always performs better than the single SAM 3 model, as shown in Tab. 9d
Generalization to new domains. We also study the generalization ability of AI verifiers. For a given new domain, the MV AI verifier is typically on par with human verifiers without any domain specific data; the EV Al annotator is typically worse than human in zero-shot evaluation, but can reach human performance with only thousands of domain specific data points.
As discussed in $B.4, using AI verifiers is effective and allows human annotators to focus on the most challenging data points, i.e. those that have poor mask quality or missing masks. This approach more than doubles the throughput of the SAM 3 data engine. As both SAM 3 and AI verifier models improve, more data be exhaustively annotated using only SAM 3 and AI verifiers. This leads to increasingly higher throughput and ensures that human annotators only work on SAM 3 failure cases.
Correction. We perform manual correction wherever needed described in phase 1.
29

---

Published as a conference paper at ICLR 2026
Input
White imaage
Image input for Al verifier (left) and UI for human annotation (right)
AI Verifier Instructions
You are an expert annotator of object segmentation masks. an image and pre-defined label, you are given mask and asked to evaluate the quality of the mask. Follow the following rules when rating the mask.
Rate the mask as "Accept" when the label accurately describes the masked object and the mask covers the object with good boundaries. We do not need masks to be pixel-perfect for this task. However, the mask should cover all important parts of the object.
2. Rate the mask as "Accept as text" when the mask covers text and the label exactly matches the masked text. The mask should cover all important parts of the text (all specified letters/punnctuationnttt).
Rate the mask as "Flag label" when the label corresponds to Race, Ethnicity, Sexual orientation, Religion, Sacio-cconomic status, Med- ical conditions. Disabilities, Derogatory terms/profanity and Animal phrases for person.
4. Rate the mask as "Whole image" when the label corresponds to the entire image. Description that refers to the whole image may describe setting (e.g., inside, outside, day. night). type of media (e.g.. flier, screenshot, photo), type image (e.g.. close up. an aerial view) and location (e.g., an airport, the woods, a bedroom).
5. Otherwise, rate the mask as "Reject".
Please give your rating directly without any explanation. Now let's start. the given figure, the left half shows fuchsia box highlighting the region of interest in the original image, and the right half shows fuchsia mask overlaid on zoom-in view of that region.
Rate the fuchsia mask the label "a computer monitor": (A). Accept as text. (B). Flag label. (C). Reject. (D). Accept. (E). Whole image.
Mask
Verification
(D). Accept.
Result
Table 21: example data point Mask Verification (either human AI verifier). The Al-verifier given two crops of the image, a zoomed-ous view where the object is highlighted via box and a zoomed-in view where the mask is highlighted. This allows better visualization of small objects, and avoids color confusion from mask overlay.The Al-verifier instructions annotators. Human annotators also given options reject the phrase the image-level vague. For Al verifier, we use the model output logits the choice tokens (c.g., A/B/C/D/B) as logits for the corresponding labels, and soft-max over the logits to get their probabilitics.
condensed version of the annotation guidelines given to human
30

---

Published as a conference paper at ICLR 2026
Input
CRIECT
Image input for Al verifier (left) and UI for human annotation (right)
AI Verifier Instructions
You are an expert annotator of object detection. For an image and a pre-defined label, you are given some boxes and asked to evaluate if the boxes are exhaustive. Follow the following rules when rating the boxes.
1. Rate the boxes as "Accept" when there are no undetected objects that match the label or the label is vague (i.e, "rent", "it"") or malformed (heavily misspelled).
2. Rate the boxes "Reject" when there undetected objects and least undetected instances can be separated.
3. Rate the boxes "Reject but unseparated" when there are undetected objects but cannot be separated. This common with groups where it is not possible to tell undetected instances apart distinguish them from others.
4. Rate the boxes as "Flag label" when the label corresponds to Race, Ethnicity, Sexual orientation, Religion, Socio-cconomic status, Medical conditions, Dis- abilities, Derogatory terms/profanity
Animal phrases for a person.
5. Rate the boxes "Whole image" when the label corresponds to the entire image. Description that refers to the whole image may describe setting (e.g.. inside, outside, day, night), type of media (e.g., flier, sereenshot, photo), type of image (c.g., close up, an acrial view) and location (c.g., an airport, the woods, bedroom).
6. Rate the boxes "Ungroundable Vague / Unsure" when the label is ungroundable or vague concept (e.g.. corner", "gap") or when you are unsure whether or not the phrase in the image.
Now let's The given figure shows of interest the original image, evaluate the box for the label computer monitor" and select your answer from the following options: (A). Reject but unseparated. (B). Whole image. (C). Reject. (D). Ungroundable Vague Unsure. (E). Accept. (F). Flag label.
multiple red boxes highlighting the region
Exhaustivity Verification
(E). Accept.
Result
Table 22: An example data point of Exhaustivity verification (either human AI verifier). For Al verifier, objects to are highlighted via boxes in the image. If there are no candidate objects, the original image is used and "one or multiple red boxes" is replaced by "zero red box" in the text prompt. For Al verifier, we use the model output logits of the choice tokens (e.g.. A/B/C/D/E/F) as the logits for the corresponding labels, and soft-mat the logits get their probabilities. The presence score from the EV AI verifier is defined as I Proh( (Aceept in boxes input). Prob( |ccept|/no masks as input) is the probability of Accept missing objects) given zero detections as input, which is equivalent to the probability of NO presence.
Attributes
Crowded
Food&& Sports Equip. MetaCLIP
SA-1B
Wiki-Common
Average
MV
EV
MV
EV
MV
EV
MV
EV
MV
EV
MV
EV
MV
EV
MV
EV
Human
72.3
81.2
72.9
82.4
76.8
76.7
87.3
72.4
79.5
72.3
73.8
91.5
81.8
Al verifier
77.1
82.6
74.6
81.3
79.4
75.1
80.1
84.7
75.3
78.8
75.9
76.8
81.3
88.4
77.7
81.1
Table 23: Human/Al verifier accuracy mask verification (MV) and exhaustivity verification (EV) tasks
31

---

Published as a conference paper at ICLR 2026
E.5 PHASE 3: SCALING AND DOMAIN EXPANSION
Data Mining. We continue the data mining approaches from Phase 2 and scale to more novel domains. In addition, we target cases that are rare in web datasets and challenging for the model: crowded scenes with high object counts and images with very small objects. To mine such images, rely on the SA-1B dataset with mask annotations and compute the "crowdedness" metric i.e. calculate IoU between pair of masks and then aggregate it over all pairs of masks. We also statistics of the number masks and mask area to identify images with high object counts and very small objects.
Proposing NPs. We continue leveraging the approach from phase We also expand concept coverage to long-tail, fine-grained concepts by extracting NPs from each image's alt-text where available and by mining concepts from the SA-Co ontology.
Proposing Masks. Unchanged from Phase 2.
Verification (Human+AD). We continue to both human and AI verifiers described in Phases 1 and 2 respectively, but primarily rely on AI verifiers to increase the data engine throughput.
Correction (Human). We perform manual correction wherever needed, described in Phase 1. Annotators are asked to correctly mask all occurrences of the given concept in the image.
E.6 PHASE 4: VIDEO ANNOTATION
In Phase 4, we extend the data engine to video. We use the same high-level stages as the image version, but with sideo-specific implementation details which are described next.
Media Pool. We curate pool of O(IM) hours of video from SA-V, SA-V internal, YouTube- IB and SA-FARI (wildlife cameras) datasets that covers diverse domains and a range of video quality.
Data Mining. To efficiently utilize human annotation resources, we developed aggressive data mining filters and selected only videos that presented the most challenging object tracking scenarios. The mining pipeline finds challenging single-shot video clips that are 5-30s long. Focusing on single-shot clips largely reduces annotation time and ambiguity originating from attempting to track objects across camera shots in edited videos. The mining pipeline consists of the following steps:
Scene and Motion Filters. First, we leverage scene boundary detection and VMAF motion scores from FFmpeg (FFmpeg developers) to identify non-static single-shot camera clips from the video pool. To further improve the precision of single-shot clip selection, we also use Shot Boundary Detection from the PySceneDetect (PySceneDeteet Developers) library;
Content Balancing. We use a iideo-specffic ontology to balance content distribution. We build the taxonomy by combining 1) frequent NPs annotated in the image data engine that tend to be associated with higher motion scores, and 2) taxonomy that emphasizes human activities, animals and transportation. We then generate a set text queries based on the video ontology and leverage PE Bolya et al. (2025) embeddings to retrieve video candidates for each text query. We propose text queries that elicit grouped objects and crowded scenes, for example "group of dogs" a text query based on the concept "dog";
Challenging Track Filter. We use an MILM (PLM (Cho et al. 2025) as a judge to select videos with challenging tracking scenarios. This is achieved by performing iideo-QA on of questions regarding the existence of various difficult scenarios, and selecting videos that receive more affirmative responses these questions;
Targeted Semantic Search. Lastly, we enhance the search for challenging scenarios by performing a video similarity search (using PE emheddings) using known challenging videos identified in human annotation as seeds.
Proposing NPs. We obtain candidate noun phrases for objects in the video.
Frame-level captioner and parser. We apply the Phase 3 captioner and parser on each video frame, opposed video level, maximize the diversity and volume of candidate noun phrases.
32

---

Published as a conference paper at ICLR 2026
NP Filtering. To keep only relevant phrases, we implement series of filters. First, filter out noun phrases that correspond to the overall scene, such as room, using a fine-tuned Llama 3.1 model. Similarly, filter out noun phrases that are too ambiguous to be masked, using the previously trained EV Al Verifier, which has been trained to classify such cases. Next, we remove noun phrases if they are present in given list of restricted noun phrases. This list contains 1) phrases that have been annotated as man-mas in previous annotation rounds, 2) phrases for which we already have lot of annotations, and 3) phrases that correspond "background" concepts, as our focus is on challenging moving objects. Next, we optionally filter out phrases that do not belong to certain pre-specified super- super-categories, such as "animal" or "vehicle" further focus on moving objects. We determine the super-category of given noun phrase using Llama 3.1 model.
NP Cleaning. The same cleaning applied as previous phases.
Proposing Masklets. We use the latest iteration of SAM 3 to generate instance masklets by prompting with the proposed noun phrases.
Maskler Generation. Initially, we use SAM 3 at the image level to process frames independently, and then propagate the masks using SAM 2. If masks detected in con-propagated frames not encompassed by the propagated masklets, they are used as starting points for new SAM 2 masklet propagations. Once SAM 3 video performance surpassed the decoupled system, the pipeline was updated to use SAM 3 alone.
Masklet Deduplication. After the masklets are obtained, we deduplicate them based on their IoU.
Masklet Filtering. We filter out the noun phrases that result in masklets containing the whole scene.
scenarios, namely videos that are relatively crowded and contain multiple objects of the same category. The last step of the pseudo- labeling pipeline filters out all noun phrases with fewer than N=3 objects, and videos that contain fewer than M=2 such noun phrases.
Filtering Out Easy Cases. We target challenging multi-
Verification and Correction (Human).
Verification. Human annotators check if the video is well pre-processed, e.g., no scene cuts, split screen, or explict content. Then they check if the noun phrase is groundable throughout the video, c.g., there are no comparison or size attributes that might be unclear, and no action attributes which might change across the timeline. Finally, they check that the masklet is challenging track yet possible to annotate i.e. focus on fast motion and highly occluded objects but which are still identifiable by human annotators and not too blurry to annotate properly.
Correction. Another annotator reviews the proposed masklets, removing those that are incorrect (improving precision), and using online SAM 2 in the loop to correct those that can be improved. Next, they check for any missing masklets, and use SAM 2 to add them if needed (improving recall). This annotation task results in two types of data: fully exhaustive tracking data where every object that matches the noun phrase is annotated, partially exhaustive tracking data, where some masklets might be missing because they are impossible to annotate (e.g.. inseparable background objects that match the noun phrase).
Exhaustiviry Confirmation To ensure data quality, a final round of exhaustivity checking is performed. there are any remaining missing masklets, they are added as necessary.
Sampling Frame Annotations. To increase the diversity and volume of the annotated video data, we also sample video frames and annotate them using the image data engine (Phase 3), where they are treated the same way other images. The sampling follows two separate strategies. The first one is just random sampling of a frame within video. This guarantees we cover the distribution of frames. The second strategy consists of first running the video data engine pipeline, and using the results to determine frames that contain many objects.
F
SA-Co DATASET AND METRIC DETAILS
F. SA-CO TRAINING DATA
SAM training data includes images and videos from many diverse sources, including existing datasets with box mask annotations. The training data consists of three image datasets and one
33

---

Published as a conference paper at ICLR 2026
video dataset. Fig. visualizes statistics on these subsets in comparison with existing open-source image and video detection and instance segmentation datasets as well as the distribution of top-level SA-Co ontology categories image datasets. More detailed statistics for each subset and comparison with open-source datasets are shown in Tab. 24 and Tab. 25 The original dataset sources by subset listed in Tab. [26]
SA-Co/HQ: High quality. This image dataset is generated by the data engine in Phases 1-3 with high quality annotations verified either by human annotators by Al verifiers that have accuracy on par with humans.
SA-Co/SYN:
Synthetic. We generate this synthetic dataset via the data engine in Phase 3, relying only on AI annotators. We use MetaCLIP images as the media pool and extract NPs from two sources: 1) alt-text captions associated with the images, captions generated by Llama4. We prompt SAM 3 using the extracted NPs to generate mask proposals. The image-NP-mase proposals are then verified by MV and EV AI verifiers resulting high-quality synthetic data. We also generate hard negatives proposals ($E.4) and verify them using the EV AI verifier resulting in exhaustive image-level negatives. This scalable system enabled large-scale synthetic data generation, resulting in 39M images, 1.7B image-NPs and 1.4B masks.
SA-Co/EXT: External. This dataset includes eighteen external datasets with existing instance mask or bounding boxes annotations. For datasets with only bounding boxes, generate instance masks with SAM 2. We further enrich these external datasets by mapping the original label to SA-Co ontology and propose additional negative labels using Wikidata hierarchy.
SA-Co/VIDEO: Video. The video dataset is collected via the data engine in phase 4 with high quality annotations. All the data in SA-Co/VIDER is verified by human annotators.
100
SACCAVIDEO
Visua Caname
SA- CM-COEXTT
100
Vicky 30k
Jeaty applleeee 3.2%
DDet
Butiic beaugy
BURST
Grs
Open Images
@0 TT-VV 2021*
buiding loccaiion
0.55
1000
famiiiing home 2.0%
Number of images (illbora)
Number Unique Videce (Thouaans))
(a)
(b)
(c)
Figure 12: (a) SAM 3 image training data statistics and comparison with existing open-source image detection and instance segmentation datasets. Bubble size denotes total number of NP- mask//box pairs. Bolded datasets are annotated with masks, others are bboxes only. Datasets with are exhaustively annotated, datasets with t are partially exhaustively annotated. (b) SAM 3 vidco training data statistics and comparison with existing open-source vidco instance segmentation datasets. Bubble size denotes total number of NP-masklet pairs. Datasets with * are exhaustively annotated. (c) Instance masks distribution among SA-Co ontology top-level categories in SA-Co/HQ and SA-Co/EXT. SA-Co/EXT incorporates several medical and microscopy image datasets, we categorize them under medical imaging in addition to categories in Tab. 20
Dataset
# NPs
# Images
# Image-NP
% Negatives
NP-bbox
# NP-mask
# masks per pair
30k
86.4K
30.1K
193.0K
312.2K
LVIS*
1.2K
120.0K
1.6M
72.7%
1.5M
1.5M
3.51
V3Det"
13.2K
213.2K
737.7K
1.6M
Visual Genome
$42.6K
108.1K
4.3M
6.3M
Open Images
600
1.7M
4.1M
13.3M
2.7M
2.79
Object365*
365
1.7M
10.1M
22.9M
SA-CwHQ
4.0M
5.2M
146.1M
88.5%
52.3M
52.3M
3.10
SA-CWEXT!
497.4K
9.3M
136.6M
71.8%
70.5M
70.5M
1.83
SA-Co/SYN
38.0M
39.4M
1.7B
74.0%
1.4B
1.4B
3.17
Table 24: Detailed statistics for image training datasets and comparison with existing open-source image detection and instance segmentation datasets. Datasets with exhaustively annotated, datasets with partially exhaustively annotated, % Negatives denotes percentage Image-NPs that are negatives.
34

---

Published as a conference paper at ICLR 2026
Dataset
# NPs
# Videos
# Video-NP
% Negatives
NP-makklet
masklets per pair
OVIS'
25
607
3.6K
4.04
YTVIS 2021"
40
3.9K
6.3K
1.61
BURST
482
6.9K
16.0K
2.33
LVVIS
1.2K
3.9K
13.8K
19.7K
1.40
SA-CoVVIDEO'
24.8K
52.5K
134.3K
26.7%
467.1K
4.75
Table 25: Detailed statistics for video training dataset and comparison with existing open-source video instance segmentation datasets. Datasets with are exhaustively annotated. Negatives denotes percentage of Vidco- NPs that are negatives.
F.2 SA-Co EVALUATION BENCHMARK
We create the Segment Anything with Concepts (SA-Co) Benchmark for evaluating promptable concept segmentation (PCS) in images and videos. Our benchmark contains images and videos paired with text labels, each annotated exhaustively with masks on all object instances that match the label. The dataset is federated, meaning that not all labels annotated for all images, but only handful of positive and negative labels are verified as ground-trutt truth per image. We add a large volume of challenging hard negative label annotations test models' ability to handle large, open vocabularies. In particular, the SA-Co/Gold benchmark has ~ 50x more unique phrases compared to existing exhaustively annotated mask dataset LVIS-test. The SA-Co benchmark covers diverse array of sub-domains including common objects, fine-grained concepts, food, art, robotics, etc. See Tab. 28 for detailed benchmark statistics and Tab. 27 for the list of sub-domains and their original sources.
In particular, the SA-Co/Gold benchmark has seven sub-domains as shown in Fig. 14 and Tab. 27 created to test different aspects of the concept and image distributions:
MetaCLIP MetaCLIP images (web-scraped) annotated with captioner-proposie noun phrases.
SA-1B SA-IB images (stock photos, more objects per image than MetaCLIP) annotated with captioner-proposed noun phrases.
Attributes MetaCLIP images annotated with attribute phrases. To better test attribute understanding. we also annotate phrases with swapped nouns, e.g., "pink rosi"=>"pin flamingo", and adjectives, c.g., "pink "rred rose."
Crowded Scenes SA-1B images filtered to select very crowded scenes, annotated with noun phrases proposed by MLLM.
Wiki-Comman MetaCLIP images annotated with labels corresponding to IK nodes from the SA-Co ontology judged to be common by an LLM. These concepts are meant to expand the vocabulary beyond frequent terms like "car", but still be recognizable non-experts, e.g., "Jeep", "bunk bed". "ballot box."
Fioi-Food& Drink MetaCLIP images annotated with labels corresponding to nodes from the Food& Drink branch the SA-Co ontology. Many are very tne-ggrined concepts like "kefir", "pastille".
SA-CoRIQ
SA-DoSYN SA-CWEXT
SUGGVIDEOEENT
MrrsCTLPP
JP Noctal
NOI
MrsCTJP
Chycrs al 9019
SA-V
Rost
DAVIS201
Pien Ther 2017
SA-18
Opintrops po
InageNei
Callayy
Ego4d
Role
MrFoa
2021
EDEX Le 20:41
GesbE
Remoondy 47 1004
Fashio
1orauie 200m
DR0ID
Pattomer
NNtuandis
YT. VT-Tempord.
PanNuse
3020
MotAAM:
SNOW
Pecceie Piree
FSC-147
Table 26: Media pool used to construct each SA-Co train subset. See Figs. domain and annotations.
13d 15 to 18 for examples of cach
35

---

Published as a conference paper at ICLR 2026
Wiki-Sports Equipment MetaCLIP images annotated with labels corresponding nodes from the Sports Equipment branch the SA-Co ontology, with many fine-gratd grained concepts like "kettlebell.""
All of the above sub-domains also have high number of hard negative annotations, see Tab. 28
SA-Co/Gold
SA-Co/Silver
SA-CoBronze
SA-Co/Bio SA-Co/VEval
MctaCLIP
BDD100k
BDD100k
Livecell
SA-V
SA-IB
DROID
EDEN
PanNuke
YT-Temporal-I1
Attributes (MetaCIIP)
Ego4D
Fashionpedia
McdSAM2 SmariGlasses
Crowded Scenes (SA-1B)
MyFoodRRep-223
Visdrone
SNOW
Wiki-Commoo (MetaCLIP)
GeoDE
WCS Camera Traps
Wiki-Food&Drink (MetaCLIP)
INaturalist-2017
Wiki-Sports Equipment (MetaCLIP)
National Gallery of Art
SA-V
YT-Temppoal--1B
Fathomnet
Table 27: Domains in each SA-Co evaluation benchmark subset. See Figs.
and 16
18
for examples of
each domain and annotations.
Dataset
NPs
Images
# Image-NP
% Negatives
NP-
% 0-shot NPs
LVTS test
1.2K
19.8K
COCO test2017
80
40.7K
ODinW-35 test
290
15.6K
26.1K
131.KK
SA-Co/Gold
51.8K
15.8K
168.9K
84.4%
126.9K
6.98%
SA-Cw/Silvr
54.6K
66.1K
1.8M
94.0%
219.8K
8.00%
SA-Co/Bronze
105.3K
32.5K
1.0M
84.9%
261.5K
57.25%
SA-Co/Bio
166
5.4K
35.9K
71.8%
264.6K
(a)
Dataset
NPs
# Videos
I Video-NP
Negatives
# NP-mmakkets
% 0-shot NPs
LVVIS test
1.2K
908
5.7K
BURST test
482
1.4K
3.4K
8.0K
SA-Co/VEval
5.2K
1.7K
10.3K
75.4%
11.2K
6.37%
(b)
Table 28: (a) Summaay statistics of SA-Co image evaluation benchmark by subsets and comparison with existing image instance segmentation benchmarks. (b) Summary statistics of SA-Co/EEval benchmark and comparison video instance segmentation benchmarks. % Negatives denotes percentage of Image-Ni Migg--Pssaa Video-NPs that negative. Percentages zero-shot NPs in each subset are reported. zero-shot NP defined phrase that has not been seen in the combined set of SA-Co/HO) SA-Co/EXT and SA-Co/VIDEO.
F.3 METRICS
We introduce the classifccation-aater F/ (cgF,) to evaluate the PCS task on images. The traditional AP (Average Precision) metric designed for closed-vocabulary detection tasks (c.g., COCO), breaks down when applied to apen-vocabulary detection with very large label spaces. While averaging AP over 80 classes is feasible, with tens of thousands most appear just once in the test set and the average dominated by noise. Computing full precision-recal curves for all labels also computationally infeasible and unnecessary for practical use cases. AP also does not account for the model calibration, which means that high-scoring models can be difficult to use practice. F1 at a fixed confidence threshold presents good alternative, however it is sensitive to high ratios of negative annotations: no extra credit is given for correctly predicting nothing for a negative, but the score is lowered by predicting false positives.
To remedy these issues we design new metrics for the PCS task. Given datapoints consisting of predicted and ground truth (media, phrase, masks) triplets we compute the following metries to measure localization and classification separately:
Localization. We measure this only on positive datapoints with at least one ground-truth mask. For one sample, assume we have N predicted masks mi,".. m. and M ground-truth masks
36

---

Published as a conference paper at ICLR 2026
m1..... m.y- We compute the IoU matrix iou., iou(m,, m,,) then deduce the optimal bipartite matching = argmax, E, a(t)- We fix an IoU threshold т, then for every prediction i, if it is matched and 101),,((1) т. then it is counted as TP (true positive), otherwise FP (false positive). Unmatched ground truths are counted as FN (false negative). We compute F 2TP++PP" + FNN for each datapoint, known as the local Fl score. We accumulate the counts of TP, FP and FN over all data points with at least one groundtruth mask, and compute "positive micro F1 score pmf We compute pmF1 for all T E [0.5, 0.95] with increments of 0.05, then average to obtain the final pmFi:
2TP:
2TP
pmf
totat
pmF =
Σ
pmfj
(1)
2TP total + FP total FN
total
10 1mpplinspace0..5,0.000
We also compute the average of the local Fl scores over all data points with at least one groundtruth mask, and obtain the "positive macro F1 score. We report both the positive micro and macro Fl scores in our score chart, and choose the positive micro score pmF, as the main metric for localization.
Classification. This metric between [-1, I] computes the ability of the model to predict one or several masks, if and only if the datapoint is positive. This can be seen as a binary prediction task at the image level ('is the object present or not?"), and crucially, in this metric we do not care about the quality of the predicted masks. If the datapoint is positive, and if the model has predicted any mask (with confidence greater than 0.5), then it is an IT TP (image level TP), otherwise an [I__FN. If the datapoint is negative, and if the model has predicted any mask, then it is an IL.FP, otherwise an IL_TN. We summarize this confusion matrix into a single metric, and measure potential imbalances with the Matthews Correlation Coefficient (MCC) as:
TP IT, TN- - IT. FP II. FN
IL MCC = VII.TP + IL.P) (IL.TP- FN) (IL.TN + IL.FP) (IL. FN)
As our main metric, we combine these two metrics to compute egF, ("classiiiattiongaated F1"), defined as
cgF, 100 pmF, IL MCC.
The PCS task is quite ambiguous in many cases, and to alleviate this issue our SA-Co/Gold subset contains three independent ground-truth annotations for each datapoint. To adapt our metric, use an oracle setting, where we compare the model's 's predictions to each ground-truth for each datapoint, and select the one that yields the best local Fl score.
F.4 HUMAN PERFORMANCE ON SA-CO
As described in $2 the PCS task is intrinsically ambiguous. Given an image-NP or video-NP pair, trained annotators can have different interpretations that are all valid. When the phrase is vague, annotators can even disagree on the presence of the NP. Hence when evaluating on the SA-Co benchmark, disagreement with ground truth does not necessarily mean the prediction is wrong. To this end, it is important to study the human-level performance (i.e. the agreement among skilled annotators) on the PCS task to facilitate interpreting model performance.
Human Performance on SA-Co/Gold. On the image benchmark, we provide three sets of annota- tions by different annotators. Fig. [4 shows examples of the three independent annotations per (phrase, image) pair for each domain in the benchmark. These annotations are done from scratch, meaning that the annotators create masks (using SAM 1) without seeing any SAM model interpretations. We define the "oracle" metric as follows to measure the upper bound of human performance. For each ima the best pair (out of all three pairs of annotations) is selected by maximizing the local Fl score minimizing the sum of false negatives (FN) and false positives (FP) when there is tie in local Fl scores. We then report the cgF, metric based on these selected best pairs using one annotation as ground truth and the other as prediction. To make the model performance comparable, the "oracle" model performance is calculated by comparing model predictions to all three annotations and selecting the best pairs.
Alternative to the "Oracle" protocol, human performance can also be measured on randomly selected pairs. Specifically, we adopt the following protocol to compute "Random Pair" human performance
37

---

Published as a conference paper at ICLR 2026
SA-Co/Gold
SA-Co/VEval
3-Reviewed Subset
YT- -emporrra
SmartGlasses
Evaluation Protocol
cgFi
ML.
pmff
egFi
рIKOTA
pDetA
pAssA
cgFi
pHIOTA
pDetA
pAssA
Oracle
76.2
0.99
77.0
85.1
88.4
84.7
92.6
73.5
83.1
75.4
92.0
Random Pair
55.5
0.87
63.4
75.9
82.0
73.8
91.3
53.4
70.2
54.9
90.2
Table 29: Human instance segmentation performance comparison between the Oracle and Random Pair protocols on SA-Co benchmark. The comparison is shown on subsets of the benchmark with three human annotations.
on SA-Co benchmark with three sets of annotations: 1) randomly choosing pair of annotations for each image/video-NP, then aggregate over all image/video-NPs to get the metric values, 2) repeating the process thousand times and reporting the 0.5 quantile for each metric. As shown in Tab, [29 there is noticeable gap between Oracle and Random Pair performance on both image and video benchmarks, suggesting that the PCS task is inherently ambiguous.
The image benchmark has large portion of hard negatives. These phrases go through human verification, but as costly to collect three sets of human annotations on the entire dataset due to the large volume, the negative noun phrases only have one ground-truth label. The human performance on these phrases estimated by collecting additional human annotations on subsample of phrases and comparing them with the initial annotation (i.e., the ground truth). Specifically, we collect additional human annotations on about one thousand image-NPs for each domain in SA-Co/Gold. Since the ground truths are all negatives, these phrases only contribute the IL MCC metric. We compute counts of II TN and II. FP on these samples, and then extrapolate these results to estimate the corresponding counts for the entire set of hard negatives. These estimated counts are then combined with image-level counts from the rest of the benchmark where NPs have three annotations get the final ILC.C.
Typically, our annotation protocol allows annotators to mark NPs as ambiguous if they are unsure. In this additional human review for the hard negatives, we remove the unsure option and prompt them to make a choice between positive and negative, thus reduce uncertainty and potential bias that could arise from ambiguous data.
Human Performance on SA-Co/VEval. Annotating videos is much more expensive than static images, so we collect only one set annotations per NP on the video benchmark. To guarantee annotation quality, these ground- -oound annotations undergo multiple rounds of human refinement. To measure human performance
way that is directly comparable to model evaluation in video PCS, we collect one additional from-scratch human annotation for every NP in the test set across all sub-domains. Human performance on video PCS task is then reported by comparing this additional annotation to the ground truth, using the same metrics as for model evaluation (cgF) and pHOTA).
Additionally, to study the gap between the Random Pair and the Oracle protocols, we collect two further human annotations (for total of three) the Ye-Temporal- IB and SmartGlasses test splits of the SA- EA-Co/VEval dataset. This allows us to verify that the gap observed in the image domain also exists in the video domain (see Tab. 29
F.5 ADDITIONAL DATASET EXAMPLES
Figs. 13 and 15 to show examples each visual domain in our image and video datasets. Fig. 14 illustrates the domains in our SA-Co/Gold evaluation benchmark and the three independent annotations per sample. Fig. [1]sshw show an example image from our synthetic data set SA-Co/SYN, with its positive noun phrases in the figure and negative noun phrases in the caption.
38

---

Published as a conference paper at ICLR 2026
alpina
Neoplastic
occa Asbestopluma
Connective
infiammatory
Fathornet
INaturalist [loen al. 2017
SNOW
Livecell
Pannuke Gomper
Katija
2021
2013
Edlund 2021
2019
bus
Sacrum Right hip
Left hip
van
Leopardus
sky tree fence
rose grass stone
Camera Tryjs LILA BC
Yu stal. 2020
Ma
2012
Zhu et al. 2021
Le ct al.
2002]
Figure 13: Per-domain examples in SA-Co/EXT. Shown with annotated phrases and masks overlaid.
39

---

Published as a conference paper at ICLR 2026
ORIGINAL IMAGE
ANNOTATOR
ANNOTATOR
ANNOTATOR
purse
SA-1B
house
MetaCLIP
Crowded Scenes
Attributes
Sports Equipment
swimfins
KARZ
KARZ MORE
KARZ MORE
KARZ MORE A
Jeep
MORE
Wiki Common
S
bread
Food&
Figure 14: Annotations in our SA-Co/Gold benchmark. Each row shows an example (image, noun phrase) pair from one of the seven domains, with masks from three independent annotators overlayed. No mask means the annotator decided that the phrase is not in the image. Dashed borders indicate special group masks that cover more than single instance, used when separating into instances is deemed difficult. Annotators sometimes disagree on precise mask borders,
number of instances (e-g.. house, broccoli, swimfins) or on whether phrase applies (Jeep). We use the GT annotations to measure the human performance on the task, to serve as upper bound for model performance.
40

---

Published as a conference paper at ICLR 2026
trundle
body tape maasure pencil the white band black and yellow pla...
Pumpkin... ...
MetaCLIP LIP ale textext Nu 20046
MctaCLIPP Camman WikiNodes
MetoCLIP Food and Drink
MetaCLIP Sports Equipment
SA-1B
Kirillow 2023
door
table
cheese bowl salad
twig black
bush drawing
button
road
the shadow
flower
column
plastic
black rabbit
gentleman
black pant
grey
the ground
pigeon
lamp
halloumi
rusty metal trough white rabbit with a
the man's outfit
blue tarp
the face
green
the white crevat
road
shutter
the couch
dlitt ground
the plant
yollow
the animal
tomato,
the carving
pumpkin seeds
the backpack
cherry tornato the halloumi cheese varietty ingradients a mix of green lettuce light-colored woode...
light fixture
vending machine
Crowded
Food
Gallery Art National Gallery of
Mohunty al. 2021
2024
Ravi all
2024
bush
pathway a shears
tire buildinng the headllght
taillight
cable the nut the hose
the window the grey wall the robotic arm the white paper bag
a stove
garden
clip machine the screw the package
black light
brick pathway
red brick house
dense, green shrub
Errttation etail
2022
Ya el 20220
Mitash all 2003
Khazatsky et al. 2014
Ranawaamy et al. 2023
Figure 15: Per-domain examples SA-Co/HQ. Shown with annotated phrases and masks overlaid.
41

---

Published as a conference paper at ICLR 2026
1
Rooo
blue bag
the
white chair
the wall
20
120
pink
black mortorbike
the building
baseboard
.
plastic
yellow dog toy
dag
pink
Figure 16: Example annotations from the SA-V media the SA-Co/VIDEO and SA-Co/VEval datascts.
42

---

Published as a conference paper at ICLR 2026
OC
a silver fid
the black knob
black burner
white piece of paper
blue-handled knife
person
Figure 17: Example annotations from the SmartGlasses media in the SA-Co/VEval dataset.
white-nosed coati
squirrel monkey
Figure 18: Example annotations from the AAFARI dataset.
43

---

Published as a conference paper at ICLR 2026
PEOPLE NETWORK
PEOPLE NETWORK
shoe
metal beam
sky
warehouse wall
cloud
high ceiling
high-visibility yellow vest
yellow and black bollard
overhead light
black pants
THE
PEOPLE NETWORK
THE PEOPLE
NETI
Coga
large orange and white truck man
light metal panel
high-visibility vest wheel
skylight
Figure 19: Example annotations on one image from the SA-Co/SYN dataset. There are 19 positive noun phrases for this image, which we visualize in 4 groups for better visualization quality. Al verifier assigned an exhaustivity label for each phrase: noun phrases in green box phrases pink means not exhaustive. There
the masks for that noun phrase exhaustive, 20 negative noun phrases that are verified and confirmed by Al verifier for this image: a black and white post, a blue and yellow post, a large red and white truck, an orange and black post, large orange and white boat, the green and red post, the large orange and white motorcycle, the orange and white bollard, the purple and yellow post, yellow sash, the Malaysian Air Boeing 737-800, display vehicle, Zamboni ice resurfacer, boot, the small folded paper package, long yellow vehicle, the small gift- srfpped package, small trailer, bunker gear, two-seat model.
44

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXPERIMENTS AND DETAILS
G.1 PCS WITH NPs ON IMAGES
This section describes the experiments in Tab. in detail. We compare to OWLv2 (Minderer et al., 2024), GroundingDino (Liu et al. (024a) and LLMDet (Fu et al.) 2025). Since they produce only bounding boxes, we convert them to masks using SAM 1 to evaluate segmentation. We also compare to APE (Shen et al. 2024) and DINO-X (Ren et al., 2025), two state-of-the-arr segmentation models,
and finally Gemini 2.5 Flash (Comanici et al.
2025) generalist I.I.M.
We report performance on LVIS Gupta et al. 2019), COCO (Lin al. 2014), COCO-O (Mao et al. 2023), Cityscapes (Cordts et al. 2016) (val set), ADE (Zhou et al. 2019), and Pascal Context (Mot- laghi et al. 2014), reporting their official metrics. For LVIS, we report AP-fixed (Dave et al. 2022). On our new SA-Co benchmark, we report the average across every split. We report cgF except for SA-Co/Bio where we report pmF, (this split does not have negatives, only localization is meaningful). On SA-Co/Gold we have three ground-truth annotations per datapoint, so we report the oracle metric and estimated human performance (human performance measurement detailed in SF.4). To evaluate semantic segmentation using SAM 3, we predict instance masks for each semantic category and filtering the predictions using the presence scores, mask scores, and mask areas to create the semantic mask per image. In Tab. 34 we include additional semantic segmentation evaluation on
ADE-150 (Zhou al. 2019) and PC-459 (Mottaghi al., 2014).
We employ the following Hugging Face model checkpoints: OWLv2,
for tiny" for gDinoo'T, and FSSEEE-Laooratorylllln large" for L.L OWI.v²* utilizes an ensemble of checkpoint weights after self-training and after fine-tuning the model on LVIS base, demonstrating improved open-world generalization compared to fine-tuning alone (Minderer et al.) 2024). We provide per-domain performance of instance segmentation for all baselines, SAM 3, and human on SA-Co/Gold in Tab. 30 and on SA-Co/Silver in Tab. 31 and Tab. 32 We also include per-domain performance for the Al verifier ablation study in Tab. 9d In Tab. 33 we compare with additional base-
for OWLv2*, gooogle/owll2--lrge-euueee
lines using "IDEA-Res "IDEAA-eseearch/grnnee wwwp:rchhhrooounnnnnnnnnoooooeeo for LLMDet-B.
for g gDnoo-B and "SSEE-Laboratory/Ilnde base"
For OWLv2, GroundingDino, and LLMDDet we swept over the detection threshold 0.1 intervals and determined the best threshold using the LVIS cgF, metric for the box detection task. Then, we applied this threshold to compute cgFi on the remaining datasets for the box detection and instance segmentation tasks. The detection threshold is set to 0.4 for LLMDet-L, LLMDet-B, and gDino-T; 0.3 for OWLv2*: and 0.2 for OWLv2 and gDino-B. For DINO-X, we find the detection threshold 0.5 gives the best cgF, metric. Additionally, we found that prompting multiple noun phrases at once for a given image greatly improved performance for GroundingDino and LLMDet, compared to prompting one noun phrase at time. For example, we prompted Groundingginno and LLMDet with 30 prompts for SA-Co/Gold and 20 prompts for SA-Co/Silver, SA-Co/Bronze, and SA-Co/Bio.
For Gemini 2.5 Flash, we run inference via the Gemini API. For each (image, text query) pair, we prompt Gemini 2.5 using the same prompt template that is used in Gemini Flash 2.5 image segmentation demo (Paul Voigtlaender, Valentin Gabeur and Rohan Doshi 2025) with the same generation settings. In addition, we prompt the model multiple times if there are any errors in generation, or parsing the result into set of masks and bounding boxes.
Manzii TL,MC
SA-IB
Cronded
Foedkinikk
Spart Eepal-
Allihezse
ILMCC
ILMCC peF.
1LMMCC
1,M00
ILMCC pxF.
prF..
g
15
015
16.2
16
6:11
136
01:0
008
0.16
11.5
17.
204
0.57
13.0
11.
0:53
111
0150
15.
0.11
13.0
0.6
-4.1
36.0
0.64
15.6
-111
OBL2
17.3
0.46
15.8
11.2
0.30
31.3
0.45
21.3
8.9
0.36
248
24,4
0.31
435
24A
0.52
47.8
25.9
0.54
46.2
15.4
862
35.5
LLMbell
65
0.21
27.5
45
025
19.4
53
0.23
22.8
2.4
0.8
13.
55
0.19
20.1
4.4
0.17
2533
22.
0.39
55.1
1.2
545
23.3
APE
164
0.90
36.9
13.6
043
30.1
32
0.22
160
12
0.35
283
22.7
0.51
45.0
31.8
0.56
56.5
26.
0.47
513
11.5
0.29
395
DINO.X
213
0.38
55.
15.
6.3
493
197
0:18
483
12.9
0.34
11.5
30.1
0.00
61.3
3844
0.41
08.
31.6
0.42
7400
97
8.88
535
Geii 75
134
0.38
15.1
46
0.7
718
11
0.11
331
8.
0.37
185
196
0.55
58.
151
0.7
53.5
184
0:30
63.1
64
611
51
SAM3
54.1
0.32
56.1
413
0381
36.5
31.7
0.86
62.5
61.1
09
63.7
51.4
0.70
01.3
555
0.89
75.8
54.9
0.7
12.0
42.5
0.70
90.
Horos
224
004
730
641
0.66
655
643
0.9
666
764
604
15.
15.3
8.9
$1.3
$0.4
0.99
85.1
912
005
84.
11.6
0.00
91.1
Table 30: Per-domain results of instance segmentation on SA-Co/Gold, *: partially trained on LVISS.
45

---

Published as a conference paper at ICLR 2026
RDDIOOK
DROID
FggoD
My Fooo
GooDE:
cgFi
IL.MCC
pmf:
cgFi
IL_MCC
pmfF
IL.MCC
pmf.
cgf
IL.MCC
pmnf;
cgFi
ML
pml;
cpFi
ML_MCC
pmnfi
pDinxT
0.12
2.2
0.15
23.5
0.10
23.6
0.52
0.05
10.3
10.5
0.34
OWL-2
11.5
0.32
33.0
15.0
0.16
32.6
11.1
0.36
30.7
7.6
0.23
33.0
20.0
0.44
45.4
27.6
0.50
55.2
OWLv2
0.23
31.1
1.3
0.25
29.0
0.32
41.8
LLMDesL
7.1
0.17
28.7
1.5
0.08
17.1
2.3
0.30
25.2
2.1
0.08
26.6
0.90
0.06
15.0
21.0
0.37
56.8
APE-D
0.24
24.5
20.9
8.9
0.30
29.6
7.0
0.26
26.5
57.5
Gemini 2,5
8,3
0.39
26.1
5.1
0.19
26.9
4.5
0.14
31.8
0.32
0.01
32.4
8.6
0.24
55.9
164
0.26
62.9
SAM
0.76
65.2
46.6
60.1
0.76
60.4
38.6
62.6
53.0
0.79
67.2
0.89
Table 31: Per-domain results of instance segmentation on SA-Co/Silver. partially trained on LVIS.
INaturalist-2011
National Gallery Art
SA-V
YT-Temporal-11
Fathomnet
egli
MCC
pml'i
cgF:
IL_MCC
pmfty
egTi
MCC
pml'
cgFi
MCC
pml
cgli
IL_MCC
pmft-
aDino-T
0.00
0.00
3.7
0.88
0.09
9.8
4.2
0.19
22.0
2.5
0.16
15.6
0.00
0.00
0.74
OWTx2*
5.6
0.14
40.3
6.7
0.31
21.7
11.5
0.32
35.8
9.9
0.38
26.2
0.07
0.01
9.3
OWLv2
3.3
0.10
33.2
5,8
0.25
23.1
10.8
0.32
33.6
9.9
0.35
28.3
+0.20
-0.01
20.7
LLMDei-L
32.7
0.46
71.2
1.8
0.13
13.7
5.0
0.19
26.3
3.2
0.16
20.0
0.65
0.04
16.9
APE-D
L11
0.10
11.2
3.1
0.21
14.7
7.6
0.26
28.8
5.8
0.28
20.8
0.1
0.01
7.2
Gemini 2.5
26.6
0.36
74.0
5.6
0.20
27.8
7.4
0.22
33.8
6.9
0.23
29.9
2.1
0.08
25.6
SAM
65.8
80.7
38.1
0.66
57.6
0.67
42.1
0.72
58.4
51.5
0.86
60.0
Table 32: Per-domain results of instance segmentation SA-Co/Silver continued. *: partially trained on LVIS.
G.2 VISUAL EXEMPLARS AND INTERACTIVITY
In Tab. 35 visual exemplar experiments, we report performance in settings: (1) text prompt only, (2) visual prompt only, and (3) both text and visual prompt. We note that (2) is quite ambiguous. For example, given a visual example of dog, one could want detect all dogs, only dogs of the same color or breed. As a result, SAM 3 performs worse on SA-Co/Gold in setting (2) compared to (1). Therefore, setting (3) better suited, as the text lifts most of the ambiguity, and the additional input box gives a hint for unfamiliar concepts.
G.3 FEW-SHOT FINE-TUNING
We evaluate SAM 3's object detection capabilities on real-world data through comprehensive zero- shot and few-shot experiments using two established benchmarks: Odin 13 Li et al. (2022a) and Roboflow- 1oooo..o--00V.,111 Robichhauu et al. (005)). These benchmarks encompass 13 and 100 diverse object detection datasets, respectively, capturing wide range of real-world scenarios with standardized train and test splits that enable fair comparison with cxisting methods.
Few-shot training and evaluation. For 11 few-shot experiments, train on all three official few-shot training splits and report mean performance with standard deviation on the test split. For srroflow-100VL.
utilize the official FSOD training splits provided by the benchmark and report numbers on the test split. We treat few-shot fine-tuning runs similarly to traditional training runs, but with some differences. We train for 40 epochs reduced learning rate that is one-tenth of the standard value on a batch size of 2. Since these benchmarks focus exclusively on object detection without mask annotations, we disable all mask-speciff components and losses during training.
W1inW13 results. Fig. 20a presents our few-shot performance on AdinW13. comparing SAM 3 against previous state-of-the-ar methods/Ren et al. (202a)); Wu et al. (024b); Xu et al. (202); Zhang
et al. (2022b). We report mean BoxAP averaged across all 13 datasets, with SAM 3 consistently achieving superior performance and establishing new state-of-the-ar results. Complete dataset- specific results for each O1inW13 dataset are provided in Fig. 20b
Roboftow-100V1 results. Tab. 36 summarizes our comprehensive evaluation across zero-shot, few-shot, and full fine-tuning settings on Roboflow-100V1 with results averaged across all 100 datasets. While SAM underperforms the current state-of-the-art Liu et al. (2023) in zero-shot evaluation, it surpasses leading methods Liu et al. (2023), Chen et al. (2024a) in both few-shot and full fine-tuning mee-ttning scenarios. This demonstrates SAM 3's strong visual generalization capabilities when provided with task-specific training data. We attribute the zero-shot performance gap to the use of specialized, dataset-specits prompts that may lack broad generalizability in Roboflow- 100VL. However, even minimal fine-tuning closes this gap and enables substantial performance improvements.
46

---

Published as a conference paper at ICLR 2026
Box Detection
LVIS
COCO
SA-Co
Model
cgl'
AP
AP
AP.
Gold egFi
Silver egFi
Bronze cgFi
Bio
pmFF,
Human
74.0
OWLv2 Minderer
2024
19.9
35.2
38.2
42.4
16.9
0.95
OWLv2*
Minderer
2024
30.2
45.5
46.1
23.9
24.5
11.0
12.0
0.08
gDino-B
al.
M124
15.8
25.7
52.5
45.5
6.0
4.2
12.2
0.90
@Dino-T
Liu
20248
15.1
20.5
45.7
35.3
3,4
2.5
7.6
0.35
LLMDet-
Tu cl
2025
39.3
42.0
55.6
49.8
6.8
6.7
14.0
0.17
LLMDet-B B
Fu et al.
2025
36.6
37.8
54.2
39.4
5.0
7.3
14.5
0.27
APE-D
Shen et al.
2024
59.61
58.31
17.3
7.7
14.3
0.00
DINO-X
et 2025
52.4f
56.0f
22.5
Gemini 2.5
Comannei et
2025
16.1
14.4
9,4
8.2
12.4
SAM 3
41.0
53.7
56.4
55.7
55.7
50.0
47.1
56.3
Table Additional evaluation on box detection. AP. corresponds to COCO-O accuracy, x: partially trained on LVIS, t: from original papers, 6: from DINO-X API. Gray numbers indicate usage of respective closed set training data (LVIS/COCO). See for for details on human performance.
Semantic Segmentation
ADE-847 mloU
ADE-150 mloU
PC-59
PC-459 mloU
Cityscapes mloU
Model
mloU
APE-D
9.2
30.0
58.5
21.8
44.2
SAM
13.8
39.0
60.8
18.8
65.2
Table 34: Additional evaluation on semantic segmentation.
Roboflow-100VL also categorizes its 100 datasets into seven dataset types; we report averages per
each such dataset type in Tab. 37
G.4
OBJECT COUNTING
We evaluate an internal SAM checkpoint on object counting benchmarks CountBench (Paiss et al.
2023) and PixMo-Count (Deitke et al., (2025) to compare with MLLMs (Wang al. 2024 Deitke
et al. 2025 Comanici et al. 2025) and detection expert models (Ren et al. 2025) ). See Tab. 38
for results. The metrics include Accuracy(%) and Mean Absolute Error (MAE). CountBench (Paiss
itt al. 2023) contains 540 images and their captions, with 2- 10 objects in each image. By removing images with unavailable links, we test on 487 images. PixMo-Coum (Deitke
2025) contains
540 images and their text descriptions in the form of simple noun phrases, with 2 to 10 objeets in each image. By removing images with unavailable links, we test on 529 images.
To evaluate MLLMs on CountBench, we apply the same question set as Molmo (Deitke et al. 2025 which is inherited from PaliGemma (Beyer al., 2024). When evaluating SAM 3 on CountBench. we modify the question sentence to the simple noun phrase. To evaluate MLLMs on PixMo-Count, we construct the question as "How many () are there in this image" or "Count the ( where 0 is the simple noun phrase provided by FixMo-Counl annotations.
find that presence token does not help SAM on counting tasks, so we do not use it. For group of objects, we find that SAM outputs predictions for both each individual and the group as whole, which contradicts the counting task.
SA-CwGold
pmfF T
pmFF
pmf T+l
Model
T-Rexx (Jiagg et SAM
2024
57.6
66.4
69.5
74.6
Table 35: Visual prompting SA- SA-Co/Gold. We report pmF, metric in different prompt types: T (text-only), I (imagc-only), and T+I (combined text and image).
47

---

Published as a conference paper at ICLR 2026
Model
Zero-shot
1-Shot
3-Shot
5-Shot
10-Shot
All
AP
GLIIPv-H
55.5
61.7 + 0.5
64.1 0.8
64.4 0.6
65.9 + 0.3
70.4
GLEE-Pro
53.4
59.4 + 1.5
61.7 + 0.5
64.3 1.3
65.6 0.4
69.0
MQ-GLIP-L
$4.1
62.4
64.2
65.4
66.6
71.3
Grounding DINO 1,5 Pro
58.7
62.4 1 1.1
66.3 1 1.0
66.9 1 0.2
67.9 0.3
72.4
SAM
61.0
66.1 0.8
69.1 +: 0.0
70.2 0.3
71.8
0.4
75.8
(a) Comparison of different models under few-shot settings 11.nW
Aerialmaritimedronn())
Aquarium
Rabbits
Egohands(g)
NAMushrooms
Packages
VoC
AP
0-Shol
20.9
44.5
80.9
70.0
89.6
74.6
65.2
1 1-Shot
31.5 2.6
46.5 0.4
82.3 + 1.6
73.4 + 0.3
94.8 = 0.6
82.9 + 1.6
69.0 + 0.9
3-Shot
35.9 = 1.0
4..
84.8
- 0.2
72.3
+ 1.0
97.7 = 1.0
88.4 + 1.7
70.1 + 0.3
S-Shol
37.7 + 0.9
+ 1.4
83.8
1 0.2
73.1
1 1.0
98.1 1 1.4
87.9 1 1.3
70.6 1 0.3
10-Shot
39.1 = 0.8
54.1 + 0.7 61.5
84.6
- 0.7 84.7
73.5
73.5
99.7 = 0.4
92.7 + 1.3
70.7 + 0.1
All
40.1
79.6
100.0
98.0
77.4
Raccoon
Shellfish
Vehicles
Pistols
Pothole
ThermalDP
13-Average
0-Shot
65.9
63.7
65.0
62.2
29.3
61.2
61.0
1-Shot
69.8 - 3.1
64.8 1.5 1.5
67.0 1.9
70.5 + 1.0
36.5 1.0
70.2
+ 2.6
66.1 + 0.8
3-Shol
77.9 1 2.5
64.0:2.1
68.0 1.0
71.5
1 0.7
0.9
77.9
2.6
69.1 1 0.0
5-Shot
79.9 + 1.6
64.8 = 2.3
67.3 0.8
73.1 + 0.9
1.1
79.9 + 3.0
70.2 + 0.3
10-Shot
84.3 1 0.3
66.1 0.8 0.8
73.1 6..0
4..4
8 1.9
71.8 + 0.4
86.4
58.8
72.1
78.8
58.6
89.7
75.8
(b) ODinW13 per-dataset results SAM 3.
Figure 20: Zero-shot and few-shot results on D1in
As a post-processing step. we perform Mon-Maximal Suppression (NMS) to remove duplicate detections. Instead of the usual Intersection-ovent -ver-Union (loU) criterion, Minimum (loM), where the area of overlap is divided by the area of the smaller mask, rather than by the area of the union. By doing this, can detect whole part situations: if a mask is fully covered by another, the IoM will be high, even if the covering mask is much bigger (which would lead to low loU). We set the IoM threshold 0.5 in our NMS process. Finally, we select the predictions with confidence higher than 0.5 as the final predictions and count the number of predictions as the counting result.
Interaction
G.5 VIDEO PCS DETAILS
In this section, we provide additional details for the video PCS evaluation (in $ and Tab. 5).
Benchmarks. We evaluate the video PCS capabilities of the SAM 3 model based
input text
prompt (similar to the -ocabbuayy video instance segmentation task <Wang et al. 2023 ) on both our collected video benchmark SA-Co/VEval and public benchmarks. For SA-Co/V Eval, evaluate separately each subset (SA-V, YT-Temporal-111 B, and SmartGlasses) based on their data sources, and report classification-gee gated F1 (egFi), ), phrase-based -based HOTA (pHOTA), and Track Every Thing Accuracy (TETA). The SA-Co/VEval benchmarks contain large number of noun phrases (5. (5. in SA-V and YT-Temporal- subsets and 4.9K in SmartGlasses), and provides each video
Zero-Shot
10-Shot
All
Model
Grounding Dino LW-DETRm
15.7
33.7
59,8
SAM3
15.2
36.5
61.6
Table 36: Comparison on Roboflow 100-VL.
48

---

Published as a conference paper at ICLR 2026
Aerial
Document
Flora-Fauna
Industrial
Medical
Other
Sports
Average
AP
0-Shot 10-Shot
24.0
12.9
23.9
9.0
2.6
17.0
15.6
15.2
35.4
35.8
39.6
40.4
25.7
35.0
40.3
36.5
All
56.9
64.2
60.1
68.4
52.5
63.8
61.5
61.6
Table 37: SAM Roboflow 100- VL results by dataset type.
CountBench
PixMo-Count
Model
Presence
IoM
MAE
Acc (96)
MAE 1
Acc (%)
SAM 3
0.34
84.8
0.34
76,9
SAM 3
0.50
83.7
0.41
75.8
SAM 3
0.11
94.0
0.21
86.3
SAM 3
0.38
89.3
0.29
85.2
Table 38: Ablation on counting results on an internal SAM checkpoint.
with list of noun phrases as text prompts. During evaluation, for each evaluation video, we prompt SAM 3 with the list of noun phrases provided for that video, as shown in Tab. 39 (a, b, c).
For public benchmarks, we evaluate on LVVIS
(Wang et al. (023). BURST (Athar et al.. 2023). BDD Yu et al. 2020), GMOT40 (aai et al. 2025 and report the official metrics each dataset (for
YTVIS (Ke et al. 2022). OVIS (Qi et al.
2022).
2021 and DeepSeaMOT Barnard et al.
DeePSeaMOT, we report the average performance over its 4 subsets). These public benchmarks are often based on set of categories, with relatively large vocabulary size in LVVIS and BURST (1196 categories in LVVIS and 482 categories in BURST) and much smaller numbers of categories in other datasets. We use the category name the text prompt, and prompt SAM 3 with all category names in the dataset on every evaluation video, as shown in Tab. (39 (d).
Video PCS Metrics. Similar to its definition in the image domain in gated Fl (cgFi) 1) metric on videos as the multiplication between the vidco-level Matthews correlation coefficient (VL_MCC) on whether the noun phrase exists in the video and the localization positive macro F1 (pmF,) on positive noun phrases. To decide whether predicted masklet matches ground-truth masklet, we measure their volume intersection-owe oonrr-ctioonover-uniion (IoU), defined by their total intersection volume divided by their total union volume over the video. When computing pmF we averaged the results over multiple volume loU thresholds from 0.5 to 0.95 with increments of 0.05, similar to how it is computed on images.
define the classification-
We also evaluate the phrase-based based HOTA (pHOTA) metric, where we compute the Higher Order Tracking Accuracy (HOTA) metric (Luiten et al. 202T) over all video-NP pairs along with their breakdown into phrase-based detection accuracy (pDetA) and phrase-based association accuracy (pAssA). As the HOTA metric originally designed for category- based variant pHOTA for open- vocabulary prompts, we remap each video-NP pair in the evaluation benchmark new unique video ID and then set all ground-truth annotations and predictions have the same category ID (i.e., the total number of video IDs after remapping equals the total number of video-NP pairs in the evaluation benchmark). That is, each video-NP pair in the benchmark is treated as an isolated sample for prediction and evaluation, and the results are aggregated over all video-NP pairs in a class-agnostic manner. More specifically, we save the remapped ground-truth annotations and the predictions as JSON files under the YTVIS format, and use the TrackEval package (Jonathon Luiten 2020) obtain the mask HOTA statistics on this remapped dataset (using the YTVIS dataset wrapper in TrackEval along with its default parameters), and report their results as PHOTA, pDetA, and pAssA. Similarly, we also evaluate the Track Every Thing Accuracy (TETA)
evaluation, to get its phrase-
metric (Li ct al. 2022c over the masklet predictions these datasets.
We compare the SAM model with several baselines, including GLEE Wu et al. 2024a) (a previous work on open-vocabulas image/video segmentation), "LLMDet detector + SAM Tracker", by replacing the Detector component SAM with recent open- vocabuuayy detector LLMDet (Fu et al. 2025), and "SAM Detector T-by-D tracker", by replacing the Tracker in SAM with an association module similar as in tracking-by-ce be-detection approaches (Wojke et al.
Baselines.
2017 Zhhan et al. 2022c).
49

---

Published as a conference paper at ICLR 2026
2024a). we follow its official implementation. Since GLEE supports taking as inputs multiple text prompt simultaneously. we evaluate it in two ways: a) prompting it with all the noun phrases from evaluation video once, denoted as "GLEE (prompted w/ all NPs at once)" in Tab. (39 and b) looping over each noun phrase in the evaluation video and prompting GLEE with one noun phrase at a time, denoted as "GLEE (prompted w/ one NP time)". We find that for open-vocabulary segmentation on videos, it is usually better to prompt GLEE with one noun phrase time instead of prompting it with all noun phrases at once.
For GLEE (Wu et al.
For "LLMDet as Detector + SAM 3 Tracker", we replace the detection outputs from the SAM 3 detector with LLMDet (Fu et al. 2025) bounding box outputs, and obtain the mask output by prompting it with the SAM 3 component. Then we apply the SAM 3 Tracker similar to how it applied over the SAM Detector output. We also note that GLEE and L.MDet have not been trained on the noun phrases in the SA-Co dataset, so their results should be seen zero-shot on the SA-Co/VEval benchmark.
For "SAM 3 Detector + T-by-D as tracker", we replace the SAM 3 Tracker with a detection-to-maskle association module as commonly used the backing-by-detectie
paradigm, e.g. Wookee et al. (2017); association module tries to match the masklets already tracked in previous frames with detected objects in the current frame, based on a dot product between the visual features of each detected object and the visual features of the past 16 frames of masklet. If a high-confidence detection isn't matched to any existing masklet, we add it as a new object and start new masklet for it. The association module is trained on the SA-Co dataset.
Zhang et al. (002cc). The detection-to-mase
Results. As shown in Tab. 39 SAM 3 largely outperforms these baselines across the benchmarks. On SA-Co/VEval with very large number of noun phrases, SAM 3 excels in both frame-level detection (pDetA) and cross-frame association (pAssA). Comparisons with "LLMDet as Detector + SAM Tracker" and "SAM 3 Detector + T-by-D as tracker" demonstrate that both the Detector module and the Tracker module in SAM 3 play a critical role in the final video performance. In public benchmarks, SAM also achieves a strong performance, including new of-th--o-----tttttt results on LVVIS and OVIS. We also note that GLEE and LLMDDet have not been trained on the SA-Co dataset, so their results should be seen as zero-shot on SA-Co/VEvva In addition, the SmartGlasses subset in SA-Co/VEval contains many egocentric videos, which might be out of the training distribution for GLEE and LLMDet.
Strategies on Temporal Disambiguation. As described in $D.3 SAM 3 adopts several strategies to address ambiguities in videos. In Tab.B3 39 we also report two other settings where we turn off all these temporal disambiguation strategies ("SAM 3 w/o any temporal disambiguation"). The results show that the disambiguation strategies boost the video PCS performance (especially under the pHOTA metric). We also find that the disambiguation strategies notably improve the qualitative outputs on videos.
G.6 PVS DETAILS
We evaluate SAM on range of Promptable Video Segmentation (PVS) tasks as in Ravi al. (2024).
Video Object Segmentation (VOS). The VOS task requires tracking an object throughout video given an input segmentation mask. As shown in Tab. 6, we compare SAM 3 with recent state-of- the-art models on the VOS task, including SAMURAI (Yang et al.) 2024). SAM2Long (Ding et al., 2024), and SeC (Zhang et al. 2025). SAM 3 brings gains on all datasets, including the challenging
MOSEv2 benchmark (Ding et al. 2025) and datasets with long videos such as LVOSv2 (Hong et al.. 2024).
Interactive Image Segmentation. We evaluate SAM on the 37-dataset benchmark introduced in Ravi et al. (2024) for the interactive image segmentation task. As shown in Tab. ? SAM 3 outperforms SAM and SAM 2 on average mlou, producing more accurate segmentation masks when prompted with or 5 clicks.
Interactive Video Segmentation. We follow interactive offline and online evaluation protocol Ravi et al. (2024) and compare SAM 3 with baseline methods, including SAM 2, SAM XMem++, and SAM + Cutie. The interactive offline evaluation involves multiple passes over the entire video, while the interactive online evaluation involves only one pass over the entire video. We use the same
50

---

Published as a conference paper at ICLR 2026
SA-CaNEEsa val NES)
SA-CWVESal
Model
VI,MCU
PASIA
1ETTA
VI,MKC
Harnnn
53.9
0,86
62.4
11.7
56.9
90.8
68.9
53.1
0.87
60.5
70.5
55.5
90.2
09.1
GLEB (poopeed wi NPS once)
0.1
0.17
0.4
7.7
2.6
28.5
6.5
0.1
0.22
0.6
8.7
288
30.9
7.1
GLEE (prompee onz NP at time)
0.2
0.05
2.5
12.4
3.9
41.7
15.7
0.1
0.05
2.2
118
3.4
43.1
14.9
LLMDei detectir SAM Tracker
1.6
0.11
14.1
30.5
13.0
72.1
29.3
2.3
0.15
14.7
30.1
12.0
76.0
28.5
SAM 3 Deseiur + Bay D trakeer
23.7
0.65
34.7
37.6
44.5
73.3
53.4
25.7
0.72
35.5
55.7
40.2
78.3
53.2
lempond dsantiiuaaion
0.36
$7.3
30.3
0.69
(a) Results SA-Co/VEval SA-V val and test
SA-CoVEnoll YT- YT-Teeppeeu--B (1.4K NPO
SA-CorNEval YT- T.Teepporll11 (1.5)
Model
VI.MDCC
pnf;
pMOTA
pllz
pAssA.
TPFA
VI,30C
penf:
PiOOT
pDls
TETA
Tlamnn
0.96
K3.8
0.7
FE (pompeat
unce)
18
0.36
6.7
6.8
16.5
GLEE goompta ond time)
2.3
0.23
0.9
8.5
LLMUel detector SAM Tracker
10.5
0:30
39.6
78.3
32.0
0.10
0:33
24.1
37.8
18.7
72.8
35.4
Devccor Tily
temionaa dsambigaation
0.84
70.1
09.8
SAM 3
50.2
0.58
57.2
70.5
00.5
82.7
70.1
50.8
0.89
57.2
69.9
60.2
K1.7
70.5
(b) Results on SA-Co/VEval YT-Temporal- 1B val and test
SA-Ca'Evvl SmartGlases val 2.2K NPS)
SA-CWY Eval SmartGlasees test (2.2K NPS)
puf,
JÉIOTA
pDesk
pAsa
TETA
ceft
pnE:
pDesk
pAsiA
Harnn
091
58.5
0.92
(propped SPS
0.15
1.6
GLEE (prmpped
0.14
13.3
0.14
1.8
0.5
SAM Deaaure T-y-ID tndder
23.2
0.34
52.4
56.2
68-4
60.6
79.7
0,88
35.1
60.0
30.6
71.7
61.2
SAM win any temporal dismbiggaation
30.9
0.1
63.5
18.1
41.2
7307
65.1
16.6
0.75
05.5
61.6
47.0
$1.2
66.0
SAM
33.5
0.76
602
79.3
65.0
36.6
0.78
46.6
50.0
65.9
(c) Results on SA Co/CallEvall SmartGlasses and test
LYVIS NP
BDD1OOX
DecSSesMOT 230NP)
CST NPS
NPO
iuAP
HOTA
HOTA
TEDA
GLES (gouppted all MP oxee)
20.8
36.0
18.4
25.4
-5.4
24.8
63.3
36.7
150
36.8
58
ppoepeed one seazi
13.9
39
2012
16.3
34.9
23.9
SAM 3 Detckar Taly Das tuakerr
359
32.9
582
39.7
56.7
35.8
56.5
55.1
314
09.6
35.5
(d) Results public benchmarks
Table 39: More details on video PCS from text prompt copen-vocabulary video instance segmentation) on Co/VEval and public benchmarks (# NPs of each benchmark in parentheses). SAM 3 excels in hoth frame-level detection (pDetA) and cross cross-frame frame association (pAssA) and largely outperforms the baselines, especially on benchmarks with very large number of noun phrases. t: GLEE and LLMDet have not been trained the SA-Co dataset, so their results should be seen as zero-shot on SA-Co/VEval.
zero-shot datasets and 3 clicks per interacted frame as in Ravi et al. (024) (see Sec. F.1.2 2 of Ravi (ct al. (2024) for details). The results are Fig. 2T where SAM 3 achieves better overall performance in both interactive offline and online evaluation.
51

---

Published as a conference paper at ICLR 2026
overage 1&E over Calteets
20ningg MRF over dattsonts
80
78
SAM
SAM2
XMmnt
Cztee
SAM Cute
zumber of anmotated frames
zumber of anmotated frames
(a) offline J&F averaged across datasets
(b) online J J&F averaged across 9 datasets
EndoVis
Virtual KITTI
Method
LVVSv2
LVAVIS
VIPSeg
(average)
XMem++
72.1
63.8
SAM + Cutie
71.8
87.6
82.1
87.1
59.4
75.2
84.4
70.3
54,3
74.7
SAM3
(c) J&J on dataset averaged over interacted frames under interactive offline evaluation
Virtual KITTI 2
Method
2018
ESD
1.V0vv2
IV.VIS
PUMaVOS
UVO
VIPSeg
VOST
(avenge)
SAM XMem++
71.4
87.8
72.9
85.2
63.7
74.7
82.5
63.9
52.7
72.8
70.4
SAM 2
77.5
88.9
87.8
88.7
72.7
78.6
85.5
74.0
65.0
79.8
SAM
79.2
89.6
89.7
87.9
73.1
77.5
84.2
75.6
67.9
80.5
(d) J&FF on each dataset averaged over 1--8 interacted frames under interactive online evaluation
Figure 21: Interactive video segmentation of SAM 3 baselines under offline and online evaluation, following the setup in Ravi al. (2024) with the same zero-shot datasets and 3 clicks per interacted frame. The J & under different numbers interactive frames are shown in (a) and (b), while the T &FF on each datasct shown in (c) and (d).
52

---

Published as a conference paper at ICLR 2026
G.6.1
ADDITIONAL MODEL OUTPUTS FOR DIFFERENT TASKS
2
Princs
2
P
EnidE
FC
cab ofricio
Figure 22: Example predictions on the SA-Co/Gold dataset. As these are model outputs, occasional errors may be present.
53

---

Published as a conference paper at ICLR 2026
hanging pot
white bag
sheet white paper
B
B
Figure 23: Example video concept segmentation predictions on the SA-Co/VEval SA-V test set (top two rows) and the SA-Co/VEval SmartGllasses test set (bottom two rows).
pepper
3 7
4
5
2
2
8 6
8
3
7
6
5
4
3
2
1
Figure 24: Example predictions on the countbench dataset.
54

---

Published as a conference paper at ICLR 2026
H SAM 3 AGENT
H.1 AGENT DESIGN
In this section, introduce SAM Agent, visual agentic system that turns natural- -anguuge segmentation requests into precise masks through dynamically querying a multimodal LLM (MLLM) and SAM 3. Given input image and request, MLLM acts as planner/controller: analyzes the image, devises a step-by-step plan, invokes SAM 3 to generate masks, inspects the results, and finalizes candidate masks. After each action, the agent receives visual and textual feedback describing the updated environment state, enabling it to revise the plan and decide the next action. This perception-action loop continues until the agent is confident has satisfied the goal (or determines that no valid mask exists), at which point it returns final set of masks. The resulting pipeline handles queries far more complex than simple noun phrases which require understanding relationships between scene elements and visual common sense.
Each action consists of calling one of several "tools". We define the following four basic tools for the MLLM to call: segment permenn_pprase,
phrase, examine_eate mack,, pareectmmsssaccee aaddddddree return, and report too maak. Among these four tools, trigger a return function and end the current task. The other two functions: segment.phran and examine_each. mask, are intermediate tools, which will either call the SAM model on a noun phrase trigger iterative process for the MLLM to examine each generated mask.
and return and report_ no mask are return tools, which will
Tool #1: Segment Phrase
(1) Definition: Use the Segment Anything model to ground all instances of simple noun phrase by generating segmentation mask(s) that cover those instances the input image. At the same time, all previously generated mask(s) will be deleted and cannot referred in future messages.
(2) Use cases: Given simple, dircct, and singular quires additional understandinghhesonnng segment_ _phrasse_pppheeeeee will try locate all object instance(s) on the raw input image match the simple noun phrase you provided. The tool will also render all the generated segmentation mask(s) onto the image for you to examine and decide the next step.
phrase (not referring expression that
("type": "object". "properties": ("text prompt": ("type": "string", "description"; "A short and simple noun phrase, e.g., rope, bird beak, speed monitor, brown handbag, person torso"]]. "required": ["texi_prompt"]]
Parameters:
Return type: new image with differently colored segmentation mask(s) rendered on a text message indicating the number of mask(s) generated by the Segment Anything 3 model for this "text.promp"" only.
Tool #2: Examine Each Mask
(1) Definition:
Use this tool when the segment.phran pphrase tool generates multiple small or overlapping mask(s), making difficult distinguish the correct mask(s). examine aaamine_echh... sww.eeaahaaaaa _mmask allows you render and examine each mask independently overlapping mask(s).
small mask(s) clearly and avoid confusing
(2) Use cases:
Sometimes there multiple small mask(s) overlapping mask(s) rendered image, making difficult to distinguish each mask from others. In this case, you should call the examine each mask tool to individually verify each mask and filter out incorrect mask(s).
(3) Parameters: None
Return type: A new image with colored segmentation mask(s) accepted by exam- ine each mask tool, and a text message indicating how many masks were accepted.
55

---

Published as a conference paper at ICLR 2026
Tool #3: Select Masks And Return
(1) Definition: Call this tool to select a subset of or all of the mask(s) rendered on the most recent image your final output. When calling select_maske mas.saasaaaa generated by previous rounds other than the most recent round in your "final answarn seaverrsssk"..... You can only use mask(s) from the most recent image in your message history.
you cannot select any mask(s)
(2) Use cases: Given image with one more segmentation mask(s) already rendered on it, select masks and return returns the set of mask(s) you select as the final output.
(3) Parameters: ("type": "object", "properties": ("final answer masks": ("type": "array", "descrip- tion": "An array of integers representing the selected mask(s) you want to choose your final output, e.g. [1, 4, 5)" "required": ("final ['tinal wnnswwrmasss"]]
(4) Return type: None (End of Conversation)
Tool #4: Report No Mask
Definition: Call this tool when you absolutely sure that there object(s) in the image that match or answer initial user input query.
Use cases: Reporting that the given image does not contain any target object(s) that match answer the initial user input query.
Parameters: None
Return type: None (End of Conversation)
After each intermediate tool call has been executed, the system will provide the MLLM with the following two pieces of information:
The user input image with all generated and currently available segmentation masks rendered on in a Set-of-Marks (Yang et al. 2023) manner. The masks are randomly colored and numbered from I to N in decreasing order of SAM confidence scores received at the time of mask generation. The set of currently available masks, combined with the original user input image, defines the environment state of the SAM 3 Agent at the current time step.
An automatically generated text message stating all changes from the previous environment state (c.g. how many masks have been generated by the segment phrase tool, or how many masks were removed by the examine each mask tool).
After analyzing the updated image with currently available masks rendered on it (current environment state) in the context of the initial user query (task goal), the MILMM must update its tool-calling plan and generate the next tool call (current action). We allow the MLLM to call each intermediate tool as many times as it needs, before arriving at a final set of segmentation masks on the input image (terminal state) that it is satisfied with.
Empirically, we observe that for especially challenging queries, SAM Agent may produce as many as 60 steps of trial and error before being satisfied with grounding outcome and calling a return tool. This results in an extremely long environment-state context history with each step containing new image. pushing both the context limit and multi-image reasoning capability of even current state-of-the-arn MLLMS.
To resolve this issue, we propose an aggressive context engineering mechanism that prunes all intermediate ande--and-rrror states between the initial user text query and the most recent agent call to the segment segment_phrase tool. We also discard previously generated masks after each tool call to the segment_phrase tool, which avoids cluttering the rendered Set-of-Marks image with redundant masks. To avoid losing important failure experience from pruned steps, we provide a continuously updated list of all previously used (and discarded) SAM 3 noun phrase prompts for the model to note.
H.2 QUALITATIVE ANALYSIS
In this section, we provide success 25 and failure 26 examples of SAM 3 Agent on the ReasonSeg (Lai et al. 2024) and RefCOCOg (Kazemzadeh al. 2014) datasets, they are currently the most
56

---

Published as a conference paper at ICLR 2026
kingdom, makes scec
features
Placee where nector from
photographing nearlty
computer disp lyyngg pages
the traneportation weticle
reflec
the person in the piccure the ownerr of blus car and warts toee
Twant to take a tipp around the works, but nead some tronsportation to cross What type transporatiin cluree reguiement?
part thispicture Furny placs?
usiin dog?
IT Pieeellleee wanted to Rcentify the veticle wiil wwr.charrrooo of this picture world contain usedul informationn?
In Nitto Scal aal.scctttt bulkdings often have entrances that consler of operdinge with curvedior boliited top. What feature in the ditture/eeemmiiee such entrance?
Figure 25: Successful examples of SAM Agent (Qwen2.5-VL 72B) the ReasonSeg Lai al. 2024 dataset for Reasoning Segmentation and ReiCOCOg (Kazemzadeh al. 2014) dataset for Referring Expression Segmentation. For each example, see the original input image (left), textual user query (bottom), and final segmentation output (if applicable) from SAM Agent (right).
57

---

Published as a conference paper at ICLR 2026
1
(a) Original Input Image
(b) SAM3 Intermediate Masks
(c) Final Selected Mask
Figure 26: Error example of SAM Agent (Qwen2.5-VI. 72B) including the original input image and SAM intermediate output masks. The textual user query is "A black object that protects you from the rain, being held by person in jeans". The agent was able to reason and find suitable grounding target for calling SAM ("black umbrella"). However, failed to select the correct mask during final mask selection due visual reasoning error by the MILMM
challenging and widely used reasoning segmentation and referring expression segmentation datasets. We also provide complete reasoning trace example of SAM 3 Agent, demonstrating how SAM Agent solves complex reasoning segmentation queries by leveraging SAM 3 for precise grounding and MILM for visual reasoning.
Empirically, we observe that SAM 3 Agent is able to handle free-form textual queries of varying spatial and logical reasoning complexity. It is able reject queries that not point to any object in the given image and to generate multiple output masks for queries that match multiple objects in the given image. The MILM backbone allows SAM Agent to robustly handle queries varying length and format. It also enables SAM 3 Agent to accurately segment fine-grained concepts that SAM 3 alone struggle with, such as specific aircraft types and fine-ge fine-graine grained food categories.
SAM3 Agent Complete Reasoning Trace Example
"role": "ayatem", "content"; ; (Syatem prompt Including tool definitions, agenz instructione, and response format.
"role": "uses", "content":
"Lye"" "inage",
"mage": gigere [Nref
"sype": "text",
"toxt"; "the stronger Mario*
"role": "assistant",
"content":
"Lype": "Lext",
"text": *ethinks There is only one image in the context. The inage shows banner with pixel art characters resembling Mario. user wants ground strenger Mario, !" which appears be the charecter the right side of the equation depicted en the banner. i will use the begment__parase function with the text prompt ("stronger Mariei" to attempt ground the target objact. An\nBafore proceeding, remind myself that each call to "segment_phraee will delete all
58

---

Published as a conference paper at ICLR 2026
previcusly generated masks, sa need to ensure plan is effective in one start by attempting to ground \"stronger Mario. wwwoo./wwwwnn/hhnnniioooooo (A"name)" V"aegmett_ohrasel,, Vparameters""; prrexxt Vatronger Mario)")] </tcol>in
\n"
"role": "user", "content":
*type": "text",
*text": *SAM3 di not generate any masks for the prompt ronger Marto', please peomentt_phease again with different, perhaps general and simple noun parase prompt."
"role": "asaistant", "content":
"Lype"; "Lext",
the centext. The pravicus attempt to ground V"stronger Maricl did not yield any masks. need to try gereral
"text"; *cthink> There is still only
simplar and Given the pixel azt style, I'll try grounding \"Mario\" to see if it captures the equation.
phrase that atill
the
of target.
remind myself segment segment_pprase pphrase will delete all previcusly generated maska, so need ensure my plan is effective in one go. will new attempt to ground \MMario, \ ("Manns/thiakk tool> (\"name>:; ""segment_phrasev. \"parameters""; ( (>"tex"_prompe>"; \"Mario!"]] </ tool>\n\"
right
"role": "uses", "content":
"type"; "Lext", "text": *SAM3 did not generate any masks for the prompt "Kario', please call sement_piraee again different, perhaps general simple noun phrase prompt."
"role": "assistant", "content":
"type": "text",
The previcus attempts to ground Vattoonge Mario'" and \MMaroo\* did not yield any maaks. style
"text"; *ethink> There is still only image
image, will tzy more general and simple noun phrase related to the character's appearance. will attempt to ground ("pixal charscteri" see it captures the target. remind mysalf that each call to "segment_parase will delete all previously generated maaks, so need to ensure ny plan is effactiva in one go. I will now attempt to ground \"pixal charactar. (M/inn//thiiiinntttto Lext Lexi_prompt V"iieel character\":) </Lool>\ala"
the pixel
(""name\": ""segment_paraee"", Vparameters (X"
"role": "user", "content";
"type": "text",
"toxt*: "The original user query was: 'the strongar Kario', and the SAM3 prompt you used was: 'pixel character', SAM3 generated maaks for the prompt xpixel character', which rendered in the latest image, please analyze this Image and decide your next action."
"type": "inage",
*smage": Figure [ (
59

---

Published as a conference paper at ICLR 2026
Gros four Business With Apps
Gros Tu viness With Apps
Grow Your fariness/ Apps
(a) Original Input Image
(b) SAM3 Intermediate Masks
(c) Final Output Mask
Figure 27: Images from the Complete Reasoning Trace Example of SAM 3 Agent (Qwen2.5-VL 72B), including images (a) and (b) from the agent context image (c) containing final agent output mask. The textual user query is "the stronger Mario". The SAM 3 Intermediate Masks are generated by SAM 3 using the text prompt "pixel character".
"role": "assistant", "content":
"type"; "text",
images the context the input Image and the ourrent output Image with tour maaka rendered. Mask corresponds to Vetronger Mario)" charectes, which is target of 3, and 4 are ast relevant to the query. in\nSince 1 have identified the correct mask, as
"text": *ethink> There
query. Masks
anaver, anawer.\alne/tee. www.:///haaioonnstoollll (A"name)** \" select_ aaasss_anoreturn \"parameters("; aerinal_anawer "anawer_maskk". [2]3] </tool>*
precsed
"role": "user",
"content":
"type": "text",
"text": "rinal Answer:"
"type": "inage", "mage"; gigure /\ref/
H.3 FULL QUANTITATIVE RESULTS
We report the full quantitative experiment results of SAM 3 Agent on ReasonSeg in Tab. 40 OmniLabel in Tab. 41 and RefCOCO Seg in Tab. 42 SAM 3 Agent achieves the best results both ReasonSeg and OmniLabel in a zero-shot manner, without training on any referring expression segmentation reasoning segmentation data. SAM 3 Agent also surpasses previous zero-shot tftth--rt- results on RefCOCO+ and RefCOCOg, and is close to best methods that leverage the training datasets. We hypothesize that on RefCOCO, where all masks come from the MSCOCO dataset and each query points to exactly one ground-truth object mask, training- based methods learn the specific dataset annotation biases. We show examples of such annotation biases in the RefCOCO-Sed datasets in Fig. 28] SAM 3 Agent, being zero-shot method, is unable to exploit these (generally undesirable) biases.
60

---

Published as a conference paper at ICLR 2026
Model
Training
Val Set
Test Set
Test (Short)
Test (Long)
Version
RES
ReasonSeg
gloU
gloU
gloU
cloU
gloU
cloU
24.3
Grounded SAM
26.0
14.5
21.3
16.4
17.8
10.8
22.4
18.6
OVSeg
GLaMM
Vicuna 7B
X
47.4
47.2
SAMHMLLM
Owen- L. 8B
SAM4MLLM
58.4
60.4
Seg-7ero
Owen2.5-VI. V. 3B
58.2
53.1
56.1
48.6
Seg Zero
Qwen2.5. VL. 7B
62.6
62.0
57.5
$2.0
X-SAM
Phi3 3.8B
56.6
32.9
57.8
41.0
47.7
48.1
$6.0
40.8
HyperSog
59.2
56.7
Kang et al.
LLaVA 1.5 7B
52.4
48.7
48.0
49.1
13B
LISA
LLaVA 7B
X
44.4
46.0
36.8
34.1
37.6
34.4
36.6
34.7
LLaVA
LISA
LLaVA 13B
X
48.9
46.9
44.8
45.8
39.9
43.3
46.4
46.5
LISA
LLaVA 13B
56.2
62.9
51.7
44.3
42.0
54.0
54.3
LISA
LlanaZ 13B
60.9
67.8
51.5
51.3
43.9
45.8
54.0
53.8
LISA
LLaVAL5 7B
53.6
52.3
48.8
47.1
48.3
48.8
49.2
48.9
5 7B
61.3
55.6
48.3
46.3
57.9
59.7
LISA
1. 13B
57.7
60.3
53.8
50.8
50.8
50.0
54.7
50.9
RSVP
LLaVA16 7B
X
59.2
56.7
56.9
50.7
47.9
42.0
58.4
$3.0
RSVP
Owen2-VL
48.5
44.3
53.0
RSVP
Geminil S-Flash
56.9
49.2
$7.1
59.2
47.3
40.2
60.2
65.6
RSVP
4o
64.7
63.1
60.3
55.4
50.4
61.9
62.5
Gemini Seg
Gemini2.5 Flash
28.3
13.3
30.6
16.5
8.0
35.0
9.5
SAM Agent
Qwen2.5.Ve 7B
X
62.2
49.1
63.0
53.5
59.4
43.5
64.1
56.2
SAM Agent
Qwen2.5 VL. 72B
74.6
65.1
70.8
70.3
55.7
71.0
66.3
SAM Agent
Llama4 Maverick
X
68.5
61.5
67.1
60.9
66.8
59.4
67.2
61.3
SAM Agent
Qwen3-VL 8B Thinking
68.5
57.4
70.2
67.3
70.5
62.5
70,2
68.6
Agent
Qwen3-VI. Thinking
73.7
75.1
73.3
SAM 3 Agent
Gemini2.5 Pro
X
77.0
66.0
74.0
72.1
75.8
67.5
73.4
73,2
Table 40: SAM 3 Agent experiments on the ReasonSeg dataset (Lai et al. (0244) for Reasoning Segmentation. Training-REs RES indicates whether the model has been fine-tuned on the Referring Expression Segmentation task. Training-ReasonSSe
indicates whether the model has been fine-tuned the ReasonSeg training set. The overall best performances
shown in bold and the best zero-shot performances for models not trained
ReasonSeg training set are underlined. Notable baselines include: SEEM (Zou al. 2023). Grounded
SAM (Ren et al. 2024b). OVSeg (Liang et al. 2023), GLaMM (Rasheed et al. 2024). SA M4MLLM (Chen
et al. 2024b), Seg-Zero Zero (Liu et al. (2255) X SAMM Wang et al. (225aa). HyperSeg Wei et al. 2024). (Kang
al. 2025). LISA (Lai 2024). RSVP (Lu et al. 2025) and Gemini-scg (Paul Voigtlaender. Valentin Gabeur
Rohan Doshi 2025)
FRANCE OF
PRANCE COO
orange piece left of previous hint
1
carrots
front bowl
Figure Examples annotation and ground truth errrr from the RefCOCO S-eg datasets Kazemzadeh) ltt al. 2014 Mao et al. 2016). For each example, see the original dataset ground truth annotation (left image), the textual user query (hottom text), and the SAM Agent (Qwen2.5-VV 72B) final segmentation output (right image). Our emror analysis reveals such annotation hias and ground truth errors account for the majority of low- predictions by SAM Agent the RefCOCO-Seg datasets.
61

---

Published as a conference paper at ICLR 2026
Model
2023 Val Set (AP 1oU=00550:0.55)
Name
Version
AP
AP-ealce
AP-deser
AP-descr-pos
AP-deaeeS
AP-Jescr-M
AP-deser-L
FIBER
FIBER-B
25.7
30.3
22.3
34.8
38.6
19.5
12.4
GLIP
GLIP-T
19.3
23.6
16.4
25.8
29.4
14.8
8.2
Zhao
GLIPT
22.2
27.2
18.8
Zhao al.
FIBER-B
28.1
32.1
25.1
36.5
DesCo
GLIP-T
23.8
27.4
21.0
30.4
GLEE
Lite
20.3
37.5
14.0
19.1
23.0
12.7
10.0
GLEE
Lite-Scale
22.7
35.5
16.7
22.3
33.7
14.3
10.2
GLEE
Plus
25.4
46.7
17.5
23.9
28.4
16.3
12.5
Real
Swin-R
36.5
$2.1
54.4
33.2
25.5
LED
Qwen2
27.9
33.9
23.7
36.2
LED
IntemL.M2-20
27.9
33.4
23,9
36.1
ROD-MLLM
Vicuna 7B
25.3
30.9
31.8
24.5
21.0
WSCI.
GLIP-T
24.3
23.9
24.7
34.
39,3
21.6
16.4
WSCI.
PIBER-H
30.5
31.6
29.5
40.3
43.7
26.3
21.3
WSCL
FIsco-FIBER
32.0
33.1
30.9
40.4
45.2
27.7
22.9
SAM 3 Agent
Qwen2.5-VL.
38.8
41.1"
36.7
42.8
52.6
34.3
26.6
Agent
Qwen2.5-VL
SAM 3 Agent
Liamad Maverick
36.5
41.1"
328
43.0
43.7
30.9
27.5
SAM 3 Agent
Owen3-VL 8B Thinking
40.9
41.1"
40.7
$2.9
53.3
39.5
32.4
3 Agent
Qwen3-VL Thinking
44.4
41.1"
48.4
59.4
58.0
41.5
Agent
Gemini2.5
Table 41: SAM Agent experiments on the OmniT.abel dataset (Schulter et al.
2023) (val 2023) for generalized
referring expression comprehension (box prediction). iindicates predictions generated by the hase SAM 3 model
without MLLM. The overall best performances
shown in bold. Notable baselines include: FIBER (Dou al. 2022). GLIP (Li al. 2022b), (Zhao al. 2024). DesCo (Li al. 2023c). GLEE (Wu al. 2024a).
Real CChn et al
2025
Zhou et al. 2025), ROD-MLLM Yin et al. 2025 and WSCL (Park et al. 2024
62

---

Published as a conference paper at ICLR 2026
Model
Training
RefCOCO
RefCOCO+
RefCOCOg
Name
Version
RES
val
testA
testB
lestA
TestB
(lT)
LISA
LLaVA 7B
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
79.2
73.8
63.6
75.7
77.0
GLaMM
Vicuna 7B
79.5
83.2
76.9
72.6
78.7
64.6
74.2
74.9
SAM4MLLM
LLaVAL6 7B
79.6
82.8
76.1
73.5
77.8
65.8
74.5
75.6
SAM-MLLM
LLaVAL66
79.8
82.7
74.7
74.6
80.0
75.5
76.4
70.6
GLEE
Pro DETRIS-L
80.0
69.6
72.9
DETRIS
81.0
81.9
79.0
75.2
78.6
70.2
74.6
75.3
UniLSeg
UniLSeg-200
80.5
81.8
78.4
72.7
77.0
67.0
78.4
79.5
UniLSog
81.7
83.2
78.3
79.3
80.5
PSALM
Phil.5 1.38
83.6
84.7
81.6
72.9
75.5
70.1
73.8
74.4
EVF-SAM
RC Extra Data
82.1
83.7
HO.0
75.2
78.3
70.1
76.8
77.4
EVE-SAM
82.4
84.2
80.2
76.5
80.0
71.9
78.2
78.3
Qwen2.5-7B
MLCD-seg
Qwen2.5-7B
83.6
85.3
81.5
82.9
75.6
79.7
80.5
HyperSeg
Phi2 2.78
84.8
85.7
83.4
79.0
83.5
75.2
79.4
78.9
X-SAM
Phi3 3.8B
85.1
87.1
83.4
78.0
81.0
74.4
83.8
83,9
GL-CLIP
ResNer-50
32.7
35.3
30.1
37.7
40.7
34.9
41.6
42.9
44,0
GL-CCLIP
VT B/32
32.9
34,9
30.1
38.4
42.1
32.7
42.0
42.0
42.7
30.5
Ref-Diff
VAE
37.2
37.2
40.5
44.0
44.5
44,3
TAS
ResNer-50
39.9
42,9
35.9
44.0
50.6
36.4
47.7
47.4
48.7
TAS
VIT-R/32
39.8
41.1
36.2
43.6
49.1
36.5
46.6
46.8
48.1
IicRPrimeE
40.2
46.5
33.9
51.6
35.3
46.0
45.1
45.8
Pseudo-RIS RIS
CRIS
39.8
44.8
33.0
42.2
46.3
34.5
43.7
43.4
43.8
Pscudo-RIS
ETRIS
41.1
48.2
33.5
44.3
51.4
35.1
46.0
46.7
46.8
LGD+DINO
VIT-B/32
49.5
$4.7
41.0
49.6
$8.4
38.6
50.3
51.1
$2.5
VLM-VG
ResNet-101
49.9
46.7
47.3
48.0
48.5
HybridGL
VIT-B/32
49,5
$3.4
45.2
43.4
.9.1
37.2
$1.3
51.6
Kang et al.
MaVA-115
74.2
76.5
70.4
62.5
65.2
56.0
64.2
68.1
Kang et
LLaVA-1.5 13B
78.9
67.1
67.7
SAM 3 Agent
Qwen2.5-VD
59.4
64.3
55.0
$1.4
$7.0
44.9
57.2
58.8
59.7
SAM 3 Agent
Dwen2.5-VLL 72B
X
71.6
74.9
66.0
64.9
70.8
$7.9
68.8
70.2
70.4
Agent
Llama4 Maverick
SAM Agent
Qwen3-VL 8B Thinking
68.6
72.3
63.9
59.3
64.5
55.6
66.4
66.9
67.8
SAM 3 Agent
Qwen3-V VL 235 Thinking
73.9
77.0
69.7
66.9
70.9
62.2
72.1
72.9
72.5
SAM 3 Agent
Gemini2.5
X 75.5
77.6
71.0
67.3
21.1
63.4
23.4
74.0
74.6
Table 42: SAM Agent experiments on the RefCOCO RefCOCO+ RefCOCOg datasets (Kazemzadeh
2014 Mao et al. (2016) for Referring Expression Segmentation (RES), metric is cloU, Training- RES indicates whether the model has been fine-tuned on the RAFCOCCORR//CCCOO//OOOOOOO bbox datasets). The overall best performances shown in bold and the best zero-shot performances for models trained the RES task underlined. SAM Agent achieves state-of-the-arr performance on ReiCocO+/ReiCOCOS in the zero-shot setting, and is close to best fine-tuned models. Notable baselines include: LISA (Lai et al. 2024), GSVA (Xia et al. 2024), GLaMM (Rasheed et al. 2024), SAM4MLLM (Chen et al. 2024bb). GLEE Wu et al. 2024a). DETRIS (Huang et al, 2025). UniL.Seg (Liu al. 2024b). PSALM (Zhang al. 2024c). EVF-SAM (Zhang RICE (Xie al. (225). MLCD-seg (An al. (024). HyperSeg (eii et al. [024]. X-SAM (Wang
segmentation datasets (notice
that nearly all MILMs were trained on Re(COCOOR=/CCCCO+/:
2024b).
2025a),
GL-CCIIP Yu et al. 2023b). CaR (Sun et al. 2024). Ref-Diff (Ni et al.) (023), TAS (Sno
et al.
2023). IteRPrimeE: (Wang et al. 2022cc), Pseudo-RIS (Yu et al. (024). L.D+DDINO (Li al. 2025
VLM
VG (Wang et al. 2025b), HybridGl. (Liu L 2025). and (Kang et al.
2025
63

---

Published as a conference paper at ICLR 2026
1 MODEL AND ANNOTATION CARDS
I.1 DATA ANNOTATION CARD
Task Formulation.
At high level. what are the subjective aspects of your task? There ambiguity the task. Annotators may have multiple valid interpectations of what should be masked for given phrase. Egg. If person is wearing backpack should mask for the phrose pperoon include the backpack? If the person standing next to a painting that contains person, should that person be masked too? accept this ambiguity in the task, and the SA ColGolld benchmark we use reviews from three different annotators help capture multiple interpretations.
What assumptions s do you make about annotators? Annotators worked full time on the annotation task, which allowed for frequently sharing feedback that led improvd annotations, Annotators were proficient in English and completed adequate research understand concepts that they were not familiar with. This research allowed us to annotate more fine- grained or specific concepts. like cur brands. Annotators were detail-oriented looking for all possible instances of the phruse the image. We focused more on annotation quality over annotation speed to allow annotators to carefully look for all instances.
How did you choose the specific wording of your task instructions? What steps, if any, were taken to verify the clarity of task instructions and wording for annotators? We provided detailed guidelines that included numerous examples of correct and incorrect unnotations. We broke down the task into different scenarios. and the expected outcome for each scenario. We made frequent guideline updates to handle umbiguities and address new corner cases surfaced by the vendor. The vendor trained the raters on the g guidelines, and QA around guideline clarifications. We met with the vendor weekly to provide feedback on annotation quality and surface common mistake patterns. This decreased repeat erross, and increased the quality of vendor QA's.
annotators ensure adoption. We maintained a log of vendor posed questions and answers
What, any, risks did your lask pose for aunotators and were they informed of the riska prior to engugement with the task? Annotators were instructed to seject objectionable content and flag phrases that were harmful or offensive to ground
5. What are the precise instructions that were provided annotators? The instructions varied for each of the annotation tasks. For images, we had annotators work on three separate tasks. 1) Verify the quality of masks for a given phrase in an image 2) Check if masks were exhaustively annotated in an image for a given phrase and 3) Add any missing masks and correct mask annotations such that instances of the phrase were masked in the image. For video, there were two separate tasks. 1) Exhaustively annotute all instunces of the phrase in the video and 2) Verify whether all instances are annotated with high-quafity masklets in the video.
Selecting Annotations.
Are there certain perspectives that should be privileged? If so, how did you seek these perspectives out? All unnotutors had a minimum of B-2 English proficieccy. Annotators had previous segmentation experience. Annotators researched fine grained concepts that they were unfamiliar with.
2. Are there certain perspectives that would be harmfid include? If so, how did you screen these perspectives ou? No.
Were sociodemographic characteristics used select annotators for your task? so, please detail the process. No.
4. If you lave any uggregated decio-denographic statisties about your annotator pool. please describe. Do you have reason to believe that sociodemo graphii characteristics of unnotators may have impacted how they annotated the data? Why why not? We worked with annotators based in APAC and EMEA. The sociodemogruphic characteristies of unnotators may have some impuct on the annotated data. Across different regions, words cun differ in their meanings and the same concept may look different across regions.
5. Consider the intended context of use of the dataset and the individuals and communities that may be impacted by a model trained on this dataset, Are these communities represented in your annotator poo!? Our annotator pool does not represent all communitiees that will use the SAM 3. Annotators researched concepts they were unfamiliar with. When annotators were unsure, they researched concepts to better understand diffcrent visual representations of the concept. If annotators were still unsure, they rejected the job as unsure in order to make sure that our annotations only contained confident responses. Annotators flagged concepts that were harmful in context of the image or the video.
Platform and Infrastructure Choices.
What annotation platforr did you utilize? At high level, what considerations informed your decision to choose this platform? ? Did the chosen platform sufficienaly meet the requirements you oullined for amutator pools? Are any aspects not covered? We used an internal annotation platform.
What, if any, communication channels did your chosen platform effer facilitate commurication with annotators? Hmw did this channel of communicatinn influence the annotation process and/or resulting annotations?
The research
QA'ed the vendors' quality team and annotators, shured feedback and met weekly with the vendor to align the guidelines, clarify ambiguities and surface common mistake patterns. The research team maintained a spreadsheet where they
answered the vendor's ' questions requiring the desired annotations for specitic jobs that were corner cases or amhiguous. The guidelines were frequently updated include new comnrr cases surfaced. These processes helped aliga the vendor to our desired output, which allowed the vendor to more effectively QA the annotators and provide per-annotator feedback which decreased repeat errors. A chat thread was also muintained between the research team and vendor.
3. How mack were annotators compensated? Did you consider any particular pay standards, when determining their compensation?
If so, please describe. The unnotators were compensated with an hourly wage set by the vendor.
Dataset Analysis and Evaluation.
1. How do you define the quality of annotations in your context, and how did you assess the quality in the dataset you constructed? Annotation quality was based more on ensuring completeness and validity of the annotator's 's interpretation. We defined quality across three axes: 1) mask quality (e.g. masks should not have holes or missing pieces) 2) mask concept correctness fe.g there should not be a musk around a dog when the concept is "cat") ) 3) mask exhaustivity (c.g. all instances including small background instances should masked). We a high har for quality, and aligned with the vendor on the requirements of a correct annotation.
64

---

Published as a conference paper at ICLR 2026
For each fask, annotators underwent 2-day training session led by vendor, followed annotating jobs from training queue. They were only eligible to move into the production annotation queues after the vendor QA or research team reviewed their annotations and upproved of the quality The vendor QA team continuously reviewed production annotations, covering covering 10% avg of annotations, ranging from 5% 20% dependent on task complexity across the duration of the program. The research team manually reviewed small subsets of the production annotations and shared feedback weekly.
ensure high data quality by letting annotators reject low confidence vague annotations. Annotators were asked to research concepts they were unfamiliar with unsure they present in the media. If, after researching, annotators still unsure the concept was considered vague (e.g. "sunlight") the annotators were instructed to reject the job as unsure.
In addition, all video annotations munually reviewed and only those that meet the criteria across all three use in training and evaluation, ensuring that the dataset contains only high qighiuulity, validated data.
accepted for
Have you conducted any analysis disagreement patterns? Io,0, what analyses did you and what were the major finding?? fijorfindngs?? Did you onalyze potential sources of disagreement? The PCS task inberently ambiguous und the sources of annotation disagreement are explained in 2 For SA Co/Gold, we recorded independent annotator responses order to capture the ambiguity in the task and measured their agreement. See the human metrics discussion &F more details.
How do the individual annotator responses relate to the final labels released in the dataser? For images, each image, phrase puir underwent three annotation tasks: 1) mask quality verification of masks pscudo labeled by SAM-3 PL masks 2) exhaustivity verification of accepted masks for the phrase and 3) manually adking missing masks until all instances of the concept were masked. For video annotations, individual annotator responses are validated and only amootations that meet the required quality criterion are accepted. The final labels released in the video subset consist exclusively of these accepted annotation.
For SA Co/Gold: Gold subsets were 3x multi multi-reviewerd reviewed. All three annotators had to accept mask as high quality for it to be included in the dataset, Image and phrase pairs that at least one the raters marked as non-exhann eoh-euhustivvely annotated in step two were sent manual annotation. All three of the annotators' responses were saved sepurate versions of the unnotations for the give image und phrase pair. This helped cupture the natural umbiguity in the task and alternative but valid interpretations of concept. For SA-Co/Silver: Silver subsets were reviewed. The first annotator's 's accepted masks were given to the exhaustivity annotator. If the exhaustivity annotator marked the phrase and its associated set of masks as exhaustive, this was the
Dataset Release and Maintenance.
believe the annotations in this dataset change over time? you plan update your dataset? The dataset contains annolations for public images or external puhlic datasets. Some images or datasets may become unavailahle over time.
you have
2. Are there any conditions or definitions that, if changed, could impact the utility of your dataset? Concepts change visually for example musks annotated with 'smartphone" may no longer represent modern day smartphone. New phrases or types of items will not be represented.
3. Will you attempt to track, impose limitations on, or otherwise influence how your dataset is used? If so, how? Our benchmark is for model evaluation, und should not be used for training.
4. Were annotators informed about how the data is externalized? If changes to the dataset are made, will they be informed?? No.
5. Is there pracess by which annotators can later chouse withdraa their data from the datases? If pleuse detail. No.
1.2 MODEL CARD
65

---

Published as a conference paper at ICLR 2026
Model Overview
Name
SAM (Segment Anything Model
Version
1.0
Date
2025
Organization
Meta SAM Team
Made type
Promptahle segmentation model
Architecture
See Section B
Repository
https:/ / github.bbb......... com/ Iacehookresearch/aaae......
License
SAM (detaned
repository)
Intended
Primary intended users
SAM 3 designed model for promtable concept segmentation (PCS) and promptable visual segmentation in images and videos. The model was primarily developed for research use SAM repository.
cases. SAM releused under the license detailed
Out-of-scope use cases
SAM 3 does not support complex text queries or queries including referring expressions. See license for restrictions.
Caveats and recommennations
SAM generally performs well zero-shot settings, will benefit from fine-tuning for niche domains. In video, the cost of inference scales lineurly with the number of ohjects being tracked. See $C for detailed discussion around limitutions.
Relevant Factors
Grmups
SAM was developed open world vocabulary. $ limitations prompis. SAM 3 was trained on diverse set of media pools that thit in instrumentation (e.g. medical instruments, underwater imagery, camera traps) and settings (e.g everyday scenes, driving, robotics, wildlifee)
Instrumentation and environment
Metries
We evaluate the performance of SAM 3 using metrics tailored to the specific task. For image tasks, we use the following metries:
PCS in Images with Text: egFi, MCC. pmtF;, AP. See for details.
PCS in Images with Exemplars: cgFi, AP. See $F. for details.
Instance Segmentation, Box Detection, Few Shot Box Detection: egti,, AP
Semantic and Interactive Instance Segmentation (SAM task): mloU
Counting: MAE, Accuracy
For video tasks, the following metrics:
PCS Videos: cgF,, PHOTA, mAP, See $G.5 for details.
VOS: J&J.
Promptable video segmentation (SAM tasky: offline average J&F, online average J J.F.F
Evaluation Data
Data sources
For promptable concept segmentation, SAM is evaluated using the SA-Co beachmark which is composed of:
SA-Co/Gold
SA-Co/Bronze
SA-Co/VEval
SA-Co/Silver
SA-Co/Bio
Each of these subsets is composed series data sources. The SA-Co benchmark annotations ure releused publicly httpa: httppp:///gthbb...p htt.ttttt.....eg.ggggg......a..t com.
details.
Escebookresearchh 5am3
Training Data
Data source
SAM 3 is trained using the following datasets:
SA-Co/HO
SA-Co/YN
SA CA-CCIVIDEO-ER EXT
SA-Co/EXT
SA-CoVVIEO
SA-IB
SA-Co/EXT, SA Co-COOVIDEO SA Co/SYN SA-Co/HO, SA CA-Co/VIDEO-ENY composed of
multiple data sources, Table 2o $. for the list of data for subset. See Table 18
D.441 for details as which datasets are used during which training stages.
Ethical Considerations
Data
See [.] for data annotution card
Cost and impact compute
Ieaase SAM trained 172k A.00 GPU hours and 86k H200 GPU hours. This
corresponds estimated 142k-176k kWH and emissions between 65.3 und 77.6 metric tons
CO2e (Patterson al. 2021 Lacoste al. (201)), The emissions from training the released
SAM model equivalent 166k miles drven by average gasoline- powered passenger
vehicle (Agency 2222).
Risks and harms
The behavior of SAM is undetined for opinionated or subjective queries (e-g. magnificent painting) and should be used make subjective judgments about objects or people. Users
should evaluate the safety of SAM 3 tailored to their specific use
Use cases
We implore users to use their best judgment.
Table 43; Model card SAM following the structure Mitchell et al.
2019
66

---

Published as a conference paper at ICLR 2026
REFERENCES
United States Environmental
Protection
Agency.
Greenhouse
gas equiv-
alencies calculator, 2022. URL
https://www https:::wwwwwwa.aaee.. www.wepp..wv/eeee...m gov/energy/
greenhouse gas
cacsss-calllaaaall ator
Xiang An, Kaicheng Yang, Xiangzi Dai, Ziyong Feng, and Jiankang Deng. Multi-label cluster discrimination for visual representation learning. In European Conference Computer Vision, pp. 428-444. Springer, 2024.
Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: benchmark for unifying object recognition, segmentation and tracking in video. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 1673,1683, 2023.
Hexin Bai, Wensheng Cheng, Peng Chu, Juehuan Liu, Kai Zhang, and Haibin Ling. Gmot-40: A benchmark for generic multiple object tracking. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition, pp. 6719-6728, 2021.
Kevin Barnard, Elaine Liu, Kristine Walz, Brian Schlining, Nancy Jacobsen Stout, and Lonny Lundsten. DeepSea MOT: A benchmark dataset for multi-object tracking deep-sea video. arXiv preprint arXiv:2509.03: 03499, 2025. doi: 00445500arxxv225000044999
Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In Proceedings of the JEEF/CVF international conference on computer vision, pp. 941-951, 2019.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP), pp. 3464-3468. leee, 2016.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024.
Alexey Bochkovskiy, Chien Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection, 2020. /ttp:////a.....o///////aaa////a/
10934
Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and Christoph Feichtenhofer. Window attention is bugged: How not to interpolate position embeddings. In International Conference on Learning Representations, 2024.
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network. arXiv:2504. 13181, 2025.
Zhi Cai, Songtao Liu, Guodong Wang, Zeming Li, Zheng Ge, Xiangyu Zhang, and Di Huang. Align-detr: Enhancing end-to-end object detection with aligned loss. In 35th British Machine Vision Conference 2024, BMVC 2024, Glasgow, UK, November 25-28, 2024. BMVA, 2024. URL
httpp:://pppe:: bwwpb024 wttp:///.///ro..ee
Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9686-9696. 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Zagoruyko.
Qiang Chen, Xiangbo Su, Xinyu Zhang, Jian Wang, Jiahui Chen, Yumpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, et al. Lw-detr: transformer replacement to yolo for real-time detection. arXiv preprint 2XXi:2406.034550 2024a.
67

---

Published as a conference paper at ICLR 2026
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu- Yu-Chiang Frank Wang. and Chu-Song Chen. Sam4mllm: Enhance multi modtl large language model for referring expression segmentation. In European Conference Computer Vision, pp. 323-340. Springer, 2024b.
Yuming Chen, Jiangyan Feng, Haodong Zhang, Lijun Gong, Feng Zhu, Rui Zhao, Qibin Hou, Ming- Ming Cheng, and Yibing Song. Re-aligning language to visual objects with an agentic workflow. In International Conference on Learning Representations, 2025.
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan. Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Hanoona Rasheed, Peize Sun, Po- Yao Huang, Daniel Bolya, Suyog Jain, Miguel Martin, Huiyu Wang, Nikhila Ravi, Shashank Jain, Temmy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Philipp Krähenbühl, Piotr Dollár, Lorenzo Torresani, Kristen Grauman, and Christoph Feichtenhofer. Perceptionlm: Open-access data and models for detailed visual understanding. 1:Xi::5504.3188, 2025.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Puttern Recognition 2016.
Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil in the details, 2022. URL httpp:://aaxiv..ot
abs 2bs// 01066
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et Molmo and pixmo: Open models. In Proceedings of the Computer
weights and open data for state-of-the Vision and Puttern Recognition Conference, pp. 91-104, 2025.
Henghui Ding. Kaining Ying. Chang Liu, Shuting He, Xudong Jiang. Yu-Gang Jiang, Philip HS Torr, and Song Bai. Mosev2: A more challenging dataset for video object segmentation in complex scenes. arXiv preprint arXiv:2508. 05630, 2025.
Kexin Ding, Mu Zhou, He Wang, Olivier Gevaert, Dimitris Metaxas, and Shaoting Zhang. A large-scale synthetic pathological dataset for deep learning-enable segmentation of breast cancer. Scientific Data, 10(1):231, 2023.
Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam 2 for long video segmentation with a training-free memory tree. arXiv preprint arXiv:2410.16266 2024.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. arXiv preprint arXiv:2208. 08984, 2022.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang. Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, and Lijuan Wang. Coarse-to-fine vision- language pre-training with fusion in the backbone, 2022. URL
2206. 07643
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
68

---

Published as a conference paper at ICLR 2026
Christoffer Edlund, Timothy R Jackson, Nabeel Khalid, Nicola Bevan, Timothy Dale, Andreas Dengel, Sheraz Ahmed, Johan Trygg. and Rickard Sjögren. Livecell large-scale dataset for label-free live cell segmentation. Nature methods, 18(9):1038-0045 2021.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Detect track and track to detect. In Proceedings of the IEEE international conference on computer vision, pp. 3038-3046, 2017.
FFmpeg developers. FFmpeg. https://EEmpeg.org/
Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan. Xihan Wei, Jingke Meng. Xiaohua Xie, and Wei-Shi Zheng. LImdet: Learning strong open- vocabulary object detectors under the supervision of large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
Jevgenij
Gamper, Navid Alemi Koohbanani, Ksenija Benes, Ali Khuram, and Nasir Rajpoot. Pannuke: open pan-cancer histology dataset for nuclei instance segmentation and classification. In European Congress on Digital Pathology, pp. 1 11-19. Springer, 2019.
Jevgenij Gamper, Navid Alemi Koohbanani, Simon Graham, Mostafa Jahanifar, Syed Ali Khurram, Ayesha Azam, Katherine Hewitt, and Nasir Rajpoot. Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003. 10778, 2020.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi. Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2022.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint 1rXiv:2104.13921 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on compuler vision and patlern recognition, pp. 16000-16009, 2022.
Byeongho Hco, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. arXiv preprint arXiv:2403.13220 2024.
Lingyi Hong. Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, et al. Lvos: A benchmark for large-scale long-term video object segmentation. arXiv preprint arXiv:2404. 19326, 2024.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo.. 12
Grant Van Hor, Oisin Mac Aodha. Yang Song. Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist challenge 2017 dataset. CoRR, abs//707..ol 7766426,2 2017. URL http:://axil httt:://axxv.oorr/abb////70
69

---

Published as a conference paper at ICLR 2026
Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. DAC-DETR: Divide the attention layers and conquer. In Advances in Neural Information Processing Systems. 2023.
Jiaqi Huang, Zunnan Xu, Ting Liu, Yong Liu, Haonan Han, Kehong Yuan, and Xiu Li. Densely connected barameter-efffcient tuning for referring image segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1653-3661, 2025.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint arXiv:2207. 2022.
Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, and Serge J. Belongie. Fashionpedia: Ontology, segmentation, and an attribute localization dataset. CoRR, abs/2004. 12276, 2020. URL
2004. 12276
Junjie Jiang. Zelin Wang, Manqi Zhao, Yin Li, and DongSheng Jiang. Sam2mot: A novel paradigm of multi-object tracking by segmentation. arXiv preprint 2rXiv:2504.045199 2025.
Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision, pp. 38-57. Springer, 2024.
Arne Hoffhues Jonathon Luiten.
Trackeval.
TrackEval 2020.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790, 2021.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9339-9350, 2025.
Kakani Katija, Eric C. Orenstein, Brian Schlining, Lonny Lundsten, Kevin Barnard, Giovanna Sainz, Oceane Boulais, Benjamin G. Woodward, and Katy Croff Bell. Fathomnet: A global underwater image training set for enabling artificial intelligence in the ocean. CoRR, abs/2109. 14646, 2021.
URL httpt:tss////ttt/t org/ httgg//bb/2105 2109. 14646
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Lei Ke, Henghui Ding, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transtiner for high-quality video instance segmentation. In European Conference on Computer Vision, pp. 731-747, Springer, 2022.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung. Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O' O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip. Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale wild robot manipulation dataset. 2024.
70

---

Published as a conference paper at ICLR 2026
Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models, 2024. URL
battps://arxi>: org/ a / 2408 12569
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yea Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popoy, Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956-198 2020.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 2019.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya lia. Lisa: Reasoning segmentation via large language model. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589, 2024.
Hoang-An Le, Partha Das, Thomas Mensink. Sezer Karaoglu, and Theo Gevers. EDEN: Multimodal Synthetic Dataset of Enclosed garDEN Scenes, In Proceedings of the IEEE/CVF Winter Conference of Applications on Computer Vision (WACV), 2021.
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating sunguage-augment visual models. Advances in Neural Information Processing Systems, 35: 9287-9301, 2022a.
Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Hu-Sheng Xu, Hongyang Li, Chun yue Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Visual in-context prompting. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12861-12877
2023a. UtLtttpss://ap: semant ientticschllast oor./ooopps
Jiachen Li, Qing Xie, Renshu Gu, Jinyu Xu, Yongjian Liu, and Xiaohan Yu. Lgd: Leveraging genera- tive descriptions for zero-shot referring image segmentation. arXiv preprint 47X4v.2504.14467, 2025.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b. URL https
12597
Liunian Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang. Desco: Learning object recognition with rich language descriptions. Advances in Neural Information Processing Systems, 36:37511-377 1-37526. 2023c.
Liunian Harold Li, Pengehuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10965-10975, 2022b.
Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In European Conference on Computer Vision, 2022c.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pp. 280-296. Springer, 2022d.
71

---

Published as a conference paper at ICLR 2026
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7061-7070, 2023.
LILA BC.
WCS
camera
traps.
URL
BtDp:://ilaa.sconne/aaaaaa
wcscameral rattmmertttae raps.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yutong Lin, Yuhui Yuan, Zheng Zhang, Chen Li, Nanning Zheng, and Han Hu. Detr doesn't tt need multi-scale or locality design. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6545 6554,, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, 2023. semant www.sccccchhaaa.. wwt.esseeaaanniiiiiaeiei
htthtttps:/
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024a.
Ting Liu and Siyuan Li. Hybrid global-local representation with augmented spatial guidance for zero-shot referring image segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29634-29643, 2025.
Yong Liu, Cairong Zhang, Yitong Wang, Jiahao Wang, Yujiu Yang, and Yansong Tang. Universal segmentation at arbitrary granularity with language instruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3459-3469, 2024b.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520 2025.
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. ledar 2023 competition on hierarchical text detection and recognition. arXiv preprint arXiv:2305.09750, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
Yi Lu, Jiawang Cao, Yongliang Wu, Bozheng Li, Licheng Tang. Yangguang Ji, Chong Wu, Jay Wu, and Wenbo Zhu. Rsvp: Reasoning segmentation via visual prompting and multi-modal chain-of-thought. arXiv preprint arXiv:2506.044 04277, 2025.
Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura LaiieTaixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, pp. 548-578, 2021.
Jun Ma, Zongxin Yang, Sumin Kim, Bihui Chen, Mohammed Baharoon, Adibvafa Fallahpour, Reza Asakereh, Hongwei Lyu, and Bo Wang. Medsam2: Segment anything in 3d medical images and videos. arXiv preprint arXiv:2504. 03600, 2025.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu. Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IÉEE conference on computer vision and pattern recognition, pp. 1 11-20, 2016.
72

---

Published as a conference paper at ICLR 2026
Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, and Hui Xue. Coco- o: A benchmark for object detectors under natural distribution shifts, 2023. URL https
7/ //arxiv. www.vvorg//aa abs/ http://bb2233.....
Tim Meinhardt, Alexander Kirillov, Laura Teixe and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8844-8854, 2022.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, al. Simple open-vocabulary object detection. In European conference computer vision, pp. 728-755. Springer, 2022.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection, 2024. URL BtLltttss///xxxx.rrrt http://aaaaa...oooaa///20000 abs.////.////33..///
09683
Chaitanya Mitash, Fan Wang, Shiyang Lu, Vikedo Terhuja, Tyler Garaas, Felipe Polido, and Manikan- tan Nambi. Armbench: object-centric benchmark dataset for robotic manipulation. arXiv preprint arXiv:2303. 16382, 2023.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.
Sharada Prasanna Mohanty, Gaurav Singhal, Eric Antoine Scuccimarra, Djilani Kebaili, Harris Héritier, Victor Boulanger, and Marcel Salathé. The food recogmition benchmark: Using deeplearn- ing to recognize food on images, 2021. Utt.httss..//x.....rrr///
2106 22101.14977.
Seong-Whan Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Scong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
National Gallery of Art. Public domain collection dataset. URL https:://///.. www. nga...... gov artworks/ ffreee/reeeeeeeee images----.... and- open-oen--- access
Minheng Ni, Yabo Zhang, Kailai Feng, Xiaoming Li, Yiwen Guo, and Wangmeng Zuo. Ref-diff: Zero-shot referring image segmentation with generative models. arXiv preprint arXiv:2308.1677 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang. Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL https: https::/ net/ Forum?id=a
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 31 3170-3180, 2023.
Kwanyong Park, Kuniaki Saito, and Donghyun Kim. Weak-to-strong compositional learning from generative models for language-basee object detection. In European Conference on Computer Vision, pp. 1-19. Springer, 2024.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Aluis-Mique Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:21044 10350, 2021.
Paul Voigtlaender, Valentin Gabeur and Rohan Doshi. mentation with Gemini 2.5. [attps://eevloopers www.lebloo.oooooooooooooo com/en/
Conversational image seg-
conversati ona 1 image------------ www.eeetttttooooneniiii----22-2a 2025.
73

---

Published as a conference paper at ICLR 2026
Bryan A Plummer, Kevin J Shih, Yichen Li, Ke Xu, Svetlana Lazebnik, Stan Sclaroff, and Kate networks for open-ended phrase detection. IEEE transactions
Saenko. Revisiting pattern analysis and machine intelligence, 44(4):2155-214 2020.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Workine-Hornung and Luc Van Gool. The 2017 davis challenge on vidco object segmentation. arXiv preprint arXiv:1704.000 00675, 2017a.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alexander Sorkine- and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675. 2017b.
PySceneDetect Developers. PySceneDetect, https: wttp:/wwwwwwneeettttccc.
Jiyang Qi, Yan Gao, Yao Hu. Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip Torr, and Song Bai. Occluded video instance segmentation: A benchmark. International Journal of Computer Vision, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B. Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Geode: a geographically diverse evaluation dataset for object recognition. In NeurIPS Datasets and Benchmarks, 2023.
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3394-3403, 2021.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing. Ming-Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009- 13018, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 0078000714.... 2024.
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300 2024a.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world world models for diverse visual tasks. arXiv preprint arXiv:2401.144 14159, 2024b.
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng. Yuda Xiong, Wenlong Liu, Zhengyu Ma. Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song. Yuhong Zhang, Hongjie Huang, Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, and Lei Zhang. Dino-x: A unified vision model for worn-world object detection and understanding, 2025. (tthtt
Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow 100-v1: multi-domain object detection benchmark for vision- Langualae models. arXiv preprint arXiv:2505.20612, 2025.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal computer vision, 115(3)2211 (553)2211-222.. 1-252, 2015.
74

---

Published as a conference paper at ICLR 2026
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaced, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A hierarchical vision transformer without the bells-and- whistles. ICML, 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01100 2019.
Samuel Schulter, Yumin Suh, Konstantinos M Dafnis, Zhixing Zhang, Shiyu Zhao, Dimitris Metaxas, et al. Omnilabel: A challenging benchmark for language-based object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11953-11962, 2023.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430-8439, 2019.
Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13193-13203, June 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2 M0Xi:2200.009644, 2021.
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan. Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012.15460, 2020.
Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip mn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13171-13182, 2024.
Yucheng Suo, Linchao Zhu, and Yi Yang. Text augmented spatial-aware zero-shot referring image segmentation. arXiv preprint arXiv:2310.18044 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL /Lle/3Ffoe2235//eeeee1000ddaadd3
wwwripe. cc/paper files//aper:20177
Papeer pdf
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Commun. doi: 10.1145/2629489. URL https
ACM, 57(10)78-88 September 2014. ISSN 0001-0782.
, //4ol..og/100.114//222233
Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, and Xiaodan Liang. X-sam: From segment anything to any segmentation. arXiv 2025a.
preprint arXiv: arXiv:2508.0460
Haochen Wang. Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. Towards ooca-vocayulary video instance segmentation. In proceedings of the IEEE/CVF international conference on computer vision, pp. 4057-4066, 2023.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024.
Shijie Wang. Dahun Kim, Ali Taalimi, Chen Sun, and Weicheng Kuo. Learning visual grounding from generative vision and language model. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 8067-8067. IEEE, 2025b.
75

---

Published as a conference paper at ICLR 2026
Yuji Wang, Jingchen Ni, Yong Liu, Chun Yuan, and Yansong Tang. Iterprime: Zero-shot referring image segmentation with iterative grad-cam refinement and primary word emphasis. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 8159 8168, 2025c.
Dante Francisco Wasmuht, Otto Brookes, Maximillian Schall, Pablo Palencia, Chris Beirne, Tilo Burghardt, Majid Mirmehdi, Hjalmar Kühl, Mimi Arandjelovic, Sam Pottie, Peter Bermant, Brandon Asheim, Yi Jin Toh, Adam Elzinga, Jason Holmberg, Andrew Whitworth, Eleanor Flatt, Laura Gustafson, Chaitanya Ryali, Yuan-Ting Hu, Baishan Guo, Andrew Westbury, Kate Saenko, and Didac Suris. The SA-FART dataset: Segment anything in footage of animals for recognition and identification. arXiv preprint arXiv:2511.15622 2025. URL htttttts: httpp/ss//axxv..oo orr/abs/ 22111 15622
Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: To- wards universal visual segmentation with large language model. arXiv preprint 1rXiv22411.77600 2024.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference on image processing (ICIP), pp. IEEE, 2017.
Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3783-3795 2024a.
Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3783-3795, 2024b.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3858-3869. 2024.
Yin Xie, Kaicheng Yang, Xiang An, Kun Wu, Yongle Zhao, Weimo Deng. Zimin Ran, Yumeng Wang, Ziyong Feng. Roy Miles, et al. Region-based cluster discrimination for visual representation learning. arXiv preprint arXiv:2507.20025, 2025.
Hu Xu, Po- Yao Huang, Xiaoqing Ellen Tan, Ching Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, et al. Altogether: Image captioning via 10.2211,, 2024a.
re-aligning alt-text. arXiv preprini
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li. Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data, 2024b. URL httptttss///xxii...t.t wttp:/wwwwwrgg//////////////// 0000099.66667 16671
N. Xu, L. Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas S. Huang. Youtube-vos: large-scale video object segmentation benchmark. ArXiv, abs/I htts/880.000 08990033727 2018.
Yifan Xu, Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang, Ke Li, and Changsheng Xu. Multi-modal queried object detection in the wild. Advances in Neural Information Processing Systems, 36:4452-4400 2023.
Cheng-Yen Yen Yang. Hsiang- Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory. arXiv preprint arXiv:2411.1192) 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mant prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:23100 11441, 2023.
Heng Yin, Yuqiang Ren, Ke Yan, Shouhong Ding, and Yongtao Hao. Rod-mllm: Towards more reliable object detection in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14358-1438 14368, 2025.
76

---

Published as a conference paper at ICLR 2026
En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release-fetee supervision for end-to-end multi-object tracking. arXiv preprint arXiv:230. 142988 2023a.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with global-local context features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19456-19465, 2023b.
Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Pseudo-ris: Distinctive
generation for referring image segmentation. In European Conference on Computer Vision, pp- 18-36. Springer, 2024.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. In CVPR, 2022.
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In European conference on computer vision, pp. 659-675. Springer, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022. URL URL Cttlhtps://// httttp://arxi...oooo org/ abs /2t.//116.0000 04560
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang. Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint (XXiv:2203303600 2022a.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022b.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in neural information processing systems, 37:71737-71767, 2024a.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng. Zehuan Yuan, Ping Luo. Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In European conference on computer vision, pp. 1-21. Springer, 2022c.
Yuxuan Zhang, Tianheng Cheng, Lianghui Zhu, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Evf-sam: Early vision- languuge fusion for text-prompted segment anything model. arXiv preprint 2r06v:2406. 2406.2076. 2024b.
Zheng Zhang, Yeyao Ma. Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pp. 74-91. Springer. 2024c.
Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He. Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Sec: Advancing complex video object segmenta- tion via progressive concept construction. arXiv preprint arXiv:2507.15852, 2025.
Shiyu Zhao, Long Zhao, Yumin Suh, Dimitris N Metaxas, Manmohan Chandraker, Samuel Schulter, et al. Generating enhanced negatives for training language-based object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13592 13602. 2024.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127 2019.
77

---

Published as a conference paper at ICLR 2026
Yang Zhou, Shiyu Zhao, Yuxiao Chen, Zhenting Wang, Can Jin, and Dimitris N Metaxas. Led: Llm enhanced vocabuluy object detection without human curated data generation. arXiv preprint arXiv:2503.13374, 2025.
Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11))7380-7399 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04139) 2020.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36: 16:19669-199797777 2023.
78


Reviews:
[{"review_id": "HmudHFyNzR", "reviewer": "Reviewer_EmE5", "summary": "This paper introduces SAM3, a unified model that can perform detection on images and tracking across frames based on given concepts or prompts. The main contributions are twofold: proposing the SAM3 method and introducing a data engine pipeline. Unlike SAM2, it can detect multiple objects based on given concepts, extending its capabilities. Furthermore, by incorporating MLLMs, it produces more precise and higher quality datasets. The results demonstrate strong generalization ability across various scenarios.", "questions": "- Motion-aware Concept\n  - In SAM3, the main focus is on simple noun phrases. For more complex or longer phrases, the paper demonstrates the use of MLLMs to handle such cases. However, in video segmentation, object descriptions often incorporate motion-aware information. In such cases, which approach could be used? Could the SAM3 Agent design still be applied to such scenarios? The current design seems to use the tracking module only to associate objects across frames in a semantic-agnostic manner.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Exhaustive Analysis\n  - The paper demonstrates its effectiveness through extensive analyses provided in the appendix.\n  - Each component is evaluated with ablation studies, and the paper even illustrates the impact of AI verification through various experiments.\n  - For both image and video datasets, the importance of each and the effects of different dataset sizes are thoroughly explored.\n- Open-sourcing\n  - The paper open sources key components, including SAM3 and the SA-Co benchmark. In particular, releasing the SA-Co benchmark contributes to the perception research community by providing challenging and diverse samples.", "weaknesses": "- Unclear Definition of Terms\n  - Throughout the paper, the term **geometric** is frequently used. What is its exact meaning here? At times, *geometric* is distinguished from *visual prompts*, but in other cases, it seems to include their meaning. Although it represents an important concept in the paper, its usage is quite vague.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MkUVfCPHlv", "reviewer": "Reviewer_N5F4", "summary": "This paper proposes SAM 3, an advanced segmentation foundation model aiming to perform “Prompted Concept Segmentation (PCS)” tasks, where text and/or image exemplars serve as prompts for concept understanding and object segmentation. The system integrates large-scale data, modular architecture design, and training techniques into a unified framework, showing strong segmentation performance across multiple benchmarks. The paper presents extensive dataset construction and evaluation.", "questions": "1.    SAM 3 presents itself as a large-scale, highly engineered system integrating architecture design, massive data curation, and multi-modality prompt handling. While its engineering quality and practical completeness are impressive, I am uncertain whether such a system-level project aligns with ICLR’s focus on methodological and theoretical innovation.\n\n   * Would the authors consider that SAM 3, with its solid engineering and cross-domain applicability, might be more suitable for a Nature-type venue, where large, impactful engineering frameworks and cross-domain demonstrations are more appreciated?\n   * Has the team considered including case studies or validations in diverse domains (e.g., medical imaging, industrial inspection, materials science, or bioinformatics) to better highlight the model’s broad real-world impact?\n\n\n2.   I acknowledge the revolutionary impact of SAM (1), which fundamentally redefined segmentation as a promptable and interactive visual understanding task. SAM 2 further extended this to the temporal domain through hierarchical memory and video modeling.\n   However, SAM 3 appears to be more of a task-level extension (image -> video -> concept) rather than a technical breakthrough.\n\n   * The progression of input and output forms (point or box to frame sequence to text or image exemplars; single-frame masks to temporal sequences to concept-level masks) seems evolutionary rather than fundamentally new.\n   * It is difficult to evaluate SAM 3’s novelty in isolation from SAM (1) and SAM (2). Its advances appear to rely heavily on the established SAM framework and infrastructure.\n   * If the entire SAM series (1, 2, and 3) were viewed as a single long-term contribution, I would rate it extremely high, possibly a 10/10. For SAM 3 alone, however, the incremental contribution ratio remains unclear. Could the authors clarify what specific conceptual or architectural innovations distinguish SAM 3 from its predecessors beyond scaling and multi-modality integration?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. Engineering Excellence and Practical Impact.\nSAM 3 demonstrates strong engineering quality and productization potential, similar to recent systems such as DeepSeek. Its framework and modular tools are robust and well implemented, showing strong potential for deployment and for industrial or cross-domain applications.\n\n2. Comprehensive Dataset and Benchmarking.\nThe data engine and dataset organization are well executed, standardized, systematic, and clear. The large-scale data collection and clearly defined evaluation pipeline demonstrate solid engineering effort.\n\n3. High Performance through Scalable Architecture.\nThe model achieves impressive segmentation results through a reasonable architectural combination, large-scale data, and effective training strategies.\n\n4. Potential for Broader Impact.\nThe SAM 3 Agent and its associated tools are solid and stable. With additional validation in medical, industrial, or molecular domains, this work could be even more suitable for a Nature-level publication due to its cross-domain applicability and engineering completeness.", "weaknesses": "1. Limited Novelty Beyond Existing Referring and Open-Vocabulary Segmentation Frameworks. While SAM 3 extends promptable segmentation to a broader “concept-level” scope, its core mechanism remains largely similar to existing referring and open-vocabulary segmentation approaches. The main novelty lies in system integration, data scaling, and multimodal prompting, rather than in algorithmic or conceptual innovation. \nThe paper’s strength is its engineering completeness and potential impact, but from a research novelty standpoint, its incremental contribution beyond prior referring segmentation frameworks (e.g., CLIPSeg (CVPR'22 [https://arxiv.org/abs/2112.10003]), SEEM (NeurIPS'23 [https://arxiv.org/pdf/2304.06718]), Grounded-SAM & Grounaded 2 (arXiv [https://arxiv.org/abs/2401.14159]) appears limited.\n2. Lack of Theoretical Insight. The paper appears more like a large-scale project than a conceptual or methodological contribution. The performance improvements mainly come from scaling data and model size, rather than introducing new conceptual insights or algorithmic innovations.\n3. Limited Definition and Discussion of “Concept”. \nThe notion of “concept” is underexplained. Since “concept” is an abstract idea, the paper should explicitly define it and clarify how PCS differs from existing works such as Spider (ICML'24 (https://arxiv.org/abs/2405.01002)) and SAM-Eva (arXiv (https://arxiv.org/abs/2412.01240)), which already distinguish between CI (context-independent) and CD (context-dependent) concepts. Without a clear definition, the term “Prompt Concept Segmentation” remains ambiguous. I suggest that the authors expand the Related Work section to provide a more detailed explanation of how “concept” has been defined and studied in prior literature. It would also be valuable to include additional experiments to demonstrate how SAM 3 performs across different types of concepts, particularly distinguishing between context-independent and context-dependent cases.\n4. Task Limitation (PCS Inference Scope). The PCS task currently handles single-image or text-based prompts but lacks the ability for batch or generalizable reasoning across multiple images or complex prompts.\n5. Lack of Novelty in Data Engine. While the data pipeline is well-organized, it is largely an engineering implementation without notable methodological innovation.\n6. Questionable Data Efficiency.\nIn Table 14, the improvement from using 20% → 100% data is comparable to the smaller-scale increments (10% → 20%, etc.), suggesting poor data efficiency and underutilization of the large dataset. Moreover, there remains a significant gap between the full-data model and the teacher model, implying room for optimization in training or architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uPJxgLdo6M", "reviewer": "Reviewer_ma9q", "summary": "This work introduces SAM 3, a new extension of the \"Segment Anything Model\" paradigm that formalizes and successfully tackles the promptable concept segmentation task.\nThis moves the field beyond single-object segmentation to the more complex challenge of detecting, segmenting, and tracking all instances matching a given concept text prompt.\nThis advance is underpinned by a scalable data engine to generate a massive, high-quality dataset.\nThe refined model architecture effectively decouples recognition and localization for better open-vocabulary segmentation.\nThe method demonstrates a good performance gain on this new task and this paper establishes the new and large-scale SA-Co benchmark.", "questions": "1. What is meant by the use of \"group masks\" as mentioned in Lines 246-248?\n1. For the evaluation of zero-shot capabilities, the authors are advised to include a performance comparison on the MESS [1] benchmark. This benchmark covers a diverse range of target domains, which would provide a more comprehensive perspective on the model's generalization abilities.\n1. The paper restricts its analysis to specific object counting benchmarks. This ignores the long-standing, highly challenging, and practical domain of crowd counting [2] (e.g., on datasets like UCF-QNRF,  JHU-Crowd, ShanghaiTech, or UCFCC50). These dense, highly-occluded scenarios are the true stress test for any model claiming robust counting abilities.\n\nREF:\n1. What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation, 2023\n1. Revisiting crowd counting: State-of-the-art, trends, and future perspectives, 2023", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The introduction of the \"presence head\" is a specific and impactful architectural contribution. This design choice directly addresses the challenge of open-vocabulary detection by decoupling the recognition from the localization, which the ablations show is effective.\n- The paper demonstrates compelling quantitative results, achieving a great performance gain on the PCS task and setting a new state-of-the-art on existing benchmarks. These results validate the efficacy of the complete system.\n- The SAM 3 model and the large-scale SA-Co benchmark are valuable. This provides a valuable new asset for the community.\n- While other works have explored text-prompted segmentation, this paper successfully integrates concept-level segmentation and tracking into a single, unified SAM model. The ability to scale this integration to such a large dataset and achieve robust performance is a notable engineering accomplishment.", "weaknesses": "Given that this paper can be viewed as the latest advancement in the SAM series, a critical point of evaluation is the extent of its novelty and extension over prior work (SAM 1 & 2). The contributions can be broadly categorized into three areas: (1) task definition, (2) data benchmark, and (3) model design.\n\n1. On the task definition: The paper extends promptable visual segmentation to promptable concept segmentation. However, the integration of text prompts (in addition to conventional interactive prompts) has already been explored by numerous existing works [1,2,3]. This makes the novelty of this specific task definition, particularly for the SAM series, appear limited.\n1. On the data benchmark: What are the specific differences and unique innovations of the data engine workflow (Sec. 4) compared to the pipelines used for SAM 1 & 2? The primary distinction appears to be the incorporation of text, which seems like a necessary adaptation for the new task rather than a fundamental innovation in the data engine itself. Furthermore, existing works [1,2,3] have also constructed large-scale datasets for the similar tasks. The paper fails to adequately discuss or differentiate its data collection process (in terms of workflow and details) from these prior efforts.\n1. On the model design: Existing research [4] has shown that SAM models can be sensitive to perturbations in interactive prompts. This paper also lacks an investigation into this aspect. How does SAM 3 perform when faced with perturbations in its text or visual prompts, such as variations in phrasing (expression deviations) or positional shifts (spatial deviations)?\n\nREF:\n1. Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos, 2025\n1. VoCap: Video Object Captioning and Segmentation from Any Prompt, 2025\n1. Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos, 2025\n1. Inspiring the Next Generation of Segment Anything Models: Comprehensively Evaluate SAM and SAM 2 with Diverse Prompts Towards Context-Dependent Concepts under Different Scenes, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gkp24HNazl", "reviewer": "Reviewer_wjcv", "summary": "This work proposes a new task called Promptable Concept Segmentation (PCS), which segments target objects through language prompts or image examples. Authors build a scalable data engine to automatically generate training data. They also present a new model, SAM3, to implement PCS. SAM3 has achieved excellent performance in both image and video segmentation tasks.", "questions": "1. Do you have the plan to release the code and model? I think it is quite import for the community.\n2. It seems that the concept of SAM3 is limited to simple text and cannot perform complex reasoning like ReasonSeg. Authors have used an agent-based approach to achieve this. However, authors did not show the performance of using SAM3 directly for complex reasoning, which I find quite curious.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Generalized prompts. Compared with previous works, PCS has a wider range of prompts and SAM3 achieves better performance.\n2. Data engine. The data engine proposed by authors can effectively expand data, including mask and text, which is beyond the capability of previous works.\n3. SAM3 achieves PCS for both images and videos through a detection-then-tracking paradigm.\n4. SAM3 achieves sota performances over several tasks and benchmarks.", "weaknesses": "1. The formulation of SAM3. This detection-then-tracking approach of SAM3 follows a two-stage format. It inevitably introduces errors and requires hyperparameter in the merging stage. The paradigm of SAM3 could be further adjusted to implement PCS in an end-to-end manner instead of a two-stage one.\n2. Complex naming conventions. The naming of training and test data in the paper is overly complex, with numerous similar names. Additionally, due to space limitations, authors did not elaborate on the meanings of different names, which makes this quite confusing.\n3. Missed citation. Many of the mentioned datasets or benchmarks are not accompanied by citations in the references.\n4. Multi-stage training, which trains different parts of SAM3. A final end-to-end training that fine-tunes the entire model, may further improve performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)