Tracking and Understanding Object Transformations
Authors: Yihong Sun
Xinyu Yang
Jennifer J. Sun
Bharath Hariharan
Venue: NeurIPS 2025 poster ()
Keywords: object tracking
transformations
state change detection
spatiotemporal reasoning
zero-shot
Primary area: applications
TL;DR: We track objects through transformations while detecting and describing these state changes.

Abstract: Real-world objects frequently undergo state transformations. From an apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects and dynamics. However, existing methods often lose track of the target object after transformation, due to significant changes in object appearance. To address this limitation, we introduce the task of Track Any State: tracking objects through transformations while detecting and describing state changes, accompanied by a new benchmark dataset, VOST-TAS. To tackle this problem, we present TubeletGraph, a zero-shot system that recovers missing objects after transformation and maps out how object states are evolving over time. TubeletGraph first identifies potentially overlooked tracks, and determines whether they should be integrated based on semantic and proximity priors. Then, it reasons about the added tracks and generates a state graph describing each observed transformation. TubeletGraph achieves state-of-the-art tracking performance under transformations, while demonstrating deeper understanding of object transformations and promising capabilities in temporal grounding and semantic reasoning for complex object transformations. Code, additional results, and the benchmark dataset are available at https://tubelet-graph.github.io.

Tracking and Understanding Object Transformations
Yihong Sun Cornell University
Xinyu Yang Cornell University
Jennifer J. Sun Cornell University
Bharath Hariharan Cornell University
Abstract
Real-world objects frequently undergo state transformations. From apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects and dynamics. However, existing methods often lose track of the target object after transformation, due to significant changes in object appearance. To address this limitation, we introduce the task of Track Any State: tracking objects through transformations while de- tecting and describing state changes, accompanied by a new benchmark dataset, VOST-TAS. To tackle this problem, we present TubeletGraph, zero-shot system that recovers missing objects after transformation and maps out how object states are evolving over time. TubeletGraph first identifies potentially overlooked tracks, and determines whether they should be integrated based on semantic and proximity priors. Then, it reasons about the added tracks and generates a state graph describ- ing each observed transformation. TubeletGraph achieves state-of-the-arr tracking performance under transformations, while demonstrating deeper understanding of object transformations and promising capabilities in temporal grounding and se- mantic reasoning for complex object transformations. Code, additional results, and the benchmark dataset are available at https:
httpp://tubele /ttpbelettteeeeeeeeeeeeeeeeeeeee graph. gi github... io.
Introduction
As the Greek philosopher Heraclitus noted, nothing is permanent but change. All around us, objects undergo transformations that can dramatically alter their appearance, geometry, and sometimes even their identities. In nature, seeds give birth to plants, chicks emerge from eggs and caterpillar metamorphoses into butterfly, while in our homes we slice apples and tomatoes, fold clothes and build up chairs from pieces of wood (Figure 1). Understanding and tracking these transformations is important for modern vision systems. For instance, embodied agents like kitchen robots need to understand object pre- and post-conditions (such the locations of sliced pieces of apples) to ground actions [22]. As another example, wildlife monitoring systems must recognize and keep track of the the butterflies emerging out their chrysalis to keep tabs on the insect population. More generally, understanding and tracking object transformations can improve capabilities in action-grounding [36, 50], video editing [24], and scene modeling for augmented reality [56].
With these motivations in mind, we seek a system that, given a video and a prompt specifying particular object, maps out how the object evolves over time, detects state changes, and tracks the resulting objects of these changes. We call this problem Track Any State. We observe that this is strictly harder problem than object tracking on the one hand (which does not care about object transformations) and recognizing state change on the other (which does not track the change pre- and post-conditions in space and time). Combining tracking with state change produces a more complete representation that is useful for downstream tasks (Figure 1, bottom).
However, even the simpler problem of tracking objects through state transformations challenging for existing methods. Object trackers of all kinds (be they based on template matching [18], optical flow [40], or supervised neural networks [35]) rely primarily objects appearance, assuming that they do not change drastically across time. However, as shown in Figure 1, transformations or state
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input Frame Prompt Object
Object Tracks
Prodicted Biato Graph
Appl sogment
Apple segment
Apple
Apple silce
Apple regreett
Apple
hele
Tulla
Tcht
Lode
Lufta
Object Tracking
Output by SAM2
Ouiput by Ours
Ourput by GPT 4.1
happened video?
A:A busterityy smerged from chrypale
Comp Mel
Macnin
Na Plenscriing about Otpecl
Buturrly sheyow
Einrrge
Intect buitarlyy
No Dijjett
No Spartceerppeee
Gitandding
Inde
Emargedd buzlellly
Figure 1: (top) Given a video and a object mask as prompt, TubeletGraph tracks the object consistently, while building state graph for each detected transformation and its resulting effect. (bottom) Compared to existing object trackers (SAM2 [35]) or video Q&A systems (GPT-4 [1]). TubeletGraph predicts complete object tracks while providing spatiotemporal grounding for the transformation.
changes can alter object appearance significantly: e.g. from a red apple a pile white flesh pieces, or from a chrysalis to an empty chrysalis shell and a butterfly. These drastic changes cause existing trackers fail in the face of these transformations, precluding any understanding of the state change.
Intriguingly, we find that the errors caused by tracking through state change are typically one-sided when object appearance changes, the model is likely to predict the initial prompt object to be "missing", leading to false negatives. This observation offers an opportunity: if we can detect when these false negatives occur, can attempt both recover the missed object and understand the transformation that caused the error in the first place. To do this, we must answer two questions. First, when and where exponentially large search space among all pixels in the video to find the missing object? Second, how can we model the underlying transformations and resolve any object ambiguity after state change? For instance, how can we name the transition and resulting objects in Figure (bottom)?
recover the missed object? In particular, how do navigate the
We answer these two questions with TubeletGraph, novel zero-shot framework for tracking and understanding object transformations in videos. First, to recover the missed object, we propose a new representation that dramatically reduces the search space. This representation tracks every entity in the video from the first frame and initiates new tracks in intermediate frames wherever there are untracked pixels. This produces soup of "tubelets". Finding the missing post-condition object of transformation then boils down to finding the right tubelet. By reasoning jointly about the semantics and spatial proximity of each tubelet, we demonstrate effective recovery the missing object.
Second, we use the emergence of these new tubelets as a marker for when state transformations happen. We then query existing multi-modal LLMs [1] to describe the transformation and the resulting objects natural language to produce a corresponding state graph. Together with the tracked tubelets, we build a complete representation of the object's evolution over time. An example TubeletGraph's output is shown Figure (top). In sum, our contributions are:
(1) We introduce Track Any State: a task of tracking objects through transformations while detecting and describing state changes, accompanied by VOST-TAS, new benchmark dataset.
(2) We propose TubeletGraph: a zero-shot framework that recovers missing objects post- transformation by using a spatiotemporal partition of the video and constructs state graph to detect and describe the underlying transformations.
We demonstrate both state-of-the-an tracking performance under transformations as well as effective detection and description of the transformation itself.

---

2 Related Works
2.1 Object Tracking
Object Tracking [37] aims to segment target object in a given video. Similar to Semi-supervised Video Object Segmentation (VOS) [46, 31, 32], the target object is specified via mask in the initial frame. In addition, recent benchmarks are proposed address more challenging scenarios, including long videos [16, 21J, crowds and occlusions [10, 13], and object motions [9, 13).
Benchmarks.
Methods. To predict consistent object tracks, prior works have mostly relied on appearance similari- ties via online feature finetuning [3, 4, 27]. template matching [17, 43, 49], or attention- memory reading [7. 28, 29, 51, 52, 6]. Recently, SAM2 [35] was proposed extend Segment Anything Model (SAM) [19] for interactive video segmentation. By incorporating memory- -tttetttttttniion mecha- nisms, SAM2 enables object tracking by establishing consistent temporal object correspondences. SAM2Long [11] extends SAM2 and addresses error accumulation in long videos by maintaining multiple candidate tracks in constrained tree search. Also, SAMURAI [48] introduces motion-based memory selections to handle crowded scenes with fast-moving or
objects. In addition, DAM4SAM [42] introduces a distractor-resolling memory to handle visually similar distractors.
While SAM2 and its variants demonstrate impressive results, they struggle when object appearance changes due transformation. In our work, first identify and reason about objects that are origi- nally missed by SAM2 due to their transformations. Upon their successfully retrieval, TubcletGraph proceeds to leverage them as markers for event boundaries [55] construct state graph describing the transformations that cause the false negative errors as well as the recovered object themselves.
2.2 Understanding Object Transformations
Understanding object transformations in videos has been well-studied. VOST [41] and VSCOS [53] propose to focus on object transformations from human actions in ego-centric datasets [8, 14). Similarly, S [5] extends the focus to objects undergoing phase (gas/iquiddsolidd) transitions. By assuming that object disorder increases through transformations, Re-VOS [5] propose to combine forward and reverse memory improve object tracking through transformations.
Beyond object tracking, DTTO [45] provides box-level annotations for transforming objects while How Too'nne [47] focuses open-worte world localization of three stages (initial, transitioning, and end states) of object transformation. Also, Where ToChanee [26] annotates
object state changes with the actionable and transformed object regions. For How ToChone [47] and Where Tohhane [26], pseudo-labels are generated from off-the-shelf vision-language systems to train video model for the respective task. Building upon the maaiiall-progresssiie state change segmentation maps, SPARTA [25] demonstrates real-world robotic manipulation capabilities such as spreading, mashing. and slicing.
In comparison, we focus on Track Any State, simultaneously tracking objects through transformations
and detecting and naming the transformation.
2.3 Vision and Language
Recently, mudti-modal systems have been proposed to integrate vision and language to understand and predict across modalities. CLIP [34] learns visual concepts from natural language captions via contrastive learning. From a shared embedding space for image and text, it enables zero-shot transfer to downstream tasks. FC-CLIP [54] further demonstrates this capability by predicting open-vocabulary segmentation using a frozen CLIP backbones. Finally, multi-modal LLMs such as GPT-4 [1] and Gemini [39] can reason about the visual/textaal queries and generate natural language responses to further aid down- strn-mtream tasks [23, 12, 44, 15, 38].
In our work, we leverage CLIP to semantically reason about candidate spatiotemporal tubelets. Furthermore, by prompting GPT-4 [1] with the retrieved candidates, TubeletGraph constructs a state graph by parsing the description of the transformation and the resulting transformed objects.
3

---

3 Method
3.1 Task Formulation
In this paper, we propose the problem of Track Any State: tracking objects through transformations while detecting and naming the transformation.
Concretely, the input is video V {1+} and binary mask Mi in frame /1 the initial object prompt. The output two-fold:
(1) A collection of tracks T [T' y.... J where each track 7 corresponds a mask M; each time step L. We allow for a collection of tracks rather than a single track because when objects undergo state change, they may break up into multiple independent parts; all of these are supposed to be tracked. Thus, T should track all segments that were created from the original object.
(2) A collection of state changes S where each state change 8 E S is represented by a tuple (t, Tpre, Tpost D). Here is the time step where the change happened, Tpre is the set of tracks involved before the change, Tpoa is the set of tracks involved after the change and D is a description of the change.
This output can be visualized as in Figure 1, where each track in T is visualized as masks of a specific color, and the set of state changes S are visualized as graph over the color-coded tracks.
Overview of approach: We now describe our approach, which we call TubeletGraph. Briefly, our approach first partitions the video into set of tubelets P, which are partial tracks by SAM2 and as such are delimited by appearance changes (Figure 2, top) We then use notions of spatial and semantic proximity middle-left). For each track that gets added, we prompt a vision-language model to name the state change and the pre- and post-effects (Figure 2, middle-right). The end result is consistent object tracks through transformation and a state graph that describes the underlying transformation and resulting objects in natural language (Figure 2, bottom).
the initial object prompt to decide which tracks to include in T (Figure
We now describe each step of this pipeline in detail.
3.2 Partitioning the Video into Tubelets
When objects undergo transformations, existing methods like SAM2 [35] often fail because (1) appearance information is no longer reliable when the object transforms, and (2) the assumption that the target object remains as a singular connected component no longer holds when the object fragments decomposes.
As result, these limitations often manifest in false negative errors. In the example of "taking sheet of foil out of the foil box" (Figure 2), the appearance and geometry of the foil box object can change drastically when a sheet of foil separates from the box. If we only track the foil box (denoted in pink contour) from the first frame, the foil sheet (an additional connected component with minimal appearance similarity) will be ignored in later frames.
To retrieve missing tracks like this and capture the full transformation of the object, we construct a spatiotemporal partition of the video P (Figure 2, top) to drastically reduce the search space. We first adopt an entity segmentation model, CropFormer (CF) [33], to obtain a complete spatial partition & of the initial frame Z!.
ε U (Mi}
where & represents the set of entity masks (including the object prompt MM1) ) frame I Then, we track each entity ef E & via SAM2 [35] to obtain a pool of tubelets Pun = {P.) where each tubelet P. = {<: represents the evolution of entity ef E E in the given video.
As one temporally progresses in the video, there will naturally be track-less regions where none of the initial tracked entities in Pinit are present. Thus Pini is an incomplete spatiotemporal partition of the video. To complete this partition, initialize the spatiotemporal partition P with Puni and
Please refer to Appendix A.3 resolving overlaps between the automatically segmented entities and Mi.

---

tpst Phanoi
Br
Prateggi-3a
Caryhara
Sobiin Kem Pork
Emply Regiondgrdde
New Tradi
Credy Regonegia
New Tracog
Track-a
GAME
SAME
SANE
Tinands
Time-Se
Tinende
Tou Frane Piuest Oajact
Theaa desc
Ny
GNORE-
scoLonn contoul
Thans obocrioo diceda
New Tww.haagg
cediaa (COLOR)
Pisuarity
Thone happon
410G 2+
CNIORE
BMG
09
@
Milanian fell See
Crasl was "alasiem Call akaer
Plonet Ga-3s
Odgect
Unnoll
Aluminum foil
Figure 2: Overview the proposed TubeletGraph. (1) Given a video and an initial prompt object mask, first partition the initial frame via CropFormer (CF) [33] and track every region forward in time via SAM2 [35]. For each empty region at later frame, we initiate a new track if an entity at that frame match with it. In the end, would obtain spatiotemporal partition of the video. (2) For each later-emerged entity region, we reason about its proximity and semantic consistency with the prompt object and only recover regions that satisfy both. (3) For each recovered region, we prompt multi-modal LLMs to describe the transformation and resulting objects. (4) From this, TubeletGraph achieves consistent tracking of transformation objects while mapping every transformation and resulting regions in a state graph representation.
incrementally add new tubelets to P by initializing a track whenever track-less regions emerge. Con- cretely, iterate through the frames and in every frame f > 1, we compute the entity segmentation & = CF.)). For each entity Eg &,, initiate a tubelet if less than Tooverage of its is covered by an existing tubelet. This new tubelet P' starting from entity èi is then added to P. This process ensures that the tubelets in P cover almost all of the pixels in the video.
The benefits the spatiotemporal partition are three-fold: (1) forces every region in the video to be associated with some partition tubelet, maximizing the likelihood of object retrieval. (2) It reduces the complexity of the searching problem by reformulating continuous problem "where the missing object in each frame" to simpler discrete problem of "which partition tubelet is a real missing object." (3) narrows down the set candidate tubelets to only the ones added after initial frame, since all initial tubelets in P that not the prompt can be immediately rejected.
5

---

3.3 Reasoning about New Candidate Entities
While P contains all tubelets that emerge after the initial frame, not every entity track discovered in a later frame is real missing object. They can be new objects introduced in later frames, existing objects that are uuder-segmentt object's s state change that we wish to recover. Thus, we need way to identify the latter from other irrelevant regions.
in the initial entity segmentation, or missing products of the target
Here, we make two assumptions about object transformations in the real-world: (1) An object's location does not change drastically in short period of time (e.g. an emerging butterfly is near its chrysalis), and (2) an object's identity and semantics cannot be significantly altered by transformations chrysalis can turn into butterfly, but not a bird).
From this, we define two requirements that a candidate tubelet in P must satisfy to be considered as a missing object: spatial proximity and semantic consistency.
Spatial Proximity, By assuming temporally smooth object motions, the tubelets that were initiated near the prompt object track are more likely to be genuine missed objects. To estimate the spatial region where the transformed object might located, we leverage the multiple candidate masks predicted by SAM2, These multiple masks {ml {mr}} )= originally intended to capture ambiguity the user prompts, can also capture the ambiguity of prompt object segmentation during transformation. For a candidate track C = {ca, Cs+1,.... .... cr } that begins at frame s, and the prompt object track P {p1. Pp...,...... pr). compute the following spatial proximity measure:
Sprox P)
max Ic. 7 |csl (E(12)33
where (mg)}- corresponds to the three candidate masks of prediction ps- Intuitively, Sprox captures the maximum overlap of the candidate track C with any of {m]}3 }==1 at the frame where the candidate first appear. We consider a candidate proximal if (C, >
Tprox-
Semantic Consistency. While the proximity prior eliminates candidate tracks that do not initiate nearby the prompt object, it is not sufficient by itself.
Consider the case of pulling out a sheet of foil (Figure 2, middlc-left). The hands emerge in view holding the foil, but they should not be considered as the prompt object due to clear inconsistent semantics. To model this, we introduce semantic consistency prior that assumes semantic alignment between candidate entity and the prompt object. For given mask M and frame J, we compute the masked CLIP [34] feature I PooI(CLIP(I), M) via mask-pooling [54].
For candidate track C {ca, C,,,,..,, CT) that begins frame s, and the prompt object track P = (P1, ...,,,,... Pr we compute the semantic similarity as:
max Ssem (sm PC (E. 11).EE f(p, 4) f(c,,1))
(3)
Syem captures the maximum pairwise similarity between the prompt track (prior to the candidate's emergence) and any mask in the candidate track. We consider a candidate semantically consistent if
Syem C, P) >
Tsem-
Reasoning with Constraints. By combining these two prior constraints, we only recover candidate tracks that are both semantically consistent and spatially proximal the prompt object. As illustrated in Figure 2 (middle-leftt, this successfully removes false-positive candidates (e.g. cooking utensils and actor's hands) while retaining the genuine candidate (e.g. foil sheet).
Formally, we predict the set of valid continuation tracks as:
y = {C P C' begins at t 0, Spaix (C, Tprox and Syem (C,P) T
(4)
By combining V with the original prompt track P, we form the final tracking result T that successfully captures the complete prompt object through transformation.
3.4 Understanding Object Transformation
After recovering all candidate tracks that satisfy the two constraints, we leverage their emergence as indicators for when state transformation have occurred. For each valid continuation track C' e V that

---

Table 1: Object Tracking Performance on VOST [41] validation set. We compare multiple variants of Tubp against base SAM2.1 and SAM2.1 (ft), which finetuned on VOST train split. ST, S, and P indicate spatiotemporal partition, semantic consistent constraint, and spatial proximity constraint, respectively. J and Jrr measure tracking performance for the entire and last 25% of the video, while P and R measure per-pixel precision and recall.
Method
ST S
P
J
J
J
J
P
R
Jtr
J
Jhr
T6
Pir
Rtr
SAM2.1 (ft) SAM2.1
54.4
46.2
53.8
73.1
70.9
65.5
36,4
25.7
35.1
61.3
53.2
45.4
48.4
40.7
50.0
63.0
71.3
54.5
32,4
21.2
34.3
54.2
58.5
34.7
/
25.7
22.4
27.2
30.6
18.6
71.5
13,9
9.8
14.1
22.8
10.8
59.4
Ours (SAM2.1)
/
/
49.2
39.7
52.6
64.7
63.7
64.8
35.8
23.5
39.9
56.7
48.6
49.3
V
50.7
41.3
53.2
67.5
67.7
63.8
36.6
23.6
40.5
59.1
54.9
47.1
50.9
41.3
53.0
68.6
68.1
63.7
36.7
23.6
40.2
60.1
55.2
47.0
begins frame s, we wish know what transformation occurred and what are the resulting objects. As shown in Figure (middle-right), we draw contours on the initial frame 4 and frame I, and query multi-modal LLMs to provide brief description for the transformation and object identity.
After parsing the natural language outputs, we construct the state graph as shown in Figure (bottom). This state graph provides rich, structured representation of object transformations throughout the video, beyond consistently tracking the prompt object through transformation.
4 Experiments
Datasets. VOST [41] is curated from ego-centric videos in Ego4D [14] and EPIC-Kitchens [8]
that contain object transformations from actor-object interactions. The validation set contains 70
videos with an average of 22.3 seconds captured 60 fps, with 1 object masklets annotated at 5 fps. VSCOS [53] is constructed in a similar fashion from EPIC-Kitchent [8]. Its validation set contains videos with an average of 7.5 seconds captured at 60 fps and object mask annotated at fps. M8-VOS [5] models object phase changes and contains limited camera motion due to its source from online videos. The entire dataset serves as evaluation, containing 479 videos, 526 masklets, with an average of 14.3 seconds captured at 30 fps. Also, we evaluate DAVIS 2017 [32] to confirm tracking performance for objects that are not undergoing transformations.
VOST-TAS (Track Any State): We introduce a new benchmark for evaluating the proposed task by manually annotating transformations in the VOST [41] val set. Each object instance includes list of transformations with temporal boundaries (start/end frames), action verb descriptions, and list of resulting objects with segmentation masks and text descriptions on the end frame per transformation. In total, it contains 57 video instances, 108 transformations, and 293 annotated resulting objects."
Implementation Details. For TubeletGraph, we adopt SAM2.1-L [35], CropFormer-Hornce 3X [33], FC-CLIP-COCO [54]. Hyperparameters for all three models kept as default and not tuned further. In addition, we adopt GPT-4.1 [1] and keep sampling temperature at O. To reason about new candidate entities, we select Tprox 0.3 and Teem 0.7 after sweeping intervals of 0.1 on VOST train split that is similar sized as VOST val and applied to other datasets without any further modification. In addition, we arbitrarily ignore any entities smaller than 1/25² of the video frame and set the coverage threshold for initiating new tracks Teneraage 0.25 without further tuning.
4.1 Object Tracking
To measure object tracking performance, we follow VOST [41] and report Jaccard I and Jir (only over last 25% frames), along with per-pixel precision P and recall R. For a more fine-grain analysis, divide each dataset into three equal subsets: small (S), medium (M), and large (L), based on the average object size throughout the video.
Pllesse refer to Appendix A.I for details regarding VOST-TAS construction and visualization.
7

---

Table Tracking Performance on VOST [41], VSCOS [53], - [5], and DAVIS17 [32]. J and J.r measure tracking performance for the entire and last 25% of the video, respectively. Best performance bolded and second bests are underlined.
Detects
VOST val
VSCOS val
-O-VOS val
DAVISI7 val
Method
Describes Changes
J Jer
Jer
J
Jer
Jer
XMem [7]
36.1 24.7
69.9
64.6
69.7
60.1
82.9
81.0
Cutie [6]
41.1 25.5
70.9
67.1
74.5
64.7
84.6
82.0
ReVOS [5]
X
41.0 25.3
75.6
66.5
86.0
84.8
SAM2 [35]
X
46.1 29.4
72.5
67.1
71.3
59.8
85.5
82.3
SAM2Long [1]]
46.4 29.1
73.0
68.6
70.2
58.7
87.1
85.5
SAM2.1 [35]
X
48.4 32.4
72.0
66.9
71.3
59.3
85.7
83.0
DAM4SAM [42]
48.8 33.6
71.3
66.0
72.2
61.3
86.2
84.2
SAMURAI [48]
49.8 34.0
71.8
66.9
72.6
61.6
85.6
82.7
Ours
V
50.9
36.7
75.9
72.2
74.1
64.1
85.6
82.6
Analysis on VOST. Table presents the comparison between variants of TubeletGraph, the base SAM2 model, and SAM2 fintuned on the VOST training set. The top half of Table first demonstrates an imbalanced error distribution for base SAM2: while precision P remains at over 70%, recall R. languishes below 55%. This gap indicates that false negative errors false positives when tracking transforming objects, further confirming our observation that appearance- driven trackers struggles primarily with missed tracks than wrong tracks. As expected, finetuning SAM2 on the VOST yields substantial improvements across the board, with notable increase in recall 54.5 65.6 while maintaining precision. While finetuning shows clear benefits, it is limited by the extensive annotation cost for each specific transformation domain which reduces generalizability. In contrast, TubeletGraph is training-free, offering zero-shot capabilities and improved generalization.
than twice frequent as
The bottom half of Table demonstrates the effectiveness of TubeletGraph. First, the proposed spatiotemporal partition is effective in providing candidate objects for retrieval. If every later-emerged object from the partition is incorporated into the prediction, the recall would greatly surpass that of the finetuned SAM2 model (+6 for R and +4 for R.y)). However, expected, this aggressive recovery comes at a cost of significant precision reduction (-52.7 for P and 477.7 for Ptr).
By introducing the proposed semantic consistency and spatial proximity constraints, we can improve this precision-rectt tradeoff. Notably, we are able to improve precision (+-49.5 for P and +44.4 for Ptr)) while minimizing reduction in recall (-7.8 for R and -12.4 for R2y). While semantic prior brings marginal improvement when proximity prior is already considered, the consistent gain suggests its necessity (e.g., rejecting false positive entities that are close the tracked object).
As result, TubeletGraph is able improve J by 2.5 points from the base SAM2 while surpassing the finetuned SAM2 in J... Finally, we obtain p-values of 0.014 for I and 0.013 for Ter from paired I-test between the base SAM2.1 and TubeletGraph, giving statistical significance to our improvement.
Main Results. Table 2 showcases a comprehensive comparison of tracking performance between our TubeletGraph and state-of-the-art baselines across four VOS benchmarks datasets Notably, TubeletGraph is the only method capable of not only tracking objects under transformations but also detecting and describing these state changes. Along with this additional capability, TubeletGraph achieves state-of-theeart performance on both VOST and VSCOS datasets, both focusing on trans- forming objects in ego-centric domains. When evaluated on M*-VOS. TubeletGraph outperforms all SAM-based variants and achieves results comparable to the best performing ReVOS Finally, we measure all method performances on DAVIS17. Encouragingly, TubeletGraph performs comparably to all baselines, indicating that approach of adding new tracks induces minimal false positives when tracking objects without transformations.
Qualitative Results. Figure 3 showcases our proposed system. Compared to prior works that miss object components due to transformations, TubeletGraph recovers the missing objects and leverage them as markers to describe the underlying transformation that caused the false negatives.
Complete tracking results for VSCOS [53] and M [5] are found in Appendix A.4

---

Ipput Prampt diject
Object Tracks
Plui reodUrng
Masbo Madooapoor.
plodii
Prodicted Graph
accerated
Prompi OFjecce
Predicted Tradks
tnba Out
1xi Cut
Cuatt Seserata
Trata Size
Besooo Soors teg
Cur
Brnsoo
Prodicted State Groph
Troozd
Brooooli Forel place
Broccoll sioce
Forst
Figure 3: Qualitative Results on VOST val. We showcase TubeletGraph's 's tracking and state graph predictions on top, with comparisons against baselines particular ending frame at the bottom.
4.2 Transformation State Graph
To evaluate state graph quality, we report precision Tp and recall TR for temporal localization within annotated transformation boundaries, and description accuracy for correctly localized action verbs (Av) and resulting objects (Ao) with IoU > 0.5. Finally, we combine these into two metries: spatiotemporal recall Hsr (correctly detects transformation within boundaries and finds all objects with loU > 0.5) and overall recall H. (additionally requiring correct action and object descriptions).
Temporal Localization. We first report precision Tp and recall Tn for temporal localization. As shown in Table 3, TubeletGraph achieves Tp = 43.1 and Ta. 20.4 on VOST-TAS. While the precision is moderate, the relatively low recall indicates that many ground truth transformations are not detected within the annotated temporal boundaries. This stems from the passive detection of trans- formations, as they are only triggered when false-negative object is recovered. transformations that do not alter object appearances, accurate tracking would prevent transformation detection.
Semantic Accuracy. We then evaluate the semantic quality of predicted action verbs (Sv:) for transformations that are correctly localized temporally, and resulting object descriptions (So) for objects that are matched with loU >> 0.5. As shown in Table 3, TubelerGraph achieves Sy = 81.8 for action verbs and So 72.3 for resulting objects, demonstrating accurate description of the VLM-based reasoning module. Finally, since Sy and So are computed only on successful matches, they represent the description quality conditional on successful temporal/spatial localization.
More details regarding metric computations found in Appendix A.2.

---

Table 3: Object tracking ablation on VOST [41] and state graph results on VOST-TAS. J and Jtr measure tracking performance, Sy and So measure semantic accuracy of the state graph. Tr and Tr measure the precision and recall for temporal localization, while Hst and H. measures the combined transformation recall. Sy and So are shown in gray as the relative accuracies vary across methods.
Tracking
Sem. Ace.
Temp. Loc.
Overall
Entity
Tubelet Track
Semantic Sim.
VLM
J
Je
Sv
So
Tp
Tr
Hsr
H
SAM [19]
49.2 34.7
81.0
72.9
36.8
19.4
10.2
0.9
Cutie [6]
47.6 33,9
91.7
80.6
32.4
11.1
6.5
2.8
DINOv2 [30]
50,9 36.6
76.2
77.1
42.0
19.4
12.0
5.6
Qwen [2]
50.9 36.7
31.8
44.6
43.1
20.4
12.0
1.9
CF [33]
SAM2.1 [35]
CLIP [54]
GPT4.1 [1]
50.9 36.7
81.8 72.3
43.1
20.4
12.0
6.5
Finally, compute spatiotemporal recall Her (correct temporal localiza- tion with every resulting object matched with loU > 0.5) and overall recall H (additionally requiring all correct semantic descriptions). As shown in Table 3, TubeletGraph achieves Hsr 12.0 and H = 6.5. This reflects the significant difficulty of transformation prediction unconstrained ego- centric videos. As the first approach to jointly tackle object tracking and state graph prediction, these results establish baseline for Track Any State and highlight clear directions for future work.
Overall Performance.
4.3 System Analysis and Discussion
Robustness. We first systematically ablate each component by while keeping other modules fixed (Table 3). When replacing CropFormer [33] with SAM automasks [19] for entity detection, tracking performance reduces by 1.7 in J. which is mainly attributed to SAM being less reliable for small objects (Table 4). Replacing SAM2. [35] with Cutie [6] for tubelet propagation results in a more significant degradation (-3.3 in J and -9.3 in TR). indicating the importance of accurate tubelet tracking. For semantic filtering, swapping CLIP [54] with DINOv2 [30] yields comparable tracking performance. Finally, replacing GPT-4.1 [1] with Qwen-2.5VL [2] dramatically impacts semantic accuracy, demonstrating high-quality VLM reasoning is critical for accurate semantic descriptions.
Additionally, we find TubeletGraph to be highly robust to the filtering thresholds Tprox and Tsem- On Mo-VOS and VSCOS, we obtain a robust range of (72.6, 74.2) and (75.1, 75.9) for J, respectively, after sweeping Fprox between 0.1 and 0.5 and Tsem between 0.5 and 0.9 in intervals of 0.1 (Table 6).
Computational Efficiency. The main efficiency bottleneck of TubeletGraph is constructing a spatiotemporal partition by tracking every spatial region, which costs on average 7 seconds per frame on VOST [41] with one NVIDIA RTX A6000 GPU. Although limiting real-time applications, TubeletGraph' 's unique capabilities to track objects while detecting and describing transformations can be very useful; e.g., producing training annotations on recorded demonstrations for robots, analyzing compliance videos on the factory floor, understanding animal developments from camera traps. In these applications, understanding and tracking object transformations is critical, and real-time is not needed. Finally, the spatiotemporal partition can be adapted to multi-object tracking with little-to-no additional cost, amortizing the computational time when tracking multiple objects simultancously.
5 Conclusion
In this work, introduce the problem of Track Any State, tracking objects through transformations while detecting and describing the transformation. We proposed TubeletGraph, a zero-shot system that recovers missing objects after transformation and leverage them "landmarks" reason and describe them. Our approach achieves state-of-the-ar tracking performance under transformation while demonstrating promising capabilities in spatiotemporal grounding of object transformations.
Limitations and Broader impacts, Beyond high computational cost, the modular design of Tubelet- Graph may pose potential challenges for systematic error attribution and diagnosis. Finally, our work does not introduce any foreseeable societal impacts, but will generally promote more robust and informative tracking systems for robotics and general vision systems."
5"
$ Pleasse refer to Appendix A.5 for additional discussions on error analysis and broader impacts.
10

---

Acknowledgement
This research is based upon work supported part by the National Science Foundation (IS--214111, IIS-2107161 and IIS-2505098). Yihong Sun is supported by an NSF graduate research fellowship.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2. 3, 7, 10, 16, 27
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 139233302..3323, 2025. 10, 18
[3] Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Luc Van Gool, and Radu Timofte. Learning what to learn for video object segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 777-794. Springer, 2020. 3
[4] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 221-230, 2017. 3
[5] Zixuan Chen, Jiaxin Li, Liming Tan, Yejie Guo, Junxuan Liang, Cewu Lu, and Yong-Lu Li. - Multi-phase,
and multi-scenery video object segmentation. arXiv preprint arXiv:2412.13800 2024. 3, 7, 18
[6] Ho Kei Cheng. Scoung Wug Oh, Brian Price, Joon- Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151-3161, 2024. 8.
[7] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an Lokinson-shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022. 3, 8
[8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for saie-kitchens-100 100. International Journal of Computer Vision, pages 1 1-23, 2022. 3,7
[9] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2694 27033, 2023.
[10] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF international conference computer vision, pages 20224-20234, 2023.
[11] Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang. Yuhang Zang. Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam for long video segmentation with training-free memory tree. arXiv preprint arXiv:2410. 16268, 2024.
[12] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, al. Palm-e: embodied multimodal language model, 2023.
[13] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374-5383, 2019. 3

---

[14] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, et al. EgoAd: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022. 3,7
[15] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14953-14962, 2023. 3
[16) Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo. Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13480-13492, 2023. 3
[17] Yuan-Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Videomatch: Matching based video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 54-70, 2018.
[18] Frédéric Jurie and Michel Dhome. A simple and efficient template matching algorithm. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 544-549. IEEE, 2001.
[19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026,
2023. 3. 10
[20] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97 97, 1955. 15, 16
[21] Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. Advances in Neural Information Processing Systems, 33:3430-3444 3-4401 2020.
[22] Weiyu Liu, Neil Nie, Ruohan Zhang, Jiayuan Mao, and Jiajun Wu. Blade: Learning composi- tional behaviors from demonstration and language. In Conference on Robot Learning (CoRL), 2024.7
[23] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-We Chang, Ying Nian Wu, Song- Chng-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 46:43447-434777 2023. 3
[24] Shao-Ping Lu, Song-Hai Zhang, Jin Wei, Shi-Min Hu, and Ralph R Martin. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics, 19(7):1218-122) 2012.
[25] Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Marthn and Kristen Grauman. Mash, spread, slice! learning to manipulate object states via visual spatial progress. arXiv preprint arXiv:2509 299 2025. 3
[26] Priyanka Mandikal, Tushar Nagarajan, Alex Stoken, Zihui Xue, and Kristen Grauman. Spoc: Sproially object state change segmentation in video, 2025. 3
[27] K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal information. IEEE transactions on pattern analysis and machine intelligence, 41(6):1515-1530 2018. 3
(28) Seoung Wug Oh, Joon- Young Lee, Kalyan Sunkavalli, and Seon Joo Kim. Fast video object segmentation by reference- guidedd mask propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7376-7385, 2018. 3
[29] Seoung Wug Oh, Joon- using space-time memory networks. In Proceedings of the IEFF/CVF international conference on computer vision, pages
Joon-Young Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation
2019. 3
12

---

[30] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/ 07/33, 2023. 10
[31] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Sorkine-Homung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
2016.
[32] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint vr04v 00675, 2017. 7.7.8
[33] Lu Qi. Jason Kuen, Weidong Guo, Tiancheng Shen, Jiuxiang Gu, Jiaya Jia. Zhe Lin, and Ming-Hsuar Yang. High-quality entity segmentation. arXiv preprint arXiv:2211.05776. 2022. 4, 5, 7, 10, 17
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021. 3,6
[35] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000 00714, 2024. 2, 3, 4, 5, 8, 10, 19
[36] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36, 2013.
[37] Giorgio Roffo, Simone Melzi, et al. The visual object tracking vot2016 challenge results. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II, pages 777-823. Springer International Publishing, 2016.
[38] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF international conference computer vision, pages 11888-11898, 2023. 3
[39] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055330 2024.
[40] Zachary Teed and Jia Deng. Raft: Recurrent all- field transforms for optical flow. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part 11 16, pages 102-419. Springer, 2020.
[41] Pavel Tokmakov, Jie Li, and Adrien Gaidon. Breaking the" object" in video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages
2023. 8, 10, 15, 17
[42] Jovana Videnovie, Alan Lukezic, and Matej Kristan. A distractor-aware memory for visual object tracking with sam2. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025. 3,
[43] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang- Chieh Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 9481-94900 2019.
[44] Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 1622:1 2023.
13

---

[45] You Wu, Yuelong Wang, Yaxin Liao, Fuliang Wu, Hengzhou Ye, and Shuiwang Li. Tracking transforming objects: A benchmark. Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 222-236. Springer, 2024. 3
[46] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube- vos: Sequence-to-ssquenee video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 585-601, 2018.
[47] Zihui Xue, Kumar Ashutosh, and Kristen Grauman. Learning object state changes in videos:
An open-world perspective, 2024. 3
[48] Cheng- Yen Yang, Hsiang- Ssiang-WWi Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory. arXiv preprint arXiv:2411.11922 2024. 3.8
[49] Linjie Yang. Yanran Wang, Xuehan Xiong. Jianchao Yang, and Aggelos K Katsaggelos. Efficient video object segmentation via network modulation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6499-6507, 2018.
[50] Zhengyuan Yang. Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. Grounding- tracking-integration. IEEE Transactions Circuits and Systems for Video Technology, 31(9):33333444 3443, 2020.
[51] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by breeroouddbbccgroode integration. In European Conference on Computer Vision, pages 332- 348. Springer, 2020. 3
[52] Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. Advances Neural Information Processing Systems, 34:2491-2502, 2021.
3
[53] Jiangwei Yu, Xiang Li, Xinran Zhao, Hongming Zhang, and Yu-Xie Yu-Xiong Wang. Video state- changing object segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14439-20448, 2023. 7, 18
[54] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chies Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215-32234, 2023. 3, 10
[55] Jeffrey M Zacks and Khena Swallow. Event segmentation. Current directions in psychological science, 16(2):80-88) 84, 2007. 3
[56) Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and -iuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825 2024.
14

---

A Appendix
A.1 VOST-TAS Dataset
Dataset Overview. We introduce VOST-TAS (TrackAnyStatee), an extended version of the VOST validation set [41] with explicit transformation annotations. Given video sequence V= { where I denotes the frame at time t. we annotate each temporal segments corresponding to object state transformations.
total, VOST-TAS contains 57 video instances, 108 transformations, and 293 annotated resulting objects. Qualitative examples are provided in Figure 4 and the full dataset is available at ttaps:///gthhen com/ www.coom///i/.. coom/YhhongSnn http//ogggnnuueeleeeeeeeeee TubeletGraph.
Dataset Details. For each video instance, we manually label corresponding annotation A = (fstums, fend, r). Here, faan denotes the initial annotation frame; fend [0, 7] denotes the terminal annotation frame; and r = { M containing the set of N transformations.
In addition, each transformation Ti is formally represented 12,4% 49 E [lsturt, fend] define the temporal start/end boundaries of the transformation; ey contains the free-text descriptions of the transformation; and O; {(Ma,, d objects, where Mi denotes the segmentation mask and di,j the textual description, both annotated at frame i.
tuple as 7i (1,, (4, vs, O.). Here,
K; is the set of K; resulting
Annotation Protocol.
We employ the following criteria to ensure consistency and quality:
Physical Separability: Resulting objects are considered distinct if they are physically separable, even if visually contiguous and semantically identical.
(2) Diversity Construint: allow diversity in the transformations, annotation process terminates when an action-object pair (v, o) occurs more than three times in row. Similarly, duplicated objects in the same video with identical descriptions and associated actions are excluded. The carly-stopped annotation would lead to terminal annotation frame tead < T.
(3) Quality Filtering: Transformations are excluded if the target object is not clearly visible during transformation or if the state change ambiguous.
A.2 Track Any State Evaluation Metrics
We evaluate state graph quality across two dimensions: temporal localization and semantic accuracy on the VOST-TAS Dataset.
Temporal Localization Metries. To assess the temporal localization of predicted transformations, we measure precision Tp and recall TR using bipartite matching between predicted timestamps and ground truth temporal ranges.
Given a video instance with ground truth annotation (tuan: tends I') and transformation intervals T = {(E) 1, vi N containing Ng transformations, we obtain all predicted transformation timestamps between tstt and lend, denoted as P = (1prod N ) From this, we construct a cost matrix C' E (0. N3x.N, where
0 if gn [1] 47]
Cjj
otherwise
We then apply the Hungarian algorithm [20] to find the optimal assignment minimizing total cost. Predictions matched with cost 0 are counted as true positives (TP), while unmatched predictions contribute false positives (FP) and unmatched ground truths to false negatives (FN). Precision Tp and recall TR are then computed as:
TP TP + FP'
TP
Tp
TA
(6)
TP FN
15

---

[Trame=0]
[rame=30]
[rrame446]
[frame=57]
Intedd Pronpe (rammeo))
Tonsetumtton 21 framsot8 20)
Tianeforraion #2; [rameed0 [0]
Innnformtion Ic (Tomme 110
anSon: <NOOE>
ocEor: remppe.geel.umiiinnsoo.. opart
arión remove poslunskinnzlle aport
aCSO1T renooveeeetuskkin..uu
Mi-1 -PROUPT>
Aij- banana
_ banana pasl
Mij-! banana
o0p benana peet
o3j1 banana
oii-2 banana pasl
di-S banona, pool
cgró berano pool
clg-d. banana, pool
[trama=0]
[rame:54]
[rram=112]]
[Trama:14]]
Propge (rammnn))
Trarolemaaiin on [Iurondd
Transformatin (Jaame60:
Tranoforation D1ame-118
acton: dIONES
ore aq senen,, draw paint
adion equaese www.sseddraa.eeeeo drapupaint
actot: squaede, dram paint
SPROMPT
troeting
abju4t cream, treating draniing
Figure 4: Examples of VOST-TAS.
Semantic Accuracy Metries. Beyond temporal localization, we also evaluate the semantic quality of predicted action verbs and resulting objects.
Action Verb Accuracy (Ay): For any predicted transformation that is correctly matched to ground
truth temporal boundary (i.e. ereed E [ ()), we assess whether the predicted action descriptions semantically match ground truth description using GPT-4. [1] with temperature 0 for deterministic evaluation. The model receives the system prompt:
"You are a highly intelligent assistant that can analyze actions text."
followed by the evaluation prompt:
"Given a particular action description '(G (GTTAACTION', similar to the verbs in this action? Please rate from -1 to 1, where -I means com- pletely unrelated, O means ambiguous, and means (PRED_ACTION) ACTION] captures the meaning of 'IGT ACTTONTION]" or is directly in it. Brief/general descriptions should still considered as +1. Please answer with single integer."
AATION
prediction considered correct if the model returns score of
Resulting Object Accuracy (Ao): For any predicted transformation that is correctly matched to ground truth temporal boundary (i.e. epred E [E] Ef), we perform Hungarian matching [20] on the IoU matrix between predicted masks and ground truth masks and collect all matched pairs with IoU > 0.5 for semantic evaluation. For each spatially matched objects, we evaluate description similarity using GPT-4.1 with system prompt:
" You are highly intelligent assistant that can analyze actions and resulting objects in text."
16

---

and the evaluation prompt:
"Given the object description "JGT_OBJEES OBJECTJ', is BRRE_OBBECTY' similar to it? Please rate from -1 to 1, where -1 means completely unrelated, 0 means ambiguous, and means '(PRED OBJECTJ' is similar. Over- or under-specifee seeeer-pecified descriptions should still be considered as +1. Please answer with single integer."
A prediction is considered correct if the model returns score of 1.
Combined Metries.
Finally, we define two holistic metrics combining temporal and semantic
evaluation:
Spatiotemporal Recall (Hsr)): For each ground truth transformation ;; = (t). fy. Us O₂) in video instance, prediction is considered spatiotemporal match if: (1) the predicted timestamp greed correctly matched within [t), 4 and (2) all K, ground truth resulting objects in O are
matched with predicted masks with IoU >> 0.5 at frame fe. From this, the spatiotemporal recall H.T is computed as:
HST of spatiotemporally matched transformations
# of ground truth transformations
Overall Recall (H): Building upon HST, a transformation is considered fully correct if it satisfies all spatiotemporal matching criteria and additionally: (I) the predicted action description achieves Av (semantic match with ground truth action), and (2) all spatially matched resulting objects achieve Ao = 1 (semantic match with ground truth object descriptions). Overall recall is computed as:
of fully correct transformations
H = # of ground truth transformations
(8)
A.3 Additional Implementation Details
Shown in Section 3.2, we compute the complete spatial partition & for the initial frame 4 as follows:
ε (M))
Naturally, combining the object prompt.N Mi with the set of masks predicted by CropFormer(CF) [33] trivial. Mi can overlap with subset of masks in CF(I,) at various degrees.
To resolve possible overlaps, we first denote the fraction of mask covered by mask as cover(a, b). Then, we introduce another coverage threshold Tremove (where Tremove Teoverage) to remove any entity ef CF(T1) with cover(ei, M1) 2 Tremove.
Coneretely, from the predicted entities CF(4) {ej, ei.... e prompt mask Mi, and coverage thresholds Teoverage and Fremove, we construct two subsets gkeep and E emodify from CF(41) as follows:
1. Keep as-is: For every predicted entity of E CF(1) with Mo) < include it in eleep without any modification.
Tcuverage- we
gkeep {ej ef CF(h), cover(e),.Mi) < Toovemge
Table 4: Object Tracking Ablation Results on VOST [41] validation set. T and Jer measure tracking performance for the entire and last 25% of the video, while P and R measure per-pixel precision and recall.
Entity
Tubelet Track
Semantic
M
P
R
Jur
J
J
J
Prr
R.r
SAM Automask
/
49.2
40.9
47.0
70.2
67.9
61.0
34.7
20.9
35.9
63.1
54.0
43.5
Cutie
47.6
40.8
66.9
59.0
22.3
31.8
53.6
41.0
DINOv2
50.9
41.3
52.4
69.2
68.3
63.2
36.6
23.6
39.4
60.8
55.4
46.5
SAM2.
CL.IP
50.9
41.3
53.0
68.6
68.1
63.7
36.7
23.6
40.2
60.1
55.2
47.0
17

---

Table Object Tracking Performance VSCOS [53] and M [5] validation We compare multiple variants of our model against base SAM2.1 and SAM2.1 (ft), which is finetuned on VOST train split. ST. S, and indicate spatiotemporal partition, semantic consistent constraint, and spatial proximity constraint, respectively. J and Jtr measure tracking performance for the entire and last 25% of the video, while P and R measure per-pixel precision and recall.
Method
ST P
J
J
JM
J
P
R
Jer
J
Jtr
Th.
Per
Rar
VSCOS
[53]
SAM2.1 (ft)
79.8
77.0
78.9
83,4
91.8
85.5
78.3
73.0
78.2
83.8
90.7
85.4
SAM2.1
72.0
61.7
76.7
77.9
90.0
76.6
66.9
53.5
72.1
75.4
88.8
71.6
/
42.5
32.9
47.6
47.2
44.9
86.5
38.2
29.0
42.5
43.4
38.6
86.4
Ours
V
75.5
68.1
78.0
80.4
87.1
84.0
71.5
61.5
75.5
77.6
85.2
82.8
(SAM2.1)
75.9
67.8
79.0
80.9
89.1
83.0
72.1
60.7
77.4
78.4
87.0
81.8
/
75.9
67.8
79.1
81.0
89.3
82.9
72.2
60.7
77.6
78.4
87.4
81.7
M [5]
SAM2.1 (ft)
74.4
67.6
79.3
78.3
88.5
79.6
64.8
57.1
70.4
69.3
82.5
71.5
SAM2.
71.3
66.8
74.7
73.9
89.7
75.1
59.3
54.4
62.0
62.8
84.8
63.5
60.9
48.8
63.1
74.5
66.6
84.0
51.4
38.1
53.8
66.4
58.6
79.3
Ours
/
72.5
65.5
76.2
77.9
85.0
81.0
62.5
54.3
66.1
69.4
77.8
73.3
(SAM2.1)
74.0
67.4
78.5
78.1
88.2
79.9
64.2
55.8
68.4
70.9
82.2
71.7
74.1
67.4
78.7
78.2
88.4
79.8
64.1
55.9
68.5
70.3
82.4
71.5
Table 6: Parameter sweep for semantic similarity threshold (Tsem) and proximity threshold (Tprox) for J. Best performance is bolded. The parameter setting tuned on the VOST train (Tsem 0.7,
Tprox 0.3) found underlined center grid.
M [5]
VSCOS [53]
Tprox
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Tem
0.5
74.1
74.1
74.0
74.0
74.0
75.5
75.8
75.9
75.8
75.8
Tem
0.6
74.1
74.1
74.0
74.1
74.0
75.5
75.8
75.9
75.8
75.8
Tsem
0.7
74.2
74.2
74.1
74.1
74.0
75.7
75.8
75.9
75.8
75.8
Tsem 0.8
73.7
73.6
73.6
73.6
73.6
75.5
75.6
75.7
75.7
75.7
Tsem
0.9
72.7
72.6
72.6
72.6
72.6
75.1
75.1
75.1
75.1
75.1
2. Modify and remove overlap: For every predicted entity of e CF(Li) with Teoverago S cover(et,
Tremoves only include the non-overlapping component of ef in gmidity (ei (ej MMM): et E CF(L1); Tooverage I cover(et, Tremove]
e emmdiiy
Thus, we obtain the final & = gkeep U & emodity U (Mi}.
A.4 Additional Evaluations
Please refer to Table for comparison results M and VSCOS. We observe largely consistent trends as found in Table 1, which underlines the generalizability of TubeletGraph. In addition, Table provides the full tracking results that are omitted in Table due to space constraints. Since the use of different VLM models does not impact tracking performance, the ablation with Qwen [2] is omitted. Finally, Table shows the full grid search over both spatial proximity and semantic consistency thresholds on V [5] and VSCOS [53]. The parameters tuned on the VOST train (7sem=0.7 0.7, 70.3= found in center grid) perform competitively across all datasets. This verifies that the hyperparameter selection generalizes well across datasets without requiring dataset-specific tuning.
18

---

This robustness further underlines the stability of our filtering mechanism without need for precise threshold calibration.
Inpurt Frame Promppt Objjcct
Prodicted Ooject Trackn
Alleds
Inpul Framee Pronppt Object
Predicted Object Tracks
Ints Sproad
Irlp Merga
Iely Mage
Egp mar
Pou
Datar on
Bater g
Beter atrears
spread out
Prodicted State Graph
Figure 5: Failure examples of TubeletGraph.
A.5 Additional Discussions.
Failure Examples and Error Analysis We first show failure examples of TubeletGraph Figure In the top example, the tape measure case is incorrectly identified a "smartphone," and the extended measuring tape is described as a "metal rod." Consequently, the transformation is described as "attach" rather than the correct "extend" or "pull out." This failure stems from the incomplete object views in the selected frames passed to the VLM, where hand occlusions prevent accurate object recognition and lead cascading error for the action description. In the bottom example, false positive omelet is incorporated into the tracking of an resulting object in the state-graph. Specifically, the object track for the correctly identified "batter stream" incorrectly laches to the irrelevant omelet later into the video. By assuming high precision underlying tracker, TubeletGraph fails remove this false positive error made by SAM2 [35]. Video failure examples can be found in https://tubelet- graph. aeraph..ithan github. ii..
More generally, found errors typically manifest (1) false positive predictions by the base tracker and (2) minor reduction in tracking recall when applying semantic and proximal constraints shown in Table 1. False positive emrrrs (1) can cause erroncous measures of semantic and proximal similarities, while reduction in tracking recall can reduce recall for temporal localization of object transformations.
Broader Impacts. TubeletGraph's ability to track objects through transformations and describe state changes has broad applications across multiple domains. In robotics, it enables learning from demonstration by automatically annotating object state changes in recorded manipulation tasks, reducing the manual annotation burden for training data collection. In scientific research, it facilitates the study of developmental processes (c.g., tracking metamorphosis in insects, growth of cell cultures) from video recordings where manual annotation would be prohibitively expensive.
As with any technology that analyzes visual data, risks arise when applied human behavior or in surveillance contexts. Understanding object transformations could potentially be misused to monitor individuals' activities in private settings without consent, or to enforce overly intrusive workplace surveillance that violates workers' privacy and dignity. In ego-centric applications particularly, the system processes first-person video that may inadvertently capture sensitive personal information or the activities of bystanders who have not consented being recorded.
19

---

Combining TubeletGraph with SAM2.1 (ft). We find that integrating (ft). with Tubelet- Graph (J 54.1) shows modest improvements over the base TubeletGraph (J 50.9). However, the improvement is smaller than expected, given the strong standalone performance of SAM2. We reason that because TubeletGraph specifically addresses false-negative predictions by incor- porating new candidate tracks lost due to object transformation. Since SAM2. (ft) is fine-tuned on VOST to minimize these false negatives, the complementary benefits are naturally reduced.
Effects of Occlusions on TubeletGraph. During occlusion events, TubeletGraph would generally add additional tubelets for the target object that re-emerges after being temporarily lost, this additional tubelet matches the semantic and proximity constraints, it will be incorporated in the tracked object. Since the proximity constraint relies on the candidate masks predicted by SAM2, the base tracker's internal candidate masks would set upperbound for the object recovery after occlusion. In terms of the transformation detection, since the VLM only observes the frames where the objects are visible, the transformation would not be described occlusion event.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce the task of Track Any State in Sec. 3.1, describe the method TubeletGraph in Sec. 3.2 to Sec. 3.4, verify our claim about false negative missed objects in Sec. 4.1, and demonstrate superior performance Tab. 1, Tab. and Tab.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sec. 4.3 and Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include all relevant implementation details for reproducibility in Sec. and Appendix A.I to Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and VOST-TAS are available at ahtt:p://ggihuub..o///loggggg TubeletGraph.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..
wwteimiissnoooliiiio onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublic//aeccccccccccccccc ggwpi/gg///////eeeeee /Ce///eeembbiiic cetsSuummisioni. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include relevant dataset information, hyperparameters and their selec- tions in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform paired t-test to confirm that the improvements with TubeletGraph is statistically significant in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include information on compute resources, time of execution, etc in Sec, 4.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tprr:////urips... cctpublic//eeeeccccccccc wtwteetetteiiieeeeeeeeeeeeeeeeee
Answer: [Yes]
Justification: Our research was conducted in complete alignment with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Sec. 5 and Appendix A.5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no risks for misuse: we present a training-frit free method without model weights and extend existing benchmarking dataset (VOST) with additional manual annotations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing works that are required by the proposed method and benchmark dataset properly cited with specified versions.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code and manual annotations of VOST-TAS with proper documentation and under MIT license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The use for GPT-4.1 [1] for constructing state graphs are detailed in Fig. 2, Sec. 3.4 and 4 and the for measuring semantic description accuracy detailed in App. A.2.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : /ttp/reuisp.../// cc/ for what should or should not be described.
/LLM)
27


Reviews:
[{"review_id": "ydtDb8f8tx", "reviewer": "Reviewer_jTZM", "summary": "While existing object trackers often fail when objects undergo transformations due to significant changes in appearance, this paper introduces TubeletGraph, a zero-shot system designed to recover these lost objects and map out how they evolve over time. The core of the method is to first partition the video into a dense set of potential tracks, or \"tubelets,\" and then intelligently filter them based on spatial proximity and semantic consistency with the original object. For each recovered track, the system uses its appearance as a marker to prompt a multi-modal LLM, which in turn generates a natural language description of the transformation to build a structured state graph. This novel framework allows TubeletGraph to achieve state-of-the-art tracking performance under transformations while also demonstrating a deeper, semantic understanding of the events taking place.", "questions": "- The paper states the filtering settings were tuned on the VOST dataset and then applied to the others. Did the authors also try to find the optimal settings for the other datasets, and if so, how much did the performance improve?\n- Can you show any examples where the LLM's description of the transformation is incorrect or vague?\n- Given that the fine-tuned SAM2.1 (ft) model demonstrates strong performance (Table 1), it would be insightful to see the results of applying the TubeletGraph pipeline on top of this specialized tracker. Did the authors run this experiment? This would help quantify the upper-bound performance of the proposed framework when combined with a domain-adapted tracker.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n- The paper defines the important problem of tracking objects undergo transformations, a case where standard trackers often fail.\n- TubeletGraph is a training-free method, so it can use the generalization power of the standard trackers like SAM2.1 without needing expensive new training data\n- By generating a state graph with descriptions from an LLM, the method provides richer information about what happened to the object, not just where it went.\n\nWeaknesses:\n- While the paper provides a quantitative evaluation for the temporal localization of transformations, it lacks a similar evaluation for the semantic correctness of the state graph. The system's ability to \"describe the transformation\" is only validated through qualitative examples, with no metrics to measure the accuracy of the LLM-generated descriptions\n-The paper reports a high computational cost, stating that its method takes \"around 8 seconds per frame\". This slow speed (roughly 0.125 frames per second) is a major concern for practical usability. While the generated state graph provides rich and useful descriptions of the transformation, as shown in the qualitative examples, this extreme computational demand calls the method's value proposition into question. For many applications, the additional benefit may not justify such a drastic performance drop compared to real-time trackers.\n- While the paper does provide an ablation study on its own filtering constraints (semantic and proximity) , it doesn't analyze the impact of the external models it depends on, such as CropFormer, SAM2.1 or CLIP. The paper lacks an analysis of how errors from these foundational models would propagate, which makes it hard to judge the system's overall robustness.\n- The paper states that the filtering settings (spatial proximity and the semantic consistency)were tuned on the VOST dataset and then applied to the others. Did the authors also try to find the optimal settings for the other datasets, and if so, how much did the performance improve? the generalization of these selected thresholds was not presented clearly", "was_revised": true, "final_justification": "The authors addressed my main concerns by conducting additional experiments and semantic evaluation.", "rebuttal": "We thank reviewer jTZM for their insightful comments and suggestions. We offer details to address the concerns below:\n\n---\n### **W1. Missing semantic evaluation for predicted state graphs**\nTo address this, we conducted additional experiments to evaluate the semantic accuracy of the predicted state graphs.\n\n**Experimental Setup**: We first manually annotate captions for each object transformation used in our temporal localization evaluation (Section 4.2). Then, we consider any predicted transformations within 2 seconds of the GT localization as temporally correct candidates for evaluation. To measure semantic accuracy, we consider both the predicted *action verb* and *resulting object state*. We ask 3 human raters to evaluate the semantic similarity between predicted descriptions and ground truth annotations on a 1-5 scale.\n\n|  |avg. rating|% of 5/5|ICC(2,k)|\n|-|:-:|:-:|:-:|\n|Verbs|4.28/5 |71.8|0.976 |\n|Result objects |4.38/5 |71.8|0.923|\n\n**Results**\n- We find high semantic correctness for both the predicted action verbs (average rating of 4.28/5 with 71.8% receiving perfect 5/5 scores) and resulting object states (average rating of 4.38/5 with 71.8% receiving 5/5 scores). \n- The reliability of the human raters are confirmed by intraclass correlation coefficients (ICC) all above 0.9.\n- This validates that the state graphs accurately capture the semantics of the object transformation.\n\nDue to the constrained rebuttal period, we are not able to extend the label set further, but we promise to extend both the semantic and temporal localization evaluation to the entire VOST val in the camera-ready.\n\n\n---\n### **W2. High computational cost**\n\nWe acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.\n\n**Value Beyond Real-Time Performance**: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots,  analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.\n\n**Potential for Amortized Efficiency**: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.\n\n**Bottleneck Analysis and Potential Improvements**: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore,  we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.\n\n\n---\n### **W3. Missing analysis of external models and error propagation**\n\nTo address this concern, we conduct ablation studies on key components and their impact on system performance.\n\n**W3.1 FCCLIP vs. DINOv2 Analysis**\nWe evaluated our method's robustness by replacing FCCLIP with DINOv2 while keeping all other components unchanged. As shown below, performance remains remarkably consistent across both feature extractors, demonstrating that our framework is robust to the choice of visual encoder without requiring hyperparameter adjustments.\n\n|Method|J|J_S|J_M|J_L|P|R|J(tr)|J(tr)_S|J(tr)_M|J(tr)_L|P(tr)|R(tr)|\n| - |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours (w/ FCCLIP)|51|41.3|53.3|68.9|68.1|63.8|36.9|23.6|40.5|60.6|55.2|47.2|\n|Ours (w/ DINOv2)|50.9|41.3|52.4|69.4|68.3|63.3|36.7|23.6|39.4|61.3|55.3|46.6|\n\n| Method  | VOST J | VOST J(tr) | VSCOS J | VSCOS J(tr) | M3-VOS J | M3-VOS J(tr) | DAVIS J | DAVIS J(tr) |\n| - |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours (w/ FCCLIP)|51|36.9|75.9|72.2|74.2|64.4|85.6|82.5|\n|Ours (w/ DINOv2)|50.9|36.7|76.2|73|74.1|64.1|85.6|82.5|\n\n\n\n**W3.2 - VLM Robustness for Semantic Evaluation**\nWe further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models, with Qwen marginally underperforming GPT4.1.\n\n|  |avg. rating|% of 5/5|ICC(2,k)|\n|-|:-:|:-:|:-:|\n|GPT4.1 - verbs|4.28/5 |71.8|0.976 |\n|GPT4.1 - result objs |4.38/5 |71.8|0.923|\n|Qwen 2.5 VL - verbs |3.92/5 |64.1|0.946 |\n|Qwen 2.5 VL - result objs|4.18/5|69.2|0.982 |\n\n\n**W3.3 - Error propagation**\nAfter analyzing the failure cases, we identified that errors typically manifest as (1) false positive predictions by the base tracker and (2) minor reduction in tracking recall when applying semantic / proximal constraints (Tab. 1). False positive errors (1) can cause erroneous measures of semantic / proximal similarities, while reduction in tracking recall can reduce recall for temporal localization of object transformations.\n\nDue to rebuttal period constraints, we will extend this analysis to include CropFormer (replacing with Mask2Former) and SAM2.1 (replacing with Cutie or SAMURAI) in the camera-ready version for a comprehensive robustness evaluation.\n\n---\n### **W4. No threshold tuning on other datasets (generalization is not shown)**\n\nWe did not tune hyperparameters on other datasets since we observed TubeletGraph's performance to be robust across varying threshold values. To validate this observation and address the generalization concern, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on both M3-VOS and VSCOS datasets.\n\n| M3-VOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 74.1  | 74.2 | 74.2 | 74.2 | 74.1 |\n| 0.6  | 74.1| 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.7  | 74.1  | 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.8 | 73.8 | 73.7 | 73.7 | 73.8 | 73.7 |\n| 0.9  | 73 | 72.8 | 72.8 | 72.8 | 72.8 |\n\n\n| VSCOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.6 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.7 | 75.7 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.8 | 75.6 | 75.7 | 75.8 | 75.8 | 75.8 |\n| 0.9 | 75| 75| 75| 75| 75|\n\n**Hyperparameter Robustness Analysis**\n- The grid search result shows stable performance across large parameter ranges, with maximum performance differences across a large set of parameter ranges being minimal (M3-VOS: 74.3 vs 72.8; VSCOS: 75.9 vs 75)\n- The parameters tuned on the VOST train (sem_thrd=0.7, prox_thrd=0.3, found in center grid) perform competitively across all datasets.\n- This verifies that the hyperparameter selection generalizes well across datasets without requiring dataset-specific tuning. This robustness further underlines the stability of our filtering mechanism without a need for precise threshold calibration.\n\n---\n### **Q1. Examples where the LLM's description of the transformation is incorrect or vague**\n\nPlease refer to the supplementary HTML page for the following representative failure cases.\n\nVideo ID: 0315_paint_cream_4\n- Error: The VLM misidentifies dropped paint cream as \"meringue\".\n- Analysis: Both share similar visual appearance (white, creamy texture). With a plain background and limited contextual cues, the VLM cannot distinguish between two.\n\nVideo ID: 7359_fold_tape_measure\n- Error: The tape measure case is misrecognized as a \"smartphone,\" and the extended measuring tape is described as a \"metal rod.\" Consequently, the transformation is described as \"attach\" rather than the correct \"extend\" or \"pull out.\"\n- Analysis: We believe that this failure stems from incomplete object views in the selected frames passed to the VLM, where hand occlusions prevent accurate object recognition and lead to cascading error for the action description.\n\nWe will provide additional failure case examples and analysis in the camera-ready version.\n\n---\n### **Q2. Applying TubeletGraph on top of SAM2.1 (ft) and quantifying upper-bound performance**\n\nThank you for the suggestion. We evaluated TubeletGraph using the fine-tuned SAM2.1 (ft) model as the base tracker below.\n\n|VOST      |J   |J_S |J_M |J_L |P   |R   |J(tr)|J(tr)_S|J(tr)_M|J(tr)_L|P(tr)|R(tr)|\n| - |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|SAM2.1(ft)|54.4|46.2|53.8|73.1|70.9|65.5|36.4 |25.7 |35.1|61.3 |53.2 |45.4 |\n|Ours |51|41.3|53.3|68.9|68.1|63.8|36.9 |23.6 |40.5 |60.6|55.2 |47.2 |\n|Ours+FT|54.1|45.6|53.8|72.7|69.4|67.7|37.4 |24.8 |38.3 |63.3 |52.5 |49.1 |\n\nWe find that the integration of SAM2.1 (ft) with TubeletGraph (Ours+FT) shows modest improvements over the base TubeletGraph. However, the improvements are smaller than expected, given the strong standalone performance of SAM2.1 (ft).\n\nWe reason that is because TubeletGraph specifically addresses false-negative predictions by incorporating new candidate tracks lost due to object transformation. Since SAM2.1(ft) is fine-tuned on VOST to minimize these false negatives caused by transformation, the complementary benefits are naturally reduced."}, {"review_id": "IT0aHF1a6y", "reviewer": "Reviewer_zLVN", "summary": "The paper introduces TubeletGraph, a new system for tracking real-world objects as they change over time, such as an apple being sliced or a butterfly emerging from a chrysalis. Traditional tracking methods often fail when an object's appearance changes significantly, causing them to lose track of the object. TubeletGraph addresses this by recovering these missing objects and building a state graph that explains what changed and when. It works without any additional training by using spatial and semantic clues to identify relevant parts of the video. It also uses a language model to describe each transforation. The system performs well on multiple datasets, surpassing existing methods in both object tracking and understanding object transformations.", "questions": "- Why is SAM2.1++ not considered?\n- Can the system handle more complicated or slow changes, not just simple ones like cutting or splitting?\n- Is it a problem that the method uses fixed rules to find missing parts of the object?\n- How reliable is the description of what happened, since it depends on GPT-4 to explain the changes?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n\n- The text is well written overall: clear, detailed, and accessible. \n- The Paper introduces a task called \"Track Any State\" that combining object tracking and transformation understanding (to be fair - not sure how new this consideration really is, see e.g.: https://arxiv.org/abs/2404.18143)\n- The approach successfully tracks objects even when they change appearance, break apart, or transform over time.\n- Recovers missing object parts using a combination of spatial proximity and semantic similarity.\n- Builds a state graph that explains what transformation occurred and when it happened.\n- Operates in a zero-shot setting without requiring additional training or fine-tuning.\n- Achieves improved performance on challenging datasets focused on object transformations.\n\nWeaknesses:\n\n- The progess seems quite incremental (and a bit manually engineered). I don't see a huge methodological advancement to be honest (however, it's not my field of research, hence a lower confidence)\n- On the considered datasets, the empirical advantage seems rather small and is only given for 2/4 datasets (performance is not terribly bad on the 2 others)\n- The paper does not compare against recent state-of-the-art such as SAM2.1++ (published at CVPR24). At first glance, SAM2.1++ seems to make quite a big step forward. Why do the authors not consider it?\n\nhttps://openaccess.thecvf.com/content/CVPR2025/papers/Videnovic_A_Distractor-Aware_Memory_for_Visual_Object_Tracking_with_SAM2_CVPR_2025_paper.pdf\n\n- Object recovery depends on fixed thresholds for semantic similarity and spatial proximity (tuned manually).\n- While the method is tested somewhat broadly, there is little qualitative or diagnostic analysis of where and why it fails.\n\nFurther comments:\n\n- This work seems to align very closely with the ideas of Zack's event segmentation theory. The authors should carefully relate their approach with this well-established theory and related research (also in ML models):\n\nZacks, J. M., & Swallow, K. M. (2007). Event Segmentation. Current Directions in Psychological Science, 16(2), 80-84. https://doi.org/10.1111/j.1467-8721.2007.00480.x (Original work published 2007)\n\n- Based on the idea of state changes the authors might also have a look at: https://arxiv.org/abs/2305.16291", "was_revised": true, "final_justification": "Based on the responses, I now lean towards a borderline accept.", "rebuttal": "We thank reviewer zLVN for their insightful comments and suggestions. We offer details to address the concerns below:\n\n---\n### **W1. Incremental progress**\n\nTo clarify, our paper proposes the following contributions:\n- We introduce a novel task of “Track Any State,” which requires simultaneously tracking objects through transformations while detecting and describing those transformations. As shown in Fig. 1 (bottom), no existing methods can both predict complete object tracks and at the same time  provide spatiotemporal grounding and a semantic description of the transformation.\n- We introduce a novel approach for recovering objects after transformations by leveraging a spatiotemporal partition of the input video.\n- Our method tackles the novel task without requiring any labeled transformation data or training, making it broadly applicable across domains.\n\n---\n### **W2. Small empirical advantage for 2/4 datasets**\n\nWe would like to point out that of the 2 datasets the reviewer is concerned about, DAVIS does not contain transforming objects. We include this benchmark to validate that our method doesn't overfit to transformation scenarios / degrade performance when objects do not undergo transformation. On the other datasets with transformations, our approach provides consistent significant improvements over SAM2.1.\n\nIn addition to a consistent, statistically significant improvement over the base SAM2.1 for tracking transforming objects, TubeletGraph does demonstrate an entirely new capability: simultaneous detecting and describing transformations. Here, we would like to note the strong performance in the temporal localization (Sec. 4.2) and semantic accuracy (shown below in *Q3. How reliable is the GTP-4’s description?*).\n\n---\n### **W3. Missing comparison with SAM2.1++**\n\nSAM2.1++(CVPR2025) was published after our submission deadline. We benchmarked it via its official repo below.\n\n| Method           | VOST J | VOST J(tr) | VSCOS J | VSCOS J(tr) | M3-VOS J | M3-VOS J(tr) | DAVIS J | DAVIS J(tr) |\n| ---------------- |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|SAM2.1  |48.4    |32.4     |72       |66.9 |71.3|59.3|85.7|83|\n|SAM2.1++|48.8    |33.6     |71.3     |66.0 |72.2|61.3|86.2|84.2|\n|Ours    |51      |36.9     |75.9     |72.2 |74.2|64.4|85.6|82.5|\n\n\nWe see that TubeletGraph consistently outperforms SAM2.1++ on all transformation-focuses datasets: VOST (+3.3 J_tr), M3-VOS (+3.1 J_tr), and VSCOS (+6.2 J_tr), while maintaining similar performance on non-transformation dataset: DAVIS (85.6 vs 86.2 J)\n\n\n---\n### **W4. Object recovery depends on fixed thresholds**\n\nWe found TubeletGraph's performance to be robust across varying threshold values. To this, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on M3-VOS and VSCOS datasets. The results demonstrate impressive stability, verifying that the hyperparameters are robust and generalize well across datasets without additional tuning efforts.\n\n| M3-VOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 74.1  | 74.2 | 74.2 | 74.2 | 74.1 |\n| 0.6  | 74.1| 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.7  | 74.1  | 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.8 | 73.8 | 73.7 | 73.7 | 73.8 | 73.7 |\n| 0.9  | 73 | 72.8 | 72.8 | 72.8 | 72.8 |\n\n\n| VSCOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.6 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.7 | 75.7 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.8 | 75.6 | 75.7 | 75.8 | 75.8 | 75.8 |\n| 0.9 | 75| 75| 75| 75| 75|\n\n\n---\n### **W5. Limited qualitative / diagnostic analysis of failure modes**\n\n**Qualitative Analysis**: We showcase representative failure cases in the supplementary HTML page.\n\n*Representative failure cases in transformation descriptions*\n\nVideo ID: 0315_paint_cream_4\n- Error: The VLM misidentifies dropped paint cream as \"meringue\"\n- Analysis: Both share similar visual appearance (white, creamy texture). With a plain background and limited contextual cues, the VLM cannot distinguish between two.\n\nVideo ID: 7359_fold_tape_measure\n- Error: The tape measure case is misrecognized as a \"smartphone,\" and the extended measuring tape is described as a \"metal rod.\" Consequently, the transformation is described as \"attach\" rather than the correct \"extend\" or \"pull out.\"\n- Analysis: We believe that this failure stems from incomplete object views in the selected frames passed to the VLM, where hand occlusions prevent accurate object recognition and lead to cascading error for the action description.\n\n*Representative failure cases in tracking performance*\n\nVideo ID: 0340_fry_omelet_1\n- Error: A false positive omelet is incorporated into the state-graph at t=6s.\n- Analysis: the flipping of a nearby irrelevant omelet causes a new tubelet to initiate, which was falsely incorporated to the original tracked batter due to high semantic and proximity similarity.\n\nVideo ID: 0357_make_pancakes_3\n- Error: False negative regions are found during object transformation.\n- Analysis: This is likely caused by a fast rotating occluder spatula altering the appearance of the transforming object rapidly. Once the transformation is complete, all associated regions are properly tracked.\n\n**Diagnostic Analysis**: We observe a reduction in recall as we improve precision via the filterings (Tab. 1). This indicates that some positive objects are lost as we filter out unrelated instances. \n\nWe promise to add more failure cases and the above discussion in the camera ready.\n\n\n---\n### **W6. Missing references**\n\nWe appreciate the references mentioned! \n\nIndeed, the event segmentation discussed by Zacks and Swallow [A] does relate to the predicted state graph well. In fact, when evaluating temporal localization, each annotated transformation does fall near an event boundary, with an altered object state indicating a low-level goal of the activity being achieved. However, not all event boundaries will contain a state-altering transformation (two people playing catch can have well-defined action boundaries without any transformation to the ball). In our task of “Track Any State,” the goal is to detect and describe any object state transformations while maintaining consistent track of the target object.\n\nWhile Voyager [B] focuses on LLM-driven embodied agent acquiring new skills in a simulated world engine without human intervention, its mechanism of prompting LLM for generating new tasks and programs is similar to our usage of prompting VLM to describe the detected transformation from before/after visual observations (Fig. 2(3)).\n\nThe benchmark by Wu et al. [C] focuses on bounding box tracking of transforming objects without any notation of detecting or describing the transformation. In our proposed “Track Any State,” the goal is both detecting and describing any object state transformations and maintaining accurate object tracks. \n\nWe will integrate these discussions in the camera-ready version.\n\n[A] Zacks & Swallow. “Event Segmentation.” Current Directions in Psychological Science (2007).  \n[B] Wang et al. “Voyager: An open-ended embodied agent with large language models.” arXiv preprint arXiv:2305.16291 (2023).  \n[C] Wu et al. “Tracking Transforming Objects: A Benchmark.”  arXiv preprint arXiv:2404.18143 (2024).\n\n\n---\n### **Q1. Can the system handle more complicated or slow changes?** \n\nYes! Please refer to the video examples in the supplementary HTML page. We list some representative examples below.\n- Video ID: 0315_paint_cream_4: dropping multiple droplets of paint cream.\n- Video ID: 0269_press_pencil_1: pressing a pencil tip until it shatters\n- Video ID: 0250_splash_liquid_2: dropping a disk into a bowl of water\n- Video ID: 0369_cut_cheese_3: cutting into a cheese ball containing both solid and liquid components.\n- Video ID: 0357_make_pancakes_3: Spreading pancake batter\n- Video ID: 7359_fold_tape_measure: Extending and folding a metal tap measure\n\n---\n### **Q2. Is it a problem that fixed rules are used to find missing object parts?**\n\nAlthough a more elaborate system may improve performance, we believe that the fixed rules/thresholds allow a very robust system without need for dataset-specific tuning (shown above in *W4. Object recovery depends on fixed thresholds*), at the same time leveraging the modeling capabilities of the underlying models (e.g., FCCLIP for semantic similarity and SAM2’s candidate masks for proximity measure).\n\n---\n### **Q3. How reliable is the GTP-4’s description?**\n\nWe found TubeletGraph to predict reliable action verbs (4.28/5) and resulting objects (4.38/5) as evaluated by human raters with very high reliability (ICC(2,k) > 0.9). For details regarding the experimental setup, please refer to *our response to Reviewer jTZM, W1*.\n\n|  |avg. rating|% of 5/5|ICC(2,k)|\n|-|:-:|:-:|:-:|\n|Verbs|4.28/5 |71.8|0.976 |\n|Result objects |4.38/5 |71.8|0.923|"}, {"review_id": "Z8paHMkC2x", "reviewer": "Reviewer_rGHV", "summary": "This paper introduces TubeletGraph, a zero-shot framework for tracking objects through state transformations while simultaneously detecting and describing these transformations. The authors formulate the \"Track Any State\" problem, which extends beyond traditional object tracking by maintaining object identity through dramatic appearance changes (e.g., apple being sliced, butterfly emerging from chrysalis). TubeletGraph creates a spatiotemporal partition of the video into \"tubelets\" using entity segmentation and SAM2 tracking, then uses semantic and spatial proximity constraints to identify missing object components after transformation. The system leverages multi-modal LLMs to generate natural language descriptions of transformations and constructs state graphs representing object evolution. Experimental validation on VOST, VSCOS, M3-VOS, and DAVIS 2017 datasets demonstrates state-of-the-art performance on transformation tracking while maintaining competitive results on standard tracking benchmarks.", "questions": "1. The reported 8 seconds per frame processing time on high-end hardware (RTX A6000) severely limits the method's practical applicability. For a tracking method to be useful in real-world scenarios like robotics applications (which the authors cite as motivations), near real-time performance is essential. The lack of thorough complexity analysis or optimization strategies makes this a critical limitation.\n\n​2. The spatial proximity assumption is a fundamental design choice that restricts the method's applicability. The inability to handle non-proximal transformations (e.g., objects that transform while moving significant distances) represents a major limitation that is neither adequately addressed nor tested in the experiments.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n\n1.The paper addresses a genuinely important and challenging problem that extends beyond traditional object tracking. The \"Track Any State\" formulation is well-motivated with clear real-world applications in robotics and augmented reality.\n\n2.The zero-shot nature of the approach is a significant advantage over supervised methods, eliminating the need for expensive transformation-specific annotations while providing better generalization across different types of object transformations.\n\nWeaknesses：\n1. Method validity\n​(1) The computational efficiency is a significant concern. The authors report 8 seconds per frame on an RTX A6000 GPU for constructing the spatiotemporal partition, which severely limits real-time applications. While they mention the complexity is near-linear with limited camera motion, this computational cost needs more thorough analysis and potential optimization strategies.\n​(2) Limited Handling of Non-Proximal Transformations. The spatial proximity prior (Sec 3.3) assumes transformed objects appear near original locations. This fails for distant transformations (e.g., a launched firework), as evidenced by no such test cases.\n2. Experimental details\n​(1) The hyperparameter selection process appears somewhat ad-hoc. The authors sweep Tau_prox and Tau_sem on VOST train split and apply the same values across all datasets without further tuning. A more principled approach to hyperparameter selection or adaptive threshold determination would strengthen the method's robustness.\n​(2) Method depends on external models (CropFormer, SAM2, FC-CLIP, GPT-4) . The paper would benefit from ablation studies showing the impact of different model choices.", "was_revised": true, "final_justification": "Given the responses and the reviews of other reviewers, I will raise my score.", "rebuttal": "We thank reviewer rGHV for their insightful comments and suggestions. We offer details to address the concerns below:\n\n\n---\n### **W1. High computational cost**\n\nWe acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.\n\n**Value Beyond Real-Time Performance**: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots,  analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.\n\n**Potential for Amortized Efficiency**: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.\n\n**Bottleneck Analysis**: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore, we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.\n\n---\n### **W2. Limited handling of non-proximal transformations**\n\nA launched firework is indeed an interesting example. We believe that this transformation does start with two proximal components (firework leaving the firework box), but due to low frame rate, the two components would appear to be non-proximal. We find these cases to be rare in day-to-day scenarios.\nFurthermore, the proximity constraint in TubeletGraph is soft, as we use the candidate masks produced by SAM2.1 to measure proximity. Therefore, as long as SAM2 can connect these non-proximal objects, our method will still succeed.\n\n\n---\n### **W3. The hyperparameter selection process appears ad-hoc**\n\nWhile we agree that a more principled approach to hyperparameter selection would improve the method's robustness, we found TubeletGraph's performance to be already very robust across varying hyperparameter values. To verify this, we conducted comprehensive grid searches over both spatial proximity and semantic consistency thresholds on M3-VOS and VSCOS datasets. The results demonstrate impressive stability, verifying that the hyperparameters are robust and generalize well across datasets without additional tuning efforts.\n\n| M3-VOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 74.1  | 74.2 | 74.2 | 74.2 | 74.1 |\n| 0.6  | 74.1| 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.7  | 74.1  | 74.2 | 74.2 | 74.3 | 74.1 |\n| 0.8 | 73.8 | 73.7 | 73.7 | 73.8 | 73.7 |\n| 0.9  | 73 | 72.8 | 72.8 | 72.8 | 72.8 |\n\n\n| VSCOS [J]    | prox_thrd=0.1 | 0.2  | 0.3  | 0.4  | 0.5  |\n| -:|:-:|:-:|:-:|:-:|:-:|\n| sem_thrd=0.5 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.6 | 75.6 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.7 | 75.7 | 75.8 | 75.9 | 75.8 | 75.8 |\n| 0.8 | 75.6 | 75.7 | 75.8 | 75.8 | 75.8 |\n| 0.9 | 75| 75| 75| 75| 75|\n\n---\n### **W4. Missing ablation studies for the impacts of different external model choices**\n\nTo address this concern, we conduct ablation studies on key components and their impact on system performance.\n\n**W4.1 FCCLIP vs. DINOv2 Analysis**\nWe evaluated our method's robustness by replacing FCCLIP with DINOv2 while keeping all other components unchanged. As shown below, performance remains remarkably consistent across both feature extractors, demonstrating that our framework is robust to the choice of visual encoder without requiring hyperparameter adjustments.\n\n|Method|J|J_S|J_M|J_L|P|R|J(tr)|J(tr)_S|J(tr)_M|J(tr)_L|P(tr)|R(tr)|\n| - |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours (w/ FCCLIP)|51|41.3|53.3|68.9|68.1|63.8|36.9|23.6|40.5|60.6|55.2|47.2|\n|Ours (w/ DINOv2)|50.9|41.3|52.4|69.4|68.3|63.3|36.7|23.6|39.4|61.3|55.3|46.6|\n\n| Method  | VOST J | VOST J(tr) | VSCOS J | VSCOS J(tr) | M3-VOS J | M3-VOS J(tr) | DAVIS J | DAVIS J(tr) |\n| - |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours (w/ FCCLIP)|51|36.9|75.9|72.2|74.2|64.4|85.6|82.5|\n|Ours (w/ DINOv2)|50.9|36.7|76.2|73|74.1|64.1|85.6|82.5|\n\n\n\n**W4.2 - VLM Robustness for Semantic Evaluation**\nWe further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models, with Qwen marginally underperforming GPT4.1.\n\n|  |avg. rating|% of 5/5|ICC(2,k)|\n|-|:-:|:-:|:-:|\n|GPT4.1 - verbs|4.28/5 |71.8|0.976 |\n|GPT4.1 - result objs |4.38/5 |71.8|0.923|\n|Qwen 2.5 VL - verbs |3.92/5 |64.1|0.946 |\n|Qwen 2.5 VL - result objs|4.18/5|69.2|0.982 |\n\n\nDue to rebuttal period constraints, we will extend this analysis to include CropFormer (replacing with Mask2Former) and SAM2.1 (replacing with Cutie or SAMURAI) in the camera-ready version for a comprehensive robustness evaluation."}, {"review_id": "9OXssJI5hL", "reviewer": "Reviewer_6G8j", "summary": "The paper proposes a zero-shot framework called TubeletGraph that can track objects through transformations, identifying potential tracks after state changes and also reasoning about the state change through a VLM. Introducing Track Any State as a new problem definition, TubeletGraph creates a full state graph for the tracked object through extreme transformations where conventional trackers can commonly fail. The method performs favorable to other methods it compares against.", "questions": "- From the ablations it looks like once you have the spatiotemporal partition + proximity constraint, the semantic consistency brings little improvement. Do the authors think this potentially is a factor due to the dataset, and that semantic constraint can show more significant improvement in different cases?\n- For reasoning, did the author’s try any open source models? I think it can be interesting to see if that changes the performance of reasoning and would make this framework more adoptable.\n- Can the authors elaborate further on the fine-tuning on SAM2 that you made during ablations? The paper mentions clear high annotation cost for object transformations, do you mean the existing annotations in the dataset, or did you annotate anything further?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written with an interesting problem covering very complex cases in object tracking.\n- To my knowledge, the paper proposes a new paradigm in Track Any State.\n- The experimental results show that for most cases, the proposed method that is zero-shot can show improvements over compared baselines/models.\nWeaknesses:\n- Creating the tubelets seems to be a very expensive process and not quite ready for practical use. (Seems like this is also discussed in the limitations) There's also several models involved, including a non-open source model (question on that below).\n- It would be interesting to see how this zero-shot method performs under other common confounding factors in object tracking such as occlusions. In these kinds of conditions, do we just generate additional tubelets if object is temporarily lost? Do we expect VLM to reason the transformation is the occlusion? Since occlusions are a fundamental issue with tracking I think it would make the paper stronger to discuss this.\n- Authors show ReVOS outperforms their method on $M^{3}$-VOS. It would be great to give more insights of why this is the only dataset where they don’t show superior performance in the Main Results section. Is it related to a property of the evaluation set that makes ReVOS especially strong on this set? Same with DAVIS17 results.\n- The temporal localization is evaluated only on 8 manually annotated videos, this seems insufficient to me. In addition, the experiment is a little confusing, it might be worth it to explain further how the varying T values were chosen for each line in the graph.", "was_revised": true, "final_justification": "After the rebuttals to all reviews and the detailed answers the authors provided, I decided to increase my rating. I think the added discussions on semantic evaluation, the robustness to thresholds/different models have provided more strength to the proposed framework. Once concern remaining is that some revisions promised by the authors can be involved and might not be ready before camera-ready (e.g. extending evaluations to the full VOST val set). However, due to the detailed answers during the rebuttal, I am hoping these will be prioritized for a stronger final paper.", "rebuttal": "We thank Reviewer 6G8j for their positive evaluation and for recognizing the motivation and novelty of our work. Below, we address the reviewer’s specific questions and concerns.\n\n---\n### **W1. High computational cost**\n\nWe acknowledge the concern regarding computational cost and agree that the current speed limits real-time applications. However, we believe TubeletGraph’s unique capabilities provide significant value despite this limitation.\n\n**Value Beyond Real-Time Performance**: TubeletGraph enables detecting and describing object state transformations beyond object tracking in videos. No existing methods perform both temporal localization and semantic description of the transformation in addition to state-of-the-art performance on object tracking under transformation. The combination of tracking and discovering, localizing and describing transformations from videos can be very useful for many applications: for example, producing training annotations on recorded demonstrations for robots, analyzing compliance and processes on the factory floor from videos, understanding animal behavior and development from camera traps and so on. In these applications, understanding and tracking object transformations is critical, and real-time processing is not needed.\n\n**Potential for Amortized Efficiency**: While the spatiotemporal partitioning is presented to be dependent on input object masks for clarity, our actual implementation (available in supplementary code) treats partition construction as a single preprocessing step that is shared across all objects of interest. This design allows the computational cost to be amortized when tracking multiple objects simultaneously. After preprocessing, TubeletGraph operates at speeds comparable to the base SAM2 model.\n\n**Bottleneck Analysis**: The primary bottleneck is in constructing the spatiotemporal partition, which requires independent SAM2 tracking for each identified spatial region. This process is inherently parallelizable. Furthermore,  we envision that future end-to-end systems can track multiple objects without incurring cost linear in the number of objects in the video. While important for practical deployment, we consider this optimization direction orthogonal to our core contributions of tracking and describing object transformations.\n\n\n---\n### **W2. Use of a non-open source models for transformation reasoning and description**\n\nWe further evaluate semantic correctness when GPT-4.1 is replaced with the open-source Qwen2.5-VL-7B-Instruct. We show comparable performance across both models below, with Qwen marginally underperforming GPT4.1 in terms of action verb description. For details regarding the experimental setup, please refer to *our response to Reviewer jTZM, W1*.\n\n|  |avg. rating|% of 5/5|ICC(2,k)|\n|-|:-:|:-:|:-:|\n|GPT4.1 - verbs|4.28/5 |71.8|0.976 |\n|GPT4.1 - result objs |4.38/5 |71.8|0.923|\n|Qwen 2.5 VL - verbs |3.92/5 |64.1|0.946 |\n|Qwen 2.5 VL - result objs|4.18/5|69.2|0.982 |\n\n---\n### **W3. What effects do occlusions have on the model?**\n\nThis is indeed a very interesting question. During occlusion events, the model would generally add additional tubelets for the object that is temporarily lost. If the additional tubelet matches the semantic and proximity constraints, it will be incorporated in the tracked object. Since the proximity constraint relies on the three candidate masks predicted by SAM2, the base tracker’s internal candidate masks would set an upperbound for the recall under occlusion. In terms of the transformation, since the VLM only sees the two frames where the initial and resulting objects are visible, it will not be able to describe the occlusion event as a transformation.\n\nNevertheless, addressing the occlusion challenge with the aid of the VLM is an interesting direction for future works, and we will add discussions on this in the final manuscript.\n\n---\n### **W4. Why does ReVOS outperform on M3-VOS?**\n\nIndeed, ReVOS outperforms our approach on M3-VOS, but it significantly underperforms on other datasets with object transformations. According to its authors, they “assume SAM2 takes more training data similar to EPIC-kitchens [5] from which the VOST is constructed.” We share that belief that M3-VOS and DAVIS17 contain far less ego motion compared to VOST. As such, the lower performance of our approach on M3-VOS compared to ReVOS can be traced to the fact that our base SAM2.1 tracker is seeing a distribution shift, compared to the ReVOS tracker. \n\nThis also explains the differences on DAVIS17. Given very similar performances for TubeletGraph and base SAM2 models, we show that our method doesn't overfit to transformation scenarios / degrade performance when objects do not undergo transformation.\n\nWe will add discussions on this in the final manuscript.\n\n---\n### **W5. Limited temporal localization evaluation and confusing choices for varying threshold values**\n\nWe acknowledge the concern. Due to the constrained rebuttal period, we are not able to expand the label set further, but we promise to extend the evaluation to the entire VOST val set in the camera ready. \n\nThe varying T values were chosen to see how much recall / precision we can gain by setting the proximity threshold to be low (0) and setting the semantic threshold to be relatively high (0.9) respectively. We found that increasing the proximity threshold does not lead to increasing precision, since SAM2’s internal candidate masks, which are what the proximity prior relies on, induce an upper bound. We will incorporate this in the final version.\n\n---\n### **W6. Semantic consistency brings little improvement upon ST partition and proximity constraint**\n\nThis is indeed a good observation. The semantic constraint does provide small, but consistent gain across all metrics when included. There are cases where semantic cases help, for instance when the tracker latches on to hands or other actors that begin interacting with the tracked object. In these cases, the proximity prior alone is not sufficient to reject the negative candidate. We will provide more example videos / discussion in the final version.\n\n---\n### **Q1. Details for fine-tuning SAM2 on VOST**\n\nNo, we do not annotate anything further. SAM2.1(ft) is directly finetuned on the existing annotations in VOST. SAM2.1 finetuning does not require the transformations to be explicitly annotated. The finetuned model SAM2.1(ft) tracks the objects in a single binary mask and does not output the transformations (unlike our approach)."}]

Author rebuttal:

Meta-review: Reviewers recommend acceptance. The paper addresses the challenging problem of tracking objects undergoing transformations, framing it as the “Track Any State” task. It introduces TubeletGraph: a training-free method that leverages strong pretrained models, and builds a state graph that localizes objects and describes their transformations over time. Reviewers found the problem formulation meaningful, the approach novel, and the paper clearly written with competitive results across multiple benchmarks.

The main concerns relate to computational efficiency—at ~8 seconds per frame, the method is far from real-time and may limit practical use. While this may be a clear bottleneck for practical application of the method, the problem formulation and solution framing remains novel and worth further explorations.

Reviewers also noted the reliance on external models, fixed hyperparameters, and limited quantitative evaluation of semantic correctness. Missing comparisons to some recent baselines (e.g., SAM2.1++) and incomplete runtime/robustness analysis were also raised. These points were addressed properly, as acknowledged by the reviewers.

Overall, despite these limitations, the novelty of the task formulation and the promising results justify acceptance, with revisions to strengthen evaluation and efficiency discussion.
Decision: Accept (poster)