Title: Tracking and Understanding Object Transformations
Venue: NeurIPS 2025
Accepted: yes
Keywords: object tracking

transformations

state change detection

spatiotemporal reasoning

zero-shot
Abstract: Real-world objects frequently undergo state transformations. From an apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects and dynamics. However, existing methods often lose track of the target object after transformation, due to significant changes in object appearance. To address this limitation, we introduce the task of Track Any State: tracking objects through transformations while detecting and describing state changes, accompanied by a new benchmark dataset, VOST-TAS. To tackle this problem, we present TubeletGraph, a zero-shot system that recovers missing objects after transformation and maps out how object states are evolving over time. TubeletGraph first identifies potentially overlooked tracks, and determines whether they should be integrated based on semantic and proximity priors. Then, it reasons about the added tracks and generates a state graph describing each observed transformation. TubeletGraph achieves state-of-the-art tracking performance under transformations, while demonstrating deeper understanding of object transformations and promising capabilities in temporal grounding and semantic reasoning for complex object transformations. Code, additional results, and the benchmark dataset are available at https://tubelet-graph.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a zero-shot framework called TubeletGraph that can track objects through transformations, identifying potential tracks after state changes and also reasoning about the state change through a VLM. Introducing Track Any State as a new problem definition, TubeletGraph creates a full state graph for the tracked object through extreme transformations where conventional trackers can commonly fail. The method performs favorable to other methods it compares against.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-written with an interesting problem covering very complex cases in object tracking.\n- To my knowledge, the paper proposes a new paradigm in Track Any State.\n- The experimental results show that for most cases, the proposed method that is zero-shot can show improvements over compared baselines/models.\nWeaknesses:\n- Creating the tubelets seems to be a very expensive process and not quite ready for practical use. (Seems like this is also discussed in the limitations) There's also several models involved, including a non-open source model (question on that below).\n- It would be interesting to see how this zero-shot method performs under other common confounding factors in object tracking such as occlusions. In these kinds of conditions, do we just generate additional tubelets if object is temporarily lost? Do we expect VLM to reason the transformation is the occlusion? Since occlusions are a fundamental issue with tracking I think it would make the paper stronger to discuss this.\n- Authors show ReVOS outperforms their method on $M^{3}$-VOS. It would be great to give more insights of why this is the only dataset where they don’t show superior performance in the Main Results section. Is it related to a property of the evaluation set that makes ReVOS especially strong on this set? Same with DAVIS17 results.\n- The temporal localization is evaluated only on 8 manually annotated videos, this seems insufficient to me. In addition, the experiment is a little confusing, it might be worth it to explain further how the varying T values were chosen for each line in the graph.\n\nQUESTIONS:\n- From the ablations it looks like once you have the spatiotemporal partition + proximity constraint, the semantic consistency brings little improvement. Do the authors think this potentially is a factor due to the dataset, and that semantic constraint can show more significant improvement in different cases?\n- For reasoning, did the author’s try any open source models? I think it can be interesting to see if that changes the performance of reasoning and would make this framework more adoptable.\n- Can the authors elaborate further on the fine-tuning on SAM2 that you made during ablations? The paper mentions clear high annotation cost for object transformations, do you mean the existing annotations in the dataset, or did you annotate anything further?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TubeletGraph, a zero-shot framework for tracking objects through state transformations while simultaneously detecting and describing these transformations. The authors formulate the \"Track Any State\" problem, which extends beyond traditional object tracking by maintaining object identity through dramatic appearance changes (e.g., apple being sliced, butterfly emerging from chrysalis). TubeletGraph creates a spatiotemporal partition of the video into \"tubelets\" using entity segmentation and SAM2 tracking, then uses semantic and spatial proximity constraints to identify missing object components after transformation. The system leverages multi-modal LLMs to generate natural language descriptions of transformations and constructs state graphs representing object evolution. Experimental validation on VOST, VSCOS, M3-VOS, and DAVIS 2017 datasets demonstrates state-of-the-art performance on transformation tracking while maintaining competitive results on standard tracking benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1.The paper addresses a genuinely important and challenging problem that extends beyond traditional object tracking. The \"Track Any State\" formulation is well-motivated with clear real-world applications in robotics and augmented reality.\n\n2.The zero-shot nature of the approach is a significant advantage over supervised methods, eliminating the need for expensive transformation-specific annotations while providing better generalization across different types of object transformations.\n\nWeaknesses：\n1. Method validity\n​(1) The computational efficiency is a significant concern. The authors report 8 seconds per frame on an RTX A6000 GPU for constructing the spatiotemporal partition, which severely limits real-time applications. While they mention the complexity is near-linear with limited camera motion, this computational cost needs more thorough analysis and potential optimization strategies.\n​(2) Limited Handling of Non-Proximal Transformations. The spatial proximity prior (Sec 3.3) assumes transformed objects appear near original locations. This fails for distant transformations (e.g., a launched firework), as evidenced by no such test cases.\n2. Experimental details\n​(1) The hyperparameter selection process appears somewhat ad-hoc. The authors sweep Tau_prox and Tau_sem on VOST train split and apply the same values across all datasets without further tuning. A more principled approach to hyperparameter selection or adaptive threshold determination would strengthen the method's robustness.\n​(2) Method depends on external models (CropFormer, SAM2, FC-CLIP, GPT-4) . The paper would benefit from ablation studies showing the impact of different model choices.\n\nQUESTIONS:\n1. The reported 8 seconds per frame processing time on high-end hardware (RTX A6000) severely limits the method's practical applicability. For a tracking method to be useful in real-world scenarios like robotics applications (which the authors cite as motivations), near real-time performance is essential. The lack of thorough complexity analysis or optimization strategies makes this a critical limitation.\n\n​2. The spatial proximity assumption is a fundamental design choice that restricts the method's applicability. The inability to handle non-proximal transformations (e.g., objects that transform while moving significant distances) represents a major limitation that is neither adequately addressed nor tested in the experiments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TubeletGraph, a new system for tracking real-world objects as they change over time, such as an apple being sliced or a butterfly emerging from a chrysalis. Traditional tracking methods often fail when an object's appearance changes significantly, causing them to lose track of the object. TubeletGraph addresses this by recovering these missing objects and building a state graph that explains what changed and when. It works without any additional training by using spatial and semantic clues to identify relevant parts of the video. It also uses a language model to describe each transforation. The system performs well on multiple datasets, surpassing existing methods in both object tracking and understanding object transformations.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n\n- The text is well written overall: clear, detailed, and accessible. \n- The Paper introduces a task called \"Track Any State\" that combining object tracking and transformation understanding (to be fair - not sure how new this consideration really is, see e.g.: https://arxiv.org/abs/2404.18143)\n- The approach successfully tracks objects even when they change appearance, break apart, or transform over time.\n- Recovers missing object parts using a combination of spatial proximity and semantic similarity.\n- Builds a state graph that explains what transformation occurred and when it happened.\n- Operates in a zero-shot setting without requiring additional training or fine-tuning.\n- Achieves improved performance on challenging datasets focused on object transformations.\n\nWeaknesses:\n\n- The progess seems quite incremental (and a bit manually engineered). I don't see a huge methodological advancement to be honest (however, it's not my field of research, hence a lower confidence)\n- On the considered datasets, the empirical advantage seems rather small and is only given for 2/4 datasets (performance is not terribly bad on the 2 others)\n- The paper does not compare against recent state-of-the-art such as SAM2.1++ (published at CVPR24). At first glance, SAM2.1++ seems to make quite a big step forward. Why do the authors not consider it?\n\nhttps://openaccess.thecvf.com/content/CVPR2025/papers/Videnovic_A_Distractor-Aware_Memory_for_Visual_Object_Tracking_with_SAM2_CVPR_2025_paper.pdf\n\n- Object recovery depends on fixed thresholds for semantic similarity and spatial proximity (tuned manually).\n- While the method is tested somewhat broadly, there is little qualitative or diagnostic analysis of where and why it fails.\n\nFurther comments:\n\n- This work seems to align very closely with the ideas of Zack's event segmentation theory. The authors should carefully relate their approach with this well-established theory and related research (also in ML models):\n\nZacks, J. M., & Swallow, K. M. (2007). Event Segmentation. Current Directions in Psychological Science, 16(2), 80-84. https://doi.org/10.1111/j.1467-8721.2007.00480.x (Original work published 2007)\n\n- Based on the idea of state changes the authors might also have a look at: https://arxiv.org/abs/2305.16291\n\nQUESTIONS:\n- Why is SAM2.1++ not considered?\n- Can the system handle more complicated or slow changes, not just simple ones like cutting or splitting?\n- Is it a problem that the method uses fixed rules to find missing parts of the object?\n- How reliable is the description of what happened, since it depends on GPT-4 to explain the changes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile existing object trackers often fail when objects undergo transformations due to significant changes in appearance, this paper introduces TubeletGraph, a zero-shot system designed to recover these lost objects and map out how they evolve over time. The core of the method is to first partition the video into a dense set of potential tracks, or \"tubelets,\" and then intelligently filter them based on spatial proximity and semantic consistency with the original object. For each recovered track, the system uses its appearance as a marker to prompt a multi-modal LLM, which in turn generates a natural language description of the transformation to build a structured state graph. This novel framework allows TubeletGraph to achieve state-of-the-art tracking performance under transformations while also demonstrating a deeper, semantic understanding of the events taking place.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The paper defines the important problem of tracking objects undergo transformations, a case where standard trackers often fail.\n- TubeletGraph is a training-free method, so it can use the generalization power of the standard trackers like SAM2.1 without needing expensive new training data\n- By generating a state graph with descriptions from an LLM, the method provides richer information about what happened to the object, not just where it went.\n\nWeaknesses:\n- While the paper provides a quantitative evaluation for the temporal localization of transformations, it lacks a similar evaluation for the semantic correctness of the state graph. The system's ability to \"describe the transformation\" is only validated through qualitative examples, with no metrics to measure the accuracy of the LLM-generated descriptions\n-The paper reports a high computational cost, stating that its method takes \"around 8 seconds per frame\". This slow speed (roughly 0.125 frames per second) is a major concern for practical usability. While the generated state graph provides rich and useful descriptions of the transformation, as shown in the qualitative examples, this extreme computational demand calls the method's value proposition into question. For many applications, the additional benefit may not justify such a drastic performance drop compared to real-time trackers.\n- While the paper does provide an ablation study on its own filtering constraints (semantic and proximity) , it doesn't analyze the impact of the external models it depends on, such as CropFormer, SAM2.1 or CLIP. The paper lacks an analysis of how errors from these foundational models would propagate, which makes it hard to judge the system's overall robustness.\n- The paper states that the filtering settings (spatial proximity and the semantic consistency)were tuned on the VOST dataset and then applied to the others. Did the authors also try to find the optimal settings for the other datasets, and if so, how much did the performance improve? the generalization of these selected thresholds was not presented clearly\n\nQUESTIONS:\n- The paper states the filtering settings were tuned on the VOST dataset and then applied to the others. Did the authors also try to find the optimal settings for the other datasets, and if so, how much did the performance improve?\n- Can you show any examples where the LLM's description of the transformation is incorrect or vague?\n- Given that the fine-tuned SAM2.1 (ft) model demonstrates strong performance (Table 1), it would be insightful to see the results of applying the TubeletGraph pipeline on top of this specialized tracker. Did the authors run this experiment? This would help quantify the upper-bound performance of the proposed framework when combined with a domain-adapted tracker.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}