Title: SAGE: A Unified Framework for Generalizable Object State Recognition with State-Action Graph Embedding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object State Recognition

Graph Embeddings

Video Understanding
Abstract: Recognizing the physical states of objects and their transformations within videos is crucial for structured video understanding and enabling robust real-world applications, such as robotic manipulation. However, pretrained vision-language models often struggle to capture these nuanced dynamics and their temporal context, and specialized object state recognition frameworks may not generalize to unseen actions or objects. We introduce SAGE (State-Action Graph Embeddings), a novel framework that offers a unified model of physical state transitions by decomposing states into fine-grained, language-described visual concepts that are sharable across different objects and actions. SAGE initially leverages Large Language Models to construct a State-Action Graph, which is then multimodally refined using Vision-Language Models. Extensive experiments show that our method significantly outperforms baselines, generalizes effectively to unseen objects and actions in open-world settings. SAGE improves the prior state-of-the-art by as much as 14.6% on novel state recognition with less than 5% of its inference time.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present SAGE (State-Action Graph Embedding), a framework for recognizing physical object state transitions in videos. The core idea is to create a unified model that can generalize to unseen objects and actions. This is achieved by decomposing object states into a set of fine-grained, language-described visual concepts. The framework leverages a Large Language Model (LLM) to construct an initial State-Action Graph, which is then refined using a Vision-Language Model (VLM). The resulting graph provides a compositional representation of states and actions, which is used to train a video transformer for the final recognition task.\n\nSTRENGTHS AND WEAKNESSES:\n## Paper Strengths\n- **Novel and Effective Conceptual Framework**: By representing states not as monolithic entities but as compositions of more primitive visual attributes, the proposed framework addresses the key challenge of generalization. This compositional approach forms a strong basis for the model's ability to recognize novel object states by leveraging familiar concepts learned from different contexts.\n- **Comprehensive Empirical Evaluation**: The authors conduct extensive experiments and provide insightful ablation studies that convincingly validate the effectiveness of their approach.\n    - *Performance*: As shown across different settings (Table 1), SAGE outperforms existing baselines on standard benchmarks.\n    - *Proven Generalization*: The results in open-world settings (Table 2 (b)), where both objects and actions are unseen during training, are impressive and directly support the paper's main claims about generalizability.\n    - *Efficiency*: The framework demonstrates a significant advantage in inference efficiency over specialist models (Table 3), which require running multiple experts when the action is unknown. This makes SAGE a more practical and scalable solution.\n    - *Insightful Ablations*: The ablation studies clearly demonstrate the contribution of each component, particularly the value of using visual concept descriptions and the Viterbi temporal decoding.\n- **Reproducibility and Clarity**: The authors' decision to release their code enhances the reproducibility of their work and provides a valuable resource.\n\n## Paper Weaknesses\n- **Insufficient Explanation of Viterbi Decoding**: The use of a constrained Viterbi decoding algorithm to refine the pseudo-labels is a key step in handling the lack of frame-level annotations. However, the explanation provided (L131-133) is very brief. While Figure 2 shows its place in the pipeline, its functional importance is understated. The paper would be significantly strengthened by a more detailed explanation accompanied by a qualitative example. For instance, a short walkthrough showing how the algorithm corrects an illogical, noisy label sequence (e.g., [Initial, End, Transitioning, End]) into a temporally coherent one would provide much-needed clarity and better highlight the value of this component.\n- **Over-reliance on LLMs without Robustness Analysis**: The SAGE graph, the core of the framework, is constructed almost entirely by an LLM. This creates a critical dependency on the quality and consistency of the LLM’s output, yet this dependency is not examined or analyzed.\n    - *LLM Sensitivity*: The authors rely on a single, powerful proprietary model (GPT-4) to generate states and concepts. Including an ablation study on the choice of LLM would be beneficial. How would the framework perform if a smaller, open-source model were used? Such an analysis would help assess the robustness of the approach and clarify how tightly its performance depends on a specific high-end LLM.\n    - *LLM Output Validation*: The authors do not address how the correctness of the LLM-generated states and visual concepts is verified. Including a brief analysis of the quality and consistency of the generated graph would be interesting.\n\nQUESTIONS:\nMy main concerns are the following:\n- Could the authors provide a more detailed explanation of the Vidertbi Decoding, ideally accompanied by a qualitative example?\n- What is the methodology for validating the accuracy and, crucially, the visual relevance of the states and concepts generated by the LLM?\n- The experiments rely on a single, powerful LLM. How dependent is the framework's overall performance on this specific choice?\n\nFor more details, please read the weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced SAGE (State-Action Graph Embeddings), a framework that improved object state recognition in videos by breaking down states into detailed visual concepts. The proposed method combined large language models and vision-language models for better generalization to new objects and actions. SAGE outperforms existing methods by up to 14.6% in recognizing novel states while being more efficient.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. SAGE outperforms existing methods by generalizing well to novel objects and actions, significantly improving object state recognition, even in open-world settings.\n2. The framework combines vision-language models (VLMs) and large language models (LLMs), leveraging multimodal pre-trained knowledge for more accurate state recognition.\n3. The framework allows for automatic addition of novel actions and objects, making it scalable to new data without extensive retraining.\n\nWeaknesses:\n1. Building the initial State-Action Graph (SAGE) requires the integration of multiple models, including pre-trained large language models (LLMs) and vision-language models (VLMs). This complex multi-step process of graph construction, followed by multimodal refinement, could make the system challenging to implement and maintain. Furthermore, the effectiveness of the framework heavily depends on the proper alignment between these models, which might not always be straightforward.\n2. While SAGE can learn from unlabeled instructional videos, the performance of the system may still be dependent on the quality and diversity of the data used. In particular, if the video data lacks sufficient variation in object states or actions, the model may struggle to generalize to unseen scenarios, reducing its overall robustness. In some real-world applications, high-quality labeled data might not always be available, limiting the system's ability to perform effectively.\n3. The framework’s reliance on multiple sophisticated models—such as LLMs and VLMs—requires substantial computational resources for both training and inference. The large-scale processing of multimodal data and the continuous refinement of the state-action graph could lead to increased memory usage and longer processing times, particularly in more complex scenarios or when dealing with large datasets. This could make it less suitable for enviroWhile SAGE is designed to generalize across novel objects and actions, there remains a risk of overfitting on certain specific objects or actions present in the training set. If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively when encountering new or highly diverse objects or actions in real-world settings. This overfitting issue could limit its scalability and applicability in dynamic, unpredictable environments.nments with limited computing power.\n4. While SAGE is designed to generalize across novel objects and actions, there remains a risk of overfitting on certain specific objects or actions present in the training set. If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively when encountering new or highly diverse objects or actions in real-world settings. This overfitting issue could limit its scalability and applicability in dynamic, unpredictable environments.\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for using LLMs and VLMs to recognize and classify object state changes in videos. The proposed SAGE (State-Action Graph Embeddings) first interprets each stage of the state change using short, natural language descriptions, then grounds these descriptions in vision via a pretrained VLM. The paper shows that this approach achieves good generalization to novel state changes, because these extracted visual concepts have compositional generalization capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The work takes a generalist approach to state change modeling, which is more practically meaningful than a specialist approach.\n2. The proposed approach has good interpretability, each state change recognition is supported by several visual concepts explained in natural language with associated probabilities.\n3. The method achieves a considerable performance improvement, especially on novel actions.\n\nWeaknesses:\n1. Clarity. The paper is not particularly clear in a few places, especially settings, datasets, and evaluations. It seems that different datasets have different forms of annotations, and the paper should spend more time and space explaining them clearly.\n2. The model in theory could distinguish fine-grained temporal segmentation of state changes (e.g., initial, transitioning, and end state). However, the experiments seem to only show state classification and action classification results. Is it because there is no ground truth for the former task of temporal segmentation?\n3. Does the model obtain better performance on the final benchmarks than the pseudo ground truth generation method (using VLM)? Which experiments could serve as a good reference for that information?\n\nQUESTIONS:\nPlease see the weakness section.\n\nTo summarize:\n1. Clear explanation of what ground truth is available at test time, and exactly how we evaluate each metric.\n2. Related to 1, is the current evaluation limited by groundtruth data format, and do we have any indication of state change temporal segmentation performance.\n3. How does the performance compare with the groundtruth generation method?\n\nI am open to increasing the score if these items are clarified.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a unified object state recognition framework called SAGE.
The method is built upon state-action graphs generated by large language models (LLMs), which transform complex object states into a sequence of fine-grained, linguistically-describable visual concepts. These representations are then optimized multimodally using a vision-language model (VLM) to enhance generalization to novel objects and unseen actions. Finally, a video transformer decodes state changes from video frames, enabling efficient and broadly applicable state recognition.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is clearly written and easy to follow. The motivation is well-grounded, and the proposed method is simple and reproducible.
- The framework introduces a structured modeling approach through state-action graphs, effectively leveraging the synergy of language and vision to enable a highly interpretable and extensible state recognition system.

Weaknesses:
- Although the inference efficiency outperforms multi-model ensemble approaches, scalability may still be a challenge for extremely large action sets due to the need for exhaustive matching between concepts and states.
- The current evaluation is limited to single-action video scenarios, leaving the method's effectiveness in more complex multi-action settings (e.g., real-world videos) unclear.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04