SAGE: A Unified Framework for Generalizable Object State Recognition with State-Action Graph Embedding
Authors: Yuan Zang
Zitian Tang
Junho Cho
Jaewook Yoo
Chen Sun
Venue: NeurIPS 2025 oral ()
Keywords: Object State Recognition
Graph Embeddings
Video Understanding
Primary area: applications
TL;DR: We propose a unified and generalizable framework for object state recognition in video data.

Abstract: Recognizing the physical states of objects and their transformations within videos is crucial for structured video understanding and enabling robust real-world applications, such as robotic manipulation. However, pretrained vision-language models often struggle to capture these nuanced dynamics and their temporal context, and specialized object state recognition frameworks may not generalize to unseen actions or objects. We introduce SAGE (State-Action Graph Embeddings), a novel framework that offers a unified model of physical state transitions by decomposing states into fine-grained, language-described visual concepts that are sharable across different objects and actions. SAGE initially leverages Large Language Models to construct a State-Action Graph, which is then multimodally refined using Vision-Language Models. Extensive experiments show that our method significantly outperforms baselines, generalizes effectively to unseen objects and actions in open-world settings. SAGE improves the prior state-of-the-art by as much as 14.6% on novel state recognition with less than 5% of its inference time.

SAGE: A Unified Framework for Generalizable Object State Recognition with State-Action Graph Embedding
Yuan Zang'
Zitian Tang'
Junho Cho² Jaewook Yoo?
Chen Sun
Brown University "Samsung Electronics
https://bron http/boown-pllln caalmm.gttuu..... cartuuu.ii//aa io/SAGE
Abstract
Recognizing the physical states of objects and their transformations within videos is crucial for structured video understanding and enabling robust real-world appli- cations, such as robotic manipulation. However, pretrained vision-language models often struggle to capture these nuanced dynamics and their temporal context, and specialized object state recognition frameworks may not generalize to unseen ac- tions objects. We introduce SAGE State-Action Graph Embeddings), a novel framework that offers a unified model of physical state transitions by decompos- ing states into fine-grained across different objects and actions. SAGE initially leverages Large Language Models to construct State-Action Graph, which is then multimodally refined using Vision-Language Models. Extensive experiments show that our method significantly outperforms baselines, generalizes effectively unseen objects and actions in open-world settings. SAGE improves the prior state-of-the-arr by as much as 14.6% % on novel state recognition with less than 5 5 of its inference time.
segguage-dsscrbeer visual concepts that are sharable
Introduction
If we turn the recipe book 100 Ways of Cooking Eggs (Filippini 1892) into videos, can a modern computer vision algorithm successfully understand all of them? The answer to this question is not as straightforward as might appear. On one hand, the same objects can exhibit a vast range visual appearances and physical states (e.g., a whole egg versus scrambled egg), especially when human actors interact with them. On the other hand, off-the-shel detection and segmentation systems tend to focus high-level object categories, largely overlooking their underlying physical states and dynamic transformations. We aim to develop a unified framework to jointly recognize object physical states and their temporal evolutions from visual cues, by learning from unlabeled instructional videos. We believe that understanding object states offers a powerful, object-centric abstraction for modeling world dynamics. This understanding is essential for structured video comprehension of objects, actions, and skills, and for enabling robots to perceive, predict, and plan when interacting with the physical world, thus providing promising pathway for robots to learn from human behaviors.
natural initial approach to achieve fine-grained understanding of object states might be to leverage vision- Laiiu-la models (VLMs), which provide detailed language descriptions for visual data from which object states could potentially be extracted. However, Newman et al. (2024) recently demonstrated that naive application of state-of-the-art VLMMs often fails adequately recognize object physical states. Alternatively, prior work has explored the use of instructional videos, which rich in object state transformations, enhance model training. However, these approaches often model object states as discrete categories (Souček et al. (2044), or train specialist models conditioned on actions (Souček al. (2022] Xue et al. (0024). making it challenging for them generalize to novel states belonging to unseen objects or actions. This highlights critical need for models that can learn more flexible and generalizable representation of object states and their dynamics.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Object State Recognition
Initial State
Transitioning State
End State
"saox prodile the naual tranattering statea deing
Conventional Object State Recognition
SAGE
leron, under el zing dizad annon
Spharizsishpps betbemmbiiggggllinng.... and sirpled
In-domain Training
In-domain Training
Spherical shape
Whole lernn (Initiaa State)
Initiel State
Bright yellow color
Slightly dimpledd texture
Manipulated by hand
Lemon under slicing, (Transitioning State)
Trensitioning State
Knife contacting surface
Surface deformation
Circular shape
+ End State
Juicy interior
(End State)
White pith and outer skin
Unseen Objects Actions
Unseen Objects / Actions
Circular shape
End State
Juicy interiar
(End State)
Juicyy cut surfaces
Figure 1: Illustration of state-action graph embedding (SAGE) construction and its application for object state recognition: Given an action, we ask LLM to describe the initial, transitioning, end states and their associated visual concepts. Whereas prior work recognizes states for specific actions, we recognize object states and actions through visual concepts in SAGE. It enables our model to generalize unseen objects / actions which share similar visual concepts with known ones.
We introduce SAGE (State-Actte Graph Embeddings), framework that leverages multimodal pre-trained knowledge, as in VLMs, yet is able to learn effectively from unlabeled videos, a strategy adopted by aforementioned work object state recognition. Our key inspiration is that in order to achieve generalizable object state recognition, the model needs to be specific about how a physical concept rendered visually. As illustrated in Figure [ SAGE decomposes an object state into collection of fine-grained visual concepts with language descriptions. Some of them interior) shared across objects and actions, facilitating generalization, others (e.g., white pith) are unique, capturing fine-grained nuances of various physical states for the same objects. In SAGE, individually, each action node (a verb-object pair) is connected to three types of state nodes: initial, transitioning, and end states. Each of these state nodes is, in turn, connected to a set of visual concept nodes that describe When considered together, visual concept nodes shared by different actions become connected, forming a comprehensive graph of visual concepts, states, and actions. We embed each concept node with multimodal knowledge from VLMs, where visually similar concepts are embedded nearby, even when some of them are unseen during training. An action node is represented by the direction from the initial state to end state in embedding space. We construct the initial SAGE graph using a pre- trainri Large Language Model (LLM), allowing nodes for novel actions objects to be added automatically. We then refine this graph based on multimodal information from a VLM (e.g., assessing if concept is visually recognizable) and by prioritizing concepts that are shared across a greater number of actions. Once constructed, we train a video transformer model to predict the text embeddings of these visual concepts from video frames, which subsequently decoded into a sequence object state and optionally action predictions.
juicy
We conduct comprehensive experiments to evaluate our approach existing object state recognition benchmarks. To make our observations scientifically rigorous, we carefully re-implement the baseline approaches using the same pre-trained vision encoder, and also compare with their reported results

---

for reference. We evaluate our method on Changelt and HowToChange benchmarks with known and novel objects. To further evaluate the model generalizability, we introduce a more challenging setup where both the action and the object in the video are unseen during training. We perform extensive ablation studies showing the contributions of individual design choices in SAGE. Our method demonstrates strong object state recognition performance, especially when generalized to novel objects and actions. Notably, SAGE outperforms the prior state-of-the-e (Xue et al., (024) across all benchmarks, yielding much as 14.6% relative precision improvement on state recognition for objects and actions unseen during training, and requiring less than 5% of its inference time when action is not provided during evaluation. Our project website is ittttss://// https:///brommm palm. github. io/SAGE
2 Related Work
Object states are defined as the physical and functional properties of objects (Liu et al. 2017 Newman et al.) 2024). Understanding them allows models capture the compositionality of objects and
their attributes (Misra et al., 2017: Isola et al., 2015: Purushwalkam et al.) 2019). Recognizing and localizing object states is essential for video understanding, as objects tend exhibit an even broader range of state variations (Filippini, 1892). and actions can often be represented as state transformations (Wang et al., 2016;) Fathi and Rehg 2013 Alayrac et al. 2017). Additionally, object states provide valuable cues for skill determination (Doughty et al. 2018 and goal completion (Deng
al. 2020). which are crucial for real-world tasks such as robotic manipulation (Gao et al. 2024).
Prior work has introduced foundation models (Radford et al. 2021 Alayrac et al. 2022 Jia et al.
2021 Wang al. 2022; Xu et al. (02T)) for rision-language understanding, pre-trained large- scale image/video and caption datasets to learn a unified representation of visual features and textual information. These models have demonstrated remarkable performance across various visual recognition and reasoning tasks. However, they struggle with recognizing object states in images (Newman al. (024) and videos (Souček et al. [022]] Xue al. 2024). as their training objectives often neglect object state transformations. To recognize object states in videos, prior work (Souček al. 2022 2024 Xue al. 2024) has proposed train elassifiers to predict the object states for each frame. However, these methods usually require knowing the action or object information during training and struggle with generalizing to novel actions or objects due to lacking unified representation of object states. We solve these issues by proposing State-Action Graph Embeddings to jointly represent object states and actions via visual concepts.
Visual concepts which represent the primitive features (e.g., colors) of objects have been widely utilized in visual computing. The visual concepts can enable visual models generalize composi-
tionally (Farhadi et al., 2209;; Nagarajan and Grauman 2018 [Stein et al.) 2024) and enhance their
interpretability (Koh al.] 2020 Espinosa Zarlenga al. (0022). Previous research (Menon and
Vondrick, 2022: Pratt et al. 2023: Zang et al. 2025) demonstrates that pre-trained VLMs, such as
CLIP (Radford et al.] 202TD, can learn visual concepts and perform zero-shot recognition based on
them. The concepts can be discovered by pre-trained LLMs (Yang et al. 2023 Zang et al. 2025). In
this work, we explore how visual concepts can be leveraged to represent object states.
3 Method
As illustrated in Figure 2, the input to our framework is sequence of uniformly sampled video frames encoded by a pre-trained, frozen vision encoder. The action (a verb-object pair) that causes the object state transition can be provided as input (as used by prior work), otherwise predicted by our framework. The outputs are categorical predictions for all frames, each of which belongs to one of initial state, transitioning state, end state, and background. Unless otherwise specified, follow the standard setup and assume that each video contains a single action, which can be relaxed when temporal action localization is applied as pre-processing step.
3.1 Base Model
We first contextualize the encoded visual embeddings v, from all video frames by first linearly projecting them to obtain 4,, which are fed to a temporal Transformer to produce the output embedding Z, An optional token heLs reserved for action prediction and transformed into ZCLS. All outputs projected into the discrete state action spaces, where cross-entropy losses are computed on
3

---

Where lemen
Lemosunder sRoing
Slood
Siced lemon
SAGE action embeddings
object state embeddings
Viterbi Decoding
Slicing lemon
Xax Ales)
2, RGG)
As AGJ
Zeis (la))
2, R6o)
Whole lemon
2, RRs)
2, RGG
Zeus (ls)
2, RGs)
Lemon under slicing
Peeling potato
R(Nv)
R(s)
Xrus R(s)
2, R(n
Sliced lemon
7ri5
2,
2,
Zoy-1
2r
Contextualized Video Encoder
Acus
h2
hy
Ay
Linear Projection
Vision Encoder
$
Figure 2: Overview of the training pipeline: Video frames are first encoded individually with pre-trained and frozen vision encoder, which are then contextualized with a temporal Transformer. We decode the object states by measuring the cosine similarities of the predicted embedding Ze and the possible state or action embeddings from SAGE.
all frames and the action prediction for model training. Frames that deemed as not containing the object of interest by VLM are filtered out as background. The overall training objective of our model weighted sum of the object state recognition loss and action recognition loss across all non-background frames, i.e., L = Litatte + aL
As illustrated in the left side of Figure specialist model trained to predict the object states for specific actions or objects, hence not generalizable to novel actions by design. user needs to specify the input action to select which specialist model use. generalist, on the other hand, is trained to predict all object states, even for those unseen during training. This may implemented by treating imitia//ransitioninn/eed states for all objects equivalently, which ignores the significant inter-class variations; also alternatively, by dynamically constructing the output projections (i.e., by concatenating template embeddings of all possible object states) given the action information, which is adopted by our proposed framework.
Specialist versus Generalist:
Learning from Unlabeled Videos: While each video in the training dataset is paired with video-level action label, there is a lack of frame-level annotation necessary for calculating the losses. Following the standard convention, we estimate the noisy "pseudo" object state labels by computing the cosine similarity between frame visual embeddings and state text embeddings using pre-trained, frozen VLM. We further propose to refine the pseudo labels by applying a temporal constraint (i.e.. initial state - transitioning state - final state), which is implemented as constrained Viterbi decoding algorithm (Viterbi] 1967). The decoding process is implemented through dynamic programming with restricted state transitions.
3.2 State-Action Graph Embeddings
We introduce State-Action Graph Embeddings (SAGE) to dynamically construct the state and action embeddings used by the base model to decode object states and actions for generalizable recognition. SAGE first decomposes an action into a state transformations, it then describes each state with a collection of visual concepts, forming tree structure. Intuitively, of visually distinctive concepts, which capture the nuance of fine-grained object states visually, as well as concepts shared by multiple object states, which facilitate the generalization towards unseen objects. The concept nodes shared by multiple states are merged, forming a densely-come cnnneee-coneeted graph. In addition, both the state nodes and action nodes should be embedded, that they can be used for the base model for prediction.
want to advocate the selection
To construct vocabulary of object states, we query an LLM to identify the initial, transitioning, and end states associated with each action the dataset. For example, the three object states involved in "slicing lemon" include "whole lemon", "slicing lemon". and "sliced lemon".
Graph Construction:

---

Before Refinement
After Refinement
Feach
Sloing
Clject 33000
Shur corongr.
Un Sat corcapts.
Figure 3:
The graph structure of SAGE before and after refinement. We zoom in on a subregion of the graph for demonstration. After refinement, the graph includes more shared visual concepts.
To extract the visual concepts of the object states, we leverage the LLM again to generate descriptive attributes for each object state. Given an object state and a prompt such as "What are the useful visual features distinguish a (state name) (object name) along with in-context examples, the LLM generates a set of visual concepts C, that characterize the object state.
Instead of directly encoding the object state names and action names, first derive the embeddings of all the visual concepts and then leverage them to compute embeddings for the object states and actions. Ideally, visually similar concepts should be embedded nearby in the text space, further enhancing the generalizability of our framework. As such, we first encode each visual concept as a distributed embedding using the text encoder Exest of pre-trained VLM. We then represent an object state as the average of its visual concept embeddings, i.e., R(s)) = TC.7 Eoec, (c), Since visual concepts are shared across different object states, this formulation provides unified embedding space, enabling our model to generalize across various object state transformations, including unseen states. Inspired by prior work on relation encoding (Bordes et al., 2013) and state-based action recognition (Wang et al. 2016), we compute the action embeddings as the difference between the embeddings of its end and initial object states, which naturally accounts for object-specific variations, as the same action might have different visual features for different objects (e.g. "boiling egg" and "boiling pasta").
Node Embedding:
By integrating object states and actions within the state-action graph, our approach captures both their visual attributes and dynamic transformations. The proposed SAGE embeddings enable a structured representation of actions while preserving the compositional nature of object states. This facilitates a more generalizable understanding of object state transitions across different actions.
Multimodal Graph Refinement: We propose refining the graph structure incorporate visual concepts that are reliably recognized by the model to enhance object state recognition accuracy, as well as concepts that are commonly shared across different object states to improve generalizability. Given an object state, we first generate an over-complete concept list with an LLM. We then rank these concepts based on their VL similarity scores with the video frames containing the object state. For concept the score is calculated as |F ΣtEF
Fueu (c), 7 where F is the set of video
frames containing the object state. We select the top-ranked ones they most reliably recognized by the model. To further promote generalizability, we prioritize concepts that are shared across multiple object states. We enforce that at least half of the selected concepts are shared. Specifically, to select k concepts, we first select the top-ranked shared concepts until A are selected and then select remaining top-ranked concepts. In practice, we select the top concepts for each object state and ensure that at least 3 of them are shared. We then construct new graph with the selected visual concepts. An illustration SAGE before and after refinement shown Figure
3.3 Inference
We follow a two-step inference which first predicts a video-level action label, and then decode the object states with SAGE graph. The first step skipped when the action provided as input.
Action Recognition: We first identify the object of interest the video. Each object is represented by the average embedding of all its associated states in SAGE. We then compute the similarity
5

---

between the object embedding and the embeddings of all video frames. The object of interest is selected as the one with the highest cumulative similarity the video frames. Next, we predict the action in the video. For all actions potentially associated with the identified object based on the SAGE knowledge graph, we compute the similarity between each action embedding and the video-level embedding ZCLS, and select the action a" with the highest similarity score.
For each frame we compute the similarities between its representation Ze and the state embeddings R(s), and then probability distribution over object states after softmax. The only difference is that we now narrow our scope of object states to only the initial, transitioning, and end states of the action a* instead of all the object states in S. SAGE. We then apply the Viterbi decoding over the state predictions to predict the object states following the temporal order in SAGE.
Object State Recognition:
4 Experiment
4.1 Experimental Setup
Datasets: We evaluate our method on two object state recognition benchmarks, Changelt (Souček) et al.) 2022) and HowtoChange (Xue et al., 2024), which consist of videos depicting single actions. The task is to predict the physical state of the object of interest in every frame given the action. While SAGE can support state recognition from multi-action videos, by replacing the video-level action classification step with temporal action localization, to the best of our knowledge there is no publicly available benchmark with multiple objects and actions for evaluation purposes.
Baselines: We compare our method with FookForTheChaapg (LFC) (Souček et al. 2022). Multi-
TaskChange (MTC) (Souček et al. 2024) and VidOSC (Xue et al. 2024). which are the state-of-the-an methods for object state recogmition in single-action videos. LFC and VidOSC train separate special- ized models for different objects/ actions. MTC trains a unified model for all objects and actions given a known vocabulary. We also evaluate zero-shot VLMs, including CLIP (Radford et al. 2021 VideoCLIP (Xu et al. 2021), and Intern Video (Wang et al. 2022) for object state recognition.
To ensure a fair comparison, we use the same pre-trained VLM, CLIP ViT-L-14 (Radford et al. 2021 which is fine-tuned with the pseudo labels and videos from HowtoChange. as the vision backbone and pscudo-label generator for both the baselines and our method. The fine-tuning helps our method better recognize the objects and actions from videos, by utilizing noisy, automatically generated supervision from the speech modality, which we hope can improve the generalization performance at object state level. To understand the performance benefits when a better sideo-language model is used, we also train our model with VideoCLIP and compare it with the reported performance of the baselines (uue et al., 2024 Souček et al. 2022 2024) in Section 4.5
Evaluation Metrics: following/Součeke et al. 2022 2024), we evaluate the methods in Precision @ 1 on the Changelt dataset. In this dataset, state precision refers to the precision of initial and end states, while action precision corresponds to the precision of transitioning states. We rename the metric as Trans. Pre @ avoid confusion. On the HowToChange dataset, we follow Xue al. (024) and evaluate the F-1 score, Precision, and Precision @ 1. See the Appendix for their detailed definition. While the object and action ground truth labels are available in the test set, they are optionally provided to the model according to the evaluation setup. The use of ground truth action and object annotation as privileged information is indicated in Table When not provided, our method infers them directly from videos as described in Section 3.3 For the remaining tables, we follow the protocol from Xue et al. (2024) where action information is provided but the object label not.
We sample video frames at 1 FPS as in baselines (uue et al. 2024 Souček et al. 2022 (2024). Frame embeddings are extracted by the vision encoder of the fine-tuned CLIP ViT-L-14 14 (Radford et al. (222]) (except in Section 4.5) and projected by a linear layer. We then process them using a three-layer Transformer (5 1i1memmmnsio hidden states, four attention heads) followed by another linear projection layer to obtain frame representations. The action loss weight is 0.1. We train the models using the Adam W optimizer with 1c-4 learning rate and 1c-4 weight decay. We use batch size of 32 on Changelt and 128 on HowToChange. We train our model for 10 epochs on each dataset. These hyper-parameters are optimized according to validation performance. In the state-action graph construction, we use OpenAl GPT-4o-mini 20224-07 -7818 as the LLM to generate visual concepts for each object state. We also experiment with an open-weight model, Qwen3-32B, and find the performance differences negligible compared to GPT-4o.
Implementation:

---

Table 1: Known object state recognition performance Changelt and How ToChange. SAGE outperforms the baselines across different settings. The unified versions of baseline models suffer substantial performance degradation, whereas model not only mitigates this issue but even surpasses specialized baseline models. *; Unified version implemented by us.
Privileged Info
Unified Model
Changelt
How HooaaCCh
Methods
Action
Object
State Pre@1
Trans, Pre@1
F1
Pre
Pre
CLIP (Radford al. 2021
0.30
0.63
0.27
0.27
0.48
VideoCLII XU ct al. 2021
0.33
0.59
0.37
0.40
0.48
Intern Video Waang
2022
0.27
0.57
0.30
0.31
0.47
LFC (Součck ct al. 2022
x
0.30
0.63
0.30
0.30
0.36
SAGE (ouns)
/
0.57
0.85
0.39
0.45
0.58
VidOSC (uue et al. 2024
/
x
x
0.52
0.83
0.37
0.40
0.53
SAGE (ours)
X
0.53
0.83
0.37
0.42
0.55
LFC
X
X
0.25
0.52
0.24
0.26
0.31
VidOSC
X
X
0.41
0.67
0.29
0.32
0.42
MTC (Souček et al. 2024
X
X
0.47
0.75
0.32
0.35
0.45
SAGE (ours)
X X
0.51
0.81
0.34
0.39
0.52
Table 2: Open-workd evaluation results on Changelt and HowtoChange. The MTC method cannot generalize to novel actions or objects because its classification heads are fixed for the states of known objects. SAGE shows robust performance in the open-world setting, while baseline models degrade significantly on unseen actions or objects. *: Results from the best specialized models.
(a) Evaluation with known actions and novel objects.
(b) Evaluation with novel actions and novel objects.
Changelt Novel Obj
HowtoChange Novel Obj
Changelt Novel Obj & Act
HowtoChunge Novel Obj & Act
Methods
Methods
State Prote
Proff
Preff
Proff
Trans. Pref1
LFC*
0.25
0.54
0.27
0.27
0.32
LFC
0.21
0.48
0.23
0.22
0.27
VidOSC
D.43
0.71
0.32
0.35
0.48
VidOSC*
0.27
0.59
0,25
0.28
0.37
SAGE (ours)
0.49
0.78
0.34
0.39
0.50
SAGE (ours)
0.45
0.70
0.31
0.35
0.45
Model Training and Time: We train the model with 8x NVIDIA V100 GPUs. It takes 30 minutes to extract visual embeddings for How TowToChange and 3.5 hours for Changelt. The training takes about 30 minutes for HowToChange and hours for Changelt.
4.2 Evaluation on Object State Recognition
We evaluate our method and baselines on both known and novel objects and actions. Prior work (Souček et al., 2022, 2024 Xue et al.) 2024) trains separate specialized models for different objects or actions, and thus struggles to generalize to novel objects and actions. In this work, SAGE enables us to train a unified model for all objects and actions and generalize to novel objects and actions.
As shown in Table naively extending baseline methods into unified models results in significant performance degradation. We reimplement the unified versions of the baselines by putting state prediction heads for all known actions and objects and on shared backbone. With SAGE, our unified model with comparable number of parameters significantly outperforms the generalist baselines, and even outperforms the specialists. More importantly, while baseline models require the privileged information of actions and objects as inputs, our model can make precise object state recognition without knowing the action and object in the video during evaluation.
Known Objects and Actions:
We evaluate our model with novel objects as explored in prior work, and propose a more challenging setup where both the action and the object in the video are novel. Among the baseline methods, LFC and MTC cannot generalize novel objects and actions because they rely on fixed- -xed-dimension classification heads trained specifically for seen object state transformations. Similarly. VidOSC cannot generalize to novel actions. To estimate their generalization ability and make a comparison, we follow Xue et al. (2024) and measure their performance upper bounds by enumerating all of their specialist models and pick the one with the best performance using the ground truth labels. 2 the model performance model maintains comparable performance on unseen objects and actions as it does on seen ones, whereas baseline models suffer significant performance drops.
Novel Objects and Actions:
novel objects and novel actions. Our
7

---

Table 3: Parameter numbers and inference runtime of different methods on the HowtoChange dataset.
When actions and objects are unknown, the specialized methods such as LFC and VidOSC must run all expert models and select the best one, which significantly increases the computational cost.
Mothods
#Paramns per Model
&Modles
Runtime w/ Known Actions and Ofjects Runtime w/ Unknown Actions and Objects (8)
LFC (Smukkk 2022
4.2M
18.2
33.8
MTC Soowe al 2027
I.
24.5
24.5
SAGETOUIS)
10.9
29.7
29.7
Table 4: Comparison of SAGE before and after refinement on How ToChange.
Known
Novel
Graph
# Concepts
F1
Pre
Pre@1
F1
Pre
Pre@)
SAGE (before refinement)
0.37
0.42
0.55
0.31
0.35
0.45
SAGE (hefore refinement)
15
0.36
0.42
0.54
0.30
0.34
0.44
SAGE (after refinement)
0.38
0.42
0.58
0.33
0.39
0.50
We compare the parameter numbers and runtimes on the HowToChange dataset of our model and baselines. We compare their inference time on 8x NVIDIA V100 GPUs. As shwon in Table 3] when the privileged information actions and objects is not provided, our method show significant advantages in inference efficiency. Unlike specialized baselines that must run all specialized models and select the best one (following Xue et al. (2024)) our approach uses a single general model for all actions and objects, resulting in substantially reduced computational cost.
Efficiency Analysis:
These findings demonstrate that our method enables the training of foundation models for object state recognition, paving the way for more scalable and generalizable solutions.
4.3 SAGE Graph Refinement
We evaluate our SAGE refinement strategy for both known and novel object states. To validate the effectiveness of our concept selection method, we compare the performance of original SAGE, SAGE with over-complete concepts and the refined SAGE with selected concepts. As shown in Table 4 the proposed refinement significantly improve the generalizability of SAGE.
In Figure we illustrate the local SAGE graph structures of five object states before and after refinement. The refined graph incorporates more visual concepts that are shared across different object states, enabling the model recognize novel object states by leveraging common concepts learned from known object states. Especially, before graph refinement, concepts that should be shared by multiple states might not be correctly assigned these states (e.g., "juice seep out" in Figure 3)
Table 5: Ablation studies on different technical designs in our method. We report the Pre scores and their performance differences (A) compared to the complete model.
Changelt
HowTnChaange
Methods
Pre@ (A)
% Pre@1 (A)
Soen
Novel Obj
Novel Obj Act
Soen
Novel Obj
Novel Obj & Act
W/o Textual Representation
49.8 (-11.6)
N/A
N/A
(-8.4)
N/A
N/A
W/o Visual Deseriptions
(-4.6)
50.3 (-6.8)
48.4 (-7.0)
50.7 (-4.1)
42.9 (-6.8)
(-6.5)
W/o Joint Training
62.3
(-2.()
47.9 (4.1)
55.5 (40.7)
(-1.5)
(-344)34) 41.3
W/o Dynamic State Locating
(-2.()).5)
(-3.4)
488 (3.2)
(1..0
(-..3)
4.4 Ablation Studies
We analyze the effectiveness of our proposed method through ablation studies by removing each component of SAGE at a time. The results are shown in Table
We remove the text embeddings of object states by not using the cosine similarity between language and vision representations for state recognition. Instead, we treat object states as discrete categories. The results indicate that removing textual representations significantly degrades model performance and disables it to work on unseen objects and actions.
State Text Embedding:
8

---

Table Evaluating SAGE with different VLMs. We observe that our approach is reasonably robust with respect to the choice of vision and text encoders.
Changelt
HowtoChange Pre
State Pre
Action Pre@e
F1
Pre
SAGE + CLIP
0.51
0,81
0.34
0.39
0.52
SAGE + SigLIP
0.53
0.82
0.34
0.42
0.56
SAGE + MetaCLIP
0.54
0.82
0.36
0.43
0,56
Table 7: Comparison between model with VideoCLIP pre-trained VLM and the reported
results of baselines. Our model achieves slightly better performance than all baselines on seen objects and significantly outperform all baselines on novel objects.
Changelt
Changelt (Open-world)
How ToCangee
Methods
State Pre
Action Pre@1
State Pre@1 I
Trans. Pre@
Fl
(%)
Precision (%)
Pre@!
seen
novel
seen
novel
seen
novel
seen
novel
seen
novel
LFC Souček et al. 2022
0.35
0.68
0.36
0.25
0.77
0.68
30.3
28.7
32.5
30.0
37.2
36.1
MTC Soutek et al.
2024
0.49
0.80
0.41
0.22
0.72
0.62
33.9
29.9
38.5
34.1
43.1
38.8
VidOSC Xie cl 2024
0.57
0.84
0.56
0.48
0.89
0.82
46.4
43.1
46.6
43.7
60.7
58.2
SAGE (Ours)
0.60
0.89
0.59
0.55
0.89
0.87
46.4
44.7
47.5
46.3
63.6
61.2
Visual Concept Descriptions: We replace the visual concept descriptions with object state names to obtain the state embeddings, and keep the pseudo labels consistent for fair comparison. The results show that removing visual concepts leads to the largest performance drop unseen objects and actions. This suggests that visual descriptions provide crucial reasoning cues for recognizing object states, especially when generalizing novel objects and actions.
Jointly Training with Action Recognition: We remove the action recognition objective during train- ing. Although this objective does not improve seen state recognition, it enhances the generalizability to unseen objects and actions. This is because unseen object states and actions may share similar relationships to seen ones, which could be learned by joint training with action recognition.
Viterbi Temporal Decoding: Instead of training the model with pseudo labels decoded with temporal constraints, we train it with pseudo labels generated by CLIP as in Xue et al. (2024). The results show that removing Viterbi decoding in pseudo label preparation leads to a significant drop in performance, suggesting that temporal constraints helps improve recogmition, particularly for generalization.
Finally, explore the use of different vision-language models to showease the robustness of our method. We adopt the visual and text encoders from SigL.P (Zhai al. 2023), which enhances CLIP's training objective, and from MetaCLIP (Xu et al. (2024). which refines CLIP's ' training data quality. We evaluate in the most challenging setting where both the action and object information are unknown. Table we observe moderate improvements when SigLIP and MetaCLIP are used, indicating the image encoder is not the main performance bottleneck.
Vision-Language Models:
4.5 Comparison with Reported SOTA Results
For thorough comparison, train our model by preplacing CLIP with VideoCLIP adopted by prior work, and compare SAGE with the reported results in baselines (Souček et al., 2022 2024 Xue al. 2024). Table shows that our unified model can perform consistently better than the reported state-of-the-an results from specialized models.
4.6 Qualitative Results
We demonstrate in Figure 4 the top-1 frame predictions of our model for videos with known and novel objects and states. For each video, we identify the top-1 frame for the initial, transitional, and final object states by selecting the frame with the highest embedding similarity to the corresponding state. We also display the visual concepts with the highest embedding similarity to each top-1 frame. The results suggest that our model can accurately recognize the object states for both known and novel objects and actions.

---

Known Objects/Acttons
Novel Objects/Actions
Indol State
Transt Stato TrAaEAA Cullnn
End State
Intial State
Transidoring State Dutlen Collen
End State
Call Thooaads
Clitring Durte
Top
Predicted
Gys colicciclllo ages
Moidd gleary supeoriice
Bully sita www.la.ltlblrdddd
sancadalige
Druggna www.eefuiluoeee
Vargaewonndee
PrelodMarguree
Pooling Orogon JNuR
Paul ni Hingssaeen
Cakefped
Soad
Pecskesilytewh
Predicted Consects
Faceried
Mirn bu hands
Wete funh oppeed
Tiik kellerydle
mids fisturpied
Whete Rlsterrene
peled fioe
equaed
Mictawce
SeapPople
Coleccrran
Detached Fomithereed
Intial State Choootae Bor
Transhioning State Choooiate under Orrking
End State
Inittal State
Transtroning State Gooaest unde Berscin
End State Sosaper Cocorut
Choooiss
Ureespoo Gononal
Top Concepts
pirtiii
Deestarvn
Wheki uP Tack
Cener ahstoiiiggggooo filligg
Shertin e' bettre
Die jassen
Siadle
Teaerdibb puricles
Whice nurd
Fastyrikks
The andifaly perticees
Intial State
Tring State Piweappll under Culting
End State Cal
Intial State Whole Duten
TransiBoring Stato Dulla rockr Cutling
End State Cct Dariat
Whole Frmssppla
Top Predicted Consepts
Whit paze
TNE ped DU
colcoracing ALEIOE
Figure 4: Examples of the top-1 frames and top-aligned visual concepts predicted by our model for the initial, transitional, and end states in the videos. In the action and object labels, text in green indicates known objects or actions and text in red indicates novel ones. In the visual concepts, the green concepts are shared across different object states while the red concepts are wrongly predicted.
5 Conclusion
In this paper, we propose a novel framework build unified models for recognizing object states in videos. We leverage pre-trained LLMs and VLMMs to build State-Action Graph Embeddings (SAGE) that decomposes object states into visual concepts. The graph structure where different objects and actions share the same visual concepts enables the model to generalize to novel objects and actions. Our model outperforms all baselines on two widely used object state transformation benchmarks, especially for open- world settings where both objects and actions novel.
Limitations: Our approach evaluated on Changelt and How ToChange benchmarks, both contain single action in each video. Although SAGE may be naturally generalized handle vidcos with multiple non non-overlapping actions via temporal action localization, cannot directly support the scenario where multiple objects are undergoing state transformations concurrently.
We further envision two natural directions for future work: First, despite our effort to build a unified model for diverse objects and actions in different scenarios, our model is designed solely for object state and action recognition. Integrating the success of SAGE into a vision foundation model would be ideal as recognizing object physical state is fundamental task of visual perception. Second, it would be highly impactful to apply our approach for downstream tasks such as tracking objects that undergo temporal abstraction and skill discovery for robotic applications.
state transformations,
10

---

Acknowledgments
We would like thank Calvin Luo, Nate Gillman, Shijie Wang. Tian Yun, and Zilai Zeng for helpful discussions. This project was supported by Samsung. Our research was conducted using computational resources at the Center for Computation and Visualization Brown University.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736.
Bean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. 2017. Joint discovery of object states and manipulation actions. In Proceedings of the IEEE International Conference on Computer Vision, pages 2127-2136.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data, Advances in neural information processing systems, 26.
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. 2020. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE.
Hazel Doughty, Dima Damen, and Walterio Mueol-Cuevas. 2018. Who's ' better? who's best? pairwise deep ranking for skill determination. Proceedings of the IEEE conference on computer vision and pallern recognition, pages 6057-6066.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. 2022. Concept embedding models: Beyond the accuracy-explainablle trade-off. Advances in Neural Information Processing Systems, 35:21400 1400-11410.......
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In 2009 IEEE conference computer vision and pattern recognition, pages 1778 1785. IEEE.
Alireza Fathi and James M Rehg. 2013. Modeling actions through state changes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2579-2586.
Alexander Filippini. 1892 One Hundred Ways of Cooking Eggs. 1. CL Webster.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. 2024. Physically grounded vision- vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462-12469. IEEE.
Phillip Isola, Joseph Lim, and Edward Adelson. 2015. Discovering states and transformations in image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383-1391.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- lsionnlanguage representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020. Concept bottleneck models. In International conference on machine learning, pages
PMLR.
Yang Liu, Ping Wei, and Song-Chun Zhu. 2017. Jointly recognizing object fluents and tasks in egocentric videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2924-2932.
11

---

Sachit Menon and Carl Vondrick. 2022. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations.
Ishan Misra, Abhinav Gupta, and Martial Hebert. 2017. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1792-1801.
Tushar Nagarajan and Kristen Grauman. 2018. Attributes as operators: factorizing unseen attribute- object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 169-185.
Kaleb Newman, Shijie Wang. Yuan Zang, David Heffren, and Chen Sun. 2024. Do pre-trained vision-language models encode object states? arXiv preprint arXiv:2409, 10488.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. 2023. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15701.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. 2019. Task- driven modular networks for zero-shot compositional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3593-3602.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 8763. PmI.R.
Tomá's Souček, Jean- Saan-Baptiste Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. 2022. Look for the change: Learning object states and state- modiifind actions from untrimmed web videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Tomáš Souček, -aptistee Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. 2024. Multi-task learning of object states and state-modifying actions from web videos IEEE Transactions on
Puttern Analysis and Machine Intelligence.
Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. 2024. Towards compositionality in concept learning. arXiv preprint arXiv:2406. 18534.
Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Iransactions on Information Theory, 13(2):260-269,
Xiaolong Wang, Ali Farhadi, and Abhinav Gupta. 2016. Actions" transformations. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2658-2667.
Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. 2022. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:22/2.03199)
Hu Xu, Gargi Ghosh, Ya- Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787-6800.
Hu Xu, Saining Xie, Xiaoqing Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang-Wan Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. 2024. Demystifying clip data. In The Twelfth International Conference on Learning Representations.
Zihui Xue, Kumar Ashutosh, and Kristen Grauman. 2024. Learning object state changes in videos:
An open-worle perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18493-18503.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch and Mark Yatskar. 2023. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-19197.
12

---

Yuan Zang, Tian Yun, Hao Tan, Trung Bui, and Chen Sun. 2025. Pre-trained vision-language models learn discoverable visual concepts. Transactions on Machine Learning Research.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVE international conference computer vision, pages 11975-11986.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We discuss in detail the contribution and scope of this paper in the Abstract and Introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose the implementation details including model architectures, hyper- parameters and pre-trained models we used in Section 4.1 Researchers can reproduce our results with the provided information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit the code in supplementary documents with the README file describing the steps to reproduce the results. We use the public datasets for experiments. The link of these datasets are provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
publicc/guudes//ddetttmiii wtw.eeddsubmmssssooooooocoo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//npps.cc/pulli//uidiee
wwtsssssssiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the hyperparameters and optimizers information in Section 4.1 We provide the details of the datasets in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We perform the Student T-test to evaluate the statistical significance of the main results. The results are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the compute resources we use for training and inference in Section and Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics. It does not include human subjects. The datasets publicly available and do not have ethic concerns. The research does not have potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss in the introduction how the proposed method can potentially benefit real-world machine learning applications such as robotics. We do identify any potential negative impacts the proposed method can bring.
17

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper uses pre-trained Vision-Language models and Large-Language Models. All of these models are properly cited. The datasets this paper uses are publicly available and properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/datases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We describe in detail how we use the LLM to construct the core component (SAGE) of this model in Section 3.2
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) (ttpre///eerrpppccc///fffeeeeeee cc///oofeeeennes/2225///... for what should or should not be described.
20

---

A Evaluation Metrics
On the Changelt dataset, we follow Souček et al. (2022 2024) and report State Precision@ for initial and end states, and Transition Precision @ for the transitioning state. Note that the transition precision was referred to as action precision in prior works (Souček et al. 2022 2024). We rename here to avoid confusion. For each of the intiaa/raanstioonng/eed states, the video frame with the top one probability predicted by a model is retrieved, where the precision is calculated as the percentage of corrected retrieved frames across all videos. Since the frame sampling strategy would affect the collection of candidate frames, we follow the same 1-FPS uniform sampling strategy as prior work so that the results comparable.
Additionally, on the How ToChange dataset, we follow Xue et al. (2024) to evaluate the F-1 score, Precision, and Precision@ @1 1. The definition of Precision @ is the same as Precision @ in Changelt, except that the initial/ransitiininn/nmm states are jointly considered in this metric. Since Precision only considers the top retrieved frames, F-1 score and Precision are used, both of which over all sampled videos frames. We calculate the F-1 score and Precision for each of the initial, transitioning, and end states and report their average over the three states. Similarly, use the same frame sampling strategy since the metries are defined with respect to all sampled frames.
computed
B SAGE Construction Details
We construct SAGE by querying the LLM to first provide the name of initial, transitioning and end states for action and then provide the visual concepts for each state. In practice, we use OpenAl 11-40--mmi-222--00-0
as the LLM and add in-context examples in the prompts to guide it. In the following, we provide an instance of the prompts for generating the states and visual concepts. We skip the in-context examples below for conciseness.
Prompt template for generating state names:
Q: What are the initial, transitional, and end states of a(n) {object] during the action {action)?
For example:
Q: What the initial, transitional, and end states of lemon during the action slicing lemon?
A:
whole lemon
lemon under slicing
sliced lemon
Prompt for generating visual concepts:
Q: What are the visual features for distinguishing a(n) (initial state name), ann) (transitional state name] and (n) (end state name}?
For example:
Q: What are the visual features for distinguishing whole lemon, lemon under slicing and a sliced lemon?
A:
Whole lemon:
spherical shape
bright yellow color
slightly dimpled texture
smooth surface
evenly distributed color
Lemon during slicing:
manipulated by hands
21

---

Action: Slicing lemon
initial, transitional [slicing] [lemon].
LLM #
Initial state Whole lemon
Transitioning state Lemon under Slicing
End state Sliced lemon
Provide the useful visual features to distinguish [whole, slicing, sliced] [lemon].
Uniformed yellow
Knife contacting surface
Circular shape
Spherical shape
Surface deformation
Juice vesicles
Slightly dimpled texture
Visible juice
White pith and outer
Figure A1:
Illustration of how the state- actton graph is constructed for single action slicing lemon
with frozen LLM. See details in Section B
knife contacting surface
surface deformation
juice beginning to escape distinct cut line forming
Lemon after slicing:
circular shape
juicy interior
white pith and outer skin
exposed flesh glistening
broken surface texture
We obtain the object state names and visual concepts by parsing the LLM generated answers and collecting the visual concepts into a list. We observe that the answers consistently follow the in-context examples individual concepts further merge similar concepts according to the CL.P text emheddings. Two concepts are considered as similar if their cosine similarity over the text embeddings higher than 0.9. We merge greedily until no concepts have similarity higher than 0.9. We use the pre-trained, frozen CLIP ViT-L-14 text encoder, and normalize the extracted embeddings. As discussed in the method section, object state embeddings are calculated by averaging their corresponding visual concept embeddings. We normalize the state embedding again after taking the average.
be extracted by splitting
the dash ("-") sign. We
C Dataset Details
We provide the details of the Changelt and How HowToChange datasets in Table A8 These datasets consist of instructional videos for daily tasks beyond cooking (e.g., dyeing T-shirt). HowToChange contains diverse set of objects, including subset of novel objects that do not appear in the training set, making it suitable for evaluating the generalization ability of models. Changelt features wider variety of actions and longer video sequences.
22

---

Table A8: Statistics Changelt and How ToChange datasets.
Datasets
#Objects
#Actions
#Videos
#Training
#Evaluation
Avg. Duration (s)
Changelt
27
35,095
34,428
HowToChange
134
20
41,499
36,075
5,424
41
Table A9: Statistical significance of the object state recognition performance between our method and the baseline methods. We report p-values and statistical significance. Results marked with v indicate statistical significance (p < 0.05).
Matrodd
Spanc
bodb mr)
3.5-37
21c-21 (/
21edd
2 (3341)
3..01
1.0e000
93
18447
144-3314)
D Statistical Significance
We conduct Students' T-test to evaluate the statistical significance of the result of our main experiments in Table 6. The results are shown in Table A9 Our model significantly outperforms LFC and MTC for both known and novel objects and significantly outperforms VidOSC for novel objects.
23


Reviews:
[{"review_id": "DSv7BfHhbR", "reviewer": "Reviewer_K9Kj", "summary": "This paper proposes a unified object state recognition framework called SAGE.\nThe method is built upon state-action graphs generated by large language models (LLMs), which transform complex object states into a sequence of fine-grained, linguistically-describable visual concepts. These representations are then optimized multimodally using a vision-language model (VLM) to enhance generalization to novel objects and unseen actions. Finally, a video transformer decodes state changes from video frames, enabling efficient and broadly applicable state recognition.", "questions": "Please see the weaknesses.", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and easy to follow. The motivation is well-grounded, and the proposed method is simple and reproducible.\n- The framework introduces a structured modeling approach through state-action graphs, effectively leveraging the synergy of language and vision to enable a highly interpretable and extensible state recognition system.\n\nWeaknesses:\n- Although the inference efficiency outperforms multi-model ensemble approaches, scalability may still be a challenge for extremely large action sets due to the need for exhaustive matching between concepts and states.\n- The current evaluation is limited to single-action video scenarios, leaving the method's effectiveness in more complex multi-action settings (e.g., real-world videos) unclear.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n### Q1: Scalability for extremely large action sets\n\nWe would like to clarify that exhaustive matching between concepts and states is **not needed** during model inference. As illustrated in Figure 2, we first compute the embedding similarity to identify the video-level action label (unless the video-level action label is given as input) and then the object state recognition is restricted to the three object states and their corresponding concepts related to this action. We observe that recognizing video-level actions does not impose computational overhead since it only needs to be performed once per video (and the action space is moderate, since it is not combinatorial with states and concepts).\n\n---\n\n### Q2: Multi-action evaluation\n\nSAGE can technically be generalized to support multi-action videos by replacing the video-level action classification step with a temporal action localization / segmentation module.\nWe agree that evaluations under multi-action settings would further demonstrate the effectiveness of our approach. Unfortunately, to the best of our knowledge there is no publicly available benchmark for recognizing object states from videos with multiple actions and objects. The most relevant existing dataset is Multiple Object States Transition (MOST) by Tateno et al., recently published in WACV 2025. This benchmark still assumes a single object is being manipulated within a video, but expands the list of object states to include those not strictly tied to actions (for example, an egg can be both “cracked” and “raw” after being cracked. In SAGE, we treat “raw” as a possible visual concept)."}, {"review_id": "DoPzghwh0i", "reviewer": "Reviewer_hicx", "summary": "This paper proposes a method for using LLMs and VLMs to recognize and classify object state changes in videos. The proposed SAGE (State-Action Graph Embeddings) first interprets each stage of the state change using short, natural language descriptions, then grounds these descriptions in vision via a pretrained VLM. The paper shows that this approach achieves good generalization to novel state changes, because these extracted visual concepts have compositional generalization capabilities.", "questions": "Please see the weakness section.\n\nTo summarize:\n1. Clear explanation of what ground truth is available at test time, and exactly how we evaluate each metric.\n2. Related to 1, is the current evaluation limited by groundtruth data format, and do we have any indication of state change temporal segmentation performance.\n3. How does the performance compare with the groundtruth generation method?\n\nI am open to increasing the score if these items are clarified.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The work takes a generalist approach to state change modeling, which is more practically meaningful than a specialist approach.\n2. The proposed approach has good interpretability, each state change recognition is supported by several visual concepts explained in natural language with associated probabilities.\n3. The method achieves a considerable performance improvement, especially on novel actions.\n\nWeaknesses:\n1. Clarity. The paper is not particularly clear in a few places, especially settings, datasets, and evaluations. It seems that different datasets have different forms of annotations, and the paper should spend more time and space explaining them clearly.\n2. The model in theory could distinguish fine-grained temporal segmentation of state changes (e.g., initial, transitioning, and end state). However, the experiments seem to only show state classification and action classification results. Is it because there is no ground truth for the former task of temporal segmentation?\n3. Does the model obtain better performance on the final benchmarks than the pseudo ground truth generation method (using VLM)? Which experiments could serve as a good reference for that information?", "was_revised": true, "final_justification": "Overall I do believe this paper is meaningful to the community with some imperfections, for example, the writing and presentation. My final rating is borderline accept.", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n### Q1: Clarity on evaluation setups\n\nAt test time, we sample video frames at 1 FPS and feed them to the model. While the object and action ground truth labels are available in the test set, they are **optionally** provided to the model according to the evaluation setting. When the object and action labels are not provided, our model predicts them according to our method in Section 3.3. In Table 1, checkmarks in the Privileged Info column indicate whether object and action labels are provided. For Table 2, 4, 5, and 6, we follow the standard protocol from prior work (e.g. VidOSC), where action information is provided but the object label is not. Given the information above,the model predicts one of the following labels for each sampled video frame: initial, transitional, end, or background.\n\nTo evaluate model performance, the test sets have frame-level annotations. Each label is one of  initial, transitional, end, or background. Following the implementation detail of evaluation metrics in Appendix A, we compute State Precision@1 and Transition Precision@1 on the ChangIt dataset and Precision, F-1 score, and Precision@1 on the HowToChange dataset. Note that we use different metrics on two datasets because previous works did so and we want to compare with them. This is **not** because two datasets have different forms of annotations.\n\nWe will include these clarifications in the revised version. \n\n--- \n### Q2: Why is temporal segmentation not evaluated?\n\nWe follow the standard evaluation protocols as proposed by each of the original datasets. The original evaluation protocol was introduced by Alayrac et al. in ICCV 2017, where the authors described that “a temporal action localization is said to be correct if it falls within the ground truth time interval”. In practice, the metric is more robust in the presence of background frames that frequently occur in the videos.\n\n---\n\n### Q3: Pseudo ground truth generation method as a baseline?\n\nWe would like to clarify that all approaches, including SAGE and the baselines, used pseudo ground truth generation as part of their pipelines. We note that there are two approaches where pseudo labels are used, as shown in Table 1:\nCLIP, VideoCLIP, and InternVideo are VLMs that directly predict pseudo labels on the **test videos**. This setup is usually referred to as “zero-shot” classification.\nSAGE, LFC, MTC, and VidOSC all use VLMs to generate pseudo labels on the **training videos**. These videos are then used to train machine learning models that make predictions on the test videos.\n\nTo answer the reviewer’s particular question: If the goal is to understand how SAGE outperforms VLMs that generate pseudo labels on the test videos, then CLIP/VideoCLIP/InternVideo in Table 1 are the proper baselines. \n\nIf the goal is to understand how SAGE outperforms models trained on videos with pseudo labels. Table 1/2/3 show the relevant comparisons, where we carefully control the pseudo label generator to be the same (CLIP ViT-L-14). Table 6 shows the comparison where the baselines can choose their own pseudo label generators (e.g. VideoCLIP, which was trained on HowTo100M)."}, {"review_id": "No92haKb5c", "reviewer": "Reviewer_LFZv", "summary": "This paper introduced SAGE (State-Action Graph Embeddings), a framework that improved object state recognition in videos by breaking down states into detailed visual concepts. The proposed method combined large language models and vision-language models for better generalization to new objects and actions. SAGE outperforms existing methods by up to 14.6% in recognizing novel states while being more efficient.", "questions": "Please see the Weaknesses.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. SAGE outperforms existing methods by generalizing well to novel objects and actions, significantly improving object state recognition, even in open-world settings.\n2. The framework combines vision-language models (VLMs) and large language models (LLMs), leveraging multimodal pre-trained knowledge for more accurate state recognition.\n3. The framework allows for automatic addition of novel actions and objects, making it scalable to new data without extensive retraining.\n\nWeaknesses:\n1. Building the initial State-Action Graph (SAGE) requires the integration of multiple models, including pre-trained large language models (LLMs) and vision-language models (VLMs). This complex multi-step process of graph construction, followed by multimodal refinement, could make the system challenging to implement and maintain. Furthermore, the effectiveness of the framework heavily depends on the proper alignment between these models, which might not always be straightforward.\n2. While SAGE can learn from unlabeled instructional videos, the performance of the system may still be dependent on the quality and diversity of the data used. In particular, if the video data lacks sufficient variation in object states or actions, the model may struggle to generalize to unseen scenarios, reducing its overall robustness. In some real-world applications, high-quality labeled data might not always be available, limiting the system's ability to perform effectively.\n3. The framework’s reliance on multiple sophisticated models—such as LLMs and VLMs—requires substantial computational resources for both training and inference. The large-scale processing of multimodal data and the continuous refinement of the state-action graph could lead to increased memory usage and longer processing times, particularly in more complex scenarios or when dealing with large datasets. This could make it less suitable for enviroWhile SAGE is designed to generalize across novel objects and actions, there remains a risk of overfitting on certain specific objects or actions present in the training set. If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively when encountering new or highly diverse objects or actions in real-world settings. This overfitting issue could limit its scalability and applicability in dynamic, unpredictable environments.nments with limited computing power.\n4. While SAGE is designed to generalize across novel objects and actions, there remains a risk of overfitting on certain specific objects or actions present in the training set. If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively when encountering new or highly diverse objects or actions in real-world settings. This overfitting issue could limit its scalability and applicability in dynamic, unpredictable environments.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n### Q1: The integration of multiple models could make the system challenging to implement and maintain\n\nIntegrating LLMs and VLMs is a common practice in multimodal learning, as it provides an **easy** approach to utilize the web-scale prior knowledge encoded by these models. Notably, all baseline approaches we compare against used VLMs to generate pseudo labels. In SAGE, the LLM is only responsible for proposing relevant visual concepts for each object state. The design is highly modular and not tied to other components in the overall framework, hence easy to implement. We invite the reviewer to check our actual implementation, which was included in the supplementary materials.\n\nOur system is also easy to maintain. Table 2 shows that our approach naturally generalizes to new actions and objects, without the need for re-training.\n\n---\n\n### Q2: Proper alignment between the models are needed.\n\nAlignment is not needed due to our modular design. As discussed in Section 3.2, the LLM is responsible for proposing an over-complete concept list, based on its “commonsense” knowledge. The concepts are then refined by a VLM based on how visually discriminative each concept is. As discussed in our response to reviewer LFZv, SAGE’s performance is robust with respect to the choice of the LLM. As shown in Table 1 and 6, SAGE achieves competitive performances when different VLMs are used, where as expected, better VLM leads to higher performance.\n\n---\n\n### Q3: \"Performance dependent on quality and diversity of the data … high-quality labeled data might not always be available\"\n\nAs pointed out by reviewer LFZv themselves, “SAGE can learn from **unlabeled** instructional videos”, which allow us to utilize diverse and large-scale training data, without relying on labels.\n\n---\n\n### Q4: LLMs and VLMs require substantial computational resources for both training and inference\n\nThe LLM is only used for graph construction at training time. The VLM is used for graph construction at training time, and visual feature extraction at training and test time. The use of the visual encoder in a VLM is a standard practice adopted by all baseline approaches.\n\nOnce the visual representations are extracted, only a lightweight temporal Transformer is needed during inference time.\n\n---\n\n### Q5: \"Overfitting on certain specific objects or actions present in the training set … If the system is exposed to a narrow range of actions or objects during training, it might fail to generalize effectively\"\n\nAs discussed earlier, our method does not rely on any labeled videos, hence can naturally utilize large-scale training data that covers a wide range of actions and objects. As shown in Table 2, our approach achieves the best performance on generalization towards novel actions and objects."}, {"review_id": "1LjclEMdyP", "reviewer": "Reviewer_SfBi", "summary": "The authors present SAGE (State-Action Graph Embedding), a framework for recognizing physical object state transitions in videos. The core idea is to create a unified model that can generalize to unseen objects and actions. This is achieved by decomposing object states into a set of fine-grained, language-described visual concepts. The framework leverages a Large Language Model (LLM) to construct an initial State-Action Graph, which is then refined using a Vision-Language Model (VLM). The resulting graph provides a compositional representation of states and actions, which is used to train a video transformer for the final recognition task.", "questions": "My main concerns are the following:\n- Could the authors provide a more detailed explanation of the Vidertbi Decoding, ideally accompanied by a qualitative example?\n- What is the methodology for validating the accuracy and, crucially, the visual relevance of the states and concepts generated by the LLM?\n- The experiments rely on a single, powerful LLM. How dependent is the framework's overall performance on this specific choice?\n\nFor more details, please read the weaknesses section.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Paper Strengths\n- **Novel and Effective Conceptual Framework**: By representing states not as monolithic entities but as compositions of more primitive visual attributes, the proposed framework addresses the key challenge of generalization. This compositional approach forms a strong basis for the model's ability to recognize novel object states by leveraging familiar concepts learned from different contexts.\n- **Comprehensive Empirical Evaluation**: The authors conduct extensive experiments and provide insightful ablation studies that convincingly validate the effectiveness of their approach.\n    - *Performance*: As shown across different settings (Table 1), SAGE outperforms existing baselines on standard benchmarks.\n    - *Proven Generalization*: The results in open-world settings (Table 2 (b)), where both objects and actions are unseen during training, are impressive and directly support the paper's main claims about generalizability.\n    - *Efficiency*: The framework demonstrates a significant advantage in inference efficiency over specialist models (Table 3), which require running multiple experts when the action is unknown. This makes SAGE a more practical and scalable solution.\n    - *Insightful Ablations*: The ablation studies clearly demonstrate the contribution of each component, particularly the value of using visual concept descriptions and the Viterbi temporal decoding.\n- **Reproducibility and Clarity**: The authors' decision to release their code enhances the reproducibility of their work and provides a valuable resource.\n\n## Paper Weaknesses\n- **Insufficient Explanation of Viterbi Decoding**: The use of a constrained Viterbi decoding algorithm to refine the pseudo-labels is a key step in handling the lack of frame-level annotations. However, the explanation provided (L131-133) is very brief. While Figure 2 shows its place in the pipeline, its functional importance is understated. The paper would be significantly strengthened by a more detailed explanation accompanied by a qualitative example. For instance, a short walkthrough showing how the algorithm corrects an illogical, noisy label sequence (e.g., [Initial, End, Transitioning, End]) into a temporally coherent one would provide much-needed clarity and better highlight the value of this component.\n- **Over-reliance on LLMs without Robustness Analysis**: The SAGE graph, the core of the framework, is constructed almost entirely by an LLM. This creates a critical dependency on the quality and consistency of the LLM’s output, yet this dependency is not examined or analyzed.\n    - *LLM Sensitivity*: The authors rely on a single, powerful proprietary model (GPT-4) to generate states and concepts. Including an ablation study on the choice of LLM would be beneficial. How would the framework perform if a smaller, open-source model were used? Such an analysis would help assess the robustness of the approach and clarify how tightly its performance depends on a specific high-end LLM.\n    - *LLM Output Validation*: The authors do not address how the correctness of the LLM-generated states and visual concepts is verified. Including a brief analysis of the quality and consistency of the generated graph would be interesting.", "was_revised": true, "final_justification": "After reviewing the rebuttal and engaging in the discussion phase with the authors, I find that they have improved the clarity of the paper and satisfactorily addressed all of my concerns. Accordingly, I am raising my final recommendation to accept.", "rebuttal": "Thank you for the thoughtful feedback. Below we provide detailed answers to your questions.\n\n---\n\n### Q1: Viterbi Decoding\nFor each frame, the model predicts the probability of being in one of three states: initial, transitional, or end. To produce the most probable and temporally consistent sequence of states, we apply Viterbi decoding with a monotonic constraint: the sequence must follow the order initial → transitional → end. The pseudocode for the constrained Viterbi decoding is provided below. We will clarify Viterbi Decoding in the revised version and provide the full algorithm in Appendix.\n\n**Input**:  \n- `T`: total number of frames  \n- `S = {0: initial, 1: transitional, 2: end}`  \n- `probs[t][s]`: probability of frame `t` being in state `s`\n\n**Initialization**:  \n- `dp[0][0] ← log(probs[0][0])`  \n- `dp[0][1] ← -∞`, `dp[0][2] ← -∞`\n\n**Recursion**:  \nFor `t = 1` to `T - 1`:  \n For `s in S`:  \n  Let `prev(s)` be allowed previous states:  \n   `prev(0) = {0}`  \n   `prev(1) = {0, 1}`  \n   `prev(2) = {1, 2}`  \n  `dp[t][s] ← max_{p in prev(s)} [dp[t-1][p] + log(probs[t][s])]`  \n  `back[t][s] ← argmax_{p in prev(s)} [dp[t-1][p] + log(probs[t][s])]`\n\n**Termination**:  \n- `y_T ← argmax_s dp[T-1][s]`\n\n**Backtrace**:  \n- For `t = T-2` to `0`:  \n  `y_t ← back[t+1][y_{t+1}]`\n\n**Output**:  \n- Sequence `y = [y_0, ..., y_{T-1}]` with enforced order: initial → transitional → end\n\n---\n\n### Q2: LLM Reliability\nTo verify the reliability of LLM, we manually annotate the states and concepts generated by LLMs. We randomly sample 100 states and 100 concepts generated by the LLM and manually annotate the precision of these states and concepts. The precision of states is 100% and the precision of concepts is 92%, which shows that the LLM can generate highly accurate descriptions of object states and concepts.\n\n### Q3: LLM Sensitivity\nTo verify that our method is not sensitive to the choice of LLM, we conduct the experiments with another LLM, Qwen3-32B. We compare the performance of SAGE with GPT4 and Qwen3 under the settings of Table 1 and Table 2 on HowToChange dataset. SAGE demonstrates similar performance when combining with different LLMs, showing that our method is not sensitive to the choice of LLM.\n\nTable 1: Object state recognition\n| Methods       | Action | Object | Unified Model | F1   | Pre  | Pre@1 |\n|---------------|--------|--------|----------------|------|------|--------|\n| SAGE (GPT4)   | ✗      | ✗      | ✓           | 0.34 | 0.39 | 0.52 |\n| SAGE (Qwen3)   | ✗      | ✗      | ✓            | 0.34 | 0.40 | 0.52 |\n\nTable 2(a): Object state recognition for novel object\n| Methods       | F1   | Pre  | Pre@1 |\n|---------------|------|------|--------|\n| SAGE (GPT4)      | 0.34 | 0.39 | 0.50 |\n| SAGE (Qwen3)    | 0.35 | 0.39 | 0.49|\n\nTable 2(b): Object state recognition for novel object&action\n\n| Methods       | Action | Object | Unified Model | F1   | Pre  | Pre@1 |\n|---------------|--------|--------|----------------|------|------|--------|\n| SAGE (GPT4)  | 0.31 | 0.35 | 0.45 |\n| SAGE (Qwen3) | 0.29| 0.34| 0.44  |"}]

Author rebuttal:

Meta-review: This paper introduces a state-action graph embedding framework for recognizing the physical states of objects and their transformations within videos. All reviewers unanimously agree that it makes significant contributions and recommend acceptance. The AC notes that one reviewer (LFZv) stated the concerns had been resolved after the rebuttal but did not update the final rating. Accordingly, the AC has considered the reviewer’s comments rather than the unchanged rating.

After careful evaluation, the AC concurs with the reviewers’ assessments and agrees that this paper addresses a timely and promising direction by establishing the state-action graph for physical state recognition and transition, which is critical for many other applications such as neuro-symbolic reasoning and robotics. Its unified framework and strong empirical results make it broadly relevant and likely to attract a wide audience. Therefore, the AC recommends this paper be selected as an oral presentation.
Decision: Accept (oral)