Predicate Hierarchies Improve Few-Shot State Classification
Authors: Emily Jin
Joy Hsu
Jiajun Wu
Venue: ICLR 2025 Poster (2025)
Keywords: few-shot state classification
predicate hierarchies
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A state classification model that encodes predicate hierarchies to generalize effectively in few-shot scenarios.

Abstract: State classification of objects and their relations is core to many long-horizon tasks, particularly in robot planning and manipulation. However, the combinatorial explosion of possible object-predicate combinations, coupled with the need to adapt to novel real-world environments, makes it a desideratum for state classification models to generalize to novel queries with few examples. To this end, we propose PHIER, which leverages predicate hierarchies to generalize effectively in few-shot scenarios. PHIER uses an object-centric scene encoder, self-supervised losses that infer semantic relations between predicates, and a hyperbolic distance metric that captures hierarchical structure; it learns a structured latent space of image-predicate pairs that guides reasoning over state classification queries. We evaluate PHIER in the CALVIN and BEHAVIOR robotic environments and show that PHIER significantly outperforms existing methods in few-shot, out-of-distribution state classification, and demonstrates strong zero- and few-shot generalization from simulated to real-world tasks. Our results demonstrate that leveraging predicate hierarchies improves performance on state classification tasks with limited data.

Published as a conference paper at ICLR 2025
PREDICATE HIERARCHIES IMPROVE FEW- STATE CLASSIFICATION
Emily Jin
Joy Hsu"
Jiajun Wu
Stanford University
Stanford University
Stanford University
ABSTRACT
State classification of objects and their relations is core to many long-horizon tasks, particularly robot planning and manipulation. However, the combinatorial explosion of possible object-predicate combinations, coupled with the need to adapt to novel real-world environments, makes a desideratum for state classification models to generalize novel queries with few examples. To this end, we propose PHIER, which leverages predicate hierarchies to generalize effectively in few-shot scenarios. PHIER
encoder, self-supervised losses that infer semantic relations between predicates, and hyperbolic distance metric that captures hierarchical structure; learns a structured latent space of image-predicate pairs that guides reasoning over state classification queries. We evaluate PHIER the CALVIN and BEHAVIOR robotic environments and show that PHIER significantly outperforms existing methods in few-shot, out-of- distiiiiibttionn state classification, and demonstrates strong zero- and few-shot generalization from simulated to real-world tasks. Our results demonstrate that leveraging predicate hierarchies improves performance on state classification tasks with limited data.
object-centric
INTRODUCTION
State classification of objects and relations is essential for a plethora of tasks, from scene understand- ing robot planning and manipulation (Migimatsu & Bohg 2022) Chen et al. 2024). Many such long-horizon tasks require accurate and varied state predictions for entities in scenes. For example, planning for "setting up the table" requires classifying whether the cup To the plate, whether the utensil I are OnTop of the table, and whether the microwave is Open. The goal of state classification is to precisely answer such queries about specific entities an image, and determine whether they satisfy particular conditions across range of attributes and relations.
However, the combinatorial space of objects (e.g., cup, plate, mi micooave) and predicates (e.g.. NextTo, TnT Open) gives rise to explosion of possible object-predicate combinations that intractable obtain corresponding training data for. In addition, real-world systems operating in dynamic environments must generalize to queries with novel predicates, often with only a few examples (Beddale & Boult 2015 Joseph et al., 2021 Ha & Song 2022). For instance, we may
"Equal contribution
Training state classification
Inferring Predicate Hierarchy
Few-shot novel predicate generalization
NextTo
Open
Closed
Toaching
OnRight
(cup, piate)
teup, bocelet
Enaudo
Containa
HoutzZo
Yes
Leamed
Closed
Novel
Onftight
(cabinet)
Figure 1: PHIER improves few-shot state classification, by encoding predicate hierarchy in joint predec-eee latent space. By encouraging such structured representations to emerge, PHIER enables strong few-shot generalization to novel predicates with few examples.

---

Published as a conference paper at ICLR 2025
not only want to classify our trained query of whether the cup is Next the plate, but more specifically, whether the cup is OnRight of the bottle, and whether the cabinet is Closed (See Figure D. Hence, an essential but difficult consideration for state classification models is to quickly learn to adapt to -ii-o---disrrbutiion queries. However, current supervised methods struggle with few-shot state classification, and pretrained large vision language models fail to accurately answer challenging spatial relationship queries in robotics environments.
To this end, we propose PHIER, a state classification model that leverages the hierarchical structure be- tween predicates to few-shot generalize to novel queries. At the core of PHIER is an image-predicate latent space trained to encode the relationship between pairwise predicates (See Figure D. Let us consider the predicates, OnRight and OnLeft-wail they describe opposite spatial relationships between objects, they are closely related semantically, as assessing them relies on the same underlying features. PHIER enforces image-predicate representations conditioned on these predicates to lie closer to one another. In addition, there exist predicate pairs with more complex relationships, such as OnRight and NextTo. We see that OnRight is more specific case of Next To rov-erifying OnRight involves recognizing whether the objects are Next each other. Features relevant to the higher-level predicate To therefore useful for reasoning about the lower-level predicate OnRight. PHIER encodes this predicate hierarchy to allow generalizable state classification.
Such predicate hierarchy can be used to quickly adapt to unseen object-predicate pairs. For example, when combining a learned predicate Next with the objects app and mi picowvae, ve, PHIER uses learned embedding of the predicate to control the features extracted from the image depicting an apple and microwave. Notably. for a more challenging state with novel predicate, such as OnRight, the model can still adapt by drawing on its relationship OnLeft and Next To to learn relevant features from limited data. PHIER leverages pairwise hierarchical relations between predicates efficiently generalize to out-of of-distributton predicates with few examples.
To perform state classification, PHIER first localizes relevant objects in the input image based on given query, then leverages an inferred predicate hierarchy to structure its reasoning over the scene. PHIER processes objects and predicates separately, learning to map object representations conditioned on the predicate into an predicates is learned through self-supervised losses based on the relationships between predicates (e.g., enRight and Next We use a large language model (LLM) to infer the pairwise predicate relations from language. However, explicit hierarchies inherently assume some discretization over predicates, which does not align with the continuous nature of representations used in deep learning (Nickel & Kiela 2017; Ganea et al., 2018). To address this, we propose using hyperbolic distance metric to encode the hierarchical structure of predicates in a continuous manner, enabling PHIER to effectively incorporate tree-like structure.
latent space. The hierarchical structure of
We evaluate PHIER on the state classification task in two robotics environments, CALVIN (Mees) let al. 2022) and BEHAVIOR (Li al. 2023a). Beyond the standard test settings, we focus on few- shot,
tasks involving unseen object-predicate combinations and novel predicates. Our results show that PHIER significantly outperforms existing methods, including both supervised approaches trained on the same amount of data and inference-only vision- language models (VLMs) trained on large corpora of yeal-world examples. PHIER improves upon the top-performing prior work in out-of-distributi tasks by 22.5 percent points on CALVIN and 8.3 percent points on BEHAVIOR. Notably, trained solely on simulated data, PHIER also outperforms supervised baselines on zero- and few-shot generalization to real-world state classification tasks by 7 percent points and 10 percent points respectively. Overall, we see PHIER as a promising solution to few-shot state classification, enabling generalization by leveraging representations grounded in predicate hierarchies.
In summary, our contributions are the following:
We introduce PHIER, a state classification model that encodes inferred predicate hierarchies in its latent space, enabling generalization to unseen preeiccte combinations and novel predicates with few examples.
We propose learning the predicate hierarchy through object-centric scene encoder, self- supervised losses that encourage pairwise predicate relations, and a hyperbolic distance metric model the hierarchical nature of predicates in continuous space.
We validate PHIER's 's performance in few-shot, out-of-distribution generalization, and zero- and few-shot real-world transfer across three state classification datasets. PHIER significantly outperforms both supervised baselines and large pretrained VLMs.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
State classification. The ability to understand object states and relationships is essential for a wide range of tasks in computer vision and robotics, including scene understanding, robotic manipulation, and high-level planning ( (Yao et al. 2018 Yuan et al. 2022). Earlier works that focus on a similar task of visual relationship detection learn to extract ob object-centric representations from raw images
and make predictions based on them (Gkioxari et al. 2018 Yao et al., 2018 Ma et al. 2022 Yuan
al. 2022). A more recent approach by Yuan al. (022)) specifically addresses state classification by extracting object-centric embeddings from RGB images and feeding them into trained networks to classify a set of predefined predicates. However, their approach relies on input images of the objects of interest and is limited to predicates seen during training, as it requires a separate network for each predicate. In contrast, our method can shw-shot generalize to unseen predicates given only the input scene and query, without any additional annotations or specific object images.
Recent advancements in robotics simulators have additionally enabled scalable training of state classification in simulation and subsequent transfer to real-world settings. Simulators such as CALVIN (Mees et al. 2022) and BEHAVIOR (Li et al., 2023a) offer varying levels of realism and widely used in the robotics community to generate large and diverse datasets, supporting data generation across a broad range of states (Ge et al.) 2024). We train our model on such simulators and show that, compared prior works, PHTER is significantly more effective zero- and few-shot generalization to real-world state classification tasks.
Duestion-answering approaches. Several key advancements in visual question answering (VQA) have been driven by the development of pretrained large vision-language models (VLMs). These models are trained on extensive image and text data, leading to a unified visual-languag represen- tation that can be applied to perform various downstream tasks (Shen et al. 2021 Li et al. 2023b OpenAl 2023). Approaches such as Viper-GPT (Surís et al., 2023) further leverage the power of foundation models by composing LLMs to generate programs that pretrained VLMs can execute. However, despite their strong general-purpor purpose capabilities, these models still struggle with questions involving visual and spatial relationships (Tong et al. 2024).
On the other hand, a separate class of VQA methods are models trained directly for the VQA task. These approaches follow general framework of extracting visual and textual features, combining them into a multimodal representation, and learning to predict the answer. Convolutional neural networks (CNNs) and transformers are widely used for feature extraction, with various techniques for fusing the features. FiLM (Perez et al. 2018 is an early, modular approach that applies a feature- wise transformation to condition the image features on the text features, while BUTD (Anderson let al. 2018) and Re- Re-Attention (Guo et al., 2020) are representative attention-based methods that localize important regions based on the question. Furthermore, many approaches, including modular, graph-based, or neurosymbolic approaches, introduce more explicit reasoning to better model the relationships and interactions between objects (Andreas et al. 2016 Yi et al. 2018 Ma et al.) [202]] Nguyen et al. 2022] Wang et al.) 2023). Our work lies in this latter class of methods, designed and trained for state classification. In contrast to prior works, we not only encode visual features and textual features of predicates but also learn to capture the inherent predicate hierarchy in joint image-predicate latent space.
Hyperbolic representations for hierarchy. In recent years, several works have explored the benefits of using hyperbolic space to model hierarchical relationships, as it is particularly well- suited for capturing hierarchical structures (Ganea et al., 2018 Nickel & Kiela] 2018). In computer vision, prior works have focused on learning hyperbolic representations for image embeddings, demonstrating their effectiveness across various tasks such as semantic segmentation and object recognition (Khrulkov et al. 2020) Liu et al., 2020 Atigh et al.) 2022) Ermolov et al. 2022 Ge et al., 2023). Hyperbolic embeddings allow models to naturally represent hierarchical relationships between various entities, such as objects and scenes or different categories, leading to improved performance and generalization in such tasks (Weng et al., 2021 Ge et al.] 2023). Several approaches further incorporate self-supervised learning to learn the underlying hierarchical structure without the need for explicit labels, instead leveraging proxy tasks such as contrastive learning (Hsu et al. 202 | Ge et al. 2023 Yue et al. 2023). Recently, Desai et al. (2023) proposed contrastive method to learn joint representations of vision and language in hyperbolic space, yielding a representation space that captures the underlying structure between images and text. Inspired by these works, PHIER lears predicate hierarchy in hyperbolic space informed by language and the pairwise relations between predicates, and uses it to conduct few-shot generalization to unseen state classification queries.
3

---

Published as a conference paper at ICLR 2025
Image and Predicate Encoder
Image-Predicate Predicate Space
Poincuré Ball
Self-Supervised Losses
Triplet
Encoder
Anchor
Posilive
Negative
RexUTo (oup, plato)
Predicate Hierarchy
Iracide
0.6
(cupy piate)
LLM
ide
Image
Regularization
Encoder
Tinst die
OnTeft 1-1
PHIER consists of three main components. The first are disentangled image and predicate encoders, which separately extract features based on the objects and predicates the state classi- fication query. The second is self-supervised learning process that injects explicit knowledge of pairwise predicate relations into the distance metric and encoder encourage encoding of the inferred predicate hierarchy. Together, these components enable few-shot generalization to unseen object-predicate pairs and novel predicates.
Figure 2:
latent space. The third is the use hyperbolic
METHOD
In this section, we describe PHIER, our model for generalizable state classification. We define PHIER's task as a binary state classification problem: given a 2D RGB image and a text query describing state (e.g., NextTo ( (cup, plate), the goal is to predict whether the predicate NextTo holds True or False for the objects, cup and p alat,, in the input image. PHIER enables efficient generalization unseen predicates, by way of inferring structured latent space of image- predicate pairs to perform reasoning over (See Figure 2). At the core of our method is a predicate hierarchy that captures the semantic relationships between predicates. This hierarchical structure is inferred through three key components:
1. An object-centric scene encoder that localizes regions corresponding to relevant objects;
2. Self-supervised losses that inject pairwise relations of predicates into the latent space;
3. A hyperbolic distance metric and an encoder that model hierarchical representations.
we describe PHIER's base architecture of object and predicate conditioning. In Section 3.2 introduce self-supervised losses that encourage representations cluster based on pairwise predicate relationships. In Section [3.3] we detail how PHIER learss hicrarchical representa- tions by embedding representations in hyperbolic space.
In Section 3.1
3.1 OBJECT-CENTRIC IMAGE ENCODER
PHIER extracts object-centric scene representation by conditioning the input image I the query. Assume the input query is Next (cup, plate). PHIER first parses into its objects O = (cup, plate) and predicate P Next Then, PHIER separately conditions the latent space on both of these components (See Figure 2}. By disentangling the full state classification query, we ensure that PHIER faithfully identifies the relevant entities; then learns focus on their key features for the given predicate's classification.
Object conditioning. The goal of object conditioning is localize regions of the image that correspond to the relevant objects in the query. This allows PHIER to focus on the objects of interest for predicting the downstream predicate condition. encode an input image I on the objects O, our encoder Fing generates an object-conditions image mask (I. O) that highlights image regions based on the relevant entities, following Zhou et al. (022)). At high level, we extract features for the image and object texts, align the image features with each object text's features to generate individual object masks, and then encode the image based on these masks. Concretely, for each object CLIP image encoder V and text encoder T (Radford et al., 2021 to obtain image features (I) e R. and object text features T(o) E RP, where D,, D2 correspond to the visual and language embedding dimensions, respectively.
O,
4

---

Published as a conference paper at ICLR 2025
To align the image and text features, we first project the image features into the same space the text features. In order to preserve the spatial structure of the image features, we initialize a 1 x 1 convolution layer C with weights from the last linear layer of the CLIP attention pooling mechanism and apply this to the image features. Then, we apply a convolution & on the image features, using the text features filters. This directly aligns image regions with the object text, highlighting the parts of the image that are the most relevant to the query. Our resulting object mask M(I,o) is defined as
M(I, = C(V(I)) & T(o).
To obtain the final image mask, we upsample each object mask to the original image dimensions, normalize the values to the range [0, 1]. and take the element- -wiee max across the individual object masks. We then apply the final mask to our image using a Hadamard product C and subsequently encode the masked image using pretrained vision transformer (ViT) encoder (Dosovitskiy 2020
Evir as follows:
ang (I. 0) Ever (I C man((nor(uupsaplleeMM) o)))))
rEO
This ensures that the representation encodes all relevant regions based on the specified objects.
Predicate conditioning. Next, PHIER conditions the latent representation Eing (I, O) on the text predicate P. to focus on features essential to the downstream classification task. To do so, we encode the predicate text P using a pre-trained BERT encoder E (Devlin 2018) and concatenate this with the masked image, resulting in final object-centric scene representation:
O,,,O,P) = concat Ping (I, O), Frex
which incorporates both object features and context relevant to the predicate.
3.2 SELF-SUPERVISED LEARNING
PHIER learns a joint mmaee-predicee space that encodes an inferred predicate hierarchy with self- supervised losses. In particular, a specific predicate such as OnLeft is encouraged to be close to a related, more general predicate such as Next as features important to closer predicates tend be more alike. At the same time, OnLeft ft should have a larger norm than Next to reflect its lower position in the hierarchy. This ensures that the features that are learned to be relevant to higher-Ievel predicates remain useful when reasoning about their lower-level children.
We introduce two self-supervised losses encourage such pairwise relationships: a predicate triplet loss based on the similarity between predicates and a norm regularization loss based on the hierarchy between predicates. For both losses, we sample triplets with unique corresponding predicates and then extract knowledge from an LLM to determine the underlying relationships between the predicates.
Predicate triplet loss. Given a predicate triplet with an anchor a, positive p, and negative n sample, the triplet loss encourages the distance between the anchor and negative predicate to be at least some margin greater than the distance between the anchor and positive predicate. Figure 2 we see that Tex is the anchor predicate, OnLeft is the positive sample, and Inside is the negative sample. The proper assignment of the samples is critical, as it directly affects how faithfully the model learns the relationship between predicate pairs. To determine the anchor, positive, and negative sample for any given predicate triplet, we query an LLM based on the semantic meanings and scene details described by each query. More concretely, for each triplet, prompt the L.M to assess the underlying relationships between the predicates. One predicate in the triplet is randomly chosen as the anchor. The I.LM is asked to determine which of the other two predicates is more similar the anchor. The anchor and the more similar predicate form a positive pair, while the anchor and the less similar predicate form negative pair. We provide the prompt template provided in Appendix [D] By extracting knowledge from an LLM, we leverage the LLM's 's explicit and extensive understanding of predicate relationships to produce meaningful triplets and guide the model toward a semantically rich image-predicate latent space.
Our formulation of the triplet loss is based on the distance between representations:
Lriipee, (a, p3
max(0, d(a, p) d(a, n) + A),
where d is the distance metric used in the latent space. We describe choice of hyperbolic distance metric in the following section. With this loss and chosen distance metric, the resulting representation space captures the similarity between predicates via their distance in the latent space.
5

---

Published as a conference paper at ICLR 2025
Norm regularization loss. While the triplet loss enforces similarity between related predicates, the norm regularization term introduces hierarchical structure to the latent space. We leverage the LLM's strong semantic understanding of predicates to infer the hierarchy among triplet, by ranking the predicates based on specificity. Specificity depends on several factors, including the variety and number of objects required by the state, the important features of the objects and relationships between the objects, the level of detail required by the state, and the semantic meaning of each description. The prompt template provided in Appendix D
Given a ranking of three predicates b, e from least to most specific, the regularization loss encour- ages the norm increase by at least some margin y as the specificity increases:
f b, = max(0, ||a|| - ||all + y + max(0, ||el| - -|bb|| + +)).
Intuitively, the regularization loss ensures that the norms of the representations reflect the implicit hierarchy between predicates.
Together, the predicate triplet loss and norm regularization loss encourage a semantically rich and structured representation space. The predicate triplet loss captures the similarity between predicates by enforcing appropriate distances between related predicates, while the norm regularization loss ensures that the hierarchical relationships are reflected in the norm of the representations. By leveraging the explicit knowledge an LLM infer both the triplet assignments and hierarchy ranking, our approach ensures that PHIER's learned representations align with the underlying predicate ontology.
3.3 HYPERBOLIC IMAGE-PREDICAL LATENT SPACE
Finally, PHIER effectively learns this inferred hierarchy of predicates through hyperbolic repre- sentations. While PHIER's s self- sulf-super losses inject semantic knowledge of predicates into its representations, PHIER further encodes the hierarchical nature of the predicates hyperbolic space. In hyperbolic space, we more easily visualize these relationships forming natural predicate hierarchy. In Figure 2 we visualize a learned predicate hierarchy in PHIER's 's latent space.
Hyperbolic space is non-Fuclidean space characterized by constant negative curvature, which allows hierarchical structure to be easily embedded. Due to hyperbolic space's ' curvature, the area of disc increases exponentially with its radius, analogous to the exponential branching of trees. This property makes hyperbolic space well-suited for modeling hierarchies, as it provides continuous representation of discrete trees. Furthermore, hyperbolic space differentiable, making easy to integrate with our model. Hence, we propose using hyperbolic distance metric for our predicate triplet loss and norm regularization loss to more effectively encode the predicate hierarchy. These hierarchical representations enable PHIER to generalize effectively to novel predicates, by inferring their representations based on their relationships to learned predicates in the hyperbolic latent space.
Poincaré ball model. In this work, we use the Poincaré ball model of hyperbolic space. The Poincaré ball is an open d-dimensional ball of radius 1, equipped with the metric tensor 9p (Ax)² ge. Here, ||- the Euclidean norm, A = I-1 - is the conformal factor, and g. is the Euclidean metric tensor (i.e., the Euclidean dot product). This induces the distance dp between two points x, y on the Poincaré ball
dp(x, v) cosh -1 (1 (1+ 2
1
y )1
2
13
On the Poincaré ball, the distance between two points captures the degree similarity between them, while the relative norm of two points reflects their hierarchical structure. Thus, the Poincaré ball is suitable space to represent the underlying predicate hierarchy, and PHIER's 's self-supervised losses such metrics to embed image- prediccaes onto the Poincaré ball.
Hyperbolic encoder. To obtain the hyperbolic map to lift the representation from Euclidean space onto the Poincaré ball and pass it through a small hyperbolic linear network (Ganea et al. 2018). For more details on these functions, we refer the reader Appendix C
representation, we use the exponential

---

Published as a conference paper at ICLR 2025
antop Iptink biock, tabie)
Cottpp (biue block, table)
Inaide 1blue block, slicer)
Inside (bibe biock,
Open (drower)
(al der)
CALVIN
OnTop Sooffee cup, tablet
Caflup Ipiate, cabTel
Inside iepple, cabínet)
Inside (coffee cup, drower)
Open (draver)
Open imicrowave)
BEHAVIOR
On:Top (bottle, cabiel
Onfop (cup, rable)
Inside (boctle, mi orooaav)]
Inside Icoffee drawer)
Open (drawer)
Open: (cabinet)
Real
Figure 3: Examples of state classification tasks from CALVIN and BEHAVIOR. The datasets span range of visual realism and complexity.
3.4 TRAINING Loss
After the hierarchical representation is learned in hyperbolic space, we apply the logarithmic map to project it back to Euclidean space, where it is passed through small MLP for state classification. We train PHIER with a binary cross entropy loss based on the ground truth labels (True or False), along with the predicate triplet loss and the norm regularization loss. Our overall loss is defined as Lreeal := Csup + Cripleet. + Bereg,y, where a, 3 are coefficients that control the strength of the triplet and regularization losses, and X, y are the margins for the two losses, respectively.
DATASET
We evaluate PHIER on established robotics environments, with three state classification datasets designed to test the following key aspects of performance: faithful understanding of entities and relations between them, few-shot generalization to out-of-distribution queries, and zero- and few-shot transfer real-worle setting. See Figure 3 for examples from each of the environments.
Simulator dataset generation. In order to evaluate our method's 's state classification performance on robotic environments, generate datasets of varying levels of realism with two widely used robotics simulators, CALVIN (Mees et al.) (202)) and BEHAVIOR (Li et al. 2023a). Both are known for their ease of and customizability, allowing
generate diverse data for various states with different objects under various lighting, camera angle, and object pose conditions. As shown in Figure 3 CALVIN is visually simplistic while BEHAVIOR is more realistic and complex. By evaluating on data from these two simulators, we assess how well various methods understand the semantics of different predicates.
Simulator dataset details. To evaluate the effectiveness of our inferred abstraction hierarchy, we define of in- distribution and -utto--distriutiion states, featuring both unary and binary relations. The -iit----iittrbutioon states involve unseen predicate-object combinations and novel predicates. See Appendix for the states in each dataset. We train on balanced dataset of 200 examples (100 True, 100 False) for each in-distribution state. We then evaluate on balanced test sets of 50 examples for each state under both in-distribution and -utto--distributiin settings.
Real-world dataset. In addition, we evaluate BEHAVIOR Vision Suite (Ge al. 2024) (see Figure 3). a complex real-world benchmark that consists of diverse scenes and distractor objects. Specifically, compared
train data, this consists of 10 unseen combinations and 10 novel predicates, with 337 total examples. We use this dataset to test our method's ability to perform zero- and few-shot real-world transfer after training simulated datasets alone.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of PHIER prior works CALVIN and BEHAVIOR datasets. We compare against trained models (T) and inference-only models pretrained on large-scale data (I), In this table, we report in-distribution (ID) test accuracy. out-of-distibution (OOD) test accuracy, and the difference in performance between the two test sets (ID-OOD). PHIER significantly outperforms prior work in the OOD setting; it is also the only method that performs similarly in ID and OOD settings.
CALVIN
BEHAVIOR
ID f
0OD
ID-OOD +
ID
OOD
ID-OOD
PHIER (Ours)
T
0.945
0.899
0.046
0.859
0.820
0.039
Re-Attention (uoo et al.
2020
T
0.959
0.674
0.285
0.828
0.652
0.176
CoarseFine (Nguyen
(0222)
T
0.878
0.624
0,254
0.766
0.636
0.130
BUTD Anderson et al. 2018
0.898
0.585
0.313
0,808
0.712
0.096
ReIViT Ma ct al. 2022
T
0.688
0.563
0.125
0.866
0.737
0,129
CLIP
Shen et al. 2021
T 0.937
0.546
0.391
0.722
0.632
0.090
FiL.M (Perez et al.
2018
T
0.798
0.489
0.309
0.753
0.583
0.170
SORNet Yuan
2022
0.943
0.773
GPT-4V OpenA|
2023)
0.587
0.563
0.024
0.661
0.706
-0.045
BLIP-2
[ 2023b
al.
0.553
0.556
-..03
0.554
0.535
0.019
ViperGPT [Suris ct al.
2023)
0.466
0.475
-0.009
0.552
0.583
-0.011
RESULTS
We evaluate PHIER on the three datasets and compare against 10 state-of-the art models, with our evaluation metric binary state classification accuracy. Our experiments show that PHIER's learned predicate hierarchy leads to significantly improved performance, especially in the challenging settings of few-shot, out-of-distribution generalization well zero- and few-shot, real-world transfer.
5.1 IMPLEMENTATION
PHIER. For our model, we use the CLIP image encoder, CLIP text encoder, and BERT text encoder image, object text, and predicate text encoders, respectively. Our hyperbolic encoder consists of two hyperbolic linear layers with output dimensions of 256 and 128, and the final small MLP is single layer. We use a 0.05 our triplet loss coefficient, À 10.0 our triplet loss margin, g 1.0 as our regularization loss coefficient, and = 0.1 as our regularization margin. We train all models for 50 epochs using the AdamW optimizer with learning rate of le using a gradual warmup scheduler and cosine annealing decay. For the few-shot setting, we provide 5 examples of each novel predicate and train for 20 epochs with the same optimizer and learning rate.
Baselines. We use 10 state-of-the-art methods as baselines, ranging from supervised methods to pretrained large vision language models (VLMs). Of the supervised methods, ReIViT (Ma et al. 2022) and SORNet Yuua et al. 2022) are recent methods designed with a focus on state classification, while
Re-Attention (Guo et al. 2020), Coarse-to-Fine (Nguyen et al. 2022), BUTD (Anderson et al. 2018
finetuned CLIP (Shen et al. 2022), and FiLM (Perez et al. 2018) are top-performing general VQA methods. The supervised models are all trained on the same data as PHIER, while the pretrained large VLMs, BLIP-2 2 (Li et al. 2023b), GPT-4V(OpenAA) 2023), and ViperGPT (Surís et al. (023), run inference-only. All methods are evaluated on both in-distribution and out-of-distribttion queries, except for SORNet its architecture does not allow classification of unseen states. Additional details on our baselines are provided in Appendix
5.2 COMPARISON PRIOR WORK
Few-shot generalization accuracy. In Table I show comparisons of PHIER and prior work on CALVIN and BEHAVIOR datasets, in the 5-shot generalization setting. We split prior works into trained supervised methods (T) and inference-only pretrained VLMs (I). While PHIER yields comparable performance to nor-performing prior works on the in-distribution test set, PHIER signifi- cantly outperforms all methods on the out-o--distiibution test set. PHIER demonstrates 22.5 percent
8

---

Published as a conference paper at ICLR 2025
Table 2:
Detailed breakdown of CALVIN and BEHAVIOR out-of-distribution results with accuracy
on unseen bbject-predicatte combinations and novel predicates.
CALVIN
BEHAVIOR
All
Unseen Comb.
Novel Pred.
All
Unseen Comb.
Novel Pred.
PHIER (Ours)
0.899
0.922
0.863
0.820
0.831
0.807
De-Attention
0.674
0.777
0.612
0.652
0.718
0.581
CoarseFine
0.624
0.665
0.557
0.636
0.670
0.600
BUTD
0.585
0.622
0.563
0.712
0.766
0.653
ReIVIT
0.563
0.591
0.515
0.737
0.793
0.676
CLIP
0.546
0.596
0.463
0.632
0.684
0.576
FiLM
0.489
0.538
0.406
0.583
0.652
0.508
point improvement compared to the top-performing prior work in the out-of-distribution CALVIN setting, and 8.3 percent point improvement the qul-of-distribution BEHAVIOR setting. Notably, PHIER sees low drop in accuracy between in-distribution and out-of-distribution test sets, which we hypothesize is due PHIER's s structured representation space enabling generalization. We report more detailed results in Table 2 specifically the out-of-distributie object-predicate pairs and novel predicates.
accuracy of few-shot unseen see that for both categories of generalization, PHIER significantly outperforms prior works. On CALVIN, PHIER improves upon the top-performing prior work by 25.1 percent points in few-shot generalization to novel predicates. On BEHAVIOR, we see a 13.1 percent point improvement.
Real world zero- and few-shot transfer accuracy. Table 3 we report the zero- and few-shot, real-world transfer results of models trained on the simulated BEHAVIOR dataset, and tested on the BEHAVIOR Vision Suite (Ge et al. 2024), a complex real-world benchmark. We compare PHIER with previous supervised methods, which have
the same amount of train data, and find that PHIER significantly outperforms prior works on this challenging sim-to-real task across both zero- and few-shot settings. We conjecture that this is because PHIER learns more robust features for images-only features core to the specified state classification task are captured, and hence enables PHIER to generalize and remain invariant to the visual details in the real world. We additionally include results from pre-trained models, though we note that these models are our upper bound, as they are trained on large-scale weal-world datasets with vast amounts of diverse data. Hence, these models inherently do not differentiate between in-distribution and generalization scenarios, their training data overlaps significantly with both. We see that PHIER outperforms ViperGPT and BLIP-2 by 6.0% and 1.4% respectively on PHIER's zero-shot setting, showing the potential for small model trained on significantly less data, to reach the performance level large pre-trained models. However, GPT-4v outperforms PHIER by 10.4%. which we hypothesize is due to its model size and dataset scale. In the few-shot setting using only two examples, PHIER's performance improves significantly and narrows the gap with GPT-4v just 0.9%. This further demonstrates that PHIER's inferred predicate hierarchy enables it to generalize efficiently to novel queries.
5.3 ABLATIONS
We ablate the components of PHIER in Table 4 Specifically, we begin by reporting the out-of- distribution results of supervised model for state classification. We then variants of PHIER, progressively adding each component: our object-centrie encoder, predicate triplet loss, norm regularization, and finally the full model with the hyperbolic distance metric.
We see that each component encourages structured and semantically relevant latent space for out-of-distribution generalization. The object-centric encoder localizes relevant objects in the scene, improving performance by 15.4 and 13.7 percent points on CALVIN and BEHAVIOR, respectively. Adding the predicate triplet loss helps PHIER encode pairwise relationships between predicates, improving performance by 13.1 and 8.2 percent point on CALVIN and BEHAVIOR. The addition of the norm regularization loss introduces hierarchical structure into the latent space, further improving performance by 6.5 and 4.7 percent points. Finally, highlight the full PHIER equipped with the hyperbolic metric, which further enforces a tree-like hierarchy to emerge in latent space and yields the strongest generalization performance.

---

Published as a conference paper at ICLR 2025
Table 3: We present and few-shot generalization results real-world test set, when trained only on the BEHAVIOR dataset. PHIER outperforms all prior supervised models.
Zero-shot
Few-shot
All
Unseen Comb.
Novel Pred.
All
Unseen Comb.
Novel Pred.
PHIER (Ours)
0.608
0.632
0.585
0.703
0.714
0.691
Re-Attention
0.377
0.415
0.341
0.413
0.458
0.368
CoarseFine
0.490
0.485
0.494
0.553
0.562
0.543
BUTD
0.418
0.427
0.409
0.456
0.464
0.448
ReIViT
0.556
0.579
0.528
0.603
0.654
0.552
CLIP
0.516
0.544
0.489
0.571
0.674
0.468
FiLM
0.459
0.480
0.438
0.513
0.542
0.484
GPT-4V
0.712
0.737
0.688
BLIP-2
0.594
0.591
0.597
ViperGPT 0.548
0.538
0.557
Table 4: Ablations of each component of PHIER and its effect on few-shot generalization.
CALVIN OOD
BEHAVIOR OOD
Supervised model
0.473
0.516
Object-centric encoder
0.627
0.653
+ Predicate triplet loss
0.758
0.735
Norm regularization loss
0.823
0.782
Hyperbolic metric (PHIER)
0.899
0.830
5.4 DISCUSSION
We propose PHIER framework for incorporating predicate hierarchies into the latent space of state classification models. One important design decision is how explicitly the hierarchy should be enforeed-PHIER softly encourages this structure with self-supervised losses, but does not impose hard constraints on the model's forward pass. We designed PHIER in this way, in order to allow hierarchical structure to emerge in the hyperbolic latent space based on the data, and capture more nuanced structure than strict, discrete predicate hierarchy could. Quantitatively, we see that PHIER retains the ability to perform well on generalization tests, while qualitatively, we can visualize PHIER's learned predicate hierarchy in the latent space.
We note that PHIER potentially limited by the accuracy of the language model in determining pairwise predicate relations. In this paper, we assume that language itself can differentiate the relationship between predicates, but there may be cases where visual cues from data also matter. Empirically, the datasets tested, we that the LLM's predictions match our expectations of what the predicate hierarchy should be. Additionally, as PHIER 's enforcement this hierarchy not explicit, it is still possible for PHIER to learn from data when the language model is incorrect. As a future direction, exploring environments where the dataset for state classification yields a unique predicate hierarchy, which we can encode through explicit enforcement in the model's forward pass, would showcase the effect of an explicitly hicrarchical version of PHIER in generalization. In addition, exploring ways of training model infer the pairwise predicate relations with weak supervision, instead of injecting relation priors through a language model, could potentially give rise fully emergent and discovered predicate hierarchy.
CONCLUSION
PHIER tackles the challenge of few-shot out-of-distribution state classification by encoding predicate hierarchies into its latent space. Our proposed model, PHIER, learns language-informed image- predicate representations to generalize to novel predicates with few examples. Our experiments on CALVIN, BEHAVIOR, and real-world test set demonstrate that PHIER significantly improves upon existing methods, particularly in highly difficult generalization cases. We show that using predicate hierarchies promising approach to enable more robust and adaptable state classification.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Weiyu Liu for providing valuable feedback. This work is in part supported by ONR N00014-23-1-253 1-2355, ONR YIP N00014-24-1-21177, ONR MURI N00014-22-1-27400 ONR MURI N00014-24-112748, NSF RI #221 1258, and AFOSR YIP FA9550-23-1-011 JH also supported by the Knight Hennessy Scholarship and the NSF Graduate Research Fellowship.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and Top-down Attention for Image Captioning and Visual Question Answering. In CVPR, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural Module Networks. In CVPR, 2016.
Mina Ghadimi Atigh, Julian Schoep. Erman Acar, Nanne van Noord, and Pascal Mettes. Hyperbolic Image Segmentation. In CVPR, 2022.
Abhijit Bendale and Terrance Boult. Towards Open World Recognition. In CVPR, 2015.
James W Cannon, William J Floyd, Richard Kenyon, Walter Parry, et Hyperbolic Geometry. Flavors of Geometry, 1997.
Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering. NeurIPS, 2020.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain lchter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. In CVPR, 2024.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyperbolic Image-text Representations. In ICML, 2023.
Jacob Devlin. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010. 2020.
Anna Dyubina and losif Polterovich. Explicit Constructions of Universal R-Trees and Asymptotic Geometry of Hyperbolic Spaces. Bulletin of the London Mathematical Society, 2001.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyper- bolic Vision Transformers: Combining Improvements in Metric Learning. In CVPR, 2022.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Neural Networks. In NeurIPS, 2018.
songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyperbolic Contrastive Learning for Visual Representations Beyond Objects. In CVPR, 2023.
Yunhao Ge, Yihe Tang, Jiashu Xu, Cem Gokmen, Chengshu Li, Wensi Ai, Benjamin Jose Martinez, Arman Aydin, Mona Anvari, Ayush K Chakravarthy, et al. BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation. In CVPR, 2024.
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and Recognizing Human- object Interactions. In CVPR, 2018.
Mikhael Gromov. Hyperbolic Groups. Essays group theory. Springer, 1987.
Wenya Guo, Ying Zhang, Xiaoping Wu, Jufeng Yang, Xiangrui Cai, and Xiaojie Yuan. Re-Attention for Visual Question Answering. In AAAI, 2020.
11

---

Published as a conference paper at ICLR 2025
Huy Ha and Shuran Song. Semantic Abstraction: Open-World 3D Scene Understanding from 2D Language Models. arXiv preprint arXiv:2207.11514 2022.
Matthias Hamann. On the Tree-Likeness of Hyperbolic Spaces. In Mathematical proceedings of the cumbridge philosophical society. Cambridge University Press, 2018.
Joy Hsu, Jeffrey Gu, Gong-Her Wu, Wah Chiu, and Serena Yeung. Capturing Implicit Hierarchical
Structure in 3D Biomedical Images with Supe-supe Hyperbolic Representations, 2021. URL
https:// arwivv org/ abs/ /tt./////..2222/2 2012. 01644
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards Open World Object Detection. In CVPR, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR: Modulated Detection for End-to-end Multi-modal Understanding. In ICCV, 2021.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic Image Embeddings. In CVPR, 2020.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian Optimization in Py Torch, 2020.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb Matthews, Ivan Villa-Renteria Jerry Huayang Tang. Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR- -III-::K A Benchmark for Embodied with 1,000 Everyday Activities and Realistic Simulation. In Karen Liu, Dana Kulic, and Jeff Ichnowski (eds.), PMLR, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Language-image Pre-training with Frozen Image Encoders and Large Language Models. In /CML, 2023b.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic Visual Embedding Learning for Zero-shot Recognition. In CVPR, 2020.
Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang. Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, and Anima Anandkumar. ReIViT: Concept-guide Vision Transformer for Visual Relational Reasoning, 2022.
Oier Mees, Lukas Hermann, Erick Rosete-Beat Beas, and Wolfram Burgard. CALVIN: A Benchmark for Connnagg--CCdiiioneee Policy Learning for Long-Horizon Robot Manipulation Tasks. IEEE, 2022.
Toki Migimatsu and Jeannette Bohg. Grounding Predicates Through Actions. In ICRA, 2022.
Binh X. Nguyen, Tuong Do Huy Tran, Erman Tjiputra, Quang D. Tran, and Anh Nguyen. Coarse- To-Fine Reasoning for Visual Question Answering. In CVPR, 2022.
Maximillian Nickel and Douwe Kiela. Poincaré Embeddings for Learning Hierarchical Representa- tions. NewrlPS, 2017.
Maximillian Nickel and Douwe Kiela. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry. In ICML, 2018.
OpenAl ChatGPT Can Now See. Hear, and Speak. http::://peeaa.com//ooo speak, 2023.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courvile. FiLM: Visual Reasoning with General Conditioning Layer. In AAAI, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In ICML., 2021.
12

---

Published as a conference paper at ICLR 2025
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation Tradeoffs for Hyperbolic Embeddings. In ICML, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How Much can CLIP Benefit Vision-and-language Tasks? arXiv preprint, 2021.
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic Neural Networks++. arXiv preprint (rXiv:2006.082 2020.
Didac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In ICCV, 2023.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Bugen Ganea. Poincare Glove: Hyperbolic Word Embeddings. arXiv preprint arXiv:1810.06546 2018.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. In CVPR, 2024.
Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec. VQA-GNN: Rea- soning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering. In ICCV,V 2023.
Zhenzhen Weng, Mehmet Giray Ogut, Shai Limonchik, and Serena Yeung. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In CVPR, 2021.
Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring Visual Relationship for Image Captioning. In ECCV, 2018.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018.
Zhou Yu, Jing Li, Tongan Luo, and Jun Yu. A PyTorch Implementation of Bottom-UppAAttentoon. https: github.. com/ wtw.L////oottom-p--attttetooooee ion opt.--/tppaatttttt..ttt.ttttl 2020.
Wentao Yuan, Chris Paxton, Karthik Desingh, and Dieter Fox. SORNet: Spatial Object-centric
Representations for Sequential Manipulation. CoRL, 2022.
Yun Yue, Fangzhou Lin, Kazunori Yamada, and Ziming Zhang. Hyperbolic Contrastive Learning. arXiv preprint arXiv:2302.01409 2023.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. In ECCV, 2022.
13

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY FOR: PREDICATE HIERARCHIES IMPROVE FEW-SHOT STATE CLASSIFICATION
The appendix is organized as the following. In Appendix A we include additional PHIER results, details, and discussion. In Appendix [ we describe implementation of baseline methods, including supervised models, pretrained large vision language models, and ablation variants. In Appendix C we present preliminaries on hyperbolic geometry. In Appendix D we detail prompts used extract knowledge of predicates from LLMs. Finally, in Appendix E. we list all states in our datasets, and show examples from the BEHAVIOR Vision Suite Ge et al. (2024).
A PHIER RESULTS AND DETAILS
A.1 MODEL DETAILS
PHIER's image and text encoders are initialized with pretrained CLIP (Radford et al. 2021) and BERT (Devlin 2018) weights, respectively. The hyperbolic linear layers are initialized following the approach of (himiizu et al. (2020). with the weights drawn from normal distribution centered at zero with a standard deviation (2nm) 1, where m and n are the input and output sizes of the layer, and the biases set to the zero vector. The linear layer in the small MLP is initialized by the standard Kaiming initialization. All of the parameters in PHIER are trainable and updated during training.
PHIER disentangles the conditioning of the image the full state classification query into two distinct ones: one that identifies the relevant objects and another that focuses on key features for the given predicate. While we use MaskCLIP to identify the relevant entities, PHIER's s contribution lies in the decomposition of the query into object and predicate components, enabling it to faithfully identify the relevant entities and extract features based on the predicate.
A.2 COMPARISON ON MANUALLY COLLECTED REAL-WORLDD DATASET
We collect a small real-world dataset with 100 examples, consisting examples for each of the out-of-distribution BEHAVIOR states, to test our method's ability to perform zero-shot real- world transfer after training on simulated datasets alone. See Figure for examples. Table ??, we observe similar trends in the BEHAVIOR Vision Suite evaluation in the main text, even with simpler dataset. PHIER significantly outperforms prior supervised baselines. However, as expected, pre-trained models trained large-scale real-world data outperform PHIER.
OnIop icottee table)
(nIop teble)
Inside (appie, Inside (cottes draver)
Open (drawer) (micromave)
esbinet)
Real world
Figure 4: Examples from our manually collected real-world dataset.
A.3 ABLATION STUDY ON EXAMPLE COUNT
We study the effect of varying the number of examples used in the few-shot setting. We added new ablation experiments with 0, 1, 2, 3, 4, 5, and 10-shot generalization performance on both CALVIN and BEHAVIOR environments. The results in Figure 5 show that PHIER consistently outperforms prior works across all numbers of examples. Notably, in the CALVIN environment, PHIER's performance plateaus as the number of examples increases, indicating that the method requires only few examples to adapt effectively unseen scenarios.
14

---

Published as a conference paper at ICLR 2025
Sew-Shot Ablation: CALVIN
Sew-Shol.oo Ablation: BEHAVIOR
Ous
Ous
Aeciitation
0.75
CouseFiiz
0.75
CousseFine
BUTD
Accumey
ReVV
Accuncy
RelViT
CLIP
0.35
0.5
Number of Examples
Number of Examples
Figure 5: Ablations varying number of exammles given in few-shot setting for CALVIN and BEHAV-
IOR environments.
A.4 ABLATION STUDY WITH REMOVED COMPONENTS
We add an ablation study that evaluates the impact of removing individual components of PHIER to evaluate their contributions. We compare PHIER with four variants, (I) without the object-centric encoder, (2) without the hyperbolic latent space, (3) without the norm regularization loss, and (4) without the predicate triplet loss. We report results in Table 5 We see that without object- centric design, performance drops significantly in both ID and OOD settings, emphasizing the importance of object-centric encoders for improved representation and reasoning. In addition, show that removing each of the self-supervised losses leads to much weaker generalization capability. Finally, observe reduced generalization performance without PHIER 's hyperbolic latent space and hyperbolic norm regularization loss, demonstrating that the hyperbolic space facilitates better handling of hierarchical relationships. These results validate that each component contributes meaningfully to PHIER's ' performance, particularly in improving OOD generalization.
Table Ablations each component of PHIER and its effect on few-shot generalization.
CALVIN
BEHAVIOR
ID t
OOD t
ID-OOD 1
ID 1
OOD
ID-OOD J
PHIER (Ours)
0.945
0.899
0.046
0.859
0.820
0.039
Object-centric encoder
0.786
0.704
0.082
0.703
0.659
0.044
Predicate triplet loss
0.867
0.601
0.266
0.774
0.624
0.150
Norm regularization loss
0.914
0.823
0.091
0.834
0.782
0.052
Hyperbolii metric
0.903
0.784
0.119
0.803
0.761
0.042
A.5 FEW-SHOT GENERALIZATION TO NOVEL OBJECTS
We expand our CALVIN and BEHAVIOR experiments to evaluate accuracy on few-shot generalization novel objects in Table 6 The queries with these novel objects listed in Table 7 As in experiments on unseen combinations and novel predicates, we observe that PHIER significantly outperforms prior baselines on unseen objects. Specifically, PHIER improves upon the top-performing prior work by 21.8 percent points on CALVIN and 13.5 percent point on BEHAVIOR. These results demonstrate that PHIER improves generalization both novel objects and predicates, further highlighting the benefit of our object-centric encoder and inferred predicate hierarchy.
A.6 VISUALIZATIONS ON THE INFERRED PREDICATE HIERARCHY
In Figure 6 visualize the joint image-predicee space for BEHAVIOR the Poincaré disk, highlighting the hierarchical semantic structure captured by PHIER's embeddings. By grouping the joint image-predicate embeddings by predicate, uncover the inferred predicate hierarchy. For
15

---

Published as a conference paper at ICLR 2025
Table 6: We present novel object generalization results of PHIER and prior works CALVIN and BEHAVIOR environments.
CALVIN
BEHAVIOR
PHIER (Ours)
0.851
0.781
Re-Attention
0.633
0.608
CoarseFine
0.562
0.632
BUTD
0.584
0.646
ReIViT
0.497
0.642
CLIP
0.506
0.595
FiLM
0.411
0.521
Table 7: All states with novel objects (bolded) in the CALVIN and BEHAVIOR datasets.
Dataset
Predicate
Object 1
Object 2
OnTop
red block
table
Stacked
red block
blue block
Stacked
red block
pink block
CALVIN
TurnedOn
led
Inside
box
bottom cabinet
Inside
can
bottom cabinet
OnTop
bottle
breakfast table
OnTopp
bottle
chair
BEHAVIOR
OnTop
box
breakfast table
OnTop
bread
breakfast table
OnTop can
chair
Open
refrigerator
instance, we see that embeddings for NextTo are positioned closer to the origin compared to those for OnLeft, accurately reflecting their hierarchical relationship- OnLe ft is of Next Furthermore, embeddings for Touching are nearest to the origin, consistent with its role the most general predicate. For example, when one object Inside or OnTop of another, they are inherently Touching. Similarly, objects that are Next To or OnLeft ft are also frequently Touching. This visualization demonstrates that PHIER captures not only semantic structure but also nuanced hierarchical relationships between predicates.
specific
We further analyze the embeddings for novel predicates after few-shot learning with only five examples. Notably, even with such limited data, PHIER successfully integrates these novel predicates into the latent space and aligns them with their learned counterparts in semantically consistent regions (e.g., Ontiggt is near OnLeft). By aligning these predicates in similar regions, PHIER is able to leverage its existing knowledge of relevant features for learned predicates (c.g., OnLeft) to reason about novel predicates (e.g.. OnRight) This alignment highlights that PHIER effectively encodes the relationships between pairwise predicates in the latent space, enabling generalization to novel predicates with minimal examples.
A.7 IN-DISTRIBUTION PERFORMANCE
Here, we discuss the in-distribution performance PHIER in Table of the main text. We note that in the in-distribution (ID) setting of CALVIN, PHIER outperforms all prior works except Re- Attention, with only a small margin 1.4%. the out-of-distribution (OOD) setting, which is our primary focus, PHIER outperforms Re-Attention by significant 22.5%. Similarly, on ID BEHAVIOR, PHIER performs comparably to top rop-performing prior works, surpassing all except ReIViT by 0.7%; however, in the OOD setting we focus on, PHIER outperforms RelViT by 8.3%. We highlight that PHIER performs comparably to top per-frrfoming prior works the ID setting, while significantly improving the OOD performance. We focus the few-shot generalization task and
16

---

Published as a conference paper at ICLR 2025
In-Distribution Predicatos
Novel Prodicates
1,50
Predicates Inodo
Pridicates
0.50
Onflight
Closed
Unitop
Unde:
Neatlo
Foushing
0.00
-0.00
+100
+1.00
1..00
-0.50
0.00
0.50
1.00
-1.00
-0.50
Dimemioo
Dimersion
Figure 6: Visualizations of the joint amage-preddccate space for BEHAVIOR the Poincaré disk, revealing that PHIER learns a meaningful predicate hierarchy. The novel predicate embeddings are visualized after few-shot learning with examples.
design our method to enforce bottlenecked representations (via joint image- pnage-predicate space), while acknowledging that this might include tradeoffs on ID performance to avoid overfitting to the train distribution.
We also analyze specific cases where PHIER underperforms on ID examples. For instance, in CALVIN, we hypothesize that PHIER may struggle with tasks that the baselines may memorize due to their less constrained representations. We show an example in Figure 7 and note that for the ID query, Turnedon (lightbulb), Re-Attention correctly predicts True, while PHIER predicts False. However, for the -ittr--distribution query, Turnedoff ff f (1 (1i (ighhbbbb, which is linguistically similar but semantically opposite, PHIER generalizes successfully while Re-Attention struggles to adapt. We conjecture that Re-Attention may predict that TurnedOn (1 (lightbulb) is True based solely the existence of the bulb at the location, instead of learning that the state of the lightbulb depends on its color (yellow is on and white is off). In contrast, we see that although PHIER's 's constrained representation may slightly limit learning capacity for ID settings, PHIER has the potential to conduct better compositional reasoning in OOD scenarios, where PHIER significantly outperforms baselines.
In-Distribution Query TurnedOn (lightbulb)
Dui-Of-Distriution Query TurnedOff (lightbulb)
Figure 7: An example of the query, TurnedOn (1ightbuub), and OOD query with novel predicate, Turnedoff (lightbulb).
A.8 OBIECT-CENTRIC ENCODER PERFORMANCE
We empirically that PHIER's object-centric encoder performs well even environments with significant distribution shifts, such as CALVIN. In Figure we show an example how the encoder localizes objects in CALVIN. To adapt environments with even larger distribution shifts where the
17

---

Published as a conference paper at ICLR 2025
performance may decrease, we note that PHIER's object-centric encoder can be finetuned with more data as well.
Input
Masked Images
Image
Red Block
Bluc Block
Figure 8: PHIER's object-centric encoder in the CALVIN environment.
18

---

Published as a conference paper at ICLR 2025
B BASELINE DETAILS
For all of our baseline methods, we preprocess our input queries by converting the states into questions using the following templates:
For unary states: "Is the (object) (predicate)
For binary states: "Is the {object 1} (predicate) the {object 2}"
B. SUPERVISED METHODS
We train all of the supervised baselines on the same training data as our method. Below, we describe each baseline and provide implementation details:
BUTD (Anderson et al., 2018). BUTD uses bottom-up attention extract image features for important image regions and then top-down attention to focus on image regions based on the input query. We follow the original method, using Faster R-CNN pretrained on Visual Genome to extract bottom-up features for the top 36 image regions. For the text features, we embed the preprocessed text queries using 300-dimension word embeddings, initialized with pretrained GloVe vectors, and a GRU. The image and text features are then fed into model, based on the Py Torch implementation of the BUTD model for VQA (uu et al. 2020
CLIP (Shen et al. 2021). We use pretrained CLIP vision and text encoders extract features for the input image and query. respectively. These features are concatenated and passed through a small 2-layer network with hidden layer of dimension 256 for state classification.
CoarseFine (Nguyen et al., 2022). Coarse to Fine learns to reason about scenes with complex semantic information by extracting image and text features at multiple levels of granularity. We follow the official implementation of the Coarse to Fine reasoning framework and use Faster R- CNN extract image-level features and GRU with 300-dimensional GloVe embeddings extract question-level features, which are then fed into the model]
FiLM (Perez et al. 2018). FiLM conditions an input image on text by applying learned transforma- tions to the image features. We use pretrained ViT-16 16 image encoder and BERT text encoder to extract image and query features. Then, FiLM layer is applied condition the image features on the query features, and the conditioned features are passed through small 2-layer network with hidden layer of dimension 256 for final prediction.
Re-Attention (Guo et al., 2020). Re-Attention introduces attention mechanism to re-attend to objects in the images, based on the answer to the question. We follow the original implementation by using a Faster R-CNN model pretrained on the Visual Genome dataset to extract object-level image features, and 51 512ddimensional LSTM initialized with 300-dimensional GloVe embeddings to extract query features
ReIViT MM et al. 2022). ReIViT enhances the reasoning ability of vision transformers by introducing concept-feature dictionary that enables efficient image feature retrieval during training. This supports global task to promote relational reasoning and a local task to learn semantic object-centric correspondences. We the official implementation, with Faster R-CNN extract image region features, MCAN-Small as our VQA model, and the KeeeeNeeet--Peeeaaininn PVTv2b2 as our vision backbone
SORNet (Yuan et al. 2022). SORNet extracts object-centric representations from input RGB images, conditioned on a set of object queries represented as images of the objects. to enable generalization to unseen objects on various spatial reasoning tasks. It performs state classification by training readout networks to predict spatial relations based on the object embeddings. For a fair comparison to our method and other baselines, we use MDETR (Kamath et al. (02T) to detect regions corresponding to object text, resize then 32 x 32, and then use them as the input object images to train SORNet.
https://githubbcom/aavaaaaCCCCC VQA
http:///tthhh..oom/gyyy//ttttttt https://gthubbccom///aaaavvv
19

---

Published as a conference paper at ICLR 2025
We train readout networks for each training state in dataset ! Since SORNet requires training separate network for each predicate, we only evaluate it on in in-distribution states.
B.2 PRETRAINED LARGE VISION LANGUAGE MODELS (VLM)
All of the pretrained large VLM baselines are evaluated inference-only.
BLIP-2 (Li et al. 2023b). We use BLIP-2 leveraging the OPT-2.7b language model and treat VQA as an open-ended answer generation problem. The input image is provided along with a query using the following format: "Question: {state query as question] Answer:"
GPT-4V (peenA 2023). We provide GPT-4V with the input image and prompt based on the following template:
Prompt Template For GPT-4V Inference
Given an image of scene, you will answer question regarding stases and relationships of objects in the scene. The question is the following:
(state query question)
need carefully examine the image, thoughtfully consider objecta in the scene, and analyze their states and relationshi before answering the question.
Provide your answer as True or False, and strictly follow this responsa format: [insert your answer as True or False here]
Answer:
Reasoning: [insert your reasoning here]
Figure 9: Prompt template for GPT-4V experiments.
ViperGPT (Surís et al. 2023). We use the official ViperGPT implementation with Blip-2 Flan-T: XXL as the pretrained model and GPT-4 for code generation. Our data is formatted according the ViperGPT specifications, with the input image and query as question.
B.3 ABLATION DETAILS
Here, we provide clear breakdown of our ablation model architectures from Table and explain how we add each component.
Supervised model. We start with a supervised baseline model, which uses an image encoder and text incoder initialized with CLIP and BERT weights, respectively. The embeddings from both encoders are concatenated and passed through small MLP with three linear layers for classification, and the full model is trained with binary cross-entropy loss based on the ground truth labels (True or False). We then progressively add each component of PHIER.
Object-centric encoder. First, we incorporate the object-centric encoder by replacing the image encoder, text encoder, and concatenation step with our proposed object-centric encoder, while retaining the MLP and loss.
+ Predicate triplet loss. Next, we introduce the predicate triplet loss by adding this term to the total loss function without changing the architecture.
+ Norm regularization loss. We further add the norm regularization loss to get the total loss function with all components, described in Section 3
+ Hyperbolic metric. Finally, we lift the scene representation to hyperbolic space using an exponential map and replace the first two linear layers in the MLP with hyperbolic linear layers of the same size. We also use the Poincaré distance metric instead the Euclidean metric in the self-supervised losses, yielding our final model (PHIER).
20

---

Published as a conference paper at ICLR 2025
C HYPERBOLIC GEOMETRY PRELIMINARY
We briefly introduce the Poincaré ball model of hyperbolic space and hyperbolic neural networks. For more detailed explanation, refer the reader to Cannon et al. (1997) and[a et al. 2018).
As discussed in the main text, the Poincaré ball is a d-dimensional ball of radius 1, Pd = for E R" ||~|| < 1}. where || is the Euclidean norm. The ball is equipped with the metric tensor Up = (A,) Se. where A |ー|| is the conformal factor and is the Euclidean metric tensor
(i.e., the Euclidean dot product). This induces the Poincaré distance dp between two points z, y E Pd
as follows:
2 V 31 11
dp v) cosh"
cosh 1 +
|
Möbius addition. On the Poincaré ball, Euclidean operations such as addition and multiplication have equivalents ensure that all operations remain within the hyperbolic space and respect its geometry. Instead using standard Euclidean addition, Möbius addition is used, which ensures that the sum of two points the Poincaré ball still lies within the ball. The Möbius addition for any two points æ, E 1pd is defined as:
(1 (zx y)
|1 P)r (1
y
2xx y)
Exponential and logarithmic maps. To perform operations in hyperbolic space, we use exponential and logarithmic maps to map Euclidean vectors to the hyperbolic space, and vice versa. For any point z E _ the closed form expression of the exponential and logarithmic maps centered around z are defined as:
A= A>=1ee11
exp_ (v)
tanh
2
|||||
log, (v) = tanh ((l 2 + vll)
2 X=
- 2 +
-|- + yl
In practice, use the maps centered at 0, expo and loga. transition between Euclidean space and the Poincaré ball.
Hyperholic neural networks. Ganea et al. (2018) proposes hyperbolic neural networks by defining hyperbolic equivalents linear maps and bias translations. The hyperbolic linear map M R 1 of any point pdd on the Poincaré ball is defined as:
[|Mx tanh -l (Vd[[=l))
Mx ||Ma||
M (x) = (1/vc) tanh
||ж||
The translation of a point pd by bias Td as:
to Au
logo(c)
+ b = expx
The hyperbolic linear layer then defined as (x) + b. To build hyperbolic neural network, one simply has to map representations to the Poincaré ball using expo- apply hyperbolic linear layers, and then map back to Euclidean space using logo
Disk area of hyperbolic space.
We provide further details on why the exponential growth of the disc area in hyperbolic space provides natural and efficient way to represent trees. Note that for regular tree with constant branching factor b, the number of nodes increases exponentially with the distance from the root, as (b + 1)6² We can embed trees in hyperbolic space, they mirror this exponential growth. For instance, in two-dimensional hyperbolic space with constant curvature K = -1, the circumference of disc with radiuds is 2x sinh r while the area of dise is 2n (cosh r - 1). Since sinh r = = e-")
and coshr = 1 (c" + t"),, both the circumference and area the disc grow exponentially with the radius.
2
This exponential growth allows us to efficiently embed tree structures in hyperbolic space: nodes that & levels from the root can be placed the hyperbolic disc with radius proportional its level
21

---

Published as a conference paper at ICLR 2025
while nodes less than & levels within the sphere. Thus, see how this property allows hyperbolic space to serve as continuous representation of discrete trees.
Connection between hyperbolic space and hierarchical structure. We highlight several prominent prior works who have made theoretical connections between hyperbolic space and trees. Mathematical works such as Gromov (1987). Dyubina & Polterovich (001), Hndaan (018)) prove that any finite tree can be embedded into a finite hyperbolic space with approximately preserved distances. key property of hyperbolic space is its exponentially growing distance, and they show that this underlying property makes hyperbolic space well-suited to model hierarchical structures. Furthermore, works such as Sala al.) (2018) and Chami et al. (2020) propose concrete approaches embed any tree in hyperbolic space with arbitrarily low distortion, establishing upper upper and lower bounds for distortion and further demonstrating the effectiveness of hyperbolic space for hierarchical modeling.
Notably, Nickel & Kiela (2017) were among the first to explore learning hierarchical representations in hyperbolic space. They found that for data with latent hierarchies, embeddings on the Poincaré ball outperform Euclidean embeddings significantly in terms of representation capacity and generalization ability. Since then, hyperbolic spaces have been increasingly explored for modeling hierarchies across various domains, including NLP (Ganea et al., 2018 Nickel & Kiela, 2018 Tifrea et al. 2018) and computer vision (Khrulkov et al. 2020 Ermolov et al. 2022). with substantial empirical evidence supporting its efficiency and suitability for modeling hierarchical structures in comparison to Euclidean space. We believe that these prior works provide strong theoretical justification and empirical support for the connection between hyperbolic space and hierarchical structure, which inspires our method.
Implementation. We implement hyperbolic encoder using the Geoopt package (Kochurov et al. 2020), which provides functions and optimization methods for hyperbolic space
hhmps//gittub..om/ggmnm/gooopppp
22

---

Published as a conference paper at ICLR 2025
D LLM PROMPTS FOR SELF-SUPERVISER LOSSES
Here, we present the prompt templates used to extract explicit knowledge of predicates from LLMs. In Figure I0] we describe the prompt used determine the assignment (anchor, positive predicate. and negative negative) for a given triplet of predicates, used in the predicate triplet loss. In Figure II we show the prompt used to determine the hierarchy among a predicate triplet based on specificity, for the norm regularization loss. We query the LLM once before training starts to retrieve the predicate triplet pairs and hierarchy, hence training is not affected by LLM queries.
Prompt Template For Predicate Triplet Assignment
You are given anchor text query that describes state of scene. Given two other text queries describing the staze of a scene, you will help decermine which of the two queries is more similar to the anchor query.
Consider the semantic meaning the states and the specific aspects the scere they descrite. Additionally ly, think about how many objects and what kinds of object properties and features you would need verify if evaluating these states against image.
The anchor query is the following: {anshor)
other queries Query 1: {query]]
Query 2:
(query2)
You must choose one of the queries as your answer. following format:
Reapond using the
Answer: [Query 1 or Query 2]
Figure 10: Prompt template for inferring the predicate relations among a triplet with GPT-4.
Prompt Template For Triplet Hierarchy Ranking
You are expert in scene anderstanding and state hierazchy determination. Given three text descriptions each outlining a potential state of a scene, your task is to establish hierarchy among these descriptions by ideatifying which one is the most general, which is the most specific, and which lies betweer.
Consider the following when determining the hierarchy: The variety and number of objects requized the state.
The important features of the objects and/or relationships between the objects.
The level of detail provided about the scene.
The semantic meaning each descriptior.
Your goal to rank these descriptions in order specificity, from least specific (1) to most specifis (3).
three descriptions are:
1.
(anchor)
2.
/queryl
3.
[query2)
You must provide your ranking using the following format:
Least Speci [content of Description 1, 2, or 3] Insermediate Specific: [content of Description 1, 3] Most Speci fic: [content of Description it 1, 2, or 3]
Figure 11: Prompt template for inferring the hierarchy among a triplet with GPT-4.
23

---

Published as a conference paper at ICLR 2025
E DATASET DETAILS
E.1 DATASET STATES
In Tables 8 and
we provide all of the states included in the CALVIN and BEHAVIOR datasets.
Table 8: All states included in the CALVIN dataset.
State Type
Predicate
Object
Object 2
Lifted
blue block
ID
OnRight
slider
ID
Open
drawer
ID
TurnedOn
lightbulb
Inside
blue block
drawer
ID
Inside
pink block
drawer
ID
OnTop
blue block
table
ID
Stacked
blue block
pink
block
ID
Stacked
blue block
red block
OOD
Closed
drawer
OOD
Lifted
pink block
00D
OnLeft
slider
00D
TurneOOff
lighthulb Inside
0OD
blue block
slider
00D
OnTop
pink block
table
0OD
Stacked
pink block
blue block
0OD
Under
table
blue block
E.2 BEHAVIOR VISION SUITE VISUALIZATIONS
We include additional examples from BEHAVIOR Vision Suite Ge al. (0044) in Figure 12
Open (cabine_) .
Under tean, Lable)
Trsi de toup, dzawer)
Inside (cup, cabinet)
OnTop (boLLle, Lable)
OnTopp Ibread, chair)
Figure 12: Visualizations state classification tasks from the real-world BEHAVIOR Vision Suite dataset.
24

---

Published as a conference paper at ICLR 2025
Table 9: All states included in the BEHAVIOR dataset.
State Type
Predicate
Object
Ohject
ID
Open
bottom cabinet
ID
Open
drawer
ID
Open
microwave
ID
Open
oven
ID
Open
top
cabinet
Inside
apple
top cabinet microwave microwave
ID
Inside
club sandwich
ID
Inside
pizza
ID
Inside
plate
bottom cabinet bottom cabinet top coffee cup cola bottle cheesecake microwave coffee cup coffce cup cola bottle cheesecake microwave coffee cup plate plate breakfast table
ID
Inside
plate
ID
NextTo
apple
ID
NextTo
coffee cup
ID
NexiTo
croissant
ID
NextTo
pizza
ID
NexiTo
plate
ID
OnLeft
apple
ID
OnLeft
coffee cup
ID
OnLeft
croissant
ID
OnLeft
pizza
ID
OnLeft
plate
ID
On'Top
apple
ID
OnTopp
cheesecake
ID
OnTop
coffee cup bottle
ID
OnTopp
countertop
ID
On'Top
plate
breakfast table
ID
Touching
apple
plate
ID
Touching
cheesecake
plate
ID
Touching
coffee cup
breakfast table
ID
Touching
bottle
breakfast table
ID
Touching
croissant
plate
OOD
Closed
bottom cabinet
OOD
Closed
drawer
OOD
Closed
microwave
00D
Closed
top cabinet
0OD
Contains
bottom cabinet
plate
0OD
Contains
drawer
plate
00D
Contains
top cabinet apple
drawer
00D
Inside
microwave
00D
Inside
coffee cup
top cahinet
00D
Inside
plate
microwave
00D
NexiTo
apple
plate
00D
NexiTo
plate
microwave
00D
OnTopp
coffee cup
plate breakfast table
00D
OnTop
apple
00D
OnToop
apple
microwave
00D
OnLeft
apple
plate
00D
OnLeft
coffee cup
apple
00D
OnRight
apple
coffee cup
00D
OnRight
coffee cup
cola bottle
00D
OnRight
plate
coffee cup
00D
Touching
apple
breakfast table
00D
Touching
coffee cup
plate
00D
Under
breakfast table
coffee cup
0OD
Under
breakfast table
plate
00D
Under
plate
apple
25


Reviews:
[{"review_id": "LTiOcS4ugG", "reviewer": "Reviewer_5wzn", "summary": "The paper proposes PHIER, a state classification model that leverages predicate hierarchies to achieve few-shot and out-of-distribution generalization for robotic state classification tasks. PHIER incorporates an object-centric encoder, self-supervised losses to infer semantic relationships between predicates, and a hyperbolic distance metric to encode hierarchical structures. The approach is tested in simulated environments (CALVIN, BEHAVIOR) and in real-world settings.", "questions": "Here are questions that align with the identified limitations:\n\n1. What factors contribute to the proposed method's lower performance compared to baseline methods on in-distribution samples? Could more detailed analysis and insights be provided?\n2. Why were pre-trained models not evaluated on ID-OOD samples or real-world samples, and what impact would their inclusion have on the results?\n3. In the ablation study, why were the components added sequentially without considering the effect of different orderings? Has an ablation analysis been conducted that removes individual components to assess their independent impact?\n4. How were the positive and negative pairs constructed, and what methodology was used to build the predicate hierarchy? Can more detailed explanations be provided?\n5. Given the small sample size (100) and limited diversity of the collected real-world dataset, how can the proposed method avoid overfitting to this specific setup? Why was it not evaluated on larger, more diverse VQA datasets that include object relations and state information?\n6. How many examples are needed for the proposed method to generalize effectively to unseen object-predicate combinations and novel predicates? Is there an ablation study examining the effect of varying the number of examples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Incorporates predicate hierarchies effectively for few-shot state classification.\n2. Demonstrates strong generalization from simulator images to real-world images.\n3. Utilizes hyperbolic space to encode complex hierarchical relationships efficiently.", "weaknesses": "1. Why does the proposed method perform worse than baseline methods on in-distribution samples? More detailed analysis and insights are needed.\n2. Why are there no results for pre-trained models on ID-OOD samples or real-world samples?\n3. In the ablation study, components are added sequentially; however, the order of addition matters. Further analysis with ablations that remove each component should be conducted.\n4. Important details are missing, such as the process for constructing positive and negative pairs and how the predicate hierarchy is built.\n5. The experimental evaluation requires improvement. While the collected real-world dataset is valuable, the sample size (100) is too small, and the scenario settings lack diversity, which may lead to overfitting. There are numerous existing VQA datasets that include object relations and state information. The proposed method should be evaluated on these larger and more diverse benchmarks.\n6. The proposed method claims to generalize to unseen object-predicate combinations and novel predicates with few examples. However, the specific number of examples used is unclear, as it is not mentioned in the paper or the appendix. And is there an ablation study analyzing the effect of varying the number of examples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42Gx7pLGOU", "reviewer": "Reviewer_vCxY", "summary": "This work suggests a novel method for state classification with enhanced few-shot generalization capabilities. The method encodes latent features extracted from an Image and a query (in the form of an object conditioned predicate) onto a hyperbolic latent space using a unique loss and regularizer to enforce explicit learning of predicate hierarchies. The authors leverage the natural hierarchical structure that emerges in hyperbolic space based on previously seen predicates in the hierarchy to quickly learn and adapt to new, unseen objects and predicates.", "questions": "- in line 228 the authors define the image encoding with a max over all object mask norms. doesn't this cause the final embedding to ignore the objects whose norm\n- shouldn't the margin hyperparameter (lambda) be chosen dynamically depending on the current triplet? Otherwise this encourages a minimal margin that lead to equal distances between values that have significantly different semantic meaning.\n- Why does the hyperbolic space's disc area growing exponentially mean that it is a continuous representation of discrete trees? There seems to be no connection between trees and the hyperbolic space according to the information provided in this work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- All deep learning components and encoding strategies are based on strong intuitions and deep understanding of the different latent spaces and tools used. \n- Empirical results show significant improvement upon current state of the art.\n- Tested on environments with varying levels of complexity and realism.\n- Well cited with a comprehensive literature review in the related work section\n- The method is clearly presented bit by bit, slowly building up the method components in a readable fashion.\n- Provides ablation tests that show the necessity of each component.", "weaknesses": "- Theoretical justification of the method is relatively weak. It would be groundbreaking to show a strongly linked connection between hyperbolic space and hierarchical structure. This point is still unclear, even though the authors provide some intuition for it.\n- Introduction lacks citations, making it difficult to understand which parts are novel and which belong to previous work (before diving in to the rest of the paper).\n- No evidence is provided for claims on that large vision models struggle with spatial relationships, nor is there mention of multi-perspective image processing with these kinds of models (which emulates 3D and improves spatial understanding).\n- The authors assume much preliminary knowledge from the reader, e.g., acronyms like CLIP, BERT, ViT, etc., are never defined. Even though these are common terms in the deep learning and NLP communities, Some readers might still not be familiar with everything which requires reference hopping and a general breaking of the reading flow.\n- tested on very small datasets which makes the results less statistically significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10EZ1lHNgA", "reviewer": "Reviewer_V1Qj", "summary": "This paper proposes PHIER, a method for few-shot state classification by learning hierarchical predicate representations in a hyperbolic latent space. The proposed method consists of three main modules:\n\n A) an object-centric scene encoder based on CLIP. \n\nB) a triplet loss for predicate similarities based on rankings retrieved from an LLM. \n\nC) A Latent space with hyperbolic structure for encoding predicate hierarchies.\n\n\nExperiments on two robotic simulation environments and a new real-world image dataset are performed to conclude the superior performance of PHIER in contrast to other supervised models or VLMs, especially in a few-shot setting.", "questions": "- From the text, it remains unclear to me if the object-centric encoder is part of the contribution or an application of Mask CLIP. Please clarify.\n- An additional visualization/analysis of the learned Poincare Ball structure as part of the Appendix would highlight the core contribution of the method and further strengthen the paper.\n- Why do you choose a Bert encoder over the CLIP text encoder?\n- How sensitive is the model to hyperparameters (loss weights) in different envs? How did you determine them?\n- How fast is the training when you have to query the LLM for hierarchy ranking during Training? Do you use some buffer to not reprompt the LLM every time?\n- Why does the  GPT4-V reasoning happen after the model gives its answer?\n- Is the ID evaluation done after few shot training or before?\n- How does the object-centric encoder work for environments with significant distribution shifts, such as CALVIN?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written, and the method is clear and easy to follow.\n- Most design choices are justified and effective, as shown with ablations.\n- Good experimental setup and results in simulation. The results regarding novel predicates show that the learned latent space is efficient, especially compared to baselines.", "weaknesses": "- The ablation section needs more details\n    - In its current state, is not clear what type of architecture the ablation baseline models utilize. What is the architecture of the supervised model? Just an image encoder and the predicate encoder followed by the supervised loss? This needs clarification. What is the hyperbolic linear layer replaced with? Simple MLPs?\n- The network architecture is unclear from the current state of the method section. What are the trainable parameters? Where exactly do you introduce new parameters in the architecture? Only to project into the latent space and back? Including this in Figure 2, for instance, would make it easier for the reader to grasp this.\n- Although the results in simulation are convincing, I have a few concerns:\n    - The evaluation scenarios are all very simple and do not include any distractors.\n    - You do not evaluate on unseen objects. The object-centric encoder should easily allow it to adapt to new unseen objects. Experiments in that regard could further strengthen the contribution of the paper.\n    - The SORNet baseline uses an Mdetr object detector. Why do you not use your object-centric descriptor to generate the reference embeddings for SORNet? This would allow for a fairer comparison. This is a minor concern as the method does not require these representations in the first place.\n    - You do not evaluate GPT4-V on the real-world benchmark. GPT4-V is trained on real-world data, presenting a strong baseline in this case.\n    - It would be very interesting to see the few-shot transfer of the method to real-world tasks. You show that few-shot transfer works well in Simulation, so showing that it also works for sim to real transfer would strengthen the contribution.\n    - The zero-shot results in real-world are not convincing. Although you outperform baselines by a large margin (which are frequently worse than random guessing), the success rate is still low. This is why I would recommend evaluating the method on few-shot transfer to real-world.\n\nOverall, the simulation results are convincing, but for this paper to have a higher impact, more challenging evaluation settings and additional real-world experiments are required. Nevertheless, the results regarding few-shot transfer to new predicates are very promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)