InfoCon: Concept Discovery with Generative and Discriminative Informativeness
Authors: Ruizhe Liu
Qian Luo
Yanchao Yang
Venue: ICLR 2024 poster (2024)
Keywords: concept discovery
robotic manipulation
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: We focus on the self-supervised discovery of manipulation concepts that can be adapted and reassembled to address various robotic tasks. We propose that the decision to conceptualize a physical procedure should not depend on how we name it (semantics) but rather on the significance of the informativeness in its representation regarding the low-level physical state and state changes. We model manipulation concepts -- discrete symbols -- as generative and discriminative goals and derive metrics that can autonomously link them to meaningful sub-trajectories from noisy, unlabeled demonstrations. Specifically, we employ a trainable codebook containing encodings --symbols -- capable of synthesizing the end-state of a sub-trajectory given the current state (generative informativeness). Moreover, the encoding corresponding to a particular sub-trajectory should differentiate the state within and outside it and confidently predict the subsequent action based on the gradient of its discriminative score (discriminative informativeness). These metrics, which do not rely on human annotation, can be seamlessly integrated into a VQ-VAE framework, enabling the partitioning of demonstrations into semantically consistent sub-trajectories, fulfilling the purpose of discovering manipulation concepts and the corresponding (sub)-goal states. We evaluate the effectiveness of the learned concepts by training policies that utilize them as guidance, demonstrating superior performance compared to other baselines. Additionally, our discovered manipulation concepts compare favorably to human-annotated ones, while saving much manual effort. The code and trained models will be made public.

Published as a conference paper at ICLR 2024
INFOCON: CONCEPT DISCOVERY WITH GENERATIVE AND DISCRIMINATIVE INFORMATIVENESS
Ruizhe Liu Qian Luo', Yanchao Yang! 1.3
'HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong Sccoool of Electronics Engineering and Computer Science, Peking University
Department of Electrical and Electronic Engineering, The University of Hong Kong 1rz360estu, pku. edu. en, {yanchaoy, gia..uo
ABSTRACT
We focus on the self-supervised discovery of manipulation concepts that can be adapted and reassembled address various robotic tasks. We propose that the decision to conceptualize a physical procedure should not depend on how we name it (semantics) but rather on the significance of the informativeness in its representation regarding the low-level physical state and state changes. We model manipulation concepts discrete symbols - as generative and discrimi- native goals and derive metrics that can autonomously link them to meaningful sub-trajectories from noisy, unlabeled demonstrations. Specifically, we employ a trainable codebook containing encodings (concepts) capable of synthesizing the end-state of a sub-trajectory given the current state generative informativeness. Moreover, the encoding corresponding to particular sub-trajectory should dif- ferentiate the state within and outside it and confidently predict the subsequent action based on the gradient of its discriminative score discriminative informa- tiveness. These metrics, which do not rely on human annotation, can be seam- lessly integrated into VQ-VAE framework. enabling the partitioning of demon- strations into semantically consistent sub-trajectories, fulfilling the purpose of dis- covering manipulation concepts and the corresponding sub-goal (key) states. We evaluate the effectiveness of the learned concepts by training policies that utilize them as guidance, demonstrating superior performance compared to other base- lines. Additionally, our discovered manipulation concepts compare favorably to human-annotated ones while saving much manual effort. Our code available at: attps://2rl rz. qithub. Lo/InfoCon/
INTRODUCTION
Conceptual development is of core importance to the emergence of human-like intelligence, ranging from primary perceptual grouping to sophisticated scientific terminologies (Sloutsky 2010). We focus on embodied tasks that require interaction with the physical environment and seek the ma- nipulation concepts that are critical for learning efficient and generalizable action policies (Lázaro- Gredilla et al.] 2019 Shao et al.) 2021). Recently, Large Language Models (LLMs) have enabled many interesting robotic applications with their reasoning capabilities that can break complex em- bodied tasks into short-horizon interactions or manipulation concepts (Singh et al.) 2023). However, LLMs are trained with an internet-scale corpus, representing a vast amount of linguistic knowledge of manipulations, but lack embodied experiences that ground these manipulation concepts to spe- eific physical states (Brohan et al. 2023 Huang et al., 2023). In contrast, it worth noting that in human development, infants initially acquire physical skills, e.g., crawling, grasping, and walking, before delving into the complexities of language, which grant the manipulation concepts described in language with groundingness in the first place (Walle & Campos 2014 [iberruss & Violi 2016).
Drawing inspiration from the human developmental process, we aim to equip embodied agents with manipulation concepts intrinsically connected to physical states without relying on additional grounding techniques. Specifically, we seek algorithms that allow agents to abstract or identify
Work done research assistant HKU.

---

Published as a conference paper at ICLR 2024
Manual Labeling
Grasp
Align
Grasp
Align
Insert
Demos
42
44
d5 G
Kgg ds (lg Manipulation
Concepts
InfoCon
Figure 1: The proposed generative and discriminative informativeness and the derived InfoCon al- gorithm discover manipulation concepts from noisy, unlabeled demonstrations, Each identified whole trajectory into trajeerrcctoess,
concept relates to sub-goal and defines the partitioning showing the process governed by the concept for achieving the sub-goal. Concepts from InfoCon share similarities with human-annotated ones while having more fine-grained semantics, which can be more beneficial for physical interactions but time-consuning label manually.
manipulation concepts from their embodied experiences, such as demonstration trajectories, with- out humans specifying and annotating the involved concepts. The discovery process serves dual purpose. Firstly, it generates set of discrete symbols that hold semantic meaning, with some po- tentially representing concepts like "grasping block" "aligning block with hole the wall." Furthermore, the discovery process should explicitly establish correspondences between the con- cepts and physical states, thereby achieving the grounding for low-level actions while minimizing annotation efforts.
Therefore, propose model manipulation concepts generative and discriminative goals. Specifically, as a generative goal, manipulation concept should help predict the goal state even though it has not been achieved yet. For example, when given the manipulation concept "grasp the block," one can already (roughly) synthesize how the scene looks when the robotic gripper grasps the block. We formulate as the informativeness between the manipulation concept and the syn- thesized ending state when the interaction implied by the manipulation concept is accomplished, which we name generative informativenesx. On the other hand, given a manipulation concept as discriminative goal, one could tell if the current state is within the process of achieving the goal state implied by the concept. For example, with the concept "place the cup under the faucet," one would assign low compatibility to the state of "pouring water the mug" compared to the correct one. Similarly, we can formulate it as the informativeness between the manipulation concept and the binary variable, indicating whether the state falls within the manipulation process, which we call discriminative informativeness. Moreover, as discriminative goal, manipulation concept should inform the following action. If an action incurs higher discriminative score (compatibility), then should be executed to accomplish the task. Accordingly, we formulate as the informativeness between the gradient of the discriminative function (conditioned on the manipulation concept) and the next action in the demonstration.
With the proposed metrics, we can train VQ-VAE Van Den Oord et al. 2017) architecture to learn the discrete representations of potential manipulation concepts from the demonstration, as well as the assignment (grounding) between the learned concepts and the physical states, even though no concept descriptions are available in any form. We examine the quality of the learned manipula- tion concepts through the automatically acquired grounding, and verify that these concepts do come with semantic meaning in terms of human linguistics Fig I We further demonstrate the usefulness of the learned manipulation concepts by using the grounded states as guidance to train manipulation policies. Experimental results show that the discovered manipulation concepts from unannotated demonstrations enable policies that surpass the state-of-the -ftte----t---tt counterparts, while achieving com- parable performance with respect to the oracle trained with human labels, which verifies the effec- tiveness of the proposed metrics and training for self-supervised manipulation concept discovery.
2 METHOD
Problem Setup We aim to characterize the multi-step nature of low-level manipulation tasks. More explicitly, assume access set of pre-collected demonstrations trajectories, i.e.,
2

---

Published as a conference paper at ICLR 2024
c [0.1] Grasp
S
a Grasp
dc
Ōs
S3
5=31
GENERATIVE GOAL
DISCRIMINATIVE GOAL
Figure 2: We characterize a manipulation concept from two conjugate perspectives. As generative goal, a manipulation concept helps synthesize the state when the physical process meant by the concept accomplished. On the other, as a discriminative goal, a manipulation concept indicates (through scoring function) whether a state lies within the process governed by it. Moreover, it informs the next action with the gradient of the scoring function, as the action taken should maximize the discriminative utility.
D D= [- and - ((s). T() which is a sequence of state-action pairs. Our goal is to partition each trajectory into semantically meaningful segments (continuous in time), while main- taining consistency between trajectories of the same task without resorting to human annotations. We call what governs the actions within particular trajectory segment the manipulation concept. And the "semantic meaning" of the manipulation concept lies in its generative and discriminative informativeness, which we discuss in Sec. 2.1 We further name the state the boundary of two segments as the key state (Tia et al.) 2023). We examine how the learned manipulation concepts align with human semantics, as well as the quality of the partitions, by evaluating the effectiveness of the derived key states physical manipulation benchmarks. We adopt the CoTPC Jia et al. (2023) framework to train concept-guided policies across different benchmarks. Next, we elaborate on the proposed partitioning metrics.
2.1 MANIPULATION CONCEPT AS GENERATIVE AND DISCRIMINATIVE GOALS
Suppose a task can be described by K manipulation concepts fax 1K with a E R.M. Or equiv- alently, a trajectory = {(s, VT= from this task could be divided into K segments {3k] with each / a continuous short-horizon trajectory and r = _ Note that we do not endow Og with any specific linguistic descriptions this moment (e.g.. "pick up cup" "place the cup under the faucet"). Instead, we treat them as symbols grounded the segments with consistency across different trajectories of the same task. In other words, assume the existence of a partition- ing function (neural network) & such that { 1 ((T). We detail the structure of & later and enable the training of & by proposing the following criteria.
As generative goal, a manipulation concept og shall, given the current state, inform the end state of BR, i.e., the key state 8ts with t = Σ [8y]. where | is the length of (sub)sequence. For example, with the manipulation concept that resembles "grasp the mug handle," one can imagine the picture depicting when the handle is firmly grasped. We formalize this metric using Shannon's mutual information:
ggen (a) = l( key [s), key E E (s)
We name the above the generative informativeness since knowing the manipulation concept can help confidently synthesize the imminent key state after 8. Please note that the manipulation concept, key state, and state are random variables depending on the trajectory, which is omitted for simplicity.
As discriminative goal, manipulation concept og should tell whether the current state is within the process described by an or not. For example, in the task to get water to drink, if o& resembles "approach the faucet," then the state before cup is grasped or the state "drinking water" should have low compatibility with ay in contrast to the states within the corresponding sub-trajectory of approaching the faucet. Thus, the knowledge of the manipulation concept ay helps distinguish the states governed by it from the other states. We characterize this phenomenon by instantiating compatibility function:
C: x - [000 [0,
(2)
such that values close to imply high compatibility of the state s with the process described by the manipulation concept a. We also write (.)) to reflect the fact that the manipulation concept
3

---

Published as a conference paper at ICLR 2024
indexes discrimination function, hence the role of discriminative goal and we call Eq. the discriminative informativeness of the manipulation concept.
Furthermore, we propose that the gradient of the compatibility function C* should be informative of the next action a. We believe, as a densely defined function, is beneficial that C not only indicates "what" the agent is doing but also "how" to do it by modulating the fluctuations of the compatibil- ity function around s. Still, considering the concept of "filling the cup with water," the action of putting the cup under the faucet shall be assigned higher compatibility than the action of moving the cup toward microwave. This illustrates that the compatibility function shall inform state changes (induced by actions) via the changes in its value, depicted through its gradient. Accordingly, we formulate this characteristic of C as:
dis CC) I(VC:a)))
ac
where VC
Ōs given the manipulation concept a. Consequently, we name Eq. 3 the actionable
informativeness of the manipulation concept as discriminative goal. We illustrate the idea of manipulation concept as both generative and discriminative goals in Fig. 2 which shows different aspects during manipulation process as represented by the proposed informativeness criteria.
In summary, we propose that manipulation concept should 1). inform the imminent key (physi- cal) state given the current state as generative goal: 2). indicate whether the current state within the process governed by itself discriminative goal, well as 3). inform the action with the gradient of the instantiated discriminative function Ca, Please note that these metrics do not count on human supervision. In other words, long we have a network & that takes in manipulation trajectory and outputs a set of continuous segments, we can train & using the proposed metrics to discover manipulation concepts (consistent partitions) shared across trajectories of different tasks. Next, we operationalize the self-supervised manipulation concept discovery by specifying the train- ing architecture and the objectives.
2.2 SELF-SUPERVISED MANIPULATION CONCEPT DISCOVERY
Network Structure of & We adapt the basic structure proposed in VQ-VAE (Van Den Oord et al., 2017] to accommodate the sequential nature of state-action sequence and the need for continuous sub-trajectories. Specifically, we employ transformer encoder ø that maps state st to a latent 2, in an autoregressive manner:
21
(4)
This autoregressive design alleviates the ambiguity in predicting the latent by supplying rich history information. Moreover, it helps smooth out noise in 2, to facilitate the partitioning of the trajectory into continuous uub-trajectories To further enhance the segmentation continuity, we devise a posi- tional encoding scheme for time f. and apply it to the latent 2,. The proposed positional encoding can capture the fact that nearby latents have good chance of being assigned to the same manipulation concept without causing over-smoothing, Please refer Sec. A. for more details. In the following. we abuse be for the latent appended with the proposed positional encoding.
Trajectory Partitioning with Manipulation Concepts To further process {=>1=> and derive the partitioning, we instantiate K trainable vectors, which serve as the manipulation concepts {az {ax)K= We then assign the state latent 2, to the concept that shares the largest similarity. More explicitly, denote as the concept assignment function, then we have:
TT(2)) arg max ox) = arg max expl(ze, Ow))
(5)
k
Ekm exp((2r, (x)
where p(z, ax) is the probability of assigning latent 2; to the concept ag, and (-, is the cosine similarity between two vectors. This assignment process allows us to group the states into segments {B i.e., binding states corresponding to the same concept, and serves as a ground to elaborate the training objectives. Note that we also need to ensure the gradient flow during training, thus, use a technique similar to the gradient preserving proposed in VQ-VAE Van Den Oord et al. 2017). Please Sec. A.5 for more details.
4

---

Published as a conference paper at ICLR 2024
traj.
ac
Polic π
ds
state action
State
Feature
traj.
Extractor
03g
(4.g
ag
S
09
C 1
Se(ar)
CA
Assiggnns
traj.
Figure 3: Training pipeline of InfoCon. Features extracted from the state-action trajectory (using 6) are compared with learnable concepts, which are grounded according to Eq. [ The generative the informative loss (Eq. trains to predict the key (end) state of sub-trajectory. The discriminative informative loss trains compatibility function C conditioned on the concept (Eq. 7 which tells whether state compatible with the concept. Moreover, actionable informativenes: loss trains policy for action prediction (Eq. 8). Together, these components enforce the grounding to be physically and semantically meaningful.
Training Objectives With the derived sub- trabttrajectoiis, we first locate the key state for every single state. Let of be the predicted concept for trajectory
time step from {ax }K and let T()) Then sf ((aa)) is
(l(a)) be the (to be achieved) key state of s from the trajectory т, {(s}, a() determined by setting ((a)) = arg min (u 2 L, a + a.+1). Next, we detail loss terms.
Generative Goal Loss. To maximize the generative informativeness (Eq. [ we employ recon- struction loss that is related to maximizing the mutual information (Hjelm et al. 2018). We in- stantiate network ( that predicts the key state from the current conditioned on the associated manipulation concept of;:
T() Σ
gen OL, di 0%)
100 ( o
st Staa))
with ||- 112 the L-2 norm. This term encourages af be informative the to-be-achieved key state by minimizing the prediction error.
Discriminative Goal Loss.
According the defining term (Eq. of manipulation concept discriminative goal, we instantiate network (C) as hyper-classifier, which decode manipu- lation concepts from {a into compatibility functions representing the discriminative goal of these concepts. More specifically, C should assign high scores to states in trajectory r' that governed by at. We formulate it as a binary classification and employ the cross entropy loss to maximize the discriminative informativeness (Eq.
Edis (a, (,C
1 K
Σ (at = aR) at=ax
1
log Ca. (5t) (az F ork)
1
Σ aipaa
log(1 Cav (st)))
Note that the quantity inside (-) is the classification error for a manipulation concept's classifier but is normalized to account for the imbalance between the positive and negative samples. Please see Sec. A.7 for details the hyper-classifier C.
To derive the loss for maximizing the actionable informativeness (Eq. we instantiate policy network т, which minimizes the prediction error of the action conditioned on the manipulation concept:
1
T(x)
acai
Adis (a, C.
(s)
h4) afll3
N
Os ==o
ac
where In = (1bt u
is the history, and we use L-2 norm for the discrepancy.
0s s=s!
Please note that the policy employed here is mainly for the purpose of manipulation concept dis-
5

---

Published as a conference paper at ICLR 2024
P&PP Cube
Stack Cube
Turn Faucet
Peg Insertion
Figure 4: Examples of the manually defined key states (concepts) in different manipulation tasks. From left to right: P&P Cube and its two key states ("Grasp". "End"), Stack Cube and its three key states ("Grasp A", "A on D", "End"). Turn Faucet and its two key states ("Contacted",, "End"). Peg Insertion and its three key states ("Grasp", "Align", "End").
covery, which is different from the policy in downstream manipulation tasks (Please see Sec. 3.1 for more details). By minimizing the above quantity, the information in the partial derivative about the next action should be maximized. Further, following Van Den Oord et al. (2017). we add term that encourages more confident assignments by Eq. i.e.
Cen (am @)
Σ k=1
Σ log(p(z) ax)),
(ot =o
at=ax
which is similar to minimizing the entropy the assignment probability function p(z, + os.).
Finally, the total training loss for self- relf-supervised manipulation concept discovery with the proposed generative and discriminative informativeness metrics can be summarized as:
_ oz, 8 + L (ax Φ; C, π) + X (a, o; C) L (a, +)),
(10)
with balancing the importance between the prediction and the classification terms. More im- plementation details can be found in Sec. A and Sec. B and we study the effectiveness of each component in the experiments.
EXPERIMENTS
3.1 EXPERIMENTAL SETTINGS
We evaluate the effectiveness of InfoCon and the derived key states on four robot manipulation tasks from ManiSkill2 (Gu et al. 2023). an extended version of ManiSkill (Mu et al.) (002T): P&P Cube: To pick up a cube, move it to a specific location, and keep still for a short while. Stack Cube:
To pick up cube and put it on the top of another cube. Turn Faucet: To turn different faucets for To pick up a cuboid-shaped peg and insert it into hole in given object. We visualize the manipulation tasks with the manually defined and grounded key states from CoTPC (Jia et al. 2023) in Fig.
enough angle values. Peg Insertion:
Training details. Following the setup of CoTPC, choose the ground truth key states) for the four tasks above for training and evaluation. Specifically, we collect 500 training trajectories for each task, 100 evaluation trajectories for P&P Cube and Stack Cube, 100 evaluation trajectories for seen faucets, 400 trajectories for unseen faucets, and 400 evaluation trajectories for Peg Insertion. With the pipeline shown in Fig. we train the model and use the state encoder (Eq. 4 along with the concept assignment module (Eq. 5) to perform the partitioning and grounding of all trajectories. We choose the last state (of sub-trajectory) as the key state of the corresponding manipulation concept. After collecting the discovered key states, we train a CoTPC policy optimizing the action prediction with the key states as guidance. We leave out the key state prediction in the policy training when the key state does not appear in the trajectory (e.g., concepts discovered for one task may not appear in another). Please see Sec. A and Sec. B for more details.
of trajectories (without
Evaluation metrics. The efficacy of our approach is mainly assessed by the task success rate of policies that are trained utilizing the key states identified via InfoCon. Additionally, for curiosity, we introduce metric for rough understanding of whether there similarities between the predicted key states and the ground truth key states (designed by humans). Suppose the time steps of ground truth key states are {1 K 1k=1, and the time steps of predicted key states are {t, K always equal to K). We propose the Human Intuition Similarity, HIS, as the following:
J== (here K" is not
K'
HIS H
1kml
arg min, (1, (2222)
(11)

---

Published as a conference paper at ICLR 2024
Table I: Success rate (%) of different methods. For the task Turn-Faucet, we provide results on three kinds of situations: seen environments (s.&sf.), unseen environments with seen faucet types (u.&sf.), and unseen environments with unseen faucet types (u.&uf. ! * marks key state labeling methods dependent on human semantics (zero-shot), and marks key state labeling methods intrinsically related to the inherent properties of the data.
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
SR( %)
seen
unseen
seen
unseen
S.&sf.
u.&sf.
u.&uf.
seen
unseen
Decision Transformer
65.4
50.0
13.0
7.0
39.4
32.0
9.0
5.6
2.0
Last State"
70.8
60.0
12.0
4.0
47.8
46.0
21.0
9.6
5.3
AWB
96.2
78.0
45.4
18.0
53,4
50.0
14.5
31.6
4.3
LLM+CLIP*
92.2
71.0
44.6
21.0
35.8
31.0
18.0
63.2
13.5
GT Key States*
75.2
70.0
58.8
46.0
56.4
57.0
31.0
52.8
16.8
InfoCon
96.6
81.0
63.0
47.0
53.8
52.0
17.8
63.6
17.8
Table 2: Human Intuition Score (HIS) of different discovery/grounding methods.
HIS
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
LLM+CLIP 57.51
93.19
84.39
164.29
AWE
0.003
2.49
14.72
9.25
InfoCon 2.60
8.98
18.03
9.05
The HIS metric quantifies the temporal distance between a given ground truth key state and the nearest subsequent predicted key state. Thus, the HIS metric measures similarity without being affected by the number of discovered key states.
3.2
MAIN RESULTS
Baselines. We first consider several popular baselines without ofhin-of-TThuglle key state predic- tion: Vanilla BC (ooabi et al., 2018). Decision Transformer (Chen et al.) (021). Behavior Trans- former (Shafiullah et al. 2022), MaskDP (Liu et al. 2022 use ground truth key states differently), Decision Diffuser (Ajay al.
2022). We further consider the baselines with different key state labeling techniques (all these baselines are trained with the CoTPC framework; see Sec. B for more training details): (1) Last State only includes the very last state of the trajectory the key state. (2) AWE (Shi et al. 2023) (3) LLM+CLIP (Di Palo et al. 2023) first uses Large Language Model (LLM) to generate subgoals for a task, then uses CLIP target the image frame scoring highest regarding each subgoal, which will serve as the key states. (4) GT Key States uses ground truth key states, which is the oracle as in (Jia al. 2023). Here, labeling methods (1), (3), and (4) can be regarded as zero-shot, which are dependent on human semantics, while methods (2) and InfoCon are intrinsically linked the inherent properties of the data.
Performance of InfoCon. We report the success rate of all baselines in Tab. (additional results of the methods without Chain-of-Thoughee key state prediction, except Decision Transformer, can be found in Sec. Cincee their performance is much worse than Decision Transformer according to Dia al. 2023). Also, as reported in Jia et al. (2023). some of the methods above, which do not use Mhain-of-Thoughhe key state prediction strategies, cannot even achieve reasonably good results in seen environments during training (overfitting). Thus, we follow CoTPC to comprehensively com- pare different baselines and report results on the seen and unseen environment seeds. As observed in Tab. I. the policies trained with our discovered manipulation key states can achieve competi- tive performance compared with CoTPC policies using ground-truth key states and other key state grounding methods. Notably, the generalization of InfoCon is also evidenced by delivering top success rates across different tasks. We further provide human intuition similarity of key states grounded by LLM+CLIP, discovered by AWE and InfoCon in Tab. 2 We find that the HIS score seems have weak correlation with the policy performance presented in Tab. [ Therefore, we treat this as signal that the effectiveness of the manipulation concepts may not align well with what human semantics endorse, which justifies the need to develop discovery and grounding methods that can learn from unannotated trajectories.
Policies key states labeled by AWE M.I.M are trained by us. Other results from
7

---

Published as a conference paper at ICLR 2024
fome 4
Fame
Frmm
P&PP Cube
GT
GT
InfoCon
InfoCon
frane
Fame
frame
Turn Faucet
Pag Insertion
GT
GT
InfoCon
InfoCon
57
Trane
Tramn TEB
Tramne 233
Tome
froe
Frome 57
hrme 67
home
frme
frame 136
Figure 5: Key states discovered and grounded by InfoCon. From top-left to bottom-right are vi- sualizations of key states of tasks: P&P Cube, Stack Cube, Turn Faucet, and Peg Insertion. Each subfigure contains frames of ground-trun truth key states at the upper part and key states discovered by InfoCon below. We align frames of ground-truth key states with the nearest subsequent key states from InfoCon by checking their timesteps.
Visualization of key states. We visualize the discovered manipulation concepts and grounded key states with InfoCon Fig. [5] As shown, the identified key states consist of states similar to the ground-truth key states from Jia et al. (2023) (Fig.
well as more fine-graine grained ones. We provide additional visualizations in Sec. D also perform an experiment to assign discovered concepts with semantically meaningful descriptions for the Peg-Insere Insertoot task in Sec.
3.3 ABLATION STUDY
We perform an ablation study investigate the characteristics of both generative and discriminative goals. Keeping other hyper-paramene parameters constant, we separately omit the loss terms associated with the discriminative goal (Eq. 7 and Eq. {} and the loss terms related the generative goal (Eq, [T]. Our evaluation considers two primary metrics: the success rate of CoTPPC policies on the four ma- nipulation tasks and the Human Intuition Similarity (HIS, Eq. TT) on those tasks.
Table 3: Ablation study of the policy performance with only generative loss discriminative losses.
Here, "All" means the InfoCon trained with both the generative and discriminative goal losses.
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
SR(% %
seen
unseen
unseen
S.&sf.
u.&sf.
u.&uf.
unseen
All Generative Goal Only
96.6
81.0
63.0
47.0
53.8
52.0
17.8
63.6
17.8
69.8
61.0
46.8
25.0
34.8
31.0
19.8
50.2
5.5
Discriminative Goal Only
72.0
62.0
57.4
23.0
39.2
43.0
17.8
64.2
10.5
The success rates of different variants are reported in Tab. When only employing the generative loss or the discriminative losses, policies tend to worse than those trained with states dis- covered by both types of losses While some models might perform well in seen environments, they often fail to deliver comparable results unseen scenarios. This reaffirms the pivotal role of proper key state identification and underscores the indispensable synergy between the generative and discriminative informativeness losses.
Table Ablation study with the human intuition similarity for either the generative or discriminative losses. Here, "All" means the InfoCon trained with both generative and discriminative losses.
HIS
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
All
2.60
8.98
18.03
9.05
Generative Goal Only
11.15
23.11
38.79
37.11
Discriminative Goal Only
3.40
6.06
28.55
11.05
context, we policies
ablated configurations
8

---

Published as a conference paper at ICLR 2024
The ablations with HIS reported in Tab. 4 When employing only the generative loss, the iden- tified key states exhibit reduced similarity to human intuition. Interestingly, the effect on intuition similarity appears less prominent when relying solely on the discriminative goal loss. We postulate that this might be attributed to the noise in the manipulation concept assignment during the initial training phase. Particularly, at the beginning of training. the groundings of key states are not always accurate. The key states may not faithfully represent goal states, rendering them suboptimal as pre- diction targets in the generative goal loss (Eq. 6 Practically, during the initial training phase, we can temporally turn off the optimization of the generative goal loss (Eq. 66 and let the discriminative goal loss rum for a few epochs. More details on this part can be found in Sec. B.2
4 RELATED WORK
Behavior Cloning in Manipulation Tasks. Training robots through cemonntrattinn-divven behavior cloning continues to be a key research area (Argall et al. 2009). Within this realm, various tech-
niques have evolved to harness the potential of imitation learning Zhang al. 2018 Rahmatizadch
et al. 2018 Fang et al., 2019). Approaches range from employing clear-cut strategies (Zeng et al.)
(022)) and more subtle methods to adopting models based on diffusion principles (Chi et al.) 2023 Pearce et al. (023). Some methodologies manage to sidestep the necessity of task-specific labels throughout training and discover robot skills (Shankar & Guptal 2023). Different from prior research, work prioritizes the autonomous extraction and understanding of manipulation concepts from unlabeled demonstrations, taking a step beyond mere replication toward genuine conceptual understanding.
2021 Shridhar et |022 Qin et al.
2020 Tanneberg et al. 202 Xu et al.
Hierarchical Planning and Manipulation Concept Discovery. Hierarchical planning is crucial in interaction and manipulation scenarios, as it underscores the need for blending broad strategies with specific, detailed actions to achieve optimal results (Hutsebaut-
et al. 2022 Yang et al. 2022] Xu et al. 2018 Jia et al. 2023). Recent years, the Chain of Thought (CoT) prompting methodology.
as depicted in Wei et al. 2022 Cheng et al.] 2022 Yao et al. 2022). further emphasizes the value of breaking down intricate tasks into a succession of more straightforward sub-tasks. By doing so, the complexity of the policies is significantly reduced, paving the way for simpler learning processes and resulting more reliable policies. However, most hierarchical planning methods lean heavily on human semantic intuition. This can manifest as direct human input or systems mimicking hu- man reasoning like LLMS Dii Palo et al., 2023). Some studies (Zambelli et al., 2021; von Hartz eet al. 2022, Sermanet et al. 2018 Morgan al. 2021 Yan et al.) 2020 Weng et al. (023) employ self-supervised discovery of manipulation concepts to circumvent such manual semantic interven-
tions, leveraging mutual information (Hausman et al. 2018 Gregor et al. 2016 significant points
of time (Neitz et al.] 2018: Jayaraman et al. 2018 Pertsch et al. 2020 Zhu et al. 2022 Caldarelli
let al. 2022), and geometry (Shi al. 2023 Morgan et al. [022]] Zhu et al. 2022 physics (Yan et al. 2020) constraints. Recent method AWE (Shi et al.) 2023) has been proposed to mitigate the
compounding error problem inherent behavioral cloning and automatically determine waypoints based on trajectory segments that can be approximated by linear motion. However, the interpretabil- ity of discovered manipulation skills in these methods remains inadequate. Our proposed InfoCon broadens the scope beyond mere planning or trajectory partitioning, with deeper understanding and abstraction of underlying manipulation concepts. InfoCon ensures not just task completion but also a richer conceptual grasp of the task itself.
CONCLUSION
We propose InfoCon, a self-supervised learning method capable of discovering task-specific manip- ulation concepts based on generative and discriminative informativeness. The concepts discovered by InfoCon share similarities with human intuition and are able to be utilized for training manipu- lation policies with comparable performance to oracles. Our work provides an idea of constructing embodied agents discovering concepts themselves other than struggling with the grounding of con- cepts that are manually specified. We hope is received as an attempt to develop an automatic mechanism for discovering useful abstract concepts. In the future, will consider exploring the possibility of methods for discovering relationships between manipulation concepts and forming structures of discovered concepts.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work is supported by the HKU-100 Award, donation from the Musketeers Foundation, the Microsoft Accelerate Foundation Models Research Program, and in part by the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust. We also like to thank Qihang Fang for helping with the human motion concept discovery experiments.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.1500 2022.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Roboties and autonomous systems, 57(5):469 483, 2009.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318. PMLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877-1901, 2020.
Edoardo Caldarelli, Philippe Wenk, Stefan Bauer, and Andreas Krause. Adaptive gaussian pro- change point detection. In International Conference on Machine Learning, pp. 2542-2571. PMLR, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-15097, 2021.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding language models in symbolic languages. arXiv preprint 2XXi:2210.02875, 2022.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Ha David, Dai Andrew, and VL Quoc. Hypernetworks. arXiv preprint arXiv, 1609, 2016.
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller. Towards unified agent with foundation models. In Workshop Rein- carnating Reinforcement Learning at ICLR 2023, 2023.
Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362-369, 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07540 2016.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskil12: A unified benchmark for generalizable manip- ulation skills. arXiv preprint arXiv:2302.044 04659, 2023.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang. Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Leam- ing Representations, 2018
10

---

Published as a conference paper at ICLR 2024
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization, arXiv preprint arXiv:1806 180.06600000 2018.
Karan Grewal, Phil Bachman, Adam
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303. 00855, 2023.
Matthias Bttsebbutt-uysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1): 172-221, 2022.
Dinesh Jayaraman, Frederik Ebert, Alexei A Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. arXiv preprint 1rXiv:1808.070 2018.
Zhiwei Jia, Fangchen Liu, Vineet Thumuluri, Linghao Chen, Zhiao Huang, and Hao Su. Chain-of- thought predictive control. arXiv preprint (rXXi:2300400776, 2023.
Miguel Greail,,,,,,lla, Dianhuan Lin, J Swaroop Guntupalli, and Dileep George. Beyond imi- tation: Zero-shot task transfer robots by learning concepts cognitive programs. Science Robotics, 4(26))eaav3150, 2019.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Mopro: Wehly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.0799 2020.
Klaus Libertus and Dominic A Violi. Sit to talk: Relation between motor skills and language development in infancy. Frontiers in psychology, 7:475, 2016.
Fangehen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. Advances in Neural Information Processing Systems, 55:12608- 12618, 2022.
Andrew S. Morgan, Walter G. Bircher, and Aaron M. Dollar. Towards generalized manipulation learning through grasp mechanies-basee features and self-supervision. IEEE Transactions on Robotics,
2021. doi: 10.
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhi- wei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107. /4483, 2021.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. Advances in Neural Information Processing Systems, 31, 2018
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, al. Imitating human behaviour with diffusion models. arXiv preprint arXiv::301. 10677, 2027,
Karl Pertsch, Oleh Rybkin. Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Dani- ilidis, Joseph Lim, and Andrew Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. In Learning for Dynamics and Control, pp. 969-979, PMLR, 2020.
Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 1083:0083- 2022.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3758-3765 IEEE, 2018.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1805. 1063, 2018.

---

Published as a conference paper at ICLR 2024
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-114.. IEEE, 2018.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing sys- tems, 35:22955-22268, 2022.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference Machine Learning, pp. 8624-8633, PMLR, 2020.
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concep2robot: Learn- ing manipulation concepts from instructions and human demonstrations. The International Jour- nal of Robotics Research, 40(12-14):1499 1434, 2021.
Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation. arXiv preprint 2.Xiv:2377.13326, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2023.
Vladimir M Sloutsky. From perceptual categories to concepts: What develops? Cognitive science, 34(7):1244 1286, 2010.
Daniel Tanneberg, Kai Ploeger, Elmar Rueckert, and Jan Peters. Skid raw: Skill discovery from raw trajectories. IEEE robotics and automation letters, 6(3):4696-4703 2021.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation, 2018.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jan Ole von Hartz, Eugenio Chisari, Tim Welschehold, and Abhinav Valada. Self-supervised learn- ing of multi-object keypoints for robotic manipulation. arXiv preprint arXiv:2205.08311 2022.
Eric A Walle and Joseph J Campos. Infant language development is related to the acquisition of walking. Developmental psychology, 50(2):336, 2014.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Yijia Weng, Kaichun Mo, Ruoxi Shi, Yanchao Yang, and Leonidas Guibas. Towards learning geo- metric eigen-lengths crucial for fitting tasks. Proceedings of the 40th International Conference on Machine Learning, pp.
2023.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3795-3802. IEEE, 2018.
Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. arXiv preprint arXiv:2307.09955. 2023.
Mengyuan Yan, Yilin Zhu, Ning Jin, and Jeannette Bohg. Self-supervised learning of state esti- mation for manipulating deformable linear objects. IEEE robotics and automation letters, 5(2): 2372-2379, 2020.
12

---

Published as a conference paper at ICLR 2024
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought im- itation with procedure cloning. Advances Neural Information Processing Systems, 35:36366- 36381, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprini urXiv:2210.03629 2022.
Martina Zambelli, Yusuf Aytar, Francesco Visin, Yuxiang Zhou, and Raia Hadsell. Learning rich touch representations through cross-modal self-supervision. In Jens Kober, Fabio Ramos, and Claire Tomlin (eds.), Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 1425-1425. PMLR, 16-18 Nov 2021. URL htttps://proceedingg mLl..peesa/v.5//2ammes http://sscccceeelaa.l.a..aaaa//s
html
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian,
Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Re- arranging the visual world for robotic manipulation. In Conference on Robot Learning, pp. 726- 747. PMLR, 2021.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5628-5635. IEEE, 2018.
Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstra- tions for long-horizon robot manipulation. IEEE Robolics and Automation Letters, 7(2):4126 4133, 2022.
13

---

Published as a conference paper at ICLR 2024
A MORE DETAILS OF ARCHITECTURE
A. TIME-STEP EMBEDDING.
For the feature 2 extracted (Eq.) from state at time-step t, we always normalize to proceed the latter concept assignment (Eq. In Eucild Space, is on high dimensional unit ball. In order to maintain this "spherical" characteristic and embed z using time-steps, we design the time-step embedding method below:
(21/T 1)x - (sin
(21/T 1)sT
cos
2+ 2A
2 2A
(12)
ISIST
where and i' the total number of time-steps.
We give an intuition explanation at Fig. 6 This embedding operation will influence more with time step close to the beginning and the end of the manipulation trajectory. making it focus more on the absolute time-step (original weight in will be relatively small), while for z with time-steps in the middle, they will receive weaker influence from this embedding operation. Reflecting on our intuition, when learning manipulation task, the information of absolute passing time is informative at the beginming and the end, but in the middle we will focus more on relative time-step order instead of absolute current time.
In Eq. 12 A is a positive hyper parameter controlling the usage of area of the whole spherical surface. When A 0, the whole spherical surface will be used for time step embedding, but when the unified time-step + of is 0 the embedded vector will always be (-1, 0) or (1, 0), which will annihilate the feature of original vector. So we use coefficient A to avoid this. The inclusion of A will still maintain the characteristic of focusing more absolute time the beginning or the end.
(m222)
dim0
dim1
Figure An example of our time-step embedding method. Here we use an example of dimension (x. v- When is at time-step I, the embedded feature vector is z' (sin (2)/T-1)s I Cos 2+2A
(2 2/ COS (2//T-1)st
COS
2+2A
2+24
This embedding method can also restrict the upper bound of cosine similarity between two feature vector when time step different:
([sin 4,, 21 cos t1], [sin 12, #2 cos tos)
sin 4 sin 42 + (#1, 22) cos 4 cos la
(13)
< sin sin + cos cos la
= cos(t) - f2)
14

---

Published as a conference paper at ICLR 2024
Here 12 E 1-3. 2. when using the embedding method in Eq. 12 (:.)) is cos similarity (since the vectors here are all unit vectors, it is same as inner product). Larger difference between t and 22 leads to smaller value of cos(t; t2), which means smaller cosine similarity.
A.2 WIIY VQ-VAE?
When discovering key states, we need to give every state in the trajectory a label and decide the key states based the labels. VQ-VAE naturally provides a process of assigning symbols for inputs, which is suitable for partition and segmentation. The remaining task involves assigning meanings to the vectors the codebook using self-supervised learning, which is achieved through the proposed generative and discriminative goal losses.
A.3 DETAILS OF CONCEPT
The concepts {ax K criminative goal:
k=1 detailed model as two vectors, one for generative goal and one for dis-
{(ar, pk)
Og will be used for concept assignment (Eq. 5 and prediction of states that achieve the goal in
generative goal loss (). and pę: is the compressed parameters for compatibility function (Eq 2 Pkc is able be transformed into simple MLP network using hyper-network (David et al. 2016
detailed design in Sec. A.7). See Sec. Sec. A.5 also for more details of usage and update of the parameters in concepts during training.
A.4 UPDATE OF PROTOTYPES
We use EMA moving (Li et al. 2020) to update the prototype of features representing concepts. when training and or assigned with set of extracted features (Eq. 4). We will update og using the average of the extracted features:
Normalize(
1
Σ
H : n12)= = k
n(*)=k
(14)
<- Normalize (Cema O# + (1 Ceraa )=
Here < Cemo < Notice that we did not update Og based on each single feature 2, since we find is inefficient when training. Our experiments use Cema 0.9.
A.5 PRESERVE GRADIENT FLOW
When assigning concepts (Eq. 5). the gradient cannot naturally propagate back to the encoder (Eq. 4.. We use technique similar VQ-VAE to achieve this. If we have a of concept fea- tures {ox Ik=1 and the extracted feature from current state is We calculate the probability of choosing certain concept using cosine similarity:
p(z a2) exp((2, a4)/T)
(15)
k== exp( (z, ax)/T)²
So we can use the soft version to preserve gradient:
K
a soft
p(z an) SG(av) antz) =a soft +SG(an(2) =0fft)
k=1
(16)
p coft
Σ
p(z ax) SG(pk) Pn(x) pofft + +SG(pn(z) soff)
Where 77x)) ) is the same concept assignment function as Eq. SG(.) is the stop gradient operation, and the definition of og and pı is same concept vectors in Sec. A.4 We only hope the gradient to adjust the selection of concept, so we stop the gradient of prototype ag and compressed parameters Pk- ag will be updated using the gradient from generative goal loss (Eq. 6), EMA (Sec. A.4). and reconstruction regularization at next Sec. A.6 Pk will be updated using the gradient from discrimi- native goal loss (Eq. 7 and Eq. 8).
15

---

Published as a conference paper at ICLR 2024
A.6 RECONSTRUCTION REGULARIZATION
To prevent over smoothing of discovered concepts (always one concept) similar to VQ-VAE, we add in a reconstruction process from extracted features original states: $ x (ail
(Here of is the assigned concept from {a to state si in т = {(s) r= from data). Since the maximum number of concepts is an hypee-paramette and we would not choose very large number, we do not hope the reconstruction process be trained well enough. The training loss for reconstruction:
N
1 N
(6
(17)
1=1
We "pretrain" InfoCon with only the reconstruction loss above and the classification entropy loss (Eq. 9) before using generative and discriminative goal loss:
Cre C
AL (a, Φ)
(18)
During experiments, find that the above loss can be used warm up the concept discovery VQ-VAE structure via pertaining to provide a good initialization. The initialization can then help achieve better convergence when the proposed generative and discriminative losses are employed for self-supervised concept discovery. To show the effectiveness of the initialization above, we provide counts of activated manipulation concepts (within the codebook the VQ- VAEV with and without the usage of this initialization when discovering key concepts. Our observation is that the above initialization can help discover more fined umed-grained manipulation concepts, as shown by the number of activated concepts. Detailed results are in Tab. below.
Table The counts of activated manipulation concepts (within the VQ-VAB codebook) of InfoCon when it is trained with (w) and without (w/o) the usage of the initialization in Eq. 18
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
rec.
4.5+0.5
7.0+2.0
5,0+2.0 2.5+1.5
7.5
w/o rec.
1.5+0.5
2.0+1.0
1.5+0.5
Notice that there are also other methods that alleviate over-smoothing in VQ-VAE (Roy al., 2018), but these methods focus on the geometry clustering characteristic of VQ- VAE, which may conflict with our design of generative goal and diseriminative goal. On the other hand, regularization with construction is empirically reasonable according to self-supervised learning.
A.7 HYPER-NEE NETWORK
We provide details of our hyper-network for compatibility function (Eq, at Fig.
B TRAINING DETAILS
B. PSEDUO CODE
See |
for the training scheme of our method.
B.2 HYPER PARAMETERS
InfoCon. We use the structure of Transformer used in (Jia al. 2023). which refers the design of Brown et al. (020). The state encoder (Eq. 4 and state re-constructor in A.6 both use 4-layer causal Transformer. The goal based policy in Eq. S Juse a I-layer causal Transformer. The predictor for generative goal in Eq. @use 2-layer causal Transformer. For hyper-network in A.7 we also use only one hidden layer in the generated goal function. The number concepts is fixed, maximum number of 10 manipulation concepts for all the tasks. The temperature in Eq. 5 and Fig. 7 is 0.1. A in Eq. 12 0.2. All the size hidden features output by Transformers and concept Teatures (ox) is 128. When training, the coefficient X, rec in Eq. 18 Eq. 10 is 0.001, 0.1, and will
16

---

Published as a conference paper at ICLR 2024
split
P
Pj
RINE
HNG"
HN
W
Reshape
Reshape
Reshape
Temperature
State
Compatiblitt
state
Wix
Wix
jaj
Cos Similerity
Sigmoid
Figure 7: Structure of our hyper-network for compatibility function (Eq. 2 The hyper-network generate hidden layers in the compatibility function, with same shape of weight and bias. We use a two layer hyper-network structure. First set of layers (HN) transform different segments of compressed parameter vectors into medium weight vectors. The weight vectors then pass through the same layer N#) and become concatenated flattened weight matrix (W:'iat) and bias (b,) for different hidden layers in the compatibility function. The compatibility score will be represent sigmoid-actoo activated value of the cosine similarity between the final hidden feature and the last segment of the compressed parameter vector (we will weight them by the temperature т, which is equal in Eq. 5)
defer optimization of geen until half of iteration for training is done. We pretrain InfoCon according to [g for 1 x 104 iteration with base learning rate 1.0 x 10 10-4 Then we train the InfoCon for each of the task with 1.6 x 10t iterations based on the pretrained model with base learning rate 1.0 10-4 After labeling the original data with key states using trained InfoCon models, we train our CoTPC 5.0 x 10-4 For the three training stages, we all use AdamW optimizer and warm-up cosine annealing scheduler which linearly increases learning rate from 0.1 of base learning rate to the base learning rate for 1000 iteration, and then decreases learning rate from base learning rate to 0.1 of base learning rate. The weight decay is always 1.0 10-3, and batch size 256. For practice, we would only use a segment of 60 states (along with actions) for every item (trajectory) in the batch.
policies for 1.8 10th iterations with base learning
Baselines. Here we only give some essential implementation details of the two baseline methods: CoTPC with LLM sub-goal and CLIP detection of key states, CoTPC with key states discovered by AWE Shi et al. (2023). Notice that after the discover of key states, the training of CoTPC policies same as description in B.2
CoTPC (LLM+CLIP). We discover that the CLIP scores of different key states in a ma- nipulation trajectory are close to each other. Thus, is hard for us set reasonable threshold to decide whether some of the states have already achieved the sub-goal. for each text description the key states, we will use the average of the minimum and maximum value of CLIP scores to decide the threshold, and select the state with minimum temporal step and larger than this threshold to be the key states of this sub-goal. (Notice that is not reasonable to select the maximum score, since the states after achieving the goal still have the feature in the description of key states. Like "grasp the cube", after this sub-goal is achieved, most of the states after it are also suitable for this description in tasks like P&P Cube and Stack Cube.).
CoTPC (AWE). In AWE research, they set thresholds for end condition of the dynamic programming of finding way-points with different manipulation tasks. Here we modify the method that it can discover a fix number (here we choose 10 to align with InfoCon) key states for all trajectories, which is more suitable since current implementation of CoTPC needs to fix the number of key states.
17

---

Published as a conference paper at ICLR 2024
Algorithm
InfoCon
Input:
demo trajectories D, {T. (s) state encoder Φ. achieved state predictor 08
a=)==1 -==1
T(
Modules:
compatilbility function hyper-networe C, discriminative goal policy concepts {ax ]==1
Output:
trained state encoder +
for i 1..,..,,NN N.1 1.
(1i) do 2) (()) of 11))))))))
for k .,,,..., K do
(G(z)))
p(1 aa)) Ek=rexp( SG((())
end for
0f = aarg max; pl p(+{+ou)
end for
for i 1,2, ...,, K do
Σg = Normalized lafta:=orall Eat= =ai z()
0% Normalizo Cema Oc + Cema) 12k)
end for for i
11,, N, 1,2, T(1) do
for k = 1,2,.... K do
expl(z) (G(x)))))))
p(z ax) Σ (cp((2),S((aa)/7
b> Recalculate with updated og.
end for $/
= arg min, (u2t,
Last state of sub-trajectory as key state
end for
Calculate Leen (a,, Φ) 08), Lelia (ax, Φz C), Ldie C; n), Lent (a, Φ).
Cree
Eq.
Eq.
Eq.
Eq
= _ + Ldis + X( + Lont) + X gree
Back propagation from
C MORE EXPERIMENT RESULTS
C.I OVER-FITTING ISSUE
In Table I noticeable discrepancy is observed in the performance of policies between seen and unseen environments, indicative an over-fitting issue. Employing the CoTPC with key states of InfoCon, conducted experiments to evaluate the influence of scaling the training dataset over- fitting. The results of these experiments are presented in Tab. [6] From these results, it evident that the impact of augmenting the training data mitigating over-fitting varies across different tasks.
Table Success rate (%) when varying scale of training set. The policy CoTPC with key states by InfoCon. For the task Turn-Faucet, we provide results on three kinds of situations: seen environ- ments (s. &sf.), unseen environments with seen faucet types (u.&sf.), and unseen environments with unseen faucet types (u.&uf.).
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
SR( %)
unseen
seen
unseen
S.&sf.
u.&sff.
u.&
seen
unseen
200 traj.
98.5
31.0
92.0
9.0
57.0
40.0
14.0
66.0
8.5
500 traj.
96.6
81.0
63.0
47.0
53.8
52.0
17.8
63.6
17.8
800 traj.
94.4
75.0
90.6
43.0
59.2
53.0
25.3
64.3
27.0
18

---

Published as a conference paper at ICLR 2024
C.2 MORE BASELINE METHODS
We provide the results of baseline methods without the usage of concepts semantic information in seen environment Tab. 7 The data borrowed from (Jia et al. 2023). According this source, Decision Transformer exhibits the best performance in unseen environments among all methods evaluated. Consequently, the performance other methods in unseen environments not provided, except for that of Decision Transformer.
Table 7: More results success rate (%) of different methods on seen environments.
SR(
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
Vanilla BC
3.8
0.0
15.6
0.0
Behavior Transformer
23.6
1.6
16.0
0.8
Decision Diffuser
11.8
0.6
53.6
0.6
MaskDP
54.7
7.8
28.8
Decision Transformer
65.4
13.0
39.4
5.6
Last State
70.8
12.0
47.8
9.6
AWE
96.2
45,4
53.4
31.6
LLM+CLIP
92.2
44.6
35.8
63.2
GT Key States
75.2
58.8
56.4
52.8
InfoCon
96.6
63.0
53.8
63.6
C.3 CONCEPTS AND CONCEPT-DRIVES POLICIES
While experiments highlight the strength of InfoCon in identifying key states, integrating this with policy generation is an exciting and unexplored area. We see a valuable opportunity for future work in developing method that not only discovers key states using InfoCon but also generates policy similar to CoTPC.
Considering the selection of policy, there not many methods that focus making use of key states in their decision makion policies with a tailored design. Here are some methods that are relevant our knowledge:
MaskDP (Liu al. 2022). This method has weakness: appending the end state into the input sequence is needed. The agent must know the exact key states based on the initial state, which makes constrained to achieving very specific goal and is not applicable to our problem setting, where the end goal state is not provided.
Modified Decision Transformer. This is method found in the work of AWE Shi et al., 2023), in which they adapt the original decision transformer Canso et al. 2021 to let it leverage key prediction during policy learning.
CoTPC (Ja et al. 2023), which is the method we mainly employ in our work.
We trained the Modified Decision Transformer and saw that policies based on InfoCon also have better performance compared with the Decision Transformer without key states using ground- truth (GT) key states. The results are in the table below.
Table 8: Success rate (%) of methods based Decision Transformer (DT for short). For the task Turn- Fau-eF we provide results on three kinds of situations: seen environments (s.&sf.), unseen environments with seen faucet types (u.&sf.), and unseen environments with unseen faucet types (u.&uf.).
P&P Cube
Stack Cube
Turn Faucet
Peg Insertion
SR( %)
seen
unseen
seen
unseen
S&&sf.
&.&.sf.
u.&uf.
seen
unseen
DT
65.4
50.0
13.0
7.0
39.4
32.0
9.0
5.6
2.0
DT GT
89.8
68.0
58.4
29.0
41.2
36.0
16.0
34.6
6.3
DT & InfoCon
95.2
70.0
62.6
32.0
48.0
48.0
17.0
52.0
7.8
19

---

Published as a conference paper at ICLR 2024
D
MORE VISUALIZATION RESULTS
We provide more visualization results of key states labeled out by InfoCon at Fig 10
E
ALIGNMENT WITH HUMAN SEMANTICS
variety of methods have been devised to identify elements similar to key states (Neitz et al.
2018 Jayaraman et al. 2018 Pertsch et al. 2020: Zhu et al. 2022 Caldarelli et al. 2022). Our
approach surpasses them by ensuring that the identified key states stem from well-defined concepts. Specifically, in our framework, these concepts pertain to the modeling of goal (Eq.
[ Eq. Eq. 3).
Furthermore, we demonstrate that the concepts encapsulated within the key states of InfoCon share similarity with intricate human semantics, substantiated by the evaluation of the Human Intuition Score (HIS, Eq. TTI.
In this section, we present further attempts and results to align the self-discovered manipulation concepts by InfoCon with human semantics for potential enhancement the explainability when facing human-robot interaction. We perform the analysis with the robotic task of "Peg- Ingertttt The major goal is to check whether we can assign reasonable and linguistically meaningful names or descriptions the discovered concepts key states.
Before we perform some qualitative analysis, we first determine the manipulation concepts discov- ered for this task by checking the activated concepts (codes) among all the codes in the codebook (in total 10) of the VQ- VA-V employed. Namely, all the trajectories of the task "Peg-Inser Inng-rnneriion are pushed through the VQ-VAE, the activated codes will be recorded for each of them, and then the activation rate for each code (concept) is computed.
Tab. provides the activation rate of each manipulation concept related to task Peg Insertion using the concept discovery VQ-VAH trained by InfoCon.
Table 9; Activation rate of each manipulation concept on task Peg Insertion from InfoCon.
Concept Index
0
3
5
6
7
8
Activation Rate (%) 46.7
65.0
100.0
1.9
91.1
100.0
95.6
82.3
100.0
3.9
The table above indicates that nearly every trajectory performing task Peg Insertion has the same set of key states activated, which means that the key states (concepts) involved in accomplishing Peg Insertion are the discovered concepts with numbering: #0, #1 #2, #4, #5, #6, #7, and #8, in total 8 vectors in the code book of VQ-VAE InfoCon. (#3 and #9 are probably noise, given their low activation rate).
Next, we use a sequence illustrate the description we assign to each of the discovered concepts, which help align with human semantics. The concept names are the following:
The gripper is positioned above the peg (discovered concept #7).
The gripper is aligned with the peg and ready to grasp (discovered concept #5).
The peg is grasped (discovered concept #0).
4. The peg is grasped and lifted (discovered concept #1).
5. The peg is aligned with the hole distantly (discovered concept #4).
The peg is aligned with the hole closely (discovered concept #6).
The peg is inserted half-way into the hole (discovered concept #8).
The peg is fully inserted (discovered concept #2).
Please check Fig. for visuals corresponding these discovered concepts. The above verifies that humans can still assign descriptions to the discovered concepts by InfoCon. At the same time, we see the potential of InfoCon to discover meaningful concepts that can be aligned with human
20

---

Published as a conference paper at ICLR 2024
semantics. Moreover, shows that InfoCon has the capabiliit of discovering fine- fine-grained concepts or key states human annotators (in CoTPC) have ignored or are unaware of. These further demonstrate the effectiveness of the proposed InfoCon for automatically discovering meaningful manipulation concepts.
The gripper positioned above the peg (discovered concept #7)
The gripper is aligned with the peg and ready to grasp (discovered concept #5)
The peg is grasped (discovered concept #0)
The peg is grasped and lifted (discovered concept #1)
The peg is aligned with the hole distantly (discovered concept #4)
The peg is aligned with the hole closely (discovered concept #6)
The peg is inserted half-way into the hole (discovered concept #8)
The peg is fully inserted (discovered concept #2)
Figure 8: Align InfoCon Concepts with Human Concepts on the Peg Insertion task.
F INFOCON WITH REAL-WORLD DATA
InfoCon can be applied in various tasks involving sequential events (from manipulation, to sound, video, natural language, log of computer systems, cache behavior of memory, and changing of climate and weather). These processes can all be regarded as trajectory with meaningful partitions in achieving specific goals. Here we provide visualization of the results of applying InfoCon on human pose sequences estimated from real-world videos. We can that it also separates video of pose motion into different key states, which can be mapped to different "human motion concepts".
GT segments:
Walk
Transition
Sit
Transition
Walk
Figure 9: InfoCon for human motion concept discovery from pose sequences estimated with real- world videos.
21

---

Published as a conference paper at ICLR 2024
fame 16
Fame
InloCon
InfoCon
frame
frame B5
frame 33
frame 45
InfoCon
InfoC.on
Trame
frame 33
frame 42
frame
frame 83
Frame 35
frame 45
frame
frame 91
frame 132
Gr
Infocon
InfoCon
frame
frame
frame
frame
frame 117
frame 32
frame
frame
frame
frame
frame 29
frame 42
InfoCon
InfoCCon
Trame 30
frame 39
frame 50
frame
frame 107
frame 26
frame 42
frame 43
frame 64
frame 92
GT
infoCon
InfoCoo
frame
frame 36
frame
frame
frame 108
frame
frame frame 39
frame
frame
frame
fame 36
InloCon
InfoCon
Trame
frame:36
37
frame
frame
frame 26
39
frame
frame 93
frame
InloCon
InfoCon
Trame
frame 38
frame 64
frame
frame 138
frame 22
frame 35
frame
frame 54
frame 78
Figure 10: More examples of labeled key states of task P&P Cube (partl).
22

---

Published as a conference paper at ICLR 2024
frame 61
Frame 127
GT
InfoCon
frame 17
frame 51
frame 63
frame 76
frame 81
frame 132
frame 136
frame 64
frame 129
GT
InfoCon
frame 17
Frame 55
frame 65
frame 78
frame
frame 129
frame 138
frame 72
frame 156
GT
InfoCon
Frame 17
Frame 48
frame 75
frame 86
frame 90
frame 160
frame 165
frame
frame 132
GT
InfoCon
frame 22
frame 51
frame 64
frame 77
frame 83
frame 132
frame 141
frame 85
frame 185
GT
infoCon
frame 22
frame 55
frame 85
frame
frame 104
frame 186
frame
Figure 11: More examples of labeled out key states of task Stack Cube (part1).
23

---

Published as a conference paper at ICLR 2024
Frame 56
Frame 132
GT
InfoCon
frame 17
frame 56
frame 60
frame 77
frame 78
frame 138
frame 141
frame 77
frame 163
GT
InfoCon
frame 17
Frame 51
frame 78
frame 92
frame
frame 165
frame 172
Frame 57
frame 125
GT
InfoCon
Frame 17
Frame 57
frame 60
frame 72
frame 77
frame 126
frame 134
frame
frame 125
GT
InfoCon
frame 13
frame 45
frame 60
frame 72
frame 77
frame 132
frame 134
frame 57
frame 123
GT
infoCon
frame 13
frame 57
frame 60
frame
frame
frame 123
frame 132
Figure 12: More examples of labeled out key states of task Stack Cube (part2).
24

---

Published as a conference paper at ICLR 2024
Tomn 112
Fome
InloCon
In InloCon
frame 119
frame
InfoCon
InfoCon
Frame 58
Frame 126
Frame 191
Frame 240
Frame
Frame 82
Frame 83
frame 151
Frame 174
InfoLon
InfoLon
Frame 119
frame
frame
Frame
Frame
frame
Frame 99
Frame
Frame 109
frame 108
InfoCon
InfoCon
Frame 55
frame 120
Frame 195
frame 225
Frame 9
frame
frame 85
frame 113
Frame 150
InfoCon
InfoLon
Frame
frame
frame
frame
Frame
frame
frame
Frame
fame 53
frame 105
InloCon
InfoCon
Trame
frame
frame
frame
Frame
frame
frame
frame
Frame
InfoCon
InfoCon
frame 2
Trame
frame 106
frame 139
frame
frame 100
frame 132
frame
frame 190
Figure 13: More examples of labeled out key states of task Turn Faucet.
25

---

Published as a aonferenn paper apperaa at (CLK 2024
frame 72
Frame 128
GT
InfoCon
frame 0
frame 35
frame 72
frame 90
frame 121
frame 132
frame 171
frame 45
Trame
GT
infoCon
Trames 5
frame 24
frame 45
frame 58
frame frame 103
frame 134
framen 53
frame 100
TT
InfoCon
frame 26
frame 28
frame 53
frame 54
frame 93
frame 111
frame 141
frame 77
frame 149
Infoonn
frame e
frame 40
frame 78
frame 84
frame 133
frame 152
frame 197
frame 64
frame 114
GT
infoCon
Frame
frame 33
frame 64
frame 79
frame 106
frame 122
frame Hame-11 161
Figure 14: More Commmper of labeled out key states of task Peg Insertion (parti)
26

---

Published as a conference paper at ICLR 2024
frame 52
Frame 102
GT
InfoCon
frame 3
frame 26
frame 52
frame 54
frame 101
frame 110
frame 145
frame 75
frame 121
GT
InfoCon
frame
Frame 34
frame 75
frame 84
frame 119
frame 131
frame 167
frame 77
frame 141
GT
InfoCon
Frame
frame 39
frame 77
frame 88
frame 135
frame 144
frame 187
frame
frame 105
GT
InfoCon
frame
frame 30
frame 66
frame 67
frame 112
frame 116
frame 147
frame 57
frame 111
GT
infoCon
frame
frame 27
frame 57
frame
frame 108
frame 115
frame 152
Figure 15: More examples of labeled out key states of task Peg Insertion (part2).
27


Reviews:
[{"review_id": "oyX0i8YW0a", "reviewer": "Reviewer_1PDE", "summary": "This manuscript proposed InfoCon, a framework that can discover concepts in manipulation tasks automatically based on information reuseness. Specifically, the authors designed an analysis system that outputting discretized concepts from an offline manipulation dataset using a few different losses: generative goal loss, discriminative goal loss. Also, the side product from the architecture is the derivative of the discriminative loss, which represents the action to perform in order to complete the task.", "questions": "1. I am a little bit confused about the motivation of discriminative goal loss. How is it trained and why it is useful for extracting information from the input states?\n2. I am not fully confident about the fairness of the comparison. It seems to me that methods including LLM+CLIP are zero-shot learning process that does not require the training data to be seen. Am I correct or it's actually not the case? If so, I would suggest to separating them in the comparison. However, I still think the proposed method has technical contributions that is worth to present.\n3. Is there any real-world example to indicate the effectiveness?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The manuscript is nicely written. I can follow most of the parts.\n2. The evaluation covers a medium size of tasks and dataset, which is not perfect (not large-scale) but I believe is enough for showcasing in robot learning area.\n3. The results look impressive that beat previous method marginally, achieving either the first or second best result in the whole table.", "weaknesses": "See question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "il4ah2IFyK", "reviewer": "Reviewer_mBdA", "summary": "The authors propose using vector quantization to discretize robot manipulation trajectories into sets of discrete sub-trajectory encodings that maximize proposed metrics on discriminative and generative informativeness.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and the motivations and technical details are clear.  There are detailed evaluations and the proposed method is compared to multiple SOTA approaches.  The authors also include an ablation study and highlight a comparison of of human interpretability in addition to policy performance.", "weaknesses": "How do the authors feel about the interpretability of manipulation concepts for human robot interaction?  Learned concepts may not be as understandable in an interaction.  I am curious about an opinion on being able to map the learned concept back to a semantic concept or how that can be integrated into the objective.  I know human intuition is covered in Table 2, and the authors state there is a weak correlation with policy performance, but there may be cases where the goal is to optimize for both.\n\nIn section 2, \"partition each trajectory into semantically meaningful segments\" should this be clarified on what \"semantically meaningful\" means?  This goes with the earlier stated motivation of moving away from human semantic discretization into self-discovered discretization that optimizes discriminative and generative informativeness.\n\nFor eq 1, how does the observed state sequence factor into the generative goal?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q06iAHfduQ", "reviewer": "Reviewer_WSFG", "summary": "This work, InfoCon, uses self-supervised learning method to discover the manipulation concepts in robotic tasks. The concepts are verified with semantic meaning in terms of human linguistics while saving much manual annotation efforts. This can be used as auxiliary task to support the encoding in the policy optimization. Experiements demonstrate that the policy trained based on these learned concepts can achieve the state-of-the-art results.", "questions": "1. As generative models, there are many VAE variants. Can you explain why you choose VQ-VAE architecture or include ablation study?\n\n2. As mentioned in the paper: the manipulation concept, key state, and state are random variables depending on the trajectory. Does the initialization of these variables affect the experimental results? Or other prototype network approaches help?\n\n3. How about the generalization capability of InfoCon? In the paper, the training and testing tasks are same: P&P Cube, Stack Cube, Turn Faucet and Peg Insertion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. InfoCon can be self-supervised given state-action trajectory without human annotation, guided by network architecture VQ-VAE and informativeness objectives. Surprisingly, the self-supervised key states even performs better than the human GT in the COTPC for policy generation.\n\n2. The robot with InfoCon can discover abstract concepts themselves other than struggling with the grounding of concepts that are manually defined.\n\n3, The concepts of generative goal and discriminative goal are novel and beneficial to the trajectory encoding, which serves as the auxiliary task for self-supervision.\n\n4. Strong results in simulation comparing to extensive baselines.", "weaknesses": "1, The proposed approach and concept of self-supervised manipulation concepts and key states seems closely related to the COTPC, as COTPC needs the key states. However, in the method description, there is no mention of COTPC. It may be possible to achieve co-optimization between policy generation and self-supervised manipulation concept. Moreover, can the proposed approach be general beneficial to policy optimization beside COTPC?\n\n2. The experience portion is a bit weak where only few tasks are evaluated. For the baseline, it should also include COTPC + other manipulation concept discovery for fair comparison.\n\n3. Lack of real robotic experiments. It is hard to judge if the proposed self-supervised approach works for the real-world complex tasks and videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aX8VwZddno", "reviewer": "Reviewer_v8jc", "summary": "This paper proposes a method for extracting different manipulation concepts (i.e. grasp, align, insert) from expert trajectories of robot manipulation tasks. In essence, the algorithm unsupervisedly learns relevant subgoals to accomplish a task, and in addition, also provides a gradient signal for actions to achieve these (sub)goals. This is done by training two components: predicting the next subgoal given the current state, and training a compatibility function that indicates how \"compatible\" a state is with the desired subgoal, using contrastive learning. The gradient of the compatibility function can then be used to select actions, i.e. which action increases the goal compatibility given the current state. The method is evaluated on 4 tasks of the ManiSkill2 benchmark, and some qualitative examples are provided of the discovered subgoals.", "questions": "- What are the standard deviations on the results in Table 1. To what extent is InfoCon actually significantly better than the other baselines?\n\n- For some tasks (i.e. P&P Cube) the GT key states underperform the other approaches. Any insight on why the ground truth key states are insufficient to efficiently execute the task, and which are the \"extra\" subgoals identified by InfoCon that might explain this gap?\n\n- The model is trained on only 500 trajectories. To what extent is this overfitting to the train set, and does this explain the large gap between seen and unseen scenarios? Would this gap be closed by just adding more trajectories?\n\n- The VQ-VAE is pretrained on the trajectories without any task-related signal. Hence, the learned subsequences are merely clustered by visual appearance, rather than semantic relevance of being a valid \"subgoal\" for a task?  Wouldn't it make sense to also adjust the codes in the codebook based on e.g. how good one can predict a particular goal and/or how well-behaved a compatibility function is?\n\n- The policy is conditioned on the current state and the gradient of the compatibility function. Wouldn't it make sense to also condition the policy on the goal state, i.e. similar to e.g. https://arxiv.org/abs/2211.13350", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Interesting ideas and approach to go from expert trajectories to subgoals, and in addition obtain policies to accomplish those subgoals.", "weaknesses": "- The experimental results don't provide standard deviations, which makes it difficult to assess if there is any significant improvement compared to the presented baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)