Learning Unsupervised Forward Models from Object Keypoints
Authors: Alireza Rezazadeh
Changhyun Choi
Venue: Submitted to ICLR 2023 (2023)

Abstract: Object-centric representation is an essential abstraction for forward prediction. Most existing forward models learn this representation through extensive supervision (e.g., object class and bounding box) although such ground-truth information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network)---an end-to-end unsupervised framework to reason about object interactions based on a keypoint representation. Using visual observations, our model learns to associate objects with keypoint coordinates and discovers a graph representation of the system as a set of keypoint embeddings and their relations. It then learns an action-conditioned forward model using contrastive estimation to predict future keypoint states. By learning to perform physical reasoning in the keypoint space, our model automatically generalizes to scenarios with a different number of objects, novel backgrounds, and unseen object geometries. Experiments demonstrate the effectiveness of our model in accurately performing forward prediction and learning plannable object-centric representations which can also be used in downstream robotic manipulation tasks.

Under review as a conference paper at ICLR 2023
LEARNING UNSUPERVISED FORWARD MODELS FROM OВJЕCT KEYPOINTS
Anonymous authors Paper under double-bling review
ABSTRACT
Object-centric representation is an essential abstraction for forward prediction. Most existing forward models learn this representation through extensive supervi- sion (c.g., object class and bounding box) although such ground-trute information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network) an end-to-end unsupervised framework to reason about ob- ject interactions based on a keypoint representation. Using visual observations, our model learns associate objects with keypoint coordinates and discovers graph representation the system as a set of keypoint embeddings and their rela- tions. It then learns tion to predict future keypoint states. By learning to perform physical reasoning scenarios with
conditioned forward model using contrastive estima-
in the keypoint space, our model automatically generalizes different number of objects, novel backgrounds, and unseen object geometries. Experiments demonstrate the effectiveness of our model in accurately performing forward prediction and learning plannable object-centric representations which can also be used in downstream robotic manipulation tasks.
INTRODUCTION
Discovering a structured representation of the world allows humans to perform wide repertoire of motor tasks such interacting with objects. The core of this process is learning to predict the response of the environment applying an action (Miall & Wolpert 1996; Wolpert & Kawato 1998). The internal models, often referred to as the forward models, come up with an estimation of the future states of the world given its current state and the action. By caseading the predictions of forward model it is also possible to plan sequence of actions that would bring the world from an initial state to desired goal state (Wolpert et al. 1998 1995).
Recently, deep learning architectures have been proposed to perform forward prediction using object-centric representation the system (Ye el al. 2020 Chen et al. 2021b Li al., 2020a Qi et al. 2020 This representation is learned from the visual observation by factorizing the scene into the underlying object instances using ground-trute truth object states (c.g., object class, position, and bounding box). We identified two major limitations in the existing work: First, they either assume access to the ground-truth object states (Battaglia et al.) 2016 Li et al. (200aa) or predict them using idealized techniques such as pre-trained object detection instance segmentation models (Ye) et al. 2020: Qi et al. 2020). However, obtaining ground truth object states is not always feasible in practice. Relying on object detection and segmentation tools, on the other hand, makes the forward model fragile and dependent on the flawless performance of these tools which is often infeasible in real-world settings. Second, factorizing the scene into a fixed number of object instances limits the generalization of the model models to scenarios with a different number of objects.
In this paper, we address both of these limitations by proposing to learn forward models using key- point representation. Keypoints represent set of salient locations of moving entities. Our model KINet (Keypoint Interaction Network) learns an unsupervised forward model three steps: (I) A keypoint extracior factorizes the scene into keypoints with supervision other than raw visual observations. (2) A probabilistic graph representation the system is learned where each node cor- responds to keypoint and edges keypoints relations. Node features carry implicit object-centric features as well as explicit keypoint state information.
(3) With probabilistic message passing, our model learns an action-conditional forward model to predict the future location of keypoints and

---

Under review as a conference paper at ICLR 2023
reconstruct the future appearances of the scene. We evaluate KINet's forward prediction accuracy and demonstrate that, by learning forward prediction in a keypoint coordinate, our model effectively re-purposes this knowledge and generalizes it to complex unseen circumstances.
Our key contributions are: (1) We introduce KINet, an end-to-end method for learning unsupervised action-conditional forward models from visual observations (2) We introduce probabilistic Interac- tion Networks for efficient message- paassing to aggregate relevant information.
(3) We introduce the GraphMPC for accurate action planning using graph similarity. (4) We demonstrate learning forward models in keypoint coordinates enables zero-shot generalization to complex unseen scenarios.
2 RELATED WORK
Unsupervised keypoint extraction. Keypoints have been used in computer vision areas such as pose tracking (Zhang et al.] 2018, Yao et al. 2019) and video prediction (Minderer et al. 2019]
Zhang et al., 2018 Xue et al. 2016 Manuelli et al., 2020). Recent work explored keypoints for
control tasks in reinforcement learning to project the visual observation to lower-dimensional key-
point space (Kulkarni et al. 2019; Chen et al. 202 [akab et al.) (018).
Forward models. The most fundamentally relevant work to ours is Interaction Networks (IN) [Battaglia et al.] [016;) (Sanchez-Gonzalee work for forward simulation (Pfaff et al., 2020 Li et al. 2019, Kipf et al. 2018 Mrowca et al.) 2018). These methods rely on the ground-truth states of objects to build explicit object represen- lations. Several approaches extended IN by combining explicit object positional information with implicit visual features from images (Watters al., 2017 Qi et al. 2020 Ye et 2020). However, two main concerns remain unaddressed. First, visual features are often ex extracted from object bound- ing boxes using object detection or segmentation models (Janner et al. 2018 Qi et al. 2020 Kipf et al. (2018) that are either pretrained on the setup (Qi et al.) (020) or use the ground truth object position (Ye et al. 2020). Second, these approaches lack generalization as they are formulated on a fixed number of objects. Minderer et al. (2019) used keypoints for video prediction given a history of previous frames. However, their dynamics model is not formulated on external action and cannot be used for action planning applications.
al.
2018) and follow-up work using graph neural net-
Action-conditional forward models. Battagli et al. (016) augments the action as an external effect augmented to the node embeddings. Ye al. (2020) included action
additional node in a fully connected graph with other nodes representing objects. For probabilistic forward models Henaff al. (2019) suggests using a latent variable dropout mechanism to properly condition the model on the action (Gal & Ghahramani 2016). In a more relevant application to ours, Yan et al. (2020) highlighted the effectiveness of contrastive estimation (Oord et al. 2018) to learn proper ac- tionable object-centric representations.
Unsupervised Forwad Models. Kipf et al. 2019) uses an object-level contrastive loss to learn object-centric abstractions in multibody environments with deterministic structures and minimal vi- sual features such as 2d shapes. In our work, we randomize the properties of the system and examine realistic objects. Veerapaneni et al. (2020) infers set of entities in the image based on their depth and predicts future entity states. Entities are mixed using weight parameters of their distance from the camera. Our work, the other hand, does not make any assumption relies on keypoints. Kossen et al. (2019) uses images to infer a set of explicit states (e.g., position and velocity) for a fixed number of objects in dynamic system predict the future state of each object using graph networks. Although learning unsupervised state representation, this method is formulated on a fixed number of objects and only tested on environments with simple 2d geometries. Li et al. (2000b) infers the causal structure and makes future predictions on a fixed dynamic system using a pretrained keypoint extractor on topview images. In our work, we experiment with other camera angles and 3d objects with random properties such as geometry and texture.
inferring depth and
3 KEYPOINT INTERACTION NETWORKS (KINET)
We assume access to observational data that consists of RGB image, action vector, and the resulting
image after applying the action: D = {[I,, ue I.+1)]. Our goal is to learn a forward model that predicts the future states of the system with no supervision above the observational data. We describe our approach in two main steps (see Figure learning to encode visual observations into keypoints and learning an action-conditioned forward model in the keypoint space.
2

---

Under review as a conference paper at ICLR 2023
{+/
St
Gi+1
fkp
Fdwd
Satate
1
f
l+11
Figure 1: KINet performs forward modeling in three major steps: extracting keypoints, inferring probabilistic graph representation, and estimating the future appearance conditioned on the action. Some arrows are simplified for clarity.
3.1 UNSUPERVISED KEYPOINT DETECTION
The keypoint detector (fkp. Fig I is mapping from visual observations to lower-dimensional set of K keypoint coordinates { h = fxp (4)) The keypoint coordinates are learned by capturing the spatial appearance of the objects in an unsupervised manner. Specificallyy, the keypoint detector receives a pair of initial and current images (To, Iτ) and uses a convolutional encoder to compute K -ddimensional feature map for each image ((To), (b()) E HH'''" The expected number of keypoints is set by the dimension K. Next, the feature maps are marginalized into a 2D convolutional image reconstruction model
keypoint coordinate {= & 1k=1...K E R We (free Fig [ with skip connections to inpaint the current image frame using the initial image and the predicted keypoint coordinates 1, - froe(Ir Lo, {=y at Jk. With this formulation, fup and free create bottleneck encode the visual observation in temporally consistent lower-dimensional set of keypoint coordinates (Kulkarni et al. 2019).
3.2 GRAPH REPRESENTATION OF SYSTEM
After factorizing the system into K keypoints, we build a graph G. -V,, Z) (undirected, no self-loop) where keypoints and their pairwise relations the graph nodes and edges. Keypoint positional and visual information are encoded into embedding of nodes (n/ )==... E V2 and
edges 6 E & We also use adjacency matrix to specify the connectivity as Z E K
where [Zi E [0, 1] specifies the probability of the edge {e²) } E E, At timestep f, node embeddings [b]] Edge embeddings contain
encode keypoint visual and positional information {nt}
relative positional information of each node pair {c.²² = [a] - 2/ || =111
3.3 PROBABILISTIC INTERACTION NETWORKS
To build a forward model, we extend the recent approaches and propose a probabilistic variation of the Interaction Networks (IN) (Battaglia et al., 2016 Sanchez- probabilistic IN is generating edge-level latent variables Zij E R that represents the edge probabil- ity [Z], = Zij. posterior network po infers the elements the adjacency matrix given the graph representation of the scene. In particular, we model the posterior as po(2i) = ofenc (Ing n!])) where (.) is the sigmoid function.
al., 2018). The core of
The probabilistic IN forward model G.+1 fiwa (Gr, u, Z) predicts the graph representation at the next timestep by taking as input the current graph representation and action (ftwa, Fig 11. The message-passing operation in the forward model can be described as, (t0}
f.(n', n ce)),
(ń*) +- fa (n >i)((R) Zi ux) where the edge-specific function fo first updates edge em- beddings, then the node-specific function fn updates each node embedding a by probabilistically aggregating its neighboring nodes ((k) information (i.e., the edge probabilities from the inferred adjacency matrix input to the neighbor aggregation step. Note that the functions fenes for and fn are MLPs.
used as weights in the neighbor aggregation). The action vector uy is also an
Recent models for forward prediction rely on fully connected graphs for message passing (Qi et al. 2020 Ye al. 2020
Li et al. 2018). Our model, however, learns to probabilistically sample the

---

Under review as a conference paper at ICLR 2023
neighbor information. Intuitively, this adaptive sampling allows the network to efficiently aggregate the most relevant neighboring information. This is specifically essential in our model since keypoints could provide redundant information if they are in very close proximity.
3.4 FORWARD PREDICTION
The state decoder (fstate. Fig transforms the predicted node embeddings of the updated graph Gi+1 to a first-order difference which is integrated once predict the position of the keypoints in the next timestep (+*+1 = {-+} + [ (I). To reconstruct the image at the next timestep,
borrow the reconstruction model free from the keypoint detector le+1
free(Lo. { 1)).
3.5 LEARNING KINET
Reconstruction loss. The keypoint detector is trained using the distance between the ground truth
image and the reconstructed image at each timestep Lee ||4 Le As suggested by Minderer et al. (2019), errors from the keypoint detector were not backpropagated to other modules of the model. This is necessary step to ensure the model does not conflate errors from image modules and reasoning modules.
Forward loss. The model is also optimized predict the next state of the keypoints. A forward loss penalizes the distance between the estimated future keypoint locations using first-order state decoder and the keypoint extractor predictions: Liwd Σx || 1 f
Inference loss. Our model is also trained to minimize the KL-divergence between the posterior and prior distributions: Linter DEL (p& (Z g) [[()) We use independent Gaussian prior p(Z)
[l,W(z,) and use reparameterization trick for training (Kingma & Welling 2013).
Contrastive loss. We use the contrastive estimation method to further enhance the learned graph representations. We add a contrastive loss (Oord al.) 2018 Yan al.. (0200) and reframe for The predicted graph
graph embeddings as: Letr - EE [log(S((++11 (( SC44
representations Gi+1 are maximally similar to their corresponding positive sample pair g == Gi+1 and maximally distant from the negative sample pairs 9 G- Vr + ++1.
1. We use simple node embedding similarity as the graph matching algorithm S(G,, Ex (nf)-(n)) The motivation behind adding a contrastive loss is aligning the graph representation of similar object configurations while pushing apart those of dissimilar configurations in the embedding space to enhance the learned graphs.
Finally, the combined loss is: L= -vee Lree + Atw Liwd + Ainter Linter + Netr Letr.
3.6 GRAPHMPC PLANNING WITH KINET
We use a learned KINet model and select actions based Model Predictive Control (MPC) al- gorithm (Fino & Levine 2017) in the graph embedding space (GraphMPC). With KINet forward prediction. we compute the predicted graph representation of the next timestep for multiple sampled actions. We then select the optimal action that produces the most similar graph representation to the goal graph representation Geoal We describe our GraphMPC algorithm with a time horizon of T as: u = arg max (S (gral), fgwa(g', (u:rr)) ())rr))): only with respect to positional states, GraphMPC allows for accurately bringing the system to goal state both explicitly (i.e, position) and implicitly (i.e, pose, orientation, and visual appearance).
}: t E [0. 7). Unlike performing conventional MPC
4 EXPERIMENTAL SETUPS
Our experiments seek address the following: (1) How accurate is our forward model? (2) Can use the model for action planning? and (3) Does the model generalize to unseen circumstances?
4.1 DATASET
We apply our approach to learn a forward model for multi-object manipulation tasks. The task involves rearranging multiple objects in the scene and bringing them to the desired goal state using
4

---

Under review as a conference paper at ICLR 2023
pushing actions. In the RealBlocks dataset, the Sawyer robot pushing dataset in Ye et al. (2020) to exemplity how our proposed framework applies to real settings. RealBlocks data includes 5K random pushing actions on blocks. For each action, an RGB image pair with the action vector in the image coordinates is captured.
We use MuloCo 2.0 (Todorov et al. 2012) generate two sets of simulated scenarios. In BlockObjs dataset, we simulate 10K episodes of random pushing on multiple objects (1-5 objects) where a simple robot end-effector applies randomized pushing for 90 timesteps per episode (Figure A.112). Each object has randomized geometry and color. We sample the object geometry from a prede- fined continuous range denoted as geom for training and geom for generalization geometries. Unseen geometries are designed to have elongated shapes to create complex out-of- distribution cases (Figure A.I.II. In YCBObjs dataset, we simulate 10K episodes (60 timesteps) of randomly pushing subset of YCB objects placed on wooden table (1-5 objects) (Calli et al.) 2013) that includes objects of daily life with diverse properties such as shape, size, and texture (Fig- ure A.I.3). We collect the 4-dimensional action vector (pushing start and end location) and RGB images before and after each action is applied. Images are obtained using an overhead (Top View) and angled camera (Angled View) (see Appendix
unseen
A.I A.I for more details).
4.2 BASELINES
We compare our approach with existing methods for learning object-centric forward models:
Forward Model (Forw): We train convolutional encoder extract visual features of the scene image (/mg) and learn forward model in the feature space.
Forward-Inverse Model (ForwIny): We train a convolutional encoder extract visual features of the scene image (Img) and jointly learn forward and inverse models Agrawal et al. (2016).
Interaction Network (IN): We follow Battaglia et al.] (2016); |Sanchzz- Gonbzlzzz al. (2018) to build an Interaction Network based on the ground truth location of the objects. Each object repre- sentation contains the ground-truth position and velocity of the objects (GT' state). This approach only applicable to scenarios where the number of objects in the scene is known and fixed.
Visual Interaction Network ( (VisIN): To replicate Ye et al. (2200); Watters et al. (2017), we train convolutional encoder to extract visual features of fixed-size bounding boxes centered on ground- truth object locations (GT state + Img). We use the extracted visual features as object representations in the Interaction Network. This approach requires prior knowledge of the number of objects.
Causal Discovery from Videos (V-CDN): Li et al. (2020b) infers the causal structure of a fixed physical system from visual observations and makes future predictions for that system. A pretrained perception module extracts keypoints that are used in an inference module to predict a graph repre- sentation for the visual observation which is then used in an IN-based dynamics module predict the future location of the keypoints.
4.3 TRAINING AND EVALUATION SETTING.
All models are trained on subset of the simulated data (BlockOhjs and YCBObjs) with objects (8K episodes: 80% training, 10% validation, and 10% testing sets). To evaluate generalization to different number of objects, we use other subsets data with 1, 2, 4, and 5 objects (~ 400 episodes for each case). We train our model separately on images obtained from the overhead camera (Top View) and the angled camera (Angled View). For RealBlocks data, we only provide qualitative results as the ground-truth location of the objects is unknown. We set the expected number of keypoints K = 6 for BlockObjs, K = 9 for YCBObjs, and = 14 for RealBlocks.
5 RESULTS
This section is organized to answer series of questions to thoroughly evaluate our model and justify the choices made in formulating approach.
5.1 DOES THE MODEL ACCURATELY LEARN A FORWARD MODEL??
First, we evaluate the forward prediction accuracy. Figure Figure[]soweer showcases qualitative results of forward predictions RealBlocks. Our model factorizes the observation into keypoint representation
5

---

Under review as a conference paper at ICLR 2023
and accurately estimates the future appearance of the scene conditioned external action. The qualitative results highlight that our model learns the effect of the action on objects as well as object- to-object interactions (see Appendix A.4 for more examples).
It
{x/}
Ut 1.
[++1
le+1
Figure 2: Qualitative results RealBlocks dataset (Ye al. 2020). Given image It, model factorizes the scene into keypoints z, and conditioned on the action it (green arrows) estimates the next keypoint coordinates 1f41 and appearance I of the scene.
Table 1:
Forward Prediction performance on BlockOhjs measured as single-step predictions error.
Mean Position Error x
Model
Supervision
Top View
Angled View
Forw
Img.
0.309
0.317=00.0
ForwIns
Img
0.29310.09
0.266
IN
GT state
0.112.0.00
0.100
VisIN
GT state Img
0. 0.10722.c0
0.12120.00
KINet (Ours)
Ing
0.12210.01
0.129
KINet- deterministic
Img
0.127+0000
0.133=0001
KINet no ctr loss
Img
0.17311002
0.169
We quantify the effectiveness of our model in comparison with Forw, ForwInv, IN, and VisIN base- lines (Table [ on BlockObjs data. We separately train and examine each model on Top View and Angled View images. The prediction error is computed as the average distance between the predicted and ground-trute truth positional states. VisIN performs the best among baseline models builds object representations with explicit ground-truth object positions and their visual features. Our model, on the other hand, achieves a comparable performance to VisIN while it does not rely on any supervi- sion beyond the scene images. Forw and FnrwInv baselines have similar supervision to ours but are significantly less accurate. This emphasizes the capability of our approach in learning a rich graph representation of the scene and accurate forward model while relaxing prevailing assumptions of the prior work on the structure of the environment and availability of ground-truth state information (see Appendix A.2 for the learned graph representations).
5.2 CAN WE USE THE MODEL IN CONTROL TASKS?
We design robotic manipulation task of rearranging a set of objects to the desired goal state using MPC with pushing actions. For all models, and initial poses and random goal configuration of objects. The planning horizon is set to T = 40 timesteps in each episode. For our model, perform GraphMPC based graph embedding similarity as described in Section 3.6) For all baseline models, we perform MPC directly on the distance to the goal. Figure a shows MPC results of BlockObjs based Top View observations. Our approach is consistently faster than the baseline models in reaching the goal configuration.
IK episodes with randomized object geometries
5.3 DOES THE MODEL GENERALIZE TO UNSEEN CIRCUMSTANCES?
One of our main motivations to learn forward model in keypoint space is to eliminate the depen- dency of the model formulation on the number of objects the system which allows for generaliza- tion an unseen number of objects, object geometries, background textures, etc.

---

Under review as a conference paper at ICLR 2023
(a) MPC BlockObje
(b) MPC
BlockObje w/ Random Domain
Distance to Goal Position
Distance to Goal Position
0.4
randeass
IN
randgen
VoIN
0.3
Foewlor
0.3
forerrun
Fore
0.2
0.2
0.1
0.1
0.0
40 0.0
0
20
20 Timestep
40
Timestep
Figure 3: MPC results BlockObjs measured as the distance goal configuration. (a) Comparison with baselines. (b) MPC for KINet trained on a fixed white background (fixederain), generalization to random backgrounds (randgen), and trained on random backgrounds (randerain).
BlockOhjs. We train KINet on 3 randomized blocks with (ga) and test for zero-shot gen- eralization to an unseen number of objects (1, 2, 4, 5), unseen object geometries (geomgen). and unseen background texture. Figure 4 shows qualitative generalization results. We separately train and examine for generalization on Top View and Angled View observations with a planning horizon of T = 80. Since our model learns to perform forward modeling in the keypoint space, it reassigns the keypoints to unseen objects and makes forward predictions. Table 2 summarizes the generaliza- tion performance. As expected, by increasing the number of objects the average distance to the goal position increases. Also, objects with out-of-distributin geometries have more distance to the goal position. Our model's generalization performance is significantly superior to Forw baseline which is simple cenfionnooddiioned video prediction model that does not depend on the number of objects and uses the same supervision as our model.
Further, we test the performance of the model on unseen background textures (i.e., the table texture). Since the keypoint extraction relies on visual features of salient objects, our model is able to perform the control tasks by ignoring the background and assigning keypoints to the moving objects. Figure b.2 compares the MPC results for the KINet trained on fixed white background (fixedrrain), zero-shot generalization of KINet trained on the fixed background to randomized backgrounds (randgen), and KINet trained directly on randomized backgrounds (randerain). As expected, al- though the MPC converges, the final distance to the goal configuration is larger for randgen. This final error is statistically at the same level of accuracy for fixedtrain and randerain- For qualitative examples, see Figure 4 (third row) and Figure A.3.2
Table 2: Generalization results measured the average distance to the goal position.
Distance Goal Position x10-3
KINet (Ours)
KINet- deterministic
Forw
Objects
gnom.
goom gen
truin
geom
goom
gnom
0.24110.0
0.31
0.25 10.02
0.3410.05
1.1810.12
1.1310.07
2
0.22+0.01
0.5820.02
0.26
0.65
1.07
1.02=0.05
3
0. 1..880000
0.21
0.19.20.0
0.28 0..00.08
0.65
0.89
Top
4
0.5 0.54.10.001
0.63
0..8
0.801o.11
1.87
1.9610.13
5
0.8622008
1.73 16
0.94
2.01 14
1.95+0.12
3. 12
0.21 20.04
0.3520.04
0.28=a0.11
0.3020.0s
1.0940.0s
1.34
Angled View
2
0.21 20.00
0.53
0.22
0.59
1.12
1.27:0.09
3
0.19=0.02
0.20=0.05
0.19=an.os
0.31
0.98+0.11
1.16=00.09
0.51+0.02
0.65=t0.07
0.57=a011
0.9620.00 na
1.33+0.15
2. 2.25=o.13
0.80
1.6410.11
1.05.10.16
2.1710.10
1.4810.12
3.202:.19
YCBObjs. We evaluate the generalization of our model on a set of realistic YCB objects (Calli et al. 2015) with challenging diverse properties such as color, texture, mass, and geometry. We train our model on a random subset of YCB objects and test for generalization to an unseen number of objects (1,2,4.5) with planning horizon of T' = 40. As shown in Figure[] our method generalizes well to an unseen number of objects and performs the control task accurately. Importantly, assigning multiple keypoints each object allows our framework to implicitly capture the orientation of each object, as well as their position without any supervision on the object pose (e.g., compare the power drill pose in Fig, 5).
We compare the performance of our framework with V-CDN Li al. 2020b) baseline which is also keypoint-based model to learn the structure of physical systems and perform future predictions and
7

---

Under review as a conference paper at ICLR 2023
To
IT
yout
Io
Ir
I gool
gromi
geom pen
pen
geottleretn
dommiilger
doonaitggn
goom you
Reom pen
Olija
geom
Top View
Angled View
Figure 4: Qualitative results of generalization BlockObjs to unseen number of objects, geometries (geom gen) and background textures (domainleen.. The green arrows are the optimal actions.
potentially generalize to an unseen number of objects. Although V-CDN is formulated to extract the causal structure of fixed system through visual observations, we pretrain its perception module our randomized YCBObjs dataset for a direct comparison to our model. Also, to perform a control task, we condition the V-CDN on the action by adding an encoding of the action vector to each keypoint embedding. Figure [ compares the MPC results (normalized to the number of objects) of our method and V-CDN both trained on subset of YCB objects. The MPC performance of both models is comparable for rearranging 3 objects (green lines). However, V-CDN is most accurate the number objects it has been trained on and significantly less accurate when generalizing to unseen number of objects. We attribute this to two reasons: (1) Although using keypoints, V-CDN formulated to infer explicit causal structure and attributes for the graph representation of fixed system setup which does not necessarily carry over unseen circumstances. (2) Unlike our method, V-CDN does not take into account the visual features in the model and only uses the keypoint positions.
5.4 ANALYSIS AND ABLATION
We justify the major choices we made to formulate the model with ablation studies. We examine two elements in our approach: the probabilistic graph representation, and the contrastive loss. We train two variants of KINet: (1) KINer deterministic with a fully connected graph instead of probabilistic
(2) KINet no ctr loss without the contrastive loss. The best forward prediction for both Top View and Angled View images is achieved when the model is probabilistic and trained with contrastive loss (Table []] The contrastive loss is an essential element in our approach to ensure the learned forward model is accurately action conditional. Also, with probabilistic graph representation. our model achieves better generalization compared to the deterministic variant. This performance gap more evident when generalizing unseen geometries (Table 2).
8

---

Under review as a conference paper at ICLR 2023
ly
goot
good
TUR
Obje
Top View
Angled View
Figure 5: Qualitative results of generalization on YCBObjs unseen number of objects (1,2,4,5) for Top and Angled View observations.
Average Goal
0.5
KINet
V. CDD
# YCB ckj
0.4
0.2
0.1
0.0
Timetop
Timetep
Figure 6: YCBObjs MPC results for generalization to varying number of objects.
CONCLUSION
In this paper, we proposed method for learning action-conditioned forward models based only on image observations. We showed that our approach effectively makes forward predictions through keypoint factorization. Also, we demonstrated that keypoint- forward model, unlike prior work, does not make assumptions about the number of objects which allows for automatic general- ization to a variety of unseen circumstances. Importantly, our model learns forward model without explicit supervision fixing the number of expected keypoints (see Appendix [A.3). However, we showed this gives more generalizability compared fixing the number of objects. We also observed inconsistency in the predicted keypoints for real-robot data scenarios where most the objects were pushed out of the image frame (see Appendix A.6p. An interesting future direction is to focus the keypoint traction module to further enhance forward models for real settings. Finally, we hope our general approach inspires future research physical reasoning in settings where ground-truth information is hard to obtain.
ground-truth object state information. One limitation in our formulation is

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physies. arXiv preprint arXiv: 160.007499. 2016.
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.0022., 2016.
Berk Calli, Arjum Singh, Aaron Walsman, Siddhartha Srinivasa, Picter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference advanced robotics (ICAR), pp- 510-517. IEEE, 2015.
Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. arXiv preprint arXiv:2106.0766 2021a.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yez Yee Kenneth Wong, Joshua Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reason- ing. arXiv preprint arXiv:2 16564, 2021b.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pp- 2786-2793, IEEE, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncer- tainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.02770 2019.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Con- ference Neural Information Processing Systems, pp. 4020-4031 2018.
Michael Janner. Sergey Levine, William Freeman, Joshua Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint arXiv: 18/2. 10972, 2018
Diederik P Kingma and Max Welling. arXiv:1312.6111 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv:191..12244 2019.
Jannik Kossen, Karl Stelzner, Marcel Hussing, Claas Voelcker, and Kristian Kersting. Struc- tured object-aware physics prediction for video modeling and planning. arXiv preprint arXiv:1910.02425, 2019.
Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. arXiv preprint arXiv:1906.1 11883. 2019.
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. Proceedingy IEEE International Conference on Robotics and Automation, 120--Myy122551211 9 2018. URL www.lv..rg/abbs//ee hlEt:://axxiv.co httt://aaxxv..og//bb//BBB88
11169
Yunzhu Li. Jiajun Wu, Jun- Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-basee control under partial observation. In ICRA, 2019.
10

---

Under review as a conference paper at ICLR 2023
Yunzhu Li, Toru Lin, Kexin Yi, Daniel M. Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. 37th Inter- national Conference Machine Learning, ICML 2020, PurFF168147-8:5883--33 URL http:://axxv..o// atsp/bbs//220 2014 13644
4 2020a.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg, Causal discov- ery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020b.
Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the fu- ture: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05005, 2020.
Chris Miall and Daniel M Wolpert. Forward models for physiological motor control. Neural networks, ((8):1265-1279) 1996.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamies from videos. arXiv preprint arXiv: 07889, 2019.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Damiel LK Yamins. Flexible neural representation for physics prediction. arXiv preprint (rXivv/806.0810 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. arXiv preprint arXiv:2008. 02265, 2020.
Alvaro Sanchez-Gonzale Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470 4479. PMLR, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference Robot Learning, pp. 1439-1456. PMLR, 2020.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30:4539-4547, 2017.
Daniel M Wolpert and Mitsuo Kawato. Multiple paired forward and inverse models for motor control. Neural networks, 11(7-9):1317-332; 1998.
Daniel M Wolpert, Zoubin Ghahramani, and Michael Jordan. An internal model for sensorimotor 1995.
integration. Science, (896
Daniel M Wolpert, R Chris Miall, and Mitsuo Kawato. Internal models in the cerebellum. Trends in cognitive sciences, 2(9):338-34) 347, 1998.
Tianfan Xue, Jiajun Wu, Katherine I. Bouman, and William T Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional networks. arXiv preprint arXiv: 1607.0260 2016.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive represen- tations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003.05436 2020.
11

---

Under review as a conference paper at ICLR 2023
Yuan Yao, Yasamin Jafarian, and Hyun Soo Park. Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 753-762, 2019.
Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. In Conference on Robot Learning, pp. 100-109, PMLR, 2020.
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised dis- covery of object landmarks as structural representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2694-2703, 2018.
12


Reviews:
[{"review_id": "qmWGGKxapHD", "reviewer": "Reviewer_2bb6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method that learns dynamics models from object-centric representations in manipulation settings. This method is unsupervised and can generalize to different numbers of objects, unseen object shapes and unseen backgrounds. They demonstrate the effectiveness of this forward models by applying it to downstream robotics control tasks with a graph MPC algorithm.", "summary_of_the_review": "This paper presents a new method on unsupervised learning of forward models based on probabilistic graph representations. I am not convinced that the novelties presented in this method are crucial. Presenting more comprehensive comparison to existing methods and other suggested baselines may help.", "strength_and_weaknesses": "Strengths:\n1. This paper presents a method to learn unsupervised forward models that can be generalized to different number of objects.\n2. The method is straightforward and easy to understand.\n\nWeaknesses:\n1. In the related work section, authors mention that Kipf et al.(2019) is very relevant to your method. Kipf(2019) is also unsupervised and doesn't require ground-truth object locations. Could you include this method as part of the baseline as well?\n2. Another natural baseline would be Minderer et al.(2019), which also learns a dynamics model base on unsupervised keypoint representations.\n3. One of the main contribution is the probabilistic graph representation enabled by the learned adjacency matrix. The insight behind this design is that keypoints that are closer in the space could provide redundant information. However in the visualization of the adjacency matrix(supplement A.2.1), I can't reach to a conclusion as in what's being learned in these graphs. Can you provide more explanation and experiments to prove that this design is essential to the method(other than KINet-deterministic)?\n4. I am curious how would the method perform with more keypoints. If there are similar number of keypoints and objects in the scene, each object is only represented by one or two keypoints. Since each object is only represented by their keypoints, is one or two keypoints enough to capture the pose and orientation information?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mMcmpTtGRHR", "reviewer": "Reviewer_85iU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised framework to learn object-centric representation via keypoint representations. The model abstract the object by keypoints and form a graph representation, where the edges model the relationship between the nodes. An action-conditioned forward model is also learned using contrastive estimation. Experiments are conducted on the rearranging task in MuJoCo as well as a real robot dataset.\n", "summary_of_the_review": "My ratings are given based on my evaluation of the strengths and weaknesses above. More justifications are expected in the keypoint-based abstraction and distinction with prior work.", "strength_and_weaknesses": "Strength:\n- The idea of combining keypoint abstraction with unsupervised representation learning is novel and the learned forward makes downstream applications like task planning possible.\n- The experiments show that the proposed model can achieve high accuracy in forward prediction and the real robot execution task.\n\nWeakness:\n- Model design. While combining keypoints and object-centric learning is new, It’s not clear what would be the benefits of choosing a coarser representation as opposed to a denser latent representation. The keypoints abstraction is a bottleneck of the proposed framework, especially in more complex settings or environments. It’s also not obvious how the number of the keypoints $K$ will affect the model's performance as it’s essentially an environment/dataset-dependent hyperparameter. The reconstruction model $f_{rec}$ also depends on $K$. It remains to be seen how the model will perform with more cluttered backgrounds.\n- Discussion about more related work. [1] is also a probabilistic object-centric and action-conditioned video prediction model, and interactions between entities are also modeled. [2] Introduces “velocity” and physical interaction and may also be used in modeling action-aware object-centric representations. What are the key differences between the proposed model in terms of the advantages and disadvantages in both representation discovery and forward prediction? \n[1] Veerapaneni R, Co-Reyes J D, Chang M, et al. Entity abstraction in visual model-based reinforcement learning[C].\n[2] Kossen J, Stelzner K, Hussing M, et al. Structured object-aware physics prediction for video modeling and planning[J].\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kIwiveUYfTn", "reviewer": "Reviewer_gVjw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use key points as internal representations of scenes and model forward dynamics with graph interaction networks. The authors used a multi-object manipulation testbed for evaluating the proposed Keypoint Interaction Network (KINet). They show that KINet can achieve similar results for forward prediction compared with a baseline graph model with ground truth object positions. The authors also show that KINet converges to the goal configuration faster than baselines and generalizes to unseen circumstances where the number of objects, geometries, and background textures are unseen during training.", "summary_of_the_review": "This paper focuses on learning keypoint-based representations for modeling the forward dynamics of a system. However, the proposed method shares similarity with previous works and the design choices seem to have limited significance as shown in a synthetic dataset setting. Therefore, I don't recommend this paper for acceptance for now. I suggest the authors polish the experiment section and design better experimental settings to show the significance of their method compared with the baselines mentioned in strengths&weakness.", "strength_and_weaknesses": "[+] The paper proposes to extract object key points as representations for modeling forward dynamics. Such an unsupervised learning setting could be treated as a starting step toward many more research areas.\n\n[+] The experiment section is well-organized with not only an evaluation of the future prediction but also a manipulation task.\n\n[-] One major concern lies in the novelty of this paper. The current method shares the same insight as in previous works like OP3[1] and V-CDN [2] since they both adopt an unsupervised key point/mask detection and then a graph message-passing paradigm for solving the dynamics learning problem. Compared with methods, this paper's novelty seems incremental as they can also handle the generalization to an unseen number of objects. The authors should consider adding proper comparison with these two methods as baselines to make a point on the significance of the current model design.\n\n[-] Meanwhile, the currently selected datasets seem simplistic and fail to illustrate the potential of the model on more complex scenes. Under this setting (2D overhead or top view), the results fail to show the effectiveness of model design (e.g. deterministic vs. probabilistic). The authors should consider settings that better justify the significance of their design, especially when other methods showed more complex scenarios like morphing clothes.\n\n[-] A minor question lies in the design for keypoint detection. Why is the current detector taking as input the pair ($I_0$,$I_t$) but not the more common one ($I_{t-1}$, $I_t$)? I don't see a clear advantage in this design since the model will be required to start prediction from a random initial configuration and always comparing with the initial state will cause trouble in long-horizon planning or prediction (i.e., forgetting in long future predictions).\n\n[1] Veerapaneni et al., \"Entity Abstraction in Visual Model-based Reinforcement Learning\", CoRL 2019.\n[2] Li et al., \"Causal Discovery in Physical Systems from Videos\", NeurIPS 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tv9ezUoF_S", "reviewer": "Reviewer_E8Ms", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present Keypoint Interaction Network (KINet) - an unsupervised learning method to associate keypoints with objects and uses a forward motion model to estimate future keypoint states. As an unsupervised method, the authors make no assumptions about access to ground truth semantic keypoint locations, but rather just use image observations and actions.", "summary_of_the_review": "Overall the paper seems novel and is well written. It is likely to be of interest to the community, however I do find the experimental results to be somewhat underwhelming (hence the final score). I’d like to see more real-world examples with more complex object and scene appearances.", "strength_and_weaknesses": "Strengths:\n\n- Not needing ground truth keypoints is an obvious win. It should make this method usable in multiple setups.\n- The keypoint extraction and graph association are object agnostic, meaning that the method should generalize to unseen objects and scenes. That is, the keypoints have no specific semantic encoding that would cause generalization to fail to other classes of unseen objects.\n\nWeaknesses:\n\n- The fixed “K” in the keypoint detector seems particularly problematic. If K is overcomplete (i.e. just some huge value) for describing the intrinsic degrees of freedom of an object, you could argue that perhaps a fixed K is OK, but generally, for articulated or deformable objects, you might not know a priori the “K” value that is needed to fully recover the pose of an object. Another argument to make is that the k-dimensional feature map is just yet another hidden layer of a larger network (that happens to include the graph representations) - in which case it’s probably a stretch claiming these are semantic keypoints at all.\n- Related to above, the authors claim that the K representation is a bottleneck, however how much of a bottleneck seems to be very much dependent on the value of “K”.\n- Addition of the contrastive loss seems like a tacked-on solution and the authors need to do a better job describing what problem it solves (or what additional regularization it adds). The ablation where it is removed seems fine, but this alone is not enough to justify its use.\n- The class of objects the proposed method is evaluated on seems particularly limiting. The real-world results are all roughly rectangular shapes on a consistent background with minimal occlusion (there’s some slightly more complicated shapes in the sim experiments, but even these are simple from an appearance perspective). I would like to see how this method performs on more complex scenes with background clutter and even on articulated or soft-bodied objects. Why is this not unreasonable or out of scope for this paper?: because the promise of this unsupervised method is that it works on any objects without needing semantic labels, however these experimental results fall short of showing that IMO. By contrast, for THIS particular application, you could very likely do simple heuristics to get a set of keypoints that would probably work just as well (e.g. color centroids, etc).\n- Nit: The Fig 2 real-robot data does not feel like “robot data” to me (I work primarily in robotics). There’s no robotic embodiment visible in the frame, no active perception, etc. For instance a human could have moved the objects between frames. This is a perfectly fine computer vision dataset, but I think it’s a stretch to claim it’s somehow robot data. 5.2 presumably uses a robot (with MPC on your forward model) to actually move the objects, so why isn’t this visible in the frames?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject