KINet: Keypoint Interaction Networks for Unsupervised Forward Modeling
Authors: Alireza Rezazadeh
Changhyun Choi
Venue: ICLR 2022 Submitted (2022)

Abstract: Object-centric representation is an essential abstraction for physical reasoning and forward prediction. Most existing approaches learn this representation through extensive supervision (e.g, object class and bounding box) although such ground-truth information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network)---an end-to-end unsupervised framework to reason about object interactions in complex systems based on a keypoint representation. Using visual observations, our model learns to associate objects with keypoint coordinates and discovers a graph representation of the system as a set of keypoint embeddings and their relations. It then learns an action-conditioned forward model using contrastive estimation to predict future keypoint states. By learning to perform physical reasoning in the keypoint space, our model automatically generalizes to scenarios with a different number of objects, and novel object geometries. Experiments demonstrate the effectiveness of our model to accurately perform forward prediction and learn plannable object-centric representations which can also be used in downstream model-based control tasks.

Under review as a conference paper at ICLR 2022
KINET: KEYPOINT INTERACTION NETWORKS FOR UNSUPERVISED FORWARD MODELING
Anonymous authors
Paper under double-blind review
ABSTRACT
Object-centric representation is essential abstraction for physical reasoning and forward prediction. Most existing approaches learn this representation through extensive supervision (e.g., object class and bounding box) although such ground- truth information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network) an end-to-end unsupervised framework to reason about object interactions in complex systems based on a keypoint rep- resentation. Using visual observations, our model learns associate objects with keypoint coordinates and discovers graph representation of the system set of keypoint embeddings and their relations. then learns an action-conditione forward model using contrastive estimation to predict future keypoint states. By learning to perform physical reasoning in the keypoint space, our model auto- matically generalizes to scenarios with different number of objects, and novel object geometries. Experiments demonstrate the effectiveness of our model to accurately perform forward prediction and learn plannable object-centric repre- sentations which can also be used in downstream model-based control tasks.
INTRODUCTION
Discovering a structured causal representation of the world allows humans to perform a wide reper- toire of motor tasks such as interacting with objects. At the core of this process lies the ability to predict the response of the environment to applying an action (Miall & Wolpert 1996 Wolpert & Kawato 1998). Such an internal model, often referred to as the forward model, aims to come up with an estimation of future states of the world given its current state and an action. By cascading the predictions of forward model it is also possible to plan sequence of actions that would bring the world from an initial state to desired goal state (Wolpert et al. 1998 1995).
Recently, various deep learning architectures have been proposed to perform forward modeling using an object-centric representation of the system (Ye et al. 2020) Chen et al.) 20216 Li et al.) 2020, Qi et al. 2020). This object-centric representation is learned from the visual observation by factorizing the scene into the underlying object instances using ground-truth object states (e.g.. object class, position, and bounding box).
We identified two major limitations in the recent work: First, existing approaches either assume access the ground-truth object states (Battaglia et al. 2016 Li al. (2020) or predict them using idealized techniques such as pre-trained object detection or instance segmentation models (Ye et al., 2020 Qi et al. 2020). However, obtaining ground truth objects state information is not feasible in practice. Relying on object detection and segmentation tools, on the other hand, makes the forward model fragile and dependent on the flawless performance of these tools. More often than not, pre- trained object detection or segmentation models suffer from poor generalization to unseen objects. Second, factorizing the scene into object instances limits the generalization of forward models to scenarios with different number of objects.
In this paper, we address both of these limitations by proposing to learn forward models using a beye object-centric representation. Keypoints represent set of salient locations of mov- ing entities. Our model KINet (Keypoint Interaction Network) learns unsupervised forward mod- eling in three major steps: (1) A keypoint extractor factorizes the scene into keypoint coordinates (2) A probabilistic graph representation of the system is inferred where each node corresponds to keypoint and edges keypoints relations.
with no supervision other than raw visual observations.

---

Under review as a conference paper at ICLR 2022
[+)]
(
Training
G2
g++1
xxp
fkep
PolA, Zo.)
I
I++1
(
free
frwd
Zr
PolZe 96-1)
U4
Gi+1
ZrZ
fstato
(#t+1)
free
Figure I: KINet Architecture. Our model performs forward modeling based on image observations in three major steps: extracting keypoint coordinates, inferring probabilistic graph representation of the system, and estimating the next state of the system conditioned on the action. Learned func- tions and distributions are the blue blocks.
Therefore, each object is represented in this graph by least distinct node. Node features carry implicit object-centric representation as well as explicit keypoint state information. (3) With prob- abilistic message passing, our model performs physical reasoning and learns action-conditional forward model to predict future locations of the keypoints. Using this prediction, it also reconstructs the future appearances the system. We evaluate KINet's forward prediction accuracy and demon- strate that, by learning the physical reasoning in keypoint coordinate, our model can effectively re-purpose this knowledge and generalize
complex unseen circumstances.
Our key contributions are as follows: (1) We introduce KINet, an end-to-end method for learning un- supervised contiinnnoniioonall forward models from visual observations using keypoint coordinates. (2) We introduce Probabilistic Interaction Networks for efficient message- message-passing in fully connected graphs by learning to adaptively aggregate subset relevant neighborhood information. (3) We introduce the GraphMPC algorithm for accurate action planning based graph similarity. (4) We demonstrate that learning forward models in keypoint coordinates enables zero-shot generalization to complex unseen scenarios.
2 RELATED WORK
Unsupervised keypoint extraction methods have been used successfully in computer vision tasks such as pose tracking (Zhang et al.) 2018 Yao et al. 2219)) and video prediction (Minderer et al. 2019 Zhang al.] 2018 Xue et al. 2016). Recent work explored keypoint extraction for control tasks in reinforcement learning environments to project the visual observation space to a lower- dimensional keypoint space (Kulkarni et al. 2019 Chen al. 2021a Jakab et al. 2018 We
extend these methods for the object manipulation setting.
Forward prediction models estimate the future state of system given its current state. There is a large line of work on using neural networks to build such models for various applications. These approaches date back to Nguyen & Widrow (1990); Grzeszczuk et al. (1998) for modeling state transition in dynamic systems. The most fundamentally relevant approach to our model is the Inter- action Network (IN) (Battaglia al. 2016, Sanchez- on using graph neural network in forward modeling (Pfaff et al. 22200; Li et al.) 2019 Kipf et al.) 2018 Mrowca et al. 2018). Although these approaches demonstrated powerful forward modeling capabilities, they often rely building explicit object representations based on ground-truth state information of the objects in the system. In an attempt to bridge this gap several methods have been proposed to IN-based models using visual features extracted from image observations. (Watters) jet al., 2017) Qi et al. 2020; Ye et al. 2020). However, two main concerns remain unaddressed in these methods. First, the visual features of objects often extracted from object bounding boxes
al. 2018)) and other follow-up work
2

---

Under review as a conference paper at ICLR 2022
using pretrained object detection segmentation model (Janner al. 2018 Qi et al. 2020: Kipf
et al. 2018) that is either pretrained on the environment (Qi et al. 2020) or assume prior knowledge of the object position YYe et al. 2020). Second, implementing these approaches entails training the model on a fixed number of objects.
Learning aetion-condittonall forward models is active challenge although the original IN in- corporates the action as an external effect by aggregating the action vector to node and edge em- beddings. For the case of probabilistic forward models HHenaff al.] (2019) interestingly suggests using a latent variable dropout to ensure proper conditioning of the forward model on the action information following Gal Ghahramani (2016). In a more relevant application our model, Yan et al. (2020) highlighted the effectiveness of contrastive estimation (Oord et al. 2018) to learn proper plannable object-centric representations.
Graph representation inference methods has been originally introduced in variational graph au- toencoders by Kipf Welling (2016). To formulate the graph structure inference module we were also inspired by Simoonvsky & Komodakis (2018) where they use variational graph autoencoder to generate small graphs of molecular structures. More relevant to our work Kipf et al. (2019) in which object-level contrastive loss has been used to learn object-centric abstractions in multibody systems. However, this work is only restricted to fixed environments with oversimplified objects such -re-dimensional geometries (Watters et al.) 2019 Stanić al. (2020). In our work, we con- sider experimenting with more complex three-dimensional objects where the objects are randomized and replicate real-world object manipulation.
3 KEYPOINT INTERACTION NETWORKS (KINET)
We assume access to observational data that consists of images, actions, and resulting images after applying actions: D {Ie, ut [1+1)} where I,,1 E R are images before and after applying action and me E TR is the action vector. Our goal is to learn a forward model that predicts future states of the system with no supervision above the observational data. In this section, we describe our approach which is composed of two main steps: learning to encode visual observations into keypoint coordinates and then learning an action-conditioned forward model in the keypoint space reason about the system and predict its future state.
3.1 UNSUPERVISED KEYPOINT DETECTION
The keypoint detector fkp is mapping from visual observations lower-dimensional set of K keypoint coordinates { k= capturing the spatial appearance of the objects in the system in an unsupervised manner. The detector design following the idea of Kulkari et al. ( et (2018), which we extend for object manipulation setting that necessitates consideration of external actions.
(xp).. The keypoint coordinates are learned through
Specifically, The keypoint detector receives a pair of initial and current image frames (Jo, It) and uses convolutional encoder to compute a K K-dimensional feature map for each image (Lo),, (1e) E #f x The expected number of keypoints in the system is captured by the dimension K. Next, each keypoint feature map is marginalized into a 2D keypoint coordi- R We convolutional image reconstruction model free with skip connections to inpaint the current image frame using the initial image and the predicted keypoint coordinates f, = Srea(Tn, {a8, })
nate {=, x² 1 k=
With this formulation, fup and free create bottleneck encode the visual observation in tem- porally consistent lower-dimensional keypoint coordinate representation which is distributed across the visual observation.
3.2 GRAPHI REPRESENTATION OF SYSTEM
The primary idea of our approach is representing the system as a graph. In particular, after factor- izing the system into K keypoints, become the graph nodes and their pairwise relations become the graph edges. Keypoint positional and visual information are encoded into the feature embeddings of nodes (n/)==l... E V, and
build directed graph G, = =V,, E,, Ar) where keypoints
3

---

Under review as a conference paper at ICLR 2022
edges {e,) & We also adjacency matrix to further specify the graph connectivity as A, RR M where {e²} E Ef [Ar]is =
At each timestep t the observational data is encoded into the graph. Node embeddings are an en- coding of the keypoint visual features and positional information extracted in the keypoint detector {u,} = x / Edge embeddings contain a relative positional information of the sender and z || z [ as suggested by Pfaff al. 2020)). We assume that
receiver nodes = there is no self-loop in the graph nodes (i.e, diagonal elements in the adjacency matrix is set to 0). Note that the graph representation is built keypoints coordinates and, hence, we do not impose any prior assumption on the number of objects in the system.
[2]
3.3 PROBABILISTIC INTERACTION NETWORKS
To build a forward model we extend the recent approaches based on Graph Neural Networks (Battaglia et al., 2016] [Sanchez- Gonzalez et al. 2018 Plaff et al.] (020) and propose a proba-
bilistic variation of the Interaction Networks (IN). The core of the probabilistic IN generating node-level latent variables Z; IRMI (Kipf & Welling, 2016 Simonovsky & Komodakis 2018). The node-level prior latent variables are independently sampled from fixed isotropic Gaussian prior distribution p(z)).
During training, the posterior distribution of the latent variable is obtained from the next timestep graph,
qo (Zelkn ( = fenc (9++1)
We use probablistic decoder to derive the posterior probability of the adjacency matrix given the latent variable. To build this decoder, the existance of edges in the graph modeled as Bernoulli variables with probabilities defined as,
po ([Ai)y 1|2 z/) = a[Sao (Gr, Z.))
where (-) is the sigmoid function. We propose to use the posterior adjacency matrix distri- bution along with the latent variables to infer a probabilistic graph representation of the system G. = (V,, &,, Ar). In particular, the inferred adjacency matrix A, pa (A.A, /z.) includes the edge probabilities. The node embeddings are also appropriately aggregated with the node-level latent variables {n [n. 2/1]==1.... K E Vj-
The probabilistic IN forward model ف.+ = Stwa (Sr, u,; Z, A,) predicts the graph representation at the next timestep by taking as input the current probabilistic graph and action. The message-passing operation in the forward model can be described as,
=
{ey} + fa n. fe( 6*), (n+}
fn
Σ E ue)
(3)
(EN(K)
where the edge-specific function fe first updates edge embeddings, then the node-specific function fa updates node embeddings using neighboring edge information. Note that the message-passing operation takes into account the node-level latent variables and the neighborhood aggregation NNk)
also performed based the probabilistic adjacency matrix.
Recent models for forward prediction often rely on fully connected graphs for message passing (Qi al. 2020; Ye et al.] 2020 Li et al., 2018). Our model, however, learns to dynamically sample the neighborhood of each node at each timestep conditioned on the latent variable. Intuitively, this adaptive sampling allows the network to efficiently aggregate long-range context information only by selecting the most relevant neighboring nodes. This is specifically essential in our model since keypoints could provide redundant information if they are in very close proximity.
3.4 FORWARD PREDICTION
Using the predicted graph representation S.+1. we interpret first-order difference of the keypoint states. The state decoder fstate transforms the predicted node embeddings to a first-order difference which is integrated once to predict the position of the keypoints in the next timestep { =
(xr) } + fatat ((,
4

---

Under review as a conference paper at ICLR 2022
To reconstruct the image at the next timestep borrow the reconstruction model free from the key- point detector. The initial image along with predicted keypoint states at the next time step produces an estimated rendering of the future appearance of the system f+41 = Srec (T, {= 4 klk=.....k)
3.5 LEARNING KINET
Reconstruction loss. The keypoint detector is trained using the L2 distance between the ground truth image and the reconstructed image each timestep Lree || 4 ll: As suggested by Minderer et al. (2119) errors from the keypoint detector were not backpropagated to other modules of the model. This
necessary step ensure the model does not conflate errors from image modules and reasoning modules.
Inference loss. Our model is also trained infer the adjacency matrix. This goal is acheived by optimizing the evidence lower bound (ELBO) (Kipl' & Welling 2016). This includes maximizing the likelihood of the adjaceney matrix given the latent distribution and minimizing the KI.-divergence between the posterior and prior latent distribution:
Linter g. (2 po (A] Z) + DRL.
(yo.(1o (1o (Z (Z))
(4)
We take independent Gaussian prior p(Z)= = II; and posterior qe (Zg)) (Z (g) distributions for node latent variables and use reparameterization trick for training (Kingma & Welling 2013).
Forward loss. The model is also optimized to predict the next state of the keypoints. A forward loss penalizes the L2 distance between the estimated future keypoint locations using first-order state decoder and the keypoint extractor's prediction:
Liwd Σ Ela+t+1 (1p((1+2))113 K
(5)
Contrastive loss. In our model, we are secking to learn representations build an action condi- tioned forward model. Therefore, to further encourage the model to learn actionable object-centric representations, we make use of the contrastive estimation method. We add contrastive loss as described in Oord et al. (2018): Yan et al. (2020) and reframe it for graph embeddings as:
S(9
Car Eoolog(
where S is graph matching algorithm. With this loss, maximize a lower bound on the mutual information of the learned graph representations. Specifically, we want to ensure that the predicted graph representations G:-1 are maximally similar to their corresponding positive sample pairs G and maximally distant from the negative sample pairs 9 We use a simple node embedding similarity as the graph matching algorithm S(G,, G2) = Σx [n?].
Finally, the combined loss of the model can be written as:
L Arer froe + Atwd Liwdd + Xinfor Linter + Aetrr Letr
3.6 GRAPIIMPC PLANNING WITH KINET
We use learned KINet model and plan actions based on Model Predictive Control (MPC) algo- rithm (Finn& & Levine 2017) in the graph embedding space (GraphMPC). We sample several actions at each timestep to apply them to the current graph representation of the system. With KINet for- ward prediction, we compute a predicted graph representation for the next timestep given a sampled action. We then take the optimal action that produces the most similar graph representation to goal graph representation gooal We describe our GraphMPC algorithm with a time horizon of as:
u = arg max S (g88aa) fiwa(G', (uur))) [0,
(8)
5

---

Under review as a conference paper at ICLR 2022
where S is graph matching algorithm. Unlike performing conventional MPC only with respect to positional states, GraphMPC allows for accurately bringing the system to goal state both explicitly (i.e, position) and implicitly (i.e, pose, orientation, and visual appearance).
4 EXPERIMENTAL SETUPS
Our experiments are motivated by the following questions: (1) Does the model accurately learn forward model? (2) Can we use the model implement action planning? and (3) How well our model generalizes to unseen circumstances?
4.1 ENVIRONMENT
We apply approach to learn forward model for multi-object manipulation tasks. The task involves rearranging multiple objects in the scene and bringing them to a desired goal state using point-to-point pushing actions. We use MuJoCo 2.0 (Todorov al. 2012) to simulate training and testing scenarios.
Specifically, generate total of 10K episodes of random object manipulations where multiple objects (1-5 objects) are present in the scene and a simple robot end-effector applies randomized pushing actions for 90 timesteps per episode. In the simulation, each object is represented bination of cuboid geoms with randomized length, width, and color to diversify the objects. We uniformly sample the randomized length and width from a predefined continuous range (geomtrain for training and geomagn for generalization to unseen geometries, see Appendix for more details). At each timestep, we only collect the 4-dimensional action vector (pushing start and end location) and RGB images of the scene before and after the action is applied. Images are obtained using an overhead camera (Top View) and an angled camera (Angled View) (see Fig 3).
com-
4.2 BASELINES
We compare our approach with previous methods on learning object-centric forward models:
Forward-Inverse Model (Forwiny): We train convolutional encoder to extract visual features of the scene image (Img) and jointly learn forward and inverse models in the feature space following Agrawal et al. (2016).
Interaction Network (IN): We follow Battaglia et al. (2016); Sanchez-Gonzalez al. (2018) to build an Interaction Network based the ground truth location of the objects. Each object is represented with a vector representation that contains the ground-truth position and velocity of the objects (GT' state). This approach is only applicable to scenarios where the number of objects in the scene known and fixed.
Visual Interaction Network (VisIN): We train convolutional encoder extract visual features of fixed-size bounding boxes centered on ground-truth object locations (GT state + Img). We use the extracted visual features object representations in the Interaction Network. This approach also requires prior knowledge of the number of objects in the scene (Kipf et al.) 2019 2017).
Watters et al.
4.3 TRAINING AND EVALUATION SETTING.
All models are trained on a subset of the simulated data where objects are present in the scene which gives total of approximately 8K episodes. We split this dataset into training (80%), valida- tion (10%), and testing (10%) sets. We report the performance of our model using the testing data. To evaluate generalization different number of objects, we use other subsets of data with 1, 2, 4, and objects (approximately 400 episodes for each case). We also measure generalization for unseen geometries where sample object length and width out of the training dimension range. We train our model separately on images obtained from the overhead camera (Top View) and the angled camera (Angled View).

---

Under review as a conference paper at ICLR 2022
Table I: Forward Prediction performance measured in position error for single-step prediction. The prediction errors are computed for models separately trained on Top and Angled View images. Our model accurately predicts the next timestep using only image observations.
Mean Position Error x10-3
Model
Supervision
Top View
Angled View
Forwinv
Img
0.293
0.266:t0.02
IN
GT state
0.112 003
0.109:+0.00
VisIN
GT state Img
0.107
0.121 10.03
KINet (Ours)
Img
0.122
0.129:10.0
KINet- deterministic
Img 0.127 1.003
0.133
KINet no ctr loss
Img
0.173
0.169
5 RESULTS
This section is organized to answer series of questions to thoroughly evaluate our model and justify the choices we made in formulating our approach.
5.1 DOES THE MODEL ACCURATELY LEARN A FORWARD MODEL?
First, we evaluate if our model can accurately perform single-step forward prediction. The predic- tion error computed as the average L2 distance between the predicted and ground truth positional states. Note that in the baseline models, the number of objects in the scene is known and fixed (N 3). However, our model does not make any assumption on the number of objects. Instead, we only set the expected number of keypoints in the system (K 6).
We demonstrate the effectiveness our model comparison with Forwlnv, IN, and VisIN baseline performances (Table [ We separately train and examine the performance of each model on Top View and Angled View images. Among baseline models, VisIN performs best since it uses ground- truth object position information along with an encoding of the object visual features to build object representations. Our model, on the other hand, achieves a comparable performance to VisIN while it does not rely on any supervision beyond the scene images.
We also compare our model against ForwInv baseline that has similar supervision for training and show that model produces forward predictions that are significantly more accurate. This empha- sizes the capability of our approach to accurately learn a forward model while relaxing prevailing assumptions of the prior methods the structure of the system and availability of ground-truth state information.
5.2 CAN USE THE MODEL IN CONTROL TASKS?
To further compare approach with baseline methods, quence of pushing actions to bring objects to a desired goal state using an MPC planning based on Top View observations. For all models, we perform 1000 trials where at each trial object geometries and initial pose are randomized and a random goal configuration of objects also generated. The planning horizon is emhedding similarity as described in Section 3.6, For all baseline models, we perform MPC directly based on the distance to the goal.
our model to predict a T-step
T 40 timesteps. For our model, we perform GraphMPC based graph
Figure 2 shows MPC results over the planning horizon. Our approach is consistently faster than baseline models reaching the goal configuration. Additionally, we compute the pixel-wise dis- tance to the goal image to evaluate whether the planned actions reach the goal configuration in terms of object appearances (i.e, orientation).
5.3 DOES THE MODEL GENERALIZE TO UNSEEN CIRCUMSTANCES?
One of our main motivations to learn a forward model in keypoint coordinate space is to eliminate the dependency of model formulation the number of objects in the system. We test for zero-

---

Under review as a conference paper at ICLR 2022
Distance to Goal Position
Distance Goal Image
KINet
0.4
0.8
IN
0.3
0.6
VisIN
ForwIny
0.2
0.4
0.2
20
40
20
40
Timestop
Timestep
Figure 2: MPC comparison with baseline models for Top View images. Distance to goal measured based on of (a) position and (b) pixel errors. Comparison with baseline models shows that model faster and more accurate for planning.
shot generalization to a different number of objects (1, 2, 4, and 5 objects) and unseen geometries (geomg..)) See Figure for qualitative generalization results. We separately train and examine for generalization on Top View and Angled View observations. Note that baseline models are not able to generalize to a different number objects because this number hard-coded in their formulation (e.g., in the number of feature channels or the number of graph nodes).
For generalizations, we set the planning horizon to T = 80. Since our model learns to perform forward modeling in the keypoint space, with zero-shot generalization, it reassigns the expected keypoints (K = 6) to unseen objects and then makes forward prediction. We observed that the keypoint extraction is, however, slightly less consistent compared to the trained scenario with objects.
We also quantitatively evaluate the quality of generalization scenarios (Table and expected by increasing the number of object the average distance to goal position increases. Also, objects with geometries that were sampled out of the training dimension range has more distance the goal position.
5.4 ANALYSIS AND ABLATION
We further justify the major choices we made to formulate the model by conducting ablation studies. Specifically, we examine two clements in our approach: the probabilistie graph representation, and the contrastive loss. We train two variants of KINet: (1) KINer deterministic in which the graph representation
probabilistic and there's 's inference the graph adjacency matrix. (2) KINet no ctr where we train the model without adding the contrastive loss.
The best forward prediction performance for both Top View and Angled View observations is achieved when the model probabilistic and trains with a contrastive loss (Table The contrastive loss is essential element in our approach to ensure the learned forward model accurately action conditional. Also, probabilistic graph representation allows for more efficient information aggre- gation of neighboring nodes. With a probablistic graph representation our model achieves higher generalization performance compare to its deterministic variant. This performance gap is more evi- dent when generalizing unseen geometries (Table 2)
CONCLUSION
In this paper, we proposed general method for learning cetion-conditioned forward models based on image observations of the system. We showed that our approach effectively performs physical reasoning by inferring the structure of the system. Additionally, we demonstrated that keypoint- based forward model makes fewer assumptions about the system. This in turn allows for au- tomatic generalization to a variety of unseen circumstances. Our method addresses a frequent is- accurate forward model without explicit supervision
with the prior work by learning ground truth object information. An interesting future direction is to focus on the keypoint extrac- tion sim2real gap further help with building reliable forward models for real settings. Finally,
8

---

Under review as a conference paper at ICLR 2022
Ia
Fr
I goot
Io
Ir
1 gool
geOlltssinn
g0om
guian
grom...in
geom
geot
Top View
Angled View
Qualitative results of generalization to different number of objects and unseen geometries for Top and Angled View observations. With zero-shot generalization, our model assigns keypoint the objects and performs forward modeling in unseen scenarios. The green arrows depict the optimal action found with GraphMPC at each timestep. See Appendix Cor detailed results.
Figure 3:
Table Generalization results measured by the average distance the goal position for different number of objects and unseen object geometries in Top and Angled View images. Using a proba- bilistic graph representation significantly improves the generalization.
Distance Goal Position x10-3
KINet
KINet deterministic
Object
geom
geom gev
geom
geom gen
0.24+0.02
0.3110.01
0.20
0.34
Top View
2
0.22+t.01
0.58
0.262
0.65+t.00
0.18+00.3
0.21 00.01
0.19
0.28++0.08
0.5440.01 4
0.
0.68
0.89
0.86+0000 5
0.
0.94+ 0.6
2.01+0.14
0.21400.4
0.35
0.28440.01
0.36400.8
Angled View
0.21+
0.53
0.22+40005
0.5940.07
0.19+0.02
0.20=0.05
0.19:+0.08
0.31+0.08
0.51+0.02
0,65+0.07
0.57=0.11
0.96+0.09
0.8910.13
1.64
10.1
2.17110110
hope general approach inspires future research on physical reasoning in settings where ground- truth information is hard to obtain.
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abbecl. Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint 1rXiv:1606.000 2016.

---

Under review as a conference paper at ICLR 2022
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. arXiv preprint arXiv:1612.00022., 2016.
Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. arXiv preprint 1061::2006.07633, 2021a.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Yee Kenneth Wong, Joshua Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reason- ing. arXiv preprint arXiv:2 16564, 2021b.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp- 2786-2793. IEFE, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Radek Grzeszczuk, Demetri Terzopoulos, and Geoffrey Hinton. Neuroanimator: Fast neural net- work emulation and control of physics-based models. Proceedings of the 25th annual confer- ence on Computer graphics and interactive techniques, 9-20, 1998.
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncer- tainty regularization for driving in dense traffic. arXiv preprint 1rXiv:1901.0270:. 2019.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Con- ference on Neural Information Processing Systems, pp. 4020-4031, 2018.
Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint 1rXiv:1812, 10972, 2018
Diederik Kingma and Max Welling. arXiv:1312.611 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 1911.12247, 2019.
Thomas N Kipf and Max Welling. arXiv:1611.07308, 2016.
Variational graph auto-encoders. arXiv preprint
Tejas Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. arXiv preprint arXiv: 1906. 11883, 2019.
Yunzhu Li, Jiajun Wu, Jun- Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. Proceedings IEEE International Conference on Robotics and Automation, 2019-May:1205-12 9 2018. URL http:// http///xivv....//i. www./oorg//e abs wtwpvv/.rrg/bbs/800000 11169
Yunzhu Li, Jiajun Wu, Jun- Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In ICRA, 2019.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel M. Bear, Daniel L. K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. 37th Interna- tional Conference on Machine Learning. ICML 2020, PartF168 1 176-77888888 5892, 2020. URL http::/arxisw www./orrgggoooo..ooo abs /200s/2000 13664
10

---

Under review as a conference paper at ICLR 2022
R Chris Miall and Daniel M Wolpert. Forward models for physiological motor control. Neural networks, 9 1996.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. arXiv preprint arXiv: 0tt::/996.0077 07889, 2019.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physics prediction. arXiv preprini arXiv: 08047, 2018.
Derrick Nguyen and Bernard Widrow. The truck backer-upper: An example of self-learning in neural networks. In Advanced neural computers, pp. 11-19, Elsevier, 1990.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 2018.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Sanchez-Gonzaler and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.034000 2020.
Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. arXiv preprint arXiv:2008.02265. 2020.
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470-4479, PMLR, 2018.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. International conference on artificial neural networks, pp. 412-422. Springer, 2018.
Alcksandar Stanić, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Hicrarchical relational infer- ence. arXiv preprint arXiv:2010.03635, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30:4539-4547, 2017.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven 09275, 2019.
exploration. arXiv preprint
Daniel M Wolpert and Mitsuo Kawato. Multiple paired forward and inverse models for motor control. Neural networks, 11(7-8):111--133) 1998.
Daniel M Wolpert, Zoubin Ghahramani, and Michael Jordan. An internal model for sensorimotor 1995.
integration. Science, (69
Daniel M Wolpert, Chris Miall, and Mitsuo Kawato. Internal models the cerebellum. Trends in cognitive sciences, (3))338-347,, 1998.
Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional networks. arXiv preprint arXiv:1607.02558 2016.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive represen- tations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003.05433 2020.
11

---

Under review as a conference paper at ICLR 2022
Yuan Yao, Yasamin Jafarian, and Hyun Soo Park. Monet: Multiview semi-supervised keypoint detection via epipolar divergence. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 753-762, 2019.
Yufei Ye, Dhiraj Gandhi, Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. In Conference on Robot Learning, pp. 100-109, PMLR, 2020.
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised dis- covery of object landmarks as structural representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2694-2703, 2018.
12

---

Under review as a conference paper at ICLR 2022
A DATA AND MUJUCO ENVIRONMENT
To generate simulated dataset we represent each object with two cuboid-shaped geoms. We initialize each simulation episode by randomize the geom Size, color, and pose. We define the training geom dimensions as geom with width range in [0.02, 0.04] and a length range in [0.02, 0.06j. We also generate dataset with elongated geoms for generalization experiments with geomgen length range [0.06, 0.18]. All geom heights are fixed 0.03. We simulate random 1-step point-to-poinn pushing actions in range [0.01, 0.05] with random initialize position in proximity of the objects [0.01, 0.03] towards uniformly sampled directions.
geom
goom gen.
Figure 4: Example of objects for training and generalization. Each object a combination of two cuboid geoms with randomly sampled dimensions.
B LEARNED GRAPH REPRESENTATION
Here, we include examples the learned graph representation with our model. Figure 5 shows an example of the inferred probabilistic graph adjacency matrix. Although it is not trivial how the structure of the scene reflected in the graph connectivity, our generalization results shows that probabilistic graph representation enables the model better generalize to unseen geometries (see Table
Figure 5: Detected keypoints and the inferred graph connectivity. Note that here we are plotting the edges with probability of more than 0.5; however, the graph representation of the scene probabilistic fully-connected graph.
To demonstrate the effectiveness of our approach for learning meaningful object-centric representa- tion of the system we compute 2D t-SNE embeddings of the learned node features our model. In these examples we can see that the learned features of the nodes that belong to the same object form distinct clusters (Fig 6).
C DETAILED MPC RESULTS
Here, we include a more detailed qualitative results of the MPC planning steps using our model for Top View (Fig and Angled View (Fig(B) observations. Using our model in the downstream control task results in faster and more accurate planning.
13

---

Under review as a conference paper at ICLR 2022
204
Figure 6: Node features 2D I-SNE plots.
Jnit
4wi
4--
+
+
Figure 7: MPC results steps for Top View observations.
14

---

Under review as a conference paper at ICLR 2022
fnit
fgoal
Figure 8: MPC results steps for Angled View observations.
15


Reviews:
[{"review_id": "w6ewb9txIjE", "reviewer": "Reviewer_nqXG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-organized\n- The problem motivation in the introduction is clearly explained and analysis of related work is quite thorough\n- I appreciate that the experiment section includes not only experiments on reconstruction and future prediction, but also a simple control task. This is important to show the applicability of the proposed method.\n\nI am generally positive about this paper. However, the main weakness is the characterization of the method in section 3. While reading, I could not find the justification for some claims and/or could not match the textual description with the notation in the formulas. Here is a list of major points in order of appearance:\n- Section 3.2 (and following): how is the adjacency matrix defined? If no external information about the environment is used, then $\\[A_t\\]_ij=1\\ \\forall t,i,j$, i.e. the graph is assumed fully-connected. If this is the case, why focussing on reconstructing the graph structure (likelihood maximization in eq5). All adjacency matrices, both initial and reconstructed, will be full of ones anyway, right? It should be noted that this was not an issue in Kipf&Welling 2016, which the paper indicates as the base for KINet, since the VGAEs were applied to graphs with a non-trivial adjacency matrix.\n- KINet remains rather inflexible wrt the number of keypoints, even though the paper tries to present it as more flexible and generalizable than other baselines and related works:\n  - Section 3.2: \"Note that the graph representation is built on keypoints coordinates and, hence, we do not impose any prior assumption on the number of objects in the system.\" but actually the number of keypoints is fixed by the number of feature maps K in $f_{kp}$.\n  - Section 5.3: \"baseline models are not able to generalize to a different number of objects because this number is hard-coded in their formulation (i.e, in the number of feature channels or the number of graph nodes)\". Actually, KINet uses a constant K in this exact way, the only difference is that K is set to 6 which is greater than the max number of objects used in the experiments.\n- The method is strongly related to Kipf&Welling 2016, as mentioned often in the paper, but at the same time there are strong inconsistencies. For example, why is the node-level prior (eq 1) conditioned on $G_t$? In Kipf&Welling the prior is a zero-centered unit-variance normal distribution that does not depend on the graph. The same definition of prior, including the dubious dependency on the graph, is repeated in section 3.5.\n- Also related to the point above, eq2 uses the graph at the next timestep $G_{t+1}$ to obtain $Z_t$. Since the training is based on predicting the next graph based on the current, how is this choice justified? Why doesn't it interfere with training? Also, if eq2 is used during training as mentioned in the text above the equation, what is the behavior during inference when the future is not accessible?\n- The neighborhood aggregation mechanism that appears in eq4 is portrayed as a main contribution of the paper (see introduction). However, I could find neither a definition nor an ablation study of such a component. The only vague explanation is at the end of section 3.3: \"Our model, however, learns to dynamically sample the neighborhood of each node at each timestep conditioned on the latent variable\". To be considered a main contribution, I would expect a more detailed description of what $N(k)$ means and how it affects the model.\n- Since the method focuses strongly on defining a graph representation for the keypoints it's weird that in the experiment section this graph structure is not inspected at all. Considering that a key component of the method is inferring the adjacency matrix $\\tilde{A}_t$, it would be good to analyze the learned matrices, characterize them in terms of sparsity, and verify whether they capture semantic information about the environment.\n- How are negative samples $G^-$ for the contrastive loss (eq7) defined? On the same topic, how is the claim \"The contrastive loss is an essential element in our approach to ensure the learned forward model is action conditional\" in 5.4 justified? How does the contrastive loss relate to the model being action conditional?\n- Also on the contrastive loss, can S be considered a graph matching algorithm? First, the algorithm assumes the same number and order of nodes between the two graphs. Isn't it possible that the encoder network outputs two isomorphic graphs that are identical expect in the order of the nodes? It would be enough for the feature maps $\\phi_k$ to specialize in specific regions of the input image. Also, the graph matching algorithm S completely disregards the adjacency matrices. The two graphs could have similar node embeddings but very different connectivity and they would be considered similar. This further shows how the graph representation and the probabilistic inference of the adjacency matrix might not be a key component of the method.\n\nOther weaknesses:\n- Using L2 distance as a reconstruction loss in image space will not scale well to higher-dimensional domains such as natural images. Of course, this paper doesn't have to solve every problem with representation learning, but since this limits the applicability of the method it should at least be mentioned.\n\nOther questions:\n- Why does the keypoint detector take as input the pair $(I_0, I_t)$ as opposed to $(I_{t-1}, I_t)$? What is the advantage in using the initial state of the system? Since the simulated tasks start from random configurations with no inherent initial state, how does the choice of $I_0$ affect the detection of keypoints, the reconstruction, and the forward model? How far in the future can the forward model run?\n- Is the dataset of 10K episodes described in 4.1 available for reproducibility?\n- Is the test set of 1000 control tasks used in 5.2 released?\n- What is the size (e.g. side length) of the simulation used in the experiments? This is important to understand if the mean position errors reported in the tables are small or large in absolute terms.\n\nMinor points:\n- Figure 1: repeated word \"are are\"\n- Section 2: \"relate work\" -> \"related work\"\n- Figure 2: x axis labels \"timetep\" -> \"timestep\", also improve font size\n- Section 5.2: is \"converging\" the right word here? Convergence sounds like a model being trained, but my understanding is that the MPC algorithm uses the learned forward model to predict the outcome of future actions and take one step. I suggest replacing with \"reaching\" that gives the idea of achieving the desired state following a trajectory.\n- Section 5.3: \"i.e. in the number of feature channels or the number of graph nodes\" should be \"e.g.\"\n- Figure 3: the reader might benefit from a little more details in caption. What does each row represent? What are the dots and arrows in the images?", "summary_of_the_paper": "The authors introduce Keypoint Interaction Network, an unsupervised framework for keypoint-based representation learning and forward modelling.\nThis framework is accompanied by auxiliary algorithms for training and quantitative experiments for evaluation.", "summary_of_the_review": "The paper gives much importance to a graph-based representation of interacting systems. However, the proposed formulation doesn't seem to leverage the graph in full. Actually, it might even be possible to describe the entire method without discussing nodes and adjacency matrices.\n\nSome elements of the method are not convincing, e.g. prior function, graph matching, reconstruction loss. Other components, e.g. the contrastive loss, are under-specified. Some claims are not well supported: flexibility of KINet wrt the parameter K and effect of the contrastive loss.\n\nOverall, the method of the paper sounds promising, but I don't recommend it for acceptance in its present form. I suggest polishing the method, re-evaluating the role of the graph representation, revising some vague claims, clarifying some definitions and formulas, and incorporating more experiments and ablation studies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vXG3_v-WfcH", "reviewer": "Reviewer_58w4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThis work propose to achieve multi-object position prediction with only using visual information like images without ground truth positions as supervision.\n\nWeakness:\n1 In general, this work is a patchwork of a condition-based generative model and a raw image preprocessing method at the input and output ends, leading to insufficient novelty. \n   1) For key point detection, Jakab et al. (2018) thoughts are followed.\n   2) The initial graph Gt with its nodes and edges is built through widely used operations, e.g., visual+position; distance\n   3) Why the approximate posterior is conditioned on the graph at next step? There is no interpretation, but this design is seem as some condition-VAE-based multi-agent prediction methods [1,2], while the only difference is that existing works infer the agents' dynamics but KINet infers the graph representation.\n   4) The message passing is not novel.\n   5) The forward prediction with a skip connection is not novel.\n\n2 Why do you use the contrastive loss? The interpretation is not clear; why it helps to learn actionable object-centric representation? How to build the negative G-?\n\n3 As for the experiments, I do not think the dataset matches the authors' claim: hard to annotation/detection and complex scenes.\n   1) The objects and scenes are quite simple. I suggest to test the methods on some real-world dataset like SDD and ETH-UCY, which have top-view image and scene information like obstacles.\n   2) In the proposed dataset, I notice that there are very large `+' object, but the method only determines a few key points for each object. So how to simulate the positional constraints imposed by the shape of an object？\n\n\n\n[1] It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction (ECCV2020)\n[2] Contextually Plausible and Diverse 3D Human Motion Prediction (ICCV 2021)", "summary_of_the_paper": "This paper introduces KINet (Keypoint Interaction Network)---an end-to-end unsupervised framework to reason about object interactions in complex systems based on a keypoint representation. There are three major steps in this model: extracting keypoint coordinates, inferring a probabilistic graph representation of the system, and estimating the next state of the system conditioned on the action.", "summary_of_the_review": "This work propose to achieve multi-object position prediction with only using visual information like images without ground truth positions as supervision. But the detailed designs are not sufficiently novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A54-tVW5ER4", "reviewer": "Reviewer_QJbD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles a crucial problem of learning the keypoints and their interactions from visual observations without supervision. The proposed method also has some novelty, such as learning probabilistic graphs and using contrastive loss to train the forward dynamics model. However, I have some concerns listed below.\n\n- The paper misses some important related work, making its contribution less clear. For example, OP3[1] and V-CDN[2] predict object masks / keypoints and use graph nets to model dynamics, also without supervision. Moreover, both methods can generalize to unseen number of objects, contradicting the claim in the paper that \"factorizing the scene into object instances limits the generalization of forward models to scenarios with a different number of objects\". I think these two methods should serve as baselines for this paper.\n- The dataset used in this paper seems a bit simplistic. Although the paper claims that \"In our work, we consider experimenting with more complex three-dimensional objects where the objects are randomized and replicate real-world object manipulation\", the actual dataset is \"obtained using an overhead camera\", making the observations essentially in 2D. In comparison, the dataset in [1] seems more challenging.\n- The learned keypoints in Figure 3 are mostly around object centers. This behavior is quite different from [3], where the keypoints can capture object morphology. This can reduce the usefulness of the learned keypoints.\n- The paper proposes to infer probabilistic graph representations. However, the pipeline seems broken. For example, in Equation 5, the node-level variables $Z$ are learned by reconstructing the adjacency matrix $A$, but $A$ is unknown and also needs to be inferred. How do you learn to infer $A$? And is this loss still a valid lower bound?\n- Also the probabilistic approach did not show much improvement over the deterministic approach (see Table 1)\n- The paper claims that the proposed model can generalize to unseen number of objects. However, if we compare Table 1 and 2, the error for generalization is an order of magnitude larger ($\\times 10^{-2}$ in Table 2 vs $\\times 10^{-3}$ in Table 1). I do not think this result supports the claim.\n\n**MINOR COMMENTS**\n\n- When you compute the prediction error for keypoints, how do you obtain the groundtruth keypoint position?\n- In GraphMPC, how do you obtain the goal state graph configuration?\n\n[1] Entity Abstraction in Visual Model-Based Reinforcement Learning. Veerapaneni et al., CoRL 2019.\n\n[2] Causal Discovery in Physical Systems from Videos. Li et al., NeurIPS 2020.\n\n[3] Unsupervised Learning of Object Keypoints for Perception and Control. Kulkarni et al., NeurIPS 2019.", "summary_of_the_paper": "The paper proposes a keypoint-based forward dynamics model for generalization to unseen number of objects. The model employs unsupervised keypoint extraction methods from the literature, infers probabilistic graph representations over the keypoints, and makes forward predictions by message passing. The model is evaluated on multi-object manipulation tasks. It is shown that the model prediction error is close to that of a graph net with access to groundtruth object positions. The model is also shown to converge to the goal configuration faster than the baselines.", "summary_of_the_review": "I recommend reject, because several claims are wrong or not well supported, and the contribution over related work is not sufficiently discussed or demonstrated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7t5g70Aq_V", "reviewer": "Reviewer_YtC3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**1. Strengths**\n\n(1). The idea of combining unsupervised keypoint detection and forward modeling seems to be new and has not been done before.\n\n(2). The idea is simple and shows its effectiveness to some previous baselines.\n\n**2. Concerns**\n\nThe reviewer does have some concerns about the model.\n\n(1). The claim on the previous method is not accurate enough that \"Relying on object detection and segmentation tools, on the other hand, makes the forward model fragile and dependent on the flawless performance of these tools. More often than not, pretrained object detection or segmentation models suffer from poor generalization to unseen objects\". Models based-on detection or segmentation proposals have not been compared in the experimental sections. Also, recent methods like MoNet[A] have been showing object proposals can be obtained from unsupervised learning. It will be interesting and necessary to compare the models based on keypoint detection and object proposals like combining the MoNet and PropNet for forward modeling.\n\n(2). The evaluation is limited on a newly-built dataset and it will be more convincing to compare the proposed methods on the previous dataset like CLEVRER, which offers more baselines for future prediction.\n\n(3). The details of the newly-built dataset is missing like the statistics of the dataset and what kind of objects it contains. It will be hard for readers to catch up how challenging the dataset is and how effective the model is.\n\n(4). From the examples in Figure 3, the dataset seems to be simple. It contains simple objects from the top-down view.  whether the dataset contains occlusion cases? Is the camera stable? Much details about the dataset is needed.\n\n[A]. Burgess C P, Matthey L, Watters N, et al. Monet: Unsupervised scene decomposition and representation[J]. arXiv preprint arXiv:1901.11390, 2019.\n", "summary_of_the_paper": "**The main idea of the paper**\n\nThis paper combines two powerful ideas, 1. unsupervised keypoint extraction for objects and 2. object-centric representation for forward modeling. KINET first detects objects' key points in the video in an unsupervised manner and extract object-centric representation for future prediction. Experiments are conducted in a new dataset built on MuJoCo and the performance in table 1 shows the proposed framework achieves better performance than the baseline methods.", "summary_of_the_review": "Since a bunch of significant baselines are absent and the details of the dataset are missing, the reviewer thinks the paper in the current version is not suitable for publishing on ICLR. The reviewer does think that a revised version will make the paper much more appealing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received four high-quality reviews. After the discussion period, all reviewers agreed that this submission is not strong enough to be accepted. Concerns include the novelty of the proposed method wrt related work and the limited experiments. The AC agrees. The AC also finds it disappointing that the authors didn't address the concerns on novelty or even discuss the related papers suggested by the reviewers in the revision. 

The recommendation is reject.
Decision: Reject