Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a keypoint-based forward dynamics model for generalization to unseen number of objects. The model employs unsupervised keypoint extraction methods from the literature, infers probabilistic graph representations over the keypoints, and makes forward predictions by message passing. The model is evaluated on multi-object manipulation tasks. It is shown that the model prediction error is close to that of a graph net with access to groundtruth object positions. The model is also shown to converge to the goal configuration faster than the baselines.
main_review: The paper tackles a crucial problem of learning the keypoints and their interactions from visual observations without supervision. The proposed method also has some novelty, such as learning probabilistic graphs and using contrastive loss to train the forward dynamics model. However, I have some concerns listed below.

- The paper misses some important related work, making its contribution less clear. For example, OP3[1] and V-CDN[2] predict object masks / keypoints and use graph nets to model dynamics, also without supervision. Moreover, both methods can generalize to unseen number of objects, contradicting the claim in the paper that "factorizing the scene into object instances limits the generalization of forward models to scenarios with a different number of objects". I think these two methods should serve as baselines for this paper.
- The dataset used in this paper seems a bit simplistic. Although the paper claims that "In our work, we consider experimenting with more complex three-dimensional objects where the objects are randomized and replicate real-world object manipulation", the actual dataset is "obtained using an overhead camera", making the observations essentially in 2D. In comparison, the dataset in [1] seems more challenging.
- The learned keypoints in Figure 3 are mostly around object centers. This behavior is quite different from [3], where the keypoints can capture object morphology. This can reduce the usefulness of the learned keypoints.
- The paper proposes to infer probabilistic graph representations. However, the pipeline seems broken. For example, in Equation 5, the node-level variables $Z$ are learned by reconstructing the adjacency matrix $A$, but $A$ is unknown and also needs to be inferred. How do you learn to infer $A$? And is this loss still a valid lower bound?
- Also the probabilistic approach did not show much improvement over the deterministic approach (see Table 1)
- The paper claims that the proposed model can generalize to unseen number of objects. However, if we compare Table 1 and 2, the error for generalization is an order of magnitude larger ($\times 10^{-2}$ in Table 2 vs $\times 10^{-3}$ in Table 1). I do not think this result supports the claim.

**MINOR COMMENTS**

- When you compute the prediction error for keypoints, how do you obtain the groundtruth keypoint position?
- In GraphMPC, how do you obtain the goal state graph configuration?

[1] Entity Abstraction in Visual Model-Based Reinforcement Learning. Veerapaneni et al., CoRL 2019.

[2] Causal Discovery in Physical Systems from Videos. Li et al., NeurIPS 2020.

[3] Unsupervised Learning of Object Keypoints for Perception and Control. Kulkarni et al., NeurIPS 2019.
summary_of_the_review: I recommend reject, because several claims are wrong or not well supported, and the contribution over related work is not sufficiently discussed or demonstrated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: **The main idea of the paper**

This paper combines two powerful ideas, 1. unsupervised keypoint extraction for objects and 2. object-centric representation for forward modeling. KINET first detects objects' key points in the video in an unsupervised manner and extract object-centric representation for future prediction. Experiments are conducted in a new dataset built on MuJoCo and the performance in table 1 shows the proposed framework achieves better performance than the baseline methods.
main_review: **1. Strengths**

(1). The idea of combining unsupervised keypoint detection and forward modeling seems to be new and has not been done before.

(2). The idea is simple and shows its effectiveness to some previous baselines.

**2. Concerns**

The reviewer does have some concerns about the model.

(1). The claim on the previous method is not accurate enough that "Relying on object detection and segmentation tools, on the other hand, makes the forward model fragile and dependent on the flawless performance of these tools. More often than not, pretrained object detection or segmentation models suffer from poor generalization to unseen objects". Models based-on detection or segmentation proposals have not been compared in the experimental sections. Also, recent methods like MoNet[A] have been showing object proposals can be obtained from unsupervised learning. It will be interesting and necessary to compare the models based on keypoint detection and object proposals like combining the MoNet and PropNet for forward modeling.

(2). The evaluation is limited on a newly-built dataset and it will be more convincing to compare the proposed methods on the previous dataset like CLEVRER, which offers more baselines for future prediction.

(3). The details of the newly-built dataset is missing like the statistics of the dataset and what kind of objects it contains. It will be hard for readers to catch up how challenging the dataset is and how effective the model is.

(4). From the examples in Figure 3, the dataset seems to be simple. It contains simple objects from the top-down view.  whether the dataset contains occlusion cases? Is the camera stable? Much details about the dataset is needed.

[A]. Burgess C P, Matthey L, Watters N, et al. Monet: Unsupervised scene decomposition and representation[J]. arXiv preprint arXiv:1901.11390, 2019.

summary_of_the_review: Since a bunch of significant baselines are absent and the details of the dataset are missing, the reviewer thinks the paper in the current version is not suitable for publishing on ICLR. The reviewer does think that a revised version will make the paper much more appealing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce Keypoint Interaction Network, an unsupervised framework for keypoint-based representation learning and forward modelling.
This framework is accompanied by auxiliary algorithms for training and quantitative experiments for evaluation.
main_review: Strengths:
- The paper is well-organized
- The problem motivation in the introduction is clearly explained and analysis of related work is quite thorough
- I appreciate that the experiment section includes not only experiments on reconstruction and future prediction, but also a simple control task. This is important to show the applicability of the proposed method.

I am generally positive about this paper. However, the main weakness is the characterization of the method in section 3. While reading, I could not find the justification for some claims and/or could not match the textual description with the notation in the formulas. Here is a list of major points in order of appearance:
- Section 3.2 (and following): how is the adjacency matrix defined? If no external information about the environment is used, then $\[A_t\]_ij=1\ \forall t,i,j$, i.e. the graph is assumed fully-connected. If this is the case, why focussing on reconstructing the graph structure (likelihood maximization in eq5). All adjacency matrices, both initial and reconstructed, will be full of ones anyway, right? It should be noted that this was not an issue in Kipf&Welling 2016, which the paper indicates as the base for KINet, since the VGAEs were applied to graphs with a non-trivial adjacency matrix.
- KINet remains rather inflexible wrt the number of keypoints, even though the paper tries to present it as more flexible and generalizable than other baselines and related works:
  - Section 3.2: "Note that the graph representation is built on keypoints coordinates and, hence, we do not impose any prior assumption on the number of objects in the system." but actually the number of keypoints is fixed by the number of feature maps K in $f_{kp}$.
  - Section 5.3: "baseline models are not able to generalize to a different number of objects because this number is hard-coded in their formulation (i.e, in the number of feature channels or the number of graph nodes)". Actually, KINet uses a constant K in this exact way, the only difference is that K is set to 6 which is greater than the max number of objects used in the experiments.
- The method is strongly related to Kipf&Welling 2016, as mentioned often in the paper, but at the same time there are strong inconsistencies. For example, why is the node-level prior (eq 1) conditioned on $G_t$? In Kipf&Welling the prior is a zero-centered unit-variance normal distribution that does not depend on the graph. The same definition of prior, including the dubious dependency on the graph, is repeated in section 3.5.
- Also related to the point above, eq2 uses the graph at the next timestep $G_{t+1}$ to obtain $Z_t$. Since the training is based on predicting the next graph based on the current, how is this choice justified? Why doesn't it interfere with training? Also, if eq2 is used during training as mentioned in the text above the equation, what is the behavior during inference when the future is not accessible?
- The neighborhood aggregation mechanism that appears in eq4 is portrayed as a main contribution of the paper (see introduction). However, I could find neither a definition nor an ablation study of such a component. The only vague explanation is at the end of section 3.3: "Our model, however, learns to dynamically sample the neighborhood of each node at each timestep conditioned on the latent variable". To be considered a main contribution, I would expect a more detailed description of what $N(k)$ means and how it affects the model.
- Since the method focuses strongly on defining a graph representation for the keypoints it's weird that in the experiment section this graph structure is not inspected at all. Considering that a key component of the method is inferring the adjacency matrix $\tilde{A}_t$, it would be good to analyze the learned matrices, characterize them in terms of sparsity, and verify whether they capture semantic information about the environment.
- How are negative samples $G^-$ for the contrastive loss (eq7) defined? On the same topic, how is the claim "The contrastive loss is an essential element in our approach to ensure the learned forward model is action conditional" in 5.4 justified? How does the contrastive loss relate to the model being action conditional?
- Also on the contrastive loss, can S be considered a graph matching algorithm? First, the algorithm assumes the same number and order of nodes between the two graphs. Isn't it possible that the encoder network outputs two isomorphic graphs that are identical expect in the order of the nodes? It would be enough for the feature maps $\phi_k$ to specialize in specific regions of the input image. Also, the graph matching algorithm S completely disregards the adjacency matrices. The two graphs could have similar node embeddings but very different connectivity and they would be considered similar. This further shows how the graph representation and the probabilistic inference of the adjacency matrix might not be a key component of the method.

Other weaknesses:
- Using L2 distance as a reconstruction loss in image space will not scale well to higher-dimensional domains such as natural images. Of course, this paper doesn't have to solve every problem with representation learning, but since this limits the applicability of the method it should at least be mentioned.

Other questions:
- Why does the keypoint detector take as input the pair $(I_0, I_t)$ as opposed to $(I_{t-1}, I_t)$? What is the advantage in using the initial state of the system? Since the simulated tasks start from random configurations with no inherent initial state, how does the choice of $I_0$ affect the detection of keypoints, the reconstruction, and the forward model? How far in the future can the forward model run?
- Is the dataset of 10K episodes described in 4.1 available for reproducibility?
- Is the test set of 1000 control tasks used in 5.2 released?
- What is the size (e.g. side length) of the simulation used in the experiments? This is important to understand if the mean position errors reported in the tables are small or large in absolute terms.

Minor points:
- Figure 1: repeated word "are are"
- Section 2: "relate work" -> "related work"
- Figure 2: x axis labels "timetep" -> "timestep", also improve font size
- Section 5.2: is "converging" the right word here? Convergence sounds like a model being trained, but my understanding is that the MPC algorithm uses the learned forward model to predict the outcome of future actions and take one step. I suggest replacing with "reaching" that gives the idea of achieving the desired state following a trajectory.
- Section 5.3: "i.e. in the number of feature channels or the number of graph nodes" should be "e.g."
- Figure 3: the reader might benefit from a little more details in caption. What does each row represent? What are the dots and arrows in the images?
summary_of_the_review: The paper gives much importance to a graph-based representation of interacting systems. However, the proposed formulation doesn't seem to leverage the graph in full. Actually, it might even be possible to describe the entire method without discussing nodes and adjacency matrices.

Some elements of the method are not convincing, e.g. prior function, graph matching, reconstruction loss. Other components, e.g. the contrastive loss, are under-specified. Some claims are not well supported: flexibility of KINet wrt the parameter K and effect of the contrastive loss.

Overall, the method of the paper sounds promising, but I don't recommend it for acceptance in its present form. I suggest polishing the method, re-evaluating the role of the graph representation, revising some vague claims, clarifying some definitions and formulas, and incorporating more experiments and ablation studies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces KINet (Keypoint Interaction Network)---an end-to-end unsupervised framework to reason about object interactions in complex systems based on a keypoint representation. There are three major steps in this model: extracting keypoint coordinates, inferring a probabilistic graph representation of the system, and estimating the next state of the system conditioned on the action.
main_review: Strength:
This work propose to achieve multi-object position prediction with only using visual information like images without ground truth positions as supervision.

Weakness:
1 In general, this work is a patchwork of a condition-based generative model and a raw image preprocessing method at the input and output ends, leading to insufficient novelty. 
   1) For key point detection, Jakab et al. (2018) thoughts are followed.
   2) The initial graph Gt with its nodes and edges is built through widely used operations, e.g., visual+position; distance
   3) Why the approximate posterior is conditioned on the graph at next step? There is no interpretation, but this design is seem as some condition-VAE-based multi-agent prediction methods [1,2], while the only difference is that existing works infer the agents' dynamics but KINet infers the graph representation.
   4) The message passing is not novel.
   5) The forward prediction with a skip connection is not novel.

2 Why do you use the contrastive loss? The interpretation is not clear; why it helps to learn actionable object-centric representation? How to build the negative G-?

3 As for the experiments, I do not think the dataset matches the authors' claim: hard to annotation/detection and complex scenes.
   1) The objects and scenes are quite simple. I suggest to test the methods on some real-world dataset like SDD and ETH-UCY, which have top-view image and scene information like obstacles.
   2) In the proposed dataset, I notice that there are very large `+' object, but the method only determines a few key points for each object. So how to simulate the positional constraints imposed by the shape of an object？



[1] It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction (ECCV2020)
[2] Contextually Plausible and Diverse 3D Human Motion Prediction (ICCV 2021)
summary_of_the_review: This work propose to achieve multi-object position prediction with only using visual information like images without ground truth positions as supervision. But the detailed designs are not sufficiently novel.
review_title: null

Meta-review: This submission received four high-quality reviews. After the discussion period, all reviewers agreed that this submission is not strong enough to be accepted. Concerns include the novelty of the proposed method wrt related work and the limited experiments. The AC agrees. The AC also finds it disappointing that the authors didn't address the concerns on novelty or even discuss the related papers suggested by the reviewers in the revision. 

The recommendation is reject.
Decision: Reject