Title: Discovering objects and their relations from entangled scene representations
Authors: David Raposo
Adam Santoro
David Barrett
Razvan Pascanu
Timothy Lillicrap
Peter Battaglia
Venue: ICLR 2017

Abstract: Our world can be succinctly and compactly described as structured scenes of objects and relations. A typical room, for example, contains salient objects such as tables, chairs and books, and these objects typically relate to each other by virtue of their correlated features, such as position, function and shape. Humans exploit knowledge of objects and their relations for learning a wide spectrum of tasks, and more generally when learning the structure underlying observed data. In this work, we introduce relation networks (RNs) - a general purpose neural network architecture for object-relation reasoning. We show that RNs are capable of learning object relations from scene description data. Furthermore, we show that RNs can act as a bottleneck that induces the factorization of objects from entangled scene description inputs, and from distributed deep representations of scene images provided by a variational autoencoder. The model can also be used in conjunction with differentiable memory mechanisms for implicit relation discovery in one-shot learning tasks. Our results suggest that relation networks are a powerful architecture for solving a variety of problems that require object relation reasoning.


Review discussion:
Are there some examples of the images used in Sec. 5.2.2?
We agree that it is helpful to include example images as used in section 5.2.2 and we have now updated the paper to include some examples (Figure 11).  Thank you for your suggestion and for your review.
1. Why do the scene descriptions have 16 rows? Sec. 2.1 states that the scene description is a m x n matrix with each row corresponding to an object and n properties of the objects.
On the other hand sec. 3.1 says that each description is a 16 row matrix 4 rows for each object type. Could you clarify the actual structural representation of the scene description matrix? Were all scenes constrained to have 16 objects (4 from each type)?
2. Why is BU a matrix of size 16 x 3 in sec. 5.2.1 (page 8), when B is a matrix of size mn x mn and U is mn x mn? Can you explain the inset figure in Fig. 5a in more detail?
3. The setup for the final category of tasks needs to be explained better. Perhaps a pictorial example would assist a lot.
Also, did this task operate on images or scene description matrices?
4. The experiments state that the baseline MLP used same size architecture as the RN. What does this mean in practice?
For instance, while comparing a {200, 200} RN with an MLP, did the MLP just have two hidden layers with 200 units each? If that is the case, the RN with object pair prior runs an MLP on each pair of objects (m^2 {200, 200} MLP) before summing and passing it to f_{\phi}. Is the number of computations consistent between such an RN and the corresponding baseline {200, 200} MLP?
5. The object-pair prior RN is stated to operate on every pair of object rows.
However, in the case of VAE features there is no guarantee that the feature is factored in terms of objects.
So how does the RN identify object pairs in this case?
6. Have you considered running experiments with  real-life image datasets with labelled object relations such as Visual Genome and HICO (Chao et al. ICCV'15)?
We thank the reviewer for their astute points of clarification and correction. We have modified the paper to address their points, and include specific details about each question below.
Q1. The scene description has 16 rows in accordance with the 16 objects in the scene (one row per object). There are 4 object types (e.g., red square, blue triangle, orange circle, yellow square), and 4 objects per type (e.g., a red square positioned at (x1, y1), a red square positioned at (x2, y2), etc.) for a total of 4*4=16 objects.
All scenes consisted of 16 objects. In principle, this was not a constraint.
Indeed we have done experiments using different numbers of objects and object types, and we find that the results remain consistent with the results presented in our paper.
Q2. BU is indeed a matrix of size mn by mn, and the text has been clarified and corrected. D (reshaped as a vector of concatenated objects) multiplied by BU is a vector of size mn. This is then reshaped back into a matrix of size m by n and provided as input to the RN. The inset in Fig. 5a, then, is correctly depicting BU.
Q3. We have added a substantial section in the appendix outlining the one-shot learning task, and have included two new figures.
The task operated on scene description matrices.
Q4. The RN with object pair prior indeed runs an MLP on each object pair -- but, crucially, it is the same MLP for each object pair (i.e., the weights are shared).
So, a {200,200} RN is comparable to a {200,200} MLP. Note: this means that the quoted “{m^2 {200, 200}}” size from the question is incorrect, as the RN MLP size is independent of the number of objects (which is, notably, one of its strengths).
We note that the goal of training models of multiple sizes was not entirely for the purpose of matching the network capacities; we also wanted to test a range of network sizes to probe for limitations, overfitting, etc. Of particular note, even the largest MLP, which had orders of magnitude more parameters than the smallest RN, was unable to perform the task (see for example, Figure 4).
Q5. This is exactly correct -- although VAEs tend to produce decorrelated latent codes, there is no guarantee that the codes are factored in terms of objects. The key to this experiment is that the RN, which needs to act on factorized object representations, can use this constraint to drive learning in a linear layer to disentangle the VAE representation.
In other words, a potentially entangled VAE representation can be disentangled into factorized object representations because of the presence of the RN.
Practically, the VAE representation is reshaped into a 16x16 matrix (which is arbitrary -- other sizes are fine), and sent through a linear layer. The linear layer output is reshaped to a matrix. The RN treats the rows of this matrix as if they were objects, even though, as you state, there is no guarantee that the rows constitute factorized objects. This implicit assumption of factorized object representations, though, acts to actually induce factorized object representations in the RN input (Figure 5a, inset).
The ability of the RN to induce factorized representations from entangled, or non-factorized representations is perhaps one of the most intriguing features of RNs.
Q6. This is a great suggestion, and we hope to work on datasets like these in the future. Given the results of this paper, we believe that the RN may be a powerful module for relation reasoning, and we see no reason why it could not be combined with powerful visual processing, NLP, and attention-based models/components, as would be required for modelling real-life datasets such as the Visual Genome and HICO datasets. In this work, we focused on developing the conceptual and architectural foundations necessary for reasoning about object relationships. Our results on the synthetic data give us sufficient reason to believe that tackling more demanding datasets should indeed be possible;
but, it would unfortunately require substantial, non-trivial insights and engineering to the perceptual components of the model. We have now included some statements in our conclusion pointing to and citing these datasets as future work.
We thank the reviewer again for their comments.
This paper proposes a relation network (RN) to model relations between input entities such as objects.  The relation network is built in two stages.
First a lower-level structure analyzes a pair of input entities.  All pairs of input entities are fed to this structure.
Next, the output of this lower-level structure is aggregated across all input pairs via a simple sum.  This is used as the input to a higher-level structure.  In the basic version, these two structures are each multi-layer perceptrons (MLPs).
Overall, this is an interesting approach to understanding relations among entities.  The core idea is clear and well-motivated -- pooling techniques that induce invariance can be used to learn relations.  The idea builds on pooling structures (e.g. spatial/temporal average/max pooling) to focus on pairwise relations.  The current pairwise approach could potentially be extended to higher-order interactions, modulo scaling issues.
Experiments on scene descriptions and images verify the efficacy of relation networks.  The MLP baselines used are incapable of modeling the structured dependencies present in these tasks.  It would be interesting to know if pooling operators (e.g. across-object max pooling in an MLP) or data augmentation via permutation would be effective for training MLPs at these tasks.
Regardless, the model proposed here is novel and effective at handling relations and shows promise for higher-level reasoning tasks.
We’ve uploaded a new version of the paper that addresses much of the reviewers’ comments and questions.
Additionally, we included a new experiment that sheds light on the modelling power of RNs.
Specifically, we tested the RN on withheld, never-before-seen scene classes, and showed that it can successfully classify scenes from these classes. These results suggest that the RN is learning something compositional about the scenes (e.g., the existence of specific object relations).
Thus, RNs can infer object relations, and combine them in unique ways to predict scene class membership. They are therefore able to generalize in a combinatorially complex object-relation space, which is a hallmark feature of compositional learning.
This paper proposes relation networks in order to model the pairwise interactions between objects in a visual scene.
The model is very straight forward, first an MLP (with shared weights) is applied to each pair of objects.
Finally a prediction is created by an MLP which operates by summing non-linear functions of these pairs of objects.
Experimental evaluation is done in a synthetic dataset that is generated to fit the architecture hand-crafted in this paper.
The title of the paper claims much more than the paper delivers. Discovering objects and their relations is a very important task.
However, this paper does not discover objects or their relations, instead, each objects is represented with hand coded ground truth attributes, and only a small set of trivial relationships are "discovered", e.g., relative position.
Discovering objects and their relationships has been tackled for several decades in computer vision (CV). The paper does not cite or compare to any technique in this body of literature. This is typically refer to as "contextual models".
Can the proposed architecture help object detection and/or scene classification? would it work in the presence of noise (e.g, missing detections, non accurate detection estimates, complex texture)? would it work when the attributes of objects are estimated from real images?
I'll be more convinced if experiments where done in real scenes. In the case of indoor scenes, datasets such as NYUv2, Sun-RGB-D, SceneNN, Chen et al CVPR 14 (text-to-image-correference) could be used. In outdoor scenes, KITTI and the relationships between cars, pedestrians and cyclist could also serve as benchmark.
Without showing real scenes, this paper tackles a too toy problem with a very simple model which does not go much further than current context models, which model pairwise relationships between objects (with MRFs, with deep nets, etc).
The reviewer has fundamentally miscategorised our paper as a contribution to computer vision, whereas in fact, the focus of our paper is reasoning about the relations between elements in a set.
Indeed, the only time the word “visual” appears in our paper is when we state that our paper is not about computer vision: “Although the term scene description alludes to visual information, this need not be the case; scenes can be entirely abstract, as can the objects that constitute the scene, and the features that define the objects”.
In addition to this high-level misunderstanding, the reviewer missed many of the technical details of the experiments and their contributions to the literature on learned representations and structure reasoning.
1) The terms "scene," "object," "relation," etc., are general concepts, and are not restricted to the subdomain of vision.
For example, objects could be planets, and relations could be gravitational attraction. Or, objects could be DNA sequences, and relations can be their sequence alignment. Reasoning about objects and their relations in this realm stands as sufficiently distinct from the problems of image understanding, segmentation, and classification as classically defined in computer vision, and the techniques used therein. We direct the reviewer to the rich literature that deals with structured data, and reasoning about entities and their relations.
Although this data can be construed as "hand-crafted," the types of problems for which these data tend to be used -- such as structure prediction, and relational reasoning -- are entirely non-trivial.
2) We'd like to emphasize the contribution that this work makes towards learning permutation invariant representations from structured data, which is directly aligned with the central motive for this conference; i.e., learning representations.
3) The claim is made that we do not discover objects and their relations. We direct the reviewer to the experiments in the latter half of the paper that provide evidence for object and relation disentanglement from entangled distributed representations generated from pixels. It can be argued that the scenes used are too "simple." To this we respond that the purpose of the experiment was to show evidence for learned disentangling from distributed representations. Producing factorized object representations is non-trivial;
indeed, many papers submitted to ICLR solely tackle this problem. Once again, these representations need not be derived from images, but can theoretically be distributed representations from other domains as well.
4) The contributions of the one-shot learning and VAE-disentangling experiments -- which are perhaps the most important, and constitute half the experiments presented -- go unacknowledged. Online one-shot learning of relations is unprecedented, and producing factorized object representations from latent VAE representations has just begun to be explored.
N/A
+ Understanding relations between objects is an important task in domains like vision, language and robotics.
However, models trained on real-life datasets can often exploit simple object properties (not relation-based) to identify relations (eg: animals of bigger size are typically predators and small-size animals are preys). Such models can predict relations without necessarily understanding them. Given the difficulty of the task, a controlled setting is required to investigate if neural networks can be designed to actually understand pairwise object relations. The current paper takes a significant step in answering this question through a controlled dataset.
Also, multiple experiments are presented to validate the "relation learning" ability of proposed Relation Networks (RN).
+ The dataset proposed in the paper ensures that relation classification models can succeed only by learning the relations between objects and not by exploiting "predator-prey" like object properties.
+ The paper presents very thorough experiments to validate the claim that "RNs" truly learn the relation between objects.
1.
In particular, the ability of the RN to force a simple linear layer to disentangle scene description from VAE latent space and permuted description is very interesting. This clearly demonstrates that the RN learns object relations.
2. The one-shot experiments again demonstrate this ability in a convincing manner. This requires the model to understand relations in each run, represent them through an abstract label and assign the label to future samples from the relationship graph.
Some suggestions:
- Is g_{\psi}(.) permutation invariant as well. Since it works on pairs of objects, how did you ensure that the MLP is invariant to the order of the objects in the pair?
- The RNs need to operate over pairs of objects in order to identify pairwise interactions.
However, in practical applications there are more complicated group interactions. (eg. ternary interaction: "person" riding a "bike" wears "helmet"). Would this require g(.) of RN to not just operate on pairs but on every possible subset of objects in the scene? More generally, is such a pairwise edge-based approach scalable to larger number of objects?
- The authors mention that " a deep network with a sufficiently large number of parameters and a large enough training set should be capable of matching the performance of a RN". This is an interesting point, and could be true in practice. Have the authors investigated this effect by trying to identify the minimum model capacity and/or training examples required by a MLP to match the performance of RN for the provided setup? This would help in quantifying the significance of RN for practical applications with limited examples.
In other words, the task in Sec. 5.1 could benefit from another plot: the performance of MLP and RN at different amounts of training samples.
- While the simulation setup in the current paper is a great first-step towards analyzing the "relation-learning" ability of RNs, it is still not clear if this would transfer to real-life datasets. I strongly encourage the authors to experiment on real-life datasets like Coco, visual genome or HICO as stated in the pre-review stage.
- Minor: Some terminologies in the paper such as "objects" and "scene descriptions" used to refer to abstract entities can be misleading for readers from the object detection domain in computer vision. This could be clarified early on in the introduction.
- Minor: Some results like Fig. 8 which shows the ability of RN to generalize to unseen categories are quite interesting and could be moved to the main draft for completeness.
The paper proposes a network which is capable of understanding relationships between objects in a scene. This ability of the RN is thoroughly investigated through a series of experiments on a controlled dataset.
While, the model is currently evaluated only on a simulated dataset, the results are quite promising and could translate to real-life datasets as well.
Thank you for your review. You raise a number of points that will help us clarify and improve the paper.
In particular, you mention that there is a possibility that the terminology used to describe object relation learning can be confused with the terminology used for object detection in computer vision, and that this may lead to confusion for readers from the object detection domain in computer vision. We will update the paper to clarify our terminology to avoid this. We will also move Figure 8 to the main text as requested.
RE: permutation invariance. g_{\psi}(o_i, o_j) is not necessarily permutation invariant. We ensure that RN's are permutation invariant by always using \sum_ij g_{\psi}(o_i, o_j) which is permutation invariant for all MLP's.
Additionally, we use all orders of object pairs as input to the MLP for g_{\psi}, ensuring that it is not sensitive to this ordering.
Nonetheless, even presenting ordered inputs shouldn’t pose a problem, since object order is random from example to example.
Thus, any potential sensitivity can be offset by augmenting the dataset, and/or randomly re-ordering the objects online.
RE: complicated object interactions.  In the most general form, a relation network can reason across relations between triplets, quadruplets etc.
For example, if we write r = f_{\phi}(\sum_{ijk} g(o_i,o_j,o_k)), we can train a relation network to consider relations across triples. An interesting future line of work is to see whether a pre-processing module can learn to group relevant objects into a single “meta-object,” obviating the need for the RN to compute more complicated N-order relations.
RE: scalability. For a pairwise RN (a RN that acts across pairs of objects), the number of parameters is independent of the the number of objects, which is an especially nice feature of our model that makes the scaling much better than vanilla MLPs. The computation time increases quadratically (each evaluation of r requires O(n^2) evaluations of g, where n is the number of objects), although this can be mitigated with parallel computation.
RE: capacities of a DNN. The point was made mainly to argue for the use of our architectural and computational prior, and we do not believe that, in practice, a DNN can perform similar to an RN. Much like how we believe that a sufficiently complex MLP can theoretically implement the operations of a CNN, in practice we do not expect this to be the case. We tried some experiments using very large MLPs, and the results are similar to those shown in the paper. It is hard to see any improvement at all, regardless of the size of the MLP.
RE: real-life datasets. We agree that this is a particularly interesting direction for further exploration. In this work, our focus was architectural and conceptual innovation rather than scaling to real-life datasets, allowing us to focus on the fundamental problem of object-relation reasoning. We hope that, armed with a powerful tool for object relation reasoning, and with powerful perceptual front-ends developed by others, we can focus on real-life datasets that may require more significant engineering efforts to blend the two.

Meta-review:
role: PC
data: comment: This paper proposes RNs, relational networks, for representing and reasoning about object relations. Experiments show interesting results such as the capability to disentangling scene descriptions. AR3 praises the idea and the authors for doing this nice and involved analysis. AR1 also liked the paper. Indeed, taking a step back and seeing whether we are able to learn meaningful relations is needed in order to build more complex systems.
 
 However, AR2 raised some important issues: 1) the paper is extremely toy; RN has a very simplistic structure, and is only shown to work on synthetic examples that to some extent fit the assumptions of the RN. 2) there is important literature that addresses relation representations that has been entirely overlooked by the authors. The reviewer implied missed citations from a field that is all about learning object relations. In its current form, the paper does not have a review of related work. The AC does not see any citations in a discussion nor in the final revision. This is a major letdown. The reviewer also mentioned the fact that showing results on real datasets would strengthen the paper, which was also brought up by AR3. This indeed would have added value to the paper, although it is not a deal breaker.
 
 The reviewer AR2 did not engage in discussions, which indeed is not appropriate. The AC does weigh this review less strongly.
 
 Give the above feedback, we recommend this paper for the workshop. The authors are advised to add a related work section with a thorough review over the relevant fields and literature.

Decision: Invite to Workshop Track