Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: N/A
review_title: N/A

final_score: rating: null
review_content: 1. Why do the scene descriptions have 16 rows? Sec. 2.1 states that the scene description is a m x n matrix with each row corresponding to an object and n properties of the objects. On the other hand sec. 3.1 says that each description is a 16 row matrix 4 rows for each object type. Could you clarify the actual structural representation of the scene description matrix? Were all scenes constrained to have 16 objects (4 from each type)?

2. Why is BU a matrix of size 16 x 3 in sec. 5.2.1 (page 8), when B is a matrix of size mn x mn and U is mn x mn? Can you explain the inset figure in Fig. 5a in more detail?

3. The setup for the final category of tasks needs to be explained better. Perhaps a pictorial example would assist a lot. Also, did this task operate on images or scene description matrices?

4. The experiments state that the baseline MLP used same size architecture as the RN. What does this mean in practice? For instance, while comparing a {200, 200} RN with an MLP, did the MLP just have two hidden layers with 200 units each? If that is the case, the RN with object pair prior runs an MLP on each pair of objects (m^2 {200, 200} MLP) before summing and passing it to f_{\phi}. Is the number of computations consistent between such an RN and the corresponding baseline {200, 200} MLP?

5. The object-pair prior RN is stated to operate on every pair of object rows. However, in the case of VAE features there is no guarantee that the feature is factored in terms of objects. So how does the RN identify object pairs in this case?

6. Have you considered running experiments with  real-life image datasets with labelled object relations such as Visual Genome and HICO (Chao et al. ICCV'15)?
review_title: Dataset construction clarification

final_score: rating: null
review_content: Are there some examples of the images used in Sec. 5.2.2?
review_title: inferring relations from pixels

final_score: rating: 7: Good paper, accept
review_content: + Understanding relations between objects is an important task in domains like vision, language and robotics. However, models trained on real-life datasets can often exploit simple object properties (not relation-based) to identify relations (eg: animals of bigger size are typically predators and small-size animals are preys). Such models can predict relations without necessarily understanding them. Given the difficulty of the task, a controlled setting is required to investigate if neural networks can be designed to actually understand pairwise object relations. The current paper takes a significant step in answering this question through a controlled dataset. Also, multiple experiments are presented to validate the "relation learning" ability of proposed Relation Networks (RN).

+ The dataset proposed in the paper ensures that relation classification models can succeed only by learning the relations between objects and not by exploiting "predator-prey" like object properties.

+ The paper presents very thorough experiments to validate the claim that "RNs" truly learn the relation between objects.
  1. In particular, the ability of the RN to force a simple linear layer to disentangle scene description from VAE latent space and permuted description is very interesting. This clearly demonstrates that the RN learns object relations.
  2. The one-shot experiments again demonstrate this ability in a convincing manner. This requires the model to understand relations in each run, represent them through an abstract label and assign the label to future samples from the relationship graph.

Some suggestions:

- Is g_{\psi}(.) permutation invariant as well. Since it works on pairs of objects, how did you ensure that the MLP is invariant to the order of the objects in the pair?
- The RNs need to operate over pairs of objects in order to identify pairwise interactions. However, in practical applications there are more complicated group interactions. (eg. ternary interaction: "person" riding a "bike" wears "helmet"). Would this require g(.) of RN to not just operate on pairs but on every possible subset of objects in the scene? More generally, is such a pairwise edge-based approach scalable to larger number of objects?
- The authors mention that " a deep network with a sufficiently large number of parameters and a large enough training set should be capable of matching the performance of a RN". This is an interesting point, and could be true in practice. Have the authors investigated this effect by trying to identify the minimum model capacity and/or training examples required by a MLP to match the performance of RN for the provided setup? This would help in quantifying the significance of RN for practical applications with limited examples. In other words, the task in Sec. 5.1 could benefit from another plot: the performance of MLP and RN at different amounts of training samples.
- While the simulation setup in the current paper is a great first-step towards analyzing the "relation-learning" ability of RNs, it is still not clear if this would transfer to real-life datasets. I strongly encourage the authors to experiment on real-life datasets like Coco, visual genome or HICO as stated in the pre-review stage.
- Minor: Some terminologies in the paper such as "objects" and "scene descriptions" used to refer to abstract entities can be misleading for readers from the object detection domain in computer vision. This could be clarified early on in the introduction.
- Minor: Some results like Fig. 8 which shows the ability of RN to generalize to unseen categories are quite interesting and could be moved to the main draft for completeness.

The paper proposes a network which is capable of understanding relationships between objects in a scene. This ability of the RN is thoroughly investigated through a series of experiments on a controlled dataset. While, the model is currently evaluated only on a simulated dataset, the results are quite promising and could translate to real-life datasets as well.
review_title: Well motivated model for relationship prediction in abstract scenes with good experimental analysis in a controlled setting.

final_score: rating: 3: Clear rejection
review_content: This paper proposes relation networks in order to model the pairwise interactions between objects in a visual scene. 
The model is very straight forward, first an MLP (with shared weights) is applied to each pair of objects. Finally a prediction is created by an MLP which operates by summing non-linear functions of these pairs of objects. 
Experimental evaluation is done in a synthetic dataset that is generated to fit the architecture hand-crafted in this paper. 

The title of the paper claims much more than the paper delivers. Discovering objects and their relations is a very important task.
However, this paper does not discover objects or their relations, instead, each objects is represented with hand coded ground truth attributes, and only a small set of trivial relationships are "discovered", e.g., relative position. 

Discovering objects and their relationships has been tackled for several decades in computer vision (CV). The paper does not cite or compare to any technique in this body of literature. This is typically refer to as "contextual models".

Can the proposed architecture help object detection and/or scene classification? would it work in the presence of noise (e.g, missing detections, non accurate detection estimates, complex texture)? would it work when the attributes of objects are estimated from real images? 
 
I'll be more convinced if experiments where done in real scenes. In the case of indoor scenes, datasets such as NYUv2, Sun-RGB-D, SceneNN, Chen et al CVPR 14 (text-to-image-correference) could be used. In outdoor scenes, KITTI and the relationships between cars, pedestrians and cyclist could also serve as benchmark. 

Without showing real scenes, this paper tackles a too toy problem with a very simple model which does not go much further than current context models, which model pairwise relationships between objects (with MRFs, with deep nets, etc). 

review_title: The paper promises much more than it delivers. It ignores completely the litterature and only tackles a toy synthetic example. Clearly below the bar

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a relation network (RN) to model relations between input entities such as objects.  The relation network is built in two stages.  First a lower-level structure analyzes a pair of input entities.  All pairs of input entities are fed to this structure.  Next, the output of this lower-level structure is aggregated across all input pairs via a simple sum.  This is used as the input to a higher-level structure.  In the basic version, these two structures are each multi-layer perceptrons (MLPs).

Overall, this is an interesting approach to understanding relations among entities.  The core idea is clear and well-motivated -- pooling techniques that induce invariance can be used to learn relations.  The idea builds on pooling structures (e.g. spatial/temporal average/max pooling) to focus on pairwise relations.  The current pairwise approach could potentially be extended to higher-order interactions, modulo scaling issues.

Experiments on scene descriptions and images verify the efficacy of relation networks.  The MLP baselines used are incapable of modeling the structured dependencies present in these tasks.  It would be interesting to know if pooling operators (e.g. across-object max pooling in an MLP) or data augmentation via permutation would be effective for training MLPs at these tasks.  Regardless, the model proposed here is novel and effective at handling relations and shows promise for higher-level reasoning tasks.


review_title: Novel modeling of entity relationships in a neural network

Meta-review: This paper proposes RNs, relational networks, for representing and reasoning about object relations. Experiments show interesting results such as the capability to disentangling scene descriptions. AR3 praises the idea and the authors for doing this nice and involved analysis. AR1 also liked the paper. Indeed, taking a step back and seeing whether we are able to learn meaningful relations is needed in order to build more complex systems.
 
 However, AR2 raised some important issues: 1) the paper is extremely toy; RN has a very simplistic structure, and is only shown to work on synthetic examples that to some extent fit the assumptions of the RN. 2) there is important literature that addresses relation representations that has been entirely overlooked by the authors. The reviewer implied missed citations from a field that is all about learning object relations. In its current form, the paper does not have a review of related work. The AC does not see any citations in a discussion nor in the final revision. This is a major letdown. The reviewer also mentioned the fact that showing results on real datasets would strengthen the paper, which was also brought up by AR3. This indeed would have added value to the paper, although it is not a deal breaker.
 
 The reviewer AR2 did not engage in discussions, which indeed is not appropriate. The AC does weigh this review less strongly.
 
 Give the above feedback, we recommend this paper for the workshop. The authors are advised to add a related work section with a thorough review over the relevant fields and literature.
Decision: Invite to Workshop Track