Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors introduce a set of very simple tasks that are meant to illustrate the challenges of learning visual relations.  They then evaluate several existing network architectures on these tasks, and show that results are not as impressive as others might have assumed they would be.   They show that while recent approaches (e.g. relational networks) can generalize reasonably well on some tasks, these results do not generalize as well to held-out-object scenarios as might have been assumed. 

Clarity:  The paper is fairly clearly written.   I think I mostly followed it.   

Quality:  I'm intrigued by but a little uncomfortable with the generalization metrics that the authors use.   The authors estimate the performance of algorithms by how well they generalize to new image scenarios when trained on other image conditions.   The authors state that ". . . the effectiveness of an architecture to learn visual-relation problems should be measured in terms of generalization over multiple variants of the same problem, not over multiple splits of the same dataset."  Taken literally, this would rule out a lot of modern machine learning, even obviously very good work. On the other hand, it's clear that at some point, generalization needs to occur in testing ability to understand relationships.  I'm a little worried that it's "in the eye of the beholder" whether a given generalization should be expected to work or not.     

There are essentially three scenarios of generalization discussed in the paper:
        (a) various generalizations of image parameters in the PSVRT dataset
        (b) various hold-outs of the image parameters in the sort-of-CLEVR dataset
        (c) from sort-of-CLEVR "objects" to PSVRT bit patterns

The result that existing architectures didn't do very well at these generalizations (especially b and c) *may* be important -- or it may not.    Perhaps if CNN+RN were trained on a quite rich real-world training set with a variety of real-world three-D objects beyond those shown in sort-of-CLEVR, it would generalize to most other situations that might be encountered.    After all, when we humans generalize to understanding relationships, exactly what variability is present in our "training sets" as compared to our "testing" situations?   How do the authors know that humans are effectively generalizing rather than just "interpolating" within their (very rich) training set?  It's not totally clear to me that if totally naive humans (who had never seen spatial relationships before) were evaluated on exactly the training/testing scenarios described above, that they would generalize particularly well either.   I don't think it can just be assumed a priori that humans would be super good this form of generalization.  

So how should authors handle this criticism?  What would be useful would either be some form of positive control.  Either human training data showing very effective generalization (if one could somehow make "novel" relationships unfamiliar to humans), or a different network architecture that was obviously superior in generalization to CNN+RN. If such were present, I'd rate this paper significantly higher.    

Also, I can't tell if I really fully believe the results of this paper.  I don't doubt that the authors saw the results they report.  However, I think there's some chance that if the same tasks were in the hands of people who *wanted* CNNs or CNN+RN to work well, the results might have been different.   I can't point to exactly what would have to be different to make things "work", because it's really hard to do that ahead of actually trying to do the work.   However, this suspicion on my part is actually a reason I think it might be *good* for this paper to be published at ICLR.  This will give the people working on (e.g.) CNN+RN somewhat more incentive to try out the current paper's benchmarks and either improve their architecture or show that the the existing one would have totally worked if only tried correctly.  I myself am very curious about what would happen and would love to see this exchange catalyzed. 

Originality and Significance:  The area of relation extraction seems to me to be very important and probably a bit less intensively worked on that it should be.  However, as the authors here note, there's been some recent work (e.g. Santoro 2017) in the area.   I think that the introduction of baselines  benchmark challenge datasets such as the ones the authors describe here is very useful, and is a somewhat novel contribution.     
review_title: review of "NOT-SO-CLEVR: VISUAL RELATIONS STRAIN FEEDFORWARD NEURAL NETWORKS"

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths:

-	There is an interesting analysis on how CNN’s perform better Spatial-Relation problems in contrast to Same-Different problems, and how Spatial-Relation problems are less sensitive to hyper parameters.

-	The authors bring a good point on the limitations of the SVRT dataset – mainly being the difficulty to compare visual relations due to the difference of image structures on the different relational tasks and the use of simple closed curves to characterize the relations, which make it difficult to quantify the effect of image variability on the task. And propose a challenge that addresses these issues and allows controlling different aspects of image variability.

-	The paper shows how state of the art relational networks, performing well on multiple relational tasks, fail to generalize to same-ness relationships.

Weaknesses:

-	While the proposed PSVRT dataset addresses the 2 noted problems in SVRT, using only 2 relations in the study is very limited.

-	The paper describes two sets of relationships, but it soon suggests that current approaches actually struggle in Same-Different relationships. However, they only explore this relationship under identical objects. It would have been interesting to study more kinds of such relationships, such as equality up to translation or rotation, to understand the limitation of such networks. Would that allow improving generalization to varying item or image sizes?

Comments:

-	In page 2, authors suggest that from that Gülçehre, Bengio (2013) that for visual relations “failure of feed-forward networks […] reflects a poor choice of hyper parameters. This seems to contradict the later discussion, where they suggest that probably current architectures cannot handle such visual relationships. 

-	The point brought about CNN’s failing to generalize on same-ness relationships on sort-of-CLEVR is interesting, but it would be good to know why PSVRT provides better generalization. What would happen if shapes different than random squared patterns were used at test time?

-	Authors reason about biological inspired approaches, using Attention and Memory, based on existing literature. While they provide some good references to support this statement it would have been interesting to show whether they actually improve TTA under image parameter variations

review_title: This paper explores how current CNN’s and Relational Networks fail to recognize visual relations in images. It firstly performs a performance analysis of CNN’s on SVRT, proposes a new visual relation challenge and shows how the proposed sort-of-CLEVR challenge can be slightly modified to break current state of the art approaches.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Quality

This paper demonstrates that convolutional and relational neural networks fail to solve visual relation problems by training networks on artificially generated visual relation data. This points at important limitations of current neural network architectures where architectures depend mainly on rote memorization.

Clarity

The rationale in the paper is straightforward. I do think that breakdown of networks by testing on increasing image variability is expected given that there is no reason that networks should generalize well to parts of input space that were never encountered before.

Originality

While others have pointed out limitations before, this paper considers relational networks for the first time.

Significance 

This work demonstrates failures of relational networks on relational tasks, which is an important message. At the same time, no new architectures are presented to address these limitations.

Pros

Important message about network limitations.

Cons

Straightforward testing of network performance on specific visual relation tasks. No new theory development. Conclusions drawn by testing on out of sample data may not be completely valid.
review_title: Important limitations of relational networks

Meta-review: This paper studies an important problem (visual relationship detection and generalization capabilities existing networks for this task). Unfortunately, all reviewers raise concerns (e.g. limited relations studied) and are largely on the fence about this paper. While this paper does not propose solutions, it does present interesting "negative results" that should get some visibility in the workshop track. 
Decision: Invite to Workshop Track