Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a neural attention model which has a learnable and differentiable sampling lattice. The work is well motivated as few previous work focus on learning the sampling lattice but with a fixed lattice. This work is quite similar to Spatial Transformer Networks (Jaderberg 2015), but the sampling lattice is learned by the model. The experiments showed that the model can learn a meaning lattice to the visual search task where the sampling lattice looks similar to human being's. 

The main concern of the paper is that experiments are not sufficient. The paper only reports the results on a modified clustered MNIST dataset. It would be more interesting if the authors could conduct  the model on real datasets, such as Toronto Face dataset, CUB bird dataset and SVHN. For example, for the Face dataset, it would be nice if the model can learn to attend different parts of the face for expression recognition, or attend different part of birds for fine-grained classification. Since the authors replied in the pre-review question that the model can learn meaningful lattice on MSCOCO dataset, I think it would be better to add that results into the paper.

Another drawback of the model is that the paper only compare with different variants of itselves. I suggest that this paper should compare with  Spatial Transformer Networks, DRAW, etc., on the same dataset to show the advantage of the learned sampling lattice.
review_title: A good apporach to attention-based models

final_score: rating: null
review_content: I am wondering if the authors ever have done an experiment varying the number of kernels? It will be very interesting to see how the performance of the translation only and translation+zoom methods scales with the number of kernels. Also, what happens when there are two object of interests in an input canvas? Does the individual \mu focuses on both object at once (which can no longer keep the primate retina layout) or scans them sequentially and still develops the primate retina layout?
review_title: interesting experiments

final_score: rating: null
review_content: This is an interesting paper! The experiments are only conducted on digits dataset. Can authors provide results on other real datasets, such as on face dataset (Zheng et al 2015, http://dl.acm.org/citation.cfm?id=2776962), SVHN dataset and CUB_200_2011 birds datasets(Jaderberg 2015, https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf)?


review_title: experiments on other real datasets

final_score: rating: null
review_content: I'd really like to see this paper drive its point home with one additional figure, which would be a side-by-side comparison of the distribution of mu and sigma that you've learned in the translation-only model (e.g. Figure 6) and a reproduced figure of how receptive fields are organized in some species' (e.g. human's) retina. Your main point with this paper is that they should qualitatively match right? Could you add it? I'd like to see e.g. human or macaque data overlayed on the scatter-plots in the lower panels of Figure 6.

If they don't match perfectly we might ask follow-up questions like "would the match improve by applying this type of training to more-realistic images, visual tasks, vision models?"
review_title: Comparison to distribution of some primate's receptive fields?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a succinct argument that the principle of optimizing receptive field location and size in a simulated eye that can make saccades with respect to a classification error of images of data whose labels depend on variable-size and variable-location subimages, explains the existence of a foveal area in e.g. the primate retina.

The argument could be improved by using more-realistic image data and drawing more direct correspondence with the number, receptive field sizes and eccentricities of retinal cells in e.g. the macaque, but the authors would then face the challenge of identifying a loss function that is both biologically plausible and supportive of their claim.

The argument could also be improved by commenting on the timescales involved. Presumably the density of the foveal center depends on the number of of saccades allowed by the inference process, as well as the size of the target sub-images, and also has an impact on the overall classification accuracy.

Why does the classification error rate of dataset 2 remain stubbornly at 24%? This seems so high that the model may not be working the way we’d like it to. It seems that the overall argument of the paper pre-supposes that the model can be trained to be a good classifier. If there are other training strategies or other models that work better and differently, then it raises the question of why do our eyes and visual cortex not work more like *those ones* if evolutionary pressures are applying the same pressure as our training objective.

Why does the model with zooming powers out-do the translation-only model on dataset 1 (where all target images are the same size) and tie the translation-only model dataset 2 (where the target images have different sizes, for which the zooming model should be tailor-made?). Between this strange tie and the high classification rate on Dataset 2, I wonder if maybe one or both models isn’t being trained to its potential, which would undermine the overall claim.

Comparing this model to other attention models (e.g. spatial transformer networks, DRAW) would be irrelevant to what I take to be the main point of the paper, but it would address the potential concerns above that training just didn’t go very well, or there was some problem with the model parameterization that could be easily fixed.
review_title: Solid hypothesis but experiments leave doubts

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presented an extension to the current visual attention model that learns a deformable sampling lattice.  Comparing to the fixed sampling lattice from previous works, the proposed method shows different sampling strategy can emerge depending on the visual classification tasks. The authors empirically demonstrated the learnt sampling lattice outperforms the fixed strategies. More interestingly, when the attention mechanism is constrained  to be translation only, the proposed model learns a sampling lattice resembles the retina found in the primate retina.  


Pros:
+ The paper is generally well organized and written 
+ The qualitative analysis in the experimental section is very comprehensive.

Cons:
-  The paper could benefit substantially from additional experiments on different datasets.
-  It is not clear from the tables the proposed learnt sampling  lattice offer any computation benefit when comparing to  a fixed sampling strategy with zooming capability, e.g. the one used in DRAW model.

Overall, I really like the paper. I think the experimental section can be improved by additional experiments and more quantitative analysis with other baselines. Because the current revision of the paper only shows experiments on digit dataset with black background, it is hard to generalize the finding or even to verify the claims in the paper, e.g.  linear relationship
between eccentricity and sampling interval leads to the primate retina, from the results on a single dataset.
review_title: Review

Meta-review: This was a borderline case. All reviewers and the AC appeared to find the paper interesting, while having some reservations. Given the originality of the work, the PCs decided to lean toward acceptance. We do encourage however the authors to revise their paper based on reviewer feedback as much as possible, to increase its potential for impact.
Decision: Accept (Poster)