Title: Emergence of foveal image sampling from learning to attend in visual scenes
Authors: Brian Cheung
Eric Weiss
Bruno Olshausen
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Supervised Learning

Abstract: We describe a neural attention model with a learnable retinal sampling lattice. The model is trained on a visual search task requiring the classification of an object embedded in a visual scene amidst background distractors using the smallest number of fixations. We explore the tiling properties that emerge in the model's retinal sampling lattice after training. Specifically, we show that this lattice resembles the eccentricity dependent sampling lattice of the primate retina, with a high resolution region in the fovea surrounded by a low resolution periphery. Furthermore, we find conditions where these emergent properties are amplified or eliminated providing clues to their function.

TL;DR: We show a foveal sampling lattice similar to those observed in biology emerges from our model and task.

Review discussion:
This is an interesting paper! The experiments are only conducted on digits dataset. Can authors provide results on other real datasets, such as on face dataset (Zheng et al 2015, http://dl.acm.org/citation.cfm?id=2776962), SVHN dataset and CUB_200_2011 birds datasets(Jaderberg 2015, https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf)?
As we mention in the introduction, we have found certain properties in the dataset/task/model are key to having foveal sampling emerge.
For example, we are currently experimenting with localizing the digit instead of classifying and find that a very different sampling lattice emerges. To draw strong scientific explanations of foveal sampling, we require our model to work in an environment where we have strong control over the factors of variation.
With that said, we are interested in more natural stimuli and have run our attention models on Toronto Faces Dataset (TFD) (used in Zheng et al 2015) and CUB_200_2011 birds dataset (used in Jaderberg et al 2015). Currently, we do not find foveal sampling to emerge for TFD when classifying facial expressions (96x96 images). We believe this is likely because the variations in this dataset/task are drastically different from our digits dataset. The faces are very well registered and important facial keypoints consistently appear in the same locations between samples.
Similarly, for the CUB_200_2011 dataset, the bird in the images are well framed and appear consistently in the central region of the image (this becomes very apparent when you compute the mean image across samples). We are currently experimenting with attempts to create realistic and similar variations in TFD as our digits dataset.
For example, starting the glimpse at a random position rather than at the center of the image.
Going through examples from SVHN (format 1), we believe similar issues will arise because the house numbers which are also fairly well registered. Currently, we are building our own dataset derived from MSCOCO which will have the very similar factors of variation present in our digits dataset while still being natural as a stimulus. Because MSCOCO images are large and well annotated, we can take specific crops of an image and control the position and size of the object of interest.
Thus we can control the factors of variation in a similar fashion to our digits dataset (aside from the type of clutter present).
At the moment, the goal of our work is to show the minimal set of properties that generates foveal sampling.
I am wondering if the authors ever have done an experiment varying the number of kernels? It will be very interesting to see how the performance of the translation only and translation+zoom methods scales with the number of kernels.
Also, what happens when there are two object of interests in an input canvas? Does the individual \mu focuses on both object at once (which can no longer keep the primate retina layout) or scans them sequentially and still develops the primate retina layout?
> I am wondering if the authors ever have done an experiment varying the number of kernels? It will be very interesting to see how the performance of the translation only and translation+zoom methods scales with the number of kernels.
Yes, we have tried multiple kernel sizes and the results appear consistent across the ranges we have experimented (64 to 256 kernels). As expected, classification performance is generally better for both the translation only and translation+zoom methods with a higher number of kernels.
Importantly, the results of the emergent foveal tiling are consistent when varying the number of kernels.
>  Also, what happens when there are two object of interests in an input canvas? Does the individual \mu focuses on both object at once (which can no longer keep the primate retina layout) or scans them sequentially and still develops the primate retina layout?
We have experimented with tasks where the attention model must choose a digit from a set of digits randomly appearing in the input image based on some form of context (similar to a 'Where's Waldo' task). We have found that given only the high level signal of localization, the attention model does not converge well to this task (regardless of glimpse mechanism).
For example, the translation+zoom model creates a zoom window which covers all objects at once. We are currently working on better signals to provide the attention model to learn to scan the objects sequentially.
I'd really like to see this paper drive its point home with one additional figure, which would be a side-by-side comparison of the distribution of mu and sigma that you've learned in the translation-only model (e.g. Figure 6) and a reproduced figure of how receptive fields are organized in some species' (e.g. human's) retina. Your main point with this paper is that they should qualitatively match right? Could you add it? I'd like to see e.g. human or macaque data overlayed on the scatter-plots in the lower panels of Figure 6.
If they don't match perfectly we might ask follow-up questions like "would the match improve by applying this type of training to more-realistic images, visual tasks, vision models?"
This study is a first step in trying to understand the relationship between active, attentional-based vision and the spatial arrangement of the retinal sampling lattice.  There is undoubtedly more work to do here, but we feel that the result obtained with this relatively simple task and stimulus domain is already indicative of some of the important trade offs.  It also demonstrates the feasibility of using gradient descent in an end to end learning scheme to derive the optimal sampling lattice.  To produce a more quantitative agreement with the primate retina you would need a fuller range of stimulus sizes from the very smallest (defined by individual cones) to the very largest (subtending 10's degrees).  And you would want to deploy this over a visual field that subtends 150 degrees or more - I.e., the equivalent of 15,000 pixels (150 deg/.01 deg/cone) - and preferably natural scenes containing multiple objects.
That is beyond the scope of the current study, and in fact this study can be seen as a necessary prerequisite for moving into that realm, which we are currently working on.
This paper proposed a neural attention model which has a learnable and differentiable sampling lattice. The work is well motivated as few previous work focus on learning the sampling lattice but with a fixed lattice. This work is quite similar to Spatial Transformer Networks (Jaderberg 2015), but the sampling lattice is learned by the model. The experiments showed that the model can learn a meaning lattice to the visual search task where the sampling lattice looks similar to human being's.
The main concern of the paper is that experiments are not sufficient. The paper only reports the results on a modified clustered MNIST dataset. It would be more interesting if the authors could conduct  the model on real datasets, such as Toronto Face dataset, CUB bird dataset and SVHN.
For example, for the Face dataset, it would be nice if the model can learn to attend different parts of the face for expression recognition, or attend different part of birds for fine-grained classification. Since the authors replied in the pre-review question that the model can learn meaningful lattice on MSCOCO dataset, I think it would be better to add that results into the paper.
Another drawback of the model is that the paper only compare with different variants of itselves. I suggest that this paper should compare with  Spatial Transformer Networks, DRAW, etc., on the same dataset to show the advantage of the learned sampling lattice.
> Since the authors replied in the pre-review question that the model can learn meaningful lattice on MSCOCO dataset, I think it would be better to add that results into the paper.
In our pre-review answer, we described constructing a dataset based on MSCOCO which would have similar controllable factors of variation to our Cluttered MNIST dataset. We have a version of the dataset developed, but we are currently working on reformulating our model to be more computationally efficient for the large images present in the MSCOCO dataset.
But such a formulation is beyond the scope of this current paper.
> Another drawback of the model is that the paper only compare with different variants of itselves. I suggest that this paper should compare with  Spatial Transformer Networks, DRAW, etc., on the same dataset to show the advantage of the learned sampling lattice.
The goal of our work was to show an emergent property of the retinal cells during learning. We are trying to explain a phenomenon observed in biology. This work is similar in spirit (and title) to Olshausen and Field 1996 where they show gabors emerge from learning dictionary features rather than compare against SIFT/HOG features for classification. Our model with zoom is essentially the supervised model of DRAW which is a specific instance of the more general glimpse formulation proposed in Spatial Transformer Networks. We are exploring the choice of the tiling issue in the context of the DRAW model where we augment it with a learnable lattice. The point is not to make a better DRAW model.
Olshausen, Bruno A. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images." Nature 381.6583 (1996): 607-609.
The paper presented an extension to the current visual attention model that learns a deformable sampling lattice.  Comparing to the fixed sampling lattice from previous works, the proposed method shows different sampling strategy can emerge depending on the visual classification tasks. The authors empirically demonstrated the learnt sampling lattice outperforms the fixed strategies. More interestingly, when the attention mechanism is constrained  to be translation only, the proposed model learns a sampling lattice resembles the retina found in the primate retina.
Pros:
+ The paper is generally well organized and written
+ The qualitative analysis in the experimental section is very comprehensive.
Cons:
-  The paper could benefit substantially from additional experiments on different datasets.
-  It is not clear from the tables the proposed learnt sampling  lattice offer any computation benefit when comparing to  a fixed sampling strategy with zooming capability, e.g. the one used in DRAW model.
Overall, I really like the paper. I think the experimental section can be improved by additional experiments and more quantitative analysis with other baselines. Because the current revision of the paper only shows experiments on digit dataset with black background, it is hard to generalize the finding or even to verify the claims in the paper, e.g.  linear relationship
between eccentricity and sampling interval leads to the primate retina, from the results on a single dataset.
> -  The paper could benefit substantially from additional experiments on different datasets.
We are currently investigating this.
But as mentioned in the response to AnonReviewer2, we need to reformulate the attention mechanism to be computationally efficient over much larger image sizes (same computational issues arise for Spatial Transformer and DRAW).
> -  It is not clear from the tables the proposed learnt sampling  lattice offer any computation benefit when comparing to  a fixed sampling strategy with zooming capability, e.g. the one used in DRAW model.
The goal of our work is to show an emergent property of the retinal cells during learning, not that one model of attention is better than another. We are trying to explain a phenomenon observed in biology. This work is similar in spirit (and title) to Olshausen and Field 1996 where they show gabors emerge from learning dictionary features.
Olshausen, Bruno A. "Emergence of simple-cell receptive field properties by learning a sparse code for natural images." Nature 381.6583 (1996): 607-609.
This paper demonstrated some very interesting emerging phenomenon of the visual attention models on a single dataset. My concern is that the foveal-like sampling pattern could be an artifact of digit dataset, which is very unlikely and could be easily disprove by experimenting on another dataset. I stand by my initial review and am willing to change the score if the emerging foveal like lattice could be verified from a different task.
This paper presents a succinct argument that the principle of optimizing receptive field location and size in a simulated eye that can make saccades with respect to a classification error of images of data whose labels depend on variable-size and variable-location subimages, explains the existence of a foveal area in e.g. the primate retina.
The argument could be improved by using more-realistic image data and drawing more direct correspondence with the number, receptive field sizes and eccentricities of retinal cells in e.g. the macaque, but the authors would then face the challenge of identifying a loss function that is both biologically plausible and supportive of their claim.
The argument could also be improved by commenting on the timescales involved. Presumably the density of the foveal center depends on the number of of saccades allowed by the inference process, as well as the size of the target sub-images, and also has an impact on the overall classification accuracy.
Why does the classification error rate of dataset 2 remain stubbornly at 24%? This seems so high that the model may not be working the way we’d like it to. It seems that the overall argument of the paper pre-supposes that the model can be trained to be a good classifier. If there are other training strategies or other models that work better and differently, then it raises the question of why do our eyes and visual cortex not work more like *those ones* if evolutionary pressures are applying the same pressure as our training objective.
Why does the model with zooming powers out-do the translation-only model on dataset 1 (where all target images are the same size) and tie the translation-only model dataset 2 (where the target images have different sizes, for which the zooming model should be tailor-made?). Between this strange tie and the high classification rate on Dataset 2, I wonder if maybe one or both models isn’t being trained to its potential, which would undermine the overall claim.
Comparing this model to other attention models (e.g. spatial transformer networks, DRAW) would be irrelevant to what I take to be the main point of the paper, but it would address the potential concerns above that training just didn’t go very well, or there was some problem with the model parameterization that could be easily fixed.
> The argument could be improved by using more-realistic image data and drawing more direct correspondence with the number, receptive field sizes and eccentricities of retinal cells in e.g. the macaque, but the authors would then face the challenge of identifying a loss function that is both biologically plausible and supportive of their claim.
We agree that these comparisons are great future directions to pursue.
In fact, we are currently investigating this model on more realistic image datasets (MSCOCO).
But there are a number of complications that will require more time to resolve.
Scaling up the model would require a more efficient scheme for computing the glimpse for the following reasons:
-The number the retinal cells needed to be comparable to actual retinal cells of the macaque would need to be far greater than 144.
-The scene image itself would also need to be scaled up to sizes larger than 100x100 which adds another multiplicative factor of computation time.
In our view one of the interesting aspects of the results reported here is that we are already obtaining a retina-like sampling lattice with a fairly simple task and set of images.  This helps us to understand the minimal factors necessary for this tiling structure to emerge as an optimal sampling strategy in an attentional system.
> Why does the classification error rate of dataset 2 remain stubbornly at 24%? This seems so high that the model may not be working the way we’d like it to. It seems that the overall argument of the paper pre-supposes that the model can be trained to be a good classifier. If there are other training strategies or other models that work better and differently, then it raises the question of why do our eyes and visual cortex not work more like *those ones* if evolutionary pressures are applying the same pressure as our training objective.
As mentioned in Section 4.1, Dataset 2 contains MNIST digits with significant resizing. The original 28x28 image can range in sizes from 9x9 to 84x84. The MNIST digit contained in the 9x9 image is even smaller. At these sizes, the target digit for Dataset 2 images can often be indistinguishable from clutter for human observers.
Furthermore, Figure 7 shows that our attention models exhibit reasonable behavior for samples from Dataset 2.
> Why does the model with zooming powers out-do the translation-only model on dataset 1 (where all target images are the same size) and tie the translation-only model dataset 2 (where the target images have different sizes, for which the zooming model should be tailor-made?). Between this strange tie and the high classification rate on Dataset 2, I wonder if maybe one or both models isn’t being trained to its potential, which would undermine the overall claim.
In regards to Dataset 1, Figure 7 shows the zooming model has the ability to allocate more cells to discern details of the digit while the translation only model can only allocate the cells formed in it’s fovea.
Our current hypothesis for the more similar performance of the Zooming and Translation Only models on Dataset 2 is the increased difficulty of the task. We noticed that the Zooming model can sometimes miss and lie off the target digit when the MNIST digit is especially small. The Translation Only model can more easily recover from a bad translation because there’s a greater chance that some cells will still lie over the target digit.

Meta-review:
role: PC
data: comment: This was a borderline case. All reviewers and the AC appeared to find the paper interesting, while having some reservations. Given the originality of the work, the PCs decided to lean toward acceptance. We do encourage however the authors to revise their paper based on reviewer feedback as much as possible, to increase its potential for impact.

Decision: Accept (Poster)