Learning generative models with visual attention
Authors: Charlie Tang, Nitish Srivastava, Ruslan Salakhutdinov
ICLR 2014

Abstract: Attention has long been proposed by psychologists as important for effectively dealing with the enormous sensory stimulus available in the neocortex. Inspired by visual attention models in computational neuroscience and by the need for deep generative models to learn on object-centric data, we describe a framework for generative learning using attentional mechanisms. Attentional mechanism propagate signals from region-of-interest in a scene to higher layer areas of canonical representation, where generative modeling takes place. By ignoring background clutter, generative model can concentrate its resources to model objects of interest. Our model is a proper graphical model where the 2D similarity transformation from computer vision is part of the top-down process. A ConvNet is used to initialize good guesses during posterior inference, which is based on Hamiltonian Monte Carlo. Upon learning on face images, we demonstrate that our model can robustly attend to face regions of novel test subjects. Most importantly, our model can learn generative models of new faces from a novel dataset of large images where the location of the face is not known.


Peer review:
[Review]:
The authors present an attentional generative model, inspired by the routing circuits model of Olshausen et al. (ref. [14]). Similar to some earlier work, attentional aspects result from the model essentially being misspecified to only represent a single object, treating everything else in the image as noise implicitly (besides the cited [18], Chikkerur et al, 2009, is another relevant example). The model consists of a Deep Belief Net (DBN) that models individual objects (here: face patches) in a canonical reference frame, and a transformation operation that scales/rotates/translates the object to be positioned in a larger image. Inference iteratively updates both the internal representations and hypothesized positioning in the image, and is based on Hamiltonian Monte Carlo (HMC) as well as a convolutional net (ConvNet), which makes initial guesses.

Pros: I find the topic interesting, and the approach original and creative. 
Cons: The paper suffers from quality and clarity issues. Moreover, this is not primarily a biological model, so the question is whether the proposed approach is promising for the machine learning application. As with other attention models, I find the evidence lacking that attention is really needed here to solve the task or that the approach works better than other, perhaps simpler alternatives.


Details:

First of all, the paper has a number of typos and grammar issues that should be fixed (three examples right in the abstract: 'enormous sensory stimulus available in the neocortex' is an awkward formulation; 'we describe for [a?] generative learning framework', 'signals from [a] region of interest').

I understand that biological attention this is not the primary focus of the paper, but I would have enjoyed a bit more discussion of the notion that attention corresponds to transforming input into an object centered frame, other than citing ref [14]. As far as I know, this is a speculative proposal and it would be nice to discuss whether there is recent evidence or theories supporting it. Similarly, the authors could have explained better how their framework is to be interpreted in terms of biological attention, if at all. For example, they appear to contrast their model to the 'covert' one of [18], and similarly the term 'gaze variables' suggests an interpretation in terms of overt eye movements. But note that the Olshausen model was supposed to model covert processes, i.e. internal routing of information with fixed retinal input. I also take it that we're not supposed to take the 'gaze' interpretation too literally, seeing as arbitrary scaling and rotations are involved...

The description of the model could be made a bit clearer. For example, the full graphical model could be displayed in Fig 1, rather than 'hiding' parts in the black box. {p} appears to be introduced twice (Section 3, paragraphs 3 and 5), and later p is used as momentum variable in HMC. I also get the impression that some parts of the model are not explained perhaps because they further complicate the model. In particular, the figure and detailed description only cover inference in a RBM. However, a DBN is what is actually used. Presumably the the hybrid directed/undirected DBN further complicates inference? 

Similarly, the architecture and training parameters of the ConvNet are not described at all, even though the latter seems to be what does the main work when it comes to localizing the faces. Speaking of which, I found iteratively predicting the transformations with the ConvNet interesting--it would be good if the authors could comment on whether this is a novel contribution or whether there are related approaches. I'm less convinced by the performance of the HMC, which mostly seems to fine-tune the window position. Note that the authors motivate the HMC in Section 3 by writing that it helps 'with jumping out of local maximum' [sic], but then justify the ConvNet in Section 4 by writing that HMC tends to get stuck in local optima...

Also Section 6.1, is the full model run (including the DBN) here? If so, what are the model parameters? The detection performance is only compared to two, presumably baseline methods. What is state-of-the-art on this dataset? Also, footnote 1 says that u was initialized to be centered. Depending on the dataset, isn't that potentially cheating?

Section 6.2: the authors state that the novelty of their approach is demonstrated here. The authors essentially first train their model with labels, and then use this first model as a face detector on a second, unlabeled dataset to localize the faces and train a second DBN. Here it was not clear to me whether they train the second DBN from scratch or merely fine-tune the first DBN pretrained with labels (they say they train the second DBN with FPCD. FPCD is for RBMs, so are they referring to layer-wise pretraining?). Either way, the main issue here is that the authors don't actually quantify the performance of the second model, so it is not clear if anything is gained with this approach, over just taking the first model that was only trained on the labeled data.

Lastly, I found Section 6.2, 'Inference with ambiguity', the most interesting bit, as it actually demonstrates the interaction between the canonical face model and the localization. Unfortunately, this part is very short. Perhaps the authors can expand this in future work.

Leftovers:

* Section 2: 'If a second level RBM is used to model the activities of the hidden units of the first layer GRBM, we can combine their energy functions to form a Deep Belief Net (DBN) [2]'. Doesn't simply combining the energies give a deep Boltzmann machine rather than a DBN?
* Section 4.1: 'the spatial frequency of the natural image signals can form many local minima': this should be expressed better.
* Section 4.1: '(e.g. 72×72) and v (e.g. 24×24)' here and elsewhere: why the 'e.g.'? Just say that this is what you are using.
* Figure 3: the caption should clarify that this is not the full inference process, only the initial part.
* Figure 4: there seem to be artifacts in the images?
* Figure 5: the fonts are too small to be readable.