Learning generative models with visual attention
Authors: Charlie Tang, Nitish Srivastava, Ruslan Salakhutdinov
ICLR 2014

Abstract: Attention has long been proposed by psychologists as important for effectively dealing with the enormous sensory stimulus available in the neocortex. Inspired by visual attention models in computational neuroscience and by the need for deep generative models to learn on object-centric data, we describe a framework for generative learning using attentional mechanisms. Attentional mechanism propagate signals from region-of-interest in a scene to higher layer areas of canonical representation, where generative modeling takes place. By ignoring background clutter, generative model can concentrate its resources to model objects of interest. Our model is a proper graphical model where the 2D similarity transformation from computer vision is part of the top-down process. A ConvNet is used to initialize good guesses during posterior inference, which is based on Hamiltonian Monte Carlo. Upon learning on face images, we demonstrate that our model can robustly attend to face regions of novel test subjects. Most importantly, our model can learn generative models of new faces from a novel dataset of large images where the location of the face is not known.


Peer review:
[Review]:
We thank the reviewer for the thoughtful review, and we look forward to improving the manuscript in the future version.

The main motivation from this paper is not using ConvNet as a STOA object detector, but to allow for learning generative models of objects of interest in large unlabelled images. A good face detector would probably do just as well on localizing faces. However, our attentional inference process is a function of v, meaning that we can attend to whatever the top-down generative model 'has in mind'. Section 6.3 demonstrates that given the same input image but with different v, our model attends to the correct subject. 

While an off-the-shelf VJ face detector would do very well, it is unclear how a face detector would perform if it were only trained on the Caltech faces (450 faces). Our model performs slightly worse when clamping 'v' to be the mean face throughout the inference process. Note that we are already initializing v with the mean face.

Qualitatively, the samples in Fig. 6b do look more like the CMU multipie faces while samples in Fig. 6a look more like faces from the Caltech faces dataset. 

Section 6.3: It is not a trivial task because the ConvNet takes as its input v and the big image and then predict the next gaze. The ConvNet is learned to shift either left or right depending on what v is. This is not the same as a template matcher with v as the template. We agree that having the 'same-sex' targets would be an interesting experiment.

The threshold was chosen by hand, high enough to have a small number of false positives, and low enough to have reasonable recall (951/1418). This threshold was used to filter out falsely localized gazes on the novel dataset. 

It is not a simple 3 layer DBM (at the top of page 4) because x is a function of u. It could be thought of as a 3 layer DBM with high-order multiplicative interactions between u, v, and x.

Please see the comment to Reviewer eb85 regarding the overloaded notation and the contradictory statements about HMC and local minima.