Learning generative models with visual attention
Authors: Charlie Tang, Nitish Srivastava, Ruslan Salakhutdinov
ICLR 2014

Abstract: Attention has long been proposed by psychologists as important for effectively dealing with the enormous sensory stimulus available in the neocortex. Inspired by visual attention models in computational neuroscience and by the need for deep generative models to learn on object-centric data, we describe a framework for generative learning using attentional mechanisms. Attentional mechanism propagate signals from region-of-interest in a scene to higher layer areas of canonical representation, where generative modeling takes place. By ignoring background clutter, generative model can concentrate its resources to model objects of interest. Our model is a proper graphical model where the 2D similarity transformation from computer vision is part of the top-down process. A ConvNet is used to initialize good guesses during posterior inference, which is based on Hamiltonian Monte Carlo. Upon learning on face images, we demonstrate that our model can robustly attend to face regions of novel test subjects. Most importantly, our model can learn generative models of new faces from a novel dataset of large images where the location of the face is not known.


Peer review:
[Review]:
Summary
This paper proposed a probabilistic framework for learning generative models where the 2D Similarity transformation is a part of the top-down process and a ConvNet is used to initialize the posterior inference. 
Though this paper is clearly interesting and important for both deep learning and computer vision community, it could potentially be improved. The main problem is that the current  submission does not include all of necessary details (please see the below), especially for the experimental section. 

Pros
-- well-written and organized
-- an unified probabilistic graphical model framework for generation and detection 
-- interesting experimental results   
Cons
The current submission lacks of some important details:
     The details of the ConvNet and its training process;
     How many steps of Gibbs samples executed in step 3 in fig.3? For a Gaussian RBM trained on face image the mixing of its markov chain could be extremely slow; 
     What do you mean 'training DBN by FPCD' in page 7? Do you mean wake sleep algorithm?
     How do you determine the threshold for logP(x|u,v)? I believe this threshold could be crucial for learning without gaze labels.
There are not any baseline for comparison in the section of experiments. Which part of the framework is more important? The DBN or the convNet? At least you could try to change DBN to a Gaussian mixture model (and/or change the ConvNet to a randomly guess). 
Is the Monte Carlo EM algorithm described in section 5 valid? Is there any guarantee? It can be seen in section 6.2 that the E-step could fail to localize faces. How could you prevent the DBN from learning those false faces.
To be honest, I am not convinced that a RBM (or DBN) trained on face images can be such a good generative model that always give higher free energy for non-face images. 

Minor comments
Last paragraph , Page 2: Combining two energy functions of RBMs forms a DBM.
Fig. 3: What do you mean about the arrows above the ConvNet in step 2 and step 4?
Section 6.2: please explain how you do inference in a DBN.