Learning generative models with visual attention
Authors: Charlie Tang, Nitish Srivastava, Ruslan Salakhutdinov
ICLR 2014

Abstract: Attention has long been proposed by psychologists as important for effectively dealing with the enormous sensory stimulus available in the neocortex. Inspired by visual attention models in computational neuroscience and by the need for deep generative models to learn on object-centric data, we describe a framework for generative learning using attentional mechanisms. Attentional mechanism propagate signals from region-of-interest in a scene to higher layer areas of canonical representation, where generative modeling takes place. By ignoring background clutter, generative model can concentrate its resources to model objects of interest. Our model is a proper graphical model where the 2D similarity transformation from computer vision is part of the top-down process. A ConvNet is used to initialize good guesses during posterior inference, which is based on Hamiltonian Monte Carlo. Upon learning on face images, we demonstrate that our model can robustly attend to face regions of novel test subjects. Most importantly, our model can learn generative models of new faces from a novel dataset of large images where the location of the face is not known.


Peer review:
[Review]:
The proposed solution is a conditional model (conditioned on a large input image I), which augments the typical RBM with two additional sets of random variables: transformations parameters $u$ and a 'steerable' visible layer $x$ (a patch of I, whose location is defined by $u$). An additional term in the energy function ensures that low-energy configurations are given to settings of $u$, for which $x$ is close to $v$ in an L2 sense. Inferring the transformation parameters in this model is thus analogous to the task of objection detection.  To shortcut this difficult optimization process the authors propose to use Hybrid Monte Carlo (HMC), but where the states $u$ are initialized via a convolutional neural network (CNN).

I find the general idea behind the paper compelling: the problems of attention and scaling up of generative models are important ones which warrant further attention. While a solid step in this direction, the experimental section does not convincingly show evidence in favor of the model, in particular, it lacks proper baselines.

Section 6.1: How do state of the art object detection / tracking algorithms do on this task ? Normalized cross-correlation and L2 template matching do not seem like appropriate baselines.

Section 6.2 aims to show that the 'novelty of our model is that it can learn on new large images of faces without label information'. Unfortunately, the authors provide little evidence in favor of this. After training a DBN and the approximate inference module on Caltech, the authors report a successfull detection rate of 951/1418, unfortunately without any context or comparisons.  How does this compare against a standard Viola-Jones detector ? How would the model perform by simply clamping 'v' to the average face image ?  I would not expect the proposed generative model to outperform a dedicated face detection module, but the baseline seems necessary to provide context.  Furthermore, how are the samples in Fig. 6b evidence that the DBN learnt the Multie-PIE dataset ?  Are the samples 'more' qualitatively similar to Multi-PIE than Caltech ?  Without being familiar with Multi-Pie, this is not at all obvious. Samples in 6b appear similar (but worse) to 6a. A quantitative analysis seems necessary.

The experiments of Section 6.3 is interesting at a high-level, but its execution seems flawed. It appears that the authors clamped 'v' to the very patch (i.e.  cropped face image from the test set) which they intend to detect (?). This seems like an all too trivial task. To highlight the benefits of having a proper generative model, v could have been clamped to e.g. the face of a different individual but of the same sex as the 'target'. A more convincing application might be with a classification RBM whereby clamping label units results in 'attending' to the corresponding areas of the image.

That being said, I do look forward to the next revision of the paper.


Other points:

* A threshold parameter is used for detection in Section 6.2. How was this parameter chosen ? Precision/Recall seems like the only relevant measure here.

* I found Section 3 to be particularly confusing, in large part due to the notation which obfuscates the relationship between $x$ and $u$. From the energy function, one could not be blamed for thinking that the model is a simple 3-layer DBM (with constant term f(u), to be rolled into the partition function).

* I also found the description of the transformation parameters and warp w a bit confusing. For instance, 'used to rotate, scale [...] the canonical v' is a bit misleading, as v never actually undergoes any transformation. At a high-level, it seems simpler to think of u as selecting a patch x of I (where the probability of u is proportional to the L2 distance between x and v) ? One possible suggestion to help with clarity would be to delay the description of the warp transform to Section 4, and describe the model in terms of a generic transform T(I, u) ?

* Overloaded notation for p (both pixel coordinate and momentum variable)

* Contradictory statements about HMC and local minima.

* lots of typos


PROS:
* novelty of model and potential for applications
* approximate inference scheme (which continues the trend of using function approximation for approximate inference)
CONS:
* weak experimental evidence (lack of proper baselines)
* clarity of presentation