Approximated Infomax Early Stopping: Revisiting Gaussian RBMs on Natural Images
Authors: Taichi Kiwaki, Takaki Makino, Kazuyuki Aihara
ICLR 2014

Abstract: We pursue early stopping that helps Gaussian Restricted Boltzmann Machines (GRBMs) to gain good natural image representations in terms of overcompleteness and data fitting. GRBMs are widely considered as an unsuitable model for natural images because they gain non-overcomplete representations which include uniform filters that do not represent sharp edges. We have recently found that GRBMs once gain and subsequently lose sharp edge filters during their training, contrary to this common perspective. We attribute this phenomenon to a tradeoff between overcompleteness of GRBM representations and data fitting. To gain GRBM representations that are overcomplete and fit data well, we propose approximated infomax early stopping for GRBMs. The proposed method enables huge performance boosts of classifiers trained on GRBM representations.


Peer review:
[Review]:
This is an interesting paper on Gaussian RBMs. The authors' explanation in Sec. 3 sounds convincing to me. The illustration in Fig. 2 (d) agrees well with my (and probably others') observation that the classification performance obtained by the features extracted by an RBM (either binary or Gaussian) does not correlate well with the test log-probabilities. Fig. 1 (a)-(d) agree as well with my own experience of training GRBMs on image patches. 

A similar idea of using the mutual information (or its upper-bound) to measure the usefulness of hidden neurons of an RBM/DBM was proposed by me and co-authors recently at ICONIP 2013 (Berglund et al., 2013). We were able to observe a similar trend of decreasing mutual information of quite a number of hidden neurons as training continues, which was also shown by the authors in Fig. 3 (B). The authors' approach of using this measure as an early-stopping criterion is very clever and seems to work well for the purpose of training a GRBM as a feature extractor. As the authors mention in Sec. 5.2.1, it is possible to use this measure for more complicated models such as DBM (which we tried in our paper), which makes this approach more interesting.

One potential point for improvement I can think of is to actually measure how other statistical quantities of a GRBM correlate with MI (or AI) as well. If the log-likelihood of a GRBM is measured, will one actually see the trend in Fig. 2 (d) (if we replace the overcompleteness with the MI or AI)? Also, since it's known that sparse RBM performs somewhat better as a feature extractor, how would the sparsity correlate with the MI or AI?

Another interesting/important point to discuss in the paper is the relationship to the autoencoders. Vincent et al. (2010) explains that 'training an autoencoder to minimize reconstruction error amounts to maximizing a lower bound on the mutual information between input X and learnt representation Y'. Does it mean that it is possible to simply use the 1-step reconstruction error as an early-stopping criterion instead of the proposed MI? If not, what would be some differences?

One minor point to consider In my opinion is that the term 'overcompleteness' is being somewhat abused without any precise definition. From the paper, I understood that the authors mean the (normalized) number of 'close-to-orthogonal' filters by 'overcompleteness', but this seems a bit too loose a definition (without any actual definition in the paper). Though, I guess the proposed measure itself can be used to measure the overcompleteness. 

= Refs = 
Berglund, M., Raiko, T. and Cho, K. Measuring the Usefulness of Hidden Units in Boltzmann Machines with Mutual Information. ICONIP 2013.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y. and Manzagol, P. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. JMLR. 2010.