Approximated Infomax Early Stopping: Revisiting Gaussian RBMs on Natural Images
Authors: Taichi Kiwaki, Takaki Makino, Kazuyuki Aihara
ICLR 2014

Abstract: We pursue early stopping that helps Gaussian Restricted Boltzmann Machines (GRBMs) to gain good natural image representations in terms of overcompleteness and data fitting. GRBMs are widely considered as an unsuitable model for natural images because they gain non-overcomplete representations which include uniform filters that do not represent sharp edges. We have recently found that GRBMs once gain and subsequently lose sharp edge filters during their training, contrary to this common perspective. We attribute this phenomenon to a tradeoff between overcompleteness of GRBM representations and data fitting. To gain GRBM representations that are overcomplete and fit data well, we propose approximated infomax early stopping for GRBMs. The proposed method enables huge performance boosts of classifiers trained on GRBM representations.


Peer review:
[Review]:
In this paper the authors observe an interesting training dynamics when training GRBMs on CIFAR patches: filters first become selective to the usual Gabor looking features but then many faint and decay to 0 towards the end of training. From this observation they propose to measure an approximation to the mutual information between input and hidden units to detect the time when most features are used. They demonstrate that this is an effective method to automatically select the best checkpoint to use for discrimination on CIFAR dataset.

Pros
- sufficient novelty: I am not aware of other works making this observation and proposing this measurement of goodness of a trained GRBM
- relevance: the problem of automatically selecting good features is very relevant many working on unsupervised learning (one of the main themes of this conference).

Cons
- the paper is not very clear. I find the authors' use of words like 'overcomplete', 'useful', etc. very vague. I will explain better below what I did not quite understand.
- technically I am not sure the paper is correct; I did not follow how the authors measure mutual information exactly. More details below.
- the impact of this work seems rather limited: the empirical validation is done on CIFAR only showing modest improvements.

I will expand here on what I did not understand or did not agree.
First of all, the fluency of English should be improved. The choice of words is often too vague. For instance, in the abstract the authors say:
“ gain non-overcomplete representations which include uniform ﬁlters that do not represent useful image features”.
What does this mean? What are 'uniform filters' or “ useful ﬁlters“? It would be nice if every part of the paper was self-explanatory, concise yet precise in its meaning.

Second, I am not convinced by the arguments about why filters decay to zero. From fig. 1, it seems to me that filters do not decay to zero at all. The range of values of filters that become low-pass shrinks, which is rather different and perhaps expected. Most energy is in the low frequencies, however hidden units are forced to be in {0,1} which forces the low-pass (less localized) filters to rescale their values to a smaller range. If the authors rescaled each filter independently they may see a different picture (please, verify if this is correct).
Besides, it would be useful to get an estimate of the log-probability of the training data during training: is that steadily increasing? The monotonic behavior of FED seems to show an overfitting effect. I am not sure the hypothesis chosen by the authors in the introduction is the only one that can explain their observations.
Similarly in sec. 3, I do not understand the authors' conclusions. The number of hidden units need not to match the dimensionality of the data. Without any regularization, the highest likelihood on the training set would be achieved by having each hidden unit represent a single training sample, and therefore, the highest likelihood can be achieved by having as many hidden units as training samples (although this would be horrible on the test set).
From this perspective, I did not understand what happened in the example of fig. 2A.
Did the authors use any regularization? How did the authors measured 'overcompleteness' and 'fitness'? 

Third, I did not understand how the authors computed mutual information.
First, mutual information assumes a joint distribution over data and hidden units. Are the authors replacing p_GRBM(v) with  p_data(v)? 
Moreover, I did not follow how the authors compute S_D(H). Shouldn't it be the entropy of the hidden units? which means marginalizing the visibles to get p_GRBM(h) and then computing the entropy. The authors should report the formula they used and explicitly mention the approximations used. In the current draft, I do not understand how S_D(H) is computed or differs from the conditional entropy term.
For the sake of clarity, I would recommend to include in the main body of the paper the definition of bar{AMI} and tilda{AMI} as well.

Finally, empirical evidence is shown only on the CIFAR dataset. It is not obvious this is a general finding/method. 

Overall, this is an interesting paper. Unfortunately, I did not fully understand it nor I was fully convinced by its claims. I recommend a good rewrite of the paper to address these concerns. In general, the authors mention that this method is able to make GRBM a better model of natural images and contrast it to other models like SS-RBM, gMRF, etc. But actually, what they propose has little to do with these other methods (and it could be applied to these methods as well). The proposed method is about extracting discriminative features by measuring a quantity that correlates with number of non-dead filters. The question is then: are there even simpler measures (like variance across samples of the corresponding filter responses or variance of the coefficients in the filters)?