Approximated Infomax Early Stopping: Revisiting Gaussian RBMs on Natural Images
Authors: Taichi Kiwaki, Takaki Makino, Kazuyuki Aihara
ICLR 2014

Abstract: We pursue early stopping that helps Gaussian Restricted Boltzmann Machines (GRBMs) to gain good natural image representations in terms of overcompleteness and data fitting. GRBMs are widely considered as an unsuitable model for natural images because they gain non-overcomplete representations which include uniform filters that do not represent sharp edges. We have recently found that GRBMs once gain and subsequently lose sharp edge filters during their training, contrary to this common perspective. We attribute this phenomenon to a tradeoff between overcompleteness of GRBM representations and data fitting. To gain GRBM representations that are overcomplete and fit data well, we propose approximated infomax early stopping for GRBMs. The proposed method enables huge performance boosts of classifiers trained on GRBM representations.


Peer review:
[Review]:
This paper is in the domain of using GRBMs to train filters which can then be convolved and pooled before an L2-SVM is used for classification. The overall idea of finding degraded classification results after prolonged RBM training is an interesting one.  The authors then proposes an information criterion for early stopping. The paper is well written and makes link between AMI and filter quality and classification performance.

The fact that GRBM lose useful filters might be due to contrastive divergence training. Also, the concept of overfitting is hard to decipher here because learning GRBM filters is not for directly reducing classification error.

The argument that validation error is too expensive to compute is only unique to a framework where a probabilistic model is used to pretrain filters, which are then used in a discriminative pipeline.  For majority of discriminative based models, this argument would not apply and early stopping could simply be performed based on validation error.