Stopping Criteria in Contrastive Divergence: Alternatives to the Reconstruction Error
Authors: David Buchaca, Enrique Romero, Ferran Mazzanti, Jordi Delgado
ICLR 2014

Abstract: Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used to decide whether the approximation provided by the CD algorithm is good enough, though several authors (Schulz et al., 2010; Fischer & Igel, 2010) have raised doubts concerning the feasibility of this procedure. However, not many alternatives to the reconstruction error have been used in the literature. In this manuscript we investigate simple alternatives to the reconstruction error in order to detect as soon as possible the decrease in the log-likelihood during learning.


Peer review:
[Review]:
This paper investigates stopping criteria for training of RBMs by contrastive
divergence (CD). Traditional reconstruction error is compared to a ratio of
probabilities, namely the probabilities of the training data divided by the
probabilities of an equal number of sampled points (with two variants of the
sampling strategy). By dividing probabilities, the partition function cancels
out, which makes computations tractable. Experiments on two toy datasets show
that the two proposed variants are overall more useful than reconstruction
error to identify the point of maximum likelihood on training data, even though
they do not work on all cases investigated here.

The problem of early stopping of RBM training is definitely a relevant one,
since the intractability of the partition function prevents properly monitoring
the likelihood. The approach suggested here, however, lacks in both theoretical
motivation and empirical evaluation, thus in my opinion is not quite ready for
publication.

On the theoretical side, the proposed criterion (eq. 8) is only heuristicallymotivated. Note in particular that a model might be able to reach an arbitraryhigh value of this criterion if P(y_i) -> 0 for some y_i, regardless or howbadly it may perform on the training x_i's. I'm also not sure why one would
necessarily take N y_i's: why not generalize it to M y_i's, using Pi_i
P(y_i)^(N/M), so as to be able to choose the sample size based on available
computational resources? Finally, I am not convinced by the choice of y_i's:
first, the sampling rules (random h or '1 - h_i') are not well motivated (there
is no guarantee that we will not sample training points), second, since they
are defined from the model parameters they lead to a set of y_i's that evolves
during training (thus the criterion may be unstable), third the y_i's are
expectations and there is no explanation on whether it makes sense for binary
RBMs.

On the empirical side, my first concern is that experiments are performed
on low-dimensional toy datasets, and there is nothing to tell us that 
behavior observed on such datasets will actually translate into higher
dimensional tasks. In particular, sampling-based methods tend to behave rather
nicely in low dimension, but may break horribly as the dimension increases...
Thus it would have been good to add experiments in high dimension, for instance
using AIS to estimate the partition function. My second concern is that only
training errors are reported: although they are definitely interesting to
monitor, someone using reconstruction error as a stopping criterion will always
use a validation set for this, and will hope to stop at a point where validation
log-likelihood is maximized. The comparisons in the paper are thus, for the
most part, uninformative, since they only use the training data.

A few more minor points:
- Eq. 5 is missing some characters
- The number of hidden units is not mentioned in the experiments
- Plots show 'reconstruction error' as something that is better when it increases,
  which is counter-intuitive for an error
- Something potentially worth discussing is that RBMs are often used for
  pre-training purpose in deep networks, and it is not clear that better
  likelihood => better pre-training (if there is work on this topic, it should
  be cited, as it is important to motivate this direction of research)
- Another application worth mentioning to this kind of technique is model
  selection (which RBM is best?) => the proposed criterion may require a bit
  of tweaking to answer this kind of question (common y_i's are needed)