Stopping Criteria in Contrastive Divergence: Alternatives to the Reconstruction Error
Authors: David Buchaca, Enrique Romero, Ferran Mazzanti, Jordi Delgado
ICLR 2014

Abstract: Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used to decide whether the approximation provided by the CD algorithm is good enough, though several authors (Schulz et al., 2010; Fischer & Igel, 2010) have raised doubts concerning the feasibility of this procedure. However, not many alternatives to the reconstruction error have been used in the literature. In this manuscript we investigate simple alternatives to the reconstruction error in order to detect as soon as possible the decrease in the log-likelihood during learning.


Peer review:
[Review]:
This paper introduces an early-stopping criteria, to address the potential degeneracy issues in CD-training of Restricted Boltzmann Machine. The method is closely related to the method of Free-Energy differences (FED), which computes the difference of log probabilities between the training set X and a held-out validation set Y. In lieu of a validation set however, the authors propose to sample Y to points far from the training data (for which the log-prob should monotonically decrease during training). To do this, the authors propose sampling Y from the conditional distribution p(x | h), where h is either drawn uniformly over binary states, or clamped to the 'complement' of the infered latent states when conditioned on training data, i.e. h = 1-h^(i) with $h^(i) ~ p(h | x^(i))$.

Given the similarities to FED, the method is not entirely novel. The idea of biasing the Monte Carlo approximation of log Z to states $v$ sampled from the conditional p(v|h) is interesting and reminiscent of [1,2]: a uniform prior over h could be a quick and dirty way to explore the space of visible configurations having some probability mass under the model. The second option seems less justified in my opinion: why use the complement of h^(i) instead of h^(i) itself ? The former would allow to quickly sample (local) perturbations of x^(i), which seems more inline with the CD training criteria (which only raises the energy of nearby configurations).

All in all the paper is interesting, but much better suited to the workshop track of the conference. The method is not entirely novel and the experiments are still very preliminary.
(1) The datasets are very small and would need to be scaled to more realistic datasets, using AIS as a proxy to the true likelihood.
(2) The evidence in favor of the proposed criteria is rather weak. The behavior of the first criterion changes completely based on the dataset, while the second criterion fails when using CD-10 (with no clear explanation why). Also, the early-stopping point obtained by (ii) on the LSE dataset remains rather approximate. Using this criterion would have yielded a much lower likelihood than possible.
(3) The authors seem unaware of the FED method and do not compare against it.

As the proposed criteria are based on heuristics, they require very solid empirical evidence to justify their use. As it stands, the paper simply does not deliver.

Other:
* paper glosses over another widely used early-stopping heuristic: classification error.
* superfluous citation to (Bengio, 2009) for efficient block Gibbs sampling of RBMs.
* latex bug for visible and hidden unit biases in Eq 5
* 'In doing so, drastic approximations [...] are performed'. Do not understand this sentence.
* 'making zeta when learning is achieved' ?? Learning is an optimization process not an discrete time event.
* 'A second possibility is to suitably compute [...] during data reconstruction'. Very clumsy way to say use 1-h^(i). The reconstruction phase conditions on h, therefore 'the value they should take during [...] reconstruction' is nonsensical.
* 'Anyway, the behavior of the proposed criteria [...] should be further studied.'. Very uninformative statement whose writing style seems *very* inappropriate for a paper submission.
* 'This westimator works well for CD1 but [not] for CD10, which is' - Missing [not]

[1] Grégoire Mesnil, Salah Rifai, Yann Dauphin, Yoshua Bengio and Pascal Vincent, Surfing on the Manifold, Learning Workshop, Snowbird, 2012.
[2] Yoshua Bengio, Grégoire Mesnil, Yann Dauphin and Salah Rifai, Better Mixing via Deep Representations, in: Proceedings of the 30th International Conference on Machine Learning (ICML'13), ACM, 2013