Stopping Criteria in Contrastive Divergence: Alternatives to the Reconstruction Error
Authors: David Buchaca, Enrique Romero, Ferran Mazzanti, Jordi Delgado
ICLR 2014

Abstract: Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used to decide whether the approximation provided by the CD algorithm is good enough, though several authors (Schulz et al., 2010; Fischer & Igel, 2010) have raised doubts concerning the feasibility of this procedure. However, not many alternatives to the reconstruction error have been used in the literature. In this manuscript we investigate simple alternatives to the reconstruction error in order to detect as soon as possible the decrease in the log-likelihood during learning.


Peer review:
[Review]:
Dear Kyunghyun  Cho,

we appreciate very much the effort you've made to carefully read our
preprint.  After discussing the topics you've raised we have tried to
work out a new version of the paper that will (hopefully) clarify some
aspects of our work. In the meantime we want to answer (at least some)
of your comments and questions.

- The proposed stopping criterion itself is not so novel, in my
  opinion. Essentially, it is a general framework in which, for
  instance, CD minimizes, where $y^{(i)}$ is obtained by a single-step
  Gibbs sampling from $x^{(i)}$. In fact, if $y^{(i)}$ is one step of
  persistent MCMC chain, the stopping criterion corresponds to the
  approximate maximum likelihood criterion used by PCD. In short, the
  proposed criterion is, in fact, what a proper MLE should minimize.

--- We are not sure to understand your comment regarding PCD.  To our
    understanding, PCD may also lead to a decreasing log-likelihood,
    and the reconstruction error is not a good stopping criterion for
    PCD either (see for instance Fischer & Igel, 2010). In PCD, the
    value of $y^{(i)}$ is different from the ones defined in our
    paper, and it is used for other purposes. Still, we agree with you
    that the idea of computing $y^{(i)}$ in a single Gibbs step (in
    whatever the recipe you use) and get a statistical estimator from
    them, is not so novel (actually CD1 does that already).  As you
    mention below, the novelty lies on the way we choose $y^{(i)}$.
    But in any case we agree we can make more emphasis on that aspect
    in a newer version of the paper.

- The novelty is, however, in the proposed ways to choose
  $y^{(i)}$. The authors proposed two alternatives. Unfortunately (as
  somewhat expected) the first method of selecting random points does
  not work, which is only natural as if it did, why would anyone use
  MCMC to estimate the model statistics of RBM?  The second
  alternative of using the visible samples conditioned on the
  *flipped* hidden states seems to work better, but the authors do not
  provide any good justification for this choice. 

--- Our justifications are more based on intuitions. We are currently
    working on the mathematical grounds behind the ideas, but this is
    still work in progress.  But in any case the experimental results
    are promising and seem to support our conclusions, which is the
    main reason why we submitted the paper.  In our mind, however, the
    idea behind  the work is slightly different from what you point
    out. We believe the new stopping criteria will work better in
    large spaces where the subset of states having a non-negligible
    probability is small. In these systems, only very precise
    combinations of weights and unit values leads to large
    probabilities.  If you change the value of the hidden units in any
    way, this delicate equilibrium is broken and you expect to get
    much lower probabilities for the visible variables. And you can do
    that by either getting the proper value of the hidden units and take
    their complementary (the criterion that works better, apparently),
    or simply by changing them at random. In the learning process, one uses
    MCMC because, in general, one wants to do the opposite. That is, to
    identify the (small) regions of large probability, which is a
    difficult task if the space is large and the set of relevant
    states is small. But that implies that identifying states with low
    probability should be much easier, and our hope is to find any of
    these by generating values of the hidden variables that are very
    different from the right ones. We believe that this is not clearly
    seen on the experiments reported because the size of the spaces
    analyzed is not that large. We plan to extend our numerical
    analysis to other, larger systems where this should be more
    evident.

- The experiments show that the second choice is superior, but since
  the experiments are rather small-scale, it is difficult to make a
  solid conclusion from them. Also, the proposed method seems to only
  work with CD_1 (see Fig. 4), but the authors' explanation on
  possible reasons for this is not clear. 

--- Actually we are mostly interested in CD_1 as it is the cheapest
    way we know (with respect to time and memory storage) to evaluate
    the correlations required at the learning stage. We must admit,
    however, that the reason why it seems to perform worse in CD_k is
    somewhat unclear to us. As we said above, we are working on the
    mathematical grounds of the method and we hope to have a more
    clear understanding of the fine details of the method in the near
    future. 

- One thing which is clearly missing from the experiments is the
  comparison of the proposed criterion against the actual function CD
  minimizes ($E_D left[ log p (x^{(i)}) - log p(y^{(i)}) ight]$,
  where $y^{(i)}$ is the one-step Gibbs sample starting from
  $x^{(i)}$.) as well as the approximate likelihood (the same thing,
  but use samples from persistent chains as $y^{(i)}$'s). Computing
  these things should be no more expensive than computing the proposed
  criterion, and may give better indication of how log-likelihood
  evolves over time. This will be interesting to see. 

--- We have some of these results, but we decided not to include them
    in the paper to limit its extensions. In any case we have seen
    that the behavior is similar to the reconstruction error, and thus
    show the same problems once compared to the likelihood itself.  In
    the paper we only included 'somewhat positive' results, but we
    agree with you that these results may be interesting to be
    discussed. 

- Of course, one last thing is, why anyone would try to train an RBM
  to be a good generative model (maximizing the log-likelihood) using
  CD in the first place, while PCD (with possibly other MCMC
  algorithms than Gibbs) is available.

 --- The paper is not trying to answer this question. We just want to
     explore new ways to improve the stopping criteria in a learning
     algorithm where the reconstruction error does not work properly
     (in general), and we have tested it in CD_1. Since the
     reconstruction error is not a good stopping criterion for PCD
     either, PCD may also benefit from other options. It would be
     interesting to see if the same stopping criteria work properly
     with PCD, but that we leave for a future work.

-Despite these seemingly negative comments, I enjoyed reading the
 paper, since it is clearly written and well motivated. A few more
 experiments showing that the proposed methods of choosing $y^{(i)}$
 are better than other possible choices (e.g., using Gibbs sampling) 
 would make the paper much stronger. 

--- Thanks a lot for the comments and suggestions. We agree with you
    that there is more room to improve, and we are already working on
    that.  But at this point we didn't want to miss the opportunity to
    submit or recent results to this conference in order to discuss
    our ideas with other people with lots of expertise in the
    field. Still we hope to follow this line and to get more
    exhaustive results in the near future.

- P. S. I wouldn't mind, if you cited my paper (IJCNN 2013) next to
 (Desjardins et al., 2010) where I also proposed PT. 

--- Consider it done