On 2012-12-30 2:45 PM, "Vlad Niculae" <notifications@github.com> wrote:
>
> In sklearn/neural_networks/rbm.py:
>
> > +        self.intercept_visible_ = np.zeros(X.shape[1], dtype=X.dtype)
> > +        self.h_samples_ = np.zeros((self.n_particles,
self.n_components),
> > +            dtype=X.dtype)
> > +
> > +        inds = np.arange(X.shape[0])
> > +        self.random_state.shuffle(inds)
> > +
> > +        n_batches = int(np.ceil(len(inds) / float(self.n_particles)))
> > +
> > +        verbose = self.verbose
> > +        for iteration in xrange(self.n_iter):
> > +            pl = 0.
> > +            if verbose:
> > +                begin = time.time()
> > +            for minibatch in xrange(n_batches):
> > +                pl_batch = self._fit(X[inds[minibatch::n_batches]])
>
> Thanks for the input @dwf, I was just about to ping you :-)
>
> I do understand contrastive divergence and the idea behind PCD, and I
also understand why @amueller doesn't like it what was unclear to me was
the dynamics of the n_particles parameter. Also I was referring to its
description in the docstring, but since I'll be writing the narratives, it
is probably important for me to understand everything too.

Ah dammit, I just realized that I had omitted actual discussion of the
number of particles. The story is basically this: in CD it is common to run
as many 1 or K step chains as there are training examples in your mini
batch. This is because (in the one step case) you've already done half the
work for that in computing the positive phase gradients, so why not
threshold and do two more matrix multiplies.

Since the particles (Markov chain states) stick around in PCD, there is no
intrinsic reason that the sums in the positive phase and negative phase
terms need to have the same number of terms - they are approximating
expectations under different distributions, and there's no reason you need
the same number of terms in each.

> @ynd's code doesn't have any stopping condition, but an n_iter parameter
(that must be optimized via grid search). I will check how you do it in
your gist.

There isn't really any good answer, I think I just use a fixed number of
iterations too.

> Since I got your attention: is there any value in increasing the k in
CD-k after some time? IIUC it helps with the density modelling part (makes
the learning closer to actual ML) but that's not really important here is
it? The implementation here should IMHO be as close to black-box as
possible, and if users are on to something good they should use or extend
the one in pylearn.

I've not really heard of annealing K upwards, but it makes sense that it
would help somewhat. It's also far more expensive than PCD :) and the best
annealing schedule is not self-evident and would probably be problem
dependent.

There are definitely valid ways to use an RBM as a density model even
though it's intractable to compute the probability of a test example (e.g.
using the free energy of a test example to do outlier detection) for which
PCD training would make a gigantic difference. I can't remember the sklearn
unsupervised API at the moment but if there is something that gives you a
scalar score for each example passed in, the free energy would be the thing
to use for this (note that since the partition function is different for
different RBM instances this can *not* be used to do model comparison, but
it can be used to judge the log probability a given model assigns to a
point up to an additive constant, and judge an individual model's relative
preference for one example over another).

It depends whether you want to leave open these use cases, or only care
about feature extraction for classification.