Title: Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM
Authors: Chun-Liang Li
Siamak Ravanbakhsh
Barnabas Poczos
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to numerical stability and quantifiability of the likelihood,  RBM is commonly used with Bernoulli units. Here, we consider an alternative member of exponential family RBM with leaky rectified linear units -- called leaky RBM.  We first study the joint and marginal distributions of leaky RBM under different leakiness, which provides us important insights by connecting the leaky RBM model and truncated Gaussian distributions.  The connection leads us to a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; -- i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations.  This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and more accurate than the commonly used annealed importance sampling (AIS).  We further demonstrate that the proposed sampling algorithm enjoys faster mixing property than contrastive divergence algorithm, which benefits the training without any additional computational cost.

TL;DR: We study fundamental property of leaky RBM. We link the leaky RBM and truncated Gaussian distribution and propose a novel sampling algorithm without additional computation cost.

Review discussion:
Please plot some samples generated by Leaky-ReLU RBM.
We will include the sampled images in the appendix. The revision will come in the next week.
However, one should note that single layer RBM does not adequately model CIFAR10 and SVHN (considered here) when compared to multilayer models.
Moreover the difference in sample quality (between leaky ReLU and conventional RBM) are not visually significant. We instead focused on quantitative evaluation of the log-likelihood in this paper (which shows a significant improvement).
Could you please clarify the notation and provide additional derivations for Sec 2 and 3 ? Typically, one works backwards from the energy function to derive the conditionals, and associated non-linearities. The work of Ravanbaksh et al 2016 seems to allow for the reverse direction (from non-linearity to energy), but I find that the authors do not provide sufficient background material for me to understand the approach. I believe the paper should as self-contained as possible. The notation and description could also be improved: f is clearly the non-linearity of the hidden units, yet it is described as "the conditional probability of the activation". What is the base measure g? \tilde{F}^* of Eq. 5 is not defined. In general, the paper could greatly benefit from a detailed appendix which shows how one goes from the leaky-relu non-linearity over h to Eqs 3-4-5.
I would also ask the authors to further discuss the projection method in the main body of the paper. What is the cost of this projection step ? If an SVD is required per gradient evaluation, this would make the algorithm cubic in the layer size, i.e. prohibitively expensive.
Regarding the results of Fig 4, why did the authors not plot the curves to convergence? A likelihood vs cpu-time curve would also help illustrate the cost of each sampling method, and account for the projection step. A direct comparison to Nair & Hinton 2010 is also noticeably absent.
1 WRT step-by-step derivation: We will revise the description of Sec 2 and Sec 3, and put more detailed derivation in the appendix. The revision will come in the next week.
2: WRT f: Note that the non-linearity $f$ of the unit translates to the mean of the conditional probability of the activation in ReLU RBM.
3 WRT base measure: The base measure g (aka carrier measure) in exponential family distributions ensures that the distribution is well-defined (this is necessary when using the conventional notation where the log-normalization constant, F, is defined to be the integral of the transfer function).
4 WRT projection step:  One of our contributions in this paper is to explicitly show that the SDP constraint is necessary for training leaky ReLU RBM when there is “unbounded” regions (see fig 1). We will move some of the Appendix C that elaborates on this to the main body. One standard way to deal with SDP constraint is using the SVD to do projection directly as we proved in Theorem 2. The reviewer is correct in pointing out the computational burden of SVD. We also observed that scaling the algorithm to thousands of hidden units is not practical.
Therefore, as we also point out in the conclusion, a more advanced optimization algorithm, such as Hsieh et al.’11, is highly motivated.
For example, the method of of Hsieh et al.’11 solves the Graphical lasso problem by avoiding the projection steps with barrier functions and  careful line search step.
However, we leave computational improvements on the projection step to future work.
5 WRT the figure 4: As we run algorithms to convergence,  the likelihood will improve as we show the Table 3. This portion of iterations were chosen for better visualization. We will add the result for comparing CPU time in the revision as requested. Please note that the complexity of CD and Mix are the almost the same. Mix only need a few more constant time steps which can be ignored compared with sampling steps. “Leaky” is more time-consuming because of computing and decomposing the covariance matrix as we discussed in Sec 5.
6 WRT Nair & Hinton’10: Based on Ravanbakhsh et al.’16, the standard ReLU does not result in valid exponential family RBM and therefore even if it defines a valid distribution we do not know a close form for its joint distribution p(v,h) and marginals p(v). This also means we cannot evaluate the likelihood of standard ReLU RBM as defined in Nair & Hinton’10. We are the first work to show the baseline performance of RBM with (leaky) ReLU activation function and demonstrate that it could outperform Boolean hidden units.
It would be nice if the authors can clarify the anti-derivative of f^-1 (right before equation 3)
Gaussian RBMs are notoriously hard to train, and the AIS estimation is depending on \sigma as well as the initial AIS factorial distribution.
Also due to the fact that the visibles is continuous, the nats are log probability DENSITIES rather than log probability, which means they are very sensitive the the residual noise model \sigma.
Infact, the Stepped sigmoid units or ReLU from Nair and Hinton 10 was motivated by how to improve Gaussian RBMs.
A better and more convincing experiment is to use bernoulli visible units buy Relu/leaky relu hiddens as you have here, but train it on MNIST and report log-likelihood in Nats.
This is well studied and a typical RBM would get around -84 nats.
In addition, samples of MNIST would be good as well.
Thanks for your feedback. We will clarify the role of the integral of f^{-1} in the text. The revision will come in next few days.
We are aware of the difficulties in training Gaussian RBM and similar to Nair&Hinton’10 our goal here is to improve Gaussian RBM.
However, as opposed to Nair&Hinton’10 we are interested in the “generative modelling” and proper evaluation of the likelihood. In all experiments, we use exponential family with “unit” dispersion parameter (corresponding to $\sigma=1$ for Gaussian), and we show consistent evaluation of Bernoulli and ReLU RBM by comparison to the “exact” partition function in some settings (see Table 2).
Note that we study the RBM with “Gaussian visible units”; we study the transformation from full-Gaussian (with efficient sampling) to Relu-Gaussian RBM and this sampling strategy as well as our truncated Gaussian interpretation does not carry over to Bernoulli visible units. Please see Section 3.1.
The authors propose a novel energy-function for RBMs, using the leaky relu max(cx, x) activation function for the hidden-units. Analogous to ReLU units in feed-forward networks, these leaky relu RBMs split the input space into a combinatorial number of regions, where each region defines p(v) as a truncated Gaussian.
A further contribution of the paper is in proposing a novel sampling scheme for the leaky RBM: one can run a much shorter Markov chain by initializing it from a sample of the leaky RBM with c=1 (which yields a standard multi-variate normal over the visibles) and then slowly annealing c. In low-dimension a similar scheme is shown to outperform AIS for estimating the partition function. Experiments are performed on both CIFAR-10 and SVHN.
This is an interesting paper which I believe would be of interest to the ICLR community. The theoretical contributions are strong: the authors not only introduce a proper energy formulation of ReLU RBMs, but also a novel sampling mechanism and an improvement on AIS for estimating their partition function.
Unfortunately, the experimental results are somewhat limited. The PCD baseline is notably absent.
Including (bernoulli visible, leaky-relu hidden) would have allowed the authors to evaluate likelihoods on standard binary RBM datasets. As it stands, performance on CIFAR-10 and SVHN, while improved with leaky-relu, is a far cry from more recent generative models (VAE-based, or auto-regressive models).
While this comparison may be unfair, it will certainly limit the wider appeal of the paper to the community.
Furthermore, there is the issue of the costly projection method which is required to guarantee that the energy-function remain bounded (covariance matrix over each region be PSD). Again, while it may be fair to leave that for future work given the other contributions, this will further limit the appeal of the paper.
PROS:
Introduces an energy function having the leaky-relu as an activation function
Introduces a novel sampling procedure based on annealing the leakiness parameter
Similar sampling scheme shown to outperform AIS
CONS:
Results are somewhat out of date
Missing experiments on binary datasets (more comparable to prior RBM work)
Missing PCD baseline
Cost of projection method
1. WRT binary dataset:
we put the result and the discussion in the revision. See Appendix F.3.  We study both toy datasets and two benchmarks, including MNIST and Caltech 101.
2. WRT PCD:
We put the result of PCD in the Appendix F.3. It shows PCD does not perform well as CD in the ReLU-RBM case though it has better performance in binary RBM case. Note that we show that the proposed “annealing leakiness” idea can also be used for learning. We observe that it has a better mixing than both CD and PCD, with no additional computation cost. Note that the additional cost of our training algorithm is due to projection (i.e. the model) rather than the new sampling procedure.
3. WRT to the projection cost:
see Appendix F.2.
Based on previous work such as the stepped sigmoid units and ReLU hidden units for discriminatively trained supervised models, a Leaky-ReLU model is proposed for generative learning.
Pro: what is interesting is that unlike the traditional way of first defining an energy function and then deriving the conditional distributions, this paper propose the forms of the conditional first and then derive the energy function.
However this general formulation is not novel to this paper, but was generalized to exponential family GLMs earlier.
Con:
Because of the focus on specifying the conditionals, the joint pdf and the marginal p(v) becomes complicated and hard to compute.
On the experiments, it would been nice to see a RBM with binary visbles and leaky ReLu for hiddens. This would demonstrate the superiority of the leaky ReLU hidden units.
In addition, there are more results on binary MNIST modeling with which the authors can compare the results to.
While the authors is correct that the annealing distribution is no longer Gaussian, perhaps CD-25 or (Faast) PCD experiments can be run to compare agains the baseline RBM trained using (Fast) PCD.
This paper is interesting as it combines new hidden function with the easiness of annealed AIS sampling, However, the baseline comparisons to Stepped Sigmoid Units (Nair &Hinton) or other models like the spike-and-slab RBMs (and others) are missing, without those comparisons, it is hard to tell whether leaky ReLU RBMs are better even in continuous visible domain.
1. WRT binary RBM result:
we put the result and the discussion in the revision. See Appendix F.3. We study both toy datasets and two benchmarks, including MNIST and Caltech 101.
2. WRT Baseline comparison (Nair & Hinton, and spike-and-slab RBM):
Based on Ravanbakhsh et al.’16, the standard ReLU does not result in valid exponential family RBM and therefore even if it defines a valid distribution we do not know a close form for its joint distribution p(v,h) and marginals p(v). This also means we cannot evaluate the likelihood of standard ReLU RBM as defined in Nair & Hinton’10. The heuristic sampling technique used in that paper samples using max(0, x+N(0, σ(x)). In expectation, this corresponds to a monotonic activation function f(x) = E[max(0, x+N(0, σ(x))], which does not have a simple analytic form.
Therefore we do not know the energy function associated with their sampling heuristic, which in turn prevents evaluation of the likelihood in that case. See also answer to a similar comment from reviewer 1. The other extension, such as spike-and-slab RBM, is based on Nair & Hinton’10. With the elementary calculation, the marginal distribution has one more integration without closed form than leaky RBM and it seems non-trivial to estimate it directly.
Therefore, the likelihood evaluation is also unknown to the community. In the original papers of both works, they all consider the classification results only.
We are the first work to show the baseline performance of RBM with the commonly-used (leaky) ReLU activation function and demonstrate that it could outperform Boolean hidden units.
3. WRT to the novelty of this paper:
The analysis of this paper is based on Ravanbakhsh et., al (2016) and Yang et., al (2012). The first contribution of this paper is connecting the leaky ReLU RBM and the notion of “union” of truncated Gaussian distribution (Section 3.1). We then propose the novel sampling strategy by utilizing the property of Gaussian distribution which benefits the partition function estimation (Section 4). The next contribution is we are able to give the baseline performance of RBM with the commonly-used (leaky) ReLU activation function in terms of the likelihood.We further show that the proposed sampling is better than the conventional Gibbs sampling by studying the optimization performance (Section 5).
The authors proposed to use leaky rectified linear units replacing binary units in Gaussian RBM.  A sampling method was presented to train the leaky-ReLU RBM. In the experimental section, AIS estimated likelihood on Cifar10 and SVHN were reported.
It's interesting for trying different nonlinear hidden units for RBM.
However, there are some concerns for the current work.
1. The author did not explain why the proposed sampling method (Alg. 2) is correct. And the additional computation cost (the inner loop and the projection) should be discussed.
2. The results (both the resulting likelihood and the generative samples) of Gaussian RBM are much worse than what we have experienced. It seems that the Gaussian RBM were not trained properly.
3. The representation learned from a good generative model often helps the classification task when there are fewer label samples. Gaussian RBM works well for texture synthesis tasks in which mixing is an important issue. The authors are encouraged to do more experiments in these two direction.
1. WRT the correctness and the computational cost:
For the correctness, please see Appendix E, where we discuss the connection between annealing the leakiness and energy. We show that annealing the leakiness is the special case of annealing the energy by properly choosing a special distributions as the initial distribution. The theory of general AIS has been studied by several literatures (e.g. Neal (1996)).
Therefore, those theoretical guarantees without strong assumptions on the initial distributions are all applicable to sampling by annealing leakiness.
For the computational concern, the inner loop is the same as standard CD except for one more flip-flops to decrease leakiness parameter in every iteration, which can be ignored compared with the projection step. The computational time of inner loop and the projection step is shown in Appendix F.2.
2. WRT the GRBM:
In Section 4.2, we mention our implementation can reproduce the exactly same results as the code provided by http://www.cs.toronto.edu/~tang/code/GaussianRBM.m
Although we tried to grid learning rate between 1e-1~1e-6 with hand-tuned learning rate decay schedule and the momentum parameter between 0.9~0.1, we cannot reproduce the result from the same group (e.g. Tang et., al (2012)).  As we mention in the paper, we conjecture the  unsatisfactory results are caused by the parameter tuning and well-designed learning weight decay. We would appreciate if anyone can provide the right parameters to us and we can rerun the experiments.This also shows the difficulty of training Gaussian RBM with Birnary units.
We thank all reviewers for their careful reading of the paper and constructive feedback. We have extensively revised the paper to address these requests for clarifications and experiments. Here, we explain some of these and point to related updates in our revision. Please check the individual replies below the reviews.

Meta-review:
role: PC
data: comment: This paper identifies a joint distribution for an RBM variant based on leaky-ReLU activations. It also proposes using a sequence of distributions, both as an annealing-based training method, or to estimate log(Z) with AIS.
 
 This paper was borderline. While there is an interesting idea, the reviewers weren't generally as excited by the work as for other papers. One limitation is that unit-variance Gaussian RBMs aren't a strong baseline for comparison, although that is the focus of the main body of the paper. An update to the paper has results for binary visibles in an appendix, although I'm not sure exactly what was done, if the results are comparable, or if there is a large cost of projection here.

Decision: Reject