Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Could you please clarify the notation and provide additional derivations for Sec 2 and 3 ? Typically, one works backwards from the energy function to derive the conditionals, and associated non-linearities. The work of Ravanbaksh et al 2016 seems to allow for the reverse direction (from non-linearity to energy), but I find that the authors do not provide sufficient background material for me to understand the approach. I believe the paper should as self-contained as possible. The notation and description could also be improved: f is clearly the non-linearity of the hidden units, yet it is described as "the conditional probability of the activation". What is the base measure g? \tilde{F}^* of Eq. 5 is not defined. In general, the paper could greatly benefit from a detailed appendix which shows how one goes from the leaky-relu non-linearity over h to Eqs 3-4-5.

I would also ask the authors to further discuss the projection method in the main body of the paper. What is the cost of this projection step ? If an SVD is required per gradient evaluation, this would make the algorithm cubic in the layer size, i.e. prohibitively expensive.

Regarding the results of Fig 4, why did the authors not plot the curves to convergence? A likelihood vs cpu-time curve would also help illustrate the cost of each sampling method, and account for the projection step. A direct comparison to Nair & Hinton 2010 is also noticeably absent.
review_title: Derivations and baselines

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors proposed to use leaky rectified linear units replacing binary units in Gaussian RBM.  A sampling method was presented to train the leaky-ReLU RBM. In the experimental section, AIS estimated likelihood on Cifar10 and SVHN were reported.

 It's interesting for trying different nonlinear hidden units for RBM. However, there are some concerns for the current work.
 1. The author did not explain why the proposed sampling method (Alg. 2) is correct. And the additional computation cost (the inner loop and the projection) should be discussed.
 2. The results (both the resulting likelihood and the generative samples) of Gaussian RBM are much worse than what we have experienced. It seems that the Gaussian RBM were not trained properly.
 3. The representation learned from a good generative model often helps the classification task when there are fewer label samples. Gaussian RBM works well for texture synthesis tasks in which mixing is an important issue. The authors are encouraged to do more experiments in these two direction.
review_title: null

final_score: rating: null
review_content: It would be nice if the authors can clarify the anti-derivative of f^-1 (right before equation 3)
Gaussian RBMs are notoriously hard to train, and the AIS estimation is depending on \sigma as well as the initial AIS factorial distribution. Also due to the fact that the visibles is continuous, the nats are log probability DENSITIES rather than log probability, which means they are very sensitive the the residual noise model \sigma.

Infact, the Stepped sigmoid units or ReLU from Nair and Hinton 10 was motivated by how to improve Gaussian RBMs.

A better and more convincing experiment is to use bernoulli visible units buy Relu/leaky relu hiddens as you have here, but train it on MNIST and report log-likelihood in Nats.
This is well studied and a typical RBM would get around -84 nats. In addition, samples of MNIST would be good as well.

review_title: Clarity and better comparisons

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a new variant of RBM, which has a nonlinearity of leaky ReLU, in contrast to the sigmoid function nonlinearity in RBM. By gradually annealing the leakiness coefficient (corresponding to from Gaussian to non-Gaussian model), the authors can sample from their model with a higher mixing rate. With the same idea annealing leakiness, they show they can estimate the partition function of the new model more accurately. 


Main comments:

The proposed model can only account for real-valued data. However, RBM is primarily used to model binary data, real-valued RBM (Gaussian-RBM) is not a well-recognized model for real-valued data. So, to demonstrate the superiority of the model, the author should also include the comparison with binary data. And it is also not enough to only compare two datasets for a newly proposed model.

The claim that the marginal distribution of visible variables is truncated Gaussian is incorrect. For a truncated normal, the values of variables are constrained to be within some region, e.g. requiring variable v from the region a1<v<a2. But here, there is no constraint on the visible v, i.e. v \in (-inf to +inf). The model just yields a marginal distribution which has a region-wise energy function, that is, for different regions, the energy functions are different. The authors should be aware of the claims of connection to truncated Gaussian.

That the authors claims that the model proposed by Nair & Hinton (2010) has no strict monotonicity and thus cannot use the Ravanbakhsh’s framework is incorrect. Nair & Hinton’s model use max(0, x+n) to introduce a ReLU-like nonlinearity. The output expectation is actually a strict monotonic increasing function of x.

Besides Nair & Hinton’s work, there is also another closely-related work ‘Unsupervised learning with truncated Gaussian graphical model’, which introduces ReLU into RBM using truncated normal.
review_title: some claims are incorrect, experiments should be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a novel energy-function for RBMs, using the leaky relu max(cx, x) activation function for the hidden-units. Analogous to ReLU units in feed-forward networks, these leaky relu RBMs split the input space into a combinatorial number of regions, where each region defines p(v) as a truncated Gaussian. A further contribution of the paper is in proposing a novel sampling scheme for the leaky RBM: one can run a much shorter Markov chain by initializing it from a sample of the leaky RBM with c=1 (which yields a standard multi-variate normal over the visibles) and then slowly annealing c. In low-dimension a similar scheme is shown to outperform AIS for estimating the partition function. Experiments are performed on both CIFAR-10 and SVHN.

This is an interesting paper which I believe would be of interest to the ICLR community. The theoretical contributions are strong: the authors not only introduce a proper energy formulation of ReLU RBMs, but also a novel sampling mechanism and an improvement on AIS for estimating their partition function. 

Unfortunately, the experimental results are somewhat limited. The PCD baseline is notably absent. Including (bernoulli visible, leaky-relu hidden) would have allowed the authors to evaluate likelihoods on standard binary RBM datasets. As it stands, performance on CIFAR-10 and SVHN, while improved with leaky-relu, is a far cry from more recent generative models (VAE-based, or auto-regressive models). While this comparison may be unfair, it will certainly limit the wider appeal of the paper to the community. Furthermore, there is the issue of the costly projection method which is required to guarantee that the energy-function remain bounded (covariance matrix over each region be PSD). Again, while it may be fair to leave that for future work given the other contributions, this will further limit the appeal of the paper.

PROS:
Introduces an energy function having the leaky-relu as an activation function
Introduces a novel sampling procedure based on annealing the leakiness parameter
Similar sampling scheme shown to outperform AIS

CONS:
Results are somewhat out of date
Missing experiments on binary datasets (more comparable to prior RBM work)
Missing PCD baseline
Cost of projection method

review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Based on previous work such as the stepped sigmoid units and ReLU hidden units for discriminatively trained supervised models, a Leaky-ReLU model is proposed for generative learning.

Pro: what is interesting is that unlike the traditional way of first defining an energy function and then deriving the conditional distributions, this paper propose the forms of the conditional first and then derive the energy function. However this general formulation is not novel to this paper, but was generalized to exponential family GLMs earlier.

Con: 
Because of the focus on specifying the conditionals, the joint pdf and the marginal p(v) becomes complicated and hard to compute.

On the experiments, it would been nice to see a RBM with binary visbles and leaky ReLu for hiddens. This would demonstrate the superiority of the leaky ReLU hidden units. In addition, there are more results on binary MNIST modeling with which the authors can compare the results to. While the authors is correct that the annealing distribution is no longer Gaussian, perhaps CD-25 or (Faast) PCD experiments can be run to compare agains the baseline RBM trained using (Fast) PCD.

This paper is interesting as it combines new hidden function with the easiness of annealed AIS sampling, However, the baseline comparisons to Stepped Sigmoid Units (Nair &Hinton) or other models like the spike-and-slab RBMs (and others) are missing, without those comparisons, it is hard to tell whether leaky ReLU RBMs are better even in continuous visible domain.


review_title: A new model of RBM is proposed, where the conditional of the hidden is a leaky ReLU. In addition an annealed AIS sampler is also proposed to test the learned models quantifiably

final_score: rating: null
review_content: Please plot some samples generated by Leaky-ReLU RBM. 
review_title: Why not plot generated samples by the Leaky-ReLU RBM? 

Meta-review: This paper identifies a joint distribution for an RBM variant based on leaky-ReLU activations. It also proposes using a sequence of distributions, both as an annealing-based training method, or to estimate log(Z) with AIS.
 
 This paper was borderline. While there is an interesting idea, the reviewers weren't generally as excited by the work as for other papers. One limitation is that unit-variance Gaussian RBMs aren't a strong baseline for comparison, although that is the focus of the main body of the paper. An update to the paper has results for binary visibles in an appendix, although I'm not sure exactly what was done, if the results are comparable, or if there is a large cost of projection here.
Decision: Reject