Title: ANNEALING GAUSSIAN INTO RELU: A NEW SAM-PLING STRATEGY FOR LEAKY-RELU RBM

Abstract: Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to its numerical stability and quantifiability of its likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of the exponential family RBM with leaky rectified linear units -called leaky RBM. We first study the joint and marginal distributions of the leaky RBM under different leakiness, which leads to interesting interpretation of the leaky RBM model as truncated Gaussian distribution. We then propose a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; -i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and far more accurate than the commonly used annealed importance sampling (AIS). We further demonstrate that the proposed sampling algorithm enjoys relatively faster mixing than contrastive divergence algorithm, which improves the training procedure without any additional computational cost.

1. INTRODUCTION:

In this paper, we are interested in deep generative models. One may naively classify these models into a family of directed deep generative models trainable by back-propagation (e.g., Kingma & Welling, 2013; Goodfellow et al., 2014) , and deep energy-based models, such as deep belief network (Hinton et al., 2006) and deep Boltzmann machine (Salakhutdinov & Hinton, 2009) . The building block of deep energy-based models is a bipartite graphical model called restricted Boltzmann machine (RBM). The RBM model consists of two layers, visible and hidden. The resulting graphical model which can account for higher-order interactions of the visible units (visible layer) using the hidden units (hidden layer). It also makes the inference easier that there are no interactions between the variables in each layer.

The conventional RBM uses Bernoulli units for both the hidden and visible units (Smolensky, 1986) . One extension is using Gaussian visible units to model general natural images (Freund & Haussler, 1994) . For hidden units, we can also generalize Bernoulli units to the exponential family (Welling et al., 2004; Ravanbakhsh et al., 2016) . Nair & Hinton (2010) propose a variation using Rectified Linear Unit (ReLU) for the hidden layer with a heuristic sampling procedure, which has promising performance in terms of reconstruction error and classification accuracy. Unfortunately, due to its lack of strict monotonicity, ReLU RBM does not fit within the framework of exponential family RBMs (Ravanbakhsh et al., 2016) . Instead we study leaky-ReLU RBM (leaky RBM) in this work and address two important issues i) a better training (sampling) algorithm for ReLU RBM and; ii) a better quantification of leaky RBM -i.e., evaluation of its performance in terms of likelihood.

We study some of the fundamental properties of leaky RBM, including its joint and marginal distributions (Section 2). By analyzing these distributions, we show that the leaky RBM is a union of truncated Gaussian distributions. In this paper, we show that training leaky RBM involves underlying positive definite constraints. Because of this, the training can diverge if these constrains are not satisfied. This is an issue that was previously ignored in ReLU RBM, as it was mainly used for pre-training rather than generative modeling.

Our contribution in this paper is three-fold: I) we systematically identify and address model constraints in leaky RBM (Section 3); II) for the training of leaky RBM, we propose a meta algorithm for sampling, which anneals leakiness during the Gibbs sampling procedure (Section 3) and empirically show that it can boost contrastive divergence with faster mixing (Section 5); III) We demonstrate the power of the proposed sampling algorithm on estimating the partition function. In particular, comparison on several benchmark datasets shows that the proposed method outperforms the conventional AIS (Salakhutdinov & Murray, 2008) in terms of efficiency and accuracy (Section 4). Moreover, we provide an incentive for using leaky RBM by showing that the leaky ReLU hidden units perform better than the Bernoulli units in terms of the model log-likelihood (Section 4).

2. RESTRICTED BOLTZMANN MACHINE AND RELU:

The Boltzmann distribution is defined as p(x) = e -E(x) /Z where Z = x e -E(x) is the partition function. Restricted Boltzmann Machine (RBM) is a Boltzmann distribution with a bipartite structure It is also the building block for many deep models (e.g., Hinton et al., 2006; Salakhutdinov & Hinton, 2009; Lee et al., 2009) , which are widely used in numerous applications (Bengio, 2009) . The conventional Bernoulli RBM, models the joint probability p(v, h) for the visible units v ∈ [0, 1] I and the hidden units

h ∈ [0, 1] J as p(v, h) ∝ exp(-E(v, h)), where E(v, h) = a v -v W h + b h. The parameters are a ∈ R I , b ∈ R J and W ∈ R I×J .

We can derive the conditional probabilities as

EQUATION

where σ(x) = (1 + e -x ) -1 is the sigmoid function.

One extension of Bernoulli RBM is replacing the binary visible units by linear units v ∈ R I with independent Gaussian noise. The energy function in this case is given by

E(v, h) = I i=1 (v i -a i ) 2 2σ 2 i - I i=1 J j=1 v i σ i W ij h j + b h.

To simplify the notation, we assume a normalized data so that a i and σ i is no longer required. The energy function is accordingly simplified to E(v, h) = v 2 2 -v W h + b h (Note that the elimination does not influence the discussion and one can easily extend all the results in this paper to the model that includes a i and σ i .).

The conditional distributions are as follows:

EQUATION

where N (µ, V ) is a Gaussian distribution with mean µ and variance V . To simplify the notation, in the following we define η j = I i=1 W ij v i + b j -that is η j is the input to the j th hidden layer neuron and similarly define ν i = J j=1 W ij h j + a i . Using this notation the conditionals in the (2) are p(v i |ν i ) = N (ν i , 1) and p(h j = 1|η j ) = σ(η j ).

2.1. RELU RBM WITH CONTINUOUS VISIBLE UNITS:

From (1) and (2), we can see that the mean of the p(h j |v) is the nonlinearity of the hidden unit at η j = I i=1 W ij v i + b j -e.g., mean of the Bernoulli unit is the sigmoid function. From this perspective, we can extend the sigmoid function to other functions and thus allow RBM to have more expressive power (Ravanbakhsh et al., 2016) . In particular, it would be interesting to use rectified linear unit (ReLU) nonlinearity, f (η j ) = max(0, η j ), for generative modeling. Nair & Hinton (2010) use an RBM with visible Gaussian unit and ReLU hidden activation functions for pretraining. They suggest sampling from max(0, η j +N (0, σ(η j )) for conditional sampling from the hidden units (compare to (2)). However, this sampling heuristic does not suggest the parametric form of the joint ReLU-Gaussian distribution. This also means we cannot evaluate it using methods such as Annealed Importance Sampling that require access to this parametric form. In fact, only strictly monotonic activation functions can derive feasible joint and conditional distributions in the exponential familly RBM and ReLU is not strictly monotonic Ravanbakhsh et al. (2016) . Similar activation functions that are monotonic are Softplus, f (η j ) = log(1 + e ηj ) and leaky ReLU (Maas et al., 2013) , defined as f (η j ) = max(cη j , η j ), where c ∈ (0, 1) is the leakiness parameter. In contrast to the ReLU RBM the joint parametric form of these two distributions are available. However, the energy (logarithm of the joint probability) in the case of Softplus activation function contains a polylogarithmic term that requires evaluation of an infinite series; see Table 1 in Ravanbakhsh et al. (2016) . For this reason, here we focus on Leaky-ReLU activation function.

By Ravanbakhsh et al. (2016) , the conditional probability of the activation, assuming the nonlinearity f (η j ), is generally defined as p(h j |v) = exp (-D f (η j h j ) + g(h j )), where D f (η j h j ) is the Bregman Divergence associated with f, and g(h j ) is the base (or carrier) measure in the exponential family which ensures the distribution is well-defined. The Bergman divergence, for strictly monotonic function f , is D f (η j h j ) = -η j h j + F (η j ) + F * (h j ), where F with d dηj F (η j ) = f (η j ) is the anti-derivative (integral) of f and F * is the anti-derivative of f -1 (i.e., f -1 (f (η)) = η); Note that due to the strict monotonicity of f , f -1 is well-defined, and F and F * are commonly referred to as conjugate duals.

Considering the leaky ReLU activation function f (η) = max(cη, η), using this formalism, the conditional distributions of hidden units in the leaky RBM simplifies to (see Appendix A.1 for details)

EQUATION

Since the visible units uses the identity function, the corresponding conditional distribution is a Gaussian 1

EQUATION

Having these two conditional distributions is enough for training a leaky RBM model using contrastive divergence (Hinton, 2002) or some other alternatives (e.g., Tieleman, 2008; Tieleman & Hinton, 2009) .

3. TRAINING AND SAMPLING FROM LEAKY RBM:

Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model is (Yang et al., 2012; Ravanbakhsh et al., 2016 )

EQUATION

where F * (v i ) and F * (h j ) are anti-derivatives of the inverses of the activation functions f (v i ) and f (h j ) for visible units v i and hidden units h j , respectively (see Section 2.1). Assuming f (η j ) = max(cη j , c) and f (ν i ) = ν i in leaky-ReLU RBM, the joint distribution above becomes (see Appendix A.2 for details)

p(v, h) ∝ exp   v W h - v 2 2 - ηj >0 h 2 j 2 + log √ 2π - ηj ≤0 h 2 j 2c + log √ 2cπ + b h   ,

1 which can also be written as p(vi|h) = exp -D f (νi vi) + g(vi) , where νi = j=1 Wijhj and and the corresponding visible marginal distribution is

f (νi) = νi and D f (νi vi) = (νi -vi) 2 and g(vi) = -log √ 2π. W 1 W 2 W 3 R1 R2 R3 R4 R5 R6 R7

p(v) ∝ exp   - 1 2 v   I - ηj >0 W j W j -c ηj ≤0 W j W j   v + ηj >0 b j W j v + c ηj ≤0 b j W j v   .

(6) where W j is the j-th column of W .

3.1. LEAKY RBM AS UNION OF TRUNCATED GAUSSIAN DISTRIBUTIONS:

From ( 6) we see that the marginal probability is determined by the affine constraints η j > 0 or η j ≤ 0 for all hidden units j. By combinatorics, these constraints divide R I (the visible domain)

into at most M = I i=1 J i convex regions R 1 , • • • R M .

An example with I = 2 and J = 3 is shown in Figure 1 . If I > J, then we have at most 2 J regions.

We discuss the two types of these regions. For bounded regions, such as R 1 in Figure 1 , the integration of (6) is also bounded, which results in a valid distribution. Before we discuss the unbounded cases, we define Ω = I -J j=1 α j W j W j , where α j = 1 ηj >0 + c1 ηj ≤0 . For the unbounded region, if Ω ∈ R I×I is a positive definite (PD) matrix, then the probability density is proportional to a multivariate Gaussian distribution with mean µ = Ω -1 J j=1 α j b j W j and precision matrix Ω (covariance matrix Ω -1 ) but over an affine-constrained region. Therefore, the distribution of each unbounded region can be treated as a truncated Gaussian distribution. The marginal distrubution can be treated as a union of truncated Gaussain distribution. Note that leaky RBM is different from Su et al. (2017) , which use single truncated Gaussian distribution to model joint (conditional) distributions and require approximated and more complicated sampling algorithms for truncated Gaussian distribution, while leaky RBM only requires to sample from Gaussian distributions.

On the other hand, if Ω is not PD, and the region R i contains the eigenvectors with negative eigenvalues of Ω, the integration of ( 6) over R i is divergent (infinite), which can not result in a valid probability distribution. In practice, with this type of parameter, when we do Gibbs sampling on the conditional distributions, the sampling will diverge. However, it is unfeasible to check exponentially many regions for each gradient update.

Theorem 1. If I -W W is positive definite, then I -j α j W j W j is also positive definite, for all α j ∈ [0, 1].

The proof is shown in Appendix 1. From Theorem 1 we can see that if the constraint I -W W is PD, then one can guarantee that the distribution of every region is a valid truncated Gaussian distribution. Therefore, we introduce the following projection step for each W after the gradient update.

argmin

W W -W 2 F s.t. I -W W 0 (7)

Theorem 2. The above projection step (7) can be done by shrinking the singular values to be less than 1.

The proof is shown in Appendix C. The training algorithm of the leaky RBM is shown in Algorithm 1. By using the projection step (7), we could treat the leaky RBM as the union of truncated Gaussian distributions, which uses weight vectors to divide the space of visible units into several regions and use a truncated Gaussian distribution to model each region. Note that the leaky RBM model is different from Su et al. (2016) , which uses a truncated Gaussian distribution to model the conditional distribution p(h|v) instead of the marginal distribution.

The empirical study about the divergent values and the necessity of the projection step is shown in Appendix D. Without the projection step, when we run Gibbs sampling for several iterations from the model, the sampled values will diverge because the model does not have a valid marginal distribution p(v). It also implies that we cannot train leaky RBM with larger CD steps without projection, which would result in divergent gradients. The detailed discussion is shown in Appendix D.

Algorithm 1 Training Leaky RBM for t = 1, . . . , T do Estimate gradient g θ by CD or other algorithms with ( 13) and ( 4), where θ = {W, a, b}. θ (t) ← θ (t-1) + ηg θ . Project W (t) by ( 7). end for

3.2. SAMPLING FROM LEAKY-RELU RBM:

Gibbs sampling is the core procedure for RBM, including training, inference, and estimating the partition function (Fischer & Igel, 2012; Tieleman, 2008; Salakhutdinov & Murray, 2008) . For every task, we start from randomly initializing v by an arbitrary distribution q, and iteratively sample from the conditional distributions. Gibbs sampling guarantees the procedure result in the stationary distribution in the long run for any initialized distribution q. However, if q is close to the target distribution p, it can significantly shorten the number of iterations to achieve the stationary distribution.

If we set the leakiness c to be 1, then (6) becomes a simple multivariate Gaussian distribution N (I -W W ) -1 W b, (I -W W ) -1 , which can be easily sampled without Gibbs sampling. Also, the projection step (7) guarantees it is a valid Gaussian distribution. Then we decrease the leakiness with a small , and use samples from the multivariate Gaussian distribution when c = 1 as the initialization to do Gibbs sampling. Note that the distribution of each region is a truncated Gaussian distribution. When we only decrease the leakiness with a small amount, the resulted distribution is a "similar" truncated Gaussian distribution with more concentrated density. From this observation, we could expect the original multivariate Gaussian distribution serves as a good initialization. The one-dimensional example is shown in Figure 2 . We then repeat this procedure until we reach the target leakiness. The algorithm can be seen as annealing the leakiness during the Gibbs sampling procedure. The meta algorithm is shown in Algorithm 2. Next, we show the proposed sampling algorithm can help both the partition function estimation and the training of leaky RBM.

Algorithm 2 Meta Algorithm for Sampling from Leaky RBM.

Sample v from N (I -W W ) -1 W b, (I -W W ) -1 = (1 -c)/T and c = 1 for t = 1, . . .

, T do:

Decrease c = cand perform Gibbs sampling by using ( 13) and ( 4) with leakiness c end for

4. PARTITION FUNCTION ESTIMATION:

It is known that estimating the partition function of RBM is intractable (Salakhutdinov & Murray, 2008) . Existing approaches, including Salakhutdinov & Murray (2008) ; Grosse et al. (2013) ; Liu et al. (2015) ; Carlson et al. (2016) focus on using sampling to approximate the partition function of the conventional Bernoulli RBM instead of the RBM with Gaussian visible units and non-Bernoulli hidden units. In this paper, we focus on extending the classic annealed importance sampling (AIS) algorithm (Salakhutdinov & Murray, 2008) to leaky RBM. Assuming that we want to estimate the partition function Z of p(v) with p(v) = p * (v)/Z and p * (v) ∝ h exp(-E(v, h)), Salakhutdinov & Murray (2008) start from a initial distribution p 0 (v) ∝ h exp(-E 0 (v, h)), where computing the partition Z 0 of p 0 (v) is tractable and we can draw samples from p 0 (v). They then use the "geometric path" to anneal the intermediate distribution as

p k (v) ∝ p * k (v) = h exp (-β k E 0 (v, h) -(1 -β k )E(v, h))

, where they grid β k from 1 to 0. If we let β 0 = 1, we can draw samples v k from p k (v) by using samples v k-1 from p k-1 (v) for k ≥ 1 via Gibbs sampling. The partition function is then estimated via Z = Z0 M M i=1 ω (i) , where

ω (i) = p * 1 (v (i) 0 ) p * 0 (v (i) 0 ) p * 2 (v (i) 1 ) p * 1 (v (i) 1 ) • • • p * K-1 (v (i) K-2 ) p * K-2 (v (i) K-2 ) p * K (v (i) K-1 ) p * K-1 (v (i) K-1 )

, and β K = 0.

Salakhutdinov & Murray (2008) use the initial distribution with independent visible units and without hidden units. We consider application of AIS to the leaky-ReLU case with E 0 (v, h) = v 2 2 , which results in a multivariate Gaussian distribution p 0 (v). Compared with the meta algorithm shown in Algorithm 2 which anneals between leakiness, AIS anneals between energy functions.

4.1. STUDY ON TOY EXAMPLES:

As we discussed in Section 3.1, leaky RBM with J hidden units is a union of 2 J truncated Gaussian distributions. Here we perform a study on the leaky RBM with a small number hidden units. Since in this example the number of hidden units is small, we can integrate out all possible configurations of h. However, integrating a truncated Gaussian distribution with general affine constraints does not have analytical solutions, and several approximations have been developed (e.g., Pakman & Paninski, 2014) . To compare our results with the exact partition function, we consider a special case that has the following form:

EQUATION

Compared to (6), it is equivalent to the setting where b = 0. Geometrically, every W j passes through the origin. We further put the additional constraint W i ⊥ W j , ∀i = j. Therefore. we divide the whole space into 2 J equally-sized regions. A three dimensional example is shown in Figure 3 . Then the partition function of this special case has the analytical form

Z = 1 2 J αj ∈{1,c},∀j (2π) -I 2   I - J j=1 α j W j W j   -1 2 .

We randomly initialize W and use SVD to make columns orthogonal. Also, we scale W j to satisfy I -W W 0. The leakiness parameter is set to be 0.01. For Salakhutdinov & Murray (2008) (AIS-Energy), we use 10 5 particles with 10 5 intermediate distributions. For the proposed method (AIS-Leaky), we use only 10 4 particles with 10 3 intermediate distributions. In this small problem we study the cases when the model has 5, 10, 20 and 30 hidden units and 3072 visible units. The true log partition function log Z is shown in Table 1 and the difference between log Z and the estimates given by the two algorithms are shown in Table 2 .

From Table 1 , we observe that AIS-Leaky has significantly better and more stable estimations than AIS-Energy especially and this gap increases as we increase the number of hidden units. AIS-Leaky achieves this with orders magnitude reduced computation -e.g., here it uses ∼.1% of resources used by conventional AIS. For example, when we increase J from 5 to 30, the bias (difference) of AIS-Leaky only increases from 0.02 to 0.13; however, the bias of AIS-Energy increases from 1.76 to 9.6. We further study the implicit connection between the proposed AIS-Leaky and AIS-Energy in Appendix E, which shows AIS-Leaky is a special case of AIS-Energy under certain conditions. J = 5 J = 10 J = 20 J = 30 AIS-Energy 1.76 ± 0.011 3.56 ± 0.039 7.95 ± 0.363 9.60 ± 0.229 AIS-Leaky 0.02 ± 0.001 0.04 ± 0.002 0.08 ± 0.003 0.13 ± 0.004 Table 3 : The log-likelihood performance of Bernoulli-Gaussian RBM and leaky RBM.

4.2. COMPARISON BETWEEN LEAKY-RELU RBM AND BERNOULLI-GAUSSIAN RBM:

It is known that the reconstruction error is not a proper approximation of the likelihood (Hinton, 2012) . One commonly adopted way to compare generative models is to sample from the model, and visualize the images to check the quality. However, Theis et al. (2016) show the better visualization does not imply better likelihood. Also, the single layer model cannot adequately model the complicated natural images (the result for Bernoulli-Gaussian RBM has been shown in Ranzato & Hinton ( 2010)), which makes the visualization comparison difficult (Appendix F has few visualization results).

Fortunately, our accurate estimate of the partition function for leaky RBM can produce a reliable quantitative estimate of the representation power of leaky RBM. We compare the Bernoulli-Gaussian RBM2 , which has Bernoulli hidden units and Gaussian visible units. We trained both models with CD-203 and momentum. For both model, we all used 500 hidden units. We initialized W by sampling from Unif(0, 0.01), a = 0, b = 0 and σ = 1. The momentum parameter was 0.9 and the batch size was set to 100. We tuned the learning rate between 10 -1 and 10 -6 . We studied two benchmark data sets, including CIFAR10 and SVHN. The data was normalized to have zero mean and standard deviation of 1 for each pixel. The results of the log-likelihood are reported in Table 3 .

From Table 3 , leaky RBM outperforms Bernoulli-Gaussian RBM significantly. The unsatisfactory performance of Bernoulli-Gaussian RBM may be in part due to the optimization procedure. If we tune the decay schedule of the learning-rate for each dataset in an ad-hoc way, we observe the performance of Bernoulli-Gaussian RBM can be improved by ∼ 300 nats for both datasets. Also, increasing CD-steps brings slight improvement. The other possibility is the bad mixing during the CD iterations. The advanced algorithms Tieleman (2008) ; Tieleman & Hinton (2009) may help.

Although Nair & Hinton (2010) demonstrate the power of ReLU in terms of reconstruction error and classification accuracy, it does not imply its superior generative capability. Our study confirms leaky RBM could have much better generative performance compared to Bernoulli-Gaussian RBM.

5. BETTER MIXING BY ANNEALING LEAKINESS:

In this section, we show the idea of annealing between leakiness benefit the mixing in Gibbs sampling in other settings. A common procedure for comparison of sampling methods for RBM is through visualization. Here, we are interested in more quantitative metrics and the practical benefits of improved sampling. For this, we consider optimization performance as the evaluation metric.

The gradient of the log-likelihood function L(θ|v data ) of general RBM models is

EQUATION

)

Since the second expectation in ( 9) is usually intractable, different approximation algorithms are used (Fischer & Igel, 2012) . In this section, we compare two gradient approximation procedures. The baselines are the conventional contrastive divergence (CD) (Hinton, 2002) and persistent contrastive divergence (Tieleman, 2008 ) (PCD). The second method is using Algorithm 2 (Leaky) with the same number of mixing steps as CD. The experiment setup is the same as that of Section 4.

The results are shown in Figure 4 . The proposed sampling procedure is slightly better than typical CD steps. The reason is we only anneals the leakiness for 20 steps. To get accurate estimation requires thousands of steps as shown in Section 4 when we estimate the partition function. Therefore, the estimated gradient is still inaccurate. However, it still outperforms the conventional CD algorithm. On the other hand, unlike the binary RBM case shown in Tieleman (2008) , PCD does not outperform CD with 20 mixing steps for leaky RBM.

The drawback of Algorithm 2 is that sampling v from N (I -W W ) -1 W b, (I -W W ) -1 requires computing mean, covariance and the Cholesky decomposition of the covariance matrix in every iteration, which are computationally expensive. We study a mixture algorithm by combining CD and the idea of annealing leakiness. The mixture algorithm replaces the sampling from N (I -W W ) -1 W b, (I -W W ) -1 with sampling from the empirical data distribution. The resulted mix algorithm is almost the same as CD algorithm while it anneals the leakiness over the iterations as Algorithm 2. The results of the mix algorithm is also shown in Figure 4 .

The mix algorithm is slightly worse than the original leaky algorithm, but it also outperforms the conventional CD algorithm without additional computation cost. The comparison in terms of CPU time is shown in Appendix F. Annealing the leakiness helps the mix algorithm explore different modes of the distribution, thereby improves the training. The idea could also be combined with more advanced algorithms (Tieleman, 2008; Tieleman & Hinton, 2009) 4 .

6. CONCLUSION:

In this paper, we study the properties of the exponential family distribution produced by leaky RBM. This study relates the leaky RBM model and truncated Gaussian distribution and reveals an underlying positive definite constraint of training leaky RBM. We further proposed a meta sampling algorithm, which anneals between leakiness during the Gibbs sampling procedure. We first demonstrate the proposed sampling algorithm is significantly more effective and efficient in estimating the partition function than the conventional AIS algorithm. Second, we show that the proposed sampling algorithm has comparatively better mixing properties (compared to CD). A few direction are worth further study; in particular we are investigating on speeding up the naive projection step; either using the barrier function as shown in Hsieh et al. (2011) or by eliminating the need for projection by artificially bounding the domain via additional constraints.

A DERIVATION OF LEAKY (RELU) RBM:

A.1 CONDITIONAL DISTRIBUTIONS For leaky RBM, the activation function of hidden units is defined as f (η j ) = max(cη j , η j ), where c ∈ (0, 1) and η j =

I i=1 W ij v i + b j . The inverse function of f is f -1 (h j ) = min(h j , h j /c

). Therefore, the anti-derivatives are

EQUATION

and

EQUATION

The activation function of Gaussian visible units can be treated as the linear unit f (ν i ) = ν i , where

ν i = J j=1 W ij h j .

Following the similar steps for deriving F and F * , we get the anti-derivatives From Ravanbakhsh et al. (2016) , the conditional distribution is defined as

F (ν i ) = 1 2 ν 2 i and F * (v i ) = 1 2 v 2 i .

EQUATION

By plugging F and F * into (12), we get the conditional distribution for leaky RBM

EQUATION

Similarly, we have p(v i |ν i ) = N (ν i , 1) with g(v i ) = -log( √ 2π).

A.2 JOINT AND MARGINAL DISTRIBUTIONS:

Given the conditional distributions p(v|h) and p(h|v), the joint distribution p(v, h) from the general treatment for MRF model given by Yang et al. (2012) is

EQUATION

By plugging F * , F * and g from Section A.1 into (14), we have

p(v, h) ∝ exp   v W h - v 2 2 - ηj >0 h 2 j 2 + log √ 2π - ηj ≤0 h 2 j 2c + log √ 2cπ + b h   ,

Then the marginal distribution is

p(v) ∝ h p(v, h)dh ∝ h exp - v 2 2 ηj >0 exp - h 2 j 2 + η j h j -log √ 2π ηj ≤0 - h 2 j 2c + h j η j -log √ 2cπ dh ∝ exp - v 2 2 ηj >0 exp η 2 j 2 ηj ≤0 cη 2 j 2 ∝ exp   - 1 2 v   I - ηj >0 W j W j -c ηj ≤0 W j W j   v + ηj >0 b j W j v + c ηj ≤0 b j W j v   .

B PROOF OF THEOREM 1:

Proof. Since W Wj α j W j W j = j (1 -α j )W j W j 0, we have W W j α j W j W j . Therefore, Ij α j W j W j I -W W 0.

C PROOF OF THEOREM 2:

Proof. Let the SVD decomposition of W and W as W = U SV and W = Ũ S Ṽ . Then we have

EQUATION

and the constraint I -W W 0 can be rewritten as 0 ≤ Sii ≤ 1, ∀i. The transformed problem has a Lasso-like formulation and we can solve it by Sii = min(S ii , 1) (Parikh & Boyd, 2014) . Also, the lower bound I i=1 (S ii -Sii ) 2 in (15) becomes a tight bound when we set Ũ = U and Ṽ = V , which completes the proof.

D NECESSITY OF THE PROJECTION STEP:

We conduct a short comparison to demonstrate the projection step is necessary for the leaky RBM on generative tasks. We train two leaky RBM as follows. The first model is trained by the same setting in Section 4. We use the convergence of log likelihood as the stopping criteria. The second model is trained by CD-1 with weight decay and without the projection step. We stop the training when the reconstruction error is less then 10 -2 . After we train these two models, we run Gibbs sampling with 1000 independent chains for several steps and output the average value of the visible units. Note that the visible units are normalized to zero mean. The results on SVHN and CIFAR10 are shown in Figure 5 . train leaky RBM with larger CD steps when we do not do projection; otherwise, we would have the diverged gradients. Therefore, the projection is necessary for training leaky RBM for the generative purpose. However, we also oberseve that the projection step is not necessary for the classification and reconstruction tasks. he reason may be the independency of different evaluation criteria (Hinton, 2012; Theis et al., 2016) or other implicit reasons to be studied.

E EQUIVALENCE BETWEEN ANNEALING THE ENERGY AND LEAKINESS:

We analyze the performance gap between AIS-Leaky and AIS-Energy. One major difference is the initial distribution. The intermediate marginal distribution of AIS-Energy has the following form:

EQUATION

Here we eliminated the bias terms b for simplicity. Compared with Algorithm 2, ( 16) not only anneals the leakiness (1 -β k )c ηj ≤0 W j W j when η j ≤ 0, but also in the case (1β k ) ηj >0 W j W j when η j > 0, which brings more bias to the estimation. In other words, AIS-Leaky is a one-sided leakiness annealing while AIS-Energy is a two-sided leakiness annealing method.

To address the higher bias problem of AIS-Energy, we replace the initial distribution with the one used in Algorithm 2. By elementary calculation, the marginal distribution becomes

EQUATION

which recovers the proposed Algorithm 2. From this analysis, we understand AIS-Leaky is a special case of conventional AIS-Energy with better initialization inspired by the study in Section 3. Also, by this connection between AIS-Energy and AIS-Leaky, we note that AIS-Leaky can be combined with other extensions of AIS (Grosse et al., 2013; Burda et al., 2015) as well.

F MORE EXPERIMENTAL RESULTS FOR SAMPLING F.1 SAMPLED IMAGES:

We show the sampled images from leaky RBM train on CIFAR10 and SVHN datasets. We randomly initialize 20 chains and run Gibbs sampling for 1000 iterations. The sampled results are shown in Figure 6 The results shows that single layer RBM does not adequately model CIFAR10 and SVHN when compared to multilayer models. The similar results for single layer Bernoulli-Gaussian RBM from Ranzato & Hinton (2010) (in gray scale) is shown in Figure 7 . Therefore, we instead focused on quantitative evaluation of the log-likelihood in Table 3 .

F.2 COMPUTATIONAL TIME BETWEEN DIFFERENT SAMPLING STRATEGIES:

The comparison in terms of CPU time of different sampling algorithms discussed in Section 5 is shown in Figure 8 . Please note that the complexity of CD and Mix are the almost the same. Mix only need a few more constant time steps which can be ignored compared with sampling steps.

Leaky is more time-consuming because of computing and decomposing the covariance matrix as we discussed in Section 5. We also report the execution time of each step of algorithms in Table 4 .

F.3 STUDY ON RELU-BERNOULLI RBM:

We study the idea of annealing leakiness on the RBM model with leaky ReLU hidden units and Bernoulli visible units. We create the toy dataset with 20, 25 and 30 visible units as shown in Figure 9 . The small datasets allow exact computation of the partition function. For each dataset, we sample 60,000 images for training and 10,000 images for testing. We use 100 hidden units and PCD to train the model. The log likelihood results are shown in Table 5 .

Compared to the Gaussian visible units case we study in Section 3, where p(v) is a multi-variate Gaussian distribution when c = 1, the partition function of p(v) in ReLU-Bernoulli when c = 1 does not have the analytical form. Therefore, we do the following two-stage alternative. We first run the standard AIS algorithm, which anneals the energy, to the distribution with leakiness c = 1.

We then change to anneals the leakiness from 1 to the target value. For the typical AIS algorithm (AIS-Energy), we use 10 4 chains with 2 × 10 4 intermediate distributions. For the proposed twostaged algorithm (AIS-Leaky), we use 10 4 chains with 10 4 intermediate distributions for annealing to c = 1 and the other 10 4 distributions for annealing the leakiness. The results are shown in Table 6 .

In Table 6 , the standard AIS algorithm (AIS-Energy) has unsatisfactory performance. We show the performance of AIS for estimating the partition function of models with different leakiness on Toy20. We use the 10 4 independent chains and 2 × 10 4 intermediate distributions. The results are shown in Table 7 . From 

:

Our GPU implementation with gnumpy and cudamat can reproduce the results of http://www.cs.toronto.edu/ tang/code/GaussianRBM.m

CD-n means that contrastive divergence was run for n steps

We studied the PCD extension of the proposed sampling algorithm. However, the performance is not as stable as CD.

Figures and Tables:

Figure 1: A two dimensional example with 3 hidden units.

= 10 J = 20 J = 30 Log partition function2825.48 2827.98 2832.98 2837.99

Figure 4: Training leaky RBM with different sampling algorithms.

Figure 5: Divergence results on two datasets.

Figure 6: Sampled images from leaky RBM.

Figure 7: Sampled images in gray-scale from Bernoulli-Gaussian RBM trained on CIFAR10 (Ranzato & Hinton, 2010).

Figure 8: Training leaky RBM with different sampling algorithms.

The true partition function for Leaky-ReLU RBM with different number of hidden units.

Table content: <table/>

The difference between the true partition function and the estimations of two algorithms with standard deviation.

Table content: <table><tr><td/><td>CIFAR-10</td><td>SVHN</td></tr><tr><td>Bernoulli-Gaussian RBM</td><td>-2548.3</td><td>-2284.2</td></tr><tr><td>Leaky-ReLU RBN</td><td>-1031.1</td><td>-182.4</td></tr></table>

Table7, we observe that the AIS performances worse when the leakiness is closer to 0. Although we observed that increasing chains and intermediate distributions could improve the performance, but the improvements are limited. The study demonstrates when the The execution time (s) of each step of algorithms (1000 iterations).

Table content: <table><tr><td/><td colspan="2">SVHN CIFAR10</td></tr><tr><td>Inner Loop of CD</td><td>96.16</td><td>96.15</td></tr><tr><td colspan="2">Inner Loop of annealing leaky 96.17</td><td>96.16</td></tr><tr><td>Projection</td><td>438.56</td><td>439.47</td></tr></table>

Peer review (iclr2017):
Based on previous work such as the stepped sigmoid units and ReLU hidden units for discriminatively trained supervised models, a Leaky-ReLU model is proposed for generative learning. Pro: what is interesting is that unlike the traditional way of first defining an energy function and then deriving the conditional distributions, this paper propose the forms of the conditional first and then derive the energy function. However this general formulation is not novel to this paper, but was generalized to exponential family GLMs earlier. Con: Because of the focus on specifying the conditionals, the joint pdf and the marginal p(v) becomes complicated and hard to compute. On the experiments, it would been nice to see a RBM with binary visbles and leaky ReLu for hiddens. This would demonstrate the superiority of the leaky ReLU hidden units. In addition, there are more results on binary MNIST modeling with which the authors can compare the results to. While the authors is correct that the annealing distribution is no longer Gaussian, perhaps CD-25 or (Faast) PCD experiments can be run to compare agains the baseline RBM trained using (Fast) PCD. This paper is interesting as it combines new hidden function with the easiness of annealed AIS sampling, However, the baseline comparisons to Stepped Sigmoid Units (Nair &Hinton) or other models like the spike-and-slab RBMs (and others) are missing, without those comparisons, it is hard to tell whether leaky ReLU RBMs are better even in continuous visible domain.