Year: 2018

Abstract:
Models parameterized by deep neural networks have achieved state-of-the-art results in many domains. These models are usually trained using the maximum likelihood principle with a finite set of observations. However, training deep probabilistic models with maximum likelihood can lead to the issue we refer to as input forgetting. In deep generative latent-variable models, input forgetting corresponds to posterior collapse—a phenomenon in which the latent variables are driven independent from the observations. However input forgetting can happen even in the absence of latent variables. We attribute input forgetting in deep probabilistic models to the finite sample dilemma of maximum likelihood. We formalize this problem and propose a learning criterion—termed reflective likelihood—that explicitly prevents input forgetting. We empirically observe that the proposed criterion significantly outperforms the maximum likelihood objective when used in classification under a skewed class distribution. Furthermore, the reflective likelihood objective prevents posterior collapse when used to train stochastic autoencoders with amortized inference. For example in a neural topic modeling experiment, the reflective likelihood objective leads to better quantitative and qualitative results than the variational auto-encoder and the importance-weighted autoencoder.

heading: 1 INTRODUCTION
text: Learning deep probabilistic models with maximum likelihood has led to many successes in density estimation, variational inference, and text and image generation (Dinh et al., 2016; Kingma & Dhariwal, 2018; Kingma & Welling, 2013; Rezende et al., 2014; Oord et al., 2016). However, maximum likelihood learning of deep models often causes the problem of input forgetting—so called because it corresponds to ignoring the input. We refer to an input here as any variable being conditioned upon—for example a covariate in supervised learning or a latent variable in deep generative latent variable models. Input forgetting corresponds to latent variable collapse in the context of latent variable models and has been discussed in several works (Burda et al., 2015; Bowman et al., 2015; Hoffman & Johnson, 2016; Chen et al., 2016; Sønderby et al., 2016; Zhao et al., 2017; Alemi et al., 2018; Dieng et al., 2018a). It also occurs when learning with Restricted Boltzmann Machines (RBMs) where all the hidden units of the RBM collapse by learning to only capture the bias in the visible units (Cho et al., 2011). Input forgetting makes posterior inference in deep generative models very difficult. In this paper we study input forgetting when the input is a latent variable and when the input is a covariate.

heading: 1.1 CONTRIBUTIONS
text: In this paper, we propose a learning criterion that does not suffer from the input forgetting problem induced by maximum likelihood when fitting deep probabilistic models. This criterion naturally arises from what we call the finite sample dilemma of maximum likelihood.
The finite sample dilemma of maximum likelihood. Consider a supervised learning setting where we have two random variables x and y. Our goal is to learn the relationship between these two random variables by fitting a conditional parametric model pθ(y |x). The learning problem corresponds to finding θ∗ that minimizes the risk function,
R(θ) = Ex∼p∗(x) [ Ey∼p∗(y |x) [l(y, fθ(x))] ] , (1)
where p∗ denotes the population distribution and l(·, ·) is a loss function that measures how different fθ(x) is from the true y. Maximum likelihood corresponds to a log loss,
l(y, fθ(x)) = − log pθ(y |x) and Rmle(θ) = −Ex∼p∗(x) [ Ey∼p∗(y |x) [log pθ(y |x)] ] . (2)
In practice we approximateRmle(θ) using a finite datasetD = (xn,yn)Nn=1 that containsN realizations of the random variables x and y. The sample-based maximum likelihood objective is
L̃(θ;D) = 1 N N∑ n=1 log pθ(yn|xn). (3)
Now consider the risk of maximum likelihood under the same conditional model pθ(y |x) when x and y are independent,
R′mle(θ) = −Ex∼p∗(x) [ Ey∼p∗(y) [log pθ(y |x)] ] . (4)
The corresponding sample-based maximum likelihood objective is
L̃′(θ;D) = 1 N N∑ n=1 1 N N∑ n′=1 log pθ(yn|x′n). (5)
The objectives L̃(θ;D) and L̃′(θ;D) in Eq. 3 and Eq. 5 are related as follows
L̃(θ;D) = N · L̃′(θ;D)− 1 N N∑ n=1 N∑ n′ 6=n log pθ(yn|x′n). (6)
Although there is no relationship between the two population risks Rmle(θ) and R′mle(θ), there is a relationship between their sample-based estimates.
When maximizing the likelihood under finite data—i.e. maximizing L̃(θ;D)—the term L̃′(θ;D) is also being maximized. However maximizing L̃′(θ;D) leads to input forgetting as the underlying data being used do not encode any relationship between x and y. To see this consider maximum likelihood with the same data but where each label y is also paired with all possible inputs x. This version of the data does not contain any relationship between x and y. Maximum likelihood using this data where x and y are independent corresponds to maximizing L̃′(θ;D). A new learning criterion for supervised learning. In light of the relationship in Eq. 6, we propose to explicitly discourage the independence behavior induced by maximizing L̃′(θ;D). We do so by maximizing the reflective likelihood (RLL). Maximizing the RLL simultaneously maximizes the likelihood of the outputs under their true associated inputs and minimizes the average log-probability of these outputs when paired with all other possible inputs. On a classification under imbalance with the MNIST dataset, the classifiers trained with RLL outperform those trained with the usual maximum likelihood objective in all the instances of class imbalance. The performance were similar when the class distribution is uniform.

heading: A new family of stochastic auto-encoders for unsupervised learning.
text: We extend the proposed RLL to unsupervised learning with stochastic auto-encoders and amortized inference. We call stochastic auto-encoders trained with the RLL objective Reflective AutoEncoders (RAEs). RAEs do not suffer from the issue of posterior collapse—a phenomenon in which the estimated approximate posterior distribution of the latent variables reduces to the prior. We evaluate RAEs on a neural topic modelling experiment and found they outperform models learned with the maximum likelihood objective both quantitatively and qualitatively.

heading: 1.2 RELATED WORK.
text: Our work is closely related to two lines of work: penalized maximum likelihood and posterior inference in deep generative models.
Traditional penalized maximum likelihood methods such as the Lasso and L2-norm regularization directly operate on the parameters of a model (Tibshirani, 1996; Hinton, 1987; Louizos et al., 2017).
These regularizers penalize the magnitude of the parameters and correspond to specific prior distributions on the parameters from the Bayesian perspective. In contrast several data-dependent regularizers have been proposed. As early as 1995, Bishop (1995) proposed to add noise to the input when training a neural network with stochastic gradient descent for maximum likelihood learning and showed this corresponds to a form of Tikhonov regularization. Several works have extended this to noise injection in the hidden units of a neural network (Srivastava et al., 2014; Maaten et al., 2013; Gal & Ghahramani, 2016; Wager et al., 2013; Dieng et al., 2018b). Our work relates to those datadependent regularizers that are explicit—in the sense that the corresponding objective function can be written as the sum of the initial objective function and an additional regularization term.
Our work also relates to the literature on posterior inference in deep generative models. Latent variable models are ubiquitous in machine learning. They often involve intractable integrals that are approximated with Markov chain Monte Carlo or variational inference. Our work relates to variational methods for deep generative models (Kingma & Welling, 2013; Rezende et al., 2014). The problem that often occurs in these settings is the latent variables are driven independent to the observations thus rendering posterior inference meaningless. Several works have discussed this issue (Burda et al., 2015; Bowman et al., 2015; Hoffman & Johnson, 2016; Chen et al., 2016; Sønderby et al., 2016; Zhao et al., 2017; Alemi et al., 2018; Dieng et al., 2018a). In this paper we identify a possible cause to this problem and adopt a regularization approach to fix it.

heading: 2 SUPERVISED LEARNING WITH REFLECTIVE LIKELIHOODS
text: We consider the same setting as Section 1.1. We have observations (xn,yn) N n=1 and our goal is to fit a parametric model pθ(y |x) to this data. As stated in Section 1.1, the usual maximum likelihood objective might lead to parameters θ that do not capture the true relationship between x and y. In this section we propose the RLL criterion that encourages parameter settings under which y is more likely when conditioning on its corresponding x than under the average of all the conditional probabilities of y given all possible inputs. The RLL criterion has two forms.
Weak reflective likelihood. The weaker form of RLL minimizes L̃′(θ;D) while maximizing the usual sample-based approximation L̃(θ;D),
L̃weakRLL(θ;D) = 1
N N∑ n=1 [ log pθ(yn |xn)− αn N N∑ n′=1 log pθ(yn |xn′) ] , (7)
where αn ≥ 0 is a hyperparameter trading off these two terms. We will discuss how to set αn at the end of this section. The objective L̃weakRLL(θ;D) provides us with a way to explicitly prefer Rmle(θ) in Eq. (1) overR′mle(θ) in Eq. (4), over an entire training set. The objective L̃weakRLL(θ;D), however, may capture the dependence between the input and the output for some examples but not necessarily for all examples. This is because the second term in Eq. 7 can be driven to its minimal value by setting the term inside the log to zero for one example. We tackle this issue by proposing a stronger version of RLL.
Strong reflective likelihood. The stronger version of RLL favors parameters that capture the relationship between x and y by discouraging the independence behavior induced by maximization of L̃′(θ;D),
L̃strongRLL(θ;D) = 1
N N∑ n=1 [ log pθ(yn |xn)− αn log 1 N N∑ n′=1 pθ(yn |xn′) ] . (8)
The second term comes from an application of Jensen’s inequality, R′mle(θ) ≤ Ey∼p∗(y) [ logEx∼p∗(x) [ pθ(y |x) ]] ,
and using the datasetD to approximate this upper bound. The inner term of this upper bound is what we call a reflective probability,
preflθ (y) = Ex∼p∗(x)pθ(y |x). (9)
This reflective probability can be used as a diagnostic measure of whether a trained model has captured the dependence between the input and output for each pair. The sample-based approximation of preflθ (y) is what appears inside the logarithm in the second term in Eq. 8.
Unlike L̃weakRLL, this stronger objective function L̃ strong RLL explicitly decreases the reflective probability of each individual example (x,y). From here on out we focus on L̃strongRLL and refer to it as RLL.
Stochastic Approximation of RLL. The objective in Eq. 8 is not amenable to large datasets. First, when the sample size N is large computing L̃strongRLL(θ;D) is intractable. Second even if we use a small subset of data we are still left with approximating preflθ (y) which is also intractable for large N . To enable maximizing RLL with large datasets, we propose to maximize
L̃strongRLL(θ;M,M ′) =
1 |M| ∑
(x,y)∈M log pθ(y |x)− αn log 1|M′)| ∑ (x′,y′)∈M′ pθ(y |x′)  , (10) where M and M′ are two minibatch of data samples. In practice we improve the efficiency by constructing the second minibatchM′ using the first minibatchM. For each pair (x,y) ∈ M, we uniformly select K examples fromM at random to constructM′.
Choice of penalty αn. We investigate two strategies to set the cofficient αn. The first strategy simply sets αn to a global fixed coefficient αn = α0, where α0 is a hyperparameter. The second strategy on the other hand dynamically adapts αn for each training example (xn,yn) according to:
αn = α(xn,yn, θ) = { α0 if log pθ(yn |xn) ≤ log p̂reflθ (yn) 0 otherwise (11)
The second strategy is motivated by the intuition that the probability assigned to a correctly paired example (xn,yn) by a model pθ should be higher than the average of the probabilities assigned to incorrectly paired examples (x,yn) for all x ∼ p∗(x). It is however unclear how much higher the probability of a correctly paired example pθ(yn |xn) should be over the reflective probability preflθ (yn), and we do not want to increase the difference if the former is already greater than the latter.

heading: 3 UNSUPERVISED LEARNING WITH REFLECTIVE LIKELIHOODS
text: We extend RLL to unsupervised learning. We do so using the auto-encoding framework (Hinton & Salakhutdinov, 2006; Vincent et al., 2008; Kingma & Ba, 2014; Rezende et al., 2014). We first discuss stochastic auto-encoders and posterior collapse before proposing RAEs—a family of stochastic auto-encoders that do not suffer from the posterior collapse issue.
Stochastic auto-encoders and posterior collapse. We position ourselves in the setting where there are global parameters θ and one latent variable z for every observation y. An observation y is generated by first drawing a latent variable z from some prior distribution p(z)—that we assume fixed—and then sampling y from the conditional distribution of y given z. This conditional distribution pθ(y | z) is parameterized by θ. We are concerned with learning the parameters θ in the presence of the latent variables. Maximum likelihood corresponds to maximizing
L(θ; p∗) = Ey∼p∗(y) log pθ(y) = Ey∼p∗(y) [ log ∫ z pθ(y | z)p(z)dz ] . (12)
The likelihood pθ(y | z) is represented as a powerful deep neural network that takes z as input. This objective Eq. 12 is intractable. Stochastic auto-encoders posit a distribution qφ(z |y) over the latents and either maximize (Burda et al., 2015)
L(θ, φ; p∗) = Ey∼p∗(y) [ logEz∼qφ(z |y) [ pθ(y | z)p(z) qφ(z |y) ]] , (13)
or its lower bound (Kingma & Ba, 2014; Rezende et al., 2014) L(θ, φ; p∗) = Ey∼p∗(y) [ Ez∼qφ(z |y) log [ pθ(y | z)p(z) qφ(z |y) ]] . (14)
The distribution qφ(z |y) is parameterized by a deep neural network that takes the observation y as input.
As discussed in several works, the objective in Eq. 14 is prone to posterior collapse. It can be rewritten as
Lvae(θ, φ; p∗) = Ey∼p∗(y) [ Ez∼qφ(z |y) log pθ(y | z)− KL(qφ(z |y)‖p(z)) ] , (15)
What effectively happens is that the KL term quickly collapses to zero early on during optimization, leading to an approximate posterior that does not capture the data.
The importance-weighted auto-encoder (IWAE, Burda et al., 2015) was introduced to learn more expressive approximate posteriors by directly optimizing the objective in Eq. (13) using K samples from the approximate posterior,
Liwae(θ; p∗) = Ey∼p∗(y) [ log 1
K K∑ k=1 [ pθ(y | zk)p(zk) qφ(zk |y) ]] . (16)
Burda et al. (2015) show this leads to less posterior collapse. However, the IWAE can still lead to posterior collapse due to the finite sample dilemma of maximum likelihood discussed in Section 1.
Reflective auto-encoders. We propose a new class of stochastic auto-encoders that prevent posterior collapse by maximizing the RLL. We reconsider the original population risk functions for maximum likelihood under both paired data,
Rmle(θ) = −Ex∼p∗(x) [ Ey∼p∗(y |x) [log pθ(y |x)] ] and its upper bound under independent data,
R′mle(θ) ≤ Ey∼p∗(y) [ logEx∼p∗(x) [pθ(y |x)] ] .
The RLL in Eq. 8 is a sample-based approximation of
LsupRLL(θ; p ∗) = Ex∼p∗(x)Ey∼p∗(y |x) [ log pθ(y |x)− α logEx′∼p∗(x′) [pθ(y |x′)] ] .
Note this is simply the difference between the original maximum likelihood objective and the log reflective probability. In the case of unsupervised learning this corresponds to
LunsupRLL (θ) = Ey∼p∗(y) [ log pθ(y)− α logEy′∼p∗(y) [pθ(y |y′)] ] , (17)
where we define pθ(y |y′) as pθ(y |y′) = ∫ pθ(y | z)pθ(z |y′)dz = Epθ(z |y′)[pθ(y | z)].
Using this definition we rewrite the objective as LunsupRLL (θ) = Ey∼p∗(y) [ logEz∼p(z)pθ(y | z)− α logEy′∼p∗(y)Epθ(z |y′)[pθ(y | z)] ] . (18)
The expectations in Eq. 18 are intractable. In this paper, we propose to approximate them using importance weighting (Burda et al., 2015). For that we define a parametric proposal distribution qφ(z |y)—we make the conditioning on y explicit to account for recognition networks as proposal distributions. We now write the different expectations as
Ez∼p(z) [pθ(y | z)] ≈ p̂θ(y) = 1
K K∑ k=1 ω(y, zk)pθ(y|zk) where zk ∼ qφ(z |y);
and Ey′∼p∗(y)Ez∼pθ(z |y′) [pθ(y | z)] ≈ Ey′∼p∗(y) ( K∑ k=1 v(y′, zk)pθ(y | zk) )
where zk ∼ qφ(z |y′). The expensive expectation over p∗(y′) is dealt with as before by using a small random subset of training examples. The importance weights ω(y, zk) and v(y′, zk) are computed as
ω(y, zk) = exp(ω̃(y, zk)) and ω̃(y, zk) = log p(zk)− log qφ(zk |y).
v(y′, zk) = exp(ṽ(y′, zk))∑K s=1 exp(ṽ(y ′, zs)) and ṽ(y′, zk) = log pθ(y′ | zk) + log p(zk)− log qφ(zk |y′).
Maximizing the objective in Eq. 18 encourages the proposal qφ to output a distinct approximate posterior distribution for each input y. We conjecture this helps avoid posterior collapse. We verify this later in our empirical study.

heading: 4 CONNECTIONS TO RANKING
text: Collobert et al. (2011) proposed a pair-wise ranking loss for natural language processing. This ranking loss is defined as
Lranking = − [ m− fθ(spos) + fθ(sneg)) ] + , (19)
where fθ is a parametric scoring function, and spos and sneg are positive and negative examples respectively. spos is often simply a training example such as a sentence, whereas sneg is a sample from a distribution conditioned on spos. The hyperparameter m is the margin.
Consider the RLL objective in Eq. (8). Consider the choice of αn in Eq. (11). Using stochastic approximation with a single sample (x,y) for the likelihood term and a single sample (x′, y) for the reflective probability we can rewrite the objective L̃strongRLL(θ;D) as
L̃strongRLL(θ;D) = log pθ(y |x)− α0 log pθ(y |x ′)
=(1− α0) log pθ(y |x)− α0 [0− log pθ(y |x) + log pθ(y |x′)]+ .
This reveals that RLL is a convex sum between the usual maximum likelihood objective and the ranking loss in Eq. (19), modulated by the global coefficient α0. It directly tackles the finite sample dilemma discussed in Section 1 by maximizing the likelihood while ensuring that the correctly paired examples are better scored than incorrectly paired examples. This in turn maximizes the dependence between the inputs and outputs.

heading: 5 APPLICATIONS
text: In this section we apply our proposed method to two different problems: MNIST digit classification under imbalance and neural topic modeling. In classification under imbalance it is hard to learn features of rare classes. Because RLL promotes a stronger dependence between inputs and outputs we expect it to perform better than the maximum likelihood objective in the presence of imbalance. We also consider neural topic models that use stochastic auto-encoders. Posterior collapse is particularly severe in text modeling. Neural topic models suffer from this problem (Srivastava & Sutton, 2017). One manifestation of posterior collapse in neural topic modeling is that all the dimensions of the topic matrix collapse to the same topic which in turn contains words that are not necessarily related to each other (Miao et al., 2016; Srivastava & Sutton, 2017). On both applications we found RLL yields better performance both quantitatively and qualitatively. It learns more useful features for rare classes in classification under imbalance on MNIST as evidenced by higher F1 scores (See fig. 1.) Finally it learns more meaningful latent variables as evidenced by lower perplexity on document completion (See Table 2.) We also found that the choice of αn is dependent on the application. For classification we found a fixed schedule for alpha to perform best. For neural topic modeling we found an adaptive schedule to perform best. The reported results for RLL correspond to the best schedule for αn.

heading: 5.1 CLASSIFICATION UNDER CLASS IMBALANCE
text: We start with MNIST which has 5,000 training examples per class, totalling 50,000 training examples. We refer to this original dataset as D0. We build 9 additional training sets. Each additional training set Dk leaves only 5 training examples for each class j < k. See Table 3 in the Appendix for all the class distributions. The validation set in each case is constructed by collecting all the data points that are left from the training set. We train a classifier on each of these constructed training sets and test it on both the original test set, in which all the classes are uniformly represented, and the matched test set, in which the class distribution matches that of the corresponding training set. We measure both F1 score and accuracy for each setting.
As a classifier, we use a deep convolutional network (LeCun et al., 1995). It has two convolutional layers and two fully connected layers. Each convolutional layer consists of 5 × 5 convolution with stride 1 followed by 2 × 2 max pooling and a rectified linear unit (ReLU, Glorot et al., 2011). The output from the final convolutional layer is flattened and processed by two fully-connected layers of 50 units. We train the classifier using the Adam optimizer with dropout regularization. We find the number of samples to estimate the second term in RLL by maximizing accuracy on the validation set. We found that in most cases one sample suffices. We choose α0 to maximize accuracy on the validation set. We found α0 = 0.5 to be the best.
In Table 1, we see that the classifiers trained with RLL always outperform those trained with the usual maximum likelihood criterion when the class distribution is imbalanced. This is the case both in terms of F1 score and accuracy on both the original (balanced) test set as well as the matched test set.

heading: 5.2 NEURAL TOPIC MODELLING
text: We use the 20NewsGroup (Joachims, 1996) for topic modelling. 20NewGroup consists of 11,314 training and 7,531 test articles. The vocabulary size is 2,000. We follow the standard preprocessing steps which include tokenization and the removal of non-UTF-8 characters and English stop words (Miao et al., 2016; Srivastava & Sutton, 2017; Card et al., 2017).
We train a neural topic model, which is an extension of latent Dirichlet allocation (LDA, Blei et al., 2003) with amortized inference. This is effectively a stochastic autoencoder with a bag-of-word input and output. The inference network qφ of this auto-encoder is a two-layer feed-forward neural network with 100 tanh units each. The output of this network is projected to a 15-dimensional vector normalized by softmax to compute the topic proportion. The likelihood pθ is a three-layer feedforward neural network that maps this topic proportion to a bag-of-words. Our architecture choice is based on the earlier observation by Srivastava & Sutton (2017) that this model is particularly prone to posterior collapse. We test both the VAE (see Eq. (15)) and the IWAE (see Eq. (16)) as baselines. These are two auto-encoders trained with different forms of the maximum likelihood objective. We use Adam (Kingma & Ba, 2014) and train each model for 200 epochs with a fixed learning rate of 0.002. For RLL, we use 5 samples to estimate the reflective probability. We choose an adaptive schedule for αn.
We measure both perplexity and the KL divergence between the estimated posterior distribution qφ and the prior distribution p(z) = N (z; 0, 12). The KL has been used as a metric for measuring posterior collapse. For perplexity, we report both the perplexity on the test set (Full PPL) and the document completion perplexity (Completion PPL, Wallach et al., 2009). Both types of perplexities are computed using importance sampling. Document completion consists in holding out some words for each document in the test set to compute the topic proportions and then evaluating perplexity on the remaining words of each document using the topic proportions learned with the held out words. It is a good way to assess the quality of the learned latent variables in topic modeling. We held out the first half of each document to compute the topic proportions and evaluated perplexity on the second half.
Similarly to the classification experiment, we observe that RAE—the stochastic auto-encoder trained with RLL—always outperforms the neural topic models trained with a maximum likelihood objective (VAE and IWAE) (see Table 2.) In addition to having a lower perplexity, RLL yields slightly higher KL divergence scores—which suggests that RLL indeed improves upon VAE and IWAE in terms of posterior collapse. We verify this further by looking at the learned topics; RLL the topics learned by RLL are as good as the topics learned by Latent Dirichlet Allocation (LDA). This is not the case for the VAE. IWAE also yields good topics but they are less sharp than the ones learned by RAE. These topics can be found in the appendix.

heading: 6 CONCLUSION
text: In this paper, we identified the finite sample dilemma when fitting probabilistic models with maximum likelihood using a finite set of observations. We conjectured this to be the cause of the input forgetting issue often encountered when fitting deep probabilistic models using maximum likelihood. We then proposed RLL—a new learning criterion that mitigates the input forgetting issue. RLL encourages a strong dependence between observations and their conditioning variables. This in turn results in better performance both in classification under imbalance and neural topic modeling with stochastic auto-encoders.

heading: 7 APPENDIX

Review summaries:
This paper proposes maximizing the “reflective likelihood,” which the authors define as: E_x E_y [log q(y|x) - \alpha log q(y)] where the expectations are taken over the data, q is the classifier, and \alpha is a weight on the log q(y) term.  The paper derives the reflective likelihood for classification models and unsupervised latent variable models.  Choices for \alpha are also discussed, and connections are made to ranking losses.  Results show superior F1 and perplexity in MNIST classification and 20NewsGroups modeling.