Learning with Reflective Likelihoods

1 INTRODUCTION . Learning deep probabilistic models with maximum likelihood has led to many successes in density estimation , variational inference , and text and image generation ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ; Kingma & Welling , 2013 ; Rezende et al. , 2014 ; Oord et al. , 2016 ) . However , maximum likelihood learning of deep models often causes the problem of input forgetting—so called because it corresponds to ignoring the input . We refer to an input here as any variable being conditioned upon—for example a covariate in supervised learning or a latent variable in deep generative latent variable models . Input forgetting corresponds to latent variable collapse in the context of latent variable models and has been discussed in several works ( Burda et al. , 2015 ; Bowman et al. , 2015 ; Hoffman & Johnson , 2016 ; Chen et al. , 2016 ; Sønderby et al. , 2016 ; Zhao et al. , 2017 ; Alemi et al. , 2018 ; Dieng et al. , 2018a ) . It also occurs when learning with Restricted Boltzmann Machines ( RBMs ) where all the hidden units of the RBM collapse by learning to only capture the bias in the visible units ( Cho et al. , 2011 ) . Input forgetting makes posterior inference in deep generative models very difficult . In this paper we study input forgetting when the input is a latent variable and when the input is a covariate . 1.1 CONTRIBUTIONS . In this paper , we propose a learning criterion that does not suffer from the input forgetting problem induced by maximum likelihood when fitting deep probabilistic models . This criterion naturally arises from what we call the finite sample dilemma of maximum likelihood . The finite sample dilemma of maximum likelihood . Consider a supervised learning setting where we have two random variables x and y . Our goal is to learn the relationship between these two random variables by fitting a conditional parametric model pθ ( y |x ) . The learning problem corresponds to finding θ∗ that minimizes the risk function , R ( θ ) = Ex∼p∗ ( x ) [ Ey∼p∗ ( y |x ) [ l ( y , fθ ( x ) ) ] ] , ( 1 ) where p∗ denotes the population distribution and l ( · , · ) is a loss function that measures how different fθ ( x ) is from the true y . Maximum likelihood corresponds to a log loss , l ( y , fθ ( x ) ) = − log pθ ( y |x ) and Rmle ( θ ) = −Ex∼p∗ ( x ) [ Ey∼p∗ ( y |x ) [ log pθ ( y |x ) ] ] . ( 2 ) In practice we approximateRmle ( θ ) using a finite datasetD = ( xn , yn ) Nn=1 that containsN realizations of the random variables x and y . The sample-based maximum likelihood objective is L̃ ( θ ; D ) = 1 N N∑ n=1 log pθ ( yn|xn ) . ( 3 ) Now consider the risk of maximum likelihood under the same conditional model pθ ( y |x ) when x and y are independent , R′mle ( θ ) = −Ex∼p∗ ( x ) [ Ey∼p∗ ( y ) [ log pθ ( y |x ) ] ] . ( 4 ) The corresponding sample-based maximum likelihood objective is L̃′ ( θ ; D ) = 1 N N∑ n=1 1 N N∑ n′=1 log pθ ( yn|x′n ) . ( 5 ) The objectives L̃ ( θ ; D ) and L̃′ ( θ ; D ) in Eq . 3 and Eq . 5 are related as follows L̃ ( θ ; D ) = N · L̃′ ( θ ; D ) − 1 N N∑ n=1 N∑ n′ 6=n log pθ ( yn|x′n ) . ( 6 ) Although there is no relationship between the two population risks Rmle ( θ ) and R′mle ( θ ) , there is a relationship between their sample-based estimates . When maximizing the likelihood under finite data—i.e . maximizing L̃ ( θ ; D ) —the term L̃′ ( θ ; D ) is also being maximized . However maximizing L̃′ ( θ ; D ) leads to input forgetting as the underlying data being used do not encode any relationship between x and y . To see this consider maximum likelihood with the same data but where each label y is also paired with all possible inputs x . This version of the data does not contain any relationship between x and y . Maximum likelihood using this data where x and y are independent corresponds to maximizing L̃′ ( θ ; D ) . A new learning criterion for supervised learning . In light of the relationship in Eq . 6 , we propose to explicitly discourage the independence behavior induced by maximizing L̃′ ( θ ; D ) . We do so by maximizing the reflective likelihood ( RLL ) . Maximizing the RLL simultaneously maximizes the likelihood of the outputs under their true associated inputs and minimizes the average log-probability of these outputs when paired with all other possible inputs . On a classification under imbalance with the MNIST dataset , the classifiers trained with RLL outperform those trained with the usual maximum likelihood objective in all the instances of class imbalance . The performance were similar when the class distribution is uniform . A new family of stochastic auto-encoders for unsupervised learning .. We extend the proposed RLL to unsupervised learning with stochastic auto-encoders and amortized inference . We call stochastic auto-encoders trained with the RLL objective Reflective AutoEncoders ( RAEs ) . RAEs do not suffer from the issue of posterior collapse—a phenomenon in which the estimated approximate posterior distribution of the latent variables reduces to the prior . We evaluate RAEs on a neural topic modelling experiment and found they outperform models learned with the maximum likelihood objective both quantitatively and qualitatively . 1.2 RELATED WORK .. Our work is closely related to two lines of work : penalized maximum likelihood and posterior inference in deep generative models . Traditional penalized maximum likelihood methods such as the Lasso and L2-norm regularization directly operate on the parameters of a model ( Tibshirani , 1996 ; Hinton , 1987 ; Louizos et al. , 2017 ) . These regularizers penalize the magnitude of the parameters and correspond to specific prior distributions on the parameters from the Bayesian perspective . In contrast several data-dependent regularizers have been proposed . As early as 1995 , Bishop ( 1995 ) proposed to add noise to the input when training a neural network with stochastic gradient descent for maximum likelihood learning and showed this corresponds to a form of Tikhonov regularization . Several works have extended this to noise injection in the hidden units of a neural network ( Srivastava et al. , 2014 ; Maaten et al. , 2013 ; Gal & Ghahramani , 2016 ; Wager et al. , 2013 ; Dieng et al. , 2018b ) . Our work relates to those datadependent regularizers that are explicit—in the sense that the corresponding objective function can be written as the sum of the initial objective function and an additional regularization term . Our work also relates to the literature on posterior inference in deep generative models . Latent variable models are ubiquitous in machine learning . They often involve intractable integrals that are approximated with Markov chain Monte Carlo or variational inference . Our work relates to variational methods for deep generative models ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . The problem that often occurs in these settings is the latent variables are driven independent to the observations thus rendering posterior inference meaningless . Several works have discussed this issue ( Burda et al. , 2015 ; Bowman et al. , 2015 ; Hoffman & Johnson , 2016 ; Chen et al. , 2016 ; Sønderby et al. , 2016 ; Zhao et al. , 2017 ; Alemi et al. , 2018 ; Dieng et al. , 2018a ) . In this paper we identify a possible cause to this problem and adopt a regularization approach to fix it . 2 SUPERVISED LEARNING WITH REFLECTIVE LIKELIHOODS . We consider the same setting as Section 1.1 . We have observations ( xn , yn ) N n=1 and our goal is to fit a parametric model pθ ( y |x ) to this data . As stated in Section 1.1 , the usual maximum likelihood objective might lead to parameters θ that do not capture the true relationship between x and y . In this section we propose the RLL criterion that encourages parameter settings under which y is more likely when conditioning on its corresponding x than under the average of all the conditional probabilities of y given all possible inputs . The RLL criterion has two forms . Weak reflective likelihood . The weaker form of RLL minimizes L̃′ ( θ ; D ) while maximizing the usual sample-based approximation L̃ ( θ ; D ) , L̃weakRLL ( θ ; D ) = 1 N N∑ n=1 [ log pθ ( yn |xn ) − αn N N∑ n′=1 log pθ ( yn |xn′ ) ] , ( 7 ) where αn ≥ 0 is a hyperparameter trading off these two terms . We will discuss how to set αn at the end of this section . The objective L̃weakRLL ( θ ; D ) provides us with a way to explicitly prefer Rmle ( θ ) in Eq . ( 1 ) overR′mle ( θ ) in Eq . ( 4 ) , over an entire training set . The objective L̃weakRLL ( θ ; D ) , however , may capture the dependence between the input and the output for some examples but not necessarily for all examples . This is because the second term in Eq . 7 can be driven to its minimal value by setting the term inside the log to zero for one example . We tackle this issue by proposing a stronger version of RLL . Strong reflective likelihood . The stronger version of RLL favors parameters that capture the relationship between x and y by discouraging the independence behavior induced by maximization of L̃′ ( θ ; D ) , L̃strongRLL ( θ ; D ) = 1 N N∑ n=1 [ log pθ ( yn |xn ) − αn log 1 N N∑ n′=1 pθ ( yn |xn′ ) ] . ( 8 ) The second term comes from an application of Jensen ’ s inequality , R′mle ( θ ) ≤ Ey∼p∗ ( y ) [ logEx∼p∗ ( x ) [ pθ ( y |x ) ] ] , and using the datasetD to approximate this upper bound . The inner term of this upper bound is what we call a reflective probability , preflθ ( y ) = Ex∼p∗ ( x ) pθ ( y |x ) . ( 9 ) This reflective probability can be used as a diagnostic measure of whether a trained model has captured the dependence between the input and output for each pair . The sample-based approximation of preflθ ( y ) is what appears inside the logarithm in the second term in Eq . 8 . Unlike L̃weakRLL , this stronger objective function L̃ strong RLL explicitly decreases the reflective probability of each individual example ( x , y ) . From here on out we focus on L̃strongRLL and refer to it as RLL . Stochastic Approximation of RLL . The objective in Eq . 8 is not amenable to large datasets . First , when the sample size N is large computing L̃strongRLL ( θ ; D ) is intractable . Second even if we use a small subset of data we are still left with approximating preflθ ( y ) which is also intractable for large N . To enable maximizing RLL with large datasets , we propose to maximize L̃strongRLL ( θ ; M , M ′ ) = 1 |M| ∑ ( x , y ) ∈M log pθ ( y |x ) − αn log 1|M′ ) | ∑ ( x′ , y′ ) ∈M′ pθ ( y |x′ ) , ( 10 ) where M and M′ are two minibatch of data samples . In practice we improve the efficiency by constructing the second minibatchM′ using the first minibatchM . For each pair ( x , y ) ∈ M , we uniformly select K examples fromM at random to constructM′ . Choice of penalty αn . We investigate two strategies to set the cofficient αn . The first strategy simply sets αn to a global fixed coefficient αn = α0 , where α0 is a hyperparameter . The second strategy on the other hand dynamically adapts αn for each training example ( xn , yn ) according to : αn = α ( xn , yn , θ ) = { α0 if log pθ ( yn |xn ) ≤ log p̂reflθ ( yn ) 0 otherwise ( 11 ) The second strategy is motivated by the intuition that the probability assigned to a correctly paired example ( xn , yn ) by a model pθ should be higher than the average of the probabilities assigned to incorrectly paired examples ( x , yn ) for all x ∼ p∗ ( x ) . It is however unclear how much higher the probability of a correctly paired example pθ ( yn |xn ) should be over the reflective probability preflθ ( yn ) , and we do not want to increase the difference if the former is already greater than the latter .

Summary: This paper proposes maximizing the “reflective likelihood,” which the authors define as: E_x E_y [log q(y|x) - \alpha log q(y)] where the expectations are taken over the data, q is the classifier, and \alpha is a weight on the log q(y) term.  The paper derives the reflective likelihood for classification models and unsupervised latent variable models.  Choices for \alpha are also discussed, and connections are made to ranking losses.  Results show superior F1 and perplexity in MNIST classification and 20NewsGroups modeling.