Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation

1 INTRODUCTION . The density ratio estimation ( DRE ) problem has attracted a great deal of attention as an essential task in data science for its various industrial applications , such as domain adaptation ( Shimodaira , 2000 ; Plank et al. , 2014 ; Reddi et al. , 2015 ) , learning with noisy labels ( Liu & Tao , 2014 ; Fang et al. , 2020 ) , anomaly detection ( Smola et al. , 2009 ; Hido et al. , 2011 ; Abe & Sugiyama , 2019 ) , twosample testing ( Keziou & Leoni-Aubin , 2005 ; Kanamori et al. , 2010 ; Sugiyama et al. , 2011a ) , causal inference ( Kato et al. , 2020 ) , change point detection in time series ( Kawahara & Sugiyama , 2009 ) , and binary classification only from positive and unlabeled data ( PU learning ; Kato et al. , 2019 ) . For example , anomaly detection is not easy to perform based on standard machine learning methods such as binary classification since anomalous data is often scarce , but it can be solved by estimating the density ratio when training data without anomaly as well as unlabeled test data are available ( Hido et al. , 2008 ) . Among various approaches for DRE , we focus on the Bregman ( BR ) divergence minimization framework ( Bregman , 1967 ; Sugiyama et al. , 2011b ) that is a generalization of various DRE methods , e.g. , the moment matching ( Huang et al. , 2007 ; Gretton et al. , 2009 ) , the probabilistic classification ( Qin , 1998 ; Cheng & Chu , 2004 ) , the density matching ( Nguyen et al. , 2010 ; Yamada et al. , 2010 ) , and the density-ratio fitting ( Kanamori et al. , 2009 ) . Recently , Kato et al . ( 2019 ) also proposed using the risk of PU learning for DRE , which also can be generalized from the BR divergence minimization viewpoint , as we show below . However , existing DRE methods mainly adopt a linear-in-parameter model for nonparametric DRE ( Kanamori et al. , 2012 ) and rarely discussed the use of more flexible models , such as deep neural networks , while recent developments in machine learning suggest that deep neural networks can significantly improve the performances for various tasks , such as computer vision ( Krizhevsky et al. , 2012 ) and natural language processing ( Bengio et al. , 2001 ) . This motivates us to use deep neural networks for DRE . However , existing DRE studies have not fully discussed using such state-of-theart deep neural networks . For instance , although Nam & Sugiyama ( 2015 ) and Abe & Sugiyama ( 2019 ) proposed using neural networks for DRE , their neural networks are simple and shallow . When using deep neural networks in combination with empirical minimization of BR divergence , we often observe a serious over-fitting problem as demonstrated through experiments in Figure 2 of Section 5 . We hypothesize that this is mainly because there is no lower bound in the empirically BR divergence approximating by finite samples , i.e. , we can achieve an infinitely negative value in minimization . This hypothesis is based on Kiryo et al . ( 2017 ) , which reports a similar problem in PU learning . While Kiryo et al . ( 2017 ) call this phenomena over-fitting , we refer to it as train-loss hacking because the nuance is a bit different from the standard meaning of overfitting . Here , we briefly introduce the train-loss hacking discussed in the PU learning literature Kiryo et al . ( 2017 ) . In a standard binary classification problem , we train a classifier ψ by minimizing the following empirical risk using { ( yi , Xi ) } ni=1 : 1 n n∑ i=1 1 [ yi = +1 ] ℓ ( ψ ( Xi ) ) + 1 n n∑ i=1 1 [ yi = −1 ] ℓ ( −ψ ( Xi ) ) , ( 1 ) where yi ∈ { ±1 } is a binary label , Xi is a feature , and ℓ is a loss function . On the other hand , in PU learning formulated by du Plessis et al . ( 2015 ) , because we only have positive data { ( y′i = +1 , X ′i ) } n ′ i=1 and unlabeled data { ( x′′j ) } n ′′ j=1 , we minimize the following alternative empirical risk : π n′ n′∑ i=1 ℓ ( ψ ( X ′i ) − π n′ n′∑ i=1 ℓ ( −ψ ( X ′i ) ) ︸ ︷︷ ︸ Cause of train-loss hacking . + 1 n′′ n′′∑ j=1 ℓ ( −ψ ( X ′′j ) ) , ( 2 ) where π is a hyper-parameter representing p ( y = +1 ) . Note that the empirical risk ( 2 ) is unbiased to the population binary classification risk ( 1 ) ( du Plessis et al. , 2015 ) . While the the empirical risk ( 1 ) of the standard binary classification is lower bounded under an appropiate choise of ℓ , the empirical risk ( 2 ) of PU learning proposed by du Plessis et al . ( 2015 ) is not lower bounded owing to the existence of the second term . Therefore , if a model is sufficiently flexible , we can significantly minimize the empirical risk only by minimizing the second term − πn′ ∑n′ i=1 ℓ ( −ψ ( X ′i ) ) without increasing the other terms . Kiryo et al . ( 2017 ) proposed non-negative risk correction for avoiding this problem when using neural networks . We discuss this problem again in Section 2 and Figure 1 . In existing DRE literature , this train-loss hacking has rarely been discussed , although we often face this problem when using neural networks , as mentioned in Section 5 . One reason for this is that the existing method assumes a linear-in-parameter model for a density ratio model ( Kanamori et al. , 2012 ) , which is not so flexible as neural networks and do not cause the phenomenon . To mitigate the train-loss hacking , we propose a general procedure to modify the empirical BR divergence using the prior knowledge of the upper bound of the density ratio . Our idea of the correction is inspired by Kiryo et al . ( 2017 ) . However , their idea of non-negative correction is only immediately applicable to the binary classification ; thus we require a non-trivial rewriting of the BR divergence to generalize the approach to our problem . We call the proposed empirical risk the non-negative BR ( nnBR ) divergence , and it is a generalization of the method of Kiryo et al . ( 2017 ) . In addition , for a special case of DRE , we can still use a lower bounded loss for DRE ( See bounded uLSIF and BKL introduced in the following section ) . However , such a loss also suffers from the train-loss hacking ( bounded uLSIF of Figure 2 and BKL-NN of Figure 4 ) . In the case , the train-loss hacking is caused because the loss sticks to the lower bound . This type of train-loss hacking is also avoided by using the proposed nnBR divergence . Our main contributions are : ( 1 ) the proposal of a general procedure to modify a BR divergence to enable DRE with flexible models , ( 2 ) theoretical justification of the proposed estimator , and ( 3 ) the experimental validation of the proposed method using benchmark data . 2 PROBLEM SETTING . Let X nu ⊆ Rd and X de ⊆ Rd be the spaces of the d-dimensional covariates { Xnui } nnu i=1 and { Xdei } nde i=1 , respectively , which are independent and identically distributed ( i.i.d . ) as { Xnui } nnu i=1 i.i.d.∼ pnu ( X ) and { Xdei } nde i=1 i.i.d.∼ pde ( X ) , where pnu ( X ) and pde ( X ) are probability densities over X nu and X de , respectively . Here , “ nu ” and “ de ” indicate the numerator and the denominator . Our goal is to estimate the density ratio r∗ ( X ) = pnu ( X ) pde ( X ) . To identify the density ratio , we assume the following : Assumption 1 . The density pnu ( X ) is strictly positive over the space X nu , the density pde ( X ) is strictly positive over the space X de , and X nu ⊆ X de . In addition , the density ratio r∗ is bounded from above on X de : R = supX∈Xde r∗ ( X ) < ∞ . Note that the assumption X nu ⊆ X de is typical in the context of DRE . For instance , in anomaly detection with unlabeled test data , X de corresponds to a sample space including clean and anomaly data and X de corresponds to a sample space only with clean data . Here , we introduce the notation of this paper . Let Enu and Ede denote the expectations over pnu ( X ) and pde ( X ) , respectively . Let Ênu and Êde denote the sample average over { Xnui } nnu i=1 and { Xdei } nde i=1 , respectively . Let H ⊂ { r : Rd → ( br , Br ) } be the hypothesis class of the density ratio , where 0 ≤ br < R < Br . 2.1 DENSITY RATIO MATCHING UNDER THE BREGMAN DIVERGENCE . A naive way to implement DRE would be to estimate the numerator and the denominator densities separately and take the ratio . However , according to Vapnik ’ s principle , we should avoid solving a more difficult intermediate problem than the target problem ( Vapnik , 1998 ) . Therefore , various methods for directly estimating the density ratio model have been proposed ( Gretton et al. , 2009 ; Sugiyama et al. , 2008 ; Kanamori et al. , 2009 ; Nguyen et al. , 2010 ; Yamada et al. , 2010 ; Kato et al. , 2019 ) . Sugiyama et al . ( 2011b ) showed that these methods can be generalized as the density ratio matching under the BR divergence . The BR divergence is an extension of the Euclidean distance to a class of divergences that share similar properties ( Bregman , 1967 ) . Formally , let f : ( br , Br ) → R be a twice continuously differentiable convex function with a bounded derivative . Then , the point-wise BR divergence associated with f from t∗ to t is defined as B̈Rf ( t∗‖t ) : = f ( t∗ ) − f ( t ) − ∂f ( t ) ( t∗ − t ) , where ∂f is the derivative of f . Now , the discrepancy from the true density ratio function r∗ to a density ratio model r is measured by integrating the point-wise BR divergence as follows ( Sugiyama et al. , 2011b ) : B̈Rf ( r ∗‖r ) : = ∫ pde ( X ) ( f ( r∗ ( X ) ) − f ( r ( X ) ) − ∂f ( r ( X ) ) { r∗ ( X ) − r ( X ) } ) dX . ( 3 ) We estimate the density ratio by finding a function r that minimizes the BR divergence defined in ( 3 ) . Here , we subtract the constant BR = Ede [ f ( r∗ ( X ) ) ] from ( 3 ) to obtain BRf ( r ∗‖r ) : = ∫ pde ( X ) ( ∂f ( r ( X ) ) r ( X ) − f ( r ( X ) ) ) dX − ∫ pnu ( X ) ∂f ( r ( X ) ) dX . ( 4 ) Here , Sugiyama et al . ( 2012 ) used r∗ ( X ) pde = pnu for removing r∗ ( X ) , which is a common technique in the DRE literature . Since BR is constant with respect to r , we have argminr BR ′ f ( r ∗‖r ) = argminr BRf ( r ∗‖r ) . Then , let us define the sample analogue of ( 4 ) as B̂Rf ( r ) : = Êde [ ∂f ( r ( Xi ) ) r ( Xi ) − f ( r ( Xi ) ) ] − Ênu [ ∂f ( r ( Xj ) ) ] . ( 5 ) For a hypothesis class H , we estimate the density ratio by solving minr∈H B̂Rf ( r∗‖r ) .

Summary: The paper studies density ratio estimation (DRE), addressing the 'train-loss hacking' problems which often arise and hamper estimation when models are too flexible. The authors propose a new risk estimator for DRE, providing a non-negative Bregman divergence estimator, with the non-negative correction. Theoretical analyses are shown with the estimation error and empirical analyses are examined in multiple machine learning problem settings.