Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation

1 INTRODUCTION . The density ratio estimation ( DRE ) problem has attracted a great deal of attention as an essential task in data science for its various industrial applications , such as domain adaptation ( Shimodaira , 2000 ; Plank et al. , 2014 ; Reddi et al. , 2015 ) , learning with noisy labels ( Liu & Tao , 2014 ; Fang et al. , 2020 ) , anomaly detection ( Smola et al. , 2009 ; Hido et al. , 2011 ; Abe & Sugiyama , 2019 ) , twosample testing ( Keziou & Leoni-Aubin , 2005 ; Kanamori et al. , 2010 ; Sugiyama et al. , 2011a ) , causal inference ( Kato et al. , 2020 ) , change point detection in time series ( Kawahara & Sugiyama , 2009 ) , and binary classification only from positive and unlabeled data ( PU learning ; Kato et al. , 2019 ) . For example , anomaly detection is not easy to perform based on standard machine learning methods such as binary classification since anomalous data is often scarce , but it can be solved by estimating the density ratio when training data without anomaly as well as unlabeled test data are available ( Hido et al. , 2008 ) . Among various approaches for DRE , we focus on the Bregman ( BR ) divergence minimization framework ( Bregman , 1967 ; Sugiyama et al. , 2011b ) that is a generalization of various DRE methods , e.g. , the moment matching ( Huang et al. , 2007 ; Gretton et al. , 2009 ) , the probabilistic classification ( Qin , 1998 ; Cheng & Chu , 2004 ) , the density matching ( Nguyen et al. , 2010 ; Yamada et al. , 2010 ) , and the density-ratio fitting ( Kanamori et al. , 2009 ) . Recently , Kato et al . ( 2019 ) also proposed using the risk of PU learning for DRE , which also can be generalized from the BR divergence minimization viewpoint , as we show below . However , existing DRE methods mainly adopt a linear-in-parameter model for nonparametric DRE ( Kanamori et al. , 2012 ) and rarely discussed the use of more flexible models , such as deep neural networks , while recent developments in machine learning suggest that deep neural networks can significantly improve the performances for various tasks , such as computer vision ( Krizhevsky et al. , 2012 ) and natural language processing ( Bengio et al. , 2001 ) . This motivates us to use deep neural networks for DRE . However , existing DRE studies have not fully discussed using such state-of-theart deep neural networks . For instance , although Nam & Sugiyama ( 2015 ) and Abe & Sugiyama ( 2019 ) proposed using neural networks for DRE , their neural networks are simple and shallow . When using deep neural networks in combination with empirical minimization of BR divergence , we often observe a serious over-fitting problem as demonstrated through experiments in Figure 2 of Section 5 . We hypothesize that this is mainly because there is no lower bound in the empirically BR divergence approximating by finite samples , i.e. , we can achieve an infinitely negative value in minimization . This hypothesis is based on Kiryo et al . ( 2017 ) , which reports a similar problem in PU learning . While Kiryo et al . ( 2017 ) call this phenomena over-fitting , we refer to it as train-loss hacking because the nuance is a bit different from the standard meaning of overfitting . Here , we briefly introduce the train-loss hacking discussed in the PU learning literature Kiryo et al . ( 2017 ) . In a standard binary classification problem , we train a classifier ψ by minimizing the following empirical risk using { ( yi , Xi ) } ni=1 : 1 n n∑ i=1 1 [ yi = +1 ] ℓ ( ψ ( Xi ) ) + 1 n n∑ i=1 1 [ yi = −1 ] ℓ ( −ψ ( Xi ) ) , ( 1 ) where yi ∈ { ±1 } is a binary label , Xi is a feature , and ℓ is a loss function . On the other hand , in PU learning formulated by du Plessis et al . ( 2015 ) , because we only have positive data { ( y′i = +1 , X ′i ) } n ′ i=1 and unlabeled data { ( x′′j ) } n ′′ j=1 , we minimize the following alternative empirical risk : π n′ n′∑ i=1 ℓ ( ψ ( X ′i ) − π n′ n′∑ i=1 ℓ ( −ψ ( X ′i ) ) ︸ ︷︷ ︸ Cause of train-loss hacking . + 1 n′′ n′′∑ j=1 ℓ ( −ψ ( X ′′j ) ) , ( 2 ) where π is a hyper-parameter representing p ( y = +1 ) . Note that the empirical risk ( 2 ) is unbiased to the population binary classification risk ( 1 ) ( du Plessis et al. , 2015 ) . While the the empirical risk ( 1 ) of the standard binary classification is lower bounded under an appropiate choise of ℓ , the empirical risk ( 2 ) of PU learning proposed by du Plessis et al . ( 2015 ) is not lower bounded owing to the existence of the second term . Therefore , if a model is sufficiently flexible , we can significantly minimize the empirical risk only by minimizing the second term − πn′ ∑n′ i=1 ℓ ( −ψ ( X ′i ) ) without increasing the other terms . Kiryo et al . ( 2017 ) proposed non-negative risk correction for avoiding this problem when using neural networks . We discuss this problem again in Section 2 and Figure 1 . In existing DRE literature , this train-loss hacking has rarely been discussed , although we often face this problem when using neural networks , as mentioned in Section 5 . One reason for this is that the existing method assumes a linear-in-parameter model for a density ratio model ( Kanamori et al. , 2012 ) , which is not so flexible as neural networks and do not cause the phenomenon . To mitigate the train-loss hacking , we propose a general procedure to modify the empirical BR divergence using the prior knowledge of the upper bound of the density ratio . Our idea of the correction is inspired by Kiryo et al . ( 2017 ) . However , their idea of non-negative correction is only immediately applicable to the binary classification ; thus we require a non-trivial rewriting of the BR divergence to generalize the approach to our problem . We call the proposed empirical risk the non-negative BR ( nnBR ) divergence , and it is a generalization of the method of Kiryo et al . ( 2017 ) . In addition , for a special case of DRE , we can still use a lower bounded loss for DRE ( See bounded uLSIF and BKL introduced in the following section ) . However , such a loss also suffers from the train-loss hacking ( bounded uLSIF of Figure 2 and BKL-NN of Figure 4 ) . In the case , the train-loss hacking is caused because the loss sticks to the lower bound . This type of train-loss hacking is also avoided by using the proposed nnBR divergence . Our main contributions are : ( 1 ) the proposal of a general procedure to modify a BR divergence to enable DRE with flexible models , ( 2 ) theoretical justification of the proposed estimator , and ( 3 ) the experimental validation of the proposed method using benchmark data . 2 PROBLEM SETTING . Let X nu ⊆ Rd and X de ⊆ Rd be the spaces of the d-dimensional covariates { Xnui } nnu i=1 and { Xdei } nde i=1 , respectively , which are independent and identically distributed ( i.i.d . ) as { Xnui } nnu i=1 i.i.d.∼ pnu ( X ) and { Xdei } nde i=1 i.i.d.∼ pde ( X ) , where pnu ( X ) and pde ( X ) are probability densities over X nu and X de , respectively . Here , “ nu ” and “ de ” indicate the numerator and the denominator . Our goal is to estimate the density ratio r∗ ( X ) = pnu ( X ) pde ( X ) . To identify the density ratio , we assume the following : Assumption 1 . The density pnu ( X ) is strictly positive over the space X nu , the density pde ( X ) is strictly positive over the space X de , and X nu ⊆ X de . In addition , the density ratio r∗ is bounded from above on X de : R = supX∈Xde r∗ ( X ) < ∞ . Note that the assumption X nu ⊆ X de is typical in the context of DRE . For instance , in anomaly detection with unlabeled test data , X de corresponds to a sample space including clean and anomaly data and X de corresponds to a sample space only with clean data . Here , we introduce the notation of this paper . Let Enu and Ede denote the expectations over pnu ( X ) and pde ( X ) , respectively . Let Ênu and Êde denote the sample average over { Xnui } nnu i=1 and { Xdei } nde i=1 , respectively . Let H ⊂ { r : Rd → ( br , Br ) } be the hypothesis class of the density ratio , where 0 ≤ br < R < Br . 2.1 DENSITY RATIO MATCHING UNDER THE BREGMAN DIVERGENCE . A naive way to implement DRE would be to estimate the numerator and the denominator densities separately and take the ratio . However , according to Vapnik ’ s principle , we should avoid solving a more difficult intermediate problem than the target problem ( Vapnik , 1998 ) . Therefore , various methods for directly estimating the density ratio model have been proposed ( Gretton et al. , 2009 ; Sugiyama et al. , 2008 ; Kanamori et al. , 2009 ; Nguyen et al. , 2010 ; Yamada et al. , 2010 ; Kato et al. , 2019 ) . Sugiyama et al . ( 2011b ) showed that these methods can be generalized as the density ratio matching under the BR divergence . The BR divergence is an extension of the Euclidean distance to a class of divergences that share similar properties ( Bregman , 1967 ) . Formally , let f : ( br , Br ) → R be a twice continuously differentiable convex function with a bounded derivative . Then , the point-wise BR divergence associated with f from t∗ to t is defined as B̈Rf ( t∗‖t ) : = f ( t∗ ) − f ( t ) − ∂f ( t ) ( t∗ − t ) , where ∂f is the derivative of f . Now , the discrepancy from the true density ratio function r∗ to a density ratio model r is measured by integrating the point-wise BR divergence as follows ( Sugiyama et al. , 2011b ) : B̈Rf ( r ∗‖r ) : = ∫ pde ( X ) ( f ( r∗ ( X ) ) − f ( r ( X ) ) − ∂f ( r ( X ) ) { r∗ ( X ) − r ( X ) } ) dX . ( 3 ) We estimate the density ratio by finding a function r that minimizes the BR divergence defined in ( 3 ) . Here , we subtract the constant BR = Ede [ f ( r∗ ( X ) ) ] from ( 3 ) to obtain BRf ( r ∗‖r ) : = ∫ pde ( X ) ( ∂f ( r ( X ) ) r ( X ) − f ( r ( X ) ) ) dX − ∫ pnu ( X ) ∂f ( r ( X ) ) dX . ( 4 ) Here , Sugiyama et al . ( 2012 ) used r∗ ( X ) pde = pnu for removing r∗ ( X ) , which is a common technique in the DRE literature . Since BR is constant with respect to r , we have argminr BR ′ f ( r ∗‖r ) = argminr BRf ( r ∗‖r ) . Then , let us define the sample analogue of ( 4 ) as B̂Rf ( r ) : = Êde [ ∂f ( r ( Xi ) ) r ( Xi ) − f ( r ( Xi ) ) ] − Ênu [ ∂f ( r ( Xj ) ) ] . ( 5 ) For a hypothesis class H , we estimate the density ratio by solving minr∈H B̂Rf ( r∗‖r ) .

Summary: The paper addresses learning the ratio between two densities from their samples, with applications to outlier detection and covariate shift adaption. An existing approach is to minimize the Bregman (BR) divergence's empirical approximation while modeling the density ratio function $r^*$ by a flexible hypothesis family, such as neural networks (NNs). A particular issue of such an approach (that the present work aims to resolve) is "train-loss hacking," meaning that the empirical loss can become arbitrarily large and negative. A new loss/objective based on BR divergence has been proposed, appearing on page 4, and is referred to as $\widehat{\text{nnBR}}_f(r)$. The major theoretical result, Theorem 1, states that minimizing the proposed objective effectively minimizes the BR divergence for sufficiently large sample sizes. Following this theorem and its corollary, the paper presents empirical evaluations, showing the new algorithm outperforms prior ones on standard datasets.