An implicit function learning approach for parametric modal regression

1 INTRODUCTION . The goal in regression is to find the relationship between the input ( observation ) variable X ∈ X and the output ( response ) Y ∈ Y variable , given samples of ( X , Y ) . The underlying premise is that there exists an unknown underlying function g∗ : X 7→ Y that maps the input space X to the output space Y . We only observe a noise-contaminated value of that function : sample ( x , y ) has y = g∗ ( x ) + η for some noise η . If the goal is to minimize expected squared error , it is well known that E [ Y |x ] is the optimal predictor ( Bishop , 2006 ) . It is common to use Generalized Linear Models ( Nelder & Wedderburn , 1972 ) , which attempt to estimate E [ Y |x ] for different uni-modal distribution choices for p ( y|x ) , such as Gaussian ( l2 regression ) and Poisson ( Poisson regression ) . For multi-modal distributions , however , predicting E [ Y |x ] may not be desirable , as it may correspond to rarely observed y that simply fall between two modes . Further , this predictor does not provide any useful information about the multiple modes . Modal regression is designed for this problem , and though not widely used in the general machine learning community , has been actively studied in statistics . Most of the methods are non-parametric , and assume a single mode jae Lee ( 1989 ) ; Lee & Kim ( 1998 ) ; Kemp & Silva ( 2012 ) ; Yu & Aristodemou ( 2012 ) ; Yao & Li ( 2014 ) ; Lv et al . ( 2014 ) ; Feng et al . ( 2017 ) . The basic idea is to adjust target values towards their closest empirical conditional modes , based on a kernel density estimator . These methods rely on the chosen kernel and may have issues scaling to high-dimensional data due to issues in computing similarities in high-dimensional spaces . There is some recent work using quantile regression to estimate conditional modes ( Ota et al. , 2018 ) , and though promising for a parametric approach , is restricted to linear quantile regression . A parametric approach for modal regression would enable these estimators to benefit from the advances in learning functions with neural networks . The most straightforward way to do so is to learn a mixture distribution , such as with conditional mixture models with parameters learning by a neural network ( Powell , 1987 ; Bishop , 1994 ; Williams , 1996 ; Husmeier , 1997 ; Husmeier & Taylor , 1998 ; Zen & Senior , 2014 ; Ellefsen et al. , 2019 ) . The conditional modes can typically be extracted from such models . Such a strategy , however , might be trying to solve a harder problem than is strictly needed . The actual goal is to simply identify the conditional modes , without accurately representing the full conditional distribution . Training procedures for the conditional distribution can be more complex . Methods like EM can be slow ( Vlassis & Krose , 1999 ) and some approaches have opted to avoid this altogether by discretizing the target and learning a discrete distribution ( Weigend & Srivastava , 1995 ; Feindt , 2004 ) . Further , the mixture requires particular probabilistic choices to be made , including the number of components , which may not be correctly specified : they might be more or less than the true number of conditional modes . In this paper , we propose a new parametric modal regression approach , by developing an objective to learn a parameterized function f ( x , y ) on both input feature and target/output . We use the Implicit Function Theorem ( Munkres , 1991 ) , which states that if we know the input-output relation in the form of an implicit function , then a general multi-valued function , under certain gradient conditions , can locally be converted to a single-valued function . We learn a function f ( x , y ) that approximates such local functions , by enforcing the gradient conditions . We empirically demonstrate that our method can effectively learning the conditional modes on several synthetic problems , and that for those same problems , scales well when the input is made high-dimensional . We also show an interesting benefit that the joint representation learned over x and y appears to improve prediction performance even for uni-modal problem , for high frequency functions where the function values changes quickly between nearby x . Finally , we show that our method provides small improvements on two regression datasets that have asymmetric distributions over the targets . The proposed approach to multi-valued prediction is flexible , allowing for a variable number of conditional modes to be discovered for each x , and we believe it is a promising direction for further improvements in parametric modal regression . 2 PROBLEM SETTING . We consider a standard learning setting where we observe a dataset of n samples , S = { ( xi , yi ) } ni=1 . Instead of the standard regression problem , however , we tackle the modal regression problem . The goal in modal regression is to find the set of conditional modes M ( x ) = { y : ∂p ( x , y ) ∂y = 0 , ∂2p ( x , y ) ∂y2 < 0 } ( 1 ) M ( x ) is in general a multi-valued function . Consider the example in Figure 1 . There are two conditional modes for a given x . For example , for x = 0 , the two conditional modes are y1 = −1.0 and y2 = 1.0 . The standard approaches to find these conditional modes involve learning p ( y|x ) or to use nonparametric methods directly estimate these conditional modes . For example , for a conditional Gaussian Mixture Model , a relatively effective approximation of these modes are the means of the conditional means . More generally , to get precise estimates , non-parametric algorithms are used , like the mean-shift algorithm ( Yizong Cheng , 1995 ) . We refer readers to Chen ( 2018 ) ; Chen et al . ( 2014 ) for a detailed nice review . These algorithms attempt to cluster points based on x and y , to find these conditional modes . Looking at the plot in Figure 1 , however , a natural idea is to instead directly learn a parameterized function f ( x , y ) that captures the relationship between x and y . Unfortunately , it is not obvious how to do so , nor how to use f ( x , y ) to obtain the conditional modes . In the next section , we develop an approach to learn a parameterized f ( x , y ) that can be used to extract conditional modes , by using the implicit function theorem . 3 AN IMPLICIT FUNCTION LEARNING APPROACH . In this section , we develop an objective to facilitate learning parametric functions for modal regression ( for multi-valued prediction ) . The idea is to directly learn a parameterized function f ( x , y ) , where the set of minimum y for a function of f ( x , y ) corresponds to the conditional modes . The approach allows for a variable number of conditional modes for each x . Further , it allows us to take advantage of general parametric function approximators , like neural networks , to identify these modal manifolds that capture a smooth relationship between the conditional modes and x . Consider learning an f ( x , y ) such that f ( x , y ) = 0 for all conditional modes and non-zero otherwise . For example , for the Circle problem , f ( x , y ) = x2 + y2 − 1 for all conditional modes y . Such a strategy—finding f ( x , y ) = 0 for all conditional modes—is flexible in that it allows for a different number of conditional modes for each x . The difficulty with learning such an f , particularly under noisy data , is constraining it to be zero for conditional modes yj and non-zero otherwise . To obtain meaningful conditional modes y1 , . . . , ymx for x , the y around each yj should be described by the same mapping gj ( x ) . The existence of such gj is guaranteed by the Implicit Function Theorem ( Munkres , 1991 ) , under one condition on f . Implicit Function Theorem : Let f : Rd × Rk 7→ Rk be a continuously differentiable function . Fix a point ( x , y ) ∈ Rd × Rk such that f ( x , y ) = 0 , for 0 ∈ Rk . If the Jacobian matrix J , where the element in the ith row and jth column is J [ ij ] = ∂f ( x , y ) [ i ] ∂y [ j ] , is invertible , then there exists open sets U , V containing ( x , y ) such that there exists a unique continuously differentiable function g : U 7→ V satisfying g ( x ) = y and F ( x , g ( x ) ) = 0 . The theorem states that if we know the relationship between independent variable x and dependent variable y in the form of implicit function f ( x , y ) = 0 , then under certain conditions , we can guarantee the existence of some function defined locally to express y given x . For example , a circle on two dimensional plane can be expressed as { ( x , y ) |x2 + y2 = 1 } , but there is no definite expression ( single-valued function ) for y in terms of x . However , given a specific point on the circle ( x0 , y0 ) ( other than y0 ) , there exists an explicit function defined locally around ( x0 , y0 ) to express y in terms of x . Notice that at y0 = 0 , the condition required by the implicit function theorem is not satisfied : ∂ ( x 2+y2−1 ) ∂y = 2y = 0 at y0 = 0 , and so is not invertible . Obtaining such smooth local functions g enables us to find these smooth modal manifolds . The conditional modes g1 , . . . , gmx satisfy f ( x , gj ( x ) ) = 0 and ∂f ( x , gj ( x ) ) ∂y 6= 0 . When training f , we can attempt to satisfy both conditions to ensure existence of the gj . The gradient condition ensures that for y locally around f ( x , gj ( x ) ) , we have f ( x , y ) 6= 0 . This encourages the other requirement that f ( x , y ) be non-zero for the y that are not conditional modes . Notice , though , that this condition is only local and may not encourage a minimal set of conditional modes . We can also add a negative sampling component that encourages all y to by default have f ( x , y ) 2 > 0 . We find empirically , however , that this addition is not necessary.We therefore pursue this simpler objective , which avoids the complications of negative sampling . Now we derive the full objective , under stochastic targets . To do so , we make some assumptions on the noise around the conditional modes . In particular , we assume that the noise around each conditional mode is Gaussian . More precisely , define ( X , Y ) def = gj ( X ) − Y ( 2 ) for gj the conditional mode for Y . Our goal is to approximate ( x , y ) with parameterized function fθ ( x , y ) for parameters θ . We assume ( X , Y ) ∼ N ( µ = 0 , σ2I ) = ( 2πσ ) −k/2 exp ( − ( X , Y ) 2 2σ2 ) ( 3 ) We also assume that ∂ ( x , y ) [ i ] ∂y [ j ] = 0 for i 6= j : the conditional mode for dimension i is not influenced by changes to the y in dimension j. Consequently , the implicit function condition becomes ∂ ( x , y [ j ] ) ∂y [ j ] 6= 0 , to ensure invertibility . In fact , for this function , we have ∂ ( x , y ) ∂y = ∂gj ( x ) − y ∂y = −1 . ( 4 ) Therefore , when learning f , we simply need to constrain ∂fθ ( x , y ) ∂y = −1 for all y . Putting this together , our goal is to minimize the negative log likelihood of fθ , which approximates a zero-mean Gaussian random variable , under this constraint which we encourage with a quadratic penalty term . This gives the following objective , where the goal is to find argminθ L ( θ ) : L ( θ ) def = n∑ i=1 fθ ( xi , yi ) 2 + ∥∥∥∥∂fθ ( xi , yi ) ∂y + 1 ∥∥∥∥2 2 ( 5 ) The same objective is used when doing prediction , but now optimizing over y . Given x∗ , we compute a y∗ in the set argminy fθ ( x ∗ , y ) 2+ ( ∂fθ ( x ∗ , y ) ∂y +1 ) 2 . These y∗ should correspond to the conditional modes , because the objective should be minimal for conditional modes . One option is to use gradient descent to find these y∗ . In all our experiments , we opted for the simple strategy of searching over 200 evenly spaced values in the range of y .

Summary: This paper considers the regression problem in scenarios in which the conditional distribution of the response variable y given the input x is multimodal. For artificially constructed datasets, in which the the conditional modes are known, performance is assessed by the RMS distance to the mode closest to each prediction.  For real-world datasets, performance is in terms of RMSE and MAE.