An implicit function learning approach for parametric modal regression

1 INTRODUCTION . The goal in regression is to find the relationship between the input ( observation ) variable X ∈ X and the output ( response ) Y ∈ Y variable , given samples of ( X , Y ) . The underlying premise is that there exists an unknown underlying function g∗ : X 7→ Y that maps the input space X to the output space Y . We only observe a noise-contaminated value of that function : sample ( x , y ) has y = g∗ ( x ) + η for some noise η . If the goal is to minimize expected squared error , it is well known that E [ Y |x ] is the optimal predictor ( Bishop , 2006 ) . It is common to use Generalized Linear Models ( Nelder & Wedderburn , 1972 ) , which attempt to estimate E [ Y |x ] for different uni-modal distribution choices for p ( y|x ) , such as Gaussian ( l2 regression ) and Poisson ( Poisson regression ) . For multi-modal distributions , however , predicting E [ Y |x ] may not be desirable , as it may correspond to rarely observed y that simply fall between two modes . Further , this predictor does not provide any useful information about the multiple modes . Modal regression is designed for this problem , and though not widely used in the general machine learning community , has been actively studied in statistics . Most of the methods are non-parametric , and assume a single mode jae Lee ( 1989 ) ; Lee & Kim ( 1998 ) ; Kemp & Silva ( 2012 ) ; Yu & Aristodemou ( 2012 ) ; Yao & Li ( 2014 ) ; Lv et al . ( 2014 ) ; Feng et al . ( 2017 ) . The basic idea is to adjust target values towards their closest empirical conditional modes , based on a kernel density estimator . These methods rely on the chosen kernel and may have issues scaling to high-dimensional data due to issues in computing similarities in high-dimensional spaces . There is some recent work using quantile regression to estimate conditional modes ( Ota et al. , 2018 ) , and though promising for a parametric approach , is restricted to linear quantile regression . A parametric approach for modal regression would enable these estimators to benefit from the advances in learning functions with neural networks . The most straightforward way to do so is to learn a mixture distribution , such as with conditional mixture models with parameters learning by a neural network ( Powell , 1987 ; Bishop , 1994 ; Williams , 1996 ; Husmeier , 1997 ; Husmeier & Taylor , 1998 ; Zen & Senior , 2014 ; Ellefsen et al. , 2019 ) . The conditional modes can typically be extracted from such models . Such a strategy , however , might be trying to solve a harder problem than is strictly needed . The actual goal is to simply identify the conditional modes , without accurately representing the full conditional distribution . Training procedures for the conditional distribution can be more complex . Methods like EM can be slow ( Vlassis & Krose , 1999 ) and some approaches have opted to avoid this altogether by discretizing the target and learning a discrete distribution ( Weigend & Srivastava , 1995 ; Feindt , 2004 ) . Further , the mixture requires particular probabilistic choices to be made , including the number of components , which may not be correctly specified : they might be more or less than the true number of conditional modes . In this paper , we propose a new parametric modal regression approach , by developing an objective to learn a parameterized function f ( x , y ) on both input feature and target/output . We use the Implicit Function Theorem ( Munkres , 1991 ) , which states that if we know the input-output relation in the form of an implicit function , then a general multi-valued function , under certain gradient conditions , can locally be converted to a single-valued function . We learn a function f ( x , y ) that approximates such local functions , by enforcing the gradient conditions . We empirically demonstrate that our method can effectively learning the conditional modes on several synthetic problems , and that for those same problems , scales well when the input is made high-dimensional . We also show an interesting benefit that the joint representation learned over x and y appears to improve prediction performance even for uni-modal problem , for high frequency functions where the function values changes quickly between nearby x . Finally , we show that our method provides small improvements on two regression datasets that have asymmetric distributions over the targets . The proposed approach to multi-valued prediction is flexible , allowing for a variable number of conditional modes to be discovered for each x , and we believe it is a promising direction for further improvements in parametric modal regression . 2 PROBLEM SETTING . We consider a standard learning setting where we observe a dataset of n samples , S = { ( xi , yi ) } ni=1 . Instead of the standard regression problem , however , we tackle the modal regression problem . The goal in modal regression is to find the set of conditional modes M ( x ) = { y : ∂p ( x , y ) ∂y = 0 , ∂2p ( x , y ) ∂y2 < 0 } ( 1 ) M ( x ) is in general a multi-valued function . Consider the example in Figure 1 . There are two conditional modes for a given x . For example , for x = 0 , the two conditional modes are y1 = −1.0 and y2 = 1.0 . The standard approaches to find these conditional modes involve learning p ( y|x ) or to use nonparametric methods directly estimate these conditional modes . For example , for a conditional Gaussian Mixture Model , a relatively effective approximation of these modes are the means of the conditional means . More generally , to get precise estimates , non-parametric algorithms are used , like the mean-shift algorithm ( Yizong Cheng , 1995 ) . We refer readers to Chen ( 2018 ) ; Chen et al . ( 2014 ) for a detailed nice review . These algorithms attempt to cluster points based on x and y , to find these conditional modes . Looking at the plot in Figure 1 , however , a natural idea is to instead directly learn a parameterized function f ( x , y ) that captures the relationship between x and y . Unfortunately , it is not obvious how to do so , nor how to use f ( x , y ) to obtain the conditional modes . In the next section , we develop an approach to learn a parameterized f ( x , y ) that can be used to extract conditional modes , by using the implicit function theorem . 3 AN IMPLICIT FUNCTION LEARNING APPROACH . In this section , we develop an objective to facilitate learning parametric functions for modal regression ( for multi-valued prediction ) . The idea is to directly learn a parameterized function f ( x , y ) , where the set of minimum y for a function of f ( x , y ) corresponds to the conditional modes . The approach allows for a variable number of conditional modes for each x . Further , it allows us to take advantage of general parametric function approximators , like neural networks , to identify these modal manifolds that capture a smooth relationship between the conditional modes and x . Consider learning an f ( x , y ) such that f ( x , y ) = 0 for all conditional modes and non-zero otherwise . For example , for the Circle problem , f ( x , y ) = x2 + y2 − 1 for all conditional modes y . Such a strategy—finding f ( x , y ) = 0 for all conditional modes—is flexible in that it allows for a different number of conditional modes for each x . The difficulty with learning such an f , particularly under noisy data , is constraining it to be zero for conditional modes yj and non-zero otherwise . To obtain meaningful conditional modes y1 , . . . , ymx for x , the y around each yj should be described by the same mapping gj ( x ) . The existence of such gj is guaranteed by the Implicit Function Theorem ( Munkres , 1991 ) , under one condition on f . Implicit Function Theorem : Let f : Rd × Rk 7→ Rk be a continuously differentiable function . Fix a point ( x , y ) ∈ Rd × Rk such that f ( x , y ) = 0 , for 0 ∈ Rk . If the Jacobian matrix J , where the element in the ith row and jth column is J [ ij ] = ∂f ( x , y ) [ i ] ∂y [ j ] , is invertible , then there exists open sets U , V containing ( x , y ) such that there exists a unique continuously differentiable function g : U 7→ V satisfying g ( x ) = y and F ( x , g ( x ) ) = 0 . The theorem states that if we know the relationship between independent variable x and dependent variable y in the form of implicit function f ( x , y ) = 0 , then under certain conditions , we can guarantee the existence of some function defined locally to express y given x . For example , a circle on two dimensional plane can be expressed as { ( x , y ) |x2 + y2 = 1 } , but there is no definite expression ( single-valued function ) for y in terms of x . However , given a specific point on the circle ( x0 , y0 ) ( other than y0 ) , there exists an explicit function defined locally around ( x0 , y0 ) to express y in terms of x . Notice that at y0 = 0 , the condition required by the implicit function theorem is not satisfied : ∂ ( x 2+y2−1 ) ∂y = 2y = 0 at y0 = 0 , and so is not invertible . Obtaining such smooth local functions g enables us to find these smooth modal manifolds . The conditional modes g1 , . . . , gmx satisfy f ( x , gj ( x ) ) = 0 and ∂f ( x , gj ( x ) ) ∂y 6= 0 . When training f , we can attempt to satisfy both conditions to ensure existence of the gj . The gradient condition ensures that for y locally around f ( x , gj ( x ) ) , we have f ( x , y ) 6= 0 . This encourages the other requirement that f ( x , y ) be non-zero for the y that are not conditional modes . Notice , though , that this condition is only local and may not encourage a minimal set of conditional modes . We can also add a negative sampling component that encourages all y to by default have f ( x , y ) 2 > 0 . We find empirically , however , that this addition is not necessary.We therefore pursue this simpler objective , which avoids the complications of negative sampling . Now we derive the full objective , under stochastic targets . To do so , we make some assumptions on the noise around the conditional modes . In particular , we assume that the noise around each conditional mode is Gaussian . More precisely , define ( X , Y ) def = gj ( X ) − Y ( 2 ) for gj the conditional mode for Y . Our goal is to approximate ( x , y ) with parameterized function fθ ( x , y ) for parameters θ . We assume ( X , Y ) ∼ N ( µ = 0 , σ2I ) = ( 2πσ ) −k/2 exp ( − ( X , Y ) 2 2σ2 ) ( 3 ) We also assume that ∂ ( x , y ) [ i ] ∂y [ j ] = 0 for i 6= j : the conditional mode for dimension i is not influenced by changes to the y in dimension j. Consequently , the implicit function condition becomes ∂ ( x , y [ j ] ) ∂y [ j ] 6= 0 , to ensure invertibility . In fact , for this function , we have ∂ ( x , y ) ∂y = ∂gj ( x ) − y ∂y = −1 . ( 4 ) Therefore , when learning f , we simply need to constrain ∂fθ ( x , y ) ∂y = −1 for all y . Putting this together , our goal is to minimize the negative log likelihood of fθ , which approximates a zero-mean Gaussian random variable , under this constraint which we encourage with a quadratic penalty term . This gives the following objective , where the goal is to find argminθ L ( θ ) : L ( θ ) def = n∑ i=1 fθ ( xi , yi ) 2 + ∥∥∥∥∂fθ ( xi , yi ) ∂y + 1 ∥∥∥∥2 2 ( 5 ) The same objective is used when doing prediction , but now optimizing over y . Given x∗ , we compute a y∗ in the set argminy fθ ( x ∗ , y ) 2+ ( ∂fθ ( x ∗ , y ) ∂y +1 ) 2 . These y∗ should correspond to the conditional modes , because the objective should be minimal for conditional modes . One option is to use gradient descent to find these y∗ . In all our experiments , we opted for the simple strategy of searching over 200 evenly spaced values in the range of y .

Summary: The paper proposes an implicit function approach to learning the modes of multimodal regression. The basic idea is interesting, and is clearly related to density estimation, which the paper should have discussed. On the other hand the tackled problem is a weaker variant of the learning multimodal output distributions in regression. What is the benefit of tackling this simpler problem, when in reality we are interested in the output distributions? In Bayesian literature (BNNs, deep GPs, Bayesian regression) multimodal outputs are routinely produced, but the paper ignores these methods. The paper should compare to these, and demonstrate the benefit of just learning the modes instead of learning the full output distributions (perhaps the mode learning gives more accurate mode assignment?).