Synthetic Reduced Nearest Neighbor Model for Regression

1 INTRODUCTION . One of the main topics of research in Machine Learning is the relation between the features and output responses Hastie et al . ( 2009 ) ; Santosa & Symes ( 1986 ) ; Tibshirani ( 1996 ) ; Criminisi & Shotton ( 2013 ) . Synthetic Reduced Nearest Neighbor ( SRNN ) models are shown to be an effective tool in determining the relationships between features of the inputs and the sub-clusters of each class in supervised learning tasks Tavallali et al . ( 2020b ) . However , existing prototype nearest neighbor models such as SRNN are constrained to classification problems , and to the best of our knowledge , there remains a gap in extending these algorithms towards regression tasks . Such regression reduced nearest neighbor models may find extensive applications in epidemiological studies Tavallali et al . ( 2020a ) ; Cisneros et al . ( 2021 ) , medical studies Criminisi & Shotton ( 2013 ) ; Graf et al . ( 2011b ; a ) , and other applied regression tasks in general Tibshirani ( 1996 ) . To address this gap in the state of the art , we propose a novel algorithm for the optimization and construction of Regression Synthetic Reduced Nearest Neighbor ( Reg-SRNN ) models . The proposed Reg-SRNN is capable of discovering various modalities of the input data , and relates those to the modalities of the output responses . The Reg-SRNN algorithm is designed to handle both single-response and multi-response regression . The multi-response regression consists of learning the relation between input samples and several ground-truth output responses . Reg-SRNN partitions the input space into piecewise constant regions , where each region is represented by a centroid and its output response . From this perspective , Reg-SRNN is similar to other piecewise constant models , such as Li & Martin ( 2017 ) ; Begon et al . ( 2017 ) ; Bertsimas & Dunn ( 2017 ) ; Tavallali et al . ( 2019 ; 2020c ) . Reg-SRNN is capable of learning an accurate relation between each cluster of the data and its corresponding output responses . Therefore , Reg-SRNN can also provide enhanced interpretablity by reducing the information content of clusters into a compressed representation manifested in their centroids . The technical contributions of this paper include the proposal of a novel initialization that by itself is competitive to other existing regression models . This is proceeded by an expectation maximization algorithm for directly minimizing the least squares error of the mode . The proposed optimization algorithm is provably convergent , and it is shown that it monotonically decreases the loss function . Therefore , the algorithm has a convergence guarantee on minimizing the loss function and achieving a local optimum.It is also worth mentioning that the algorithm does not cycle . The proposed optimization algorithm consists of two steps and is inspired by K-means algorithm Lloyd ( 1982 ) . One step is the assignment step and is composed of finding samples assignments and proper output response of the centroid . Second step is the update step where the centroid is optimized such that the loss function is decreased . The centroid step is affected by all the samples and we will show that this update step is a kind of NP-hard weighted binary classification problem . The update step is computed through a surrogate objective function that is similar to SVM . We establish that the algorithm is efficient because of its linear computational complexity . Finally , the model is evaluated on various datasets with various sizes and dimensionalities , the results of which demonstrate that Reg-SRNN is capable of competing and even over-matching similar regression models . Accordingly , the main contributions of this paper are as follows : • We propose a novel algorithm for initialization of SRNN models to extend their application to regression tasks . • We develop an optimization algorithm for regression SRNN models with guarantees on convergence . • Through experimental evaluation , we demonstrated the feasibility of our proposed regression SRNN model in filling the gap between more complex models ( such as random forests ) and basic and interpretable models such as linear regression and decision trees . 2 RELATED WORK . A regression task consists of learning the relation between samples of the input space and a numerical output space . More specifically , regression is a supervised learning task of mapping inputs ( independent variable X ) to the output Y , which is a continuous vector ( Y ∈ Rd ) . If the dimensionality of the output d ≥ 2 , the task is known as multi-response regression . Regression has been the workhorse of numerous fields Tai ( 2021 ) , and various regression models have been developed and expanded fundamentally over the recent decades Hastie et al . ( 2009 ) . This expansion has been so rampant such that listing all such models and their relationships is a difficult task and is out of the scope of this work . However , a brief review of the recent models is presented in this paper . A common objective function for regression is to minimize the least squares error : ||Ŷ − Y ||2 ( 1 ) Where Ŷ is the prediction . According to the Gauss-Markov theorem Gauss ( 1823 ) , the least squares error can be an unbiased linear model of minimum variance of the data under certain assumptions . Ordinary least squares may fail to properly predict outcomes if it is applied to settings where the Gauss-Markov assumptions are not held . Therefore , it is important to understand the assumptions and occasionally apply the proper changes to the objective function of equation 1 to modify the model Tai ( 2021 ) . Manifestations of such changes include imposing regulations or constraints over the objective function . The literature on ordinary least squares estimation has extensively dealt with some of the well-known concerns that might violate the assumptions , such as Ridge Hoerl & Kennard ( 1970a ; b ) , Lasso Tibshirani ( 1996 ) , Elastic Net Zou & Hastie ( 2005 ) , trees Quinlan ( 2014 ) , forest Breiman ( 2001 ) , boosting Bühlmann & Yu ( 2003 ) and others . Common regression models include bagging , boosting , random forest Criminisi & Shotton ( 2013 ) , oblique trees Murthy et al . ( 1994 ) ; Norouzi et al . ( 2015 ) ; Heath et al . ( 1993 ) , and regression SVM Drucker et al . ( 1997 ) . In the context of regression trees , various approaches of inducing a tree are presented in the literature . Most decision tree induction methods are concentrated on the splitting criterion used at the growing phase of the tree Ikonomovska et al . ( 2011 ) ; Levatić et al . ( 2014 ) . Application of decision tree algorithms to multi-response regression has been previously considered in the literature Breiman et al . ( 1984 ) ; De ’ Ath ( 2002 ) . In Breiman et al . ( 1984 ) ; Quinlan ( 1986 ) , authors consider training a decision tree for each individual output response . However , such approach constructs a large model specially if the number of output responses are high . Another approach proposed in De ’ Ath ( 2002 ) consists of constructing a single decision tree for all the output responses . In other words , the model predicts all the output values simultaneously through a single decision tree . However , a model for all the outputs might not be sufficient Kocev et al . ( 2009 ) because they train model for single response rather than the true problem which is a multi-response regression . Authors in Kocev et al . ( 2009 ) have explored two approaches to the multi-response regression problem by comparing learning a model for each output separately ( i.e. , multiple regression trees ) , and learning one model for all outputs simultaneously ( i.e. , a single multi-target regression tree ) . In order to improve predictive performance , Kocev et al . ( 2013 ) has also considered two ensemble learning techniques , namely , bagging Breiman ( 1996 ) ; Liang et al . ( 2011 ) and random forests Breiman ( 2001 ) for regression trees and multi-target regression trees . The results showed that multi-target regression trees created more accurate and compact models . A related topic to the problem of this paper is nearest neighbor regression . Nearest neighbor regression and local estimators are well-established methods in the literature of ordinary univariate location estimators ( Benedetti ( 1977 ) ; Stone ( 1974 ) ; Tukey et al . ( 1977 ) . However , as per our extensive search , there remains a gap in prototype nearest neighbor approaches to regression . The only work that considered a similar model and optimization to SRNN was Huang ( 2017 ) . However , the proposed algorithm does not have guarantee of convergence or achieving some sort of optimum solution . 3 PROPOSED METHOD . 3.1 PRELIMINARIES . Assume a dataset consisting of tuples ( xi , yi ) where x , y and i represent input features , output responses and index number . Each tuple represent a data xi and its corresponding output response yi . Here , xi ∈ R D and yi ∈ R d. The Regression Synthetic Reduced Nearest Neighbor ( Reg . SRNN ) consists of K tuples of synthetically produced centroids/prototypes ( cj , ŷj ) where c , ŷj and j represent the centroid ’ s point in the input space , output prediction and index . At the inference time , the Reg . SRNN operates like a nearest neighbor model where the centroids are used as the samples . The problem of training Reg . SRNN is as follows : min { ( cj , ŷj ) } K1 N ∑ i=1 ||yi − ŷj∗ i ||2 s.t j∗i = argmin { j } K 1 d ( xi − cj ) ( 2 ) where d ( . ) is a distance metric . Through this paper we use the l-2 norm as the distance metric : d ( xi − cj ) = √ ||xi − cj ||2 ( 3 ) Essentially the prediction of the model consists of the output prediction of closest centroid to the input sample . Officially , we define Reg . SRNN as follows : NN ( x ) = K ∑ j=1 yjI ( x ∈ Rj ) ( 4 ) Where , NN ( . ) represents a nearest neighbor function of the K centroids . I ( . ) is an indicator function that produces 1 if the input x is in the region of Rj . Rj represents the region where the closest centroid to the points in that region is cj . 3.2 INITIALIZATION . Numerical optimization algorithms require initialization ( cite num opt ) . In this paper , we propose a novel initialization for the regression SRNN . Previous work on the initialization of SRNN models consisted of learning a K-means model for each class of the data . For example , in case of M classes and K centroids , K M centroids are learned for each class as initialization of the SRNN Kusner et al . ( 2014 ) ; Wang et al . ( 2016 ) ; Zhong et al . ( 2017 ) ; Tavallali et al . ( 2020b ) . However , such approach is not applicable to the regression . This initialization also have close ties with naive Bayes and density estimation Silverman ( 2018 ) . Here , we expand this initialization to the case of Reg . SRNN . Intuitively , the output responses can consist of several modalities . In other words , it is possible that the output responses are generated from several distributions . The clusters of such distributions can be approximated by running a K-means over the output space ( M centroids ) . Assume that Sm represents the set of samples assigned to each output cluster . Next step consists of learning KN|Sm| centroids over the input features of the Sm for all M clusters . In other words , we learn centroids over the input features of each output cluster relative to the population of that cluster . The found centroids at the second step are used as initialization for the Reg . SRNN . At this step , ŷj is found using the following formula : ŷj = mean ( yi ∈ Sj ) ( 5 ) where Sj represents the set of samples that are assigned to j th centroid . Sj essentially consists of samples where jth centroid is the closest centroid to them . mean ( . ) represents the average of its input set . Note that Sj ∈ Rj .

Summary: The paper presents a novel regression model based on Nearest Neighbor. In particular authors adapt the synthetic reduced NN model used for classification to the regression case. The main intuition is to simplify the regression problem by finding clusters of inputs and targets. Authors presents some theoretical properties of the method as well as few experimental results.