A Hypergradient Approach to Robust Regression without Correspondence

1 INTRODUCTION . Regression analysis has been widely used in various machine learning applications to infer the the relationship between an explanatory random variable ( i.e. , the input ) X ∈ Rd and a response random variable ( i.e. , the output ) Y ∈ Ro ( Stanton , 2001 ) . In the classical setting , regression is used on labeled datasets that contain paired samples { xi , yi } ni=1 , where xi , yi are realizations of X , Y , respectively . Unfortunately , such an input-output correspondence is not always available in some applications . One example is flow cytometry , which is a physical experiment for measuring properties of cells , e.g. , affinity to a particular target ( Abid & Zou , 2018 ) . Through this process , cells are suspended in a fluid and injected into the flow cytometer , where measurements are taken using the scattering of a laser . However , the instruments are unable to differentiate the cells passing through the laser , such that the correspondence between the cell proprieties ( i.e. , the measurements ) and the cells is unknown . This prevents us from analyzing the relationship between the instruments and the measurements using classical regression analysis , due to the missing correspondence . Another example is multi-object tracking , where we need to infer the motion of objects given consecutive frames in ∗Equal Contributions . †Yujia Xie , Simiao Zuo , Tuo Zhao are affiliated with Georgia Institute of Technology . Emails : { xieyujia , simiaozuo , tourzhao } @ gatech.edu . Yixiu Mao is affiliated with Shanghai Jiao Tong University . Email : 956986044myx @ gmail.com . Hongteng Xu is affiliated with Gaoling School of Artificial Intelligence , Renmin University of China , and Beijing Key Laboratory of of Big Data Management and Analysis Methods . Email : hongtengxu @ ruc.edu.cn . Xiaojing Ye is affiliated with Georgia State University . Email : xye @ gsu.edu . Hongyuan Zha is affiliated with School of Data Science , Shenzhen Institute of Artificial Intelligence and Robotics for Society , the Chinese University of Hong Kong , Shenzhen . Email : zhahy @ cuhk.edu.cn . a video . This requires us to find the correspondence between the objects in the current frame and those in the next frame . The two examples above can be formulated as a shuffled regression problem . Specifically , we consider a multivariate regression model Y = f ( X , Z ; w ) + ε , where X ∈ Rd , Z ∈ Re are two input vectors , Y ∈ Ro is an output vector , f : Rd+e → Ro is the unknown regression model with parameters w and ε is the random noise independent on X and Z . When we sample realizations from such a regression model , the correspondence between ( X , Y ) and Z is not available . Accordingly , we collect two datasets D1 = { xi , yi } ni=1 and D2 = { zj } nj=1 , and there exists a permutation π∗ such that ( xi , zπ ( i ) ) corresponds to yi in the regression model . Our goal is to recover the unknown model parameterw . Existing literature also refer to the shuffled regression problem as unlabeled sensing , homomorphic sensing , and regression with an unknown permutation ( Unnikrishnan et al. , 2018 ) . Throughout the rest of the paper , we refer to it as Regression WithOut Correspondence ( RWOC ) . A natural choice of the objective for RWOC is to minimize the sum of squared residuals with respect to the regression model parameter w up to the permutation π ( · ) over the training data , i.e. , minw , π L ( w , π ) = ∑n i=1 ‖yi − f ( xi , zπ ( i ) ; w ) ‖22 . ( 1 ) Existing works on RWOC mostly focus on theoretical properties of the global optima to equation 1 for estimating w and π ( Pananjady et al. , 2016 ; 2017b ; Abid et al. , 2017 ; Elhami et al. , 2017 ; Hsu et al. , 2017 ; Unnikrishnan et al. , 2018 ; Tsakiris & Peng , 2019 ) . The development of practical algorithms , however , falls far behind from the following three aspects : •Most of the works are only applicable to linear regression models . • Some of the existing algorithms are of very high computational complexity , and can only handle small number of data points in low dimensions ( Elhami et al. , 2017 ; Pananjady et al. , 2017a ; Tsakiris et al. , 2018 ; Peng & Tsakiris , 2020 ) . For example , Abid & Zou ( 2018 ) adopt an Expectation Maximization ( EM ) method where Metropolis-Hastings sampling is needed , which is not scalable . Other algorithms choose to optimize with respect to w and π in an alternating manner , e.g. , alternating minimization in Abid et al . ( 2017 ) . However , as there exists a strong interaction between w and π , the optimization landscape of equation 1 is ill-conditioned . Therefore , these algorithms are not effective and often get stuck in local optima . • Most of the works only consider the case where there exists an exact one-to-one correspondence between D1 and D2 . For many more scenarios , however , these two datasets are not necessarily well aligned . For example , consider D1 and D2 collected from two separate databases , where the users overlap , but are not identical . As a result , there exists only partial one-to-one correspondence . A similar situation also happens to multiple-object tracking : Some objects may leave the scene in one frame , and new objects may enter the scene in subsequent frames . Therefore , not all objects in different frames can be perfectly matched . The RWOC problem with partial correspondence is known as robust-RWOC , or rRWOC ( Varol & Nejatbakhsh , 2019 ) , and is much less studied in existing literature . To address these concerns , we propose a new computational framework – ROBOT ( Regression withOut correspondence using Bilevel OptimizaTion ) . Specifically , we propose to formulate the regression without correspondence as a continuous optimization problem . Then by exploiting the interaction between the regression model and the data correspondence , we propose to develop a hypergradient approach based on differentiable programming techniques ( Duchi et al. , 2008 ; Luise et al. , 2018 ) . Our hypergradient approach views the data correspondence as an operator of the regression , i.e. , for a given w , the optimal correspondence is π̂ ( w ) = arg minπ L ( w , π ) . ( 2 ) Accordingly , when applying gradient descent to ( 1 ) , we need to find the gradient with respect to w by differentiating through both the objective function L and the data correspondence π̂ ( w ) . For simplicity , we refer as such a gradient to “ hypergradient ” . Note that due to its discrete nature , π̂ ( w ) is actually not continuous inw . Therefore , such a hypergradient does not exist . To address this issue , we further propose to construct a smooth approximation of π̂ ( w ) by adding an additional regularizer to equation 2 , and then we replace π̂ ( w ) with our proposed smooth replacement when computing the hyper gradient of w. Moreover , we also propose an efficient and scalable implementation of hypergradient computation based on simple first order algorithms and implicit differentiation , which outperforms conventional automatic differentiation in terms of time and memory cost . ROBOT can also be extended to the robust RWOC problem , where D1 and D2 are not necessarily exactly aligned , i.e. , some data points inD1 may not correspond to any data point inD2 . Specifically , we relax the constraints on the permutation π ( · ) ( Liero et al. , 2018 ) to automatically match related data points and ignore the unrelated ones . At last , we conduct thorough numerical experiments to demonstrate the effectiveness of ROBOT . For RWOC ( i.e. , exact correspondence ) , we use several synthetic regression datasets and a real gated flow cytometry dataset , and we show that ROBOT outperforms baseline methods by significant margins . For robust RWOC ( i.e. , inexact correspondence ) , in addition to synthetic datasets , we consider a vision-based multiple-object tracking task , and then we show that ROBOT also achieves significant improvement over baseline methods . Notations . Let ‖·‖2 denote the ` 2 norm of vectors , 〈· , ·〉 the inner product of matrices , i.e. , 〈A , B〉 =∑ i , j AijBij for matrices A and B. ai : j are the entries from index i to index j of vector a . Let 1n denote an n-dimensional vector of all ones . Denote d ( · ) d ( · ) the gradient of scalars , and ∇ ( · ) ( · ) the Jacobian of tensors . We denote [ v1 , v2 ] the concatenation of two vectors v1 and v2 . N ( µ , σ2 ) is the Gaussian distribution with mean µ and variance σ2 . 2 ROBOT : A HYPERGRADIENT APPROACH FOR RWOC . We develop our hypergradient approach for RWOC . Specifically , we first introduce a continuous formulation equivalent to ( 1 ) , and then propose a smooth bi-level relaxation with an efficient hypergradient descent algorithm . 2.1 EQUIVALENT CONTINUOUS FORMULATION . We propose a continuous optimization problem equivalent to ( 1 ) . Specifically , we rewrite an equivalent form of ( 1 ) as follows , minw minS∈Rn×n L ( w , S ) = 〈C ( w ) , S〉 subject to S ∈ P , ( 3 ) where P denotes the set of all n× n permutation matrices , C ( w ) ∈ Rn×n is the loss matrix with Cij ( w ) = ‖yi − f ( xi , zj ; w ) ‖22 . Note that we can relax S ∈ P , which is the discrete feasible set of the inner minimization problem of ( 3 ) , to a convex set , without affecting the optimality , as suggested by the next theorem . Proposition 1 . Given any a ∈ Rn and b ∈ Rm , we define Π ( a , b ) = { A ∈ Rn×m : A1m = a , A > 1n = b , Aij ≥ 0 } . The optimal solution to the inner discrete minimization problem of ( 3 ) is also the optimal solution to the following continuous optimization problem , minS∈Rn×n〈C ( w ) , S〉 , s.t . S ∈ Π ( 1n,1n ) . ( 4 ) This is a direct corollary of the Birkhoff-von Neumann theorem ( Birkhoff , 1946 ; Von Neumann , 1953 ) , and please refer to Appendix A for more details . Theorem 1 allows us to replace P in ( 3 ) with Π ( 1n,1n ) , which is also known as the Birkhoff polytope1 ( Ziegler , 2012 ) . Accordingly , we obtain the following continuous formulation , minw minS∈Rn×n〈C ( w ) , S〉 subject to S ∈ Π ( 1n,1n ) . ( 5 ) Remark 1 . In general , equation 3 can be solved by linear programming algorithms ( Dantzig , 1998 ) . 2.2 CONVENTIONAL WISDOM : ALTERNATING MINIMIZATION . Conventional wisdom for solving ( 5 ) suggests to use alternating minimization ( AM , Abid et al . ( 2017 ) ) . Specifically , at the k-th iteration , we first update S by solving S ( k ) = arg minS∈Π ( 1n,1n ) L ( w ( k−1 ) , S ) , and then given S ( k ) , we update w using gradient descent or exact minimization , i.e. , w ( k ) = w ( k−1 ) − η∇wL ( w ( k−1 ) , S ( k ) ) . However , AM works poorly for solving ( 5 ) in practice . This is because w and S have a strong interaction throughout the iterations : A slight change to w may lead to significant change to S. Therefore , the optimization landscape is ill-conditioned , and AM can easily get stuck at local optima . 1This is a common practice in integer programming ( Marcus & Ree , 1959 ) .

Summary: The authors proposed a novel method for regression problems with outliers. The main idea is to first propose a mixed-integer optimization problem for the regression problem and then and the optimization procedure of finding the solutiuon of the problem differentiable, and the objective function of the problem are also be rephrased as a differentiable function. Based on this, an end-to-end learning approach can be established.