Co-manifold learning with missing data

1 INTRODUCTION . Dimension reduction plays a key role in exploratory data analysis , data visualization , clustering and classification . Techniques range from the classical PCA and nonlinear manifold learning to deep autoencoders ( Tenenbaum et al. , 2000 ; Roweis & Saul , 2000 ; Belkin & Niyogi , 2003 ; Coifman & Lafon , 2006 ; Vincent et al. , 2008 ; Rifai et al. , 2011 ; Kingma & Welling , 2014 ) . These techniques focus on only one mode of the data , often the observations ( columns ) which are measurements in a high-dimensional feature space ( rows ) , and exploit correlations among the features to reduce the dimension of the feature vectors and obtain the underlying low-dimensional geometry of the observations . Yet for many data matrices , for example in gene expression studies , recommendation systems , sensor networks , and word-document analysis , correlations exist among both observations and features . In these cases , we seek a method that can exploit the correlations among both the rows and columns of a data matrix to better learn lower-dimensional representations of both . Biclustering methods , which extract distinct biclusters along both rows and columns , give a partial solution to performing simultaneous dimension reduction on the rows and columns of a data matrix . In certain settings , however , assuming a bi-clustering model is too restrictive and results in breaking up smooth geometries into artificial disjoint clusters that do not match the actual structure of the data . This can occur when the true geometry is one of overlapping rather than disjoint clusters , for example in word-document analysis ( Ahn et al. , 2010 ) , or when the underlying structure is not one of clusters at all but rather a smooth manifold ( Gavish & Coifman , 2012 ) . Thus , we consider a more general viewpoint : data matrices possess geometric relationships between their rows ( features ) and columns ( observations ) such that both modes lie on low-dimensional manifolds . Furthermore , the relationships between the rows may be informed by the relationships between the columns , and vice versa . Several recent papers ( Gavish & Coifman , 2012 ; Ankenman , 2014 ; Mishne et al. , 2016 ; Shahid et al. , 2016 ; Mishne et al. , 2017 ; Yair et al. , 2017 ) exploit this coupled relationship to coorganize matrices and infer underlying row and column embeddings . Further complicating the story is that such matrices may suffer from missing values , due to measurement corruptions and limitations . Missing values can sabotage efforts to learn the low dimensional manifold underlying the data . Specifically , kernel-based methods rely on calculating a similarity matrix between observations , whose eigendecomposition yields a new embedding of the data . As the number of missing entries grows , the distances between points are increasingly distorted , resulting in poor representation of the data in the low-dimensional space ( Gilbert & Sonthalia , 2018 ) . Matrix completion algorithms assume the data is low-rank and fill in the missing values by fitting a global linear subspace to the data . Yet , this may fail when the data lies on a nonlinear manifold . Manifold learning in the missing data scenario has been addressed by a few recent papers . Nonlinear Principle Component Analysis ( NLPCA ) ( Scholz et al. , 2005 ) uses an autoencoder neural network , where the middle layer serves to learn a low-dimensional embedding of the data , and the trained autoencoder is used to fill in missing values . Missing Data Recovery through Unsupervised Regression ( Carreira-Perpin & Lu , 2011 ) first fills in the missing data with linear matrix completion methods , then calculates a non-linear embedding of the data and incorporates this embedding in an optimization problem to fill in the missing values . Recently Gilbert & Sonthalia ( 2018 ) proposed MR-MISSING which first calculates an initial distance matrix using only non-missing entries and then uses the increase-only-metric-repair ( IOMR ) method to fix the distance matrix so that it is a metric from which they calculate an embedding . None of these methods consider the co-manifold setting , where the coupled structure of the rows and the columns can be used to fill in the data , and to calculate an embedding . In this paper , we introduce a new method for performing joint dimension reduction on the rows and columns of a data matrix , which we term co-manifold learning , in the missing data setting . We build on two recent lines of work on co-organizing the rows and columns of a data matrix ( Gavish & Coifman , 2012 ; Mishne et al. , 2016 ; 2017 ) and convex optimization methods for performing coclustering ( Chi et al. , 2017 ; 2018 ) . The former provide a flexible framework for jointly organizing rows and columns but lacks algorithmic convergence guarantees . The latter provides convergence guarantees but does not take full advantage of the multiple scales of the data revealed in the solution . In the first stage of our approach , rather than inferring biclusters at a single scale , we use a multiscale optimization framework to fill in the data at fine to coarse scales while imposing smoothness on both the rows and the columns . The scales of the solutions are encoded in a pair of joint cost parameters along the rows and columns . Next , we define a new multi-scale metric based on the filled-in matrix across all scales , which we then use to calculate nonlinear embeddings of the rows and columns . Thus our approach yields three results : a collection of smoothed estimates of the matrix , pairwise distances on the rows and columns that better estimate the geometry of the complete data matrix , and corresponding nonlinear embeddings ( see Figure 1 ) . We will demonstrate in experimental results that our method reveals meaningful representations in coupled datasets with missing entries , whereas other methods are capable of revealing a meaningful representation only along one of the modes . The paper is organized as follows . We present the optimization framework in Section 2 , the new multi-scale metric for co-manifold learning in Section 3 and experimental results in Section 4 . 2 CO-CLUSTERING AN INCOMPLETE DATA MATRIX . We seek a collection of complete matrix approximations of a partially observed data matrix X ∈ Rm×n that have been smoothed along their row and columns to varying degrees . This collection will serve in computing row and column multi-scale distances to better estimate the pairwise distances of the complete data matrix . Let [ m ] denote the set of indices { 1 , . . . , m } , and let Θ ⊆ [ m ] × [ n ] be a subset of the indices that correspond to observed entries of X , and let PΘ denote the projection operator ofm×nmatrices onto an index set Θ , i.e . [ PΘ ( X ) ] ij is xij if ( i , j ) ∈ Θ and is 0 otherwise . We seek a minimizer U ( γr , γc ) of the following function . f ( U ; γr , γc ) = 1 2 ‖PΘ ( X ) − PΘ ( U ) ‖2F + γrJr ( U ) + γcJc ( U ) . ( 1 ) The quadratic term quantifies how well U approximates X on the observed entries , while the two roughness penalties , Jr ( U ) and Jc ( U ) , incentivize smoothness across the rows and columns of U . The nonnegative parameters γr and γc tune the tradeoff between how well U agrees with X over Θ and how smooth U is with respect to its rows and columns . By tuning γr and γc , we obtain estimates of X at varying scales of row and column smoothness . In this paper , we use roughness penalties of the following forms Jr ( U ) = ∑ ( i , j ) ∈Er Ω ( ‖Ui· −Uj·‖2 ) and Jc ( U ) = ∑ ( i , j ) ∈Ec Ω ( ‖U·i −U·j‖2 ) , ( 2 ) where Ui· ( U·i ) denotes the ith row ( column ) of the matrix U . The index sets Er and Ec denote the edge sets of row and column graphs that encode a preliminary data-driven assessment of the similarities between rows and columns of the data matrix . The function Ω , which maps [ 0 , ∞ ) into [ 0 , ∞ ) , will be explained shortly . Variations on the optimization problem of minimizing ( 1 ) have been previously proposed in the literature . When there is no data missing , i.e . Θ = [ m ] × [ n ] and Ω is a linear mapping , minimizing the objective in ( 1 ) produces a convex biclustering problem ( Chi et al. , 2017 ) . Additionally , if either γr or γc is zero , then we obtain convex clustering ( Pelckmans et al. , 2005 ; Hocking et al. , 2011 ; Lindsten et al. , 2011 ; Chi & Lange , 2015 ) . If we take Ω to be a nonlinear concave function , problem ( 1 ) reduces to an instance of concave penalized regressionbased clustering ( Pan et al. , 2013 ; Marchetti & Zhou , 2014 ; Wu et al. , 2016 ) . The convergence properties of our co-clustering procedure will rely on the following two assumptions . Assumption 2.1 The row and column graphs Er and Ec are connected , i.e . the row graph is connected if for any pair of rows , indexed by i and j with i 6= j , there exists a sequence of indices i → k → · · · → l → j such that ( i , k ) , . . . , ( l , j ) ∈ Er . A column graph is connected under analogous conditions . Assumption 2.2 The function Ω : [ 0 , ∞ ) 7→ [ 0 , ∞ ) is ( i ) concave and continuously differentiable on ( 0 , ∞ ) , ( ii ) vanishes at the origin , i.e . Ω ( 0 ) = 0 , ( iii ) is increasing on [ 0 , ∞ ) , and ( iv ) has finite directional derivative at the origin . Figure 4 in Appendix A plots an example of a function Ω that satisfies Assumption 2.2 . For concreteness , in the rest of this paper , we use the following function Ω Ω ( z ) = 1 2 ∫ z 0 1√ ζ + dζ , ( 3 ) where is a small positive number , e.g . 10−12 . The key feature of Ω in ( 3 ) , which satisfies Assumption 2.2 , is that when it is used in the roughness penalties ( 2 ) small differences between rows and columns are penalized significantly more than larger differences resulting in more aggressive smoothing of small noisy variations and leaving intact more significant systematic variations . Specifically , functions that satisfy Assumption 2.2 are not differentiable at the origin and therefore incentivize sparsity in the differences in the rows and columns . Consequently , small noisy variations between pairs of rows and columns are eliminated completely for sufficiently large γr and γc . This is in contrast with commonly used quadratic penalties . For example , replacing Jr ( U ) and Jc ( U ) by quadratic row and column penalties Jr ( U ) = 1 2 ∑ ( i , j ) ∈Er wij‖Ui· −Uj·‖22 and Jc ( U ) = 1 2 ∑ ( i , j ) ∈Ec w̃ij‖U·i −U·j‖22 , ( 4 ) gives a version of matrix completion on graphs ( Kalofolias et al. , 2014 ; Rao et al. , 2015 ) , where wij and w̃ij are fixed row and column weights inferred from the data . Shahid et al . ( 2016 ) also use quadratic row and column penalties to perform joint linear dimension reduction on the rows and columns of a data matrix . Penalties like the ones given in ( 4 ) , unlike the ones considered in Assumption 2.2 , smooth out more significant systematic variations more aggressively and shrink , but do not completely eliminate , small noisy variations . A simple concrete example illuminates the differences between roughness penalties considered in this paper and commonly used existing penalties . Let U1· = ( 1 0 ) T and U2· = ( 1 + δ 0 ) T and take Ω to be as defined in ( 3 ) . Then , ‖U1· −U2·‖2 = δ , Ω ( ‖U1· −U2·‖2 ) ≈ √ δ , and ‖U1· −U2·‖22 = δ2 . Suppose there is a small difference between the first and second rows of U , e.g . δ = 10−4 . Then ‖U1· −U2·‖2 = 10−4 , Ω ( ‖U1· −U2·‖2 ) ≈ 10−2 , and ‖U1· −U2·‖22 = 10−8 . Thus , small differences are penalized the most using the concave Ω and the least by the convex quadratic penalty . Suppose there is a large difference between the first and second rows of U , e.g . δ = 104 . Then ‖U1· −U2·‖2 = 104 , Ω ( ‖U1· −U2·‖2 ) ≈ 102 , and ‖U1· −U2·‖22 = 108 . Thus , large differences are penalized the least using the concave Ω and the most by the convex quadratic penalty . The practical consequence of the differences highlighted by the example above is that the convex penalties , either when Ω is linear or quadratic , do not introduce enough smoothing for small differences and too much smoothing for large differences . Indeed , ( Pan et al. , 2013 ; Marchetti & Zhou , 2014 ; Wu et al. , 2016 ) showed that superior clustering results could be had , when only Jr or Jc is used , by using concave Ω. Chi et al . ( 2017 ) also showed that empirically that the solution to the convex biclustering problem tended to identify too many biclusters and consequently also introduced a one-step reweighted convex biclustering refinement , which recovered the true biclusters more accurately in simulation experiments . The reweighting refinement can be seen as taking a single step in an iterative algorithm for minimizing ( 1 ) inexactly when Ω is concave ( Chi et al. , 2018 ; Chi & Steinerberger , 2018 ) . As our method combines biclustering of incomplete data at different scales , or values of γr and γc , consequently we extend the reweighting refinement introduced by Chi et al . ( 2017 ) in Section 2.1 . Our problem formulation ( 1 ) is distinct from related problem formulations in the following ways : 1 . Rows and columns of U are simultaneously shrunk towards each other as the parameters γr and γc increase . Note that this shrinkage procedure is fundamentally different from methods like the clustered dendrogram , which independently cluster the rows and columns as well as alternating partition tree construction procedures ( Gavish & Coifman , 2012 ; Mishne et al. , 2016 ) . 2 . Our ultimate goal is not to perform matrix completion ( though this is a by-product of our approach ) but rather to perform joint row and column dimension reduction . 3 . Our work generalizes both Shahid et al . ( 2016 ) and Chi et al . ( 2017 ) in that we seek the flexibility of performing non-linear dimension reduction on the rows and columns of the data matrix , i.e . a more general manifold organization than a co-clustered structure . 4 . Instead of determining an optimal single scale of the solution as in Shahid et al . ( 2016 ) ; Chi et al . ( 2017 ) , we recognize that the multiple scales of the different solutions can be aggregated to better estimate the underlying geometry , similar to the tree-based Earth mover ’ s distance proposed in Ankenman ( 2014 ) ; Mishne et al . ( 2017 ) .

Summary: This paper presents a joint learning method for filling missing value and bi-clustering. The method extends (Chi et al. 2017), using a penalized matrix approximation. The proposed method is tested on three data sets, where two are synthetic and one small real-world data matrix. The presented method is claimed to be better than two classical approaches Nonlinear PCA and Diffusion Maps.