Unlabeled Principal Component Analysis

1 Introduction . 1.1 Motivation . In principal component analysis , a cornerstone of machine learning and data science , one is given a data matrix X̃ , assumed to be a corrupted version of a ground-truth data matrix X∗ = [ x∗1 · · · x∗n ] ∈ Rm×n , typically but not necessarily assumed to have low rank , and the objective is to estimate X∗ or the column-space S∗ ⊂ Rm of X∗ . The most common types of corruptions that have attracted interest in modern studies are additive sparse perturbations [ 6 , 47 ] , outlier data points that lie away from S∗ [ 43 , 40 ] , and missing entries , the latter also known as low-rank matrix completion [ 7 , 4 ] . Recently , permutations have been emerging as another type of data corruption , typically set in the context of linear regression , where the correspondences between the input and the output data have been partially distorted or even are entirely unavailable [ 38 , 39 , 22 , 17 , 30 , 31 , 46 , 19 , 41 , 36 ] . There , one is given a point x∗ of a linear subspace S∗ , but only up to a permutation of its coordinates , say x̃ = Π∗x∗ with Π∗ an unknown permutation , and the goal is to find x∗ from the data x̃ , S∗ . This Unlabeled Sensing [ 38 , 39 ] problem has many potential machine learning applications , e.g. , record linkage [ 30 , 31 ] , visual [ 26 , 27 ] or textual [ 5 , 28 , 29 ] permutation learning , and matching problems in neuroscience [ 21 ] and biology [ 1 , 42 ] . While methods for unlabeled sensing rely on knowledge of the source subspace S∗ , this is not always known in practice . On the other hand , data of the form X̃ = [ x̃1 , . . . , x̃n ] ∈ Rm×n with x̃j = Π∗jx∗j an unknown permutation of an unknown point x∗j ∈ S∗ , are often available , thus raising the question of whether S∗ can be estimated from X̃ . An important example of this situation is record linkage [ 13 , 20 , 3 ] , where the objective is to integrate data from independent sources , x̃1 , . . . , x̃n ∈ Rm , for subsequent data analysis . Since the entries of different records x̃i ’ s are collected separately , the data matrix X̃ is unlabeled in the sense that , the entries of its i-th row do not necessarily correspond 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to the same entity . Such kind of unlabeled data X̃ also arise in the context of data privatization , where the data provider anonymizes the original data X∗ by permuting each column of X∗ prior to release [ 11 , 16 ] . Data re-identification is a concern since companies with privacy policies , health care providers , and financial institutions may release the collected data after anonymization . Understanding the fundamental limits of re-identifying the original data X∗ from the released ones X̃ is essential for striking a balance between data privacy and data preservation , the subject of a plenary talk in the 2019 International Conference on Machine Learning [ 2 ] . 1.2 Contributions . In this paper we consider the recovery of X∗ from its unlabeled version X̃ , which we term Unlabeled Principal Component Analysis ( UPCA ) . We make the following contributions : • We establish that as long as r : = rank ( X∗ ) < min { m , n } and X∗ is generic ( see Definition 1 ) , then up to a permutation of its rows , X∗ is the only matrix of rank less than or equal to r that is compatible with X̃ . This asserts that UPCA is a well-posed problem , since the inherent ambiguity of whether X̃ comes from X∗ or a row-permuted version of X∗ is in most cases practically harmless . • We establish that in this basic formulation UPCA is a purely algebraic problem , by exhibiting a polynomial system of equations parametrized by X̃ , whose solutions are all the rowpermutations of X∗ ; solving the UPCA problem amounts to obtaining one such solution . • Inasmuch as solving this polynomial system is in principle NP-hard , we introduce an efficient algorithmic pipeline for the practically relevant case where a significant part of the data have undergone the same permutation , while the rest of the points have been permuted arbitrarily ; in the case of record linkage this would correspond to one of the records having much larger size than the others . The first stage of the pipeline employs PCA methods with robustness to outliers [ 43 , 33 , 25 , 45 , 37 , 48 , 18 ] to produce an estimate Ŝ of S∗ from X̃ ; the second stage of the pipeline uses unlabeled sensing methods [ 30 , 32 , 36 , 23 ] to furnish an estimate X̂ of X∗ from Ŝ and X̃ . • We introduce a simple but efficient algorithm for unlabeled sensing based on least-squares with recursive filtration ( Algorithm 2 ) . • We assess our algorithmic pipeline on synthetic data , face images , educational and medical records , with encouraging results . 1.3 Related work . Unlabeled sensing . There is a large literature on application-specific problems that involve lack of correspondences , e.g . in computer vision or statistics ; here we just review four recent methods for unlabeled sensing that will be used in this paper . In unlabeled sensing one is given a subspace S∗ ⊂ Rm of dimension r and a point x̃ which is some permuted version of a point x∗ ∈ S∗ and the goal is to recover x∗ from S∗ and x̃ . A critical distinction among methods in the literature is the sparsity level α of the permutation , that is the ratio of coordinates that are moved by the permutation . The case of dense permutations ( α = 1 ) is extremely challenging , with existing methods only able to handle small ranks r. We consider two methods known to perform best in this regime . The first one is the algebraic-geometric AIEM method of [ 36 ] . This has linear complexity in m and instead concentrates its effort on solving a polynomial system of r equations in r variables to produce an initialization for an expectation maximization algorithm . Currently , this method is efficient for r ≤ 5 and intractable otherwise . A very different method is CCV-Min of [ 23 ] , which proceeds via branch- & -bound together with concave minimization and can handle ranks r ≤ 8 . For sparse permutations ( small α ) dealing with higher ranks becomes possible [ 30 , 32 ] . The ` 1-RR algorithm of [ 30 ] applies an ` 1 robust linear regression relaxation , and it works when α ≤ 0.5 . Another approach is the Pseudo-Likelihood method ( PL ) of [ 32 ] , which fits a two-component mixture density for each entry of x̃ , one accounting for fixed data and the other for permuted data . The fitting is done via a combination of hypothesis testing , reweighed least-squares , and alternating minimization . Empirically , PL can tolerate up to α = 0.7 but it is sensitive to the particular basis of S∗ used to generate x̃ . Robust PCA with outliers . PCA methods with robustness to outliers will also play a role in this paper . Among a large literature , we review four modern methods . In that context , it is assumed that X̃ can be partitioned into inlier points that lie in an unknown r-dimensional subspace S∗ ⊂ Rm and outlier points that lie away from S∗ , and the goal is to recover S∗ from X̃ . The Outlier Pursuit ( OP ) method of [ 43 ] decomposes X̃ via convex optimization into the sum of low-rank and column-sparse parts . The convex method of [ 45 ] , a successor of [ 33 ] , referred to as Self-Expr , solves a self-expressive elastic net problem so that each x̃j is expressed as an ` 2-regularized sparse linear combination of the other points . The self-expressive coefficients define transition probabilities of a random walk on a graph , and the average of the t-step transition probability distributions is used as a score for inliers vs. outliers , with higher scores expected for the former . The Coherence Pursuit ( CoP ) method of [ 25 ] is based on the following simple but effective principle : with X̃−j the matrix X̃ with column j removed , for each x̃j one computes its coherence X̃ > −jx̃j with the rest of the points . As it turns out , inliers tend to have coherences of higher ` 2-norm than outliers , and the top r x̃j ’ s are taken to span Ŝ . Finally , the Dual Principal Component Pursuit ( DPCP ) of [ 37 ] solves an ` 1 non-smooth problem on the Grassmannian via iterated-reweighed least-squares ( IRLS ) , to compute an orthonormal basis for the orthogonal complement of the subspace . A dual formulation with a randomized singular value decomposition incorporated in the IRLS procedure is known as Fast Median Subspace [ 18 ] . In sharp contrast to other convex robust-PCA methods , DPCP has been shown in [ 48 , 9 ] to tolerate as many outliers as the square of the number of inliers . 2 Theoretical Foundations . 2.1 Problem Formulation . Let us denote by Pm the set of all permutations of coordinates of Rm . We let X∗ = [ x∗1 · · · x∗n ] ∈ Rm×n be our ground-truth data matrix with rank r < min { m , n } and column-space S∗ = C ( X∗ ) , and we suppose that the available data are X̃ = [ x̃1 · · · x̃n ] = [ Π∗1x∗1 · · ·Π∗nx∗n ] ∈ Rm×n , ( 1 ) where each Π∗j ∈ Pm is an unknown permutation . Let Pnm = ∏ i∈ [ n ] Pm be n ordered copies of Pm , where [ n ] = { 1 , . . . , n } . For π = ( Π1 , . . . , Πn ) ∈ Pnm we set π ( X̃ ) = [ Π1x̃1 · · ·Πnx̃n ] . We pose Unlabeled Principal Component Analysis ( UPCA ) as the following rank minimization problem : min π∈Pnm rankπ ( X̃ ) ( 2 ) First , note that ( 2 ) never has a unique solution , because if π = ( Π1 , . . . , Πn ) is a solution , then so is π ′ = ( ΠΠ1 , . . . , ΠΠn ) , where Π ∈ Pm is any permutation . This reveals an inherent ambiguity of UPCA : it is only possible to recover X∗ from X̃ up to a permutation ΠX∗ of its rows . On the other hand , this is rather harmless in many situations , since ΠX∗ is the same dataset as X∗ except that the row-features appear now in some different order . Thus , our hope in formulating ( 2 ) is that the only solutions are of the form π = ( ΠΠ∗1 > , . . . , ΠΠ∗n > ) with Π ranging across Pm and Π∗j as in ( 1 ) . However , without any other assumptions on the data X∗ , there could in principle be additional undesired permutations that also give rankX∗ , or even worse , the minimum rank in ( 2 ) could be lower than r = rankX∗ . Our results show that for generic enough data , such pathological situations do not occur , and the only solutions to ( 2 ) are the ones associated with row-permutations of X∗ .

Summary: This paper introduces unlabeled principal component analysis (UPCA), which refers to the reconstruction of the ground-truth data matrix X^* from its unlabeled version \tilde{X}, whose each column is generated from a permutation of the coordinates of the corresponding column of X^*. The authors first establish that UPCA is well-posed by showing that if X^* is sufficiently "generic", then the solution to the UPCA presented in Eq. (2) applied to \tilde{X} is X^* (up to a global permutation matrix). The authors then establish that X^* (up to a global permutation matrix) is the unique solution of a polynomial system of equations parametrized by \tilde{X}. Based on these theoretical results, the authors propose a two-stage algorithmic pipeline for UPCA for a practically relevant case where only a fraction of the data has been permuted. The first stage uses robust PCA methods to estimate \hat{S} of the ground-truth column space S^* from \tilde{X}. The second stage uses unlabeled sensing methods to estimate \hat{X} of the ground-truth data matrix X^* from \hat{S} and \tilde{X}. The authors also introduce a novel algorithm for unlabeled sensing based on least-squares with recursive filtration (LSRF).