Unlabeled Principal Component Analysis

1 Introduction . 1.1 Motivation . In principal component analysis , a cornerstone of machine learning and data science , one is given a data matrix X̃ , assumed to be a corrupted version of a ground-truth data matrix X∗ = [ x∗1 · · · x∗n ] ∈ Rm×n , typically but not necessarily assumed to have low rank , and the objective is to estimate X∗ or the column-space S∗ ⊂ Rm of X∗ . The most common types of corruptions that have attracted interest in modern studies are additive sparse perturbations [ 6 , 47 ] , outlier data points that lie away from S∗ [ 43 , 40 ] , and missing entries , the latter also known as low-rank matrix completion [ 7 , 4 ] . Recently , permutations have been emerging as another type of data corruption , typically set in the context of linear regression , where the correspondences between the input and the output data have been partially distorted or even are entirely unavailable [ 38 , 39 , 22 , 17 , 30 , 31 , 46 , 19 , 41 , 36 ] . There , one is given a point x∗ of a linear subspace S∗ , but only up to a permutation of its coordinates , say x̃ = Π∗x∗ with Π∗ an unknown permutation , and the goal is to find x∗ from the data x̃ , S∗ . This Unlabeled Sensing [ 38 , 39 ] problem has many potential machine learning applications , e.g. , record linkage [ 30 , 31 ] , visual [ 26 , 27 ] or textual [ 5 , 28 , 29 ] permutation learning , and matching problems in neuroscience [ 21 ] and biology [ 1 , 42 ] . While methods for unlabeled sensing rely on knowledge of the source subspace S∗ , this is not always known in practice . On the other hand , data of the form X̃ = [ x̃1 , . . . , x̃n ] ∈ Rm×n with x̃j = Π∗jx∗j an unknown permutation of an unknown point x∗j ∈ S∗ , are often available , thus raising the question of whether S∗ can be estimated from X̃ . An important example of this situation is record linkage [ 13 , 20 , 3 ] , where the objective is to integrate data from independent sources , x̃1 , . . . , x̃n ∈ Rm , for subsequent data analysis . Since the entries of different records x̃i ’ s are collected separately , the data matrix X̃ is unlabeled in the sense that , the entries of its i-th row do not necessarily correspond 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to the same entity . Such kind of unlabeled data X̃ also arise in the context of data privatization , where the data provider anonymizes the original data X∗ by permuting each column of X∗ prior to release [ 11 , 16 ] . Data re-identification is a concern since companies with privacy policies , health care providers , and financial institutions may release the collected data after anonymization . Understanding the fundamental limits of re-identifying the original data X∗ from the released ones X̃ is essential for striking a balance between data privacy and data preservation , the subject of a plenary talk in the 2019 International Conference on Machine Learning [ 2 ] . 1.2 Contributions . In this paper we consider the recovery of X∗ from its unlabeled version X̃ , which we term Unlabeled Principal Component Analysis ( UPCA ) . We make the following contributions : • We establish that as long as r : = rank ( X∗ ) < min { m , n } and X∗ is generic ( see Definition 1 ) , then up to a permutation of its rows , X∗ is the only matrix of rank less than or equal to r that is compatible with X̃ . This asserts that UPCA is a well-posed problem , since the inherent ambiguity of whether X̃ comes from X∗ or a row-permuted version of X∗ is in most cases practically harmless . • We establish that in this basic formulation UPCA is a purely algebraic problem , by exhibiting a polynomial system of equations parametrized by X̃ , whose solutions are all the rowpermutations of X∗ ; solving the UPCA problem amounts to obtaining one such solution . • Inasmuch as solving this polynomial system is in principle NP-hard , we introduce an efficient algorithmic pipeline for the practically relevant case where a significant part of the data have undergone the same permutation , while the rest of the points have been permuted arbitrarily ; in the case of record linkage this would correspond to one of the records having much larger size than the others . The first stage of the pipeline employs PCA methods with robustness to outliers [ 43 , 33 , 25 , 45 , 37 , 48 , 18 ] to produce an estimate Ŝ of S∗ from X̃ ; the second stage of the pipeline uses unlabeled sensing methods [ 30 , 32 , 36 , 23 ] to furnish an estimate X̂ of X∗ from Ŝ and X̃ . • We introduce a simple but efficient algorithm for unlabeled sensing based on least-squares with recursive filtration ( Algorithm 2 ) . • We assess our algorithmic pipeline on synthetic data , face images , educational and medical records , with encouraging results . 1.3 Related work . Unlabeled sensing . There is a large literature on application-specific problems that involve lack of correspondences , e.g . in computer vision or statistics ; here we just review four recent methods for unlabeled sensing that will be used in this paper . In unlabeled sensing one is given a subspace S∗ ⊂ Rm of dimension r and a point x̃ which is some permuted version of a point x∗ ∈ S∗ and the goal is to recover x∗ from S∗ and x̃ . A critical distinction among methods in the literature is the sparsity level α of the permutation , that is the ratio of coordinates that are moved by the permutation . The case of dense permutations ( α = 1 ) is extremely challenging , with existing methods only able to handle small ranks r. We consider two methods known to perform best in this regime . The first one is the algebraic-geometric AIEM method of [ 36 ] . This has linear complexity in m and instead concentrates its effort on solving a polynomial system of r equations in r variables to produce an initialization for an expectation maximization algorithm . Currently , this method is efficient for r ≤ 5 and intractable otherwise . A very different method is CCV-Min of [ 23 ] , which proceeds via branch- & -bound together with concave minimization and can handle ranks r ≤ 8 . For sparse permutations ( small α ) dealing with higher ranks becomes possible [ 30 , 32 ] . The ` 1-RR algorithm of [ 30 ] applies an ` 1 robust linear regression relaxation , and it works when α ≤ 0.5 . Another approach is the Pseudo-Likelihood method ( PL ) of [ 32 ] , which fits a two-component mixture density for each entry of x̃ , one accounting for fixed data and the other for permuted data . The fitting is done via a combination of hypothesis testing , reweighed least-squares , and alternating minimization . Empirically , PL can tolerate up to α = 0.7 but it is sensitive to the particular basis of S∗ used to generate x̃ . Robust PCA with outliers . PCA methods with robustness to outliers will also play a role in this paper . Among a large literature , we review four modern methods . In that context , it is assumed that X̃ can be partitioned into inlier points that lie in an unknown r-dimensional subspace S∗ ⊂ Rm and outlier points that lie away from S∗ , and the goal is to recover S∗ from X̃ . The Outlier Pursuit ( OP ) method of [ 43 ] decomposes X̃ via convex optimization into the sum of low-rank and column-sparse parts . The convex method of [ 45 ] , a successor of [ 33 ] , referred to as Self-Expr , solves a self-expressive elastic net problem so that each x̃j is expressed as an ` 2-regularized sparse linear combination of the other points . The self-expressive coefficients define transition probabilities of a random walk on a graph , and the average of the t-step transition probability distributions is used as a score for inliers vs. outliers , with higher scores expected for the former . The Coherence Pursuit ( CoP ) method of [ 25 ] is based on the following simple but effective principle : with X̃−j the matrix X̃ with column j removed , for each x̃j one computes its coherence X̃ > −jx̃j with the rest of the points . As it turns out , inliers tend to have coherences of higher ` 2-norm than outliers , and the top r x̃j ’ s are taken to span Ŝ . Finally , the Dual Principal Component Pursuit ( DPCP ) of [ 37 ] solves an ` 1 non-smooth problem on the Grassmannian via iterated-reweighed least-squares ( IRLS ) , to compute an orthonormal basis for the orthogonal complement of the subspace . A dual formulation with a randomized singular value decomposition incorporated in the IRLS procedure is known as Fast Median Subspace [ 18 ] . In sharp contrast to other convex robust-PCA methods , DPCP has been shown in [ 48 , 9 ] to tolerate as many outliers as the square of the number of inliers . 2 Theoretical Foundations . 2.1 Problem Formulation . Let us denote by Pm the set of all permutations of coordinates of Rm . We let X∗ = [ x∗1 · · · x∗n ] ∈ Rm×n be our ground-truth data matrix with rank r < min { m , n } and column-space S∗ = C ( X∗ ) , and we suppose that the available data are X̃ = [ x̃1 · · · x̃n ] = [ Π∗1x∗1 · · ·Π∗nx∗n ] ∈ Rm×n , ( 1 ) where each Π∗j ∈ Pm is an unknown permutation . Let Pnm = ∏ i∈ [ n ] Pm be n ordered copies of Pm , where [ n ] = { 1 , . . . , n } . For π = ( Π1 , . . . , Πn ) ∈ Pnm we set π ( X̃ ) = [ Π1x̃1 · · ·Πnx̃n ] . We pose Unlabeled Principal Component Analysis ( UPCA ) as the following rank minimization problem : min π∈Pnm rankπ ( X̃ ) ( 2 ) First , note that ( 2 ) never has a unique solution , because if π = ( Π1 , . . . , Πn ) is a solution , then so is π ′ = ( ΠΠ1 , . . . , ΠΠn ) , where Π ∈ Pm is any permutation . This reveals an inherent ambiguity of UPCA : it is only possible to recover X∗ from X̃ up to a permutation ΠX∗ of its rows . On the other hand , this is rather harmless in many situations , since ΠX∗ is the same dataset as X∗ except that the row-features appear now in some different order . Thus , our hope in formulating ( 2 ) is that the only solutions are of the form π = ( ΠΠ∗1 > , . . . , ΠΠ∗n > ) with Π ranging across Pm and Π∗j as in ( 1 ) . However , without any other assumptions on the data X∗ , there could in principle be additional undesired permutations that also give rankX∗ , or even worse , the minimum rank in ( 2 ) could be lower than r = rankX∗ . Our results show that for generic enough data , such pathological situations do not occur , and the only solutions to ( 2 ) are the ones associated with row-permutations of X∗ .

Summary: - The authors describe and tackle the problem of unlabelled principal component analysis, a variant of PCA where the columns of the data matrix might have been permuted arbitrarily with different permutations for each column. The problem shares some of its motivation with the easier problem of unlabelled sensing, where the principal subspace is known. - The authors divide the problem of UPCA into two natural steps: i) Identifying a rank-r subspace S* where all data points lie if permitted correctly, and; ii) mappings the permuted data points onto the identified subspace. These two steps can be naturally addressed by existing lines of work: robust PCA for former and unlabelled sensing for the latter. - The authors use a algebraic framework to formulate and solve the problem. The main theoretical result is a powerful one: under genericity of the true solution, then any solution of the rank minimization formulation of the problem recovers the ground truth up to a permutation. - A combinatorial solution is prohibitively expensive. One could consider a polynomial system solver, but the authors correctly point out that scalability and robustness to noise would pose separate challenges and leave that to future work.  - Instead, the authors focus on specific subclasses of the UPCA problem where only a fraction of the entries are permuted. Under this assumption it is indeed possible to have efficient solutions of the problem, and the proposed two-stage workflow seems well suited. - Finally, the authors provide extensive experiments on their methodology on synthetic and real datasets.