Manifold Alignment via Feature Correspondence

We propose a novel framework for combining datasets via alignment of their associated intrinsic dimensions . Our approach assumes that the two datasets are sampled from a common latent space , i.e. , they measure equivalent systems . Thus , we expect there to exist a natural ( albeit unknown ) alignment of the data manifolds associated with the intrinsic geometry of these datasets , which are perturbed by measurement artifacts in the sampling process . Importantly , we do not assume any individual correspondence ( partial or complete ) between data points . Instead , we rely on our assumption that a subset of data features have correspondence across datasets . We leverage this assumption to estimate relations between intrinsic manifold dimensions , which are given by diffusion map coordinates over each of the datasets . We compute a correlation matrix between diffusion coordinates of the datasets by considering graph ( or manifold ) Fourier coefficients of corresponding data features . We then orthogonalize this correlation matrix to form an isometric transformation between the diffusion maps of the datasets . Finally , we apply this transformation to the diffusion coordinates and construct a unified diffusion geometry of the datasets together . We show that this approach successfully corrects misalignment artifacts , and allows for integrated data . 1 INTRODUCTION . In biology and other natural science settings we often have the problem that data are measured from the same system but with different sensors or in different days where sensors are calibrated differently . This is often termed batch effect in biology and can include , for example , drastic variations between subjects , experimental settings , or even times of day when an experiment is conducted . In such settings it is important to globally and locally align the datasets such that they can be combined for effective further analysis . Otherwise , measurement artifacts may dominate downstream analysis . For instance , clustering the data will group samples by measurement time or sensor used rather than by biological or meaningful differences between datapoints . Recent works regard the two datasets as views of the same system and construct a multiview diffusion geometry but all of them require at least partial bijection , if not full one , between views ( Lafon et al. , 2006 ; Ham et al. , 2003 ; 2005 ; Wang & Mahadevan , 2008 ; Tuia & Camps-Valls , 2016 ) . Other work directly attempt to match data points directly in ambient space , or by local data geometry and these can be very sensitive to differences in sampling density rather than data geometry ( Haghverdi et al. , 2018 ) . Here , we present a principled approach called harmonic alignment to for correct this type of effect based on the manifold assumption . The manifold assumption holds that high dimensional data originates from an intrinsically low dimensional smoothly varying space that is mapped via nonlinear functions to observable high dimensional measurements . Thus , we assume that the datasets are from transformed versions of the same low dimensional manifold . We learn the manifolds separately from the two datasets using diffusion geometric approaches and then find an isometric transformation to map from one manifold to the other . Note that we are not aligning points to points . Indeed there may be sampling differences and density differences in the data . However , our manifold learning approach uses an anisotropic kernel that detects the geometry of the data to align rather than point-by-point matching which is done in other methods . Our method involves first embedding each dataset separately into diffusion components , and then finding an isometric transformation that aligns these diffusion representations . To find such transformation , we utilize the duality between diffusion coordinates and geometric harmonics that act as generalized Fourier harmonics in the graph space . The diffusion components are eigenvectors of a Markov-normalized data diffusion operator , whose eigenvalues indicate frequency of the eigenvector . We attempt to find a transformation from one set of eigenvectors to another , via feature correspondences in the data . While datapoint correspondences may be difficult or impossible to obtain since many biological measurements are destructive , feature correspondences are often available . For instance single-cell measurements of cells from the same device , thus containing counts for the same genes , albeit affected by batch differences . Thus when corresponding features are transformed via the graph Fourier transform ( GFT ) into diffusion coordinates , the representations should be similar , with potentially small frequency-proximal perturbations . For instance , slowly varying features across the manifold should be load to low-frequency eigenvectors of the Markov matrix . This insight allows us to create a correlation matrix between the eigenvectors of one dataset to another based on correlation between feature loadings to the eigenvectors . However , since we know that eigenvectors represent frequency harmonics , we need not compute the entire correlation matrix but rather only the near-diagonal values . This implies that the two manifolds must be perturbed such that low and high frequency eigenvector space are similar . We then find a linear transformation between that maps the eigenvectors of one space into those of the other that maximizes these correlations by orthogonalizing this matrix . This transformation allows us to align the two datasets with each other . Finally , given an aligned representation , we build a robust unified diffusion geometry that is invariant batch effects and sample-specific artifacts and low-pass filter this geometry to denoise the unified manifold . Thus , in addition to aligning the manifolds our method denoises the manifolds as well . We demonstrate the results of our method on artificial manifolds created from rotated MNIST digits , corrupted MNIST digits , as well as on single-cell biological data measuring peripheral blood cells . In each case our method successfully aligns the manifolds such that they have appropriate neighbors within and across the two datasets . We show an application to transfer learning where a lazy classifier trained on one dataset is applied to the other dataset after alignment . Further , comparisons with recently developed methods such as the MNN-based method of Haghverdi et al . ( 2018 ) show significant improvements in performance and denoising ability . 2 HARMONIC ALIGNMENT . A typical and effective assumption in machine learning is that high dimensional data originates from an intrinsic low dimensional manifold that is mapped via nonlinear functions to observable high dimensional measurements ; this is commonly referred to as the manifold assumption . Formally , let Md be a hidden d dimensional manifold that is only observable via a collection of n d nonlinear functions f1 , . . . , fn : Md → R that enable its immersion in a high dimensional ambient space as F ( M ) = { f ( z ) = ( f1 ( z ) , . . . , fn ( z ) ) T : z ∈ Md } ⊆ Rn from which data is collected . Conversely , given a dataset X = { x1 , . . . , xN } ⊂ Rn of high dimensional observations , manifold learning methods assume its data points originate from a sampling Z = { zi } Ni=1 ∈ Md of the underlying manifold via xi = f ( zi ) , i = 1 , . . . , n , and aim to learn a low dimensional intrinsic representation that approximates the manifold geometry ofMd . A popular approach towards this manifold learning task is to construct diffusion geometry from data ( Coifman & Lafon , 2006 ) , and embed data into diffusion coordinates , which provide a natural global coordinate system derived from Laplace operator over manifold geometries , as explained in Section 2.1 . However , this approach , as well as other manifold learning ones , implicitly assumes that the feature functions { fj } nj=1 represent data collection technologies ( e.g. , sensors or markers ) that operate in a consistent manner on all samples in Z . While this assumption may be valid in some fields , biological data collection ( and more generally , data collection in empirical sciences ) is often highly affected by a family of phenomena known as batch effects , which introduce nonnegligible variance between different data batches due to various uncontrollable factors . These include , for example , drastic variations between subjects , experimental settings , or even times of day when an experiment is conducted . Therefore , in such settings , one should consider a collection of S samples { X ( s ) } Ss=1 , each originating from feature functions { f ( s ) j } nj=1 that aim to measure the same quantities in the data , but are also affected by sample-dependent artifacts . While each sample can be analyzed to find its intrinsic structure , their union into a single dataset X = ⋃S s=1X ( s ) often yields an incoherent geometry biased by batch effects , where neither the relations between samples or within each sample can be clearly seen . To address such artifacts , and constrct a unified geometry of multiple batches ( i.e. , samples or datasets ) together , we propose to first embed each batch separately in diffusion coordinates , and then find an isometric transformation that aligns these diffusion representations . In order to find such transformation , we utilize the duality between diffusion coordinates and geometric harmonics that act as generalized Fourier harmonics , as shown in graph signal processing ( Shuman et al. , 2013 ) . As explained in Section 2.2 , this duality allows us to capture cross-batch relations between diffusion coordinates , and orthogonalize the resulting matrix to provide a map between batch-specific diffusion representations . Finally , given an aligned representation , we build a robust unified diffusion geometry that is invariant to both batch effects and batch-specific artifacts . While our approach generalizes naturally to any number of batches , for simplicity , we focus our formulation here on the case of two batches . 2.1 DIFFUSION GEOMETRY & GRAPH HARMONICS . The first step in our approach is to capture the intrinsic geometry of each batch X ( s ) using the diffusion maps method from Coifman & Lafon ( 2006 ) , which non-linearly embeds the data in a new coordinate system ( i.e. , diffusion coordinates ) that is often considered as representing a data manifold or more generally a diffusion geometry over the data . The diffusion maps construction starts by considering local similarities defined via a kernel K ( x , y ) , x , y ∈ X ( s ) that capture local neighborhoods in the data . We note that a popular choice for K is the Gaussian kernel e− ‖x−y‖2 σ , where σ > 0 is interpreted as a user-configurable neighborhood size . Next , these similarities are normalized to defined transition probabilities p ( x , y ) = K ( x , y ) ‖K ( x , · ) ‖1 that are organized in an N × N row stochastic matrix P that describes a Markovian diffusion process over the intrinsic geometry of the data . Finally , a diffusion map is defined by taking the eigenvalues 1 = µ1 ≥ µ2 ≥ · · · ≥ µN and ( corresponding ) eignevectors { φj } Nj=1 of P , and mapping each data point x ∈ X ( s ) to an N dimensional vector Φt ( x ) = [ µt1φ1 ( x ) , . . . , µ t NφN ( x ) ] T , where t represents a diffusion-time ( i.e. , number of transitions considered in the diffusion process ) . In this work , we denote the diffusion map for the entire datasetX ( S ) as Φ ( s ) t . We refer the reader to Coifman & Lafon ( 2006 ) for further details and mathematical derivation , but note that in general , as t increases , most of the eigenvalue weights µtj , j = 1 , . . . , N , become numerically negligible , and thus truncated diffusion map coordinates ( i.e. , using only nonnegligible ones ) can be used for dimensionality reduction purposes . Much work has been done in various fields on applications of diffusion maps as a whole , as well as individual diffusion coordinates ( i.e. , eigenvectors of P ) , in data analysis ( Farbman et al. , 2010 ; Barkan et al. , 2013 ; Mahmoudi & Sapiro , 2009 ; Angerer et al. , 2015 ; Haghverdi et al. , 2016 ) . In particular , as discussed in Coifman & Lafon ( 2006 ) and Nadler et al . ( 2006 ) , the diffusion coordinates are closely related to eigenvectors of Laplace operators on manifolds , as well as their discretizations as eigenvectors of graph Laplacians , which were studied previously , for example , in Belkin & Niyogi ( 2002 ) . Indeed , the similarities measured in K can be considered as determining edge weights of a graph structure defined over the data . Formally , we define this graph by considering every data point in X as a vertex on the graph , and then defining weighted edges between them via an N × N adjacency matrix W with Wi , j = K ( xi , xj ) , i , j = 1 . . . N . Then , the ( normalized ) graph Laplacian is defined as L = I−D−1/2WD−1/2 with D being a diagonal degree matrix ( i.e. , with Di , i = ∑N j=1 Wi , j ) . Finally , it is easy to see that L = I −D1/2PD−1/2 , and thus it can be verified that the eigenvectors of L can be written as ψj = D1/2φj , with corresponding eigenvalues λj = 1− µj . It should be noted that if data is uniformly sampled from a manifold ( as considered in Belkin & Niyogi , 2002 ) , these two sets of eigenvectors coincide and the diffusion coordinates can be considered as Laplacian eigenvectors ( or eigenfunctions , in continuous settings ) . A central tenet of graph signal processing is that the Laplacian eigenfunctions { ψj } Nj=1 can be regarded as generalized Fourier harmonics ( Shuman et al. , 2013 ; 2016 ) , i.e. , graph harmonics . In- Algorithm 1 Manifold Alignment via Feature Correspondence Input : Datasets X = { X ( 1 ) , X ( 2 ) } where X ( s ) has N ( s ) observations by d ( s ) features Output : Aligned graph Laplacian L ( Y ) . 1 : for X ( s ) ∈ X do 2 : Compute the anisotropic weight matrix W ( s ) ( Section 2.2 ) and degree matrix D ( s ) 3 : Construct the normalized graph Laplacian L ( s ) and its truncated eigensystem Λ̄ ( s ) = diag [ λ ( s ) i ] N ( s ) i=2 , Ψ̄ ( s ) = [ ψ ( s ) i ] N ( s ) i=1 , with L ( s ) ψ ( s ) i = λ ( s ) i ψ ( s ) i and Λ 0 4 : Compute the diffusion map Φ ( s ) = e−tΛ̄ ( s ) Ψ̄ ( s ) 5 : The spectral domain wavelet transform tensor Ĥ ( s ) i , j , k ( Equation 1 ) . 6 : end for 7 : Compute intraband harmonic correlations between each dataset M′ : , : ,k ( Section 2.2 ) . 8 : Compute the total interband correlation M = ∑τ k=1 M ′ : , : ,k. 9 : Orthogonalize M via SVD , T = UV T 10 : Construct the transformed matrix E = [ Φ ( 1 ) e−tΛ̄ ( 1 ) Ψ̄ ( 1 ) T e−tΛ̄ ( 2 ) Ψ̄ ( 2 ) TT Φ ( 2 ) ] . 11 : Embed E using a Gaussian kernel to obtain L ( Y ) . deed , a classic result in spectral graph theory shows that the discrete Fourier basis can be derived as Laplacian eigenvectors of the ring graphs ( see , e.g . Olfati-Saber , 2007 , Proposition 10 ) . Based on this interpretation , a graph Fourier transform ( GFT ) is defined on graph signals ( i.e. , functions f : X ( s ) → R over the vertices of the graph ) as f̂ ( λj ) = 〈f , ψj〉 , j = 1 , . . . , N , similar to the definition of the classic discrete Fourier transform ( DFT ) . Further , we can also write the GFT in terms of the diffusion coordinates as f̂ ( λk ) = 〈f , D1/2φj〉 , given their relation to graph harmonics . Therefore , up to appropriate weighting , the diffusion coordinates can conceptually be interpreted as intrinsic harmonics of the data , and conversely , the graph harmonics can be considered ( conceptually ) as intrinsic coordinates of data manifolds . In Section 2.2 , we leverage this duality between coordinates and harmonics in order to capture relations between data manifolds of individual batches , and then them in Section 2.3 to align their intrinsic coordinates and construct a unified data manifold over them .

Summary: The authors pointed out that the measurements in biology and natural science suffer from batch effects such as the variations between batches of data measured at different times or by different sensors. In order to analyze different batches of data, an alignment or a calibration is frequently needed. The authors propose to use that though there is variation among different batches, these batches all share an underlying intrinsic manifold structure, which may admit a set of alignable coordinates.