A General Framework for Manifold Alignment

Manifold alignment has been found to be useful in many fields of machine learning and data mining. In this paper we summarize our work in this area and introduce a general framework for manifold alignment. This framework generates a family of approaches to align manifolds by simultaneously matching the corresponding instances and preserving the local geometry of each given manifold. Some approaches like semi-supervised alignment and manifold projections can be obtained as special cases. Our framework can also solve multiple manifold alignment problems and be adapted to handle the situation when no correspondence information is available. The approaches are described and evaluated both theoretically and experimentally, providi ng results showing useful knowledge transfer from one domain to another. Novel applications of our methods including identification of topics shared by multiple document collection s, and biological structure alignment are discussed in the pap r.  In many areas of machine learning and data mining, one is often confronted with situations where the data is in a high dimensional space. Directly dealing with such high dimensional data is usually intractable, but in many cases, the manifold structure underlying the data may have a low intrinsic dimensionality. Manifold alignment builds conn ections between two or more disparate data sets by aligning their underlying manifolds and provides knowledge transfer across the data sets. More formally, given data sets Xk = {x 1 k, · · · , x mk k } wherek ∈ {1, · · · , c} (assumingXk is from manifoldXk)) together with a small fraction of samples labelled with known correspondences, we want to find a correspondence between them. Directly working with the original data instances can be quite difficult, since they ar e in high dimensional spaces and might be defined by totally different features. A solution is to map Xk for differentks to a new space, where instances xa andx j b can be directly compared for any two data sets Xa andXb. Manifold alignment can be used to transfer knowledge across domains. Real-world applications include automatic machine transl tion (Diaz & Metzler 2007), cross-lingual information retrieval, representation and control transfer in Markov dec iCopyright c  Existing manifold alignment approaches can be categorized into two types. The first type (illustrated in Figure 1(A)) includes diffusion map-based alignment (Lafon et al. 2006) and Procrustes alignment (Wang & Mahadevan 2008). These approaches first map the data sets to low dimensional spaces reflecting their intrinsic geometries usi ng a standard (linear like LPP (He & Niyogi 2003) or nonlinear like Laplacian eigenmaps (Belkin & Niyogi 2003)) dimensionality reduction approach. Then some components (like rotational and scaling components) are removed from one set so that the alignment of two sets can be achieved. In this type of alignment, the computation of lower dimensional embedding is done in a unsupervised way (without considering the purpose of alignment), so the resulting embeddings of the two data sets might be quite different. The second type (illustrated in Figure 1(B)) includes semi-supervise d alignment (Ham et al. 2005), manifold projections (Wang & Mahadevan 2009) and semi-definite alignment (Xiong et al. 2007). Semi-supervised alignment first creates a joint manifold representing the union of the given manifolds. Then it maps the joint manifold to a lower dimensional l tentspace preserving local geometry of each manifold, and matching instances in correspondence. Semi-supervised alignment i s based on eigenvalue decomposition. Semi-definite alignment solves a similar problem using a semi-definite programming framework. Manifold projections is a linear approximation of semi-supervised alignment. It directly bui lds connections between features rather than instances and can naturally handle new test instances. One situation that arises in many real world applications is that the given manifolds (defined by totally different fea tures) need to be aligned with no available correspondence information. Solving this problem is rather difficult, if no t impossible, since there are two unknown variables in this problem: the correspondence and the transformation. One such example is control transfer between different Markov decision processes (MDPs), where we want to align state spaces of different tasks. Here, states are usually defined by different features for different tasks and it is hard to fin d correspondences between them. Another example is knowledge transfer between document collections in different la nguages. In this task, correspondence information is usuall y