Domain Adaptation via Low-Rank Basis Approximation

1 INTRODUCTION . Supervised learning and in particular classification is an essential task in machine learning with a broad range of applications . The obtained models are used to predict the label of unseen test samples . A basic assumption in supervised learning is that the underlying domain or distribution is not changing between training and test samples . If the domain is changing from one task to a related but different task , one would like to reuse the available learning model . Domain differences are quite common in real-world scenarios and eventually , lead to substantial performance drops ( Weiss et al. , 2016 ) . A domain adaptation example is the classification of web pages . A classifier is trained in the domain of university web pages with a word distribution according to universities . In the test scenario , the domain has changed to non-university web pages , where the word distribution is related , but may not be similar to the training distribution . More formally , let Z = { z1 , . . . , zm } ∈ Rd be source data sampled from the source domain distribution p ( z ) and let X = { x1 , . . . , xn } ∈ Rd be target data from the target domain distribution p ( x ) . Traditional machine learning assumes similar distributions , i.e . p ( z ) ∼ p ( x ) , but domain adaptation assumes different distributions , i.e . p ( z ) 6= p ( x ) and appears in the web page example where Z could be data of university websites and X are data of non-university websites . Multiple domain adaptation techniques have been already proposed , following different strategies and improving the prediction performance of underlying classification algorithms in test scenarios ( Weiss et al. , 2016 ; Pan & Yang , 2010 ) . The common domain adaptation approach requires a large number of source or target samples , which is indeed a disadvantage of many domain adaptation approaches and is not guaranteed in restricted environments where labeling is expensive ( Weiss et al. , 2016 ) . Further , learning a matrix for the alignment of domain distributions does not necessarily lead to effective adaptation shown by ( Raab & Schleif , 2018 ) , where domain differences are explicitly solved by overriding the basis of one domain with another and is competitive to current state of the art domain adaptation algorithms ( Gong et al. , 2012 ; Long et al. , 2015 ; 2013 ; Pan et al. , 2011 ; Fernando et al. , 2013 ; Sun et al. , 2016 ) . The main contribution of this work is to improve the Basis-Transfer ( Raab & Schleif , 2018 ) approach by deriving a more simplified closed-form solution of the least-squares problem and enhance this by a Nyström based dimensionality reduction . The resulting method has a better prediction performance and makes it easer to apply . Further , it is the fastest domain adaptation algorithm in terms of computational complexity , while maintaining its excellent performance by using only a subset of source and target data . It determines a target subspace representation for both domains and transfers target basis information to source data via Nyström Singular Value Decomposition ( SVD ) with class-wise sampling . Further , we show that Nyström approximation is well suited for nonsquare landmark matrices , in contrast to common decomposition ( Nemtsov et al. , 2016 ) by applying post-processing . The rest of the paper is organized as follows : We give an overview of related work in section 2 . The underlying mathematical concepts are given in section 3 . The proposed approach is discussed in section 4 , followed by an experimental part in section 5 , addressing the classification performance and computational time . A summary with a discussion of open issues is provided in the conclusion at the end of the paper . More detailed derivations , description of experiments , implementation details and additional results can be found in the appendix . 2 RELATED WORK . In general , homogeneous transfer learning ( Weiss et al. , 2016 ) approaches , or domain adaptation ( DA ) , distinguish roughly between the following strategies : Methods implementing the symmetric feature adaptation ( Weiss et al. , 2016 ) are trying to find a common latent subspace for source and target domain to reduce marginal distribution differences , such that the underlying structure of the data is preserved in the subspace . A baseline approach for symmetric feature adaptation is the Transfer Component Analysis ( TCA ) ( Pan et al. , 2011 ) . TCA finds a suitable subspace transformation called transfer components via minimizing the Maximum Mean Discrepancy ( MMD ) in the Reproducing Kernel Hilbert Space ( RKHS ) . The Geodesic Flow Kernel ( GFK ) finds a particular subspace by embedding original data onto a Grassmannian manifold , matches source and target on the geodesic flow and determines a suitable domain adaptation kernel by applying the kernel trick ( Gong et al. , 2012 ) . The relational-knowledge adaptation aims to find some relationship between source and target data ( Weiss et al. , 2016 ) . Transfer Kernel Learning ( TKL ) ( Long et al. , 2015 ) is a recent approach , which approximates a kernel of training data K ( Z ) by test kernel K ( X ) via the Nyström kernel approximation . However , recent studies conclude that the asymmetric transformation of source data is a promising approach in domain adaptation due to the better classification performance ( Sun et al. , 2016 ; Fernando et al. , 2016 ; Raab & Schleif , 2018 ; Long et al. , 2013 ) . Possibly , because a classifier is trained on the source label information in the target domain ( Sun et al. , 2016 ) . The good performance has led to a variety of unsupervised asymmetric domain adaptation algorithms ( Elhadji-Ille-Gado et al. , 2017 ; Shao et al. , 2014 ; Blitzer et al. , 2011 ; Thopalli et al. , 2019 ) , to name a few . The asymmetric feature adaptation approaches try to transform source domain data in the target ( subspace ) domain to match the source to target subspace and is most related to our work . In comparison to the symmetric feature adaptation approaches , there is no shared subspace , but only the target space ( Weiss et al. , 2016 ) . The Joint Distribution Adaptation ( JDA ) ( Long et al. , 2013 ) solves divergences in marginal distributions similar to TCA , but aligning conditional distributions with pseudo-labeling techniques . The Subspace Alignment ( SA ) ( Fernando et al. , 2013 ) computes a target subspace representation based on the correlation between target and source subspace projectors and hence aligning source and target data . The Correlation Alignment ( CORAL ) ( Sun et al. , 2016 ) technique transfers secondorder statistics of the target domain into whitened source data . Given these methods , there is an overall trend in domain adaptation in the direction of non-linear problem formulations , indirect alignment of matrix differences and learning a suitable subspace for the source domain in the target domain and have similar problem statements to ours . However , in this work , we do not learn a suitable source subspace transformation , but explicitly override the structural information of source domain with the target one . With this , we model the source subspace domain as part of the target subspace . The considered domain adaptation methods have approximately a complexity of O ( n2 ) where n is the most significant number of samples concerning target or source . These algorithms pursue transductive adaptation ( Pan & Yang , 2010 ) , because some unlabeled test data must be available at training time . These transfer-solutions can not be directly used as predictors , but instead , are wrappers for classification algorithms . The respectively used baseline classifier is the Support Vector Machine ( SVM ) . 3 PRELIMINARIES . We introduce the basics of the Nyström kernel approximation in section 3.1 , which is the foundation of the Nyström based Singular Value Decomposition in section 3.2 . The Nyström-SVD is used for creating the subspace transformation of Nyström Basis Transfer in section 4.1 . 3.1 NYSTRÖM APPROXIMATION . The computational complexity of calculating kernels or eigensystems scales with O ( n3 ) where n is the sample size ( Williams & Seeger , 2001 ) . Therefore , low-rank approximations and dimensionality reduction of data matrices are popular methods to get a better computational performance . In this scope , however not limited to it , the Nyström approximation ( Williams & Seeger , 2001 ) is a reliable technique to accelerate eigendecomposition or approximation of general symmetric matrices ( Gisbrecht & Schleif , 2015 ) . It computes an approximated set of eigenvectors and values based on a usually much smaller sample matrix . The landmarks are typically picked at random , but advanced sampling concepts could be used as well ( Talwalkar et al. , 2012 ) . The approximation is exact if the sample size is equal to the rank of the original matrix and the rows of the sample matrix are linear independent ( Gisbrecht & Schleif , 2015 ) . In general , the Nyström approximation technique assumes a symmetric matrix K ∈ Rn×n with a decomposition of the form K = [ A B C D ] , ( 1 ) with A ∈ Rs×s , B ∈ Rs× ( n−s ) , C ∈ R ( n−s ) ×s and D ∈ R ( n−s ) × ( n−s ) . The submatrix A is called the landmark matrix containing s randomly chosen rows and columns from K and has the Eigenvalue Decomposition ( EVD ) A = UΛU−1 . Where eigenvectors are U ∈ Rs×s and eigenvalues are on the diagonal of Λ ∈ Rs×s . The remaining approximated eigenvectors Û of K as the part C or BT , are obtained by the Nyström method with ÛΛ = CU . Combining U and Û the full approximated eigenvectors of K are Ũ = [ U Û ] = [ U CUΛ−1 ] ∈ Rn×s . ( 2 ) The right part of the EVD ( Ũ−1 ) of K can be obtained via Nyström similar to equation 2 by Ṽ = [ U−1 Λ−1U−1B ] . ( 3 ) Combining equation 2 , equation 3 and Λ , the matrix K is approximated by K̃ = ŨΛṼ = [ U CUΛ−1 ] Λ [ U−1 Λ−1U−1B ] . ( 4 ) The Nyström approximation error is given by the Frobenius Norm between ground truth and reconstructed matrices , i. e. Eny = ||K̃−K||F , with bounds proven by Gittens & Mahoney ( 2013 ) . 3.2 GENERAL MATRIX APPROXIMATION . Another application of the Nyström method is the approximation of the Singular Value Decomposition , which generalizes the concept of matrix decomposition with the consequence that respective matrices must not be squared . Let K ∈ Rn×d be a rectangular matrix with the decomposition as in equation 1 . The SVD of the landmark matrix is given by A = LSRT where L are left and R are right singular vectors . S are positive singular values . The left and right singular vectors for the non-symmetric part C and B are obtained via Nyström techniques and are defined as L̂ = CRS−1 and R̂ = BTLS−1 respectively ( Nemtsov et al. , 2016 ) . Applying the same principal as for Nyström-EVD , K is approximated by K̃ = L̃SR̃T = [ L L̂ ] S [ R R̂ ] = [ L CRS−1 ] S [ R S−1LTB ] . ( 5 )

Summary: In this work, the authors improve the work (Raab & Schleif, 2018) by (1) reducing the computational complexity, (2) neglecting the sample size requirement, and (3) achieving a low-rank projection through Nystrom approximation. More specifically, the feature dimensionality is reduced by using only s biggest eigenvalues and eigenvectors, and the sample size is coordinated through Nystrom approximation. Class-wise sampling is used for the source, and uniform sampling is used for the target. Experimental studies on three datasets have been done.