Domain Adaptation via Low-Rank Basis Approximation
Authors: Christoph Raab
Frank-Michael Schleif
Venue:  (2020)
Keywords: Domain Adaptation
Basis Transfer
Transfer Learning
Low Rank Approximation
Nyström Approximation

Abstract: Domain adaptation focuses on the reuse of supervised learning models in a new context. Prominent applications can be found in robotics, image processing or web mining. In these areas, learning scenarios change by nature, but often remain related and motivate the reuse of existing supervised models.
While the majority of symmetric and asymmetric domain adaptation algorithms utilize all available source and target domain data, we show that efficient domain adaptation requires only a substantially smaller subset from both domains. This makes it more suitable for real-world scenarios where target domain data is rare. The presented approach finds a target subspace representation for source and target data to address domain differences by orthogonal basis transfer. By employing a low-rank approximation, the approach remains low in computational time. 
The presented idea is evaluated in typical domain adaptation tasks with standard benchmark data.

Under review as a conference paper at ICLR 2020
DOMAIN ADAPTATION VIA LOW-RANK BASIS AP- PROXIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain adaptation focuses on the reuse of supervised learning models in new context. Prominent applications can be found in robotics, image processing or web mining. these areas, learning scenarios change by nature, but often remain related and motivate the reuse of existing supervised models. While the majority symmetric and asymmetric domain adaptation algorithms utilize all available source and target domain data, we show that efficient domain adaptation requires only a substantially smaller subset from both domains. This makes it more suitable for real-world scenarios where target domain data is rare. The presented approach finds a target subspace representation for source and target data to address domain differences by orthogonal basis transfer. By employing low-rank approximation, the approach remains low in computational time. The presented idea is evaluated in typical domain adaptation tasks with standard benchmark data.
INTRODUCTION
Supervised learning and in particular classification is essential task in machine learning with broad range of applications. The obtained models are used to predict the label of unseen test sam- ples. A basic assumption in supervised learning is that the underlying domain distribution is not changing between training and test samples. If the domain is changing from one task to related but different task, one would like to reuse the available learning model. Domain differences are quite common in real-world scenarios and eventually, lead to substantial performance drops (Weiss et al., 2016).
A domain adaptation example is the classification of web pages. A classifier is trained in the domain of university web pages with word distribution according to universities. the test scenario, the domain has changed to non-university web pages, where the word distribution is related, but may be similar to the training distribution. More formally, let
{z...... Zmn E IRd be source data sampled from the source domain distribution p(z) and let X = {x ( Xa) E Rd be target data from the target domain distribution p(x). Traditional machine learning assumes similar distri- butions, i.c. p(2) p(r), but domain adaptation assumes different distributions, i.e. p(2) # p(x) and appears in the web page example where Z could be data of university websites and X are data of non-university websites.
Multiple domain adaptation techniques have been already proposed, following different strategies and improving the prediction performance of underlying classification algorithms in test scenarios (Weiss et al., 2016; Pan & Yang. 2010). The common domain adaptation approach requires large number of source or target samples, which is indeed a disadvantage of many domain adaptation ap- proaches and is not guaranteed in restricted environments where labeling is expensive (Weiss et al., 2016). Further, learning a matrix for the alignment of domain distributions does not necessarily lead to effective adaptation shown by (Raab& & Schleif, 2018), where domain differences are explicitly solved by overriding the basis of one domain with another and competitive to current state of the art domain adaptation algorithms (Gong al., 2012; Long et al., 2015; 2013; Pan et al., 2011; Fernando et al., 2013; Sun et al., 2016).
The main contribution of this work is to improve the Basis-Transfe (Raab & Schleif, 2018) ap- proach by deriving a more simplified closed-fon solution of the least-squares problem and enhance this by Nyström based dimensionality reduction. The resulting method has a better prediction performance and makes easer to apply. Further, it is the fastest domain adaptation algorithm in terms of computational complexity, while maintaining its excellent performance by using only

---

Under review as a conference paper at ICLR 2020
subset of source and target data. It determines target subspace representation for both domains and transfers target basis information to source data via Nyström Singular Value Decomposition (SVD) with class wise sampling. Further, show that Nyström approximation is well suited for non- square landmark matrices, in contrast to common decomposition (Nemtsov et al., 2016) by applying post-processing.
The rest of the paper is organized as follows: We give an overview of related work in section 2. The underlying mathematical concepts are given in section 3. The proposed approach is discussed in section 4, followed by an experimental part in section 5, addressing the classification performance and computational time. summary with a discussion of open issues is provided in the conclusion at the end of the paper. More detailed derivations, description of experiments, implementation details and additional results can be found in the appendix.
RELATED WORK
In general, homogeneous transfer learning (Weiss et al., 2016) approaches, or domain adaptation (DA), distinguish roughly between the following strategies: Methods implementing the symmer- ric feature adaptation (Weiss et al., 2016) trying find common latent subspace for source and target domain to reduce marginal distribution differences, such that the underlying structure of the data is preserved in the subspace. baseline approach for symmetric feature adaptation is the Transfer Component Analysis (TCA) (Pan et al., 2011). TCA finds a suitable subspace transfor- mation called transfer components via minimizing the Maximum Mean Discrepancy (MMD) in the Reproducing Kernel Hilbert Space (RKHS). The Geodesic Flow Kernel (GFK) finds a particular subspace by embedding original data onto a Grassmannian manifold, matches source and target on the geodesic flow and determines suitable domain adaptation kernel by applying the kernel trick (Gong et al., 2012).
adaptation aims to find some relationship between source and target data (Weiss et al., 2016). Transfer Kernel Learning (TKL) (Long et al., 2015) is a recent approach, which approximates kernel of training data KZ)) by test kernel K ((X) via the Nyström kernel approximation.
The iiaational-kowwlede
However, recent studies conclude that the asymmetric transformation of source data is promising approach in domain adaptation due to the better classification performance (Sun et al., 2016; Fer- nando et al., 2016; Raab & Schleif, 2018; Long et al., 2013). Possibly, because a classifier is trained on the source label information in the target domain (Sun et al., 2016). The good performance has led variety of unsupervised asymmetric domain adaptation algorithms (Elhadji-llee-in) 2017: Shao et al., 2014; Blitzer et al., 2011; Thopalli et al., 2019), to name few. The asymmetric feature adaptation approaches try to transform source domain data in the target (subspace) domain to match the source to target subspace and is most related to our work. In comparison to the symmet- ric feature adaptation approaches, there no shared subspace, but only the target space (Weiss et al., 2016). The Joint Distribution Adaptation (JDA) (Long et al., 2013) solves divergences in marginal distributions similar to TCA. but aligning conditional distributions with pscudo-labeling techniques. The Subspace Alignment (SA) (Fernando et al., 2013) computes target subspace representation based on the correlation between target and source subspace projectors and hence aligning source and target data. The Correlation Alignment (CORAL) (Sun et al., 2016) technique transfers second- order statistics of the target domain into whitened source data. Given these methods, there is an overall trend in domain adaptation in the direction of non-linear problem formulations, indirect aligoment of matrix differences and learning a suitable subspace for the source domain the target domain and have similar problem statements ours. However, in this work, we do not learn a suitable source subspace transformation, but explicitly override the structural information of source domain with the target one. With this, we model the source sub- space domain as part of the target subspace.
al.,
The considered domain adaptation methods have approximately a complexity of O(n²) where n is the most significant number of samples concerning target source. These algorithms pursue transductive adaptation (Pan & Yang, 2010), because some unlabeled test data must be available at training time. These transfer-souutons cannot be directly used predictors, but instead, are wrap- pers for classification algorithms. The respectively used baseline classifier is the Support Vector Machine (SVM).
2

---

Under review as a conference paper at ICLR 2020
3 PRELIMINARIES
We introduce the basics of the Nyström kernel approximation in section 3.1, which is the foundation of the Nyström based Singular Value Decomposition in section 3.2. The Nyström-SVD is used for creating the subspace transformation of Nyström Basis Transfer in section 4.1.
3.1 NYSTRÖM APPROXIMATION
The computational complexity of calculating kernels or eigensystems scales with (nn²) where n is the sample size (Williams & Seeger, 2001). Therefore, low-rank approximations and dimensionality reduction of data matrices are popular methods to get a better computational performance. In this scope, however not limited to it, the Nyström approximation (Williams & Seeger, 2001) is reliable technique to accelerate eigendecomposition or approximation of general symmetric matrices (Gis- brecht Schleif, 2015).
It computes approximated set of eigenvectors and values based on usually much smaller sample matrix. The landmarks are typically picked at random, but advanced sampling concepts could be used as well (Talwalkar et al., 2012). The approximation is exact if the sample size is equal to the rank the original matrix and the rows of the sample matrix are linear independent (Gisbrecht & Schleif, 2015).
In general, the Nyström approximation technique assumes a symmetric matrix K E R with a decomposition of the form
A
B D
K
(1)
with A E R' B R=x(n-s), C E (mn=s)* and D E ((a=s) The submatrix A is called the landmark matrix containing randomly chosen rows and columns from K and has the Eigenvalue Decomposition (EVD) A = UAU Where eigenvectors are U E R*xs and eigenvalues are on the diagonal of A E R The remaining approximated eigenvectors of as the part C or B", are obtained by the Nyström method with UA = CU. Combining U and Ú the full approximated eigenvectors of K are
U
U Ü
R
(2)
CUA
The right part of the EVD (Ú-1) of K can be obtained via Nyström similar to equation 2 by
[UIU-1
A-
(3)
Combining equation 2, equation and A, the matrix K is approximated by
U K ÜAV CUA-1 A A A
The Nyström approximation error is given by the Frobenius Norm between ground truth and recon- structed matrices, i. c. Eny K k| with bounds proven by Gittens & Mahoney (2013).
3.2 GENERAL MATRIX APPROXIMATION
Another application of the Nyström method is the approximation of the Singular Value Decompo- sition, which generalizes the concept of matrix decomposition with the consequence that respective matrices must not be squared.
Let K E Raxd be a rectangular matrix with the decomposition as in equation 1. The SVD of the landmark matrix is given by A LSR" where L are left and R are right singular vectors. S are positive singular values. The left and right singular vectors for the non-symmenn con-symmetric part and are obtained via Nyström techniques and are defined as L = CRS-1 and R B gTLS- respectively (Nemtsov et al., 2016).
Applying the same principal as for Nyström-EVD, K is approximated by
K = LSR
L
S S
R =
CRS
L
[
S
(5)
3

---

Under review as a conference paper at ICLR 2020
3.3 POLAR DECOMPOSITION
The Polar decomposition (Higham, 2005) theorem is used to validate the proposed subspace trans- formations PCA transformations in section 4 and the class-wise sampling strategy section 4.2. is a universal decomposition applicable to an arbitrary matrix and is defined as X = QP. Where RSR" with as singular values, L and R are left and right singular vectors
Q LR" and respectively. If X is square matrix, the decomposition is unique and U is orthogonal and a rotation matrix. P is positive semi-definite and contains the scaling factors of X.
Theorem (Reusing of Eigensystem). Let X Raxd and K XX" with EVD K UAU-! and Singular Value Decomposition (SVD) of K LSR". Taking the square root of K and using the Polar Decomposition with
K = =UAA-L}
(LR' RSR (LSR")* = LS
(6)
then the eigenvectors and square root eigenvalues of K are singular vectors and values of X respec- lively.
3.4 GERSCHGORIN THEOREM
The Gerschgorin theorem (Varga, 2004) provides geometric structure to bound eigenvalues to so- called dises for complex square matrices, but also generalize complex square matrices. The work of Qi (1984) expands the Gerschgorin circles to so-called Gerschgorin type circles for singular values:
Theorem 2 (Gerschgorin Type Bound for Singular Values (Qi, 1984)). Given the matrix X E Raxd with n, d 2 1, the singular values {s:] of X are the range of
= {p. + |rils,
n.
Where P. [aas and the range Ta is defined
= max
Eagle
|2
i= I....,n.
(8)
j=1,j
By this, it is possible to estimate the numerical range of singular values of X and is used in the error bound given in section 4.3.
BASIS TRANSFER
The task of domain adaptation is to align distribution differences. That means that underlying statis- ties will be similar afterwards. The key idea of the approach presented in the following that explicit alignment of data matrices without any knowledge of statistics and distribution of data is possible and successful. It is based on the work of Schölkopf et al. (1998), which proposed that if two kernel matrices are similar, they follow similar distributions. Hence, we make the domain data as similar as possible by overriding the syntactic structure, which will result in similar kernels. This also applies to arbitrary rectangular data matrices of linear kernel, because it can be reconstructed by using theorem 1. Hence, the method is also applicable in the original feature space without requiring positive semi definite kernel.
The recent Basis- Transfer (BT) (Raab & Schleif. 2018) approach already showed great transfer capabilities and performance by aligning E Raxd and Z R.xd with small error in terms of the Frobenius norm. However, this required equal samples sizes = m. In BT, the following optimization problem was considered:
min MZT X||F
M.T
Where M and T are transformation matrices, drawing the source (Z) closer to target (X) data. A solution of equation 9 is found in closed-fome summarized in three steps (Raab & Schleif, 2018): First, normalize data to standard mean and variance. It is assumed that this will align marginal distributions Euclidean space without considering label information (Raab & Schleif, 2018). Sec-
ond, compute SVD of source and target data, i.e. Z LzS₂RZ with Lz Rmxm and
4

---

Under review as a conference paper at ICLR 2020
R2 E Raxd And = LxSRRR with Lx Rx and Rx Rddd Note that we denote x as source/target related matrix respectively, Next, the approach assumes Sz ~ Sx in terms of the Frobenius norm due similar domains providing similar scaling factors by singular values and normalization. Finally, compute a solution for equation 9 by solving the linear equations. One obtains M = Lx LxL2 and T = R Note that L R I. Finally, trans- fer operation is applied such that the source matrix is approximated by using the available target information by
Ž = MZT Lx L L2S R LxS
(10)
With 2 E R as approximated source data, used for training.
In the following, the work (Raab & Schleif, 2018) is hereby substantially improved and we propose a Nyström based version with following improvements: Reduction of computational complexity, neglecting sample size requirements and achieve a low-rank projection via Nyström approximation with class-wise sampling and implicit dimensionality reduction.
Recap equation 9 and consider the slightly changed optimization problem
min M
|MZ -X
(11)
Where a transformation matrix M must be found, which is again obtained in closed-form. Note this is a transfer learning interpretation of the orthogonal procrustes problem (Schönemann, 1966). Because we apply a dimensionality reduction technique, just the left-sided transformation matrix must be determined, derived as follows: Based on the relationship between SVD and EVD, the Principal Component Analysis (PCA) can be rewritten in terms of SVD. Consider the target matrix with SVD:
X (Rx Sx [
R
(12)
Where Rx E Rdxx as eigenvectors and $ E Rex as eigenvalues of X X. By choosing only the biggest eigenvalues and corresponding eigenvectors the dimensionality of X is reduced by
X, XR, = L,S.R,R, = L,S.-
(13)
With L, E Raxe S. E R X, E R is the reduced target matrix. Hence, only a left sided transformation in equation 11 is required. because the right sided transformation is omitted in equation 13. Note, that for equation 12 a linear covariance or kernel is used,
However, this procedure requires a complete data matrix corresponding singular values and scales the worst case with O(n²) (Williams & Seeger, 2001). In the following, we apply Nyström-SVD and show that only subset of the data is required, which simultaneously reduces computational complexity and eliminates the need to examine all singular values.
4.1 NYSTRÖM BASIS TRANSFER
Let Z and X have valid decomposition given as in equation Note for clarity the Nyström notation is used as in section 3.2. For a Nyström-SVD we sample from both matrices values obtaining landmarks matrices A 2 = L2S2R E Rx and Ax LxSx E R*x Based on Nyström-SVe in equation 5, the dimensions reduced as in equation 13 keeping only most relevant data structures
Lx
Sx = [ Lx
Lx C Sx E Rax.
X, = LxSx =
(14)
Hence, is sufficient to only compute a SVD of Ax instead of X with s < m, d and therefore is considerably lower in computational complexity. Analogy, we approximate source data by Z, LyS2 E mx Since again assume Sz S2 due to data normalization and domain similarities, solving the linear equation as a possible solution for equation 11. leads to M LxL Plugging it back we obtain
z,
Lx R
(15)
Where again a basis of target subspace transfers structural information into the source domain. The matrix Z, is used for training and X, is used for testing. We showed equation 13, that equation 14 and equation 15 are valid PCA transformations. By definition of SVD follows L = LxLxx = I and Lx is an orthogonal basis. Therefore, equation 14 and equation 15 are orthogonal transformations. In particular, equation 15 transforms the source data into the target subspace and
5

---

Under review as a conference paper at ICLR 2020
projects it onto the principal components of X. If data matrices X and Z are standardized', the geometric interpretation is rotation of source data w.r.t to angles of the target basis. The relationship between BT and NBT is in the number of samples used by Nyström. If the rank of the data matrices is equal to 8 the approximation is exact and it became a subspace version of BT.
According to (Weiss et al., 2016), it is asymmetric transfer approach. Further, it is transductive (Pan & Yang. 2010), where unlabeled target data is needed at training time. Subsequently this approach is denoted Nyström Basis Transfer (NBT).
But uniform sampling may not be optimal for Nyström, given labeled data in a classification task (Schleif al., 2018). Therefore, we integrate class-wise sampling the following.
4.2 SAMPLING STRATEGY FOR NYSTRÖM
The standard technique to create Nyström landmark matrices is sample uniform find clusters in the data matrix (Talwalkar et al., 2012). In supervised learning, sampling should utilize class- wise sampling to properly include class-depending attributes of matrix into the approximation (Schleif et al., 2018). This is especially necessary for |||||| > 2. However, a decomposition as in equation 1, required for Nyström-SVD is intractable with class-wise sampling, because respective matrices are non-square: Let Z E Rmxd with m # d and landmark indices I {ii,...,.. with least one in > d. then by definition undefined and especially true for n d. Therefore, sample rows class-wise forming A E Read instead of A 2 R Using the insights of theorem 1, we implicitly obtain SVD of square landmark matrix, i.e. ASAS" K% R by computing A LzSyRT Hence, we can proceed without non-square landmark matrix and using the SVD of A analogy in equation 15. Therefore, it is possible to sample from the whole range of source data and by application of the Polar decomposition, the standard decomposition as in equation 1 is not required. The resulting singular values and vectors utilized for successive Nyström approximations.
The sampling from test data X is done uniformly row-wise, because of missing class information and SVD is obtained analogy via A = Lx R² and used in equation 14. However, the possible range of singular values of A is naturally not equal to Az. Utilizing theorem 2 it is easy to show that $% * Sz. It scales approximated matrices Z, (equation 15) different through $y and accurate normalization of matrix cannot be guaranteed. Therefore, we apply a post- processing correction and apply z-normalization against the standard convention after NBT. The singular vectors also have an approximation error. However, both subspace projections are based on the same transformation matrix, hence making identical error and as result, the error should not affect the classification. The process is given in the figure 2 (appendix A.2) and showing the alignment of source to the target domain. Further, shows the mechanics of the approach: The structural similarities stay the same, while the scaling is changed leading to a high similarity after the post-pnn post-processing correction.
4.3 PROPERTIES OF NYSTRÖM BASIS TRANSFER
The computational complexity of Nyström Basis Transfer (NBT) is composed of economy-size
SVD of landmark matrices A and A with complexity O(22²). The matrix inversion of diagonal matrix S -1 in equation 14 can be neglected. Remaining & matrix multiplications are Oks?² contributing to the overall complexity of NBT which is O(s²) with < n. m, d. This makes NBT the fastest domain adaptation solution in terms of computational complexity in comparison to discussed methods in section 2.
The difference between source and target domain after NBT, i.c. approximation error of source by target domain is bounded by
ENET = X, Z.ll= & IV D21l < X Z|
(16)
Where || S V is the bounding of the normalized singular vectors and D(-) the maximum sum of the singular values obtained by the numerical estimation, given in theorem 2, of X, and Z, respectively. The equation 16 shows that NBT has a lower norm compared to BT or original data and proofs that the matrices are aligned during NBT, reducing the distribution differences. The
Experimental data standardized mean zero and variance in the preprocessing.

---

Under review as a conference paper at ICLR 2020
respective proof is given in appendix A.1.
The sut-le extension for unseen target/source samples, c.g. x R., analog to equa- tion 14. Based equation 13 subspace projection via (approximated) right singular vec- tors is also valid. Hence, sample can be projected into the subspace via x, xR Rx XX LLTBx and be evaluated by an arbitrary classifier learned in the subspace. The pseudo code of NBT is shown in Algorithm
Algorithm 1 Nyström Basis Transfer (NBT)
Require: Z as m sized training; as n sized test set; Y as m sized training label vector; s as number of landmarks parameter.
Ensure: New Source Zg; new Target X.;
I: A : Cx matrix dea.tixxddcoommostiioooX..
> According to section 4.2
2: $2 =AV((A)):
D. SVD of landmark matrix of Z
3: Lx.8% X Rx =
SVD of landmark matrix of X
4: Lx Lx Cx "I]"
> According to equation 14
5: X, LxSO X
> According to equation 14
6: Z, S
According to equation 15
7: Z, X, =Z normalization(2.
b Per Matrix; Effect shown in figure
EXPERIMENTS
We follow the experimental design typical for domain adaptation algorithms (Long et al., 2015; Gong al., 2012; Long al., 2013; Pan & Yang, 2010; Pan et al., 2011; Sun et al., 2016; Mahadevan et al., 2019; Fernando et al., 2013). crucial characteristic datasets for domain adaptation is that domains for training and testing and test classes have the same top category or source. The classes itself are subcategories or subsets. The parameters for respective methods* determined for best performance terms of accuracy via grid search. A detailed description of experiments can be found in appendix A.4. In the experiments, o
different but related. This relation exists because the train
the SVM independent of being a baseline underlying classifier for domain adaptation methods uses the RBF-Kernn The tests are conducted on the datasets groups Reuters, Newsgroup and Calteeha A detailed dataset description is given in appendix A.3.
5.1 PERFORMANCE COMPARISON
The experiments are cumulated three dataset group, which Reuters, Newsgroup and Office- Caltech. The results are summarized in table 1. It shows the mean errors of the experiments per dataset group. To determine statistically significant differences, the two-sided Kolmogorov- Smirnoff- with a p-value of 1% is applied. The marks statistical significance compared to NBT. The NBT is compared baseline SVM and standard domain adaptation methods.
The NBT and BT methods have excellent performance and outperform every other algorithm by far and are overall similar. Looking at the datasets individually, the only competitive approach SA, however, in the overall rating, all non-BT methods are statistically worse.
Note that BT and NET are very similar at Office-Caltech, this caused because within NBT the rank of given image data matrices is equal to the number of samples drawn by the Nyström approximation. Hence, the approximation exact and we use the most important r singular vec- tors and values. BT and NBT are overall much better than remaining approaches, caused by fol- lowing possible reasons: Aligning differences of Frobenius norm between domains using target cubsspace)-transfommat
is sufficient as a problem statement and overriding of source basis is successful approach. Further, by Schölkopf et al. (1998) it will also align differences in subsequent kernels. Explicit integration of non- lineariyy or kernel statistic alignments does not necessarily lead to more knowledge transfer. But, second-order statistics but also structural and geometric informa- tion should be transferred.
Source code and datasets hosted at
http://iirrrrrr//ooommm git

---

Under review as a conference paper at ICLR 2020
02
Classification Error
60
23
Classiication
60
16
10
50
Number of Landmarks
Number of Landmarks
(a) Reuters
(b) Office- Office-Caltech
Figure I: Relationship between number of landmarks and mean classification error over all Reuters datasets, shown in left figure and mean over all Office-Caltech datasets, shown in right figure.
TCA
JDA
TKL
GFK
SA
CORAL
BT
Dataset
SVM
NBT
2011
2013
2015
2012
2013
2016
2018
Reuters
32.1
31.8*
33.1*
23.2*
35.0*
7.6
35.7*
3.5
1.9
Newsgroup
17.8*
14.7 *
17.4*
6.7
22.5*
2.3
28.1 *
2.6
2.3
Office-Caltech
55.5 *
51.5
51.4
49.2
55.3
68.7
52.6 *
20.7
20.8
Overall Mean 35.1 *
32.7
34.0
26.4 *
37.6 *
26.2
38.8 *
8.90
8.3
Table 1: Result of experiments as mean error in percent over each dataset group. Bold marks winner. * marks statistical differences with a p-value of 1% against NBT. The standard deviation was omitted, since it is the same for all and on average around 2-3%.
The sensitivity of the number of landmarks on prediction error as the only parameter of NBT is demonstrated in figure 1. It shows comparison of the number of landmarks and the mean classifi- cation error over Reuters and Office-Caltech datasets, At Reuters, it is monotonically decreasing convex function with decreasing error by increasing the number of landmarks. This supports the Nyström error expectation of approximating the real rank has many local minima. We assume this indicates that only subset of features is relevant and corre- late for classification and remaining features are noise. A time comparison is given in the appendix A.5 showing that NBT approach is the overall fastest domain adaptation approach.
matrix. However, the image function
6 CONCLUSION
We proposed a low-rank domain approximation algorithm called Nyström Basis Transfer. It trans- fers source and target data into the target subspace, requiring only subset of domain data from both domains. The dimensionality reduction, paired with smart class-wise sampling showed its reliability and robustness in this study. Validated common domain adaptation tasks and data, it showed an outstanding performance both in absolute and statistical values. Additionally, it is lowest com- putational complexity compared discussed solutions. NBT is extension of earlier versions of Basis Transfer via Nyström Methods and is a low-rank and fast domain adaption solution.
In future work, deep transfer learning, real-world and other domain adaptation datasets should be considered. A more comprehensive discussion of the Nyström approximation error with the pro- posed decomposition should be theoretically evaluated and compared current Nyström techniques.
We encountered numerical issues at experiments, therefore the shown results are taken from (Mahade-
al., 2019) with similar sampling scheme.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
John Blitzer, Dean Foster, and Sham Kakade. Domain adaptation with coupled subspaces. Journal of Machine Learning Research, 15:173-181, 2011.
Wenyuan Dai, Gui-Rong Xue, Qiang Yang, and Yong Yu. Co-clustering based classification for out-of-domain documents. In Proceedings of the 13th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, San Jose, Califormia, USA, August 12-15, 2007, pp- 210, 2007.
Nassara Elhadji-lle-Gado Edith Grall-Maes, and Malika Kharouf. Transfer Learning for Large Scale Data Using Subspace Alignment. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), volume 2018-Januar, pp. 1006-1010. IEEE, dec 2017.
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual do- main adaptation using subspace alignment. Proceedings of the IEEE International Conference on Computer Vision, pp. 2960-2967, 2013.
Wen-sheng Chu Fernando, De Torre and Jeffery F Cohn. Selective Transfer Machine for Per- sonalized Facial Action Unit Detection. IEFE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 39(3):529-545, 2016.
Andrej Gisbrecht and Frank Michael Schleif. Metric and non-metric proximity transformations linear costs. Neurocomputing. 167:643-657, 2015.
Alex Gittens and Michael W. Mahoney. Revisiting the Nystrom Method for Improved Large-Scale Machine Learning. Journal of Machine Learning Research, 17:11711--11765 2013.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemee for unsupervised domain adaptation. In Proceedings of the IEEE Computer Somputtrr-ccietty Conference on Computer Vision and Pattern Recognition, pp. 2066-2073, 2012.
Nicholas J. Higham. Computing the Polar Decomposition with Applications. SIAM Journal on Scientific and Statistical Computing, 7(4): 160-1174, 2005.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. Proceedings of the IEEE International Conference on Computer Vision, pp. 2200-2207, 2013.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Sinno Jialin Pan, and Philip Yu. Adaptation regularization: A general framework for transfer learning. IEEE Transactions Knowledge and Data Engineering, 26(5):
2014.
Mingsheng Long, Jianmin Wang, Jiaguang Sun, and Philip S. Yu. Domain invariant transfer kernel learning. IEEE Transactions on Knowledge and Data Engineering, 1519-155 2015.
Sridhar Mahadevan, Bamdev Mishra, and Shalini Ghosh. Unified Framework for Domain Adap- tation using. In Michele Berlingerio, Francesco Bonchi, Thomas Gärtner, Neil Hurley, and Geor- giana Ifrim (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 1-17, Cham, 2019. Springer International Publishing.
Arik Nemtsov, Amir Averbuch, and Alon Schelar. Matrix compression using the Nyström method. Intelligent Data Analysis, 20(5):997-1019, may 2016.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22 13495193333, 2010.
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210, 2011.
Liqun Qi. Some simple estimates for singular values matrix. Linear Algebra and Its Applica- fions, 56: 109 1984.

---

Under review as a conference paper at ICLR 2020
Christoph Raab and Frank Michael Schleif. Sparse transfer classification for text documents. In Frank Trollmann and Anni- Yasmin Turhan (eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11117 LNAI, pp. 169-181. Springer International Publishing, 2018.
Frank-Michael Schleif, Andrej Gisbrecht, and Peter Tiño. Supervised low rank indefinite kernel approximation using minimum enclosing balls. Neurocomputing, 318:213-226, 2018.
Bernhard Schölkopf, Alexander Smola, and Klaus Robert Müller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation, 10(5): 1299-1319, 1998.
Peter H Schönemann. generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Ming Shao, Dmitry Kit, and Yun Fu. Generalized transfer subspace learning through low-rank constraint. International Journal of Computer Vision, 109(1-2):74-99 2014.
Hyun Je Song and Scong Bae Park. An adapted surrogate kernel for classification under covariate shift. Applied Soft Computing Journal, 69:435 442, 2018.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of Frustratingly Easy Domain Adaptation. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pp. 2056-2066 pp. 2058-2066 2016.
Ameet Talwalkar, Sanjiv Kumar, and Mehryar Mohri Sampling Methods for the Nyström Method. Journal of Machine Learning Research, 13:981-1006, 2012.
Kowshik Thopalli, Rushil Anirudh, Jayaraman J. Thiagarajan, and Pavan Turaga. Multiple Subspace Alignment Improves Domain Adaptation. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3552-3556. IEEE, may 2019.
Richard S. Varga. Gersgorin and His Circles, volume 36 of Springer Series in Computational Mathematics. Springer Berlin Heidelberg, 2004.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang, A survey of transfer learning. Journal of Big Data, 3(1):9, 2016. ISSN 1196-1115.
Christopher Williams and Matthias W. Seeger. Using the Nyström Method to Speed Up Kernel Machines. In T K Leen, G Dietterich, and V Tresp (eds.), NIPS Proceedings, volume 13, pp. 682-688. MIT Press, 2001.
A APPENDIX
A.1 NORM BOUND NYSTRÖM BASIS TRANSFER
This section discusses the approximation error of Nyström Basis Transfer in terms of the Frobenius norm of the matrix difference shown in equation 16. First, we clarify some definitions used in the subsequent theorem. In this work, we define the Frobenius norm of a matrix X E Raxdd as
X
12 By using theorem 2 we can bound the norm of the singular values |||||
of X by the square root of the squared sum of the numerical range given by
D |S|| S
Σ (p: + 1r.1)2
(17)
Finally, assume that the non-zero singular vectors are normalized and have by definition unit length. Therefore, the norm of a singular vector matrix with n singular vectors can be bounded by Vn.
10

---

Under review as a conference paper at ICLR 2020
Theorem 3. Given two rectangular matrices X,Z
R with n.d 1 and
rank(X), ramk(Z) > The norm |Z, -x,1l 111pp lar vectors L E R=xx is bounded by
in the subspace R induced by normalized singu-
ENET = Z2 X S V - D2 < -
F
(18)
Proof. We start by proofing the first part given by ||, - xx II S Vnl D3 - DEll. Using equa- tion 14, equation 15 and equation 16 the respective term can be rewritten in terms of a SVD-PCA using only the biggest & singular values. Further, substituting M LxL2 leads to
LzSz ZZ. F |MZ X = L - Lx
- ||xS2 LxSxll (19)
Where we assume 1 v 8 < n. This can be simplified applying the Cauchy-Schward inequality twice with
| - Lx - | || - S - | (I|S2 ll ||S ll
(20)
The norm of the singular values Sz, Sx E R*x* can be bounded by equation resulting in
|| S D2
Σ (pt + +ri),, ||Sx|| S D&
Eos + +r712
(21)
Where p and rf are the bound of a singular value of Z5 and pt and rf the bound of singular value of X., Further, the singular vectors Lx are normalized and thus can be bounded by ||xl| S vT. Ap- plying both to equation 20 leads to
Sx Lx L |F (||l || x Vnl D
(22)
The second term / Dx - - < Z X F is obtained analogy. Given the SVD of the original matrices we obtain
Iz - X F. = R - Lx Sx RxllF
(23)
By again using the Cauchy-Schward inequality we obtain
IZ x - |z || R IF - |[ [Sxl |
(24)
Because Lz and Lx have same size they are bounded by v and because Rz, Rx E Bad and are also normalized, the norm can be bounded by ||o]| I vd respectively. Because we are not in the subspace R but Rd with < d and the singular values are S2. Sx E Raxd the numerical bound becomes
12
D
Σ (pf + 1r11),
D =
Σ 1=1
(pF + +172.
(25)
Applying the bounds to equation 24 we obtain
|| ||>]| [Rzlr
|x |Sxll [R
(26)
Simplifying this term further leads to
|Z-X||= S VnvalDz> II
(27)
Because we assume and minimum one non-zero singular vector Re,) this at least
vnIDs - DE1l < Z X F - vavalD> - D
(28)
The result from the above allows the conclusion that the norm of the subspace matrices is smaller than the norm of the original matrices. Note that because BT (Raab & Schleif, 2018) is applied in Rd, this allows the conclusion that EN < EBT. If the norm of the difference of two matrices is small, they are similar and if they are numerically similar, we can conclude that the distributions are also similar (Song& & Park, 2018). Finally, this is the reason why NBT aligns the distribution of two given matrices. However, note that similar distributions not necessarily means a good classification performance in terms of accuracy by an arbitrary classifier in transfer learning setting.
11

---

Under review as a conference paper at ICLR 2020
A.2 PROCESS OF NYSTRÖM BASIS TRANSFER
In this section, an illustration of Nyström Basis Transfer (NBT) is given. Figure 2 shows the process of Nyström Basis Transfer. The first row shows the samples of Nyström create the approximated set of subspace projectors. The second row shows the data after the subspace projection. The similarity in structure but dissimilarity in scaling, as discussed in section 4.3 is visible. The last row shows the data after applying post projection and the difference is not visible anymore.
A.3 DATASET DESCRIPTION
The study consists of 24 benchmark datasets, already preprocessed and taken from (Gong et al., 2012), (Long al., 2015) and (Long et al., 2014).
Reuters-21578' (Long et al., 2015): A collection of Reuters news-wire articles collected in 1987. The text is converted to lower case, words are stemmed and stopwords are removed. With the Document Frequency (DF)-Threshold of 3, the numbers of features are cut down. Finally, Term- Frequency Docesss-t Docquenn-Frequeency (TFIDF) is applied for feature generation (Dai et al.,, 2007). The three top categories organization (orgs), places and people are used in our experiment. To create a transfer problem, a classifier not tested with the same categories as it trained on, g- it is trained on some subcategories of organization and people and tested on others. Therefore, six datasets are generated: orgs vs. places, orgs vs. people, people vs. places, places vs. orgs, people vs. places and places vs. people. They are two-class problems with the top categories the positive and negative class and with subcategories as training and testing examples.
20-Newsgroup' (Long et al., 2014): The original collection has approximately 20.000 text docu- ments from 20 Newsgroups and is nearly equally distributed in 20 subcategories. The top four categories comp, rec, talk and sei and containing four subcategories each. We follow data sampling scheme introduced by Long et al. (2015) and generate 216 cross domain datasets based on subcategories: Let C' be top category and {C1, C2, C3, C4) E C' are subcategories and analogy K with {K1, K2, K3, K4) E K. Select two subcategories each, e. C1,, C2, K1 and K2, train a classifier, select another four and test the model on it. The top categories are respective classes. Fol- lowing this, 36 samplings per top
are possible, which are in total 216 dataset samplings. This is summarized as mean over all test runs as comp vs rec, comp vs talk, comp vs sci, rec vy sci, rec vs talk and sci vs talk. This version of 20-Newsgroup has 25,804 TF-IDF features within 16.021 documents (Long et al., 2015).
Caltech- (attech-256-oo 356-Office (Gong et al., 2012): The first, Caltech (C) is an extensive dataset of images and contains 30.607 images within 257 categories. The Office dataset a collection of images drawn from three sources which are from amazon (A), digital SLR camera (DSLR) and webcam (W). They vary regarding camera, light situation and size, but ten similar object classes, e. g. computer or printer, extracted for a classification task. Duplicates more than 15 similar Scale Invariant Feature Transform (SIFT) in common (Gong et al., 2012). The final feature extraction is done with Speeded Up Robust Features Extraction (SURF) and en- coded with 800-bin histograms. Finally, the twelve sets are designed to be trained and tested against each other by the ten labels (Gong et al., 2012).
removed, as well as images which have
Note to reproduce the results of this paper, one should use the linked version of datasets with the same choice of subcategories. A summary of all datasets is shown in table 2. Regardless of the dataset, data have been standardized to standard mean and variance. The experiments are summa- rized as the mean over 20 runs with 2-fold sampling, with separated samplings for source and target data.
A.4 IMPLEMENTATION DETAILS
The parameters for the respective domain adaptation algorithms are obtained via grid-search for best performance on a dataset type. meaning text with Reuters and Newsgroup and image with Office vs Caltech. All algorithms using the SVM with an ReF-Kemel, except GFK which suggest k nearest
Phttp://wee www.ddddiiidiidde wwtp:/www.....eeeeeeeeeeeeee
"https://people werkele....eeey........... edu/ whoo:hoofffooooooom hator
12

---

Under review as a conference paper at ICLR 2020
2.05
0.08
0.06
D.04
0.04
0.04
800
800
2.01
600
400
400
200
200
(a) Target Samples
(b) Source Samples
0.06
0.05
0
0.01
0.06
4.05
0.15
0.04
0.2
0.1
100
100
40.15
(c) Target Subspace after NBT
(d) Source in Subspace after NBT
20
10
0
-0
-20
-15
200
200
100
100
Samples
Dimensions
Samplas
Dimensions
(e) Normalization of Target
(I) Normalization of Source
Figure 2: Process of Nyström Basis Transfer with ten landmark samples applied Caltech Ama- zon image dataset as a surface plot. The left column shows the target and the right column shows source data. The first row shows samples used for creating NBT transformations. The differences are clearly visible. The second row shows the data after transformation. Note the similarities in structure but differences in scale, based on the approximation error of singular values. The last row contains the normalization correction and differences are hardly visible with the bare eye. This is finally used for training and testing. Note that this is a toy example approximation unsuitable for proper classification due to small landmark size, shown in figure 1. Best viewed in color.
13

---

Under review as a conference paper at ICLR 2020
Office-Caltech Dataset
Text Datasets
#Samples
#Features
#Labels
#Samples
#Features
#Labels
Names
Names
Caltech (C)
1123
Comp
4857
Amazon(A) A)
958
Rec
3968
800
10
25804
2
DSLR (D)
295
Sci
3946
Webcam (W)
157
Talk
3250
Orgs 1237
People
1208
4771
Places
1016
Table 2: Overview of dataset characteristics containing numbers of samples, features and labels. Datasets are categorized into two data types. e. text and image. The horizontal line separates the datasets into groups namely Sattech-Officee Newsgroup and Reuters.
SVM
Dataset
TCA
JDA
TKL
GFK
SA
CORAL
BT
NBT
Baseline
Reuters
0.06
0.86
0.36
0.40
3.11
0.87
8.38
0.94
0.50
Newsgroup 1.35
21.39
4.79
2.80
214.40
59.70
705.77
34.49
2.64
Office-Caltech
0.05
0.64
0.45
1.09
0.38
0.28
0.37
0.20
0.09
Overall 0.48
7.51
1.77
1.21
72.54
20.22
238.38
11.87
1.08
Table 3: Result of experiments shown in mean time in seconds per dataset group. All considered methods using baseline classifier, hence the SVM not compared as standalone solution.
neighbor classifier. The C' parameter of the SVM is set to 10 for all experiments. For SVM the LibSVM is used. The TCA also has one parameter which gives the subspace dimensions and is set μ 50 for both datasets. The JDA has two model parameters. First, the number of subspace bases k, which is set to 100. Second, the regularization parameter À is set to 1 for both sets.
The TKL approach has the eigenvalue dumping factor { parameter, which was set 2 for the text datasets and 1.1 for the image datasets. For the GFK solution, the parameter, number of subspace dimensions is set to 20 for image and 40 for text data. The SA subspace dimension parameter is set to 5 for every dataset. The CORAL has no free parameters and CGCA is not evaluated, because of numerical issues. The BT approach has no free parameter and for NBT, the subspace dimension is set to 500 for text and 210 for images.
A.5 TIME COMPARISON
The mean time results seconds of the cross-validation study per data set group are shown in the table 3. Note that SVM is the underlying classifier for the compared approaches and is presented for the baseline and not listed for stand-alone comparison but included into the time measurement of domain adaptation approaches. In the overall comparison, NBT is the fastest compared to the discussed solutions and in contrast to BT, the required time is magnitude smaller, especially at Newsgroup, which a high dimensional dataset. This is an expected effect due Nyström approx- imation. Individually, the NET is the fastest algorithm at the Newsgroup and CaltceCC data set. Reuters, JDA is the fastest. summary, the differences between the winning algorithm and other algorithms are very small. However, GFK and CORAL are the slowest methods with factor of at least 90 compared to the fastest approach for Newsgroup and are clear outliers this time comparison.
A.6 DETAILS OF PREDICTION PERFORMANCE
The detailed version of the results of the experiments are shown in table 4. It is an extended version of table of the main paper.
14

---

Under review as a conference paper at ICLR 2020
TCA
JDA 2013
TKL 2015
GFK
SA
CORAL
Dataset
SVM
BT
NBT
2011
2012
2013
2016
Orgs vs People
23.6
24.0
25.3
19.5
27.8
28.8
1.2
People vs Orgs
21.5
20.9
24.5
13.0
28.7
4.3
30.2
1.9
1.2
Orgs Place
31.9
29.0
29.5
23.5
34.7
6.5
35.6
3.4
1.7
Place vs Orgs
34.8
32.7
34.0
18.3
35.2
7.9
37.6
4.2
2.8
People vs Place
39.6
40.9
40.6
30.9
41.6
7.6
41.3
4.1
2.7
Place vs People
41.2
43.0
44.6
34.1
42.1
12.5
40.6
6.0
2.7
Reuters Mean
32.1*
31.8*
33.1
23.2*
35.0
7.6
35.7*
3.5
1.9
Comp vs Rec
12.7
8.1
7.8
3.0
16.3
1.8
23.0
0.3
0.7
Comp vs Sci
24.5
26.3
27.1
9.5
24.3
4.8
28.2
0.3
0.8
Comp Talk
5.1
2.9
4.2
2.4
22.3
0.9
26.8
4.8
3.9
Rec vs Sci
23.7
17.3
23.9
5.1
24.4
29.5
0.1
Rec vs Talk
18.7
13.6
15.2
5.6
23.2
1.8
29.8
5.1
3,9
Sci vs Talk
21.7
20.1
26.1
14.6
24.6
2.9
31.0
5.0
4.1
Newsgroup Mean
17.8*
14.7
17.4*
6.7
22.5*
2.3
28.1
2.6
2.3
C vs A
50.3
49.0
46.3
48.5
61.3
48.6
50.4
12.3
12.4
C vs W
58.9
59.5
55.1
53.6
63.3
81.3
60.9
20.5
20.5
Css
54.3
55.7
53.4
54.2
59.7
61.8
32.8
32.8
A VS C
58.8
57.7
57.5
57.1
63.8
50.2
58.1
38.5
38.5
A W
66.0
63.5
57.8
58.1
65.8
73.2
60.0
20.5
20.5
D
59.2
61.5
62.6
56.8
64.5
74.2
61.8
34.3
34.3
Dis C
76.8
71.3
68.7
65.5
73.4
78.1
67.4
36.3
36.3
D VS A
72.1
67.8
66.5
64.0
68.4
70.5
64.5
4.7
4.8
vs W
24.4
23.8
18.9
23.6
22.3
45.4
18.9
32.8
32.8
vs
74.4
68.6
68.0
64.4
70.8
68.3
33.4
W v
75.2
65.3
66.4
63.6
69.7
73.7
66.3
6.8
6.8
W D
52.9
31.7
27.7
23.5
33.1
33.2
25.8
19.8
19.8
Office-Caltech Mean
55.5
51.5
51.4
49.2
55.3
68.7
52.6
20.7
20.8
Overall Mcan
35.1 * 32.7
34.0 *
26.4 s
37.6 a:
26.2+
38.8
8.9
8.3
Table 4:
Result of cross-validation test shown mean error in percent per dataset. Mean over dataset group at the end of each section. Bold marks winner. marks statistical differences with a p-value of 1% against NBT. The study shows that none of listed algorithms statistically significant better as NBT.
15


Reviews:
[{"review_id": "SJeQ8PccYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is mainly an improvment of the basis transfer model, by introducing the Nystrom approximation and a NBT model is formulated. This paper lacks of research motivation and solid experimental validation.\nThe authors claim \"it is the fastest domain adaptation algorithm in terms of computational complexity\", which is not very convinced.\nI cannot observe new knowledge in domain adaptation, except the Nystrom approximation technique used.\nThe equation (9) is similar to the representational based transfer model, where the T matrix is just the reconstruction matrix, because X and Z have different number of samples.\nThis paper written is a little poor, and the novelty of NBP is not clearly claimed.\nThe experimental effectiveness is weak and have no improvement compared with BT 2018 in image dataset.\nIn deep learning era, could you discuss what is the value in deep transfer learning or deep domain adapatation?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkgkZ4H6tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors improve the work (Raab & Schleif, 2018) by (1) reducing the computational complexity, (2) neglecting the sample size requirement, and (3) achieving a low-rank projection through Nystrom approximation. More specifically, the feature dimensionality is reduced by using only s biggest eigenvalues and eigenvectors, and the sample size is coordinated through Nystrom approximation. Class-wise sampling is used for the source, and uniform sampling is used for the target. Experimental studies on three datasets have been done.\n\nThis paper should be rejected because (1) the paper lacks important latest references on domain adaptation, (2) the paper misuses the notations that makes the paper is not easy to follow, (3) the algorithm is not well justified either by theory or experiments, and (4) the presentation should be further polished. Here are some detailed comments:\n\n(1)\tA lot of recent deep domain adaptation methods are missing. These deep works achieve the state-of-the-art results on many transfer benchmark tasks. Without comparison with them (the paper does not mention any deep works), it is very unconvincing to conclude the paper makes new contributions to the transfer community. \n(2)\tIn line 2 of page 5, the authors claim that BT assumes S_Z ~ S_X, which is not true. The key idea of BT is to construct new source data using target basis and source eigenvalues. Similarly, the authors make the same claim in section 4.1 for NBT, which is also not valid. \n(3)\tThe notations in eq. (12) and (13) are very misleading. Eq. (12) follows the notations of the first line in page 5, but in the first line below eq. (12), why R_X \\in R^{d \\times s}, S_X^2 \\in R^{s \\times s}? I understand that X_s is the low dimensional X, then X_s = XR_s with R_s is the dimensionality reduction matrix whose size is d \\times s (using biggest s eigenvectors is fine). With this, eq. (13) is incorrect as X should be L_X*S_X*R_X^T, but not L_s*S_s*R_s^T. Moreover, it is also unclear why X is decomposed into product of two matrices in this work, is there any benefit of doing so for transfer purpose? \n(4)\tIn section 4.1, A_Z and A_X have exactly same form with X and Z, i.e., L_Z*S_Z*R_Z^T and L_X*S_X*R_X^T, please clarify. How eq. (14) X_s = \\tide{L_X}*S_X come from? Is it the same as eq. (13)? \n(5)\tThe title highlights low-rank, but it is not very clear how low-rank matters in the proposed method. I do not find contents stating the low-rank property of the proposed algorithm in the main technical sections.  \n(6)\tRegarding section 4.2, what is the benefit of using class-wise sampling for the source? Have you tried to use uniform sampling for both the source and target domains?\n(7)\tThe bound in eq.(16) is not very meaningful as s << n, m, d. Moreover, I am also not convinced by the claim this reduces the distribution differences, please give more theoretical justifications.\n(8)\tRegarding the experimental studies, why not use accuracy as many existing works do? The baselines are all subspace-based papers, and are out-of-the-date. The latest subspace papers, e.g., JGSA and MEDA, should be included. Moreover, deep methods are completely missing, which makes the empirical evaluation much less convincing. The improvements of NBT to BT are very marginal, 0.6. \n(9)\tSome typos and unclear points (please further polish the paper): \n(a)\tThe last sentence of para 2, X and Z should be data, not features/\n(b)\tPara 3, it is unclear what are the implicit alignment and explicit alignment of domain distributions.\n(c)\tThe first sentence in section 2, it should be homogeneous. \n(d)\tPage 5 above eq. (15), it should be S_Z ~ S_X.\n(e)\tSection 4.2 the second line, it should be “in the data matrix”\n(f)\tThe second last line of page 6, it should be inequality (16).\n(g)\tSome references miss page information.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bklsscm6Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.\tThis manuscript is mainly based on the previous work BT (Raab and Schleif, 2018). The novelty seems to be too limited.\n2.\tThe proposed method NBT improves the computional efficiency of BT. The Datasets used in the experiments are not representative. More experiments should be conducted to demonstrate its efficiency on large-scale Datasets, such as VisDA and Offic-Home Dataset.\n3.\tThe precise results of CGCA are not provided, which is unfair. Besides, there is no comparsion between the proposed methods and the state-of-the-art deep learning-based methods.The experimental results seems unconvincing.\n4.\tTypos:\n(1)\t‘Out BT approach has no free parameter…’ in page 14. Here ‘Out’ means ‘Our’?\n(2)\t‘…NBT is fastest at Newsgroup und Image data set. In page 14.’\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers have scored this paper  as 1/1/3 and they have not increased their rating after the rebuttal and the paper revision. The main criticism revolves around the choice of datasets, missing comparisons with the existing methods, complexity and practical demonstration of speed. Other concerns touch upon a loose bound and a weak motivation regarding the low-rank mechanism in connection to DA. On balance, the authors resolved some issues in the revised manuscripts but reviewers remain unconvinced about plenty other aspects, thus this paper cannot be accepted to ICLR2020.
Decision: Reject