Subspace Distribution Alignment for Unsupervised Domain Adaptation

Traditional machine learning algorithms often fail to generalize to new input distributions, causing reduced accuracy. Domain adaptation attempts to compensate for the performance degradation by transferring and adapting source knowledge to target domain. Existing unsupervised methods project domains into a lower-dimensional space and attempt to align the subspace bases, effectively learning a mapping from source to target points or vice versa. However, they fail to take into account the difference of the two distributions in the subspaces, resulting in misalignment even after adaptation. We present a unified view of existing subspace mapping based methods and develop a generalized approach that also aligns the distributions as well as the subspace bases. Background. Domain adaptation, or covariate shift, is a fundamental problem in machine learning, and has attracted a lot of attention in the machine learning and computer vision community. Domain adaptation methods for visual data attempt to learn classifiers on a labeled source domain and transfer it to a target domain. There are two settings for visual domain adaptation: (1) unsupervised domain adaptation where there are no labeled examples available in the target domain; and (2) semisupervised domain adaptation where there are a few labeled examples in the target domain. Most existing algorithms operate in the semi-superised setting. However, in real world applications, unlabeled target data is often much more abundant and labeled examples are very limited, so the question of how to utilize the unlabeled target data is more important for practical visual domain adaptation. Thus, in this paper, we focus on the unsupervised scenario. Most of the existing unsupervised approaches have pursued adaptation by separately projecting the source and target data into a lowerdimensional manifold, and finding a transformation that brings the subspaces closer together. This process is illustrated in Figure 1. Geodesic methods [2, 3] find a path along the subspace manifold, and either project source and target onto points along that path [3], or find a closed-form linear map that projects source points to target [2]. Alternatively, the subspaces can be aligned by computing the linear map that minimizes the Frobenius norm of the difference between them, a method known as Subspace Alignment [1]. Approach. The intuition behind our approach is that although the existing approaches might align the subspaces (the bases of the subspaces), it might not fully align the data distributions in the subspaces as illustrated in Figure 1. We use the firstand second-order statistics, namely the mean and the variance, to describe a distribution in this paper. Since the mean after data preprocessing (i.e. normalization) is zero and is not affected 10 20 30 40 50 60 70 80 90 100 25 30 35 40 45 Subspace Dimension A c c u ra c y Mean Accuracy of k−NN with k=1 NA SA SDA−TS GFK SDA−IS 10 20 30 40 50 60 70 80 90 100 25 30 35 40 45 Subspace Dimension A c c u ra c y Mean Accuracy of k−NN with k=3 NA SA SDA−TS GFK SDA−IS Figure 2: Mean accuracy across all 12 experiment settings (domain shifts) of the k-NN Classifier on the Office-Caltech10 dataset. Both our methods SDA-IS and SDA-TS outperform GFK and SA consistently. Left: k-NN Classifier with k=1; Right: k-NN Classifier with k=3. 10 20 30 40 50 60 70 80 90 100 15 20 25 30 Subspace Dimension A c c u ra c y Mean Accuracy of k−NN with k=1