T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

1

Domain-Sum Feature Transformation For
Multi-Target Domain Adaptation
Takumi Kobayashi1,21National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Japan

2

University of Tsukuba
Tsukuba, Japan

takumi.kobayashi@aist.go.jp

Lincon S. Souza1
lincon.souza@aist.go.jp
2

Kazuhiro Fukui

kfukui@cs.tsukuba.ac.jp

Abstract
Domain adaptation effectively transfers a learner from a source domain to a target domain. Recent deep methods are based on detailed comparison between a pair of source
and target domains, which makes it less applicable to multiple domains. In this paper,
we address the domain adaptation on the basis of subspace which provides more robust
metric. We analyze the subspace methods in domain adaptation to theoretically derivea subspace-based feature transformation in an efficient form of simple summation. It
intrinsically contributes to closing a gap between source and target subspaces in an endto-end deep framework. Besides, due to the robust representation of subspace and thesimple transformation, the proposed method naturally deals with multiple domains bothfor source and target in contrast to previous approaches. Multi-target domain adaptation
especially provides efficient inference to process multiple target domains by only a single model. In the experiments on visual domain adaptation tasks, the proposed method
exhibits favorable performance in a scenario of the multi-target domains.

1

Introduction

It is costly to annotate a huge number of samples required to train deep neural networks(DNNs), which limits the annotation only to samples of a source domain, while test samples
can be drawn from a different target domain. Domain adaptation [26, 33] fills a gap between
the source and target domain so as to effectively transfer the learner.
Deep domain adaptation [26] based on end-to-end trainable DNNs has attracted keen
attention in recent years. It leverages DNN of favorable transferability [1, 29] to transfer
classifiers trained on a source domain into a target domain by means of regularization toalign the source distribution with the target one. The regularization losses are formulated,for example, by adversarial techniques [5, 12, 13, 15, 23] for minimizing the discrepancy
between source and target domains. Statistical moment matching also contributes to regularization through minimizing the discrepancy of mean (MMD) [14], content moment [31] andhigher-order moment [17, 20]. While they globally adapt the source domain to the target,some works have recently addressed local sub-domain adaptation [25, 28, 32]. Class-related© 2023. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.

2

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION
Principal component subspaceenhanced

Domainsum

Psrc , Ptarget1 , Ptarget2 } are uniformly aggregated to
Figure 1: Multiple domain subspaces {Pproduce our transformation, which enhances the principal component subspace consistentacross domains. The principal component subspace is enlarged through end-to-end learning.
local distributions are useful for minutely aligning feature representation across the source
and target domains; diversity over class categories also enhances the adaptation [2].
While those approaches exploit detailed characteristics of feature distributions, we focus
on a subspace representation, a more general structure of features than the distributions;distribution and its finer structures related to class discrimination are contained in a subspace.
Thus, the subspace is suitable for robustly describing a domain due to the following two
reasons. (1) Subspace is an unsupervised representation and less dependent on the details of
distributions, such as moments and local structures; for example, a subspace resorts only toeigenvectors of a matrix ∑i x i x >i , while moment-based statistics are additionally related to theeigenvalues. (2) Subspace is estimated robustly against sampling-related issues particularly
found in mini-batches; the subspace computed from few samples, even drawn from a domain
in a biased way, is contained in the (inherent) subspace of that domain, while the detailed
characteristics of distribution are poorly estimated due to such sampling issues.
The subspace representation attracted attention in shallow adaptation approaches [7, 8,
10, 21, 22]. In [8], the Grassmann manifold is introduced to describe the domain shift
from the subspace viewpoint mathematically; a domain is shifted along a geodesic path on
the manifold from the source-domain subspace to the target subspace. It leads to geodesic
flow kernel (GFK) [7] by theoretically formulating the feature transformation in a closed
form. Those approaches provide transformation of input features by means of subspaces.
They, however, demand complex computation processes less suitable for end-to-end learning. Multiple subspaces per domain are dealt with in [22], though resorting to greedy subspace matching for a shallow domain adaptation in a scenario of single source and single
target.
In this paper, by analyzing the subspace method, we propose an effective feature transformation method to reduce discrepancy between source and target subspaces in a deep
framework. The proposed method is theoretically derived from the geodesic flow kernel(GFK) [7, 8] and is formulated in an efficient form simple enough to be differentiable. Our
feature transformation is capable of bridging a gap between source and target subspaces
intrinsically through end-to-end feature learning in a deep domain adaptation framework.
The method uniformly aggregates domain-specific subspaces, regardless of source or target, as shown in Fig. 1. Therefore, we can naturally address the adaptation for multiple
target domains which is a more generalized scenario than the standard setting of singletarget domain. The multi-target domain adaptation (DA) renders a single versatile model
applicable to classifications on multiple target domains without pre-identifying the target
domain of an input sample nor preparing multiple models tailored for respective domains ina memory-consuming way. Thus, by the multi-target adaptation, we can enjoy a simple andefficient inference. While multi-target DA is addressed in a shallow framework [30], someT. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

3

deep approaches are applied to this task by stacking multiple networks [6] and learning multiple teacher models [16] in less efficient ways. In contrast, our method is formulated ina simple yet effective way through summation of subspaces, thus being applicable even to
multi-source and multi-target adaptation. Besides, by resorting to the subspace, the proposed
method is complementary to the other deep domain adaptation approaches based on regularization of distributions, which are contained in subspaces. It is applicable in conjunction
with the regularization losses for further boosting performance.

2

MethodWe first review the geodesic flow kernel (GFK) [7] which describes domain shifts from a
Grassmannian viewpoint, and then formulate the proposed feature transformation method.

2.1

Subspace-based domain adaptation

Training samples on a source domain S are equipped with class label yi ∈ {1, · · · ,C} andSfeature representation xSi ∈ Rd to build a labeled set of {xxSi , yi }ni=1. Our objective is to construct a classifier discriminating C classes on a target domain T which provides only sample
T
without any annotation. The target domain contrasts with the source one
features {xxTj }nj=1such as in terms of image styles (Fig. 3), thereby making it hard to directly transfer a classifier optimized on the source domain. The domains are characterized by subspaces such thatxSi ≈ U S U S> xSi = PS xSi , xTj ≈ U T U T> xTj = PT xTj , ∀i, j,(1)where U S ∈ Rd×r and U T ∈ Rd×r are orthonormal bases of rank r for the source- and targetdomain subspaces, respectively; they are practically computed by applying singular-value
decomposition (SVD) to X S = [xxS1 , · · · , xSnS ] and X T = [xxT1 , · · · , xTnT ]. From a geometric viewpoint, those subspaces are represented by two points on a Grassmann manifold, which aresmoothly connected via a geodesic path on the manifold [8]. By using the complementary
U S ∈ Rd×d−r s.t. U S>ŪU S = 0 , the geodesic flow [7] is formulated as
subspace basis Ū
U S S̄S Σ (t) , t ∈ [0, 1],
U (t) = U S S Γ (t) − Ū(2)where S ∈ Rr×r and S̄S ∈ Rd−r×r are orthonormal matrices given by the generalized SVD of>U S>U T = S Γ T > , ŪU S U T = −S̄S Σ T > ,(3)which uses the orthonormal matrix T ∈ Rr×r . The diagonal matrix Γ and Σ are composed of {cos θk }rk=1 and {sin θk }rk=1 based on the canonical angles {θk }rk=1 between thesource U S and the target subspace U T . The parametric form (2) is thus given by Γ (t) =U S and U (t=1) = U T .
diag({cos(tθk )}rk=1 ) and Σ (t) =diag({sin(tθk )}rk=1 ) to satisfy U (t=0) =U
The geodesic flow (2) provides spectral features between source and target domains
via projection U (t)U >(t) x . While uniform sampling on the geodesic path produces a fixeddimensional representation [8], geodesic flow kernel [7] is given by the integral on the path;k(xxi , x j ) = x >i Gx j , G =

Z 1
t=0

U S S , ŪU S S̄S ]U (t)U >(t) = [U




Λ1 Λ2
U S S , ŪU S S̄S ]> ,
[U
Λ2 Λ3

(4)

4

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

where the diagonal matrices Λ 1 ∼ Λ 3 are
n

n

n

sin(2θk ) orcos(2θk )−1 orsin(2θk ) or
Λ 1 =diag 1+, Λ 2 =diag, Λ 3 =diag 1−
.
2θk
2θk
2θk
k=1
k=1
k=1
(5)
1
1
The matrix G 2 , square root of G , renders an explicit feature transform of GFK as x̂x = G 2 x .

2.2

Domain-sum transformation based on subspace projection

The above approach leverages a geodesic path (2) to produce effective feature transformation in a Grassmannian manner. It, however, demands rather complicated computational
processes, especially in generalized SVD (3), to construct the geodesic flow, which hinders
us from incorporating the GFK into end-to-end learning. Thus, we propose a simpler formulation exploiting the subspace approach with a theoretical connection to GFK. It also gives
interpretation and another formulation to GFK from a viewpoint of feature transformation.
We roughly discretize the integral (4) to a sum of two boundary points (source and target);H=∑ U (t)U >(t) = U (t=0)U >(t=0) +UU (t=1)U >(t=1) = U U > +UU U > .
S

S

T

T

(6)

t∈{0,1}

In spite of the rough approximation, the domain-sum (DS) matrix H is closely related to the
GFK matrix G as follows; our propositions are proved in the supplementary material.
Proposition 1 GFK matrix G (4) and DS matrix H (6) are similarly eigen-decomposed as




Ψ+
Φ+
U + ,UU −]U + ,UU − ]> , H = [UU + ,UU −]U + ,UU − ]> ,G = [U
[U
[U
(7)
Ψ−
Φ−
where
U S S ±UU T T ), Ψ ± =diag({1±sinc θk }rk=1 ), Φ ± =diag({1±cos θk }rk=1 ), (8)U ± =colnorm(Uusing column-wise normalization operator colnorm to ensure orthonormality and sinc(θ ) =sin(θ )
θ . Difference between G and H is only in the functions of sinc and cos applied to thecanonical angles {θk }rk=1 to construct eigenvalues Ψ and Φ .
As shown in Fig. 2a, the eigenvectors U + associated with the eigenvalues Φ + (and Ψ + )are sum of canonical vectors U S S and U T T , indicating principal component subspaces consistent across the source and target domains, while U − reflects the difference subspaces todiscriminate two subspaces U S and U T [3, 4]. Thus, the projection via U + would extract
more effective features shared by two domains than U − . These analyses clarify that eventhe simple form H in (6) provides similar transformation to G in GFK (4). Namely, theyboth enhance the projection into the principal component subspace by the larger weights(eigenvalues) of 1 + sinc θ in G and 1 + cos θ in H while suppressing projection onto thedifference subspaces via the smaller weights of 1 − sinc θ and 1 − cos θ . These eigenvalues are compared in Fig. 2b depicting that sinc function further emphasizes/suppresses theprincipal/difference subspaces than cos. We can improve H based on cos by enhancing theweights on U +/− as 2Φ+
H = H 2 = (UU S U S> +U
U T U T> )2 = [UU + ,UU −]U + ,UU − ]> ,
Ĥ
[U
(9)
Φ−2

5

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION
Principal

Difference

backbone proj.
network headtransformation

norm-th source domain

subspace
extraction
Source domains

(a)

(b)

regularization
loss

Differencesubspaces-th target domain

backbone proj.

norm

Principal component
subspaces-th sourceclassification
loss

subspace

extraction
network head
Figure 2: Eigen-decomposition of
DS matrix H (6) and GFK matrix G(4). In (a), u ±k , s k and t k are the k-th Target domains
column vector of the matrices U ± , S
Figure 3: Proposed deep multi-domain adaptation.
and T in (3).

of which eigenvalues Φ2± = diag[{(1 ± cos θk )2 }rk=1 ] are depicted in Fig. 2b compared to theH;other weighting. Thus, we propose the feature transformation using Ĥ
H x j = (HH x i )> H x j leads to our domain-sum feature transformation (DSFT)• k(xxi , x j ) = x >i Ĥ
ofU S U S> +U
U T U T> )xxx̂x = H x = (U(10)
1

without applying SVD (3) nor square root of matrix in contrast to G 2 in GFK.
• Weights Φ 2± enhance the principal component subspace U + and suppress the differencesubspace U − more effectively than Φ ± due to (1 + cos θ )2 ≥ 1 + cos θ and (1 − cos θ )2 <1 − cos θ .
• The domain-sum matrix H uniformly aggregates domain subspaces in disregard of domain
type, source or target. So, it is naturally extendable to multiple domains byM

H = ∑ U mU >
m,

(11)

m=1

where M is the number of domains both for source and target, and U m indicates the m-th
domain subspace basis.

2.3

Deep multi-domain adaptation

The simple differentiable formulation (11) inspires us to embed the feature transformation
into an end-to-end framework of deep domain adaptation. We can naturally cope with multiple domains both for source and target, as shown in Fig. 3.
The proposed transformation is applied on top of the backbone feature extractor (Fig. 3).
The features produced by the backbone are normalized into unit L2 -norm for better feature
representation learning [9, 11]. During training, we apply computationally stable SVD [27]U m }Mto mini-batch samples for extracting domain subspaces {U
m=1 , which are aggregated into
the domain-sum matrix H in (11); the details about computing subspaces is shown in supplementary material. Input feature vector xS of source domain is transformed by x̂xS = H xS
which is then fed into a classifier module to finally produce the classification loss of softmax
cross-entropy using class labels of the source samples. Domain subspaces are close to each

6

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

other through the end-to-end learning as shown below, and thus at inference a target-domain
sample can be simply passed to the classifier without transformation.
The domain-sum transformation (10) endows the feature representation with high similarity across source and target domains as described in Sec. 2.2. Besides, the following
proposition ensures that it contributes to closing a domain gap through end-to-end optimizing feature representation x .
Proposition 2 Suppose backbone DNN can flexibly produce a feature vector x which is norwc }Cc=1 , it can produce {xxi }ni=1malized as kxxk2 = 1; for a well separable classifier W = {w>>
PS + PT )xxi ≥ w c (PPS + PT )xxi ∀c, where PS is a r-rank subspace projection
such that w yi (Pmatrix for x i , i.e., x i = PS x i , and PT is an arbitrary subspace of rank r. We partition a feaPS )|kxxk2 = 1, ∠(P
PS , PT ) = θ } where ∠ is an operator
ture space into S(θθ ; PT ) = {xx ∈ span(Pto measure canonical angles. We define a softmax loss `CE optimized w.r.t x ∈ S(θθ ; PT ) and
W , given θ and PT asn`CE (θθ ; PT ) =min nPT )}i=1 ,WW{xxi ∈S(θθ ;P− ∑ logi=1w>
PS + PT )xxi )
exp(w
yi (P
.
>
PS + PT )xxi )exp(∑c w c (P(12)

Then, we have the following relationship between the loss and the canonical angle θ ;θ ∗ ≤ θ ⇒ `CE (θθ ∗ ; PT ) ≤ `CE (θθ ; PT ).

(13)

This proposition shows that, if we learn the model with respect to θ of subspace angles, thesoftmax loss `CE is reduced by closing the angular gap. In particular, at the global minimumθ ∗ = arg minθ `CE (θθ ; P T ), we have θ ∗ = 0 ; if ∃k, θk∗ > 0, we can further reduce the lossby θ̂k∗ < θk∗ , which contradicts the global optimality of θ ∗ and induces θ ∗ = 0 . Therefore,
the proposition indicates that the transformation via H = PS + PT intrinsically works formatching subspaces across domains via simply minimizing a softmax loss in the end-to-end
learning. It is applicable to multiple source and target domains. Thus, the domain-sumtransformation is well compatible with our deep multiple domain adaptation in Fig. 3 .
We can further improve the proposed method in the following two points.
M
P m = U mU >Regularization. As shown in Proposition 2, subspaces {Pm }m=1 get close toeach other as end-to-end training proceeds. To further enhance the subspace matching, we
propose the following regularization loss;`reg = −

r
H k2F
kH
1
1
(i, j)=−
cos2 θk .
∑∑rM(M−1) M − 1 rM(M − 1) i6= j k=1(14)

By minimizing the regularization loss `reg together with the classification loss `CE via `CE +η `reg , the subspaces are well aligned to minimize the canonical angles {θθ (i, j) }Mi, j ; we set thebalancing weight to η = 1.
Classifier subspace. Classification also considers matching between classifier vectors W
and a feature vector x . Thus, we can facilitate learning by increasing similarities not only
among domains but also between domains and classifier W . For that purpose, we can add>
W ) isthe classifier subspace U W U W
into the domain-sum matrix H (11) where U W = svd(Wa classifier subspace basis; thereby, the domain-sum matrix is slightly modified toM>
H = UWUW+ ∑ U mU >
m.
m=1

(15)

7

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

Table 1: Performance results (accuracy, %) of deep domain adaptation on Office-31 dataset
in a scenario of single source and single target; we report averaged accuracy across various
one-to-one adaptation. Detailed performances are shown in the supplementary material.

Raw
Auto-Corr. (16)

I
1
A2
1
2
T

− 21

C CS
CORAL [21]Principal component U +U >+1
GFK (7)G2
1Sum-of-subspaces (6) H 2
Ours (9)
H

2.4

Avg.

(b) Ablation study for extension methods (Sec. 2.3).
Subspace similarity

Trans. (10) Reg. (14) Cls. Sub. Avg.

81.94
21.27
74.20
80.81
84.58
84.73
85.24

X
X
X
X

X
X
X(η = 2)
X

X

81.94
84.45
85.24
85.56
84.59
86.47

Subspace Similarity

Method(a) Comparison
Trans.
matrixRaw
Regularization
Transform
Transform + Regularization
Transform + Regularization+ Classifier Subspace
0

10

20

30

Training epoch

4050Discussion

In [10], GFK is simplified in a completely different way from ours by replacing the GFKmatrix G (4) with an auto-correlation matrix written as
U T ΛT2 U T> ,A = X S X S> + X T X T> = U S ΛS2 U S> +U(16)

where ΛS /T is singular value of X S /T . In contrast to the domain-sum matrix H (6), thematrix A contains statistics of the source and target distribution via ΛS and ΛT , thereby
highly biasing the feature transform to some principal directions. Besides, the dependency
on statistics degrades robustness of subspaces discussed in Sec. 1.
The second-order statistics is also employed in the method of CORAL [20, 21]. For shal1−1

low adaptation, CORAL [21] provides transformation of source features as CT2 CS 2 xS using
covariance matrices CS /T . As in the above approach (16), it is based on the second-order
statistics, which are estimated less robustly than subspaces. CORAL is extended to deep
adaptation by reformulating the concept of covariance matching into a regularization loss
CS −C
CT k2F without feature transformation. In contrast, our transformation works cooperakC
tively with the regularization (14) in a deep framework.
It is noteworthy that Proposition 1 reformulates GFK matrix G (4) into a simpler form
based on eigen-decomposition of H in (7). The eigen-decomposition also provides us with aprojection U +U >+ onto the principal component subspace, which is the special case of H bymodifying cos θ → 1 on eigenvalues in (8). It completely ignores the difference subspaces
U − , while our H exploits all of them with proper weighting based on canonical angles{θk }rk=1 ; those transformations are empirically compared in the comparison experiment ofTable 1a.

3

Experimental results

We apply the proposed domain-sum feature transformation to visual domain adaptation tasks
in which images are drawn from multiple modalities (Fig. 3). The detailed experimental
settings and results are shown in the supplementary material.
Dataset. Methods are tested on Office-31 [19], Office-home [24], Adaptiope [18] and DomainNet [17] datasets, which poses image classification across multiple domains.

8

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

Table 2: Performance comparison (accuracy, %) in multi-target deep domain adaptation;e.g., “A→D,W” indicates transfer from source of Amazon to targets of DSLR and Webcam.
Office-31 [19]Office-Home [24]
A→
C,P,R

Avg.

C→
A,P,R

P→
A,C,R

R→
A,C,P

Avg.

A,C→
P,R

P,R→
A,C

A,P→
C,R

C,R→
A,P

A,R→
C,P

C,P→
A,R

Avg.

Domain Concat.

W→
A,D

raw
DANN
BNM
SCDA
DSAN
Ours-c80.96 82.00 82.87 81.94
81.59 80.03 82.67 81.43
86.62 78.34 80.70 81.89
86.09 76.38 81.37 81.28
88.17 80.94 81.75 83.62
88.38 84.89 85.67 86.31

56.44 58.25 58.37 65.62 59.67
47.04 50.39 54.99 63.60 54.00
39.70 48.45 52.46 63.38 51.00
44.72 49.67 52.58 62.80 52.44
45.88 49.19 53.12 62.84 52.76
67.50 67.04 63.22 68.62 66.59

62.59 57.14 64.22 67.12 59.28 61.64 62.00
60.08 59.58 64.63 65.22 57.54 58.60 60.94
58.11 59.39 61.63 64.83 53.90 61.16 59.84
59.92 57.85 63.46 65.53 57.60 60.61 60.83
60.08 58.47 64.00 65.00 57.76 60.75 61.01
77.72 63.55 70.80 75.94 71.12 73.47 72.10

Multi.

D→
A,W

DANN
BNM
SCDA
DSAN
Ours

85.55 80.22 83.95 83.24
86.06 75.49 82.00 81.18
86.81 76.18 82.05 81.68
85.83 76.97 81.73 81.51
88.37 85.03 86.28 86.56

47.30 51.87 53.80 63.83 54.20
41.59 48.60 51.73 63.25 51.29
46.29 51.02 53.19 64.47 53.74
37.43 46.03 50.66 62.58 49.17
67.83 67.65 64.00 69.73 67.30

50.84 57.27 53.21 67.14 62.73 53.92 57.52
50.51 58.28 53.67 69.59 62.41 56.03 58.41
51.19 57.83 54.87 69.65 62.70 57.01 58.87
39.16 56.00 51.74 68.86 61.26 53.20 55.04
76.94 64.94 71.11 75.97 71.08 73.40 72.24

Joint method

A→
D,W+DANN 89.14 85.25 86.17 86.86+BNM 92.81 86.33 87.38 88.84+SCDA 89.95 85.19 86.01 87.05+DSAN 90.95 85.41 86.55 87.64

66.82 68.90 65.49 71.61 68.21
69.17 70.93 66.74 71.18 69.51
67.90 67.93 64.32 69.84 67.50
68.23 68.66 65.01 70.01 67.98

77.39 66.59 73.20 77.61 71.34 72.04 73.03
78.91 66.90 72.37 77.48 72.51 75.33 73.92
77.13 65.28 71.25 76.28 71.25 73.72 72.49
78.09 65.28 71.15 76.40 71.45 73.61 72.66

Adaptiope [18]Avg.

DomainNet [17]C,I,P→ Q,R,S→
Q,R,S
C,I,P

Avg.

C,I→
P,Q→
P,Q,R,S C,I,R,S

R,S→
C,I,P,Q

Avg.

Domain Concat.

S→
P,R

raw
54.30 55.55 43.28 51.05
DANN [5] 58.75 56.13 45.02 53.30
BNM [2] 54.50 55.07 45.72 51.76
SCDA [13] 52.31 54.75 42.04 49.70
DSAN [32] 53.65 55.13 43.74 50.84
Ours-c57.77 61.39 53.14 57.43

8.43 11.97 10.20
7.47 11.99 9.73
8.14 12.15 10.15
8.30 11.83 10.07
7.04 11.21 9.13
28.92 33.13 31.02

6.40 9.22 10.76 8.79
5.35 7.81 10.57 7.91
6.34 9.17 10.70 8.74
6.32 9.13 10.69 8.71
4.64 8.45 10.50 7.86
26.22 27.80 30.00 28.00

Multi.

R→
P,S

Ours

60.09 64.59 57.68 60.79

42.23 44.30 43.26

39.10 38.99 36.67 38.25

Joint method

P→
R,S

+DANN
+BNM
+SCDA
+DSAN

64.40 71.18 62.12 65.90
62.24 65.89 59.63 62.59
60.54 64.61 57.21 60.78
61.38 65.23 58.42 61.68

41.02 43.74 42.38
42.18 44.14 43.16
42.24 44.23 43.24
42.23 44.25 43.24

38.21 38.73 36.09 37.67
39.52 39.18 36.76 38.49
39.47 39.22 36.73 38.47
39.10 39.01 36.72 38.28

Evaluation. We evaluate methods in an unsupervised adaptation framework. A classifierand backbone CNN are trained on source-domain samples equipped with annotation labels
and then the optimized classifier is transferred to classify samples on target domains; during training, we are incapable of accessing labels of target-domain samples. Classification
accuracies (%) are measured on the test-domain samples.

3.1

Performance analysis

We analyze the proposed method on Office-31 dataset in a simple scenario of single-source
single-target domain adaptation for the ease of comparison. We apply the method in theframework of end-to-end learning as shown in Fig. 3. A backbone ResNet-50 is combinedwith projection head to produce 256-dimensional features which are followed by L2 normalization; the ResNet-50 is pre-trained on ImageNet and is fine-tuned in the end-to-end
learning for domain adaptation. We use batch sizes of 32 with the subspace rank of r = 32.
• Transformation method. Our method of x̂x = H x (10) is compared to the other typesof transformation methods discussed in Sec. 2.4. While the methods of CORAL [21] andT. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

9auto-correlation [10] are straightforwardly applicable, we leverage Proposition 1 to embed
GFK [7] in this deep architecture. The performance comparison in Table 1a shows thatsubspace-based transformation is superior to those containing second-order statistics of feature distribution, and the proposed method outperforms the others including GFK. In thisdeep adaptation, CORAL [21] significantly degrades performance as eigenvalues embedded in the transformation matrix would be less stably estimated in a mini-batch. Thus,
DeepCORAL [20] get rid of the CORAL transformation and reformulate it as regularizationof matching covariances to improve performance; in this case, the DeepCORAL produces
81.85% (Avg). On the other hand, subspaces are robustly estimated on a mini-batch to produce favorable performance. The proposed method requires less computation cost than the11others which additionally apply eigen-decomposition (7) to compute U + , G 2 and H 2 .
• Extension method. We then analyze the extension methods described in Sec. 2.3. Table 1breports performance results in an ablation manner. The raw approach excludes transformation and any extension methods, exploiting only source samples for training. In the otherapproaches, target-domain samples participate in end-to-end learning through the subspace
PT = U T U T> which is embedded in the DS matrix H (6) for transformation (10) and/or
regularization (14). Table 1b demonstrates that both the transformation and the regularization contribute to matching subspaces for improving performance. The subspace similarities
based on canonical angles for those approaches are also shown in Table 1b. Interestingly, theraw approach align two subspaces to some extent even by processing source samples only.
Through the end-to-end training, feature representation is biased toward the object categories
which are shared among source and target domains, thereby increasing the similarity of thosesubspaces. In accordance with Proposition 2, our domain-sum feature transformation improves the subspace matching; it produces higher similarity than the raw approach. While
the regularization (14) enhances subspace matching in both the raw approach and our transformation approach, it produces better classification performance in our framework; 84.45%(raw+reg.) vs 85.56% (ours+reg.). Domain-sum transformation by H (6) naturally lets thetarget subspace PT join in a classification loss and thus facilitates to learning discriminative
feature representation. Combination of the transformation and the regularization further improves performance as well as provides favorable matching of subspaces. The performance,
however, is degraded by the larger regularization weight η = 2 than the standard setting ofη = 1. Considering that our feature transformation intrinsically induces subspace matching
as shown in Proposition 2, it is enough to set small weight of η = 1 to slightly inject theregularization effect into training. By adding the classifier subspace into the transformation
matrix H , performance is further improved. It also augments subspace matching as shownin Table 1b. Matching feature subspaces with a classifier subspace facilitates learning.

3.2

Multi-target deep domain adaption

The method is then compared to the other approaches on tasks of multi-target deep domain
adaptation. It is noteworthy that the multi-target adaptation provides efficient inference inwhich the single model of classifier/feature representation learned on the source domain(s)is applicable across multiple domains without identifying domain of an input sample nor
switching the model according to the target domain. We apply four types of regularization
methods based on adversarial technique (DANN) [5], diversity of classifiers (BNM) [2], adversarial alignment of predictions (SCDA) [13] and local domain adaptation (DSAN) [32].
Our method is equipped with the extension techniques (Sec. 2.3, Table 1b). For fair com-

10

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

parison, we apply the same training protocol to the same backbone and projection headproducing features (Fig. 3) which are subject to regularization and/or transformation for enhancing domain adaptation. Table 2 shows performance comparison where each column
shows performance averaged over the multiple target domains.
As those methods other than ours are not designed to cope with multiple target domains,
we concatenate multiple source/target domains in terms of mini-batch to mimic single-source
single-target adaptation; the approach is denoted by “Domain Concat” in Table 2. For comparison, the brute-force approach to consider all pairs of source and target domains is applied,
as denoted by “Multi” in Table 2, but there is less clear performance difference between those
two approaches. While the proposed method (“Ours-c” in Table 2) shows superior performance to the others even in the concatenation scheme, our method works more effectively
in the multi-domain approach. It exploits characteristics of multiple domains by means ofrespective domain subspaces and effectively aggregates them in the DS matrix (11).
Our method based on subspaces is compatible to the distribution-related regularization
losses of the other adaptation methods; as discussed in Sec. 1, the subspace matching facilitates to align distributions which are contained in the subspaces. In our framework (Fig. 3),
the regularization loss can be applied to raw feature representation x in a manner of domain
concatenation for computational efficiency. The performance is further improved by the joint
methods with ours as shown in Table 2.

4

Conclusion

We have proposed a simple yet effective feature transformation based on subspaces for reducing the discrepancy among diverse domains. The method is theoretically derived from
GFK [7] and is formulated in an efficient form of domain-sum matrix H . The feature transformation contributes to closing a gap between source and target subspaces in the frameworkof end-to-end learning. The proposed method is so simple as to naturally deal with multiple
domains via aggregating domain subspaces in disregard of domain types, source or target,
therefore addressing a general domain adaptation for multiple target domains. The experimental results on visual domain adaptation tasks demonstrate that the proposed methodfavorably improves performance in a deep multi-target framework.

References[1] Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, and Atsuto Maki. Factorsof transferability for a generic convnet representation. IEEE Transactions on PatternAnalysis and Machine Intelligence, 38(9):1790–1802, 2016.
[2] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian.
Towards discriminability and diversity: Batch nuclear-norm maximization under label
insufficient situations. In CVPR, pages 3941–3950, 2020.
[3] Kazuhiro Fukui and Atsuto Maki. Difference subspace and its generalization for
subspace-based methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(11):2164–2177, 2015.

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION

11[4] Kazuhiro Fukui, Naoya Sogi, Takumi Kobayashi, Jing-Hao Xue, and Atsuto Maki.
Discriminant feature extraction by generalized difference subspace. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 45(2):1618–1635, 2023.
[5] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle,
François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(1):2096–2130,
2016.
[6] Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, andVladimir Pavlovic. Multiple subspace alignment improves domain adaptation. IEEE
Transactions on Image Processing, 29:3993–4002, 2020.
[7] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for
unsupervised domain adaptation. In CVPR, pages 2066–2073, 2012.
[8] Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object
recognition: An unsupervised approach. In ICCV, pages 999–1006, 2011.
[9] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H.
Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel
Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, andMichal Valko. Bootstrap your own latent: A new approach to self-supervised learning.
In NeurIPS, pages 21271–21284, 2020.
[10] Xifeng Guo, Wei Chen, and Jianping Yin. A simple approach for unsupervised domain
adaptation. In ICPR, pages 1566–1570, 2016.
[11] Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: The marginal valueof training the last weight layer. In ICLR, pages 5822–5830, 2018.
[12] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko,
Alexei A. Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain
adaptation. In ICML, pages 1989–1998, 2018.
[13] Shuang Li, Mixue Xie, Fangrui Lv, Chi Harold Liu, Jian Liang, Chen Qin, and Wei Li.
Semantic concentration for domain adaptation. In ICCV, pages 9102–9111, 2021.
[14] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In ICML, pages 97–105, 2015.
[15] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional
adversarial domain adaptation. In NeurIPS, pages 1647–1657, 2018.
[16] Le Thanh Nguyen-Meidine, Madhu Kiran, Jose Dolz, Eric Granger, Atif Bela, andLouis-Antoine Blais-Morin. Unsupervised multi-target domain adaptation throughknowledge distillation. arXiv, 2007.07077, 2020.
[17] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.
Moment matching for multi-source domain adaptation. In ICCV, pages 1406–1415,
2019.

12

T. KOBAYASHI, L. SOUZA, K. FUKUI: DOMAIN-SUM FEATURE TRANSFORMATION[18] Tobias Ringwald and Rainer Stiefelhagen. Adaptiope: A modern benchmark for unsupervised domain adaptation. In WACV, pages 101–110, 2021.
[19] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual categorymodels to new domains. In ECCV, pages 213–226, 2010.
[20] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain
adaptation. In ECCV Workshop, pages 443–450, 2016.
[21] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, pages 2058–2065, 2016.
[22] Kowshik Thopalli, Rushil Anirudh, Jayaraman J. Thiagarajan, and Pavan Turaga. Multiple subspace alignment improves domain adaptation. In ICASSP, pages 3552–3556,
2019.
[23] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pages 2962–2971, 2017.
[24] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR,
pages 5018–5027, 2017.
[25] Jindong Wang, Yiqiang Chen, Han Yu, Meiyu Huang, and Qiang Yang. Easy transfer
learning by exploiting intra-domain structures. In ICME, pages 1210–1215, 2019.
[26] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
[27] Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Robust differentiable svd. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5472–5487, 2021.
[28] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chena. Learning semantic representations for unsupervised domain adaptation. In ICML, pages 5423–5432, 2018.
[29] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable arefeatures in deep neural networks? In NeurIPS, pages 3320–3328, 2014.
[30] Huanhuan Yu, Menglei Hu, and Songcan Chen. Multi-target unsupervised domain
adaptation without exactly shared categories. arXiv, 1809.00852, 2018.
[31] Werner Zellinger, Edwin Lughofer, Susanne Saminger-Platz, Thomas Grubinger, andThomas Natschlager. Central moment discrepancy (cmd) for domain-invariant representation learning. In ICLR, 2017.
[32] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian,
Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1713–1722,
2021.
[33] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu,
Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedingsof the IEEE, 109(1):43–76, 2021.