ENSEMBLE OF TRANSFER COMPONENT ANALYSIS FOR DOMAIN ADAPTATION IN
HYPERSPECTRAL REMOTE SENSING IMAGE CLASSIFICATION
Junshi Xia, Naoto Yokoya, Akira Iwasaki
Research Center for Advanced Science and Technology, The University of Tokyo,
153-0041 Tokyo, Japan
ABSTRACTIn this work, we address the problem of unsupervised domain transfer learning via an ensemble strategy in the contextof classification between multiple hyperspectral images. The
objective of domain adaption is to assign the label to an imageof interest (the target image) using the labeled samples in thesource image. The proposed method is based on the rotationbased ensemble and transfer component analysis (TCA). In
this method, the feature space in both source and target image is divided into several disjoint feature subsets. Then, thefeatures induced by the TCA technique in the source domain
are used as the input space to a random forest (RF) classifier.
Finally, the results achieved by each step are fused by a majority vote. We compare the proposed method, ensemble ofTCA (E-TCA), with a regular RF and an RF with the reduced
features by the TCA. Experiments on the real hyperspectral
image acquired over a Japanese mixed forest show remarkable cross-image classification performances.
Index Terms— Domain adaptation (DA), transfer component analysis (TCA), ensemble learning, hyperspectral image classification.
1. INTRODUCTION
The quality of reference samples plays an important role inproviding high precision classification results of remotelysensed images. However, collecting and labeling sufficient
data are very labor-intensive and tedious [1]. An effective
technique is the domain adaptation and transfer learning inwhich we can use the labeled samples from another dataset toenhance the performance. In particular, classifiers trained on
the source image with the labeled datasets are used to predictthe labels of the target image [2].
In general, the data in both source and target domains
share the same task but with different distributions. According to whether labeled information used or not in the target domain, transfer learning techniques are divided into twogroups: unsupervised and supervised [3]. In this work, we focus on unsupervised transfer learning because of the common
settings of real-world applications.

In the last decade, researchers have investigated a variety of approaches to domain adaptation into two categories:
1) modifying the trained classifier and 2) making a featurebased representation. Considering the first category, Bruzzone et al. [4, 5] re-trained a maximum likelihood classifier
or multiple classifier systems by adjusting the parameters
to update the land cover maps. The second category aims
at learning a common feature representation that minimized
the cross-domain difference. Representative work that follows the second strategy is the transfer component analysis(TCA) [6]. Matasci et al. [7] investigated the performance ofTCA and extended the TCA to the semi-supervised version
for cross-domain hyperspectral and high spatial resolution
remote sensing images. Experimental results demonstrate theclear improvements over the TCA and semi-supervised TCA(SSTCA) when compared to other methods.
Ensemble learning combines multiple based learners toshow a significantly better performance that the individual
learner [8]. Benefiting from the superiority, the ensemble
learning methods have attracted increasing attention in the remote sensing community over the last several years [9]. In
order to make a successful ensemble learning method, weshould increase the accuracy of the base learners and the diversity within the ensemble. Such approach was exploited in
the rotation-based ensemble for hyperspectral image classification, obtaining remarkable classification results [10, 11].
Inspired by the merits of rotation-based ensemble andTCA, we proposed a novel ensemble of TCA (E-TCA) for
domain adaptation in hyperspectral remote sensing imageclassification.
The remainder is organized as follows. In Section II, wegive the main description of TCA. The proposed ensembleof TCA (E-TCA) is presented in Section III. The experiments
and the conclusion are given in Section IV and V, respectively.
2. TRANSFER COMPONENT ANALYSISnSLet us denote DS = {XS , YS } = {xSi , ySi }i i be the set ofnTsource training samples and XT == {xTi }i i be the set of
unlabeled target samples. The objective of domain adaptation
classifier is to assign a label yTi to each pixel of XT .

TCA [6] aims at learning a shared subspace between thesource and target domains based on a reproducing kernel
Hilbert space (RKHS) using maximum mean discrepancy(MMD) [12].
TCA finds a projection matrix W by solving the following
eigenvalue decomposition problem
KHKw = λ(KLK + µI)w(1)
KS,S KS,T
where K =, KS,S , KS,T , KT,S , KT,T are
KT,S KT,Tthe kernel matrices (with the elements of Ki,j = exp(−θ||xi −xj ||2 )). H = I − 11> /(nS + nT ). The elements of matrix L
is defined as
2

if xi , xj ∈ XS
1/nS2Li,j = 1/nT
if xi , xj ∈ XT−1/nS nT
otherwise
The projection matrix W can be obtained from the d eigenvectors corresponding to the d largest eigenvalues achievedby (1).
The d transfer components for a new test sample xtest iscomputed by Ktest W, where Ktest is the kernel matrix between the test sample and the (nS + nT ) samples.
3. ENSEMBLE OF TCA
The performance of ensemble learning method relies on two
essential components: the accuracy of base classifier and thediversity within the ensemble [8]. Rotation-based ensemble isa recent effective strategy to construct the ensemble [10, 11].
In this work, we introduce this strategy into the domain
adaption in hyperspectral remote sensing image classification. Thus, we propose the ensemble of TCA (E-TCA),
which combines the advantages of TCA and rotation-based
ensemble together.
The training steps of the proposed method is summarized
as follows (seen in Algorithm 1)• the feature space in both source and target domain is
randomly split into K (K = D/M ) disjoint subspaceand each space contain M features.
• TCA is applied to each subspace to achieve the projection matrix Wk and kernel matrices.
• the new training set of the source domain is formed byconcatenating M extracted features that are obtainedby rotating the source training set using the aforementioned kernel matrices and the projection matrix.

In the prediction phase, the kernel matrices between the target
and the sum of source and target training samples is generated
firstly. Then, the new transformed dataset of target samples isclassified by the ensemble, and the final result is assigned tothe corresponding class by using a majority voting rule.
Algorithm 1 E-TCA
Training phasenS
Input: {XS , YS } = {xSi , ySi }i i : source training samples.
XT : target samples. T : number of classifiers, K: numberof subsets, M : number of features in a subset, L: baseclassifier. The ensemble L = ∅.
Output: The ensemble L
1: for i = 1 : T do
2:
randomly split the features in the source and target domains into K subsets.
3:for j = 1 : K do
4:form the new source training set XSi,j and target
samples XTi,j corresponding to the subset of features5:
using TCA to transform [XSi,j XTi,j ] with the aim ofobtaining the coefficients Ri,j = w1i,j , · · · , wMi,j
6:
calculate the kernel matrices by XSi,j , Ktraini,j =K(XSi,j , [XSi,j XTi,j ])7:
end fornew8:
the=i by: Fih features extracted will be given>
Ktrain>i,1 Ri,1 , · · · , Ktraini,K Ri,K

train a RF classifier Li using {Fnew, Y}i
add the classifier to the current ensemble, L = L ∪ Li .
11: end for
Prediction phase
TInput: The ensemble L = {Li }i . XT : target samples. Rotation matrix: R.
Output: class label YT
1: for i = 1 : T do
2:for j = 1 : K do
3:generate the kernel matrices Ktesti,j=K(XTi,j , [XSi,j XTi,j ])test
4:generate
the test features of target samples, Fi =>>Ktesti,1 Ri,1 , · · · , Ktesti,k Ri,K
5:
end for6:
run the classifier Li using Ftest
as inputi
7: end for
8: calculate the confidence of each sample x in XT
for each class and assign the class label p(y|x) =PT1test) to the class with the largest confii=1 p(yi |FiTdence.
9:10:• an individual random forest (RF) classifier is trained on
the new source training set.

4. DATASETS AND RESULTS• the above process is repeated T times and the ensemble
with T RFs are obtained.

A Japanese mixed forest, namely Tama Forest Science Garden located in the western region of Tokyo, is selected as theAB

35°65' NAB

0139°28' E

0100

Meters

500

MetersFig. 1. (Left) The false color composite image of hyperspectral data. (Right) Enlarged RGB images of (A) Source and (B)target images with the corresponding ground references
87

Table 1. Class name and number of reference samples in thesource and target images.

Broadleaf

Source
875
3327
282
733
380
403
948
1083
987

Target
2120
165
813
286
382
352
256
422
768

study area. The hyperspectral dataset with 72 bands was acquired by the CASI-3 sensor. The ground sampling distanceis 1 m [13]. The left image of Fig. 1 is the false color composite image obtained by the hyperspetral image (R: 654 nm;
G: 552 nm; B: 449 nm). The right image of Fig. 1 is theenlarged RGB images of source (A) and target (B) areas andthe respective distributions of the reference with two majorspecies (nine classes).
For the TCA and E-TCA, θ is set to be the median Euclidean distance among all used samples. The label samples
in both source and target images are used to find the projection achieved by TCA. Random Forest (RF) classifier is used
to train on the labeled samples in the source domain and predict the labels in the target image. For the RF, the number oftrees is set to be 50 and the number of features in a subset is488647.5

OA(%)

NameJapanese cypress
Japanese cedar
Momi fir
American sweetgum
Japanese bigleaf magnolia
Painted maple
Chinese evergreen oak
Japanese blue oak
Konara oak

86.5

OA(%)

Conifer

No.
12
3
4
5
6
78
949

48.5

85.5

4746.5
85
46
84.5
45.5

84

45
1015

202530

Number of features in a subset (M)

(a)

35

1015

20253035

Number of features in a subset (M)(b)

Fig. 2. RF classification performance on the target image byusing the E-TCA with different numbers of features in a subset (M ). (a) Two major species. (b) 9 classes.
Table 2. Classification results with two major species.
NameRFTCA E-TCA
Conifier
84.25 87.12
88.77
Broadleaf 83.86 82.16
88.57
OA
84.08 85.19
86.63
AA84.05 84.87
86.38

set to be the square root of the number of the used features.
For the TCA, the range of extracted components is from 2 to30 and we only report the best results.
Fig. 2 depicts the influence of the number of features inTable 3. Classification results with 9 classes.
No.
RFTCA E-TCA
161.56 73.54
79.25
48.48 90.91
92.12
237.38 11.4412.1436.71 36.36
48.25
45
31.94 39.01
39.79
67.10
6.25
10.23
31.25 28.52
29.69
78
17.54 25.12
24.88
30.60 28.52
30.86
9OA 37.49 44.48
48.08
40.80
AA 30.28 37.74

a subset (M ) on the RF classification performance for two
major species and nine classes, respectively. The optimum
classification accuracy is reached at the small values of M .
Thus, M = 15 is used to present the class individual accuracies, which is shown in Tables 2 and 3. It can be seen that theproposed E-TCA shows better performance than RF and TCAin both global accuracies (OA and AA) and class-specific accuracies (7 out of 9). For two major species, E-TCA achieves
2.55% and 1.44% improvements over RF and TCA, respectively. For nine classes, the improvements are 10.59% and3.60%, respectively.
5. CONCLUSION
In this paper, we propose a novel transfer learning method for
hyperspectral image classification. We cast the transfer learning problem as an ensemble strategy with transfer component
analysis (TCA). Experimental results show that our methodoutperforms RF and TCA approaches. In the future, we will
extend our method to semi-supervised scenarios.

[3] Ming Shao, Dmitry Kit, and Yun Fu, “Generalizedtransfer subspace learning through low-rank constraint,”
International Journal of Computer Vision, vol. 109, no.
1, pp. 74–93, 2014.
[4] L. Bruzzone and D. Fernández-Prieto,“Unsupervised retraining of a maximum likelihood classifier forthe analysis of multitemporal remote sensing images,”
IEEE Trans. Geosci. Remote Sens., vol. 39, no. 2, pp.
456–460, 2001.
[5] L. Bruzzone and R. Cossu, “A multiple-cascadeclassifier system for a robust and partially unsupervised
updating of land-cover maps,” IEEE Trans. Geosci. Remote Sens., vol. 40, no. 9, pp. 1984–1996, 2002.
[6] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via transfer component analysis,” IEEE
Trans. Neural Netw., vol. 22, no. 2, pp. 199–210, Feb
2011.
[7] G. Matasci, M. Volpi, M. Kanevski, L. Bruzzone, andD. Tuia, “Semisupervised transfer component analysis
for domain adaptation in remote sensing image classification,” IEEE Trans. Geosci. Remote Sens., vol. 53, no.
7, July 2015.
[8] L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms, Wiley-Interscience, 2004.
[9] P. Du, J. Xia, W. Zhang, K. Tan, Y. Liu, and S. Liu,“Multiple classifier system for remote sensing image
classification: A review,” Sensors, vol. 12, no. 4, pp.
4764–4792, 2012.
[10] J. Xia, J. Chanussot, P. Du, and X. He, “Rotation-Based
Ensemble Classifiers for High-Dimensional Data,” inFusion in Computer Vision, Bogdan Ionescu, Jenny
Benois-Pineau, Tomas Piatrik, and Georges Quénot,Eds., pp. 135–160. Springer, 2014.

6. ACKNOWLEDGMENT
The authors would like to that KAKENHI Grant Number24360347 and 16H04587, and JSPS KAKENHI Grant Number 16F16053 for supporting this work.
7. REFERENCES[1] J. Xia, J. Chanussot, P. Du, and X. He, “(Semi-) supervised probabilistic principal component analysis for hyperspectral remote sensing image classification,” IEEE
J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 7,
no. 6, pp. 2224–2236, June 2014.
[2] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng., vol. 22, no. 10,pp. 1345–1359, Oct 2010.

[11] J. Xia, J. Chanussot, P. Du, and X. He, “Spectralspatial classification for hyperspectral data using rotation forests with local feature extraction and Markov
random fields,” IEEE Trans. Geosci. Remote Sens., vol.
53, no. 5, pp. 2532–2546, 2015.
[12] K. M. Borgwardt, A Gretton, M. J. Rasch, H Kriegel,
B Scholkopf, and A. J. Smola, “Integrating structured
biological data by kernel maximum mean discrepancy,”
Bioinformatics, vol. 22, no. 14, pp. e49–e57, 2006.
[13] T. Matsuki, N. Yokoya, and A. Iwasaki, “Hyperspectraltree species classification of Japanese complex mixed
forest with the aid of LiDAR data,,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 8, no. 5, pp.
2177–2187, May 2015.