© 2017 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or lists, orreuse of any copyrighted component of this work in other works.

Title: Two-stream Deep Architecture for Hyperspectral Image Classification
This paper appears in: IEEE Transactions on Geoscience and Remote Sensing
Date of Publication: 2017
Author(s): Siyuan Hao; Wei Wang; Yuanxin Ye; Tingyuan Nie; Lorenzo Bruzzone
Volume:Page(s):
DOI: 10.1109/TGRS.2017.2778343

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

1

Two-stream Deep Architecture forHyperspectral Image Classification
Siyuan Hao, Wei Wang, Yuanxin Ye, Member, IEEE, Tingyuan Nie, Lorenzo Bruzzone, Fellow, IEEE

Abstract—Most traditional approaches classify hyperspectral
image pixels only relying on the spectral values of the input
channels. However, the spatial context around a pixel is also
very important and can enhance the classification performance.
In order to effectively exploit and fuse both the spatial context
and spectral structure, we propose a novel two-stream deeparchitecture for hyperspectral image classification. The proposed
method consists of a two-stream architecture and a novel fusion
scheme. In the two-stream architecture, one stream employs thestacked denoising auto-encoder (SdAE) to encode the spectral
values of each input pixel, and the other stream takes as input
the corresponding image patch and deep convolutional neuralnetworks (CNNs) are employed to process the image patch. In thefusion scheme, the prediction probabilities from two streams are
fused by adaptive class-specific weights, which can be obtainedby a fully-connected layer. Finally, a weight regularizer is added
to the loss function to alleviate the overfitting of the class-specificfusion weights. Experimental results on real hyperspectral images
demonstrate that the proposed two-stream deep architecture canachieve competitive performance compared with the state-of-art
methods.
Index Terms—Hyperspectral image classification, Two-streamarchitecture, Convolutional neural networks, Stacked denoisingauto-encoder, Class-specific fusion, Deep learning, Remote sensing.

I. I NTRODUCTIONCOMPARED with natural color images, hyperspectral
images (HSI) contain more channels, and can provide
not only more detailed spectral information but also coarsegrained spatial context information. When the spatial context
is involved in the classification task, the accuracy can be
improved significantly. Therefore, a growing number of scientists have taken into account the spatial context information
for hyperspectral image classification [1]. The realted works
mainly focus on two directions: 1) extraction of more robust
and invariant features, such as hand-crafted features and deep
features [2], [3]. 2) fusion of the spectral and spatial featuresfrom the complex HSI for classification.
In spatial-spectral classification, the spatial feature extraction is an important problem to be addressed. The strategies
Manuscript received Nov.19, 2016; revised Dec. 26, 2016. (Corresponding
author: Yuanxin Ye.)
Siyuan Hao and Tingyuan Nie were with the College of Communication andElectronic Engineering, Qingdao University of Technology, Qingdao 266520,
China. E-mail: lemonbananan@163.com, tynie@qut.edu.cn.
Lorenzo Bruzzone and Wei Wang were with the Department of Information
Engineering and Computer Science, University of Trento, Italy. E-mail:lorenzo.bruzzone@ing.unitn.it, wei.wang@unitn.it
Yuanxin Ye was with the Faculty of Geosciences and Environmental
Engineering, Southwest Jiaotong University, Chengdu 610031, China. E-mail:yeyuanxin@home.swjtu.edu.cn

for extracting spatial features can be generally divided into thefollowing groups [4], [5]:Neighborhood window: Spatial features of each pixel
can be calculated by averaging all the pixels in theneighborhood window.
• Segmentation: The parcels from a segmentation map
can be considerd as the homogeneous neighborhood, andcan be employed for the extraction of spatial context
information. For instance, Tarabalka et al. performed
segmentation to define the spatial structures [6].
• Markov random field (MRF): The spatial term in theenergy function of MRF is used for feature extraction,and MRF can well characterize the relationship between
local and global properties of an image [7].
• Morphological and texture features: Morphological
profiles (MPs) and Gabor filter are the most commonlyused and promising methods to characterize spatial context information. MPs were first introduced for hyperspectral images as described in [8]. Then, MPs were further improved with the extended morphological profiles(EMPs) [9]. The attribute profiles (APs) and the extended
multi-attribute profiles (EMAPs) are also efficient torepresent the context information [10]. Shi and Healey
employed a group of Gabor filters to extract spatial
features at different scales and orientations from HSI
obtaining higher classification accuracy [11].
• Others: Apart from the above four groups, other descriptors have also been proposed in the literature. For
example, pairwise hyperspectral angle (PHA) can simultaneously explore the context information across theneighboring bands and neighboring pixels [12].
•However, all the methods mentioned above can only derive
hand-crafted features instead of deep features, which canbe extracted using deep learning networks. The deep features, if properly extracted, can be more discriminative than
handcrafted features for the problem of hyperspectral image
classification.
During last decade, the representative works in [13], [14]have brought revolution to deep learning methods and havealso promoted their development in the field of machinelearning and pattern recognition. In addition, deep learning
methods have demonstrated the great potential in the fieldof remote sensing for extracting features, which are abstract,robust and invariant to most local changes of the input values [15]. For instance, Aptoula et al. extracted the attributeprofiles from the raw data and then gave them as input toconvolutional neural networks. In this way, the handcraftedIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

features can be further enhanced by the convolutional neuralnetworks and better serve the classification task [16]. Chen etal. employed the stacked autoencoders to extract the highlevel semantic features. They also proved that the semantic features outperformed the hand-crafted features for HSI
classification [2]. After that, more researchers started to add
the spatial information in deep learning architectures. Thus,a spatial deep auto-encoder, which integrated a similarity
regularization term into the loss function, was proposed to
extract discriminative spatial-spectral features [17]. Han et al.
proposed to efficiently represent the spatial-spectral featuresusing the unsupervised convolutional sparse auto-encoder [18].
However, the training computational cost for these methods
is high. To address this problem, Romero et al. employed
deep unsupervised convolutional networks to compute sparsefeature representations. They also revealed that the deep
convolutional networks are good at extracting discriminative
high-level semantic features [19]. A two-stream convolutional
network was first presented for action recognition [20]. Yang etal. applied this two-channel CNN to the hyperspectral image
classification task. However, the input of their spectral CNN
channel is vector-based, which means that all the convolutional
and pooling operators in this channel work on 1-D features.
Besides, they do not train the network from scratch. Theirparameters of the bottom and middle layers are transferredfrom an auxiliary domain [21]. In general, if the network canbe trained from scratch, the extracted features can be taskspecific and thus serve better the hyperspectral image task [22].
After feature extraction, fusion is another important stepfor the classification task [23], [24]. The conventional fusion
is implemented either on feature-level (early fusion) or on
decision score-level (late fusion) [25]. The fusion in [26]was based on the feature level, where the deep features werefused by stacking them directly. 3D convolutional neural networks (3D-CNNs) also implemented the fusion at the featurelevel by processing the spectral and spatial features at thesame time [27], [28]. For the late fusion, the most common
strategy is the uniform weight fusion, which assigns uniform
weight values to each probability matrix. For example, Li etal. performed the uniform weight fusion on the probability
outputs of each classifier. However, they extracted the handcrafted spatial features by LBP or Gabor filters [29]. Yang etal. obtained the probability matrices using two channels andconsidered them as the corresponding spectral and spatial deep
features. Then they fed the stacked feature into the fullyconnected layer [21]. However, in term of each classifier, thisfusion strategy assumes that the class features make equivalent
contributions and the fusion weights share a fixed uniform
value. Hence, it fails to take into account the difference of thediscrimination capabilities of the features, and the correlation
among the features is also ignored.
In order to solve the problems mentioned above, we propose
a two-stream deep architecture, which contains a class-specificadaptive fusion scheme. In the two-stream architecture, thefirst stream (spectral stream) is a stacked denoising autoencoder that is used to encode the spectral features, while theother stream (spatial stream) consists of a deep convolutional
neural network that is employed to extract deep spatial fea-

2

tures. In the fusion scheme, the prediction probabilities fromtwo streams are aggregated by the class-specific fusion.
Compared with the original two-stream architecture in [20],the main of the proposed method: (1) It is based on adifferent optimized architecture: the architecture in [20] only
contains ConvNets. Our two-stream architecture contains not
only CNNs, but also SdAE to extract the spatial and spectralfeatures. Furthermore, in our work, we employ a softmax layer
for classification which is integrated together with the twostream architecture and the classification layer and the twostream architecture can be trained jointly in an end-to-end
manner. (2) It is optimized to address the HSI classification
task: our method is proposed for the HIS classification task,
while the architecture is designed for the task of action
recognition in videos in [20]. In the latter, the inputs are
the RGB video frames and the stacked optical flow images,
whereas we take as input the spatial image patches and pixelwise spectral values. As there are plenty of training samples
in action recognition task, in [20] authors employed very deeparchitectures, such as VGG. However, the training samples arevery limited in HIS classification task. Therefore, we designed
an appropriate light CNNs for the HIS classification task.
The proposed architecture has the following advantages:(1) Deep spectral and spatial features can be extracted byeach stream, which can be trained from scratch. Therefore, the deep features can be learned specifically for theclassification task.
(2) A class-specific fusion scheme is presented to learn theclass fusion weights adaptively taking into account thecorrelation among different classes.
(3) To alleviate the overfitting of the fusion weight values, theloss function is improved by adding a weight regularizer.
The rest of the paper is organized as follows. Section II
briefly reviews the background on deep learning architectures.
Section III introduces the proposed method. Section IV evaluates the effectiveness of the proposed method using two
hyperspectral datasets. Section V analyzes the experimental
results and draws the conclusion of the paper.
II. BACKGROUND ON DEEP LEARNING ARCHITECTURES
A. Denoising Auto-encoder (DAE)
An auto-encoder (AE) is a multi-layer neural network with
one input layer, one output layer and one hidden layer. AE
first takes as input a signal and maps it into the hidden layer
through a deterministic mapping function. Then the output
layer reconstructs the signal by minimizing the reconstruction
error. Thus, more complex and abstract nonlinear representations can be hierarchically learned from the hidden layer. The
mapping function could either be a nonlinear function (e.g.,
sigmoid, ReLU) or a linear function. When a linear mapping
function is applied, the auto-encoder will behave like a PCA.
A denoising auto-encoder (DAE) [30] is an extension of
the standard AE with a similar structure as shown in Fig.1.
There are two steps in DAE, i.e., encoding and decoding, bothof which are implemented by nonlinear mappings. However,differently from AE, the input of DAE is corrupted by adding
noise, while the output is the original signal without noise.

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

Input layer

3

Output layer

Hidden layer

Input with noiseŏŏ

Encoding

Decoding

Fig. 1. Structure of denoising auto-encoder.

In this way, the learnd DAE is capable of recovering signal
from the noisy input. The learnt representation via DAE ismore robust than that of AE. Therefore, DAE has received
significant amount of attention in the research community,
especially for speech recognition. For example, Ueda et al.
proposed a DAE variant for robust distant-talking speech
recognition [31]. Lu et al. presented a model by ensembling
DAEs, which effectively focuses on local transformations [32].
S. Araki et al. introduced a multi-channel DAE-based speechenhancement approach [33]. DAE model can be also applied
to human pose-based action recognition [34].
Recently, deep learning has also been successively applied
to the field of remote sensing. The remote sensing scientists
use AE/DAE to extract features for subsequent classification
and unmixing [2], [35]–[37]. In [2], AE was used to extract
features for classification, whilst Chen et al. employed multiple DAEs to extract deep features [36]. The autoencoder cascade was presented in [37], which concatenated a marginalized
denoising autoencoder and a non-negative sparse autoencoder
for unmixing.

hand-crafted features [43]. Subsequently, some variants of
SdAE were also proposed. For instance, the marginalized
denoising autoencoder was presented, which marginalized therandom corruption to yield the optimal reconstruction weights,
and these weights were computed in a closed-form [42].
This method does not use back-propagation for the parametertuning, and it results in a relatively low training cost. The
marginalized SdAE was further combined with probabilistic
matrix factorization to learn the sparse representation for thecollaborative filtering [40]. For addressing the hyperspectral
image classification task, the methods in [41], [44] used
the spectral representation extracted by SdAE instead of theraw data, which can well reflect the nonlinearity in a highdimensional data space.
C. Convolutional Neural Networks (CNNs)
The Convolutional Neural Networks (CNNs) are biologically inspired variants of Multi-layer Perceptions. The mostwidely used layers in CNNs are vision layers, activation layers
and loss layers. A representive structure of CNNs is shownin Fig.2. In the vision layers, the image patches with spatial
context information are taken as the input. They are first
convolved with a set of kernels with the same size, and thenthe pooling operation is carried out. The activation function is
implemented to learn the nonlinear representations [19]. The
activation operators are usually element-wise, such as sigmoid,
ReLU and hyperbolic functions. In the loss layer, the mostwidely used loss is the softmax cross-entropy loss.

map 0
map 0

B. Stacked Denoising Auto-encoder (SdAE)
Several DAEs can be stacked into a deep network by feeding
the output from the hidden layer of the previous DAE as
the input to the next DAE. This deep network is named
as the stacked denoising auto-encoder (SdAE). The SdAE
is usually used to extract high-level representation of input
in the noisy scenarios, and then SdAE is fine-tuned for thesubsequent classification task. The training process can bedivided into two stages: unsupervised pre-training of each
DAE and supervised fine-tuning for the whole SdAE. In thestage of pre-training, the greedy layer-wise training is used foreach DAE. In the stage of fine-tuning, the bottom layers of the
SdAE are first initialized by the pre-trained parameters. Next,
a logistic regression layer is added on the top of the network.
Then, the network is further tuned by the logistic regression
layer. By minimizing the classification error from the logistic
regression layer, the deep semantic features can be learned.
SdAE has been widely used in many applications [38]–[42].
For example, SdAE was exploited to improve the accuracyof transfer learning in [39]. Glorot et al. applied SdAE for
the sentiment analysis task and demonstrated that these deep
features, when combined with a simple linear SVM classifier,yielded the state-of-the-art performance outperforming the

map 1

…

map M1

vision layers

map 1

…
…

map M2

Activation layers

loss layers

Fig. 2. Structure of convolutional neural networks.

The latest researches show that features extracted by adeeper network are more abstract and thus have better performances in the classification task. Therefore, many deeper
convolutional networks have been constructed in recent years,
such as AlexNet (8 layers) [45], VGG-Net (16-19 layers) [46],GoogLeNet (22 layers) [47] and residual net (152 layers) [48].
These deep networks require lots of training data to learn theparameters. However, the collection for the labeled training
data of hyperspectral images is very laborious and expensive, and the limited training data will cause the over-fitting
problem. Therefore, even if these deeper networks have asignificant advantage in extracting features, shallow and small
networks are more appropiate for hyperspectral images. This
problem will be further discussed in Section IV.

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

4Classes
5000

…

4500

Output…

3500

Sampling
pixel by pixel

…

3000

…

…

…

2500

…

Samples

Input

4000

…

2000

1500

1000

0

20

40

60

80

100

120

140

160

180

…

SoftMax

200

Pixel

Feature Extraction via SdAE

Probability Matrix

Classspecificfusion…

Classes

CNN

…

Input

OutputCNN…

SoftMax

Samples

Sampling
Patches…

CNN…

SoftMax
Raw data

Patches

Feature Extraction via CNN

Probability Matrix

Fig. 3. Overview of the proposed two-stream deep architecture. The stream on the top is the spectral stream which is a stacked denoising autoencoder. The input for this stream are the spectral pixels. The bottom stream is the spatial stream which is a deep convolutional neural network.
The input for the spatial stream are the image patches.

CNNs have shown an explosive popularity in image classification [45], [49]. For instance, Ciresan et al. presented
a fast, fully parameterizable GPU implementation of CNNs
variants for image classification. It has very good generalization capability and high speed. However, it is computationallyprohibitive on CPUs [50]. In order to carry out CNNs on CPUs
with great efficiency, an end-to-end CNNs architecture and a
3-dimension CNN-based feature extraction model were presented in [51] and [52], respectively. These models can boost
the discriminative capability and robustness of the extraced
features. In addition, a mixed deep learning model composedby both deep auto-encoder and CNNs was proposed [22]. It
first employed a pre-trained CNNs from an auxiliary domain(RGB images) for feature extraction. Then a deep auto-encoder
was used to process the deep features. These two steps were
implemented independently. Thus, pre-trained CNNs have apoor adaptive capability for HSI datasets since they are neither
trained or fine-tuned using HSIs.

III. P ROPOSED A RCHITECTURE
A major challenge in hyperspectral image classification
is how to fully and efficiently extract and fuse the spectral
information of a single pixel and its spatial correlations with
the neighbouring pixels in the same local context. In order toaddress this issue, we propose a two-stream deep architecture
which incorporates and fuses spectral and spatial information.
As shown in Fig.3, the proposed two-stream deep architecture
consisits of two streams (i.e. the spectral stream and the spatial
stream) and a class-specific fusion architecture. The spectral
stream is a SdAE that handles the spectral pixels, while thespatial stream is composed of CNNs which are in charge ofprocessing the spatial image patches. The class-specific fusion
scheme can learn the fusion weight values of different classes
adaptively, and these weights can be further optimized in the
training process.

A. Spectral stream
DAE is the basic component for SdAE. Given a hyperspectral image pixel x 2 Rd represented in d-dimension, it should
be first corrupted into x̃ by setting some elements of x tozeros or adding a Gaussian noise before feeding it into DAE.
The hidden representation of x̃ is computed by:(1)

h = s(Wx̃ + b),

where h 2 Rd1 , and d1 is the number of units in the hidden
layer. W and b are the encoding weight and the corresponding
bias vector, respectively. s is the activation function (e.g., thesigmoid, ReLU). We use the ReLU in this stream. The hidden
representation is then mapped back to reconstruct z without
noise using the following function:z = s(W0 h + b0 )(2)

where W0 and b0 are the decoding weight and bias vector. The
reconstruction error can be measured in many ways, dependingon the distribution assumptions on the input dataset. In thispaper, we choose cross-entropy to compute the reconstruction
error, and minimize it to learn the optimal parameter ⇥⇤ ={W, W0 , b, b0 } by:⇥⇤ = min⇥dX[xk log zk + (1xk ) log(1

zk )]

(3)

k=1

DAEs can be stacked to form SdAE by feeding the hidden
representations of the previous DAE layer as the input to thecurrent DAE layer. The spectral stream (on the top row in Fig.
3) analyzes the spectral similarity encoding by SdAE, whichcontains one input layer, three DAE layers and one output
layer. For parameter optimization, we train the first DAE layer(encoding and decoding) and obtain the parameters ⇥1 , andthen feed the hidden representation of the first DAE layer tothe second DAE layer. The remaining two DAE layers can betrained in the same manner and the parameters ⇥2 , ⇥3 can belearned.

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

5

Different from other works in which the feature extraction(e.g., HOG) and classifier (e.g., SVM) training processes are
implemented separately, the proposed spectral stream can be
fine-tuned in an end-to-end manner using the semantic labels
as supervision. The labels of test set can be immediately
predicted by the softmax layer. Therefore, the efficiency ishigher than those of other one-versus-all classifiers.
B. Spatial stream
As shown in the bottom row of Fig.3, the spatial streamis CNNs which are used to extract spatial features. The
convolutional layer generates the feature maps by convolving
the output (i.e., feature maps) of its previous layer with a setof kernels. Then, the pooling layer is applied to obtain more
abstract and transition invariant features. Pooling operator
can greatly reduce the computational cost and increase thegeneralization ability of the model [53].
In order to take advantage of the spatial context information,
image patches centered at the corresponding pixels from thespectral stream are cropped and fed into CNNs. Since thenumber of these labelled image patches is usually quite small
compared with that of the available natural images databases,
we build a shallow CNN architecture with two convolutional
layers to process these image patches. A more detailed studyof the optimal number of convolutional layers is available in
Section IV-D.
Let x 2 Rd represent the hyperspectral image pixel with ddimension, and S 2 Rd⇥m⇥m denote the image patch centeredat x, where the spatial size of the image patch at each channel
is m ⇥ m. The kernel size for the two convolutional layers
is set to [k1 , k2 ] and their kernel numbers are set to n1 andn2 , respectively. The kernel size for two pooling layers is set
to [s1 , s2 ]. Formally, the first layer can be expressed as an
operation F1 :
F1 (S) = max(0, W1 ⇤ S + B1 )(4)

where W1 and B1 denote the kernels and biases, and ‘⇤’
denotes the convolutional operation. Here, W1 corresponds
to n1 kernels with size d ⇥ k1 ⇥ k2 . The output of the firstlayer is composed of n1 feature maps.
The convolutional layer reduces the image size to [m k1 +1, m k2 +1], and the pooling layer further reduces the featuremaps to the size of [ m sk11 +1 , m sk22 +1 ]. The operation of thesecond convolutional layer is:
F2 (S) = max(0, W2 ⇤ F1 (S) + B2 )(5)

where W2 contains n2 kernels having size n1 ⇥ k1 ⇥ k2 , andB2 has a dimension equal to n2 .
Then the image size is reduced to [f1 , f2 ], and f1 =( m sk11 +1
k1 + 1)/s1 , f2 = ( m sk22 +1
k2 + 1)/s2 . For
the fully-connected layer, the feature maps obtained from theprevious layer should be reshaped to a 2-dimension matrix,
because the fully-connected layer can only operate on 2dimension matrices. The used expression is as follows:
F3 (S) = s(W3 F2 (S) + B3 )(6)

where W3 is a matrix with the size [n2 ⇥ f1 ⇥ f2 , n3 ]. B3
is a n3 -dimensional vector, where n3 is the number of hidden
units of the fully-connected layer.
Similar to the spectral stream, the spatial stream can also
adjust the parameters depending on the distribution of the input
data, and works in an end-to-end manner. It is worth notingthat the two streams can be fine-tuned jointly, and the featuresfrom the two streams can be efficiently taking into accounttheir different informative content and correlation. This canbenefit the overall classification task.
C. Class-specific Fusion
Late fusion is one of the most effective schemes to enhance classification performance through combining prediction scores of multiple classifiers, each of which can be trainedby a specific type of features. The proposed two streams are
implemented using the spectral SdAE and spatial CNNs, andeach stream can directly obtain the predicted probability scorewith respect to the input. Therefore, a robust prediction andbetter performance can be achieved by fusing the different
classification results.
Differently from the traditional uniform weight fusion, we
attempt to learn the fusion weight values of different classes
adaptively when training the whole network. Let p1 and0
p2 2 R1⇥N represent the probability vectors obtained fromthe spectral and spatial streams, and N 0 be the number ofclasses. The correspongding fusion weight is represented by0!i = {!i1 , !i2 , ..., !ij , ..., !iN }(i 2 1, 2) where !ij is the jth class fusion weight of the i-th stream. We can express the
fused vector as follows:p · ! = p1

!1 + p 2 !2
2 1
!1 0 · · ·6 0 !12 · · ·
6
..
6 ..
..
6 .
.
.
6
6
0
0 ···
1
N0 6 0= [p11 , · · · , pN
1 ,p2 , · · · , p2 ] 6! 1
0
···
2
6
6 0 !22 · · ·
6
6 .
..
..
4 ..
.
.
0
0 ···

3
0
0 7
7
.. 7
. 7
7
N0 7
!1 7
0 7
7
0 7
7
.. 7
. 5
!2N

(7)

0

where p is the stacked probability vector formed by p1 and p2 ,00
!2R2N ⇥N is a matrix composed of the class fusion weights,
and represents the element-wise product.
By analyzing (7), we can observe that the target of fusion
is to learn the optimal class fusion weight values, whichis equivalent to the formula of learning the weights of afully-connected layer. To this end, we use a fully-connected
layer to construct the class-specific fusion architecture. A
softmax layer is added on the top of the class-specific fusionarchitecture. In this way, the class fusion weights can be learnt
and adjusted adaptively via minimizing the loss function ofthis softmax layer. Instead of assigning random values tothe weights of the fully-connected layer, we only initialize
the elements on the diagonal while the other elements are
simultaneously set to 0. This initial constraint will be lossen
during the training process, which means that the correlation

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

6

among different classes can also be taken into account in thefusion process.
To alleviate the risk of overfitting of the fusion weights, aregularizer is integrated into the loss function of the softmax
layer. The overall loss function is given as follows:! = min loss(p, y; !) + k!k2!

(8)

where loss(·) is the original loss function of softmax layer, yrepresents the ground-truth labels, and k·k2 is the l2 norm.
The conventional fusion is always independent from thefeature learning process, and the fusion weights are set manually. Differently, in our work, the feature fusion is integrated
together with the feature learning pipeline. Therefore, thefusion weights can be learned adaptively by referring to thefeatures that are learned and the features can also be tuned byreferring to the fusion weights. In this way, the joint trainingof the overall architecture can boost the HIS classification
performance.
IV. E XPERIMENTS
A. Datasets description and parameter setting
The experiments to evaluate the effectiveness of the proposed two-stream deep architecture have been conducted on
two hyperspectral images.
The first hyperspectral image is the Indian Pines. It contains
220 spectral bands and has a size of 145⇥145 pixels, with eachpixel measuring approximately 20m by 20m on the ground.
20 spectral bands are removed due to the noise and water
absorption phenomena. Sixteen mutually exclusive classes areincluded. Some classes in this dataset have very few samples,
thus we split the dataset into training and testing sets with theratio of 3:2. The three-channel false-color composition and thereference land-cover map of Indian Pines are shown in Fig. 4.
The second hyperspectral image is the University of Pavia.
Water absorption bands are removed, and the original 115
bands are reduced to 103 bands. Fig.5 shows the three-channel
false-color composition and the reference land-cover map ofUniversity of Pavia, which mainly includes 9 classes (i.e.,asphalt, meadows, trees, metal sheets, bare soil, bitumen,bricks, shadows and gravel). For this dataset, we set the ratioof the training set and testing set to 1:9.

(a)Asphalt Bricks(b)

Gravel Meadows Shadows Bitumen Bare soil Trees Metalsheet

Fig. 5. Pavia University dataset: (a) three-channel false-color composition(bands 16, 27, and 45 for RGB) (b) reference land-cover.

For the spectral stream, the network weights are learnt
using the mini-batch Stochastic Gradient Descent (SGD) withrespect to the loss defined in (3). SdAE was composed of three
DAE layers, in which the number of hidden units for each
DAE layer was set to 102 . On the top of the spectral stream, asoftmax layer was added to predict the scores of samples. The
corruption level was selected from the set of [0.1, 0.2, 0.3],
because larger values may lead to the overfitting problem.
Following [2], the number of epochs for pre-training and finetuning were both set to 103 . In the process of SGD, the initial
learning rates in the pre-training and fine-tuning phases wereboth set to 10 1 , and the weight decay was 0.5. The learning
rates are decreased every 102 iterations, and become almost
one thousandth of the initial value when pre-training and finetuning are terminated. For the spatial stream, two convolutional
layers, one fully-connected layer and one softmax layer werestacked to construct the CNNs model, in which the number ofhidden units in fully-connected layer was 102 , and the softmax
layer was used to predict the scores of samples by the deep
sptial features. For the class-specific fusion architecture, theinitialization of the weights in the fully-connected layer is
described in Section III-C.
The performance evaluation metrics are the overall accuracy(OA), the average accuracy (AA), the F1-score and the Precision. To avoid biased estimation, ten independent tests werecarried out using Theano and Keras on a computer equippedwith an Intel Core i5 Processor at 2.70-GHz.
B. Analysis of Class-specific fusion

a
Soybeans-min till
Soybeans-clean till
Soybeans-no till
Hay-windrowedb
Bldg-Grass-Tree-Drives
Grass/pasture-mowed
Grass/pasture
Stone-steel towers

Corn-min till
Corn-no till
Corn
Grass/trees

Oats
Wheat
Alfalfa
Woods

Fig. 4. Indian Pines dataset: (a) three-channel false-color composition(bands 17, 27, and 50 for RGB) (b) reference land-cover.

Fig.6 shows the evolution of the class fusion weight values
on the dataset of the University of Pavia. The fusion weightmatrix is composed of !1 for the spectral stream (on the top)
and !2 for the spatial stream (at the bottom). The size of thefusion weight matrix for the considered dataset is 18 ⇥ 9. As
stated in Section III-C, we only assigned the initial values of
0.5 to the diagonal of the weight matrix of the fully-connected
layer (see Fig.6 (a)). In the testing phase, we selected 10%
samples from each class as the training set. We can observethat as the number of training epochs increases the values inIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

Spectral reflectanceConstruction in 500 epochs

Construction in 1000 epochs

Spectral reflectancethe fusion weight matrix also evolve accordingly. Comparing
the values in the first column (which corresponds to the fusionweights of the first class) of the two weight matrices, one cansee that at the beginning !11 = !21 = 0.5 (Fig.6 (a)), while after
1000 training epochs !11 and !21 decrease to 0.4082 and 0.429(Fig.6 (b)), respectively. At the same time, the value locatedat (row 4, column 1) increases to 0.2919, which means thatthe features of the fourth class in the spectral stream greatly
contribute to the classification of the samples from the firstclass. Similar behaviours also take place in other columns.
The observation of the fusion weight demonstrates that in the
training phase the class-specific fusion scheme can adaptively
tune the weights of each class by taking into account thecorrelations between them.

7(a)Spectral bands
(b)

(c)

Fig. 7. Reconstruction of a sample in Indian Pines with different training
epochs: (a) a randomly selected test sample, (b) AE, (c) DAE.
Fig. 6. Evolution of the class fusion weights for Pavia University dataset:(a) initial weights, (b) weights obtained after 1000 training epochs.

D. Effect of the network depth

C. Analysis of the performance of DAE
In order to prove that the spectral features extracted by DAE
are more robust, we implemented the following test using theIndian Pines dataset. We randomly selected 60% and 20% of
the total samples as the training and validation sets to learn
the parameters ⇥⇤ = {W, W0 , b, b0 }, and the remaining ones
were used as the test set. W0 was constrained such that W0 =WT , and W was initialized by uniformly sampling from therange [ 0.5, 0.5]. The reconstructed signals by AE (the middle
column in blue) and DAE (the last column in red) after 500
and 1000 training epochs are shown in Fig.7.
The first row in Fig. 7 shows the ground truth curve, andthe reconstructed spectral curves of AE and DAE after 500
epochs. We can observe that the reconstructed signals cannot
completely match with the original one with a limited numberof training epochs. The curves obtained by AE and DAE
are similar, and DAE has the capability to reconstruct signal
under noisy conditions. Then, we compare the results in eachcolumn. We can observe that the performance of DAE is
improved when the number of the training epochs increases
from 500 to 1000. By analyzing the second row we canobserve that the performance of DAE (last column) is better
than that of AE (middle column), which can be noted in the
spectral bands between [0, 25] and [100, 200]. The spectral
signals located in these spectral bands increase significantly
and move closer to the original signal. Therefore, we canconclude that DAE can be a good alternative to AE to extract
spectral features that are more robust. This is the main reason
for which we choose the DAE to stack the multi-layer network.

The depth of a network plays a very important role. Usually,
deeper architecture can extract more abstract features, whichis crucial for the classification performance. In this part, we
conducted the following two experiments to find out the
optimal network depths for the spectral stream and the spatial
stream on different datasets.
1) Effect on the spectral stream: The network depth in
SdAE corresponds to the number of DAE layers. Considering
the high computational cost of training for fully-connected
networks, we only evaluated the depth from 1 to 5 layers.
The spatial size of the input patches is fixed to 7 ⇥ 7. All theother settings remained the same as in Section IV-A. Then we
predicted the labels of the test set, and listed the correspondingclassification accuracy for the dataset of Indian Pines and the
University of Pavia in Tab. I and Tab. II, respectively.
As shown in Tab. I, we can see that the performance
generally shows an upward trend with a deeper network. Itreaches the peak values (98.65% OA, 99.22% AA, 98.65%
F1-score and 98.68% Precision) with a depth of 3. We canalso observe that the accuracy fluctuates when the depth isgreater than 3. When the depth is set to 5, the accuracy rises
again, but is still inferior compared with the depth of 3. Fromthese observations, we can conclude that a deeper SdAE helps
to improve the performance. However, after a given value,
increasing the depth decreases the performance because of the
problem of overfitting. A 3-layer SdAE is good enough for the
Indian Pines dataset. From Tab. II, we can observe that a 4layer SdAE is better suited to the University of Pavia dataset,and the OA, AA, F1-score and Precision can reach 97.33%,
95.01%, 97.32% and 97.39%, respectively. Therefore, we use
a 3-layer SdAE for the Indian Pines and a 4-layer one for theIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

University of Pavia in the final two-stream deep architecture.
TABLE IE FFECT OF THE DEPTH ON THE SPECTRAL STREAM (I NDIAN P INES ).
Depth (layers)
1
2
3
4
5

OA
86.96
94.88
98.65
98.50
98.65

AA
90.22
96.09
99.22
99.15
99.21

F1-score
86.57
94.85
98.65
98.50
98.65

Precision
87.36
94.91
98.68
98.53
98.68

TABLE IIE FFECT OF THE DEPTH ON THE SPECTRAL STREAM (PAVIA U NIVERSITY ).
Depth (layers)
1
2
3
4
5

OA
94.95
95.74
96.20
97.33
96.52

AA
90.73
92.68
93.15
95.01
94.69

F1-score
94.77
95.74
96.17
97.32
96.52

Precision
94.99
95.80
96.24
97.39
96.69

2) Effect on the spatial stream: Similar to the spectral
stream, we also used the same dataset to train and test the
spatial stream using the different depths. The network depth
varied in the range from 1 to 5. We fixed the size of spatial
patch to 7 ⇥ 7. Considering the relatively small spatial size of
the Indian Pines image (i.e., 145 ⇥ 145 pixels), we removed
the pooling operator behind the convolutional layer. For a faircomparison, the kernal size of all the convolutional layers
was set to 2 ⇥ 2, and its kernel number was set to 50. The
classification results for the two datasets are summarized in
Tab. III and Tab. IV.
From Tab. III, we can derive the following observations:1) The OA generally increases until depth = 3, reaching thepeak of 98.65%. Then, OA fluctuates slightly as the number oflayers increases further. 2) CNNs with three layers can produce
the largest AA, which suggests that a deeper CNNs model canwell learn the distribution of classes. 3) The optimal F1-socre
and Precision can be achieved at depth = 3. Therefore, we
select CNNs with three layers for the Indian Pines dataset
in the following tests. In addition, we can also observe from
Tab. IV that the highest OA, AA and recall can be obtained
when depth = 2 for the University of Pavia dataset.
Obviously, each stream is heavily influenced by the network
depth and the selection of network depth for each stream plays
an important role in the performance obtained.

8

TABLE IV
E FFECT OF THE DEPTH ON THE SPATIAL STREAM (PAVIA U NIVERSITY ).
Depth (layers)
1
2
3
4
5

OA
93.16
97.33
96.24
95.32
95.92

AA
85.46
95.01
93.99
93.36
93.91

F1-score
92.46
97.39
96.21
95.37
95.92

Precision
93.23
97.39
96.30
95.37
95.94

chitecture. In this section, we compare the classification accuracy (OA) of different spatial sizes with different amountof training data. For the Indian Pines dataset, we randomly
selected training data from the whole dataset with a ratio
r , which was set to 10%, 20%, 30%, 40%, 50%, 60%. The
ratio r of the Pavia University dataset varied in the set{1%, 3%, 5%, 10%, 15%, 20%}. The spatial size varied in theset {3⇥3, 5⇥5, 7⇥7}, and the kernel size of two convolutional
layers was 2 ⇥ 2. For the small spatial size (i.e., 3 ⇥ 3), weremoved the pooling layer because of the rapid downsamplingof image size. The OA curves are shown in Fig.8.

(a)

TABLE IIIE FFECT OF THE DEPTH ON THE SPATIAL STREAM (I NDIAN P INES ).
Depth (layers)
1
2
3
4
5

OA
97.92
97.54
98.65
98.31
97.78

AA
98.56
98.81
99.22
98.95
97.43

F1-score
97.92
97.54
98.65
98.31
97.79

Precision
97.94
97.61
98.68
98.33
97.85

E. Effect of the spatial size
The spatial size s of the input image patches is anotherimportant parameter for the proposed two-stream deep ar-(b)

Fig. 8. Accuracy curves for different spatial sizes versus different training
sample ratios: (a) Indian Pines dataset (b) Pavia University dataset.

From Fig. 8 (a), one can observe that the curves of s =7 ⇥ 7 and s = 5 ⇥ 5 have a similar performance improvement
as the training set ratio increases. More specifically, the OA
curve of s = 7 ⇥ 7 increases monotonically from 89.02%(r = 10%) to 98.65% (r = 60%). When r increases fromIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

9TABLE VC OMPARISON OF CLASSIFICATION ACCURACIES PROVIDED BY DIFFERENT METHODS (I NDIAN P INES DATASET ).
Class
Alfalfa
Corn-no till
Corn-min till
Corn
Grass/pasture
Grass/trees
Grass/pasture-mowed
Hay-windowed
Oats
Soybeans-no till
Soybeans-min till
Soybeans-clean till
Wheat
Woods
Bldg-Grass-Tree-Drives
Stone-steel towers
AA
OA

No. of train
33
861
501
141
299
449
9
294
12
580
1480
369
127
777
228
57
-

(a)

SC3 SVM
91.98
52.67
68.15
78.63
90.48
91.48
89.74
87.46
95.00
71.01
56.39
73.40
99.21
92.30
62.81
95.09
80.99
72.42

SdAE
100.00
73.91
72.97
88.46
91.53
93.75
100.00100.00100.00
87.96
88.25
62.42
97.44
98.25
80.00
100.00
89.68
86.04

(b)

(f)
Soybeans-min till
Soybeans-clean till

SS-LPSVM
98.89
75.45
75.08
95.30
93.40
95.72
95.38
97.22
100.00
82.77
65.37
87.30
99.62
97.74
88.89
96.84
90.31
84.11

3D-CNNs
96.67
94.62
98.68
97.75
98.54
97.40
100.00
99.45
87.50
97.86
98.43
95.65
100.0099.63
96.64
100.00
97.43
97.77

(c)

(g)
Soybeans-no till
Hay-windrowed

CNNs
100.00
92.61
90.81
94.23
96.61
95.83
100.00100.00100.00
95.81
96.91
85.23
100.0099.30
96.67
100.00
96.50
95.36

MLRpr
96.30
80.33
68.11
100.00
93.16
99.87
42.31
100.00
0.00
75.83
96.27
94.79
100.00
99.77
75.26
97.89
82.49
89.61

ppMLRpr
98.15
95.12
97.48
99.15
95.98
99.60
57.69
100.00
0.00
90.60
99.47
99.19
99.06
99.85
99.74
93.68
89.05
97.41

(d)

(h)
Bldg-Grass-Tree-Drives
Grass/pasture-mowed

ppMLR
98.15
98.47
98.44
97.86
95.17
99.73
100.00100.00
95.00
96.49
98.46
98.53
99.53
99.77
99.21
100.00
98.43
98.50

(e)

(i)

Grass/pasture
Stone-steel towers

Corn
Grass/trees

Proposed
100.0095.35
98.75
100.00100.0099.32
100.00100.00100.00100.0098.03
100.00
97.87
99.62
98.53
100.00
99.22
98.65

(j)
Corn-min till
Corn-no till

Alfalfa
Woods

Oats
Wheat

Fig. 9. Visual classification results for the Indian Pines dataset: (a) reference land-cover (b) SC3 SVM (c) SS-LPSVM (d) SdAE (e) CNNs (f) 3D-CNNs(g) ppMLR (h) MLRpr (i) ppMLRpr (j) proposed method.

10% to 60%, the OA curve of s = 5 ⇥ 5 raises 10.39% and aslight performance drop takes place at r = 40%. As expected,
the curve of s = 3 ⇥ 3 first increases until r = 50%, and thendecreases linearly. If we fix the training ratio r = 40%, we
can observe that the OA curve increases rapidly by increasing
spatial size. It reaches the peak of 97.80% at r = 40%, andthe improvement is more relevant from s = 3⇥3 to s = 5⇥5.
The behavior of the OA curve for the Pavia University dataset(Fig. 8(b)) is similar to that of the OA curve in Fig. 8(a),that is, the OA curves generally raise when the training set
ratio increases. The curves of s = 7 ⇥ 7 and s = 5 ⇥ 5 havea significant advantage over the curve of s = 3 ⇥ 3. With
respect to the curve of s = 7 ⇥ 7, the OA first increases from87.14% (r = 1%) to 97.50% (r = 10%). If the ratio continues
to increase, similar OAs are obtained (96.86% at r = 15%).
Then a slight fluctuation is observed at r = 20%. Differently

from the Indian Pines dataset, when only a small amount oftraining samples is available, the proposed method can stillachieve an accurate performance, e.g., OA at r = 10% canreach 97.50% for the curve of s = 7 ⇥ 7. The suitable spatial
size is s = 7 ⇥ 7 or s = 5 ⇥ 5. We also add the pooling
operation behind the convolutional layer since it can bringmore nonlinearity to the network model. Thus, s = 7 ⇥ 7 will
be adopted in the proposed architecture.
F. Comparsion with other methods
Tab. V reports the classification accuracy for different
methods when applied to the Indian Pines dataset. The
spatial-spectral classification methods (SC3 SVM, SS-LPSVM,
ppMLR, MLRpr and ppMLRpr) in [54]–[57] focus on the
integration of spatial context information in classification for
the improvement of accuracy. Therefore, they were selected asIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

10TABLE VIC OMPARISON OF CLASSIFICATION ACCURACIES PROVIDED BY DIFFERENT METHODS (PAVIA U NIVERSITY DATASET ).
Class
Asphalt
Meadows
GravelTrees
Metal sheets
Bare soil
Bitumen
Bricks
ShadowsAA
OA

No. of train
664
1865
210
307
135
503
133
369
95
-

SS-LPSVM
94.18
76.14
76.04
95.17
99.55
68.64
88.05
74.31
99.68
85.75
80.88

(a)

SC3 SVM
55.04
68.50
47.93
61.46
88.03
38.34
93.46
77.19
98.84
69.86
64.16

SdAE
87.32
91.48
0.00
83.87
97.99
24.33
1.99
81.48
98.66
63.01
74.70

(b)

(f)

Bricks

3D-CNNs
94.05
98.44
79.42
97.47
100.00
89.36
86.67
86.65
97.76
92.20
94.35

(c)

(g)
Asphalt

CNNs
92.00
97.69
74.33
93.84
100.00
83.09
78.45
76.17
95.99
87.95
91.26

Meadows

MLRpr
83.36
94.96
52.08
97.00
99.36
81.20
72.79
85.38
98.95
85.01
88.31

(d)

(h)
Gravel

ppMLR
91.71
97.48
66.33
95.52
99.82
82.80
76.65
82.89
98.82
88.00
91.42

BitumenBare soil

Trees

Proposed
97.47
99.92
83.80
98.98
100.00
97.75
77.44
96.65
99.65
94.63
97.50

(e)

(i)

Shadows

ppMLRpr
91.51
95.04
53.15
95.89
98.82
85.61
72.88
82.62
98.82
86.04
89.85

(j)
Metal sheets

Fig. 10. Visual classification results for the Pavia University dataset: (a) reference land-cover (b) SC3 SVM (c) SS-LPSVM (d) SdAE (e) CNNs (f)3D-CNNs (g) ppMLR (h) MLRpr (i) ppMLRpr (j) proposed method.

the benchmark methods, and the procedure for their parametersetting can be found in the corresponding references. Besides,
we also provide comparisons with other deep learning methods(including SdAE, CNNs and 3D-CNNs [27]). The spatial sizeof the input image patches for CNNs and 3D-CNNs was set
to 7 ⇥ 7. For the spectral SdAE stream, we used the 3-layer or
4-layer network according to the different datasets, and eachhidden layer had 100 units. For the spatial CNNs stream, thekernel size of convolutional layer was 2 ⇥ 2. Because larger
kernel size in pooling layer will result in a dramatic image size
shrinkage, we set the kernel size of pooling layers to 2 ⇥ 2.
From Tab. V, we can observe that the proposed method

achieves significantly higher OA, with gains of 3.29%, 9.04%,
12.61%, 14.54% and 26.23% over CNNs, MLRpr, SdAE, SSLPSVM, SC3 SVM, respectively. The OA of the proposed
method is slightly better compared with ppMLRpr, ppMLR
and 3D-CNNs. The AA of the proposed method can reachthe maximum value of 99.22%, suggesting that the proposed
deep architecture can effectively reflect the distribution information of the data. CNNs, 3D-CNNs and ppMLR obtain good
AA results, i.e., 96.50%, 97.43% and 98.43%, respectively.
Nevertheless, their AA results are lower than those of the
proposed method. We can also observe in Tab. V that only
9 and 12 samples were selected as the training sets for theIEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING

11

Grass/pasture-mowed and Oats, respectively. The accuraciesof SC3 SVM, MLRpr and ppMLRpr for these two classes
are greatly influenced by the number of training samples.
However, the accuracies of these classes obtained by the
proposed method can both achieve 100%. For illustrative purposes, the classification maps are shown in Fig.9. Comparedwith Fig.9(j), the noise in Fig.9(b)-Fig.9(d) is quite evident.
The samples of Soybeans-no till in the upper left corner are
misclassified into Soybeans-clean till in Fig.9(g)-Fig.9(i). The
class of Oats can best demonstrate the advantages of the
proposed method.
The OA, AA and individual class accuracy using the
University of Pavia dataset are shown in Tab. VI. The OA
and AA of the proposed method are 97.50% and 94.63%,
which are significantly higher than those of other classification
methods. The accuarcy for each individual stream (i.e., SdAE
and CNNs) both have an improvement after the integrationby the proposed two-stream architecture. The classification
accuracy of 3D-CNNs (94.35% OA) is 3.15% lower than thatof the proposed method. The visualization of the classification
results are shown in Fig. 10. By analyzing the figure, one canobserve that other methods often incur in errors on the Baresoil areas, whereas the proposed two-stream architecture canclassify the regions accurately.
TABLE VII
P- VALUES ( IN PERCENT ) AND WIN / TIE / LOSS COUNTS OF THE PROPOSEDMETHOD VERSUS OTHER COMPETITIVE METHODS BASED ON OA.
Dataset

3D-CNNs

ppMLR

W/T/L

Indian Pines
Pavia University
W/T/L

0.72/0.36/99.64 (W)
0.65/0.33/99.67 (W)
2/0/0

38.25/19.13/80.87 (T)0.68/0.34/99.66 (W)
1/1/0

1/1/0
2/0/0
-

We have further implemented the Paired t-test at 95%
significance level to illustrate that the proposed method isstatistically better than the baselines (i.e., 3D-CNNs andppMLR). The results are shown in Tab. VII. The original
hypothesis is that the OA of baseline is compared with theproposed method, where p = p1 /p2 /p3 is composed of p1(the OA of the baseline equal to that of the proposed method),
p2 (the OA of the baseline higher than or equal to that ofthe proposed method) and p3 (the OA of the baseline lower
than or equal to that of the proposed method), respectively.
We can see that the proposed method achieves 2 wins, 0
tie and 0 loss when compared to 3D-CNNs, and 1 win, 1
tie and 0 loss when compared to ppMLR. Therefore, fromthe perspective of statistics, our method has a better overall
performance compared with the baselines.
G. Anlysis of the computational efficiency
The analysis of the computational time was carried out on
the Pavia University dataset, and the experimental platform
was a computer equipped with an Intel Core i5 Processor at
2.70-GHz CPU. We took the test set as the validation set, andthe batch size was set to 128. For each epoch, the proposed
method takes 12s to train the model, in which pre-training
and fine-tuning consume 9s and 3s, respectively. To achieve

the best parameters, we need to increase the number of training
epochs, which is set to 1000 for the considered dataset. The
computational efficiency is higher than the traditional methods
as the code is better suited to run on GPU. The GPU-based
SVM [58] takes about 127.80s, which is 6 times faster thanthat on a standard CPU-based SVM. However, it is still muchslower than the softmax classifier. Therefore, the proposedarchitecture is computationally efficient in the testing phase,and consumes less time than either the GPU-based SVM or the
CPU-based SVM. However, if training can be implemented
through the GPU-cuda accelerated Caffe library, the wholetraining time can be further cut down as shown in [16].
V. C ONCLUSION
In this paper, we propose a novel two-stream deep architecture with a class-specific fusion scheme. In the two-streamarchitecture, the stacked denoising auto-encoder is adopted to
encode the spectral features, and deep convolutional neuralnetworks are employed to extract deep spatial features. In
the fusion architecture, the prediction probabilities from two
streams are aggregated by a class-specific fusion strategy,
which learns the class fusion weights adaptively, and takes
into account the correlation among different classes. Experimental results demonstrate that the proposed architecture canachieve competitive performance compared with the state-ofart methods.
To sum up, this paper introduces a novel deep learning
architecture for the problem of hyperspectral image classification. However, the training of the proposed method which
includes the SdAE and CNNs is time-consuming. Therefore,
further research can be conducted to explore more efficientcomputational schemes for the proposed two-stream deeparchitecture, and explore possible semi-supervised learningtechniques.
ACKNOWLEDGMENT
The authors would like to thank Prof. P. Gamba forprovidng the dataset of Pavia. This research is supportedby the Natural Science Foundation of Shandong Province,(No. ZR2017PF004), the National Natural Science Foundationof China (No. 61701272 and No. 61572269) and the Fundamental Research Funds for the Central Universities (No.
2682016CX083).
R EFERENCES[1] G. Camps-Valls, L. Gomez-Chova, J. Munoz-Mari, J. Vila-Frances,
and J. Calpe-Maravilla, “Composite kernels for hyperspectral image
classification,” IEEE Geoscience and Remote Sensing Letters, vol. 3,
no. 1, pp. 93–97, Jan 2006.
[2] Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, “Deep learning-basedclassification of hyperspectral data,” IEEE Journal of Selected Topics
in Applied Earth Observations and Remote Sensing, vol. 7, no. 6, pp.
2094–2107, 2014.
[3] K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, “Deepsupervised learning for hyperspectral data classification through convolutional neural networks,” in 2015 IEEE International Geoscience andRemote Sensing Symposium (IGARSS). IEEE, 2015, pp. 4959–4962.
[4] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson, “Advances in hyperspectral image classification: Earth monitoring withstatistical learning methods,” IEEE Signal Processing Magazine, vol. 31,
no. 1, pp. 45–54, Jan 2014.

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING[5] A. Plaza, J. A. Benediktsson, J. W. Boardman, J. Brazile, L. Bruzzone,
G. Camps-Valls, J. Chanussot, M. Fauvel, P. Gamba, and A. Gualtieri,“Recent advances in techniques for hyperspectral image processing,”
Remote Sensing of Environment, vol. 113, no. 1, pp. S110–S122, 2009.
[6] Y. Tarabalka, J. Chanussot, and J. A. Benediktsson, “Segmentation andclassification of hyperspectral images using minimum spanning forest
grown from automatically selected markers,” IEEE Transactions on
Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 40, no. 5,
pp. 1267–1279, Oct 2010.
[7] G. Moser, S. B. Serpico, and J. A. Benediktsson, “Land-cover mappingby markov modeling of spatial-contextual information in very-highresolution remote sensing images,” Proceedings of the IEEE, vol. 101,
no. 3, pp. 631–651, March 2013.
[8] J. A. Palmason, J. A. Benediktsson, J. R. Sveinsson, and J. Chanussot,“Classification of hyperspectral data from urban areas using morphological preprocessing and independent component analysis,” in Proceedings.
2005 IEEE International Geoscience and Remote Sensing Symposium,2005. IGARSS ’05., vol. 1, July 2005, pp. 4 pp.–.
[9] J. A. Benediktsson, J. A. Palmason, and J. R. Sveinsson, “Classificationof hyperspectral data from urban areas based on extended morphological profiles,” IEEE Transactions on Geoscience and Remote Sensing,vol. 43, no. 3, pp. 480–491, March 2005.
[10] M. D. Mura, J. A. Benediktsson, B. Waske, and L. Bruzzone, “Extended
profiles with morphological attribute filters for the analysis of hyperspectral data,” International Journal of Remote Sensing, vol. 31, no. 22, pp.
5975–5991, 2010.
[11] M. Shi and G. Healey, “Hyperspectral texture recognition using a
multiscale opponent representation,” IEEE Transactions on Geoscienceand Remote Sensing, vol. 41, no. 5, pp. 1090–1095, May 2003.
[12] Y. Yuan, J. Lin, and Q. Wang, “Dual-clustering-based hyperspectral band
selection by contextual analysis,” IEEE Transactions on Geoscience andRemote Sensing, vol. 54, no. 3, pp. 1431–1445, 2016.
[13] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm
for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554,
jul 2006.
[14] D. P. Yoshua Bengio, Pascal Lamblin and H. Larochelle, Greedy LayerWise Training of Deep Networks. MIT Press, 2007, pp. 153–160.
[15] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A
review and new perspectives,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, Aug 2013.
[16] E. Aptoula, M. C. Ozdemir, and B. Yanikoglu, “Deep learning with attribute profiles for hyperspectral image classification,” IEEE Geoscienceand Remote Sensing Letters, vol. 13, no. 12, pp. 1970–1974, 2016.
[17] X. Ma, H. Wang, and J. Geng, “Spectral-spatial classification of hyperspectral image based on deep auto-encoder,” IEEE Journal of Selected
Topics in Applied Earth Observations and Remote Sensing, vol. PP,
no. 99, pp. 1–13, 2016.
[18] X. Han, Y. Zhong, and L. Zhang, “Spatial-spectral classification basedon the unsupervised convolutional sparse auto-encoder for hyperspectral
remote sensing imagery,” ISPRS Annals of the Photogrammetry, Remote
Sensing and Spatial Information Sciences, vol. III-7, pp. 25–31, 2016.
[19] A. Romero, C. Gatta, and G. Camps-Valls, “Unsupervised deep feature
extraction for remote sensing image classification,” IEEE Transactions
on Geoscience and Remote Sensing, vol. 54, no. 3, pp. 1349–1362,
March 2016.
[20] K. Simonyan and A. Zisserman, “Two-stream convolutional networks
for action recognition in videos,” CoRR, vol. abs/1406.2199, 2014.
[21] J. Yang, Y. Zhao, C. W. Chan, and C. Yi, “Hyperspectral image
classification using two-channel deep convolutional neural network,” inGeoscience and Remote Sensing Symposium, 2016, pp. 5079–5082.
[22] E. Othman, Y. Bazi, N. Alajlan, H. Alhichri, and F. Melgani, “Using
convolutional features and a sparse autoencoder for land-use scene
classification,” International Journal of Remote Sensing, vol. 37, no. 10,
pp. 2149–2167, 2016.
[23] K. T. Lai, D. Liu, S. F. Chang, and M. S. Chen, “Learning sample specific weights for late fusion,” IEEE Transactions on Image Processing,vol. 24, no. 9, pp. 2772–2783, Sept 2015.
[24] D. Liu, K. T. Lai, G. Ye, M. S. Chen, and S. F. Chang, “Sample-specificlate fusion for visual category recognition,” in Computer Vision and
Pattern Recognition (CVPR), 2013 IEEE Conference on, June 2013, pp.
803–810.
[25] P. K. Atrey, M. A. Hossain, A. E. Saddik, and M. S. Kankanhalli,“Multimodal fusion for multimedia analysis: a survey,” Multimedia
Systems, vol. 16, no. 6, pp. 345–379, 2010.
[26] Y. Chen, C. Li, P. Ghamisi, X. Jia, and Y. Gu, “Deep fusion of remotesensing data for accurate classification,” IEEE Geoscience and Remote
Sensing Letters, vol. PP, no. 99, pp. 1–5, 2017.

12[27] Y. Li, H. Zhang, and Q. Shen, “Spectral-spatial classification of hyperspectral imagery with 3d convolutional neural network,” Remote Sensing,vol. 9, no. 1, p. 67, 2017.
[28] F. Palsson, J. R. Sveinsson, and M. O. Ulfarsson, “Multispectral andhyperspectral image fusion using a 3-d-convolutional neural network,”
IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 5, pp. 639–
643, 2017.
[29] W. Li, C. Chen, H. Su, and Q. Du, “Local binary patterns and extreme learning machine for hyperspectral imagery classification,” IEEE
Transactions on Geoscience and Remote Sensing, vol. 53, no. 7, pp.
3681–3693, 2015.
[30] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in
ICML08, 2008, pp. 1096–1103.
[31] Y. Ueda, L. Wang, A. Kai, and B. Ren, “Environment-dependent
denoising autoencoder for distant-talking speech recognition,” EURASIP
Journal on Applied Signal Processing, p. 92, Dec. 2015.
[32] L. Xugang, T. Yu, M. Shigeki, and H. Chiori, “Ensemble modelingof denoising autoencoder for speech spectrum restoration,” in INTERSPEECH2014, 2014, pp. 885–889.
[33] S. Araki, T. Hayashi, M. Delcroix, M. Fujimoto, K. Takeda,
and T. Nakatani, “Exploring multi-channel features for denoisingautoencoder-based speech enhancement,” in 2015 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), April
2015, pp. 116–120.
[34] A. Budiman, M. I. Fanany, and C. Basaruddin, “Stacked denoising
autoencoder for feature representation learning in pose-based action
recognition,” in 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), Oct 2014, pp. 684–688.
[35] Z. Lin, Y. Chen, X. Zhao, and G. Wang, “Spectral-spatial classification of
hyperspectral image using autoencoders,” in Information, Communications and Signal Processing (ICICS) 2013 9th International Conference
on, 2013, pp. 1–5.
[36] X. Chen, M. Li, and Y. Xiaoquan, “Stacked denoise autoencoder based
feature extraction and classification for hyperspectral images,” Journalof Sensors, p. 10, 2016.
[37] R. Guo, W. Wang, and H. Qi, “Hyperspectral image unmixing using
autoencoder cascade,” in Hyperspectral Image and Signal Processing:
Evolution in Remote Sensing (WHISPERS), 2015 7th Workshop on.
IEEE, 2015, pp. 1–4.
[38] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol,“Stacked denoising autoencoders: Learning useful representations in a
deep network with a local denoising criterion,” J. Mach. Learn. Res.,
vol. 11, pp. 3371–3408, Dec. 2010.
[39] C. Kandaswamy, L. M. Silva, L. A. Alexandre, R. Sousa, J. M. Santos,
and J. M. de S, “Improving transfer learning accuracy by reusing stacked
denoising autoencoders,” in 2014 IEEE International Conference on
Systems, Man, and Cybernetics (SMC), Oct 2014, pp. 1380–1387.
[40] S. Li, J. Kawale, and Y. Fu, “Deep collaborative filtering via marginalized denoising auto-encoder,” in The ACM International, 2015, pp. 811–
820.
[41] Y. Liu, G. Cao, Q. Sun, and M. Siegel, “Hyperspectral classification via
learnt features,” in Image Processing (ICIP), 2015 IEEE International
Conference on, Sept 2015, pp. 2591–2595.
[42] M. Chen, Z. E. Xu, K. Q. Weinberger, and F. Sha, “Marginalized
denoising autoencoders for domain adaptation,” Computing Research
Repository, vol. abs/1206.4683, 2012.
[43] X. Glorot, A. Bordes, and Y. Bengio, “Domain adaptation for large-scalesentiment classification: A deep learning approach,” in ICML, 2011.
[44] C. Wang, P. Zhang, Y. Zhang, L. Zhang, and W. Wei, “A
multi-label hyperspectral image classification method with deeplearning features,” in Proceedings of the International Conference on
Internet Multimedia Computing and Service, ser. ICIMCS’16. New
York, NY, USA: ACM, 2016, pp. 127–131. [Online]. Available:
http://doi.acm.org/10.1145/3007669.3007742[45] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.
[46] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014.
[Online]. Available: http://arxiv.org/abs/1409.1556[47] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper withconvolutions,” CoRR, vol. abs/1409.4842, 2014. [Online]. Available:
http://arxiv.org/abs/1409.4842

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING[48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available:
http://arxiv.org/abs/1512.03385[49] ——, “Spatial pyramid pooling in deep convolutional networks forvisual recognition,” IEEE transactions on pattern analysis and machine
intelligence, vol. 37, no. 9, pp. 1904–1916, September 2015.
[50] D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber, “Flexible, high performance convolutional neural networks forimage classification,” in IJCAI 2011, Proceedings of the International
Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain,
July, 2011, pp. 1237–1242.
[51] S. Yu, S. Jia, and C. Xu, “Convolutional neural networks for hyperspectral image classification,” Neurocomputing, pp. –, 2016.
[52] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction
and classification of hyperspectral images based on convolutional neuralnetworks,” IEEE Transactions on Geoscience and Remote Sensing,vol. 54, no. 10, pp. 6232–6251, Oct 2016.
[53] H. Liang and Q. Li, “Hyperspectral imagery classification using sparse
representations of convolutional neural network features,” Remote Sensing, vol. 8, p. 99, jan 2016.
[54] B. C. Kuo, C. S. Huang, C. C. Hung, Y. L. Liu, and I. L. Chen,“Spatial information based support vector machine for hyperspectral
image classification,” in Geoscience and Remote Sensing Symposium,
2010, pp. 832–835.
[55] W. Liguo, H. Siyuan, W. Qunming, and W. Ying, “Semisupervisedclassification for hyperspectral imagery based on spatialspectral label
propagation,” ISPRS Journal of Photogrammetry and Remote Sensing,vol. 97, pp. 123–137, 2014.
[56] J. Li, M. Khodadadzadeh, A. Plaza, X. Jia, and J. M. Bioucas-Dias,“A discontinuity preserving relaxation scheme for spectral-spatial hyperspectral image classification,” IEEE Journal of Selected Topics inApplied Earth Observations and Remote Sensing, vol. 9, no. 2, pp. 625–
639, Feb 2016.
[57] J. Li, J. Bioucas-Dias, and A. Plaza, “Hyperspectral image segmentation
using a new bayesian approach with active learning,” Geoscience andRemote Sensing, IEEE Transactions on, vol. 49, no. 10, pp. 3947–3960,
oct. 2011.
[58] A. Athanasopoulos, A. Dimou, V. Mezaris, and I. Kompatsiaris, “Gpuacceleration for support vector machines,” in WIAMIS 2011: 12th
International Workshop on Image Analysis for Multimedia Interactive
Services, Delft, The Netherlands, April 13-15, 2011, 2011.

13