International Journal of Machine Learning and Cybernetics
https://doi.org/10.1007/s13042-019-01004-6

ORIGINAL ARTICLE

Transfer channel pruning for compressing deep domain adaptation
modelsChaohui Yu1,2 · Jindong Wang3 · Yiqiang Chen1,2 · Xin Qin1,2Received: 29 March 2019 / Accepted: 19 August 2019
© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Abstract
Deep unsupervised domain adaptation has recently received increasing attention from researchers. However, existing methods
are computationally intensive due to the computational cost of convolutional neural networks (CNN) adopted by most work.
There is no effective network compression method for such problem. In this paper, we propose a unified transfer channel
pruning (TCP) method for accelerating deep unsupervised domain adaptation (UDA) models. TCP method is capable ofcompressing the deep UDA model by pruning less important channels while simultaneously learning transferable featuresby reducing the cross-domain distribution divergence. Therefore, it reduces the impact of negative transfer and maintains
competitive performance on the target task. To the best of our knowledge, TCP method is the first approach that aims at
accelerating deep unsupervised domain adaptation models. TCP method is validated on two main kinds of UDA methods:
the discrepancy-based methods and the adversarial-based methods. In addition, it is validated on two benchmark datasets:
Office-31 and ImageCLEF-DA with two common backbone networks - VGG16 and ResNet50. Experimental results demonstrate that our TCP method achieves comparable or better classification accuracy than other comparison methods while
significantly reducing the computational cost. To be more specific, in VGG16, we get even higher accuracy after pruning
26% floating point operations (FLOPs); in ResNet50, we also get higher accuracy on half of the tasks after pruning 12%
FLOPs for both discrepancy-based methods and adversarial-based methods.
Keywords Unsupervised domain adaptation · Transfer channel pruning · Accelerating

1 Introduction
Deep neural networks have significantly improved the performance of diverse machine learning applications. However, in order to avoid overfitting and achieve better performance, a large amount of labeled data is needed to train a
deep network. Since the manual labeling of massive training
data is usually expensive in terms of money and time, it isurgent to develop effective algorithms to reduce the labelingworkload on the domain to be learned (i.e. target domain).

* Yiqiang Chenyqchen@ict.ac.cn
1

Beijing Key Laboratory of Mobile Computing and Pervasive
Device, Institute of Computing Technology, Chinese
Academy of Sciences, Beijing, China

2

University of Chinese Academy of Sciences, Beijing, China

3

Microsoft Research Asia, Beijing, China

A popular solution to solve the above problem is called
transfer learning, or domain adaptation [32, 45, 46], whichtries to transfer knowledge from well-labeled domains (i.e.
source domains) to the target domain. Specifically, unsupervised domain adaptation (UDA) is considered more challenging since the target domain has no labels. The key is
to learn a discriminative model to reduce the distribution
divergence between domains.
Traditional methods perform adaptation by reweighting samples from the source domain [2, 19], or seeking an
explicit feature space transformation that transforms thesource and target samples into the same feature space [31,46]. Recent studies have indicated that deep networks canlearn more transferable features for domain adaptation [5,48]. The latest advances have been achieved by embedding
domain adaptation modules in the pipeline of deep featurelearning to extract domain-invariant representations [8, 23,
25, 35, 41]. This is because that they take advantages of
CNN (convolutional neural networks) to learn more transferable representations [23] compared to traditional methods.

13Vol.:(0123456789)

International Journal of Machine Learning and Cybernetics

Fig. 1  The framework of the proposed transfer channel pruning (TCP) method

Popular CNN architectures such as AlexNet [21], VGGNet[38], and ResNet [14] are widely adopted as the backbone
networks for deep unsupervised domain adaptation methods.
Then, knowledge can be transferred to the target domain by
reducing the cross-domain distance such as maximum mean
discrepancy (MMD) [31] or KL divergence [31].
Recently, adversarial learning [11] has been successfully
embedded into deep networks to reduce distribution discrepancy between the source and target domains. Prior advanced
adversarial adaptation methods [8, 29, 35, 43] have shownpromising results in several domain adaptation tasks. Existing adversarial domain adaptation methods either learn asingle domain discriminator to align the global source andtarget distributions, or pay attention to align subdomains
based on multiple discriminators. For instance, domainadversarial Neural network (DANN) [8, 9] focuses on the
global adversarial learning, while multi-adversarial domain
adaptation (MADA) [35] pays attention to the subdomain
adaptation by training several domain classifiers.
Unfortunately, it is still challenging to deploy these deep
UDA models on resource constrained devices such as mobile
phones since there is a huge computational cost requiredby these methods. In order to reduce resource requirement
and accelerate the inference process, a common solution is
network compression. Network compression methods mainly
include network quantization [36, 49], weight pruning [13,15, 28], and low-rank approximation [4, 12]. Especiallychannel pruning [15, 28], which is a type of weight pruning
and compared to other methods, it does not need special
hardware or software implementations. In addition, it can
reduce negative transfer by pruning some redundant channels, therefore it is a good choice for compressing deep UDA
models.
However, it is not feasible to apply the above network
compression methods directly to the UDA problems. The
reasons are two-fold. Firstly, these compression methods areproposed to solve supervised learning problems, which is notsuitable for the UDA settings since there are no labels in thetarget domain. Secondly, even if we can acquire some labels
manually, applying these compression methods directly to
UDA will result in negative transfer [32], since they fail13

to consider the distribution discrepancy between the source
and target domains. Currently, there is no effective network
compression method for UDA.
In this paper, we propose a unified network compression
method called transfer channel pruning (TCP) for accelerating deep unsupervised domain adaptation models. The
general framework of our TCP method is shown in Fig. 1.
Starting from a deep unsupervised domain adaptation basemodel, TCP method iteratively evaluates the importance ofchannels with the transfer channel evaluation module andremove less important channels for both source and target domains. TCP method is capable of compressing thedeep UDA model by pruning less important channels while
simultaneously learning transferable features by reducing thecross-domain distribution divergence. Experimental results
demonstrate that TCP method achieves better classification
accuracy than other comparison pruning methods while
significantly reducing the computational cost. To the bestof our knowledge, our TCP method is the first approach toaccelerate the deep UDA models.
To summarize, the contributions of this paper are as
follows:(1)

We present our TCP method as a unified approach
for accelerating deep unsupervised domain adaptation models. TCP method is a generic, accurate, andefficient compression method that can be easily implemented by most deep learning libraries.
(2) TCP method is able to reduce negative transfer byconsidering the cross-domain distribution discrepancy
using the proposed transfer channel evaluation module.
(3) Extensive experiments on two public UDA datasets
demonstrate the significant superiority of our TCP
method.

2 Related work
Our work is mainly related to unsupervised domain adaptation and network compression.

International Journal of Machine Learning and Cybernetics

2.1 Unsupervised domain adaptation
Unsupervised domain adaptation (UDA) is a specific areaof transfer learning [32, 45], which is to learn a discriminative model in the presence of the domain-shifts between
domains. The main problem of UDA is how to reduce thedomain shift between the source and target domains. There
are many methods to tackle this problem: traditional (shallow) learning and deep learning.
Traditional (shallow) learning methods have several
aspects: (1) Subspace learning. Subspace alignment (SA)[7] aligns the base vectors of both domains and subspace
distribution alignment (SDA) [40] extends SA by adding
the subspace variance adaptation. Gong et al. proposed
the Geodesic Flow Kernel (GFK) [10] to sample indefinite
points along the geodesic flow between domains. CORAL[39] aligns subspaces in second-order statistics. (2) Distribution alignment. Pan et al. proposed the transfer component
analysis (TCA) method to align the marginal distributions
between domains. Based on TCA, joint distribution adaptation (JDA) [24] is proposed to match both marginal andconditional distributions. Later works extend JDA by adding regularization, structural consistency [17] and domain
invariant clustering [42]. But these works treat the two distributions equally and fail to leverage the different importanceof distributions. Recently, Wang et al. proposed the manifold
embedded distribution alignment (MEDA) [45, 46] approachto dynamically evaluate the different effect of marginal andconditional distributions and achieved the state-of-the-art
results on domain adaptation.
As for deep learning methods, CNN can learn nonlinear
deep representations and capture underlying factors of variation between different tasks [1]. These deep representations
can disentangle the factors of variation, which enables thetransfer of knowledge between tasks.
Recent works on deep UDA approaches can be mainly
summarized into two cases [34]. The first case is the discrepancy-based deep UDA approach, which assumes that finetuning the deep network model with labeled or unlabeled target data can diminish the shift between the two domains. The
most commonly used methods for aligning the distribution
shift between the source and target domains are maximum
mean discrepancy (MMD), correlation alignment (CORAL)[41], Kullback-Leibler (KL) divergence [50], among others.
The second case can be referred to as the adversarial-based
approach. In this case, a domain discriminator that classifies whether a data point is drawn from the source or target domain is used to encourage domain confusion throughan adversarial objective to minimize the distance between
the source and target distributions [8]. Adversarial learning has been explored in generative adversarial networks(GANs) [11]. In addition, generative multi adversarial network (GMAN) [6] extends GANs to multiple discriminators

including formidable adversary and forgiving teacher, whichsignificantly eases model training.
Recent works on deep UDA embed domain-adaptation
modules into deep networks to improve transfer performance[9], where significant performance gains have been obtained.
UDA has wide applications in computer vision [16, 23] andnatural language processing and is receiving increasing
attention from researchers. In this paper, we concentrate on
accelerating both the discrepancy-based and adversarialbased deep UDA approaches.
As far as we know, no previous UDA approach has
focused on the acceleration of the network.

2.2 Network compression
These years, for better accuracy, designing deeper and wider
CNN models has become a general trend, such as VGGNet[38] and ResNet [14]. However, as the CNN grow bigger,
it is harder to deploy these deep models on resource constrained devices. Network compression becomes an efficient
way to solve this problem. Network compression methods
mainly include network quantization, low-rank approximation and weight pruning. Network quantization is good atdecreasing the presentation precision of parameters so as toreduce the storage space. Low-rank approximation reduces
the storage space by low-rank matrix techniques, which is
not efficient for point-wise convolution [3]. Weight pruning
mainly includes two methods, neural pruning [13, 22] andchannel pruning [15, 27, 28].
Channel pruning methods prune the whole channel eachtime, therefore it is fast and efficient than neural pruning
which removes a single neuron connection each time. It isa structured pruning method, compared to network quantization and low-rank approximation, it does not introduce
sparsity to the original network structure and also does not
require special software or hardware implementations. It
has demonstrated superior performance compared to othermethods and many works [15, 27, 28] have been proposedto perform channel pruning on pre-trained models with different kinds of criteria.
These above pruning methods mainly aim at supervised
learning problems, by contrast, there have been few studies
for compressing unsupervised domain adaptation models. As
far as we know, we are the first to study how to do channel
pruning for deep unsupervised domain adaptation.
The TCP method is primarily motivated by [28], while
our work is different from it. TCP method is presented forpruning unsupervised domain adaptation models. To be
more specific, we take the discrepancy between the source
and target domains into consideration, therefore we canprune the less important channels not just for the source
domain but also for the unlabeled target domain. We call13

International Journal of Machine Learning and Cybernetics

this Transfer Channel Evaluation, which is highlighted inyellow in Fig. 1.

3 Transfer channel pruning
In this section, we introduce the proposed Transfer Channel
Pruning (TCP) method.

3.1 Problem definition
Definition 1 In unsupervised domain adaptation, we arensgiven a source domain Ds = {(𝐱is , ysi )}i=1
of ns labeled examt nt
ples and a target domain Dt = {𝐱j }j=1 of nt unlabeled examples. Ds and Dt have the same label space, i.e. The marginal
distributions between two domains are different, i.e.
Ps (𝐱s ) ≠ Pt (𝐱t ). The goal of deep UDA is to design a deepneural network that enables learning of transfer classifiers
y = fs (𝐱) and y = ft (𝐱) to close the source-target discrepancy and can achieve the best performance on the target
dataset.
For a pre-trained deep UDA model, its parameters can bedenoted as 𝐖. Here we assume the lth convolutional layer has
an output activation tensor 𝐚l of size of hl × wl × kl , where kl
represents the number of output channels of the lth layer, andhl and wl stand for the height and width of feature maps ofthe lth layer, respectively. Therefore, the goal of TCP method
is to prune a UDA model in order to accelerate it with comparable or even better performance on the target domain.
In this way, we can obtain smaller models that require lesscomputation complexity and memory consumption, which
can be deployed on resource constrained devices.

3.2 Motivation
We compress the deep UDA model using model pruning
methods for their efficiency. A straightforward model pruning technique is a two-stage method, which first prunes themodel on the source domain with supervised learning andthen fine-tune the model on the target domain.
It is worth noting that there is not prior work to compress UDA models, as far as we know, we are the first to
study how to do channel pruning for UDA models and thesplitting two-stage method is probably the easiest method tothink of and be applied to compress UDA models. Because
existing channel pruning methods are usually for compressing supervised learning models, thus it is natural to thinkthat channel pruning method can be applied to the source
domain first with supervised learning and then fine-tune forthe unlabelled target domain with the pruned model by UDA
methods. However, negative transfer [32] is likely to happen

13

during this pruning process since the discrepancy between
the source and target domains is ignored.
In this work, we propose a unified transfer channel pruning (TCP) method to tackle such challenge. TCP method iscapable of compressing the deep UDA model by pruning
less important channels while simultaneously learning transferable features by reducing the cross-domain distribution
divergence. Therefore, TCP method reduces the impact ofnegative transfer and maintains competitive performance on
the target task. In short, TCP method is a generic, accurate,and efficient compression method that can be easily implemented by most deep learning libraries.
To be more specific, Fig. 1 illustrates the main idea of
our TCP method. There are mainly three steps. Firstly,
TCP method learns the base deep UDA model throughbase model building. The base model is fine-tuned with thestandard UDA criteria. Secondly, TCP method evaluates
the importance of channels of all layers with the transfer
channel evaluation and performs further fine-tuning. Specifically, the convolutional layers, which usually dominate thecomputation complexity, are pruned in this step. Thirdly,
TCP method iteratively refines the pruning results and stops
after reaching the trade-off between accuracy and FLOPs(i.e. computational cost) or parameter size.

3.3 Base model building
In this step, we build the base UDA models with deep neuralnetworks.
3.3.1 Deep adaptation networks
Deep neural networks have been successfully used in UDA
with state-of-the-art algorithms [9, 23, 43] in recent years.
Previous studies [43, 48] have shown that the features
extracted by deep networks are general at lower layers, while
specific at the higher layers since they are more task-specific.
Therefore, more transferable representations can be learned
by transferring the features at lower layers and then fine-tune
the task-specific layers. During fine-tuning, the cross-domain
discrepancy can be reduced by certain adaptation distance.
Since our main contribution is not designing new deep UDA
networks, we build the base model like most discrepancybased deep UDA approaches [23, 41, 44]. In the following,
we will briefly introduce the main idea of the base model,
and more details can be found in the original paper of these
discrepancy-based deep UDA approaches.
As shown in Fig. 2, we learn transferable features via
several convolutional and pooling layers (the blue and purple blocks). Then, the classification task can be accomplished with the fully-connected layers (the yellow blocks).
MMD (maximum mean discrepancy) [31] is adopted as theadaptation loss in order to reduce domain shift. MMD has

International Journal of Machine Learning and Cybernetics
Fig. 2  The basic architecture ofdeep adaptation networks

been proposed to provide the distribution difference between
the source and target datasets. In addition, it has been widely
utilized in many UDA methods [23, 25, 30]. The MMD loss
between two domains can be computed as� 1 ∑ 𝜙(𝐱 ) − 1 ∑ 𝜙(𝐱 )�2ij � ,Lmmd = �nt
� ns 𝐱i ∈Ds�𝐱j ∈Dt��H(1)

where H denotes RKHS (Reproducing Kernel Hilbert
Space) and 𝜙(⋅) denotes some feature map to map the original samples to RKHS.
3.3.2 Domain‑adversarial neural networks
In this case, a domain discriminator that classifies whether
a data point is drawn from the source or target domain isused to encourage domain confusion through an adversarial objective to minimize the distance between the source
and target distributions [8]. Domain adversarial adaptation
methods borrow the idea of GAN to help learn transferable
features.
As shown in Fig. 3, we learn transferable features via
domain adversarial learning. Our basic domain-adversarial
neural network consists of three parts: the feature extractor Gf (the blue blocks), the label classifier Gy (the green
blocks) and the domain discriminator Gd (the orange blocks).
The adversarial learning procedure is a two-player game,where the first player is the domain discriminator Gd trained
to distinguish the source domain from the target domain,
and the second player is the feature extractor Gf that tries
to confuse the domain discriminator by extracting domaininvariant features. The two players are trained adversarially: the parameters 𝜃f of feature extractor Gf are learned by
maximizing the loss of domain discriminator Gd , while theparameters 𝜃d of Gd are trained by minimizing the loss ofthe domain discriminator. In addition, the loss of the label
classifier Gy is also minimized. The loss function can beformalized as:() 1 ∑ ( ( ( )) )L G G 𝐱 , yi
L 𝜃f , 𝜃y , 𝜃d =ns 𝐱 ∈D y y f iis∑( ( ( )) )𝜆−Ld Gd Gf 𝐱i , di ,ns + nt𝐱i ∈(Ds ∪Dt )(2)

where 𝜆 is a trade-off parameter. Gy and Ly are the label classifier and its loss, Gd and Ld are the domain classifier and its
loss, and di is the domain label of the input sample 𝐱i.
Several popular architectures can serve as the backbone
network of the base model, such as AlexNet [21], VGGNet[38], and ResNet [14]. After obtaining the base model, we
can perform channel pruning to accelerate the model.

3.4 Transfer channel evaluation
The goal of transfer channel evaluation is to iteratively evaluate the importance of output channels of layers in orderto prune the K least important channels. Here K is controlled by users. In the pruning process, we want to preserveand refine a set of parameters 𝐖′ , which represents thoseimportant parameters for both source and target domains.
Let L(Ds , Dt , 𝐖) be the cost function for UDA and 𝐖� = 𝐖
at the starting time. For a better set of parameters 𝐖′ , wewant to minimize the loss change after pruning a channel 𝐚l,i.
This can be considered as an optimization problem. Here we
introduce the absolute difference of loss:
Label Classifier
Feature Extractor

Softmax

Domain Discriminator
GRL

Fig. 3  The basic architecture of deep domain-adversarial neural networks

13International Journal of Machine Learning and Cybernetics(3)
which means the loss change after pruning the ith channel ofthe lth convolutional layer. In addition, we want to minimize|𝛥L(𝐚l,i )| by selecting the appropriate channel 𝐚l,i . Pruning
will stop until a trade-off between accuracy and pruning
object (FLOPs or parameter size) has been achieved.
However, it is hard to find a set of optimal parameters in
one go, because the search space is 2|𝐖| which is too huge tocompute and try every combination. Inspired by [28], ourTCP method solves this problem with a greedy algorithm
by iteratively removing the K least important channels at
each time.

|𝛥L(𝐚l,i )| = |L(Ds , Dt , 𝐚l,i ) − L(Ds , Dt , 𝐚l,i = 0)|,

Criteria is the criterion for judging the importance of channels. Since the key to channel pruning is to select the least
important channel, especially for UDA, we design the criteria of TCP method carefully. There are many heuristic criteria, including the L2-norm of filter weights, the activation
statistics of feature maps, mutual information between activations and predictions and Taylor expansion, etc. Here wechoose the first-order Taylor expansion as the base criteria
since its efficiency and performance has been verified in[28] for pruning supervised learning models. Compared with
our TCP method, we also take pruning as an optimization
problem, however, the objective we want to optimize is the
final performance on the unlabeled target dataset. Therefore
we design our criteria in a different way which is better for
pruning deep UDA models.
According to Taylor’s theorem, the Taylor expansion atpoint x = a can be computed as:P∑
f (p) (a)
p=0

p!

(4)

(x − a)p + Rp (x),where p denotes the pth derivative of f(x) at point x = a andthe last item Rp (x) represents the pth remainder. To approximate |𝛥L(𝐚l,i )|, we can use the first-order Taylor expansion
near 𝐚l,i = 0 which means the loss change after removing
𝐚l,i , then we can get:f (𝐚l,i = 0) = f (𝐚l,i ) − f � (𝐚l,i ) ⋅ 𝐚l,i +|𝐚l,i |22

⋅ f �� (𝜉),

(5)

|𝐚 |2

where 𝜉 is a value between 0 and 𝐚l,i , and l,i2 ⋅ f �� (𝜉) is a
Lagrange form remainder which requires too much computation, therefore we abandon this item for accelerating thepruning process. Then back to Eq. (3), we can get:L(Ds , Dt , 𝐚l,i = 0) = L(Ds , Dt , 𝐚l,i ) −

13

|
| 𝜕L
|
|
⋅ 𝐚l,i |,G(𝐚l,i ) = |𝛥L(𝐚l,i )| = ||| 𝜕𝐚l,i||(7)

which means the absolute value of product of the activation and the gradient of the cost function, and 𝐚l,i can becalculated as:𝐚l,i =

hl w lN1 ∑ 1 ∑ ∑ p,q𝐚 ,N n=1 hl × wl p=1 q=1 l,i(8)where N is the number of batch size.

3.4.1 Criteria

f (x) =

Then, we combine Eqs. (3) and (6) and get the criteria G of
TCP method:𝜕L⋅𝐚 .
𝜕𝐚l,i l,i(6)3.4.2 Loss function of TCP method
To make our TCP method focus on pruning UDA models,
we simultaneously take the source domain and the unlabeled target domain into consideration. The loss function of TCP method consists of two parts, Lcls (Ds , 𝐖) andLmmd (Ds , Dt , 𝐖). Here, Lcls (Ds , 𝐖) is a cross-entropy loss
which denotes the classification loss on source domain andcan be computed as:N

Lcls (Ds , 𝐖) = −C

1 ∑∑P log(hc (xis ))N i=1 c=1 i,c(9)where C is the number of classes of source dataset, Pi,c is theprobability of xis belonging to class c, and hc (xis ) denotes theprobability that the model predicts xis as class c. In addition,
Lmmd (Ds , Dt , 𝐖) denotes the MMD loss between the source
and target domains that presented in Eq. (1). The total loss
function can be computed as:

L(Ds , Dt , 𝐖) = Lcls (Ds , 𝐖) + 𝛽Lmmd (Ds , Dt , 𝐖),
where

𝛽 =

4
i

1 + e−1⋅ ITER

− 2.

(10)

(11)

Here, 𝛽 is a dynamic value which takes values in (0, 1).
i ∈ (0, ITER) where ITER is the number of pruning iterations. We design 𝛽 in this way for two main reasons, on the
one hand, during the early stage of pruning, the weights
have not converged and keep unstable therefore the Lmmd is
too large and makes the pruned model hard to converge. On
the other hand, in the rest of the pruning process, the Lmmd
becomes more important that can guide the pruned model
to focus more on the target domain. Therefore the criteria of
TCP method can be computed as:International Journal of Machine Learning and Cybernetics| 𝜕L (D , 𝐖)|
⋅ 𝐚sl,iG(𝐚l,i ) =| cls ss
|
𝜕𝐚
l,i
|
𝜕L (D , D , 𝐖) t ||+𝛽 mmd st t
⋅ 𝐚l,i |,
|
𝜕𝐚l,i
|

4 Experimental analysis(12)

where 𝐚sl,i and 𝐚tl,i denote the activation with source data andtarget data respectively.

3.5 Iterative refinement
After the transfer channel evaluation, each channel is sorted
according to G(𝐚l,i ) and the K least important channels are
removed after each pruning iteration. Then, a short-termfine-tuning is adopted to the pruned model for 5 epochs to
help the model to converge and the pruning is done after a
trade-off between accuracy and FLOPs or parameter sizehas been achieved. This step is done iteratively to prune the
network.
The learning procedure of TCP method is described in
Algorithm 1. Specifically, for a given UDA baseline 𝐖 ,firstly fine-tune the UDA baseline until the best performance achieved on the target dataset. Then, sort the channels according to their importance which is measured by
criteria Eq. (12) and remove the K least important channels.
Next, perform short-term fine-tuning to the pruned modelfor 5 epochs to help the model to converge. These steps aredone iteratively to prune the network. Finally, a long-termfine-tuning is adopted to increase the performance of the
final pruned model 𝐖′.

In this section, we evaluate the performance of TCP method
via experiments on pruning deep unsupervised domain adaptation models. We evaluate our approaches for VGGNet [38]and ResNet [14] on two popular datasets—Office-31 [37]and ImageCLEF-DA.1 All our methods are implemented
based on the PyTorch [33] framework and the code will be
released soon at [47].

4.1 Datasets
4.1.1 Office‑31
This dataset is a standard and maybe the most popularbenchmark for unsupervised domain adaptation. It consistsof 4110 images within 31 categories collected from everyday objects in an office environment. It consists of threedomains: Amazon (A), which contains images downloaded
from https​://www.amazo​n.com/, Webcam (W) and DSLR(D), which contain images respectively taken by web cameraand digital SLR camera under different settings. We evaluate all our methods across six transfer tasks on all the threedomains A→ W, W→ A, A→ D, D→ A, D→ W and W→ D.
4.1.2 ImageCLEF‑DA
This dataset is a benchmark dataset for ImageCLEF 2014
domain adaptation challenge, and it is collected by selecting the 12 common categories shared by the following

Algorithm 1 TCP: Transfer Channel Pruningt ntsInput: Source domain Ds = {(xsi , yis )}ni=1 , target domain Dt = {xj }j=1 , the baselineW.
Output: A pruned model W for deep unsupervised domain adaptation.
1: Fine-tune the unsupervised domain adaptation baseline until the best performanceachieved on the unlabeled target dataset;
2: for iteration i do
3:
Sort the importance of channels by criteria Eq. (12) and identify less significant
channels;
4:
Remove the K least important channels of the layers;
5:
Short-term fine-tune;
6:
if the trade-off between accuracy and FLOPs or parameter size has achieved
then
7:
break
8:
end if
9: end for
10: Long-term fine-tune;
11: return pruned model W .

1

http://image​clef.org/2014/adapt​ation​

13

International Journal of Machine Learning and Cybernetics

Fig. 4  The sensitivity analysis of the effect of K

public datasets and each of them is considered as a domain:
Caltech − 256 (C), ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P) and Bing (B). There are 50 images ineach category and 600 images in each domain. We evaluate all methods across six transfer tasks following existing
work [8, 23, 46]: I→ P, P→ I, I→ C, C→ I, P→ C and C→ P.
Compared with Office-31, this dataset is more balanced andcan be a good comparable dataset to Office-31.

4.2 Implementation details
We mainly compare three methods: (1) Two_stage: which is
the most straightforward method that applies channel pruning
to the source domain task first, then fine-tune for the target
domain task with the pruned model. (2) TCP_w/o_DA: Our
TCP method without the MMD loss, here we call it domain
adaptation (DA) loss. Which also means 𝛽 = 0 all the timein Eq. (10). (3) TCP: Our full TCP method with DA loss.
We evaluate all the methods on two popular backbone
networks: VGG16 [38] and ResNet50 [14]. As baselines,
VGG16-based and ResNet50-based are the original models
that are not pruned. As for VGG16-based model, it has 13
convolutional layers and 3 fully-connected layers. We prune
all the convolutional layers and the first fully-connected layer
and we only use the activations of the second fully-connected layer as image representation and build the MMD
loss which is shown in Fig. 2. In addition, as for ResNet50based model, we use similar settings as VGG16-based modelwith a few differences. Because of the shortcut and residual
branch structure, we only prune the inside convolutional layers of each bottleneck block. The MMD loss is built withthe only fully-connected layer. Moreover, we also take the
Batch Normalization (BN) [20] layers into consideration andreconstruct the whole model during pruning.
In practice, all the input images are cropped to a fixed size
224 × 224 and randomly sampled from the resized image
with horizontal flip and mean-std normalization. At first, we
fine-tune all the UDA models on each unsupervised domain

13

adaptation tasks for 200 epochs with learning rate from 0.01
to 0.0001 and the batch size = 32. During pruning, we set
K = 64 which means 64 channels will be removed after eachpruning iteration. Here we set K = 64 because we want tospeed up the pruning process while maintaining high accuracy. In fact, this is a tradeoff between accuracy and pruning
time. We analyze the effect of K value on the accuracy andpruning time of the model and the result is shown in Fig. 4.
As can be seen, if we set K to a smaller value, we mayachieve better performance on the target domain, but we alsoneed more time to do the pruning and fine-tuning procedure
to get the same FLOPs reduction. In addition, if we set K toa larger value, the pruning process can be accelerated while
the accuracy on the target domain can not be guaranteed.
After that, extra 5 epochs are adopted to help the pruned
model to converge. In addition, we follow [18, 26] to prune
the baseline with different compression rate. The VGG16based baseline is pruned with 26% and 70% FLOPs reduced
while the ResNet50-based baseline is pruned with 12% and
46% FLOPs reduced. ResNet50 has lower compression ratesince the bottleneck structure stops some layers from being
pruned. As for the VGG16-based and ResNet50-based baselines, they are the original well fine-tuned UDA models without pruning. To be more specific, as for the VGG16-based
baseline, we build the UDA model as Fig. 2. The backbone
netowrk is VGG16 and MMD loss is built on the second
fully-connected layer to compare and align the distribution
between source and target domains. Then we fine-tune the
UDA model for 200 epochs with learning rate from 0.01 to
0.0001 and batch size = 32. In addition, as for the ResNet50based baseline, the structure is similar to the VGG16-based
baseline, we use ResNet50 as the backbone network and
MMD loss is built on the only fully-connected layer.
We follow standard evaluation protocol for UDA and use
all source examples with labels and all target examples without labels [10]. The labels for the target domain are onlyused for evaluation. We adopt classification accuracy on
the target domain and parameter reduction as the evaluation metrics: higher accuracy and fewer parameters indicate
better performance.

4.3 Results and analysis
Discrepancy-based UDA model pruning Firstly, we evaluate our TCP method on the discrepancy-based base model.
we first evaluate all the tasks on Office-31 dataset. The
results are shown in Table 1. As can be seen, the full TCP
method outperforms other methods in accuracy and F1score under the same compression rate (FLOPs reduction)and can reduce more parameters, and the bold indicates thebest results under the same compression rate. It is important and interesting that TCP method achieves even better
performance than the baseline model (which is not pruned).

46%

12%

70%

26%

32.4%
33.5%
37.7%
56.2%
56.4%
58.4%

96.7
97.0
98.2
93.0
95.8
96.3

0.734
0.783
0.801
0.622
0.732
0.754

0.968
0.974
0.983
0.928
0.969
0.970

31.4%
35.5%
36.2%
56.3%
56.4%
58.0%

31.2%
32.7%
36.2%
68.0%
61.3%
69.8%
–

75.8
79.8
81.8
65.5
75.1
77.4

0.949
0.964
0.965
0.869
0.894
0.896
0.969

94.5
95.5
96.1
88.1
89.2
89.5
97.1

0.684
0.714
0.757
0.563
0.528
0.651
0.790

69.4
73.0
76.1
57.1
53.5
74.1
80.3

29.4%
29.7%
36.8%
63.3%
62.5%
69.3%
–

W→D

99.5
100
99.8
98.7
99.2
100

99.0
99.3
99.8
96.3
97.9
98.8
99.2
0.997
1.000
0999
0.991
0.992
1.000

0.993
0.996
0.999
0.966
0.975
0.986
0.993

A→D

35.5%
34.5%
37.0%
57.3%
56.6%
57.1%

30.6%
29.2%
32.6%
64.5%
57.5%
65.2%
–

0.694
0.758
0.764
0.530
0.617
0.652
0.801

29.3%
25.8%
35.9%
61.0%
54.8%
66.2%
–

76.0 0.776 30.4%
77.1 0.796 36.2%
77.9 0.785 36.9%
64.9 0.642 56.0%
70.8 0.705 55.8%
72.0 0.716 59.0%

69.8
75.8
76.2
55.0
61.8
65.9
78.9

D→A

0.392
0.453
0.483
0.304
0.341
0.345
0.639

32.5%
29.3%
37.1%
66.2%
62.6%
68.5%
–
48.0 0.461 28.3%
47.8 0.471 34.5%
50.0 0.494 35.0%
34.0 0.338 57.2%
34.2 0.343 56.4%
36.1 0.363 57.8%

42.7
45.9
47.9
31.8
35.3
35.6
64.3

W→A

0.429
0.482
0.512
0.314
0.323
0.365
0.603

28.3%
29.6%
39.8%
63.1%
58.8%
68.2%
–
50.1 0.482 29.4%
52.6 0.511 33.1%
55.5 0.531 36.9%
38.9 0.373 57.3%
41.5 0.376 56.6%
46.3 0.443 58.5%

47.2
50.4
51.2
32.7
34.5
38.5
62.3

Average
Param↓

0.576
0.728
0.748
0.497
0.613
0.649
0.667

30.2%
29.4%
36.4%
64.3%
59.6%
67.9%
–
74.4 0.736 31.2%
75.7 0.755 34.5%
77.2 0.766 36.7%
65.8 0.649 56.7%
69.4 0.686 56.4%
71.3 0.708 58.1%

70.4
73.3
74.5
60.2
62.0
67.1
80.3

74.5 0.740 –Param↓ Acc F1

55.2 0.541 –Param↓ Acc F1

54.1 0.528 –Param↓ Acc F1

72.3 0.734 –Param↓ Acc F1

97.5 0.976 –Param↓ Acc F1

94.0 0.949 –Param↓ Acc F1

D→W

74.0 0.714 –Acc F1FLOPs↓ A→W

The bold values denote the best performance under the same compression rate of different tasks

Here, FLOPs ↓ and Param ↓ denote the decrement of FLOPs and parameter size compared with the baseline, F1 and Acc are the F1 score and accuracy on the target domain. The base model isa discrepancy-based model shown in Fig. 2

VGG16-based (baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP
ResNet50-based(baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP

ModelsTable 1  The performance on Office-31 dataset (VGG16-based and ResNet50-based)

International Journal of Machine Learning and Cybernetics

13

13

46%

81.3
80.5
82.6
71.8
73.0
77.5

0.807
0.788
0.818
0.691
0.719
0.765

31.5%
34.6%
36.5%
56.5%
57.4%
58.0%

29.9%
34.2%
32.2%
62.9%
65.7%
66.0%

92.1
92.0
92.5
85.2
85.5
88.6

88.5
89.0
90.5
80.6
80.5
81.8
92.3
0.915
0.922
0.919
0.841
0.848
0.881

0.878
0.885
0.893
0.798
0.798
0.808
0.930

88.5 0.879

Param↓ Acc F1

I→C

C→I

34.4%
33.8%
35.5%
54.2%
55.4%
56.2%

27.5%
33.1%
35.1%
64.3%
66.5%
65.7%

76.5
76.1
80.8
68.2
67.7
71.6

64.5
77.1
77.8
57.7
55.1
59.8
83.3
0.729
0.729
0.788
0.645
0.640
0.692

0.605
0.771
0.775
0.544
0.516
0.579
0.825

77.0 0.756

Param↓ Acc F1

The bold values denote the best performance under the same compression rate of different tasks

29.4%
33.5%
37.5%
53.2%
55.4%
57.2%

0.706
0.715
0.750
0.638
0.651
0.657

71.8
73.0
75.0
65.6
66.6
67.8

12%

70%

0.765
0.789
0.837
0.679
0.673
0.680
0.863

77.7
79.5
80.5
69.2
69.1
69.5
82.2

0.669
0.689
0.720
0.564
0.598
0.605
0.749

68.0
70.5
72.0
58.6
61.0
61.9
74.8

29.0%
33.6%
39.0%
65.8%
55.4%
66.7%

80.0 0.791

71.3 0.709

Param↓ Acc F1

P→I

26%

Acc F1FLOPs↓ I→P

The base model is a discrepancy-based model

VGG16-based (baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP
ResNet50-based(baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP

ModelsTable 2  The performance on ImageCLEF-DA dataset (VGG16-based and ResNet50-based)

32.4%
31.2%
36.7%
54.0%
55.5%
58.5%

29.0%
34.5%
36.3%
57.0%
63.3%
68.7%

64.0
64.3
66.2
57.4
55.5
57.7

56.0
62.2
64.8
43.2
47.0
49.7
70.0
0.621
0.621
0.640
0.541
0.529
0.555

0.536
0.608
0.646
0.401
0.457
0.491
0.721

61.1 0.587

Param↓ Acc F1

C→P

29.2%
30.1%
36.6%
51.5%
53.6%
55.7%

29.3%
35.0%
36.6%
67.8%
66.5%
68.8%

0.822
0.842
0.893
0.743
0.705
0.753
0.924

Average

26.9%
33.1%
34.5%
61.5%
65.8%
67.2%

0.712
0.764
0.794
0.621
0.624
0.653
0.835

28.6%
33.9%
35.6%
63.2%
63.9%
67.1%

Param↓

78.2 0.769 31.2%
78.7 0.771 33.2%
80.6 0.795 36.7%
71.1 0.688 53.9%
70.8 0.692 55.7%
73.8 0.723 57.3%

73.0
77.2
78.9
63.9
63.9
66.4
82.1

77.5 0.766

Param↓ Acc F1

84.0 0.835 30.8%
86.3 0.853 36.3%
86.5 0.857 37.6%
78.1 0.775 54.0%
77.0 0.768 57.1%
79.5 0.786 58.2%

83.3
85.1
87.5
74.5
70.9
75.9
89.8

87.2 0.875

Param↓ Acc F1

P→CInternational Journal of Machine Learning and Cybernetics

46%

35.1%
36.3%
40.5%
59.7%
60.1%
62.3%

96.4
96.7
98.1
90.6
93.2
93.9

0.782
0.809
0.846
0.669
0.773
0.810

77.3
81.2
84.1
67.4
78.1
80.5

12%

70%

0.960
0.957
0.985
0.897
0.916
0.931

0.951
0.957
0.976
0.880
0.905
0.925
0.966

95.3
96.2
97.5
89.1
89.6
91.8
96.9

0.704
0.761
0.765
0.578
0.559
0.734
0.815

71.2
75.8
77.6
59.8
56.2
75.7
82.0

30.2%
30.8%
38.7%
65.6%
65.2%
71.1%

94.8 0.945

76.8 0.750

Param↓ Acc F1

D→W

26%

Acc F1FLOPs↓ A→W

34.5%
37.3%
38.2%
59.3%
59.7%
61.8%

31.9%
33.5%
37.9%
70.0%
63.9%
71.9%

99.6
99.9
99.8
98.9
99.6
100

99.1
99.5
100
97.7
98.6
99.5
99.1
0.980
0.997
0.995
0.979
0.995
1.000

0.990
0.994
1.0
0.957
0.981
0.993
0.978

98.3 0.979

Param↓ Acc F1

W→D

37.8%
36.9%
39.2%
60.7%
59.4%
60.3%

31.7%
30.5%
33.9%
66.7%
59.3%
67.3%

77.1
78.3
78.8
65.6
71.1
73.0

71.5
76.3
77.8
57.9
63.7
71.3
79.7
0.758
0.776
0.793
0.644
0.709
0.728

0.708
0.716
0.795
0.572
0.658
0.721
0.783

73.8 0.743

Param↓ Acc F1

A→D

33.4%
37.2%
38.1%
57.3%
56.5%
61.0%

30.3%
26.8%
36.7%
63.3%
56.8%
67.9%

49.3
48.2
51.4
35.5
36.1
37.8

44.3
47.5
54.4
33.5
37.2
38.5
68.2
0.503
0.478
0.510
0.340
0.355
0.369

0.433
0.480
0.528
0.310
0.364
0.375
0.680

56.2 0.566

Param↓ Acc F1

D→A

29.8%
35.7%
36.8%
58.9%
58.1%
59.8%

33.6%
29.9%
38.2%
68.6%
64.4%
69.8%

51.8
53.7
56.9
39.7
43.1
47.7

49.7
52.5
53.6
33.7
36.5
38.4
67.4

Average

29.6%
30.3%
40.8%
65.6%
61.5%
70.3%

The bold values denote the best performance under the same compression rate of different tasks

The base model is an adversarial-based model shown in Fig. 3

0.713
0.735
0.765
0.603
0.636
0.685
0.812

31.7%
29.8%
37.2%
65.9%
62.1%
69.7%

Param↓

75.6 0.749 32.9%
76.7 0.758 35.5%
78.8 0.782 37.6%
66.7 0.655 58.3%
70.3 0.696 58.1%
72.8 0.716 60.7%

71.9
74.2
76.8
62.2
64.3
69.2
82.2

76.4 0.757

Param↓ Acc F1

0.509 30.3%
0.532 34.6%
0.562 37.2%
0.399 59.0%
0.429 58.3%
0.458 60.2%

0.492
0.507
0.528
0.322
0.351
0.364
0.650

57.4 0.564

Param↓ Acc F1

W→A

Here, FLOPs ↓ and Param ↓ denote the decrement of FLOPs and parameter size compared with the baseline, F1 and Acc are the F1 score and accuracy on the target domain

VGG16-based (baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP
ResNet50-based(baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP

ModelsTable 3  The performance on Office-31 dataset (VGG16-based and ResNet50-based)

International Journal of Machine Learning and Cybernetics

13

13

46%

85.8
84.8
87.1
73.6
75.2
79.3

0.850
0.837
0.863
0.721
0.743
0.791

33.5%
36.0%
38.2%
57.3%
58.4%
59.3%

31.3%
35.8%
33.5%
64.8%
67.0%
67.8%

96.3
95.9
96.9
89.0
89.4
91.8

90.1
91.5
92.9
81.8
81.6
82.9
96.2
0.962
0.947
0.972
0.864
0.882
0.909

0.887
0.910
0.927
0.805
0.807
0.825
0.955

90.2 0.897

Param↓ Acc F1

I→C

C→I

36.6%
35.4%
37.8%
56.5%
57.4%
58.6%

29.5%
35.1%
37.3%
66.5%
68.0%
67.7%

79.8
79.0
84.5
70.0
69.6
72.8

66.6
80.2
82.5
59.5
57.7
61.5
87.0
0.786
0.793
0.855
0.683
0.686
0.719

0.647
0.788
0.819
0.587
0.568
0.614
0.862

79.5 0.802

Param↓ Acc F1

The bold values denote the best performance under the same compression rate of different tasks

30.8%
34.9%
38.8%
54.8%
56.5%
58.3%

0.717
0.722
0.745
0.653
0.667
0.682

72.1
73.2
75.8
66.7
67.5
68.6

12%

70%

0.778
0.801
0.813
0.719
0.708
0.717
0.844

79.6
80.8
82.3
72.2
71.9
72.8
86.0

0.695
0.718
0.738
0.580
0.602
0.607
0.741

70.2
72.4
74.7
58.6
61.0
61.9
75.0

30.0%
34.6%
39.9%
66.7
56.4%
67.9%

81.9 0.825

73.0 0.727

Param↓ Acc F1

P→I

26%

Acc F1FLOPs↓ I→P

The base model is an adversarial-based model

VGG16-based (baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP
ResNet50-based(baseline)
Two_stage
TCP_w/o_DA
TCP
Two_stage
TCP_w/o_DA
TCP

ModelsTable 4  The performance on ImageCLEF-DA dataset (VGG16-based and ResNet50-based)

34.4%
33.8%
38.6%
56.0%
57.3%
59.9%

31.6%
36.6%
38.0%
59.4%
65.6%
69.9%

66.9
70.3
72.6
61.5
58.2
58.9

57.8
65.2
67.3
46.0
50.1
52.3
74.3
0.653
0.692
0.722
0.605
0.575
0.582

0.558
0.653
0.672
0.455
0.491
0.513
0.731

63.6 0.626

Param↓ Acc F1

C→P

31.6%
32.3%
38.1%
53.3%
55.2%
57.2%

31.3%
36.5%
37.7%
69.5%
68.6%
70.8%

0.847
0.876
0.895
0.766
0.709
0.782
0.902

Average

27.6%
34.7%
35.6%
63.0%
67.7%
69.6%

0.735
0.791
0.811
0.652
0.647
0.676
0.839

30.2%
34.2%
36.8%
65.8%
65.9%
69.2%

Param↓

80.8 0.802 33.3%
81.2 0.809 35.6%
82.5 0.838 37.9%
73.9 0.725 54.6%
73.7 0.729 57.6%
76.0 0.754 59.4%

75.2
79.6
81.6
65.7
65.7
68.5
85.0

79.8 0.794

Param↓ Acc F1

86.3 0.844 32.0%
88.0 0.867 38.1%
88.9 0.873 39.0%
83.1 0.822 56.4%
82.3 0.819 59.0%
84.8 0.840 60.5%

86.1
88.2
89.9
76.3
71.8
79.5
91.5

89.9 0.891

Param↓ Acc F1

P→CInternational Journal of Machine Learning and Cybernetics

International Journal of Machine Learning and Cybernetics
Fig. 5  The pruning result ontask A→W with more compression rate(a) VGG16-based
This is probably because some redundant channels in thebase model are removed thus negative transfer is reduced.
Especially for the results of ResNet50-based models, ourResNet50-based baseline achieves comparable or better
performance compared with the discrepancy-based deep
UDA approaches result of DDC [44] and DAN [23] in [25].
However, we can get better performance on half of the tasks
and we even get 100% on task W→D after 46% FLOPs havebeen reduced.
Then, we evaluate our methods on ImageCLEF-DA dataset
and the results are shown in Table 2. We can draw the sameconclusion that our TCP method performs better on all tasks
on ImageCLEF-DA dataset. We get higher accuracy thanthe baseline on all the VGG16-based experiments after 26%
FLOPs have been reduced, and we also get higher accuracy
on the target dataset on half of the tasks on ResNet50-based
experiments after 12% FLOPs have been reduced, comparedwith the baseline which is almost the same as the discrepancybased deep UDA approaches results of DAN in [25].
Adversarial-based UDA model pruning As for the adversarial-based base model, we also evaluate all the transfer
learning tasks on Office-31 and ImageCLEF-DA datasets.
The results are shown in Tables 3 and 4, which indicates that
our TCP method outperforms other methods under the same
compression rate (FLOPs reduction) and can reduce moreparameters while performing pruning on the adversarial-basedmodel. It is worth noting that TCP method achieves evenbetter performance than the VGG16-based baseline model(which is not pruned). In addition, we also get higher accuracy
on the target dataset on most of the tasks on ResNet50-based
experiments after 12% FLOPs have been reduced. This isprobably because some redundant channels in the base model
are removed thus negative transfer is reduced.
Apart from Tables 1, 2, 3 and 4, Fig. 5 shows the comparison for all methods. Moreover, we also add a Random method
which randomly removes a certain number of channels toachieve the same reduction of FLOPs. Combining these results,
more conclusions can be made. (1) Compared with Two_stage,(b) ResNet50-based

TCP method is more efficient because it is a unified framework and treat the pruning as a single optimization problem,
while Two_stage is a split method and it does not take the target
domain into consideration while pruning. (2) Compared with
TCP_w/o_DA, the full TCP method uses the transfer channelevaluation to represent the discrepancy between the source andtarget domains. We try to remove those less important channels for both source and target domains and reduce negativetransfer by reducing domain discrepancy. (3) As can be seenfrom Fig. 5, our TCP method outperforms other methods on
unlabeled target dataset under different compression rate. (4)Our TCP method outperforms other methods on pruning two
mainly kinds of UDA models: discrepancy-based models andadversarial-based models. This indicates that TCP method is
generic, accurate, and efficient, which can dramatically reducethe computational cost of a deep UDA model without sacrificing the performance.

4.4 Acceleration analysis
Our proposed TCP method is capable of solving the acceleration problem of UDA models. In this section we show more

Table 5  The acceleration analysis on task A→D
ModelsVGG16-base (baseline)
TCP
TCP
ResNet50-base (baseline)
TCP
TCP

FLOPs↓
(%)
26
70
12
46

Time (ms)
0.263
0.187
0.168
1.260
0.981
0.888

Speed-up
(%)
28.90
36.10
22.10
29.50

Here, FLOPs ↓ denotes the decrement of FLOPs compared with thebaseline, time means the average processing time of one image during
inference. speed-up is the speed-up ratio compared with the baselinemodel. The base model is a discrepancy-based model shown in Fig. 2

13

International Journal of Machine Learning and Cybernetics
Table 6  The acceleration analysis on task A→D
ModelsFLOPs↓(%)

VGG16-base (baseline)
TCP
TCP
ResNet50-base (baseline)
TCP
TCP

26
70
12
46

Time (ms)
0.244
0.200
0.181
1.350
1.200
0.912

Speed-up
(%)
18.00
25.80
11.10
32.40

Here, the base model is an adversarial-based model shown in Fig. 3

details to present the acceleration degree of our TCP method
compared with the baselines. Here we perform acceleration
analysis on task A→D, and other tasks are the same. We
evaluate our TCP method on the discrepancy-based basemodel and record the inference time to calculate the average inference time and speed-up ratio. Our experiments run
on a NVIDIA TITAN XP GPU and the result is shown in
Table 5. As can be seen, as for VGG16-based baseline, it canbe accelerated by 28.90% and 36.10% after 26% and 70%
FLOPs have been reduced with TCP respectively. In addition, as for ResNet50-based baseline, it can be accelerated(a) Baseline(b) Two stage

by 11.10% and 32.40% after 12% and 46% FLOPs have been
reduced respectively. Our TCP method is able to accelerate
the discrepancy-based UDA models.
We can draw the same conclusion that TCP method iscapable of accelerating the adversarial-based UDA models,
and the results are shown in Table 6.

4.5 Visualization analysis
To evaluate the effectiveness of TCP method in reducing
negative transfer, in Fig. 6, we follow [5] to visualize themodel activations of task A→W pruned by different methods using t-SNE [5]. Fig. 6a shows the results of ResNet50based baseline without pruning on the source domain. Figure 6b–d denote the result of ResNet50-based models on
the target domain, which have been pruned by 12% FLOPs
with our three methods Two_stage, TCP_w/o_DA and TCP
respectively. The colored digits represent the ground truthof the examples, therefore the number is from 0 to 30, which
denotes target dataset has 31 categories. Here we randomly
pick 10 categories to visualize. As can be seen, the target
categories are discriminated much more clearly with themodel pruned by our TCP method. This suggests that our(c) TCP w/o DA(d) TCP

Fig. 6  The t-SNE visualization of network activations. a Is generated by ResNet50-based baseline without pruning on source domain. b–d Are
generated by ResNet50-based (with 12% FLOPs pruned) with our three methods on target domain respectively. Best view in color(a) Office31: A→W(b) ImageCLEF-DA: I→P

Fig. 7  The pruned structure of all the 13 convolutional layers of VGG16-based network on different dataset for deep unsupervised domain adaptation. Best view in color

13

International Journal of Machine Learning and Cybernetics

TCP method is effective in learning more transferable features by reducing the cross-domain divergence.
To explore if there is any pattern in the structure of thepruned models, we show the structure of pruned models
on task A→W and I→P in Fig. 7 with TCP method, anddifferent color denotes different compression rate (FLOPs
reduction). The dark blue bars represent channels of the 13
convolutional layers of the VGG16-based baseline before
pruning. In addition, the light blue, green, orange and redbars represent the channels of pruned model under different
compression rate respectively. As can be seen, higher layers
have more redundancy than lower layers in VGG16-basedmodels, and our TCP method prefer pruning the channels ofhigher layers. This is reasonable for unsupervised domain
adaptation because the lower layers usually encode many
common and important features for both source and target
domains. Moreover, with more parameters in higher layers,
our TCP method thus can prune more parameters under thesame compression rate. The same result can be observed on
ResNet50-based models.

5 Conclusion and future work
In this paper, we propose a unified Transfer Channel Pruning (TCP) method for accelerating deep unsupervised
domain adaptation models. TCP method is capable ofcompressing the deep UDA model by pruning less important channels while simultaneously learning transferable
features by reducing the cross-domain distribution divergence. Therefore, it reduces the impact of negative transfer
and maintains competitive performance on the target task.
TCP method is a generic, accurate, and efficient compression method that can be easily implemented by most deeplearning libraries. Experimental results on two main kindsof UDA methods (the discrepancy-based methods and theadversarial-based methods) demonstrate the significant
superiority of our TCP method over other methods.
In the future, we plan to apply our TCP method to heterogeneous UDA problems.
Acknowledgements This work is supported in part by National Key
Research & Development Plan of China (No. 2017YFB1002802),NSFC (No. 61572471), and Beijing Municipal Science & Technology
Commission (No. Z171100000117017).

References
1. Bengio Y, Courville A, Vincent P (2013) Representation learning:a review and new perspectives. IEEE Trans Pattern Anal Mach
Intell 35(8):1798–1828
2. Borgwardt KM, Gretton A, Rasch MJ, Kriegel HP, Schölkopf B,
Smola AJ (2006) Integrating structured biological data by kernel
maximum mean discrepancy. Bioinformatics 22(14):e49–e57

3. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. arXiv preprint pp. 1610–02357
4. Denton EL, Zaremba W, Bruna J, LeCun Y, Fergus R (2014)
Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in neural information processing
systems, pp 1269–1277
5. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature forgeneric visual recognition. In: ICML. pp 647–655
6. Durugkar I, Gemp I, Mahadevan S (2016) Generative multi-adversarial networks. arXiv preprint arXiv​:1611.01673​
7. Fernando B, Habrard A, Sebban M, Tuytelaars T (2013) Unsupervised visual domain adaptation using subspace alignment. In:
Proceedings of the IEEE international conference on computer
vision. Sydney, Australia, pp 2960–2967. http://www.iccv2​013.
org/
8. Ganin Y, Lempitsky V (2014) Unsupervised domain adaptation
by backpropagation. arXiv preprint arXiv​:1409.7495
9. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial
training of neural networks. J Mach Learn Res 17(1):2096-30
10. Gong B, Grauman K, Sha F (2013) Connecting the dots withlandmarks: discriminatively learning domain-invariant features
for unsupervised domain adaptation. In: ICML. pp 222–230
11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D,
Ozair S, Courville A, Bengio Y: Generative adversarial nets. In:
Advances in neural information processing systems. pp 2672–
2680 (2014)12. Han S, Mao H, Dally WJ (2015) Deep compression: Compressing deep neural networks with pruning, trained quantization andhuffman coding. arXiv preprint arXiv​:1510.00149​
13. Han S, Pool J, Tran J, Dally W (2015) Learning both weights andconnections for efficient neural network. In: Advances in neural
information processing systems. pp 1135–1143
14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning forimage recognition. In: Proceedings of the IEEE conference on
computer vision and pattern recognition. Las Vegas, USA, pp
770–778. http://cvpr2​016.thecv​f.com/
15. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating
very deep neural networks. In: International conference on computer vision (ICCV), vol. 2. Venice, Italy. http://iccv2​017.thecv​
f.com/
16. Hoffman J, Tzeng E, Park T, Zhu JY, Isola P, Saenko K, Efros
AA, Darrell T (2018) Cycada: cycle-consistent adversarial domain
adaptation. In: ICML
17. Hou CA, Tsai YHH, Yeh YR, Wang YCF (2016) Unsupervised
domain adaptation with label and structural consistency. IEEE
Trans Image Process 25(12):5552–5562
18. Hu Y, Sun S, Li J, Wang X, Gu Q (2018) A novel channel pruning method for deep neural network compression. arXiv preprint
arXiv​:1805.11394​
19. Huang J, Gretton A, Borgwardt K, Schölkopf B, Smola AJ (2007)
Correcting sample selection bias by unlabeled data. In: Advancesin neural information processing systems. pp 601–608
20. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep
network training by reducing internal covariate shift. In: ICML
21. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances inneural information processing systems. pp 1097–1105
22. Lin J, Rao Y, Lu J, Zhou J (2017) Runtime neural pruning.
In: Advances in neural information processing systems. pp
2181–2191
23. Long M, Cao Y, Wang J, Jordan MI (2015) Learning transferable
features with deep adaptation networks. In: ICML
24. Long M, Wang J, Ding G, Sun J, Yu PS (2013) Transfer featurelearning with joint distribution adaptation. In: Proceedings of the13International Journal of Machine Learning and Cybernetics

25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.

36.
37.

38.

IEEE international conference on computer vision. Sydney, Australia, pp 2200–2207. http://www.iccv2​013.org/
Long M, Zhu H, Wang J, Jordan MI (2017) Deep transfer learningwith joint adaptation networks. In: ICML
Luo JH, Wu J (2018) Autopruner: An end-to-end trainable filter
pruning method for efficient deep model inference. arXiv preprint
arXiv​:1805.08941​
Luo JH, Wu J, Lin W (2017) Thinet: A filter level pruning methodfor deep neural network compression. arXiv preprint arXiv​:1707.06342​
Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2017) Pruning
convolutional neural networks for resource efficient inference. In:
ICLRMotiian S, Jones Q, Iranmanesh S, Doretto G (2017) Few-shotadversarial domain adaptation. In: Advances in neural information
processing systems. pp 6670–6680
Pan SJ, Kwok JT, Yang Q (2008) Transfer learning via dimensionality reduction. AAAI 8:677–682
Pan SJ, Tsang IW, Kwok JT, Yang Q (2011) Domain adaptation via transfer component analysis. IEEE Trans Neural Netw
22(2):199–210
Pan SJ, Yang Q et al (2010) A survey on transfer learning. IEEE
Trans Knowl Data Eng 22(10):1345–1359
Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, LinZ, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch
Patel VM, Gopalan R, Li R, Chellappa R (2015) Visual domain
adaptation: a survey of recent advances. IEEE Signal Process Mag
32(3):53–69
Pei Z, Cao Z, Long M, Wang J (2018) Multi-adversarial domain
adaptation. In: Thirty-second AAAI conference on artificial intelligence. New Orleans, Louisiana, USA. https​://aaai.org/Confe​
rence​s/AAAI-18/
Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net:
imagenet classification using binary convolutional neural networks. In: ECCV. Springer, pp 525–542
Saenko K, Kulis B, Fritz M, Darrell T (2010) Adapting visual
category models to new domains. In: European conference on
computer vision. Springer, Hersonissos, Heraklion, Crete, Greece,
pp 213–226. https​://www.ics.forth​.gr/eccv2​010/intro​.php/
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR

13

39. Sun B, Feng J, Saenko K (2016) Return of frustratingly easy
domain adaptation. AAAI 6:8
40. Sun B, Saenko K (2015) Subspace distribution alignment for
unsupervised domain adaptation. In: BMVC. pp 24–1
41. Sun B, Saenko K (2016) Deep coral: Correlation alignment for
deep domain adaptation. In: European conference on computer
vision. Springer, Amsterdam, The Netherlands, pp 443–450.
http://www.eccv2​016.org/
42. Tahmoresnezhad J, Hashemi S (2017) Visual domain adaptation
via transfer feature learning. Knowl Inf Syst 50(2):585–605
43. Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial
discriminative domain adaptation. In: Proceedings of the IEEE
conference on computer vision and pattern recognition. Honolulu,
Hawaii, USA, pp 7167–7176. http://cvpr2​017.thecv​f.com/
44. Tzeng E, Hoffman J, Zhang N, Saenko K, Darrell T (2014) Deep
domain confusion: maximizing for domain invariance. arXiv preprint arXiv​:1412.3474
45. Wang J, Chen Y, Hao S, Feng W, Shen Z (2017) Balanced distribution adaptation for transfer learning. In: Data mining (ICDM),
2017 IEEE International Conference on IEEE. New Orleans,
USA, pp 1129–1134. http://icdm2​017.bigke​.org/
46. Wang J, Feng W, Chen Y, Yu H, Huang M, Yu PS (2018) Visual
domain adaptation with manifold embedded distribution alignment. In: 2018 ACM Multimedia Conference on Multimedia Conference. Seoul, Korea, pp 402–410. https​://acmmm​.org/2018/
47. Wang J, et al.: Everything about transfer learning and domain
adapation. http://trans​ferle​arnin​g.xyz
48. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural
information processing systems. pp 3320–3328
49. Zhou A, Yao A, Guo Y, Xu L, Chen Y (2017) Incremental network
quantization: Towards lossless cnns with low-precision weights.
arXiv preprint arXiv​:1702.03044​
50. Zhuang F, Cheng X, Luo P, Pan SJ, He Q (2015) Supervised representation learning: transfer learning with deep autoencoders. In:
IJCAI. pp 4119–4125
Publisher’s Note Springer Nature remains neutral with regard tojurisdictional claims in published maps and institutional affiliations.