DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

1

Domain Adaptation Regularization forSpectral Pruning
Laurent Dillard1,†

1

Behold.ai
New York, USA

2

DENSO CORPORATIONTokyo, Japan

3

The University of Tokyo
Tokyo, Japan

4

AIP-RIKENTokyo, Japan

laurent.dillard@gmail.com

Yosuke Shinya2
yosuke.shinya.j7r@jp.denso.com

Taiji Suzuki3,4
taiji@mist.i.u-tokyo.ac.jp

Abstract
Deep Neural Networks (DNNs) have recently been achieving state-of-the-art performance on a variety of computer vision related tasks. However, their computational costlimits their ability to be implemented in embedded systems with restricted resources orstrict latency constraints. Model compression has therefore been an active field of research to overcome this issue. Additionally, DNNs typically require massive amountsof labeled data to be trained. This represents a second limitation to their deployment.
Domain Adaptation (DA) addresses this issue by allowing knowledge learned on one
labeled source distribution to be transferred to a target distribution, possibly unlabeled.
In this paper, we investigate on possible improvements of compression methods in DA
setting. We focus on a compression method that was previously developed in the context
of a single data distribution and show that, with a careful choice of data to use during
compression and additional regularization terms directly related to DA objectives, it ispossible to improve compression results. We also show that our method outperforms anexisting compression method studied in the DA setting by a large margin for high compression rates. Although our work is based on one specific compression method, we also
outline some general guidelines for improving compression in DA setting.

1

Introduction

In the past years, Deep Neural Networks have attracted much attention in vision tasks. Indeed, they led to significant improvements in many problems including image classificationand object detection [21, 30]. This was mainly achieved by the development of complex
new architectures based on Convolutional Neural Networks (CNNs) [14, 34, 41]. Thesearchitectures typically involve a huge number of parameters and operations for inference,
making them ill-suited to being deployed in constrained environments. Several different
strategies have been developed in order to circumvent this issue by compressing the models
into smaller ones achieving similar performances. The main methods can be categorizedc 2020. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
†Work done as an intern at AIP-RIKEN.

2

𝑚×𝑘

𝑘×𝑛

Non
Linearity𝑚×𝑘

Non
Linearity

𝑚×𝑛

𝑛×𝑝

Uncompressed
LayersNon
Linearity

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

𝑛×𝑝

DALR

𝑘×𝑝

Ours

Figure 1: We propose a compression methodwith additional regularization terms for Domain Adaptation (DA) setting. Our method
can maintain accuracy without fine-tuning after
compression, and can achieve higher accuracy
than prior work (DALR [26]). DALR is basedon low-rank matrix decomposition, while ourmethod is built on Spectral Pruning [39].

into the following schemes: selectively pruning parameters [13, 23], distillation [1, 3, 16],
quantization [8, 46], or low-rank factorization and sparsity [17, 35]. Another main challenge
for DNNs is to remain efficient on data coming from a target distribution similar yet not
identical to the training data. A popular technique is to use a model pre-trained on a large
source dataset and fine-tune its parameters on the desired target dataset. However it may bethat no labels are available on the target data. In that case, training requires unsupervised DA
techniques [2, 7, 33].
In this work, we seek to adapt compression objectives to the specific setting of DA.
Concretely, we are provided with both a source and a, possibly unlabeled, target distribution.
We are interested in compressing DNNs with high accuracy on the target distribution. Someworks have investigated this specific setting. The work in [26] is strongly related to ours but
only focuses on the case where a model is fine-tuned on a target dataset. Their method yields
great results in this setting because it is data-dependent. That is, compression depends on the
input data distribution. In this paper, we focus on a compression method that is also datadependent. The novelty of our work is to go further in the analysis of how the data affects
compression and to add a new regularization term. This regularization term is directly related
to a DA objective. It basically favors nodes that are non domain discriminative. This proves
useful as the more similar the features extracted are, between source and target distribution,
the more discriminative ability learned on the source distribution will also apply to the target
distribution. Finally we show that our extended compression method compares favorably
to [26] on various cases of fine-tuning a pre-trained model on a target dataset.

2

Related work

Here we present a brief review of the main strategies used in model compression. One canfind in [4] a more in-depth analysis of the different methods. (1) Pruning. These methods
focus on finding weights of the network that are not crucial to the network’s performance.
Early works used magnitude-based pruning which removed weights with the lowest values [13]. This method was extended by various works yielding significant results [10, 23].
In this work we focus on a pruning method [39], which is based on spectral analysis of thecovariance matrix of layers [38, 40]. (2) Network distillation. These approaches are basedon a teacher-student learning scheme [1, 3, 16]. A compressed, small network will learn
to mimic a large network or ensemble of large networks. This is efficient because the student model learns based on a combination of true labels, which are hard encoded, and soft
outputs of the teacher model. Those contain valuable information about similarity between
classes. Distillation can also train the student network to mimic the behavior of intermediate
layers [31]. (3) Parameter quantization. Network quantization is based on finding an efficient representation of the network’s parameters by reducing the number of bits required toDILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

3

represent each weight. Various strategies can achieve this objective: [8, 46] applied k-means
scalar quantization to the weight values. Other work use a fixed (8 or 16) bit representation of the weights while maintaining the accuracy of the original model [11, 44]. In [5] a
network is constrained to use only binary weights while still achieving high accuracy. (4)Low-rank factorization. The idea behind low-rank factorization and sparse techniques is todecompose layers into smaller ones with fewer parameters. For convolutional layers, maintechniques rely on the fact that performing a convolution with the original filters can be approached by a linear combination of a set of convolutions with base filters [6, 35]. In [26]truncation of Singular Value Decomposition (SVD) is used to factorize the weight matrices
of dense (fully connected) layers.
Some works explore pruning methods in DA (or transfer learning) setting [25, 27, 36, 42,
48, 49]. Unlike our work, most existing methods depend on iterative fine-tuning. Althougheach fine-tuning step may mitigate information loss caused by each pruning step, strong
dependence on fine-tuning increases the risk of overfitting (see [24] for more discussion).
In contrast to the iterative fine-tuning approach, the Domain Adaptive Low Rank Matrix Decomposition (DALR) method [26] can maintain accuracy without fine-tuning after
compression. DALR and our method are data-dependent, which makes them suited for DA
setting. Since DALR uses SVD, both are based on spectral analysis. An important difference
between DALR and our method is their structural modification. We describe the difference,
considering that uncompressed two layers have weight matrices of dimension m×n and n× p
as shown in Fig. 1. For compressing the first layer, DALR produces two smaller layers thathave weight matrices of dimension m × k and k × n (k is kept rank). First, it is noted that, toactually reduce the number of parameters in the network, the parameter k must verify the following condition: k(m + n) < mn. DALR affects only the layer being compressed since theoutput shape remains unchanged. Conversely, our method does not create additional layers
but also affects the input dimension of the following layer. Second, DALR is only designed
to compress dense layers while our method can be applied to both dense and convolutional
layers. Finally, in [26], the authors only mention using k as a given input to compression.
Therefore compression and validation with various k will be needed unless a prior knowledge of how much the layer is compressible is known. Our method helps to avoid this issue,
using an information retention ratio parameter defined in Section 3.1. It should be noticed
however that it is possible to reproduce a similar parametrization in DALR by computing theratio of kept singular values over all singular values of the decomposition.

3

Spectral pruning with DA regularization

Our work was built on a layer-wise compression method [39], which is based on spectral
analysis. The method does not require any regularization during training, but is applicable
to a pre-trained model. We will first detail this method then introduce our DA regularizer.

3.1

Principle

The method compresses the network layer per layer and can be applied to both dense andconvolutional layers. It aims at reducing the dimension of the output of a layer such that the
information of the input of the next layer remains mostly unchanged. To compress a layer,
it selects the most informative nodes (neurons) or feature map channels (corresponding toconvolutional filters). See Fig. 3 in [12] and Fig. 2 in [15] for visual explanations of pruning

4

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

dense and convolutional layers, respectively. For brevity, we will avoid the distinction of thelayer types in the remainder of this paper and refer only to nodes for both cases.
More formally, let us denote x the input to the network and suppose that we are given
a training data (xi , yi )ni=1 with size n. Let us consider a model that is a sequence of convolutional and fully connected layers. We denote by f l and η l respectively the mapping
function and activation function of a layer l. Finally we denote by φ l (x) ∈ Rml the representation obtained at layer l where ml is the number of nodes in the layer l. Therefore we haveφ l (x) = η l ( f l (φ l−1 (x))). The objective of the method is to find for each layer l, the subset
of nodes J ⊂ {1, . . . , ml } that minimizes the following quantity:Ê[kφ l (x) − ÂJ φJl (x)k2 ],(1)
where Ê[.] denotes the empirical expectation, φJl (x) denotes the vector composed of the components of φ l (x) indexed by J, and ÂJ is a matrix to recover the whole vector φ l from its
sub-vector φJl as well as possible.
For a fixed J, it is easy to minimize the objective (1) with respect to ÂJ as ÂJ = Σ̂F,J Σ̂−1
J,J ,where Σ̂ := Ê[φ l (x)φ l (x)> ](= 1n ∑ni=1 φ l (xi )φ l (xi )> ) is the empirical covariance matrix andF := {1, . . . , ml } is the full set of indexes. Then φ l (x) can be approximated by ÂJ φJl (x)which is equivalent to pruning the nodes of layer l that do not belong to J and reshaping theweights of the next layer. Let us denote Wl+1 the weight matrix if the next layer is a fully
connected layer, Tl+1 the weight tensor if it is a convolutional layer. In that case we denoteby the four dimensional tuple (oc , ic ,W, H) the shape of the weight tensor where oc , ic ,W, H
stand respectively for output channels, input channels, width, height. The reshaping of theweights of the next layer is done the following way. For fully connected layers, we obtaine l+1 = W l+1 ÂJ . For convolutional layers, it is done by first
the reshaped weight matrix as W
reshaping Tl+1 to a (W, H, oc , ic ) tensor. Therefore each point of the filter grid is associatedew,h = Mw,h ÂJ . The
to a filter matrix Mw,h ∈ Roc ×ic that will be reshaped by the operation: M
tensor is then reshaped back to its original form (oc , ic ,W, H) where ic has been modified by
the process.
Once we obtained the optimal ÂJ for a fixed J, then we minimize the objective (1) withrespect to J. By substituting the optimal ÂJ to the objective function (1), it can be re-written
as minJ Tr[Σ̂F,F − Σ̂F,J Σ̂−1
J,J Σ̂J,F ]. The minimand in the right hand side is zero for J = F andTr[Σ̂F,F ] for J = 0.
/ Hence we may consider the following “ratio” of residual as an information
retention ratio r to measure the relative goodness of J:
Tr(Σ̂F,J Σ̂−1
J,J Σ̂J,F )r=.
(2)
Tr(Σ̂F,F )The higher the ratio the more information computed by the layer will be retained. It is
no greater than 1 since the denominator is the best achievable value with no cardinality
constraint. Compression can therefore be parametrized by a required information retention
ratio parameter α and the final optimization problem translates to minJ⊂F |J| s.t. r ≥ α. To
solve the optimization problem, the authors of [39] proposed a greedy algorithm where J isconstructed by sequentially adding the node that maximizes the information retention ratio.
It is worth noticing that though the method is aimed at compressing each layer sequentially, using α allows more flexibility than methods with a fixed compression rate for eachlayer. Since we do not impose constraints on the cardinality of J, compression will adapt to
the inherent compressibility of the layer. We argue that this makes α an easy to tune and intuitive hyperparameter and, if necessary, can easily be combined with cardinality constraints.

5

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

3.2

Domain Adaptation regularization

In DA setting, informative nodes could be different between source and target domains. To
adjust this difference, we propose an additional regularization term to select better nodes.
As shown by [2], one of the main challenges of DA is to find a representation discriminative with respect to the classification task but not discriminative between domains. The
general idea is that, by matching the two distributions, the classification knowledge learnedon the labeled source data can be transferred to the target data. Therefore, we propose to use
a measure of the alignment of the source and target features distributions for selecting nodes
during compression. We call our method Moment Matching Spectral Pruning (MoMaSP).
Previous work involving covariance alignment achieved high accuracy in DA tasks [37].
Following those observations, we define our regularization term as the following
RJ = kÊx∼X s [φJl (x)] − Êx∼X t [φJl (x)]k + kSJ,J

s
t
(CJ,J
−CJ,J
)kF ,

(3)

where X s , X t denotes the source and target distributions, Cs ,Ct respectively the source and1t Ct )− 4 ,detarget empirical covariance matrices, S a scaling matrix defined by Si j = (Ci,ij, jnotes the element wise multiplication, and k · kF means the Frobenius norm of a matrix. Thisquantity measures the discrepancy between the source and target distributions by using the
up-to second order statistics. The first term measures the discrepancy of mean and the second
term measures the discrepancy of the (scaled) second order moment. Hence, this regularization term induces a moment matching effect and the two distributions become similar on theselected nodes. We may use the MMD criterion (kernel-based discrepancy measure) [9] tocapture not only the first and second moments but also all higher moments instead, but wefound that it takes too much computational time. Although the kernelized alignment loss [19]could speed-up computation, source and target samples per class need to be few.
Instead of the criterion (3), we also consider the following alternative formulation:

R j = kÊx∼X s [φ jl (x)] − Êx∼X t [φ jl (x)]k + kS j,F(Csj,F −Ctj,F )k.

(4)

The first formulation (3) depends on the whole subset J while the second one computes
information more specific to each node. The second formulation is more computationallyefficient since the R j ’s can be computed only once for each candidate index. The first one
needs to recompute it for every candidate index at each step of the procedure because of its
dependence on J.

3.3

Intuition behind our regularization

The intuition behind our regularization is the following. In DA setting where few or no labels
are available on the target distribution, the discriminative knowledge is learned mainly on thesource distribution. Thus the feature distributions on both domains should be aligned for thediscriminative knowledge learned on the source domain to apply to the target domain. When
comparing the information retention ratios of Eq. 2 for different nodes, we observed thatmany of them had very close values. This means that many nodes capture approximately thesame amount of information, in the sense of total variance. Our intuition was that when these
relative differences are too small, they are not significant to discriminate between nodes andare more likely the result of noise. Therefore a better criterion should be used to differentiate
nodes that capture a same amount of variance. Since our compression method is designed tobe applied on models that have been trained using DA methods, it is natural to use a criterion
related to DA objectives. We choose to compare nodes based on how well their feature
distributions on source and target domains are aligned. Nodes leading to a better alignment

6

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

should be favored as they will allow for a better transfer of discriminative knowledge fromsource to target domain. Our method realizes this via a moment matching type regularization.

3.4

Practical implementation

We present in Algorithm 1 a pseudo code for a Algorithm 1: MoMaSP
practical implementation of our method using the Input : Σ̂F,F : Empirical covariance matrix,
α : Required information retention ratio
first formulation of our regularizer. If the sec- Output: J : Subset of selected nodes
Function
FindSubset(Σ̂F,F , α)
ond formulation is used, the regularization term for
J ← 0/T ← Tr(Σ̂F,F )each node is computed only once before enteringr←0C←F
the while loop as it does not depend on J. At eachwhile r < α dofor j ∈ C do
step of the algorithm, we denote by C the set of canJj ← J ∪ jdidate indexes. To select which node to add to J,
Compute RJ j
Tr(Σ̂F,J j Σ̂−1
J j ,J j Σ̂J j ,F )the ratio from Eq. (2) is computed for each candiVj ←Tdate index. We denote by V the vector where each
end


RJ
coordinate corresponds to the ratio value of a cani ← arg max V j − λ · σ (V ) · max 0 (Rj )
J j0
j
j∈C
didate index. Similarly, we denote by RJ the vector
J ← J ∪ {i}r←Viwhere each coordinate correspond to the regularizC ← C \ {i}ending term associated to a candidate index.
end
Without regularization, the index to add to J issimply given by i = arg max j∈C V j . Using regularization, the index to add to J is chosen as()
RJ j
i = arg max V j − λ · σ (V ) ·,(5)
max j0 (RJ j0 )j∈Cwhere σ (V ) is the standard deviation of (V j ) j∈C . The values of RJ j are rescaled to be in
the same range as V j ’s by max normalization. Multiplying by V ’s standard deviation, σ (V ),ensures that the regularization will be applied at the right scale. It should only slightly
modify the relative ordering of the values in V as the main criteria for selection must remain
the information ratio. Indeed, only taking into account the regularization term would favornodes that output a constant value across both distributions. The hyperparameter λ allows
for more control over the trade-off between those two terms. We use 1 as the default λ in our
experiments. The max normalization and scaling make λ easier to tune. This compression
is applied after training and can therefore be combined with any DA method.

4

Experiments on digits images

In this section, we conduct experiments with a model trained on digits datasets, using a DA
technique [33]. The source dataset is the SVHN dataset [28] and the target dataset is the
MNIST dataset [22], which are standard for DA with adversarial training [7, 33, 43].

4.1

Model

Considering the relative simplicity of the task, we used a custom model composed of a 4 layers feature generator (3 convolutional + 1 dense) and a 2 layers classifier (see Supplementary
Material for details). To train the model we used the DA technique presented in [33] which
utilizes “maximum classifier discrepancy." Briefly, the adaptation is realized by adversarial
training of a feature generator network and two classifier networks. The adversarial training

7

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

forces the feature generator to align the source and target distribution. Contrary to othermethods relying on adversarial training with a domain discriminator, this method considers
the decision boundary while aligning the distributions, ensuring that the generated features
are discriminative with respect to the classification task at hand.

Figure 2: Target accuracy over compression
rate for three different settings of data used
as input. Results are presented as mean +/standard deviation based on 10 iterations.
Test data
SourceTargetFirst layer nodes specificity
SourceTarget
None
0.79
0.77
0.72
0.65
0.84
0.71

Table 1: Mean activation rates.

4.2

Figure 3: Comparison using only target dataor each formulation of our DA regularization. Results are presented as mean +/- standard deviation based on 10 iterations.
C.R. (%)

96.0

96.5

97.0

97.5

98.0

Target (%)

94.6

94.0

92.4

90.2

84.4

98.5
68.7

Reg J (%)
Reg j (%)

95.0
95.2

94.1
94.9

93.7
94.3

91.1
90.6

83.3
89.3

72.7
77.5

Table 2: Accuracy on target test dataset fordifferent compression rate (C.R.).

Data choice for compression

The uncompressed model was trained using both train splits and evaluated on the test splitof MNIST. It reached an accuracy of 96.64%, similar to the results obtained in the original
paper [33]. We then apply compression on the trained model. During compression, only
data of the train splits was used and the compressed models were evaluated on the MNIST
test split. Since the method is data-dependent, the choice of data to use to compute theempirical covariance matrix should be taken care of. Three different settings were tested: (1)Target: Using 14,000 samples from the target distribution; (2) Target/Source: Using 7,000
samples from the target distribution and 7,000 samples from the source distribution; and(3) Target+Source: Using 14,000 samples from the target distribution and 7,000 additional
samples from the source distribution. The results obtained are presented in Fig. 2. Using
only target samples to compute the covariance matrix shows the best performance.
To give a better understanding of this result, we conducted an analysis about how thenodes activation pattern depends on data distribution as follows: we compressed the firstlayer of a trained network using either only target data or only source data. We then compared
the activation of nodes that were selected only in one of the two cases, in other words, nodes
specific to target or source data. We show the results in Table 1.
As expected, nodes selected using source data had a significantly higher activation rate on
the source distribution and conversely for target specific nodes. As a control case, we added
activation of nodes that were selected in neither of the two settings. Those do not show any
significant difference in their activation. Interestingly, this difference was no longer appearing when comparing activation of the last fully connected layer, because DA training aligns
the distributions of extracted features. This experiment sheds light on how the input data
affects the compression of data-dependent methods. In case of a model trained on different

8

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

distributions with DA, early layers contain nodes that are specific to each distribution. Thus
it is critical for compression to use data coming exclusively from the distribution the model
will be applied to. Partially using source data forces compression in the early layers to selectsource specific nodes to the expense of target specific nodes leading to poorer results.

4.3

Regularization for compression

Finally we compared the results of the best baseline, using only target data, with adding
the regularization term introduced in Section 3. The results are presented in Fig. 3 andsummarized in Table 2. It appears that the second formulation of our regularizer gives thebest performance. This is probably due to the fact it focuses on node level information andnot on the whole subset J, giving better ability to discriminate against specific nodes.
Compared to the baseline, our regularizer leads to significant improvements in compression performance. We observe up to 9% bumps in accuracy. Yet, we notice that the baseline
performs slightly better for very high compression rates. We conjecture that reducing information loss becomes more important for such compression rates and our regularization
with λ =1 is too strong. There is room for improvement by tuning the hyperparameter λdepending on the compression rate or developing methods for adaptive regularization.

5

Experiments on natural images

In this section, we compare our method (MoMaSP) with the factorization based compression
method (DALR) in [26]. We reproduce the same setting as in their experiment: a VGG19
model pre-trained on the ImageNet dataset [32] is fine-tuned on different target datasets.

5.1

Experimental settings

We establish our comparison based on the following three datasets used in [26] experiments:
Oxford 102 Flowers [29], CUB-200-2011 Birds [45], and Stanford 40 Actions [47].
For all three datasets, we first trained uncompressed models by fine-tuning from an ImageNet pre-trained VGG19 model. In the fine-tuning before compression, we trained theweights of the fully connected layers while keeping the weights of the convolutional layers
frozen to their pre-trained value on ImageNet like [26].
We then compressed the models. The input data to compression was always composedof 4,000 samples of the train split, except for the Oxford 102 Flowers where the whole train
split was used. Additional 4,000 randomly sampled images from the ImageNet train split
were used for DA regularization. Note that our method and DALR [26] do not need thelabels of target datasets in this compression phase. After that we optionally fine-tuned thefully connected layers of compressed models with target labels.
We used the Adam optimizer [18] with a batch size of 50, a learning rate of 10−4 , and aweight decay of 5×10−4 for training models. See Supplementary Material for details.

5.2

VGG19 fc7 compression

We first evaluated the compression on the last fully connected layer (fc7) of the model,
containing 4,096 nodes, because the fc7 compression is a main evaluation setting in [26].
For each trial we report the results both with and without fine-tuning of the classifier layers

9

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNINGkparams (%)
SVD w/o FT.
SVD w/ FT.
DALR w/o FT.
DALR w/ FT.
Ours w/o FT.
Ours w/ FT.

4
0.20
5.2
13.7
8.6
18.9
68.9
67.3

8
0.39
12.8
36.7
31.1
48.2
69.3
68.9

16
0.78
30.2
56.6
55.1
64.4
70.0
70.3

32
1.56
54.6
65.9
66.9
70.3
71.2
69.9

64
3.13
65.2
70.0
71.0
70.7
72.0
70.8

128
6.25
67.6
70.8
72.8
72.3
72.5
70.6

k
params (%)
SVD w/o FT.
SVD w/ FT.
DALR w/o FT.
DALR w/ FT.
Ours w/o FT.
Ours w/ FT.

4
0.20
4.1
17.3
5.6
19.0
58.3
57.9

8
0.39
12.6
39.1
23.1
40.9
58.2
57.3

16
0.78
30.3
51.2
49.3
54.0
58.2
57.7

32
1.56
46.9
57.2
57.8
59.8
58.8
58.2

64
3.13
54.8
58.8
60.1
59.3
59.3
58.2

128
6.25
60.5
58.5
60.9
59.5
60.2
59.7

Table 3: VGG19 fc7 compression on Oxford Table 4: VGG19 fc7 compression on CUB102 Flowers. Test dataset accuracy results 200-2011 Birds. Test dataset accuracy results(%). Original accuracy: 73.1%.
(%). Original accuracy: 61.5%.
kparams (%)
SVD w/o FT.
SVD w/ FT.
DALR w/o FT.
DALR w/ FT.
Ours w/o FT.
Ours w/ FT.

4
0.20
17.1
45.4
20.7
46.8
69.0
70.5

8
0.39
31.0
62.0
43.3
63.3
70.2
70.4

16
0.78
54.5
68.6
65.0
69.6
71.8
71.7

32
1.56
67.3
72.8
73.0
73.5
73.1
72.9

64
3.13
72.2
73.1
74.3
72.9
73.9
73.7

128
6.25
72.7
73.7
74.7
72.9
73.8
74.0

Accuracy (%)
LayersFT. C.R. #params(%)
conv fc
Oxford CUB Stanford
348.1 43.0
57.7
DALR
84.9 21.1 M
3 3
62.2 52.8
67.4
3
48.1 37.1
61.0
Ours
85.0 21.1 M
3 3
63.9 55.1
69.0
3
3
60.6 51.4
69.2
Ours
85.4 20.4 M
3
3 3
71.4 58.3
72.6

Table 5: VGG19 fc7 compression on Stan- Table 6: VGG19 full compression on each
ford 40 Actions. Test dataset accuracy results dataset.
(%). Original accuracy: 74.6%.
after compression. We also report the results of compression using basic SVD on fc7 weightmatrix as a baseline to further illustrate the advantage of using data-dependent methods. To
compare our method with DALR, a dimension k was set for the compression using DALR
then k0 , the dimension to keep in our method resulting in an equal number of parameters,
was determined accordingly. See Supplementary Material for details.
The results are presented in Tables 3, 4, and 5. We show the relative numbers of parameters in the fc7 layer compressed by DALR in the “params” rows in the tables as in the DARL
paper [26]. In all experiments, our method maintains high accuracy even for high compression rates. In such cases it outperforms DALR by a large margin. However, for lower
compression rates DALR consistently compares favorably to our method though the difference is arguably small. In most cases, fine-tuning does not improve much the performanceof any of the two methods, except for DALR for high compression rates. This phenomenon
implies that the learning rate and/or the dropout rate we used are too high especially for our
method (see additional fine-tuning results in Supplementary Material for details).

5.3

VGG19 full compression

It is important to notice that, contrary to DALR, our method is able to compress both convolutional and dense layers. To further demonstrate this advantage, we proceeded to thecomparison of fully compressing the VGG19 network using both methods. We conducted
the experiment on all three datasets. DALR was applied to the fully connected layers. The
dropout layers of models compressed by our method were disabled in fine-tuning, because
our method reduce the input dimension of the dropout layers unlike DALR, resulting in
performance degradation. See Supplementary Material for other details.
Results are presented in Table 6. For all three datasets, our method consistently achieves
a better compression rate (C.R.) while reaching test accuracy 5.2%–12.5% higher than DALR.
If we apply our method to the fully connected layers only, the difference in accuracy issmaller but still favorable to our method in many cases. Although further improvement

10

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

would be preferable to keep representations for many classes (CUB-200 without fine-tuning),
in practice, it is not necessary to overcompress the fully connected layers by our method.

5.4

Discussion

Our method is drastically better for high compression rates, while DALR is slightly better for
low compression rates. We conjecture that this is mainly due to the fact that the two methods
modify differently the network (see Fig. 1). DALR affects only one layer. Therefore if
the compression is too important, it reaches a critical point where not enough information
can possibly be retained. Our method affects two successive layers therefore spreading theeffect of compression and avoiding this critical point for high compression rates. On theother hand, our method needs to consider nonlinearity between two layers and uses a greedy
algorithm [39]. Thus only affecting one layer is better to maintain high accuracy for lower
compression rates, because the output of the one layer can be optimally approximated.
The optimization problem to solve in DALR admits a closed form solution, making it
a very fast compression method. Compared to DALR, it is a limitation of our method torequire an iterative optimization. However, the extra computational time is usually a few
hours on 1 GPU (Tesla V100) in our experiments. Furthermore, an iterative process is usedin the DALR paper [26] to determine compression rate pairs for fc6 and fc7, and it needs
iterative accuracy evaluation. Pruning methods based on iterative fine-tuning approach also
take time for pruning, fine-tuning, and evaluation. Therefore our method is practical enough.

6

Conclusions

In this paper, we investigated compression of DNNs in the DA setting. As shown by [26],using a data-dependent method is crucial in order to achieve good results. In that matter,our work shows that the input data to calculate information retention ratio for compression
should only come from the distribution on which the model will be applied to, the target
distribution in our case. This is because adding samples from another distribution will force
compression in the early layers to select nodes that are specific to this distribution. However,
we show that source data can still be used for regularization to improve nodes selection. Thisis done by comparing the first and second order statistics of the node’s feature distributions
on each of the source and target data. This criterion serves as a measure of the alignmentof the two distributions which directly relates to DA objectives. Therefore we denote thismeasure as a DA regularizer. We evaluated this regularization on a spectral pruning method
introduced in [39] and obtained significant improvements on its compression results. Finally we compared our regularized compression method with the factorization based methodof [26] on real world image datasets. Our method compares favorably on all three datasets,
leading to significant improvements in retained accuracy for high compression rates.
Although our work focused on one compression method, we argue that using first andsecond order statistics of feature distributions to measure the alignment between source andtarget features and using it as a criterion for compression can be applied to other methods.
This work can therefore serve as a first example and practical implementation of this idea.
Applying our method to unsupervised DA setting [20, 48] is an interesting future direction.
Acknowledgements TS was partially supported by JSPS KAKENHI (26280009, 15H05707,
and 18H03201), Japan Digital Design, and JST-CREST.

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

11

References[1] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS. 2014.
[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, andJennifer Wortman Vaughan. A theory of learning from different domains. Machine
Learning, 2010.
[3] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression.
In KDD, 2006.
[4] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal
Processing Magazine, 2018.
[5] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.
[6] Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS.
2014.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle,
François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial trainingof neural networks. JMLR, 2016.
[8] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. Compressing deep
convolutional networks using vector quantization. arXiv:1412.6115, 2014.
[9] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, andAlexander Smola. A kernel two-sample test. JMLR, 2012.
[10] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficientdnns. In NIPS. 2016.
[11] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deeplearning with limited numerical precision. In ICML, 2015.
[12] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS. 2015.
[13] Stephen Jose Hanson and Lorien Y. Pratt. Comparing biases for minimal network
construction with back-propagation. In NIPS. 1988.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning forimage recognition. In CVPR, 2016.
[15] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deepneural networks. In ICCV, 2017.
[16] Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neuralnetwork. arXiv:1503.02531, 2015.

12

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING[17] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional
neural networks with low rank expansions. In BMVC, 2014.
[18] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In
ICLR, 2015.
[19] Piotr Koniusz, Yusuf Tas, and Fatih Porikli. Domain adaptation by mixture of alignments of second- or higher-order scatter tensors. In CVPR, 2017.
[20] Piotr Koniusz, Yusuf Tas, Hongguang Zhang, Mehrtash Harandi, Fatih Porikli, and Rui
Zhang. Museum exhibit identification challenge for the supervised domain adaptation
and beyond. In ECCV, 2018.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification withdeep convolutional neural networks. In NIPS. 2012.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[23] Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. Optimization basedlayer-wise magnitude-based pruning for DNN compression. In IJCAI, 2018.
[24] Bingyan Liu, Yao Guo, and Xiangqun Chen. WealthAdapt: A general network adaptation framework for small data tasks. In ACMMM, 2019.
[25] Jiaming Liu, Yali Wang, and Yu Qiao. Sparse deep transfer learning for convolutional
neural network. In AAAI, 2017.
[26] Marc Masana, Joost van de Weijer, Luis Herranz, Andrew D. Bagdanov, and Jose M.
Alvarez. Domain-adaptive deep network compression. In ICCV, 2017.
[27] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning
convolutional neural networks for resource efficient inference. In ICLR, 2017.
[28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y.
Ng. Reading digits in natural images with unsupervised feature learning. In NIPS
Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[29] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over alarge number of classes. In Indian Conference on Computer Vision, Graphics & Image
Processing, 2008.
[30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards
real-time object detection with region proposal networks. In NIPS, 2015.
[31] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo
Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In ICLR, 2015.
[32] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C.
Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV,2015.

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING

13[33] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum
classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. In ICLR, 2015.
[35] Amos Sironi, Bugra Tekin, Roberto Rigamonti, Vincent Lepetit, and Pascal Fua. Learning separable filters. TPAMI, 2015.
[36] Xavier Suau, Luca Zappella, and Nicholas Apostoloff. Filter distillation for network
compression. In WACV, 2020.
[37] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
[38] Taiji Suzuki. Fast generalization error bound of deep learning from a kernel perspective.
In AISTATS, 2018.
[39] Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma
Wachi, So Hirai, Masatoshi Yukishima, and Tomoaki Nishimura. Spectral pruning:
Compressing deep neural networks via spectral analysis and its generalization error. In
IJCAI-PRICAI, 2020.
[40] Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for
non-compressed network: unified generalization error analysis of large compressible
deep neural network. In ICLR, 2020.
[41] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper
with convolutions. In CVPR, 2015.
[42] Frederick Tung, Srikanth Muralidharan, and Greg Mori. Fine-Pruning: Joint finetuning and compression of a convolutional network with Bayesian optimization. In
BMVC, 2017.
[43] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
[44] Vincent Vanhoucke, Andrew Senior, and Mark Z. Mao. Improving the speed of neuralnetworks on CPUs. In NIPS Workshop on Deep Learning and Unsupervised Feature
Learning, 2011.
[45] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.
The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001,
California Institute of Technology, 2011.
[46] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized
convolutional neural networks for mobile devices. In CVPR, 2016.
[47] Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and
Li Fei-Fei. Human action recognition by learning bases of action attributes and parts.
In ICCV, 2011.

14

DILLARD, SHINYA, SUZUKI: MOMENT MATCHING SPECTRAL PRUNING[48] Chaohui Yu, Jindong Wang, Yiqiang Chen, and Zijing Wu. Accelerating deep unsupervised domain adaptation with transfer channel pruning. In IJCNN, 2019.
[49] Yang Zhong, Vladimir Li, Ryuzo Okada, and Atsuto Maki. Target aware network
adaptation for efficient representation learning. In ECCV Workshop on Compact andEfficient Feature Representation and Learning, 2018.