Domain-invariant Learning using Adaptive Filter Decomposition
Authors: Ze Wang
Xiuyuan Cheng
Guillermo Sapiro
Qiang Qiu
Venue:  (2020)

Abstract: Domain shifts are frequently encountered in real-world scenarios. In this paper, we consider the problem of domain-invariant deep learning by explicitly modeling domain shifts with only a small amount of domain-specific parameters in a Convolutional Neural Network (CNN). By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we show for the first time, both empirically and theoretically, that domain shifts can be effectively handled by decomposing a regular convolutional layer into a domain-specific basis layer and a domain-shared basis coefficient layer, while both remain convolutional. An input channel will now first convolve spatially only with each respective domain-specific basis to ``absorb" domain variations, and then output channels are linearly combined using common basis coefficients trained to promote shared semantics across domains. We use toy examples, rigorous analysis, and real-world examples to show the framework's effectiveness in cross-domain performance and domain adaptation. With the proposed architecture, we need only a small set of basis elements to model each additional domain, which brings a negligible amount of additional parameters, typically a few hundred.

Under review as a conference paper at ICLR 2020
DOMAIN-INVARIANT LEARNING USING ADAPTIVE FILTER DECOMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain shifts are frequently encountered in real-world scenarios. In this paper, we consider the problem of domain-invariant deep learning by explicitly model- ing domain shifts with only small amount of domain-specific parameters in a Convolutional Neural Network (CNN). By exploiting the observation that con- volutional filter can be well approximated as a linear combination of small set basis elements, we show for the first time, both empirically and theoretically, that domain shifts can be effectively handled by decomposing a regular convo- lutional layer into a domain-specific basis layer and a domain-shared basis coef- ficient layer, while both remain convolutional. An input channel will now first convolve spatially only with each respective domain-specitn specific basis to "absorb" domain variations, and then output channels are linearly combined using com- mon basis coefficients trained to promote shared semantics across domains. We use toy examples, rigorous analysis, and real-world examples to show the frame- work's 's effectiveness in cross-domain performance and domain adaptation. With the proposed architecture, we need only a small set of basis elements to model each additional domain, which brings negligible amount of additional parame- ters, typically a few hundred.
INTRODUCTION
Training supervised deep networks requires large amount of labeled training data; however, well- trained deep networks often degrade dramatically testing data from significantly different do- main. In real-world scenarios, such domain shifts are introduced by many factors, such as different illumination, viewing angles, and resolutions. Research topics such as transfer learning and domain adaptation are studied to promote invariant representations across domains with different levels of availabilities of annotated data.
Recent efforts on learning cross-domain invariant representations using deep networks generally fall into two categories. The first is to learn common network with constraints encouraging invariant feature representations across different domains Long et al. (2015) (016b); Tzeng et al. 2014). The feature invariance is usually measured by feature statisties like Maximum Mean Dis- crepancy (MMD), or feature discriminators using adversarial training Ganin et al. (2016). While these methods introduce additional model parameters, the effectiveness largely depends on the degree of domain shifts. The other direction is to explicitly model domain specific characteris- tics with multi-stream network structure where different domains are modeled by corresponding sub-networks at the cost of significant extra parameters and computations Rozantsev al, (2018b). Regularizations are imposed among sub-networks to encourage common semantics across domains.
In this paper, we model domain shifts through domain-adaptive filter decomposition (DAFD) with layer branching. At branched layer, decompose each filter over small set of basis elements to model intrinsic domain characteristics, while enforcing shared cross- doos-ndomain coef- ficients to align invariant semanties. A regular convolution is now decomposed into two steps. First, a domain-speciff basis convolves spatially only each individual input channel for shift "correction." Second, the "corrected" output channels are weighted summed using domain-shared basis coeffi- cients (1 x convolution) to promote common semantics. When domain shifts happen in space, we rigorously prove that such layer-wise "correction" by the same spatial transform applied to bases suffices align the learned features.
Comparing to the existing subnetwork-bardt properties: First, only a very small amount of additional trainable parameters are introduced to explicitly model each domain, i.e., domain-specific bases. The majority of the parameters in the
methods, the proposed method has several appealing

---

Under review as a conference paper at ICLR 2020
network remain shared across domains, and learned from abundant training data to effectively avoid overfitting. Furthermore, the decomposed filters reduce overall computations significantly com- pared to previous works, where computation typically grows linearly with the number of domains.
We conduct extensive real-world face recognition, image classification, and segmentation experi- ments, and observe that, with the proposed method, invariant representations across domains are consistently achieved without compromising the performance of individual domain.
Our main contributions are summarized as follows:
We propose Nomain-invariaae representation learning through bases decomposition with layer branching, where domain-specinf bases are learned to counter domain shifts, and semantic alignments are enforced with cross-domain common basis coefficients.
We both theoretically prove and empirically observe that by stacking the bases-decomposed
branched layer, invariant representations across domains achieved progressively.
The majority of network parameters remain shared across domains, which alleviates the demand for massive annotated data from every domain, and introduces only small amount additional computation and parameter overhead. Thus the proposed approach serves as efficient way for domain invariant learning.
Soarce Ispai
Target Ixpui
Scarce lipes
Turget Japut
Souma Tegusl
Torget teper
8
Conv layer
Branched
Coery
lurgec
Scurre Floes
Taryst Fibnos
Conv layer
Langet
FC layers Claseffer
FC lavers + Classifier
layers Classifier
(a) Regular CNN
(b) Basic Branching
(c) Branching with DAFD
(d) Feature space in (a)
(e) Feature space in (b)
Feature space in (c)
Figure 1: Three candidate architectures considered for domain-invarian representation learning. In (a), set of common network parameters are trained to model both source and target domains. In (b), the domain characteristics are explicitly modeled by two of convolutional filters in each convolutional layer. Our approach is illustrated in (c) where domain-adaptive bases are learned to "absorb" domain shifts, while the decomposition coefficients
shared across domains promote common semantics. The feature space of the three candidate architectures, MNIST + SVHN, are visualized using t-SNE Maaten & Hinton (2008) in (d), (e), (f), respectively. The obtained cross- domain invariance can be clearly observed in (f).
2 RELATED WORK
Recent achievements domain-invarians learning generally follow two directions. The first direc- tion learning a single network, where parameters are shared across domains, while the network encouraged produce domain-invariant features by minimizing additional loss functions in the network training Ganin et al. (2016); Long et al. (2013) 2016a b); Tzeng et al. (2014). The Maxi- mum Mean Discrepancy (MMD) et al. 2007). and MIK-MMD/Gree
et al. (012)) ) in Long ct al. (2015), are adopted as the discrepancy metric among domains. Beyond the first order statistic, second-order statistics are utilized in Hoffman et al. 2014). Besides the hand-crafted distribution distance metrics, Ganin al.) (2016): Tzeng al. (2015); Long al. (2018) resort to adversarial training and achieve superior performances. Various distribution alignment methods, e.g., Wu et al. (2019) Kumar al. (2018). are proposed improve the invariant feature learning. While effective in certain scenarios, the performance of learning invariant features using a shared network is largely constrained by the degree of domain shift discussed in Rozantsev al. (008b)). Meanwhile,
2

---

Under review as a conference paper at ICLR 2020
some recent works like Wu et al. (2019) ; Zhao al. (2019) suggest important insights whether is sufficient to do domain adaptation by invariant representation and small empirical source risk, which shed light exploring more effective alignment methods that are robust to common issues like different marginal label distributions.
Another popular direction modeling each domain explicitly using auxiliary network structures. Based on the hypothesis that explicitly modeling what is unigue to each domain can improve model's ability to extract domain-invariant features, e.g.. [Bousmalis et al. (2016) proposes feature representation by two components where domain similarities and shifts are modeled by a private component and shared component separately. A completely two-stream network structure with no shared parameters is proposed in Rozantsev et al. (2088)) where auxiliary residual networks trained to adapt the layer parameters of the source domain to the target domain. Chang et al. (2019) proposes attacking domain shifts by domain-specific batch normalization, which we believe compatible with the proposed domain-specific filter decomposition for better performance. An- other popular direction for domain adaptation is to remap the input data between the source and the target domain for domain adaptation Murez et al. 7) Hu et al. (2018); [Hoffman et al. (2018), which not included in the discussion since Finally, learning invariance is of relevance beyond domain adaptation, e.g., in the field of causal inference Bühlmann 2018).
focusing on learning invariant feature space.
DOMAIN-ADAPII FILTER DECOMPOSITION
A straightforward way to address domain shifts is to learn from multi-domain training data single network as in Figure [(a) However, the lack of explicitly modelling of individual domains often sults in unnecessary information loss and performance degradation as discussed in Rozantsev et al.) (0188). Thus, often simultaneously observe underfitting for domains with abundant training data, and overfitting for domains with limited training. In this section, we start with a simplistic ped- agogical formulation, domain- layer branching in Figure [(b] where domain shifts modeled by a respective branch filters in a layer, one branch per domain. Each branch is learned only from domain-specine data, while non-branched layers are learned from data from all domains. We then propose to extend basic branching domain characteristics are modeled by -epecff bases, and shared decomposition coefficients enforced align cross-domain semantics.
decompdee branching as in Figure [(c) where
3.1 PEDAGOGICAL BRANCHING FORMULATION
We start with the simple-minded branching formulation in Figure l(b) To model the domain- specific characteristics, at the first several convolutional layers, we dedicate separate branch to each domain. Domain shifts modeled by independent set of convolutional filters in the branch, trained respectively with errors propagated back from the loss functions of source and target domains. For supervised learning, the loss function is the cross-entropy for each domain. For unsu- pervised learning, the loss function for the target domain can be either the feature statistics loss or the adversarial loss. The remaining layers are shared across domains. We assume one target domain and one source domain our discussion, while multiple domains are supported. Note that, though we adopt the source target naming convention in the domain adaptation literature, we address here general comain-invariant learning problem.
Domain Domain-adaptive branching is simple and straightforward, however, it has the following drawbacks: First, both the number of model parameters and computation are multiplied with the number of domains. Second, with limited target domain training data, we can experience overfitting in deter- mining a large amount of parameters dedicated to that domain. Third, no constraints are enforced to encourage doss-n shared semantics. We address these issues through layer branching with the proposed bases decomposition.
3.2 BASES-DECOMPOSEE BRANCHING
To simultaneously counter domain shifts and enforce cross-domain shared semantics, we decompose each convolutional filter in branched layer into domain-specific bases, and cross-domain shared coefficients, as illustrated in Figure [(c)
3

---

Under review as a conference paper at ICLR 2020
W
V
W,
Target
Coos-danat Coefikiinta
Tilom
(a) Decomposition over pre-fixed bases
Qiu
(h) The proposed domain-adaptive filter decomposi- tion.
al.
2018).
Figure 2: Illustrations of the proposed domain adaptivee filter decomposition.
convolutional filter can be well approximated as a linear combination of small set of basis elements Qiu et al. (2018). In Figure 2(a) convolutional filter W with size of x L x C' x C' is decomposed into a truncated expansion with pre-fixed bases in the spatial domain. The truncated bases & with size L x L x K are predefined and remain fixed in training. where K is a hyperparameter, usually a small number, e.g.,
Filter Decomposition.
indicating the number of basis elements used in each layer. The trainable parameters are thus reduced to the decomposition coefficients with size of x C' x C. With bases decomposition, a reduction rate of K. complexity and computation. This parameter saving is not the only virtue of basis decomposition, such decomposition is natural to extract both invariance across domains and adaptation to specific domains detailed next.
T2 is achieved both model
In our approach, as shown in Figure 2(b) we decompose source and target domain filters over domain-specific bases, with decomposition coefficients shared across domains. Specifically, branched layer, the source domain filter W, and target domain filter W. of size L x L x C' x C, are decomposed into vi, X a and a4 a, where v., and 1/2 with size of L L x K are the domain- adaptive bases for source and target domains, respectively; and R Rx xC denotes the common decomposed coefficients shared across domains. Different from Qiu et al. (2018). the domain- adaptive bases independently learned from the corresponding domain data to model domain shifts, and the shared decomposed coefficients are learned from the massive data from multiple domains. Note that thanks this proposed structure, only small amount of additional parameters is required here to model each additional domain, typically a few hundred.
each
With the above Noomiin-adaptia filter decomposition, at each branched layer, a regular convolution is now decomposed into two: First, a domain-specinf basis convolves each individual input channel for domain shift "correction." " Second, the "corrected"" output channels are weighted summed using domain-sharee basis coefficients (1x1 convolution) to promote common semantics. A toy example presented in Figure A.I for illustrating the intuition behind the reason why manipulating bases alone can address domain shifts. We generate target domain data by applying two trivial operations source domain images: First, every x non-overlapping patch in each image locally rotated by 90%. Then, images are negated by multiplying with -1. Domain-ine invariant features are observed by manipulating bases alone. We will rigorously prove in Section 4 why such layer-wise "correction" aligns features across domains, and present real-world examples in the experiments.
Parameters and Computation Reduction. Suppose that both input and output features have the same spatial resolution of W W, in each forward pass in regular convolutional layer, there are totally W x C' C (2T.2 + 1) flops for each domain. While in our model, each domain C') flops, where K is the number of basis elements. For parameters, there are totally x C" C x I?2 parameters in a regular convolutional layer where D is the number of domains which is typically 2 in our case. In our model, each layer has only K x (C" x C D L2) parameters. Taking VGG-16 Simonyan & Zisserman (2014) an example with an input size of 224 x 224, a regular VGG-16 with branching, Fig ((b) and Rozantsev et al. 2018a b). requires adding 14.71M parameters and 15.38G flops in convolutional layers handle each additional domain. With the here proposed method (Fig [(c). a VGG- 16 only requires adding
only introduces 2 x C' x 2T2
702 parameters and 10.75G flops to handle one additional domain (K=6).
PROVABLE INVARIANCE WITH ADAPTIVE BASES
In this section, we theoretically prove that the features produced by the source and target networks from domain-transferred inputs can be aligned by the proposed framework of only adjusting multi-
4

---

Under review as a conference paper at ICLR 2020
layer bases, assuming generative model of the source and target domain images via CNN. Since convolutional generative networks are a rich class of models for domain transfer Hu et al. (2018); Murez et al. (2017), our analysis provides theoretical justification of the proposed approach. Di- verse examples in the experiment section shows the applicability the proposed approach is poten- tially larger than what is proved here. All proof' are in the supplementary material.
Filter Transform via Basis Transform. Let and Wy be the filters in the branched convolu- tional layer for the source and target domains respectively, and similarly denote the source and target bases by fh and d/k. In the proposed bases decomposition DA architecture, the source and target domain filters are linear combinations of the domain-specific basis elements with shared decomposition coefficients, namely
W. (.)
Σ au (u),,,(u);
wa(u) Σ [antralu)
Certain transforms of the filter can be implemented by only transforming the bases, including
(1) A linear correspondence of filter values. Let X TR + RR be a linear mapping. by linearity, applies (.(u)) (e) = ((w(u)).
(u) ( (w))
E.g. the negation X(E) = X(E), as shown in Figure A.1
(2) The transformation induced by a displacement of spatial variable, i.c. "spatial transform" of filters, defined as (w)
w(u - 7 (u)), where displacement field. Note that the dependence on spatial variable in a filter is via the bases, thus VA,. - WK,t = D.th,, applies w, - wt = DyW,,
IR? -> IR2 is differentiable
If such filter adaptations are desired in the branching network, then it suffices to branch the bases while keeping the coefficients ag shared, as implemented in the proposed architecture shown in Figure [c). A fundamental question is thus how large is the class of possible domain shifts that can be corrected by these "allowable" filter transforms. In the rest of the section, we show that if the domain shifts in the images are induced from a generative CNN where the filters for source and target differ by a sequence of allowable transforms, then the domain shift can be provably eliminated by another sequence of filter transforms which can be implemented by bases branching only.
Correction of a Single Filter Transform. We first analyze the "symmetric" correction of one filter spatial transform D, in one layer. The inclusion of linear correspondence transform is more direct (see comments after Theorem D. For technical reasons, we assume that the displacement field T is a small distortion, namely || V < 1, and then D, is invertible. Example includes rotation by small angle and small factor rescaling (dilation).
For simplicity
only consider one input and output channel in each of the multiple convolutional layers. The argument extends to multiple channels by modifying the boundedness condition of the filters. Then the forward mapping in one convolutional layer can be written as = a(x + b), where x is the input activation, y/ the output, w is the filter, b the constant bias, and a is the nonlinear activation function, e.g., ReL.U. As we take continuous formulation in the analysis, the activations x and y are assumed to be smooth functions supported on domain Ω C R², typically Ω = [-1, 112. The filter w is a function supported on 2 B, B being the unit disk, and 2 is layer scale (diameter of filter patches). The 1-norm of a function is defined be ||=||| = See [=(u) du.
Lemma 1. Suppose that the two filters w, I are supported on 2 B and 2 B respectively, o R + R is non-expansive, Dr is spatial transform where T is odd, i.e., r(-u) -T(U), and
T x < <. Then
| * D, * arlx * w) * DJ'
loo {(25- + 2 V/Vlll + 4[[x]]1}-
where db denotes the nonlinea function with the bias. The second term vanishes if (Ia - T) is rigid motion, e.g., rotation.
Provable Invariance under a Generative Model of Images. Stacking the approximate commut- ing relation, Lemma [] in multiple layers allows correct a sequence of filter transforms in previous convolutional layers by another sequence of "syymmetric" ones. This means that if we impose a con- volutional generative model on the source and target input images, and assume that the domain transfer results from a sequence of spatial transforms of filters in the generative net, then by correct- ing these filter transforms in the subsequent convolutional layers we can guarantee the recovery of
5

---

Under review as a conference paper at ICLR 2020
the same feature mapping. The key observation is that the filter transfers can be implemented by bases transfer only.
We summarize the theoretical assumptions as follows:
(Al) The in any layer is non-expansive.
(A2) In the generative net (where layer is indexed by negative integers), wh w(-1) = Dow! where D; = DT. n. odd and |V S E < & for all / 1, L. The biases in the target generative net are mildly adjusted accordingly due to technical reasons (to preserve the "baseline output" from zero-input, c.f. the proof).
(A3) In the generative net, |/w---0 1 feed-forward convolutional net taking the generated images as input, called "feature net": The source net filters have ||w! |I S 1 for 1 = 1, 1.2..., and same with Dyw? (!) which will set to be wr Also, w and wi are both supported on 2 B for I = 1,,...
1 for all and so is w--1) Dew Same for the
One can show that w|l1 = ||||| when (la p) is rigid motion, and generally ,|D,, [||||]| S </V//>>/|/o/l] which is negligible when e is small. Thus (A3) the boundedness of the 1- norm of the source and target filters imply one another exactly or approximately. The boundedness of 1-norm of the filters preserves the non-expansiveness of the mapping from input to output in a convolutional layer, and in practice is qualitatively preserved by normalization layers. Also, as a typical setting, (A3) assumes that the scales ji in the generative net (the (-l)-th layer) and the feature net (the 1-th layer) are matched, which simplifies the analysis and can be relaxed.
Theorem 1. Suppose that X, and X, are source and target images generated by I-layer generative
CNN nets with source and target filters w. (1) w. (1) respectively from the common representation
h. Under (A1)-(A3), the output at the L-th layer of the target feature CNN from X,, by setting
( = D in all layers which can be implemented by bases branching, approximates that of the source feature CNN from X, up error which is bounded in I-norm by
4
2t 2 + +2/|/Ah
and the second term vanishes if (Ta - TI) are rigid motions, e.g., rotation.
Note that typically ji S... ji and 2d2 < L, and E, 2 is proportional to the diameter of S and thus an (11 constant. The result can be extended to allow a linear transform of the filter values, namely Dw(u)=
w(D,wu)). and correct the target filters fe to be (D= In Lemma 1 commutes with the convolution by linearity, and also with o by adjusting the bias parameter If needed. Unlike D, which becomes D.=' when applying to f. À remains as X, thus X is needed in the symmetric basis transfers to correct the domain shift. Proofs are in the supplementary material.
5 EXPERIMENTS
In this section, we perform extensive experiments to evaluate the performance of the proposed domain-adaptive filter decomposition. We start with the comparisons among the 3 architectures listed in Figure T on two supervised tasks. To demonstrate the proposed framework as one princi- pled way for domain-invariant learning, we then conduct a set domain adaptation experiments. There we show, by simply plugging the proposed domain filter decomposition into regular CNNs used in existing domain adaptation methods, we consistently observe performance improvements, which illustrates that the framework can be successfully applied with multiple leading deep network architectures and very different datasets and applications.
5.1 ARCHITECTURE COMPARISONS
We start with two supervised tasks performed on the three architectures listed in Figure I regular CNN (AI), basic branching (A2), and branching with domain-adaptive filter decomposition (A3). In each layer of the network with the proposed Nomain-adaptiv filter decomposition, the different domain features convolve spatially with the corresponding domain sppciffc bases first then the out- put features are linearly combined by the cross-domain basis coefficients simultaneously. For both tasks, the networks are trained end-to-end with a summed loss for domains, and the domain-specific bases are only updated by the error from the corresponding domain, while the basis coefficients are updated by the joint error across domains.

---

Under review as a conference paper at ICLR 2020
Scales
Source domain
Target domain
Methods
VIS Ace (%)
NIR Ace (%)
NIR+VIS (%)
0.1
0.05
0.005
0.1
0.05
0.005
A1
75.57
52.71
98.44
A2
98.4
99.2
96.4
98.6
97.6
81.6
814
78.4
49.6
61.0
A2
94,46
87.50
98.58
A3
99.4
98.8
98.8
85.6
82.2
64.4
A3
97.16
95.03
99.15
(a) Accuracy (%) on both domains for super-
(b) Cross-domain simultaneous face recognition on NIR-
vised domain adaptation.
VIS-2.0.
Table I: Comparisons on supervised domain adaptaion and cross-domain simultaneous face recog- nition. A1, A2, and A3 corresoond regular CNN, basic branching, and branching with domain- adaptive filter decomposition shown in Figure respectively.
Supervised Domain Adaptation on Digits. The first task is supervised domain adaptation, where adopt challenging setting by using MNIST handwritten digit dataset as the source domain, and SVHN as the target domain. We perform a series of experiments by progressively reducing the annotated training data for the target domain. We start the comparisons at 10% of the target domain labeled samples, and end at 0.5% where only 366 labeled samples are available for the target domain. The results test set for both domains are presented in Table 3(a) It is clearly shown that when training the target domain with small amount of data, network with basic branching suffers from overfitting to the target domain because of the large amount of domain specific parameters. While regular CNN generates well on target domain, the performance on source domain degrades when the number of target domain data is comparable. A network with the proposed domain-adaptive filter decomposition significantly balances the learning of both the source and the target domain, and achieves best accuracies on both domains regardless of the amount of annotated training data for the target domain. The feature space of the three candidate architectures visualized in Figure |
Supervised Simultaneous Cross-domain Face Recognition. To show the generality of the pro- posed domain-adaptive filter decomposition for various tasks, we further perform a supervised cross- domain (RGB and near infrared) simultaneous face recognition experiment in the supplementary material Section B and the quantitative comparisons are in Table where we can clearly observe that DAFD performs superiorly even with missing input domain. Note that A2 requires additional 14.71M parameters over AI, while our method requires only 0.0007M as shown in Table A.
5.2 EXPERIMENTS ON STANDARD DOMAIN ADAPTATIONS
In this section, we perform extensive experiments unsupervised domain adaptation. Note that the objective of the experiments in this section is not to validate the proposed domain comain-adaptive filter decomposition just another new method for domain adaptation. Instead, since most of the state-of- the-art domain adaptation methods adopt the regular CNÑ (A1) with completely shared parameters for domains, we show the compatibility and the generality of the proposed domain-adaptiog filter decomposition by plugging it into underlying domain adaptation methods, and evaluate the effec- tiveness by retraining the networks using exactly the same setting and observing the performance improvement over the underlying methods. Diverse real-world domain shifts including different sensors, different image sources, and synthetic images, and applications on both classification and segmentation are examined in these experiments. Together with the experiments in the previous section, this further stresses the plug-and-play virtue of the proposed framework.
In practise, instead of learning independent source and target domain bases, we learn the residual between the source and the target domain bases. The residual is initialized by full zeros, and trained by loss for encouraging invariant features in the underlying methods, c.g., the adversarial loss in ADDA Tzeng et al. (2017). We consistently observe that this stabilizes the training and promotes faster convergence.
Digit Classification. We perform experi- ments on three public digits datasets: MNIST,
Table 2: Accuracy (%) on Digits for unsupervised domain adaptation.
USPS, and Street View House Numbers (SVHN), with three transfer tasks: USPS to MNIST (U- + M), MNIST USPS (M- - U), and SVHN to MNIST (S - M). Classification accuracy on the target domain test set samples is adopted as the metric for measuring the per- formance. We perform domain-adaptive domain decomposition on -fate-of-the-art methods DANN
Methods
M-+U U U
5 M
Avg.
DANN
73.9
ADDA
$9.4
90.1
76.0
85.1
CDAN+E
95.6
98.0
89.2
94,3
DANN DARD
92.0
95.2
K2.1 (11.1%
89.8
ADDA
91.4
82,9
(5.5% 1)
CDAN+E DAFD
96.8
98.8
96.6
97.4 (3.2% 1)
7

---

Under review as a conference paper at ICLR 2020
Ganin et al. (2016), ADDA Tzeng al. (2017), and CDAN+E Long al. (018)). Quantitative comparisons are presented Table
Office-31. Office-31 is one of the most widely used datasets for visual domain adaptation, which has 4,652 images and 31 categories collected from three distinct domains: Amazon (A), Webcam (W), and DSLR (D). We evaluate all methods six transfer tasks A - W, D W, W - D, A + D. D + A. and W + A. Two feature extractirs, Alex Krizhevsky et al. (2012) and ResNet He et al. (20T6) are adopted for fair comparisons with underlying methods. Specifically, ImageNet initialization are widely used for ResNet in the experiments with Office-31, and we consistently ob- serve that initialization is important for the training on Office-31. Therefore, when training ResNet based networks with domain- -aaaiiiv-ddiiieeeeeeee filter decomposition, we initialize the feature extractor using parameters decomposed from ImageNet initialization. The quantitative comparisons are in Table 3
Table 3: Accuracy (%) on Office-31 for unsupervised domain adaptation (AlexNet and ResNet).
Method
A
-
Avg.
AlexNet (no adaptation)
61.6
95.420.3
99.0+002
63.8=0.5
51.1=06
49,8+0.4
70.1
AlexNer
DANN Ganin
MAANnGaninee et al. 2016
73.0+0.5
9644+0.3
99.2+0.3
72.3-0.3
53.4-04
51,2+0.5
74.3
ADDA 17eng al 2
73.5+0.6
96.240..4
98.8+-0.4
71.640.4
54.6-0.55
53.5-40.6
74.7
DANN + DAFD
74.4=0.3
97.1+0.4
99.1+0.4
74.2=0.3
56,8:0.55
53.1+0.7
75.8 (2.3% f) 76.7 (2.7% 1)
ADDA DAFD
77.2+0.5
97.9+0.44
98,5+0022
73.2+0.4
55.4+006
57,8+0.5
ResNet 50 adapiation)
68.410.2
96.710..1
99.3110.1
68.9.10.2
60.5
60.7.10.3
76.1
DANN Ganin et al.
2016
82.0:0.4
96.910.2
99.1100.1
79.7:0.4
68.2:0.4
67.410.55
82.2
ResNet
ADDA Tzene
2017
86.2=0.5
96.2+0.3
98.420.3
71,8=0.3
69.5=0.4
68.9+0.5
82.9
CDAN+ Long ar 2018
94.1+0.1
98.64+0.1
100.0+:00
92.9-0.2
71.0=0.3
69 09.3+0.3
87.7
DANN DAFD
86.4+0.4
96 9-.022
99. 99.2+-0.1
84,4+0.4
70.5+00.4
68 88.84
84.35 (2.3% f)
ADDA
86.8
97.740.11
98.41-0.1
80.5
71.110.4
69.1
83.9 (1.2% ()
CDAN+E DAPD
95.6.10.1
98.8.10.1
100.0:0.00
93.5.10.2
76.6.10.5
71.310.4
89.3 (18%% 1)
Image Segmentation. Beyond image classification tasks, we perform a challenging experiment on image segmentation to demonstrate the generality of the proposed commiin-adaptiv filter decom- position. We perform unsupervised adaptation from the GTA dataset Richter et al. (2016) (images generated from video games) to the Cityscapes dataset Cordts et al. (2016) (real-world images), which has a significant practical value considering the expensive cost on collecting annotations for image segmentation in real-world scenarios. Two underlying methods FCNs in the wild Hoffman al. (2016) and AdaptSegNet Tsai al. (2018) are adopted for comprehensive comparisons, Based on the underlying methods, all the convolutional layers are decomposed using domain-adaptive fil- ter decomposition, and all the transpose-convolute quantitative results in Table 4 we use petereettion-over-niiei FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively, as the evaluation metric. As with the previous examples, our method improves all state-of-the-an architectures. Qualitative results are shown in Figure A.2 and data samples are in Section C.T
layers are kept sharing by both domains. For
TP i.e., IoU TP+FF+PN. where TP,
Table Unsupervised DA for semantic segmentation: GTA- + Cityscapes
Welah
Mc
(HER)
1700
26.0
14.5
661
3.
12M
13
60
30
3M
415
245
10
RU0
131
15
80
80
PDN WLb (W0G)
30.1
764
23.4
48.
164
24
165
145
33
162
313
68.
441
42
314
50
25
86
15
33
12.4
36.5
162
766
564
35.
266
152
145
85.
14.9
154
165
17.6
143
135
151
3.6
131
5.1
MaeskepMe-O000--00 DAPO
35-4 SMP
16.1
35.3
79.
238
185
255
314
164
6.1
122
06.
28.
341
11.2
180
212
11
114
6 CONCLUSION
We proposed to perform Nomain-invariinn learning through domain- filter decomposition. To model domain shifts, convolutional filters in a deep convolutional network are decomposed over aadaiieedaptive bases to counter domain shifts, and cross-domain basis coefficients are constrained to unify common semantics. We present the intuitions of countering domain shifts by adapting bases through toy examples, and further provide theoretical analysis. Extensive experiments on multiple tasks validate that, by stacking nomain-adaptivy branched layers with filter decomposition, complex domain shifts in real-world scenarios can be bridged to produce domain-invariant representation, which are reflected by both experimental results and feature space visualizations, all this at virtual no additional memory or computational cost when adding domains.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances Neural Information Processing Systems, pp. 343- 351, 2016.
Peter Bühlmann. Invariance, causality and robustness. arXiv preprint arXiv: 10/2.082// 2018.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In JEEE Conference on Computer Vi- sion and Patter Recognition, pp. 7354-7362, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Putlern Recognition, pp. 3213-3223, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-2)) 2030, 2016.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex J Smola. A ker- nel method for the two-sample- paoualle-problm.. In Advances in Neural Information Processing Systems, pp. 513-520, 2007.
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in Neural Information Processing Systems, pp. 1205-1213, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Judy Hoffman, Sergio Guadarrama, Eric S Tzeng, Ronghang Hu, Jeff Donahue, Ross Girshick, Trevor Darrell, and Kate Saenko. Lsda: Large scale detection through adaptation. In Advances in Neural Information Processing Systems, pp. 3536-3544, 2014.
Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell, FCNs in the wild: Pixel-level adver- 2016.
sarial and constraint-basce adaptation. arXiv preprint
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistee adversarial domain adaptation. In International Conference Machine Learning, pp. 1994-2003. 2018.
Lanqing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Duplex generative aaversarial network for unsupervised domain qdaptation. In IEEE Conference on Computer Vision and Puttern Recogni- fion, pp. 1498-1507, 2018
Alex Krizhevsky, Ila Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Free- man, and Gregory Wornell. Co-regularized alignment for unsupervised domain adaptation. In Advances in Neural Information Processing Systems, pp. 9345-9356, 2018
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features 2015.
with deep adaptation networks. arXiv preprint 1eXXiv:1502.220
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. arXiv preprint arXiv:1605.066 0066600006666... 2016a.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, pp. 136- 144, 201

---

Under review as a conference paper at ICLR 2020
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances Neural Information Processing Systems, pp. 1640-1650, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research, 9NNvv)2559-2665 2008.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. arXiv preprint arXiv: 1712.0047, 13, 2017.
Qiang Qiu, Xiuyuan Cheng, Robert Calderbank, and Guillermo Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. International Conference on Machine Learning, 2018.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pp. 102-118. Springer, 2016.
Artem Rozantsev, Mathieu Salzmann, and Pascal Fua. Beyond sharing weights for deep domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018a.
Artem Rozantsev, Mathieu Salzmann, and Pascal Fua. Residual parameter transfer for deep domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, number CONF, 2018b.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. arXiv preprint 1xXiv:170..8840 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7472-7481, 2018.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In IEEE International Conference on Computer Vision, pp. 4068-4076, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pp. 4, 2017.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically--elxxd distribution alignment. In International Conference on Machine Learn- ing, pp. 6872-6881, 2019.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recogni- 2018
tion, pp.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp- 7523-7532, 2019.
10

---

Under review as a conference paper at ICLR 2020
A
TOY EXPERIMENT
Source Domain
Target Domain
304
Inpulss
ICH
Branched Layer I
Source Buscs
Larget Bases (Rotation)
Cross Domatt Decompostion Confficiens
Layer Feature Maps
Branched Layer
Sourco Boses
Thrger Doses (Negation)
Croes Domain Decompostion Confiiciens
Layer Fenture Maps
Figure A.1: Visualization of the toy example. The columns visualize the inputs, features, and bases of the source domain and the target domain, respectively. Only the output feature in the first channel of each convolutional layer is visualized for comparison. Domain invariant features, the last row, are obtained by manually adapting source domain bases to generated target domain bases.
B SUPERVISED CROSS-DOMAIN FACE RECOGNITION
Besides standard domain adaptation, the proposed domain-adaptive filter decomposition can be ex- tended to general tasks that involves more than one visual domain. Here we demonstrate this by performing experiments on supervised cross-domain face recognition. We adopt the VIR-VIS 2.0, which consists of 17,580 NIR (near infrared) and VIS (visible light) face images of 725 subjects, and perform cross-domain face recognition. We adopt VGGI6 the base network structure, branch all the convolutional layers with the proposed domain-adaptive filter decomposition, and train the network from scratch. In each convolutional layer, two bases trained for modeling the NIR and the VIS domain, respectively. Specifically, one VIS image and one NIR image are fed simulta- neously to the network, and the feature vectors of both domains cross-domain feature, which is further fed into a linear classifier for classifying the identity. While the training is conducted using both domains simultaneously, we test the network under three set- tings including feeding single domain inputs only (VIS Acc and NIR Acc Table (bb) and both domain inputs NVI Acc in Table 3(b).. Quantitative comparisons demonstrate that branching with the proposed DAFD performs superiorly even with a missing input domain.
averaged to produce the final
C DATASET SAMPLES AND QUALITATIVE RESULTS
C.I UNSUPERVISED DA FOR IMAGE SEGMENTATION
For the segmentation experiments in Section 5.5, provide more qualitative results in Figure A.2 and some dataset samples in Figure A.3
D COMPUTATION AND PARAMETERS
In Table A.1 we provide comparisons on additional parameters and computation introduced by one domain with and without the proposed domain- adaative filter decomposition. The comparison reveals that addaiieadaptive filter decomposition not only delivers superior performances but also saves both parameters and computation significantly.

---

Under review as a conference paper at ICLR 2020
(a) Target domain image. (b) Before adaptation.
(c) After adaptation.
(d) Ground truth.
Qualitative results for domain adaptation segmentation. The samples are randomly selected from the validation subsets of Cityscapes.
Figure A.2:
(a) Source domain (GTA: video game images).
(b) Target domain (Cityscapes: real-world images).
Figure A.3: Dataset samples for segmentation experiments (video games + street views).
E PROOFS IN SECTION 4
Proof of Lemma We establish a few facts:
Fact 1. /V guarantees that, p 1T -,
|| 1/, 11 < V
(A.1)
where Jf det (V f) denotes the determinate of the Jacobian matrix of the mapping f R R The inequality can be verified by elementary calculation. When is a rigid motion then the r.h.s of equation A. zero.
12

---

Under review as a conference paper at ICLR 2020
Table A.I: Comparisons on additional parameters and computation introduced by one extra domain.
Comparisons are performed on VGG-16, with basis elements and the input size of 224 x 224.
Model
Regular VGG
VGG with DAFD
Parameters
14.71M 15.38G
0.0007M
Flops
10.75G
Fact p invertible, and odd symmetry of implies that p and thus p=1 are odd, namely (-u) = p (u).
Define
vi(u) 0b(x D,w) f(u)
J σb (Sr
((u+t=2)u(o())dd f( ((-o)dy
R22
(f - (8)ldž f( f(-v)do
32
ob
R2
and
(1())== := / σb
(f z x(u+ - f(-o)dv.
We have that
/
(w) (i(1))
+ -=)
f(-v))dāde (by os non-expansive)
32 R22
I 4 VT
JR3 /
I + (8)) w (2)l dääd (by Fact 1)
and thus
11 - 9 < 4 1|
(A.2)
When p is a rigid motion, 91 91.
Also, let
(2(u)= oz w) f())
- ob
æ(u + 2))(((()) f(-p f(-p-)(u)) (v))do (by Fact 2)
E2
zlu + p(o) 2)00(2)dz) f(-o))Jp(o)lda
σb
E2
and
(fe x(u (()) -)) /( s(-ū)dū.
(2))
RR
Similar to the proof of equation
A.2 one can verify that
l|ye
Ú2]1 S 4/V || ||
(A.3)
and the bound is zero when is rigid motion.
It remains to bound 0: Bzll1. Note that by og being non-expansive again
[i - (u)l < /
R2 J |x(u
v -- I x(u p(v) - =))
(A.4)
We claim that
- (2)) æuu + p(v) 2) < /V + 22r ()))|/lll
(A.5)
uniformly for and z. If true, with equation A.4 gives that
[ (u) - du( < V 2 + [
which proves the lemma together with equation A.2 and equation A.3
13

---

Under review as a conference paper at ICLR 2020
Proof of equation A.5
We verify that for any fixed v, 2,
uuutuup--0 - xuu+ p(t) =) = (x));
(A.6)
R2
by a direct calculation:
(l.h.s) - Vx (v - (p(1) - 2)]
(
Σ | p (=)]-
Then, combined with that v E 2 B thus |e| I 2 and 2 E 2 B and thus (2) S
V- 20' I 22' (r(0) = 0 by that T is odd, and then |=))-(>)) s | l (z))). (=))) the
r.h.s of equation A.6 2(2 2 V> /Vall, which proves equation A.5
Proof of Theorem I. We need a slightly generalized form of Lemma 1, which inserts multiple plain convolutional layers between * and *f. presented in Lemma 2
Under the setting of the theorem, in the generative CNNs.
X, = a(... -0 * w!- + b5 b(-1))
w! (1)-) + b)-1) (-1) bl-1))
(A.7)
X, af....o. oh h * w (-L)_ + 1(-2)).
wi
(A.8)
where wy and 0 are defined by,
L, 11
w Diw!! #0 w/
(1)
+ b/ 11 w + D
(A.9)
The notation a00) stands for the 1-th layer output in the target net from the input in the bottom ((-L)-
th) layer as ()-L) h, a((0) Xx, and a for that from zero input the bottom. In the feature CNNs, the L-th layer outputs are
F. a... (X, (t + (LD) w (L) + +b(())
(A.10)
F. al o (X, (1) + (()) w
(L) + b((
(A.11)
where for 1 1= 1,... T.
1 n D 6 b
The proof is by applying Lemma recursively the pair of layers indexed by and -1, from 1 to Denote wo! by (1) then w () ( where D_ = D: = Da I = 1, 1, ..,.. L. We also denote 6.") by b(2) and keep notation b:² for negative L.
First, 1 = 1, in the target net,
{(1) == ((a(((11) x D1w(-1) + -(- * D1 Diw + b60))
Use the centering -) = ((11) - -i((-)) it can be written as
,(1) (aa(a(-1) * (-1 + + = D
+ ((= D Dww + b(1))
(A.12)
= =o(a(T!(")) * (- (5 w 1) + b(11)) Diw(() (1) + (()) (by equation A.9
(A.13)
Applying Lemma (or Lemma for this case), taking ff11) w11 + b(-1) as the effective "b", we have that (using the non-expansiveness of a to take outside the last σ)
(1) ((o(a((-) * wo(") + 1n ((-1) * (-1) + b((1)) * ()) + (()) + (1)
(A.14)
a ((((((11) * no (-1) + b ((1)) * (1) + B()) + (1)
(A.15)
= (1) +,))) (1)
(A.16)
where, since (111), (1)) are supported on 2 B,
||(0 48 2 \ 11 + 21136 21166-111
(A.17)
Next,
((2) o((()) 12w + b(2))
(A.18)
((((L)) ((1))= Dz ) + b(2)) (by equation AA
(A.19)
=(((1) (2) + b(2)) + (1))
(A.20)
14

---

Under review as a conference paper at ICLR 2020
where ||-0 ||(() and observe the same bound equation A.17 since neither +w
(Lemma () nor applying o with bias expands the I-norm. Using the brief notation 0j to denote the non-linear mapping with biases 60), consider
02(((1) D2w(l)) 02(01(0-1(((11) w w(-1) wlll) D2wll2)
=0(00(001(0((E-> Dzwl -22) + bl-2)) wo w(l)) D2wll))
(2(00(0-1(0(EE Dzwl (2) + 2/) * w((-)) + 1r 2))
+/
= wlll) D2wll)), (by equation A.9
by Lemma 2 it equals (using the non con-expansiveness. of 02 take (3)) outside)
(2(01(0-1(0(E (-2-) - w- 2)) * w (1(11)) w w())) w (2)) + n(2)
= (0-00(0-_(0(( * w(12) + ol 2()))) * w(-1) * w(l) * w (2)) + (2)
== 2l2) +2)
where
||(2
4s
2 [2 + 2
(A.21)
Inserting back to equation A.20
. = =
(1)' (2)
thus || ((2) || bounded by the sum of equation A.17 and equation A.21
Continue the process, g(()) denotes the 1-th layer output in the source CNN (after 7 times correction in the target CNN) by feeding f(-t-1) from the (-()-th layer, where f(-(-1) is the output in the
(un-corrected) generative target CNN after the first (L - 1) layers. By that (--L) = x(-L) = h, and
that F2 = ((1), F. (("), repeating the argument times gives that
IF, Fill1 S 4E E
|1 111)
and when (Is P() are rigid motions, the 2nd term for each / vanishes.
We claim that
Claim 3. For I = - L, ... 1, V (1) || < Vh//,, and || s ||l|/
which suffices to prove the theorem.
Proof of Claim 3: No that in the bottom layer x((L) = ((-L) = h. For Fo - -...,,
|.s()) 1 = || - =
= (ā wi (1-1)) (1( w. (1-1))1 1) 1|2
< (|-(0-1) 1) ((( =000) wi (1-1)) 1|
(by that or non-expansive)
< (| - (by that |ml!
(
(1-1) 1- || < 1 and Lemma 3 ii)
Recursing the inequality gives that || 11 S ||/]]]1 Similarly,
V 7 ( || (V(o) w. (1(
< (V[ * w. 1111 (by that aj does not increase total variation)
[|V()(1-1) w/ (((-1))
I || |[ (by that (wo(-11) I 1 and Lemma 3
and thus |
10) 111 < /Vhl11. This proves Claim 3.
Lemma 2. Suppose filters w. fi. .ms f satisfy that the 1-norm are all bounded by 1, and
w and f are supported on B. The sequence of as, denoting non-linear function with bias, for
I 0..... m are non-expansive. D, spatial transform where T is odd and [V T I E < Then
mml (... σi (ao(x D+w) {1) fmn)
15

---

Under review as a conference paper at ICLR 2020
approximates
a. (0o w) * [i) ... fm) = Dj'1
up to an error whose I-norm bounded by
4s 2 Væ + 21|x111),
and the second term vanishes if (La - 7) is rigid motion.
Proof of Lemma 2 The proof uses the same technique in the proof of Lemma 1. Omitting sub- script R²2 in the integral, let
(u) = aml S....11) ool x(u adw+tr uu + ... + um + p (z)) dz)
f( f(-o))dv1) (m(--mm) (mmm )f u()-)
(u(u)) f f /mm()
.....1 ...ll/..001/ /
ool 2(u +
+ vm + pp - (2))w(2))d)
(--1)) .... fm( (m(-um) um (-um) )5( By Fact 1, that of are all non-expansin non-expansive
that the 1 n-norm of all the filters are bounded by 1,
/ lva(u) - 8n (u)ldu & 4=/|=//1-
Also,
v2(u)= = f Smmlf ... oal o(u + ... + um + p(v) - 2))(())22)
f( f(-vi)dus) ... (m((-m))dum (odddum((-u)lddde
02(u)= = fmml) mml ...01l) ool x(u +uu+u 0s ... + Um + p(v) - =))o(z)d2)
f(-vi)du1) (m((-m)ddm ) (-uuddo.
Similarly,
(v2(4) - (2(u)ldu S 4=||==11-
Same as before, with p being a rigid motion, 1181 - d1ll and ll32 d2ll are both zero. remains to bound 0/ - d2ll2- Observe that del (els((v))ddn))) |/ (-Um) den
f | (u) j2(u))du <
du|z(u + 01 +... ... Um p-- x(u + 01 ... + um + p(v) 2)}, (A.22)
and similarly in proving Lemma 1, one verify that for any fixed U1, Um, v, 2,
f |rru + 01
2m
(2)) x(u + in + +.... + Um + p(v) 2) du
V V boolo /o p (z)l < 22 + Vall..
Inserting back to equation A.22 and again by that the 1-norm of all the filters are bounded by 1, have that || U2ll1 S 4 Va
Lemma 3. Let and be smooth and compactly supported on R.? then (i) I| w|l1 S || /w]la-
(ii) (I w)ll V
Proof of Lemma
3 For (i)
I(u v)w(t)dulduc /
/ læ(u v)llw()))dudr
|| + w|ll
|
For (ii), g2 R22
J82 R2
/ Vol
[v
x(u ())=))w(o)du)))
Jzz
R2
/
Vuz(u- v)
Vuzluu - ())(o-))lloo))ddddd
Jga JR2
16


Reviews:
[{"review_id": "HyxcQKPMqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to do/contribute.\n* The paper proposes to perform domain adaptation via separating domain-specific and cross-domain features, by what is referred to as \"domain-adaptive filter decomposition\". Each domain contributes its own share of features to be combined by a subsequent common layer. The method was benchmarked against competing methods on simple classification tasks and a hard semantic segmentation task.\n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\nWeak Accept.\n* I think the paper was very well written, the explanations were clear and the technical contributions seem sound.\n* The experiments were satisfying for the most part. I would have wanted to see MNIST->SVHN for the unsupervised case as well, as this is a particularly hard one.\n\nProvide supporting arguments for the reasons for the decision.\n* Please do not use the Office dataset, it is commonly used in unsupervised domain adaptation papers, especially older ones, but it's hard to tell anything about proposed methods from this dataset as there is label pollution and not enough samples per class to be used with neural nets.\n* For GTA->Cityscapes you are missing a few works eg, the CYCADA work. Also please use citations in the tables if you did not yourselves run experiments (as to make it clear that experimental protocols also might be slightly different etc). \n\nProvide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n* Tables 1 & 2: It'd be better to not refer to methods as A1,2,3 but rahter with some specific names or explicitly describe what these abbreviations mean in the caption of the table.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeF6eIsFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to learning domain invariant representations using the adaptive decomposition of the convolutional filters. The approach is similar to methods that use multi-stream networks (a stream for each domain), but using the filer decomposition scheme, the authors avoid the issue of excessive increase in the number of parameters typical in fully multi-stream architectures. \n\nThis is achieved by learning a separate basis for convolutional filters for each domain while sharing the basis coefficients across domains. This encourages shared semantics across domains while maintaining a balance between the network expressiveness and the computational complexity.\n\nThe authors argue that the basis learned for each domain can be understood as correction/alignment mappings to bring together the representations of each domain. A toy example presented is convincing and shows the correction basis learned in a simple synthetic case. Theoretical arguments are provided to show that the proposed correction scheme covers a large range of possible domain shifts.\n\nThe authors also show that plugging the decomposition scheme into existing CNN based unsupervised domain adaptation algorithms results in consistent improvements across methods and datasets. \n\nHow does the method compare to approaches that learn to adapt the representations using conditional/adaptive batch norm [1,2]? \n\nOverall, the paper was well motivated and easy to read. The methods appear to be a useful addition to tools available for domain invariant learning.\n\n[1] Chang, Woong-Gi, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. \"Domain-Specific Batch Normalization for Unsupervised Domain Adaptation.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362. 2019.\n[2] Kumar, Abhishek, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Freeman, and Gregory Wornell. \"Co-regularized alignment for unsupervised domain adaptation.\" In Advances in Neural Information Processing Systems, pp. 9345-9356. 2018.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xOyyzgqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a way to decompose features for better domain adaptation via learning domain-invariant representations. The main advantage of the proposed approach is that by only introducing a few model parameters, the proposed approach could quickly adapt to new domains. Similar idea has been proposed in [1], where the authors propose to use domain-specific encoders to extract domain-invariant features. Compared with [1], the main novelty of this paper lies in a new feature decomposition of the traditional convolution layer.\n\nMy main concern is that this paper seems to miss a significant line of recent work on learning domain-invariant representations [2-3]. Basically, it has been shown that invariant representations provably hurt generalization on the target domain when the marginal label distributions are different between the source and target domains. Note that such result also holds when different feature extractors are used in source and target domains, so the model proposed in this manuscript is also subject to such inherent tradeoffs. \n\n\n[1].    Domain Separation Networks, NIPS 2016.\n[2].    On Learning Invariant Representations for Domain Adaptation, ICML 2019.\n[3].    Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment, ICML 2019.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a domain-adaptive filter decomposition method via separating domain-specific and cross-domain features, towards learning invariant representations for unsupervised domain adaptation.

Overall, this well-written paper is well motivated with a better technique for learning invariant representations using convolutional filters. Nonetheless, reviewers still have major concerns: 1) the novelty of the paper may be marginal given the significant line of recent work on learning domain-invariant representations; 2) when the label distributions differ, learning invariant representations can only lead to worse target generalizations; 3) the provided theory has an unclear connection to the presented filter decomposition method. The paper can be strengthened by further discussions on how to mitigate the aforementioned negative results. 

Hence I recommend rejection.
Decision: Reject