Title: DOMAIN-INVARIANT LEARNING USING ADAPTIVE FILTER DECOMPOSITION

Abstract: Domain shifts are frequently encountered in real-world scenarios. In this paper, we consider the problem of domain-invariant deep learning by explicitly modeling domain shifts with only a small amount of domain-specific parameters in a Convolutional Neural Network (CNN). By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we show for the first time, both empirically and theoretically, that domain shifts can be effectively handled by decomposing a regular convolutional layer into a domain-specific basis layer and a domain-shared basis coefficient layer, while both remain convolutional. An input channel will now first convolve spatially only with each respective domain-specific basis to "absorb" domain variations, and then output channels are linearly combined using common basis coefficients trained to promote shared semantics across domains. We use toy examples, rigorous analysis, and real-world examples to show the framework's effectiveness in cross-domain performance and domain adaptation. With the proposed architecture, we need only a small set of basis elements to model each additional domain, which brings a negligible amount of additional parameters, typically a few hundred.

1. INTRODUCTION:

Training supervised deep networks requires large amount of labeled training data; however, welltrained deep networks often degrade dramatically on testing data from a significantly different domain. In real-world scenarios, such domain shifts are introduced by many factors, such as different illumination, viewing angles, and resolutions. Research topics such as transfer learning and domain adaptation are studied to promote invariant representations across domains with different levels of availabilities of annotated data.

Recent efforts on learning cross-domain invariant representations using deep networks generally fall into two categories. The first one is to learn a common network with constraints encouraging invariant feature representations across different domains Long et al. (2015; 2016b) ; Tzeng et al. (2014) . The feature invariance is usually measured by feature statistics like Maximum Mean Discrepancy (MMD), or feature discriminators using adversarial training Ganin et al. (2016) . While these methods introduce no additional model parameters, the effectiveness largely depends on the degree of domain shifts. The other direction is to explicitly model domain specific characteristics with a multi-stream network structure where different domains are modeled by corresponding sub-networks at the cost of significant extra parameters and computations Rozantsev et al. (2018b) . Regularizations are imposed among sub-networks to encourage common semantics across domains.

In this paper, we model domain shifts through domain-adaptive filter decomposition (DAFD) with layer branching. At a branched layer, we decompose each filter over a small set of domain-specific basis elements to model intrinsic domain characteristics, while enforcing shared cross-domain coefficients to align invariant semantics. A regular convolution is now decomposed into two steps. First, a domain-specific basis convolves spatially only each individual input channel for shift "correction." Second, the "corrected" output channels are weighted summed using domain-shared basis coefficients (1×1 convolution) to promote common semantics. When domain shifts happen in space, we rigorously prove that such layer-wise "correction" by the same spatial transform applied to bases suffices to align the learned features.

Comparing to the existing subnetwork-based methods, the proposed method has several appealing properties: First, only a very small amount of additional trainable parameters are introduced to explicitly model each domain, i.e., domain-specific bases. The majority of the parameters in the network remain shared across domains, and learned from abundant training data to effectively avoid overfitting. Furthermore, the decomposed filters reduce the overall computations significantly compared to previous works, where computation typically grows linearly with the number of domains.

We conduct extensive real-world face recognition, image classification, and segmentation experiments, and observe that, with the proposed method, invariant representations across domains are consistently achieved without compromising the performance of individual domain.

Our main contributions are summarized as follows:

• We propose domain-invariant representation learning through bases decomposition with layer branching, where domain-specific bases are learned to counter domain shifts, and semantic alignments are enforced with cross-domain common basis coefficients. • We both theoretically prove and empirically observe that by stacking the bases-decomposed branched layer, invariant representations across domains are achieved progressively. • The majority of network parameters remain shared across domains, which alleviates the demand for massive annotated data from every domain, and introduces only a small amount of additional computation and parameter overhead. Thus the proposed approach serves as an efficient way for domain invariant learning. 

2. RELATED WORK:

Recent achievements on domain-invariant learning generally follow two directions. The first direction is learning a single network, where parameters are shared across domains, while the network is encouraged to produce domain-invariant features by minimizing additional loss functions in the network training Ganin et al. (2016) ; Long et al. (2015; 2016a; b) ; Tzeng et al. (2014) . The Maximum Mean Discrepancy (MMD) Gretton et al. (2007) , and MK-MMD Gretton et al. (2012) in Long et al. (2015) , are adopted as the discrepancy metric among domains. Beyond the first order statistic, second-order statistics are utilized in Hoffman et al. (2014) . Besides the hand-crafted distribution distance metrics, Ganin et al. (2016) ; Tzeng et al. (2015) ; Long et al. (2018) resort to adversarial training and achieve superior performances. Various distribution alignment methods, e.g., Wu et al. (2019) ; Kumar et al. (2018) , are proposed to improve the invariant feature learning. While effective in certain scenarios, the performance of learning invariant features using a shared network is largely constrained by the degree of domain shift as discussed in Rozantsev et al. (2018b) . Meanwhile, some recent works like Wu et al. (2019) ; Zhao et al. (2019) suggest important insights on whether it is sufficient to do domain adaptation by invariant representation and small empirical source risk, which shed light on exploring more effective alignment methods that are robust to common issues like different marginal label distributions.

Another popular direction is modeling each domain explicitly using auxiliary network structures.

Based on the hypothesis that explicitly modeling what is unique to each domain can improve a model's ability to extract domain-invariant features, e.g., Bousmalis et al. (2016) proposes feature representation by two components where domain similarities and shifts are modeled by a private component and a shared component separately. A completely two-stream network structure with no shared parameters is proposed in Rozantsev et al. (2018b) , where auxiliary residual networks are trained to adapt the layer parameters of the source domain to the target domain. Chang et al. (2019) proposes attacking domain shifts by domain-specific batch normalization, which we believe is compatible with the proposed domain-specific filter decomposition for better performance. Another popular direction for domain adaptation is to remap the input data between the source and the target domain for domain adaptation Murez et al. (2017) ; Hu et al. (2018) ; Hoffman et al. (2018) , which is not included in the discussion since we are focusing on learning invariant feature space. Finally, learning invariance is of relevance beyond domain adaptation, e.g., in the field of causal inference Bühlmann (2018).

3. DOMAIN-ADAPTIVE FILTER DECOMPOSITION:

A straightforward way to address domain shifts is to learn from multi-domain training data a single network as in Figure 1 (a). However, the lack of explicitly modelling of individual domains often results in unnecessary information loss and performance degradation as discussed in Rozantsev et al. (2018b) . Thus, we often simultaneously observe underfitting for domains with abundant training data, and overfitting for domains with limited training. In this section, we start with a simplistic pedagogical formulation, domain-adaptive layer branching as in Figure 1 (b), where domain shifts are modeled by a respective branch of filters in a layer, one branch per domain. Each branch is learned only from domain-specific data, while non-branched layers are learned from data from all domains.

We then propose to extend basic branching to bases-decomposed branching as in Figure 1 (c), where domain characteristics are modeled by domain-specific bases, and shared decomposition coefficients are enforced to align cross-domain semantics.

3.1. PEDAGOGICAL BRANCHING FORMULATION:

We start with the simple-minded branching formulation in Figure 1 (b). To model the domainspecific characteristics, at the first several convolutional layers, we dedicate a separate branch to each domain. Domain shifts are modeled by an independent set of convolutional filters in the branch, trained respectively with errors propagated back from the loss functions of source and target domains. For supervised learning, the loss function is the cross-entropy for each domain. For unsupervised learning, the loss function for the target domain can be either the feature statistics loss or the adversarial loss. The remaining layers are shared across domains. We assume one target domain and one source domain in our discussion, while multiple domains are supported. Note that, though we adopt the source vs. target naming convention in the domain adaptation literature, we address here a general domain-invariant learning problem.

Domain-adaptive branching is simple and straightforward, however, it has the following drawbacks: First, both the number of model parameters and computation are multiplied with the number of domains. Second, with limited target domain training data, we can experience overfitting in determining a large amount of parameters dedicated to that domain. Third, no constraints are enforced to encourage cross-domain shared semantics. We address these issues through layer branching with the proposed bases decomposition.

3.2. BASES-DECOMPOSED BRANCHING:

To simultaneously counter domain shifts and enforce cross-domain shared semantics, we decompose each convolutional filter in a branched layer into domain-specific bases, and cross-domain shared coefficients, as illustrated in Figure 1(c ). In our approach, as shown in Figure 2 (b), we decompose source and target domain filters over domain-specific bases, with decomposition coefficients shared across domains. Specifically, at each branched layer, the source domain filter W s and target domain filter W t of size L × L × C × C, are decomposed into ψ s × a and ψ t × a, where ψ s and ψ t with a size of L × L × K are the domainadaptive bases for source and target domains, respectively; and a ∈ R K×C ×C denotes the common decomposed coefficients shared across domains. Different from Qiu et al. (2018) , the domainadaptive bases are independently learned from the corresponding domain data to model domain shifts, and the shared decomposed coefficients are learned from the massive data from multiple domains. Note that thanks to this proposed structure, only a small amount of additional parameters is required here to model each additional domain, typically a few hundred.

With the above domain-adaptive filter decomposition, at each branched layer, a regular convolution is now decomposed into two: First, a domain-specific basis convolves each individual input channel for domain shift "correction." Second, the "corrected" output channels are weighted summed using domain-shared basis coefficients (1×1 convolution) to promote common semantics. A toy example is presented in Figure A.1 for illustrating the intuition behind the reason why manipulating bases alone can address domain shifts. We generate target domain data by applying two trivial operations to source domain images: First, every 3×3 non-overlapping patch in each image is locally rotated by 90 o . Then, images are negated by multiplying with -1. Domain-invariant features are observed by manipulating bases alone. We will rigorously prove in Section 4 why such layer-wise "correction" aligns features across domains, and present real-world examples in the experiments.

Parameters and Computation Reduction. Suppose that both input and output features have the same spatial resolution of W × W , in each forward pass in a regular convolutional layer, there are totally 

W 2 × C × C × (2L 2 + 1)

4. PROVABLE INVARIANCE WITH ADAPTIVE BASES:

In this section, we theoretically prove that the features produced by the source and target networks from domain-transferred inputs can be aligned by the proposed framework of only adjusting multi-layer bases, assuming a generative model of the source and target domain images via CNN. Since convolutional generative networks are a rich class of models for domain transfer Hu et al. (2018) ; Murez et al. (2017) , our analysis provides a theoretical justification of the proposed approach. Diverse examples in the experiment section shows the applicability of the proposed approach is potentially larger than what is proved here. All proofs are in the supplementary material.

Filter Transform via Basis Transform. Let w s and w t be the filters in the branched convolutional layer for the source and target domains respectively, and similarly denote the source and target bases by ψ k,s and ψ k,t . In the proposed bases decomposition DA architecture, the source and target domain filters are linear combinations of the domain-specific basis elements with shared decomposition coefficients, namely

w s (u) = k a k ψ k,s (u), w t (u) = k a k ψ k,t (u).

Certain transforms of the filter can be implemented by only transforming the bases, including

(1) A linear correspondence of filter values. Let λ : R → R be a linear mapping, by linearity,

ψ k,s (u) → ψ k,t (u) = λ(ψ k,s (u)) applies w s (u) → w t (u) = λ(w s (u)). E.g. the negation λ(ξ) = -λ(ξ), as shown in Figure A.1.

(2) The transformation induced by a displacement of spatial variable, i.e., "spatial transform" of filters, defined as D τ w(u) = w(u -τ (u)), where τ : R 2 → R 2 is a differentiable displacement field. Note that the dependence on spatial variable u in a filter is via the bases, thus ψ k,s → ψ k,t = D τ ψ k,s applies w s → w t = D τ w s . If such filter adaptations are desired in the branching network, then it suffices to branch the bases while keeping the coefficients a k shared, as implemented in the proposed architecture shown in Figure 1 (c). A fundamental question is thus how large is the class of possible domain shifts that can be corrected by these "allowable" filter transforms. In the rest of the section, we show that if the domain shifts in the images are induced from a generative CNN where the filters for source and target differ by a sequence of allowable transforms, then the domain shift can be provably eliminated by another sequence of filter transforms which can be implemented by bases branching only.

Correction of a Single Filter Transform. We first analyze the "symmetric" correction of one filter spatial transform D τ in one layer. The inclusion of linear correspondence transform is more direct (see comments after Theorem 1). For technical reasons, we assume that the displacement field τ is a small distortion, namely ∇τ ∞ 1, and then D τ is invertible. Example includes rotation by a small angle and a small factor rescaling (dilation).

For simplicity we only consider one input and output channel in each of the multiple convolutional layers. The argument extends to multiple channels by modifying the boundedness condition of the filters. Then the forward mapping in one convolutional layer can be written as y = σ(x * w + b), where x is the input activation, y is the output, w is the filter, b is the constant bias, and σ is the nonlinear activation function, e.g., ReLU. As we take a continuous formulation in the analysis, the activations x and y are assumed to be smooth functions supported on domain Ω ⊂ R 2 , typically Ω = [-1, 1] 2 . The filter w is a function supported on 2 j B, B being the unit disk, and 2 j is layer scale (diameter of filter patches) . The 1-norm of a function is defined to be x 1 = R 2 |x(u)|du. Lemma 1. Suppose that the two filters w, f are supported on 2 jw B and 2 j f B respectively. σ : R → R is non-expansive, D τ is a spatial transform where τ is odd, i.e., τ (-u) = -τ (u), and

|∇τ | ∞ < 1 5 . Then σ b (x * D τ w) * f -σ b (x * w) * D -1 τ f 1 ≤ 2|∇τ | ∞ w 1 f 1 (2 jw + 2 j f ) ∇x 1 + 4 x 1 ,

where σ b denotes the nonlinear function with the bias. The second term vanishes if (I d -τ ) is a rigid motion, e.g., rotation.

Provable Invariance under a Generative Model of Images. Stacking the approximate commuting relation, Lemma 1, in multiple layers allows to correct a sequence of filter transforms in previous convolutional layers by another sequence of "symmetric" ones. This means that if we impose a convolutional generative model on the source and target input images, and assume that the domain transfer results from a sequence of spatial transforms of filters in the generative net, then by correcting these filter transforms in the subsequent convolutional layers we can guarantee the recovery of the same feature mapping. The key observation is that the filter transfers can be implemented by bases transfer only.

We summarize the theoretical assumptions as follows:

(A1) The σ in any layer is non-expansive, (A2) In the generative net (where layer is indexed by negative integers), w

(-l) t = D l w (-l) s , where D l = D τ l , τ l is odd and |∇τ l | ∞ ≤ ε < 1 5 for all l = 1, • • • , L.

The biases in the target generative net are mildly adjusted accordingly due to technical reasons (to preserve the "baseline output" from zero-input, c.f. the proof). (A3) In the generative net, w (-l) s 1 ≤ 1 for all l, and so is w

(-l) t = D l w (-l) s

. Same for the feed-forward convolutional net taking the generated images as input, called "feature net": The source net filters have w 

| D τ w 1 - w 1 | ≤ c|∇τ | ∞ w 1 which

is negligible when ε is small. Thus in (A3) the boundedness of the 1norm of the source and target filters imply one another exactly or approximately. The boundedness of 1-norm of the filters preserves the non-expansiveness of the mapping from input to output in a convolutional layer, and in practice is qualitatively preserved by normalization layers. Also, as a typical setting, (A3) assumes that the scales j l in the generative net (the (-l)-th layer) and the feature net (the l-th layer) are matched, which simplifies the analysis and can be relaxed. Theorem 1. Suppose that X s and X t are source and target images generated by L-layer generative CNN nets with source and target filters w

(-l) s , w (-l) t

respectively from the common representation h. Under (A1)-(A3), the output at the L-th layer of the target feature CNN from X t , by setting w

(l) t = D l w (l)

s in all layers which can be implemented by bases branching, approximates that of the source feature CNN from X s up to an error which is bounded in 1-norm by

4ε ( L l=1 2 j l ) ∇h 1 + 2L h 1 ,

and the second term vanishes if (I d -τ l ) are rigid motions, e.g., rotation.

Note that typically j 1 ≤ • • • ≤ j L and 2 J L ≤ 1, and l 2 j l is proportional to the diameter of Ω and thus an O(1) constant. The result can be extended to allow a linear transform of the filter values, namely Dw(u) = λ(D τ w(u)), and correct the target filters f t to be λ -1 (D τ f s ). In Lemma 1, λ commutes with the convolution by linearity, and also with σ by adjusting the bias parameter if needed. Unlike D τ which becomes D -1 τ when applying to f , λ remains as λ, thus λ -1 is needed in the symmetric basis transfers to correct the domain shift. Proofs are in the supplementary material.

5. EXPERIMENTS:

In this section, we perform extensive experiments to evaluate the performance of the proposed domain-adaptive filter decomposition. We start with the comparisons among the 3 architectures listed in Figure 1 on two supervised tasks. To demonstrate the proposed framework as one principled way for domain-invariant learning, we then conduct a set of domain adaptation experiments. There we show, by simply plugging the proposed domain filter decomposition into regular CNNs used in existing domain adaptation methods, we consistently observe performance improvements, which illustrates that the framework can be successfully applied with multiple leading deep network architectures and very different datasets and applications.

5.1. ARCHITECTURE COMPARISONS:

We start with two supervised tasks performed on the three architectures listed in Figure 1 , regular CNN (A1), basic branching (A2), and branching with domain-adaptive filter decomposition (A3). In each layer of the network with the proposed domain-adaptive filter decomposition, the different domain features convolve spatially with the corresponding domain-specific bases first, then the output features are linearly combined by the cross-domain basis coefficients simultaneously. For both tasks, the networks are trained end-to-end with a summed loss for domains, and the domain-specific bases are only updated by the error from the corresponding domain, while the basis coefficients are updated by the joint error across domains. 1 .

Supervised Simultaneous Cross-domain Face Recognition. To show the generality of the proposed domain-adaptive filter decomposition for various tasks, we further perform a supervised crossdomain (RGB and near infrared) simultaneous face recognition experiment in the supplementary material Section B, and the quantitative comparisons are in Table 3 (b), where we can clearly observe that DAFD performs superiorly even with a missing input domain. Note that A2 requires additional 14.71M parameters over A1, while our method requires only 0.0007M as shown in Table A .1.

5.2. EXPERIMENTS ON STANDARD DOMAIN ADAPTATIONS:

In this section, we perform extensive experiments on unsupervised domain adaptation. Note that the objective of the experiments in this section is not to validate the proposed domain-adaptive filter decomposition as just another new method for domain adaptation. Instead, since most of the state-ofthe-art domain adaptation methods adopt the regular CNN (A1) with completely shared parameters for domains, we show the compatibility and the generality of the proposed domain-adapting filter decomposition by plugging it into underlying domain adaptation methods, and evaluate the effectiveness by retraining the networks using exactly the same setting and observing the performance improvement over the underlying methods. Diverse real-world domain shifts including different sensors, different image sources, and synthetic images, and applications on both classification and segmentation are examined in these experiments. Together with the experiments in the previous section, this further stresses the plug-and-play virtue of the proposed framework.

In practise, instead of learning independent source and target domain bases, we learn the residual between the source and the target domain bases. The residual is initialized by full zeros, and trained by loss for encouraging invariant features in the underlying methods, e.g., the adversarial loss in ADDA Tzeng et al. (2017) . We consistently observe that this stabilizes the training and promotes faster convergence. Digit Classification. We perform experiments on three public digits datasets: MNIST, USPS, and Street View House Numbers (SVHN), with three transfer tasks: USPS to MNIST (U → M), MNIST to USPS (M → U), and SVHN to MNIST (S → M). Classification accuracy on the target domain test set samples is adopted as the metric for measuring the performance. We perform domain-adaptive domain decomposition on state-of-the-art methods DANN Ganin et al. (2016) , ADDA Tzeng et al. (2017), and CDAN+E Long et al. (2018) . Quantitative comparisons are presented in Table 2 .

Office-31. Office-31 is one of the most widely used datasets for visual domain adaptation, which has 4,652 images and 31 categories collected from three distinct domains: Amazon (A), Webcam (W), and DSLR (D). We evaluate all methods on six transfer tasks A → W, D → W, W → D, A → D, D → A, and W → A. Two feature extractors, AlexNet Krizhevsky et al. (2012) and ResNet He et al. ( 2016) are adopted for fair comparisons with underlying methods. Specifically, ImageNet initialization are widely used for ResNet in the experiments with Office-31, and we consistently observe that initialization is important for the training on Office-31. Therefore, when training ResNet based networks with domain-adaptive filter decomposition, we initialize the feature extractor using parameters decomposed from ImageNet initialization. The quantitative comparisons are in Table 3 .

Table 3 : Accuracy (%) on Office-31 for unsupervised domain adaptation (AlexNet and ResNet). Image Segmentation. Beyond image classification tasks, we perform a challenging experiment on image segmentation to demonstrate the generality of the proposed domain-adaptive filter decomposition. We perform unsupervised adaptation from the GTA dataset Richter et al. ( 2016 A TOY EXPERIMENT 

Method A → W D → W W → D A → D D → A W → A Avg. AlexNet AlexNet (

B SUPERVISED CROSS-DOMAIN FACE RECOGNITION:

Besides standard domain adaptation, the proposed domain-adaptive filter decomposition can be extended to general tasks that involves more than one visual domain. Here we demonstrate this by performing experiments on supervised cross-domain face recognition. We adopt the NIR-VIS 2.0, which consists of 17,580 NIR (near infrared) and VIS (visible light) face images of 725 subjects, and perform cross-domain face recognition. We adopt VGG16 as the base network structure, branch all the convolutional layers with the proposed domain-adaptive filter decomposition, and train the network from scratch. In each convolutional layer, two bases are trained for modeling the NIR and the VIS domain, respectively. Specifically, one VIS image and one NIR image are fed simultaneously to the network, and the feature vectors of both domains are averaged to produce the final cross-domain feature, which is further fed into a linear classifier for classifying the identity. While the training is conducted using both domains simultaneously, we test the network under three settings including feeding single domain inputs only (VIS Acc and NIR Acc in 

D COMPUTATION AND PARAMETERS:

In Table A .1, we provide comparisons on additional parameters and computation introduced by one extra domain with and without the proposed domain-adaptive filter decomposition. The comparison reveals that domain-adaptive filter decomposition not only delivers superior performances but also saves both parameters and computation significantly. 

E PROOFS IN SECTION 4:

Proof of Lemma 1. We establish a few facts:

Fact 1. |∇τ | ∞ < 1 5 guarantees that, ρ := I d -τ , ||Jρ| -1|, ||Jρ -1 | -1| ≤ 4|∇τ | ∞ , (A.1)

where Jf = det(∇f ) denotes the determinate of the Jacobian matrix of the mapping f : R 2 → R 2 . The inequality can be verified by elementary calculation. When ρ is a rigid motion then the r.h.s of equation A.1 is zero. 

-ρ -1 (-u) = ρ -1 (u). Define y 1 (u) := σ b (x * D τ w) * f (u) = R 2 σ b R 2 x(u + v -z)w(ρ(z))dz f (-v)dv = R 2 σ b R 2

x(u + v -ρ -1 (z))w(z)|Jρ -1 (z)|dz f (-v)dv and ŷ1 (u

) := R 2 σ b R 2

x(u + v -ρ -1 (z))w(z)dz f (-v)dv.

We have that x(u + ρ(ṽ) -z)w(z)dz f (-ṽ)|Jρ(ṽ)|dṽ and ŷ2 (u

|y 1 (u) -ŷ1 (u)| ≤ R 2 R 2 |x(u + v -ρ -1 (z))||w(z)| |Jρ -1 | -1 |f (-v)|dzdv (by σ b non-expansive) ≤ 4|∇τ | ∞ R 2 R 2 |x(u + v -ρ -1 (z))

) := R 2 σ b R 2

x(u + ρ(ṽ) -z)w(z)dz f (-ṽ)dṽ.

Similar to the proof of equation A.2, one can verify that y 2 -ŷ2 1 ≤ 4|∇τ | ∞ x 1 w 1 f 1 , (A.3) and the bound is zero when ρ is a rigid motion. τ f up to an error whose 1-norm is bounded by 4ε 2 j ∇x 1 + 2 x 1 , and the second term vanishes if (I d -τ ) is a rigid motion.

Proof of Lemma 2. The proof uses the same technique as in the proof of Lemma 1. Omitting subscript R 2 in the integral, let

y 1 (u) = σ m ( • • • σ 1 ( σ 0 ( x(u + v 1 + • • • + v m + v -ρ -1 (z))w(z)|Jρ -1 |dz) f (-v 1 )dv 1 ) • • • f m (-v m )dv m )f (-v)dv, ŷ1 (u) = σ m ( • • • σ 1 ( σ 0 ( x(u + v 1 + • • • + v m + v -ρ -1 (z))w(z)dz) f (-v 1 )dv 1 ) • • • f m (-v m )dv m )f (-v)dv.

By Fact 1, that σ j are all non-expansive and that the 1-norm of all the filters are bounded by 1, 

|y 1 (u) -ŷ1 (u)|du ≤ 4ε x 1 . Also, y 2 (u) = σ m ( • • • σ 1 ( σ 0 ( x(u + v 1 + • • • + v m + ρ(v) -z)w(z)dz) f (-v 1 )dv 1 ) • • • f m (-v m )dv m )f (-v)|Jρ|dv, ŷ2 (u) = σ m ( • • • σ 1 ( σ 0 ( x(u + v 1 + • • • + v m + ρ(v) -z)w(z)dz) f (-v 1 )dv 1 ) • • • f m (-v m )dv m )f (-v)dv.

|x(u + v 1 + • • • + v m + v -ρ -1 (z)) -x(u + v 1 + • • • + v m + ρ(v) -z)|du ≤ ∇x 1 |∇τ | ∞ |v -ρ -1 (z)| ≤ ε2(2 j + 2 j ) ∇x 1 .

Inserting back to equation A.22, and again by that the 1-norm of all the filters are bounded by 1, we have that ŷ1 -ŷ2 1 ≤ 4ε2 j ∇x 1 . Lemma 3. Let x and w be smooth and compactly supported on R 2 , then (i) x * w 1 ≤ x 1 w 1 .

(ii) ∇(x * w) 1 ≤ ∇x 1 w 1 .

Proof of Lemma 3. For (i),

x * w 1 = R 2 | R 2 x(u -v)w(v)dv|du ≤ R 2 R 2 |x(u -v)||w(v)|dudv = x 1 w 1 . For (ii), ∇(x * w) 1 = R 2 |∇ u ( R 2 x(u -v)w(v)dv)|du = R 2 | R 2 ∇ u x(u -v)w(v)dv|du ≤ R 2 R 2 |∇ u x(u -v)||w(v)|dudv = ∇x 1 w 1 .

:



Figures and Tables:

Figure 1: Three candidate architectures considered for domain-invariant representation learning. In (a), a set of common network parameters are trained to model both source and target domains. In (b), the domain characteristics are explicitly modeled by two sets of convolutional filters in each convolutional layer. Our approach is illustrated in (c) where domain-adaptive bases are learned to "absorb" domain shifts, while the decomposition coefficients are shared across domains to promote common semantics. The feature space of the three candidate architectures, MNIST → SVHN, are visualized using t-SNE Maaten & Hinton (2008) in (d), (e), (f), respectively. The obtained crossdomain invariance can be clearly observed in (f).

(a) Decomposition over pre-fixed bases Qiu et al. (2018).

The proposed domain-adaptive filter decomposition.

Figure 2: Illustrations of the proposed domain-adaptive filter decomposition.Filter Decomposition. A convolutional filter can be well approximated as a linear combination of a small set of basis elementsQiu et al. (2018). In Figure2(a), a convolutional filter W with a size of L×L×C ×C is decomposed into a truncated expansion with pre-fixed bases in the spatial domain. The truncated bases ψ with a size L × L × K are predefined and remain fixed in training, where K is a hyperparameter, usually a small number, e.g., K = 5, indicating the number of basis elements used in each layer. The trainable parameters are thus reduced to the decomposition coefficients a with a size of K × C × C. With bases decomposition, a reduction rate of K L 2 is achieved in both model complexity and computation. This parameter saving is not the only virtue of basis decomposition, such decomposition is natural to extract both invariance across domains and adaptation to specific domains as detailed next.

flops for each domain. While in our model, each domain only introduces W 2 × C × 2K(L 2 + C) flops, where K is the number of basis elements. For parameters, there are totally D × C × C × L 2 parameters in a regular convolutional layer where D is the number of domains which is typically 2 in our case. In our model, each layer has only K × (C × C + D × L 2 ) parameters. Taking VGG-16 Simonyan & Zisserman (2014) as an example with an input size of 224 × 224, a regular VGG-16 with branching, Fig 1(b) and Rozantsev et al. (2018a;b), requires adding 14.71M parameters and 15.38G flops in convolutional layers to handle each additional domain. With the here proposed method (Fig 1(c)), a VGG-16 only requires adding 702 parameters and 10.75G flops to handle one additional domain (K=6).

for l = 1, 2 • • • , and same with D l w are both supported on 2 j l B for l = 1, • • • , L. One can show that D τ w 1 = w 1 when (I d -ρ) is a rigid motion, and generally

) (images generated from video games) to the Cityscapes dataset Cordts et al. (2016) (real-world images), which has a significant practical value considering the expensive cost on collecting annotations for image segmentation in real-world scenarios. Two underlying methods FCNs in the wild Hoffman et al. (2016) and AdaptSegNet Tsai et al. (2018) are adopted for comprehensive comparisons. Based on the underlying methods, all the convolutional layers are decomposed using domain-adaptive filter decomposition, and all the transpose-convolutional layers are kept sharing by both domains. For quantitative results in Table 4, we use intersection-over-union, i.e., IoU = TP TP+FP+FN , where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively, as the evaluation metric. As with the previous examples, our method improves all state-of-the-art architectures. Qualitative results are shown in Figure A.2, and data samples are in Section C.1.

Figure A.1: Visualization of the toy example. The two columns visualize the inputs, features, and bases of the source domain and the target domain, respectively. Only the output feature in the first channel of each convolutional layer is visualized for comparison. Domain invariant features, the last row, are obtained by manually adapting source domain bases to generated target domain bases.

(a) Target domain image. (b) Before adaptation. (c) After adaptation. (d) Ground truth.

Figure A.2: Qualitative results for domain adaptation segmentation. The samples are randomly selected from the validation subsets of Cityscapes.

Figure A.3: Dataset samples for segmentation experiments (video games → street views).

||w(z)||f(-v)|dzdv (by Fact 1) and thusy 1 -ŷ1 1 ≤ 4|∇τ | ∞ x 1 w 1 f 1 . (A.2) When ρ is a rigid motion, y 1 = ŷ1 . Also, let y 2 (u) := σ b (x * w) * D -1 τ f + v -z)w(z)dz f (-ρ -1 (v))dv (by Fact 2)

It remains to bound ŷ1 -ŷ2 1 . Note that by σ b being non-expansive again|ŷ 1 (u) -ŷ2 (u)| ≤ R 2 R 2 |x(u + v -ρ -1 (z)) -x(u + ρ(v) -z)||w(z)|dz|f (-v)|dv. (A.4) We claim that R 2 |x(u + v -ρ -1 (z)) -x(u + ρ(v) -z)|du ≤ |∇τ | ∞ 2(2 jw + 2 j f ) ∇x 1 (A.5)uniformly for v and z. If true, with equation A.4 it gives thatR 2 |ŷ 1 (u) -ŷ2 (u)|du ≤ |∇τ | ∞ 2(2 jw + 2 j f ) ∇x 1 w 1 f 1which proves the lemma together with equation A.2 and equation A.3.approximates σ m (• • • σ 1 (σ 0 (x * w) * f 1 ) • • • * f m ) * D -1

Similarly,|y 2 (u) -ŷ2 (u)|du ≤ 4ε x 1 .Same as before, with ρ being a rigid motion, y 1 -ŷ1 and y 2 -ŷ2 are both zero.It remains to bound ŷ1 -ŷ2 1 . Observe that|ŷ 1 (u) -ŷ2 (u)|du ≤ • • • dv|f (-v)|dv m |f (-v m )| • • • dv 1 |f (-v 1 )|dz|w(z)| du|x(u + v 1 + • • • + v m + v -ρ -1 (z)) -x(u + v 1 + • • • + v m + ρ(v) -z)|,(A.22)    and similarly as in proving Lemma 1, one can verify that for any fixed v 1 , • • • , v m , v, z,

Comparisons on supervised domain adaptaion and cross-domain simultaneous face recognition. A1, A2, and A3 correspond to regular CNN, basic branching, and branching with domainadaptive filter decomposition shown in Figure1, respectively. Supervised Domain Adaptation on Digits. The first task is supervised domain adaptation, where we adopt a challenging setting by using MNIST handwritten digit dataset as the source domain, and SVHN as the target domain. We perform a series of experiments by progressively reducing the annotated training data for the target domain. We start the comparisons at 10% of the target domain labeled samples, and end at 0.5% where only 366 labeled samples are available for the target domain. The results on test set for both domains are presented in Table3(a). It is clearly shown that when training the target domain with small amount of data, a network with basic branching suffers from overfitting to the target domain because of the large amount of domain specific parameters. While regular CNN generates well on target domain, the performance on source domain degrades when the number of target domain data is comparable. A network with the proposed domain-adaptive filter decomposition significantly balances the learning of both the source and the target domain, and achieves best accuracies on both domains regardless of the amount of annotated training data for the target domain. The feature space of the three candidate architectures are visualized in Figure

Table content: <table><tr><td>Scales A1 A2 A3</td><td>Source domain 0.1 0.05 0.005 0.1 0.05 0.005 Target domain 98.4 96.4 98.0 81.6 80.2 61.0 99.2 98.6 97.6 81.4 78.4 49.6 99.4 98.8 98.8 85.6 82.2 64.4</td><td>Methods VIS Acc (%) NIR Acc (%) NIR+VIS (%) A1 75.57 52.71 98.44 A2 94.46 87.50 98.58 A3 97.16 95.03 99.15</td></tr><tr><td colspan="2">(a) Accuracy (%) on both domains for super-</td><td>(b) Cross-domain simultaneous face recognition on NIR-</td></tr><tr><td colspan="2">vised domain adaptation.</td><td>VIS-2.0.</td></tr></table>

Accuracy (%) on Digits for unsupervised domain adaptation.

Table content: <table><tr><td>Methods</td><td colspan="2">M → U U → M</td><td>S → M</td><td>Avg.</td></tr><tr><td>DANN</td><td>-</td><td>-</td><td>73.9</td><td/></tr><tr><td>ADDA</td><td>89.4</td><td>90.1</td><td>76.0</td><td>85.1</td></tr><tr><td>CDAN+E</td><td>95.6</td><td>98.0</td><td>89.2</td><td>94.3</td></tr><tr><td>DANN + DAFD</td><td>92.0</td><td>95.2</td><td>82.1 (11.1% ↑)</td><td>89.8</td></tr><tr><td>ADDA + DAFD</td><td>91.4</td><td>94.8</td><td>82.9</td><td>89.7 (5.5% ↑)</td></tr><tr><td>CDAN+E + DAFD</td><td>96.8</td><td>98.8</td><td>96.6</td><td>97.4 (3.2% ↑)</td></tr></table>

Unsupervised DA for semantic segmentation: GTA → Cityscapes We present the intuitions of countering domain shifts by adapting bases through toy examples, and further provide theoretical analysis. Extensive experiments on multiple tasks validate that, by stacking domain-adaptive branched layers with filter decomposition, complex domain shifts in real-world scenarios can be bridged to produce domain-invariant representation, which are reflected by both experimental results and feature space visualizations, all this at virtual no additional memory or computational cost when adding domains.

Table content: <table><tr><td>Methods</td><td>IoU</td><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Class-wide IoU</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>ro a d</td><td>s id e w a lk</td><td>b u il d in g</td><td>w a ll</td><td>fe n c e</td><td>p o le</td><td>t-li g h t</td><td>t-s ig n</td><td>v e g</td><td>te rr a in</td><td>s k y</td><td>p e rs o n</td><td>ri d e r</td><td>c a r</td><td>tr u c k</td><td>b u s</td><td>tr a in</td><td>m b ik e</td><td>b ic y c le</td></tr><tr><td>No Adapt (VGG)</td><td>17.9</td><td>26.0</td><td>14.9</td><td>65.1</td><td>5.5</td><td>12.9</td><td>8.9</td><td>6.0</td><td>2.5</td><td>70.0</td><td>2.9</td><td>47.0</td><td>24.5</td><td>0.0</td><td>40.0</td><td>12.1</td><td>1.5</td><td>0.0</td><td>0.0</td><td>0.0</td></tr><tr><td>No Adapt (ResNet)</td><td>36.6</td><td>75.8</td><td>16.8</td><td>77.2</td><td>12.5</td><td>21.0</td><td>25.5</td><td>30.1</td><td>20.1</td><td>81.3</td><td>24.6</td><td>70.3</td><td>53.8</td><td>26.4</td><td>49.9</td><td>17.2</td><td>25.9</td><td>6.5</td><td>25.3</td><td>36.0</td></tr><tr><td>FCN WLD (VGG)</td><td>27.1</td><td>70.4</td><td>32.4</td><td>62.1</td><td>14.9</td><td>5.4</td><td>10.9</td><td>14.2</td><td>2.7</td><td>79.2</td><td>21.3</td><td>64.6</td><td>44.1</td><td>4.2</td><td>70.4</td><td>8.0</td><td>7.3</td><td>0.0</td><td>3.5</td><td>0.0</td></tr><tr><td>AdaptSegNet (VGG)</td><td>35.0</td><td>87.3</td><td>29.8</td><td>78.6</td><td>21.1</td><td>18.2</td><td>22.5</td><td>21.5</td><td>11.0</td><td>79.7</td><td>29.6</td><td>71.3</td><td>46.8</td><td>6.5</td><td>80.1</td><td>23.0</td><td>26.9</td><td>0.0</td><td>10.6</td><td>0.3</td></tr><tr><td>AdaptSegNet (ResNet)</td><td>42.4</td><td>86.5</td><td>36.0</td><td>79.9</td><td>23.4</td><td>23.3</td><td>23.9</td><td>35.2</td><td>14.8</td><td>83.4</td><td>33.3</td><td>75.6</td><td>58.5</td><td>27.6</td><td>73.7</td><td>32.5</td><td>35.4</td><td>3.9</td><td>30.1</td><td>28.1</td></tr><tr><td>FCN WLD + DAFD</td><td>32.7 (20.7% ↑)</td><td>76.4</td><td>36.7</td><td>68.8</td><td>17.6</td><td>5.8</td><td>11.1</td><td>13.9</td><td>2.9</td><td>80.0</td><td>24.4</td><td>69.1</td><td>47.5</td><td>4.3</td><td>74.4</td><td>14.1</td><td>6.3</td><td>0.0</td><td>2.1</td><td>0.0</td></tr><tr><td>AdaptSegNet (VGG) + DAFD</td><td>36.4 (4.0% ↑)</td><td>86.7</td><td>35.3</td><td>78.8</td><td>22.8</td><td>14.5</td><td>23.9</td><td>21.9</td><td>18.2</td><td>82.1</td><td>32.2</td><td>66.8</td><td>49.6</td><td>10.1</td><td>81.2</td><td>19.6</td><td>27.1</td><td>1.1</td><td>11.4</td><td>4.2</td></tr><tr><td>AdaptSegNet (ResNet) + DAFD</td><td>45.0 (6.1% ↑)</td><td>88.2</td><td>38.5</td><td>8.12</td><td>25.0</td><td>23.8</td><td>22.9</td><td>35.1</td><td>14.4</td><td>84.9</td><td>34.1</td><td>79.9</td><td>59.5</td><td>29.1</td><td>75.5</td><td>30.1</td><td>35.2</td><td>2.9</td><td>28.7</td><td>29.1</td></tr><tr><td colspan="2">6 CONCLUSION</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="21">We proposed to perform domain-invariant learning through domain-adaptive filter decomposition.</td></tr><tr><td colspan="21">To model domain shifts, convolutional filters in a deep convolutional network are decomposed over</td></tr><tr><td colspan="21">domain-adaptive bases to counter domain shifts, and cross-domain basis coefficients are constrained</td></tr><tr><td colspan="4">to unify common semantics.</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>(b)) and both</td></tr></table>

1: Comparisons on additional parameters and computation introduced by one extra domain. Comparisons are performed on VGG-16, with 6 basis elements and the input size of 224 × 224. is invertible, and odd symmetry of τ implies that ρ and thus ρ -1 are odd, namely

Table content: <table><tr><td>Model</td><td colspan="2">Regular VGG VGG with DAFD</td></tr><tr><td>Parameters</td><td>14.71M</td><td>0.0007M</td></tr><tr><td>Flops</td><td>15.38G</td><td>10.75G</td></tr><tr><td>Fact 2. ρ</td><td/><td/></tr></table>

Peer review (iclr2020):
The paper proposes an approach to learning domain invariant representations using the adaptive decomposition of the convolutional filters. The approach is similar to methods that use multi-stream networks (a stream for each domain), but using the filer decomposition scheme, the authors avoid the issue of excessive increase in the number of parameters typical in fully multi-stream architectures. This is achieved by learning a separate basis for convolutional filters for each domain while sharing the basis coefficients across domains. This encourages shared semantics across domains while maintaining a balance between the network expressiveness and the computational complexity. The authors argue that the basis learned for each domain can be understood as correction/alignment mappings to bring together the representations of each domain. A toy example presented is convincing and shows the correction basis learned in a simple synthetic case. Theoretical arguments are provided to show that the proposed correction scheme covers a large range of possible domain shifts. The authors also show that plugging the decomposition scheme into existing CNN based unsupervised domain adaptation algorithms results in consistent improvements across methods and datasets. How does the method compare to approaches that learn to adapt the representations using conditional/adaptive batch norm [1,2]? Overall, the paper was well motivated and easy to read. The methods appear to be a useful addition to tools available for domain invariant learning. [1] Chang, Woong-Gi, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. *Domain-Specific Batch Normalization for Unsupervised Domain Adaptation.* In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7354-7362. 2019. [2] Kumar, Abhishek, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Freeman, and Gregory Wornell. *Co-regularized alignment for unsupervised domain adaptation.* In Advances in Neural Information Processing Systems, pp. 9345-9356. 2018.