Domain-invariant Learning using Adaptive Filter Decomposition

1 INTRODUCTION . Training supervised deep networks requires large amount of labeled training data ; however , welltrained deep networks often degrade dramatically on testing data from a significantly different domain . In real-world scenarios , such domain shifts are introduced by many factors , such as different illumination , viewing angles , and resolutions . Research topics such as transfer learning and domain adaptation are studied to promote invariant representations across domains with different levels of availabilities of annotated data . Recent efforts on learning cross-domain invariant representations using deep networks generally fall into two categories . The first one is to learn a common network with constraints encouraging invariant feature representations across different domains Long et al . ( 2015 ; 2016b ) ; Tzeng et al . ( 2014 ) . The feature invariance is usually measured by feature statistics like Maximum Mean Discrepancy ( MMD ) , or feature discriminators using adversarial training Ganin et al . ( 2016 ) . While these methods introduce no additional model parameters , the effectiveness largely depends on the degree of domain shifts . The other direction is to explicitly model domain specific characteristics with a multi-stream network structure where different domains are modeled by corresponding sub-networks at the cost of significant extra parameters and computations Rozantsev et al . ( 2018b ) . Regularizations are imposed among sub-networks to encourage common semantics across domains . In this paper , we model domain shifts through domain-adaptive filter decomposition ( DAFD ) with layer branching . At a branched layer , we decompose each filter over a small set of domain-specific basis elements to model intrinsic domain characteristics , while enforcing shared cross-domain coefficients to align invariant semantics . A regular convolution is now decomposed into two steps . First , a domain-specific basis convolves spatially only each individual input channel for shift “ correction. ” Second , the “ corrected ” output channels are weighted summed using domain-shared basis coefficients ( 1×1 convolution ) to promote common semantics . When domain shifts happen in space , we rigorously prove that such layer-wise “ correction ” by the same spatial transform applied to bases suffices to align the learned features . Comparing to the existing subnetwork-based methods , the proposed method has several appealing properties : First , only a very small amount of additional trainable parameters are introduced to explicitly model each domain , i.e. , domain-specific bases . The majority of the parameters in the network remain shared across domains , and learned from abundant training data to effectively avoid overfitting . Furthermore , the decomposed filters reduce the overall computations significantly compared to previous works , where computation typically grows linearly with the number of domains . We conduct extensive real-world face recognition , image classification , and segmentation experiments , and observe that , with the proposed method , invariant representations across domains are consistently achieved without compromising the performance of individual domain . Our main contributions are summarized as follows : • We propose domain-invariant representation learning through bases decomposition with layer branching , where domain-specific bases are learned to counter domain shifts , and semantic alignments are enforced with cross-domain common basis coefficients . • We both theoretically prove and empirically observe that by stacking the bases-decomposed branched layer , invariant representations across domains are achieved progressively . • The majority of network parameters remain shared across domains , which alleviates the demand for massive annotated data from every domain , and introduces only a small amount of additional computation and parameter overhead . Thus the proposed approach serves as an efficient way for domain invariant learning . 2 RELATED WORK . Recent achievements on domain-invariant learning generally follow two directions . The first direction is learning a single network , where parameters are shared across domains , while the network is encouraged to produce domain-invariant features by minimizing additional loss functions in the network training Ganin et al . ( 2016 ) ; Long et al . ( 2015 ; 2016a ; b ) ; Tzeng et al . ( 2014 ) . The Maximum Mean Discrepancy ( MMD ) Gretton et al . ( 2007 ) , and MK-MMD Gretton et al . ( 2012 ) in Long et al . ( 2015 ) , are adopted as the discrepancy metric among domains . Beyond the first order statistic , second-order statistics are utilized in Hoffman et al . ( 2014 ) . Besides the hand-crafted distribution distance metrics , Ganin et al . ( 2016 ) ; Tzeng et al . ( 2015 ) ; Long et al . ( 2018 ) resort to adversarial training and achieve superior performances . Various distribution alignment methods , e.g. , Wu et al . ( 2019 ) ; Kumar et al . ( 2018 ) , are proposed to improve the invariant feature learning . While effective in certain scenarios , the performance of learning invariant features using a shared network is largely constrained by the degree of domain shift as discussed in Rozantsev et al . ( 2018b ) . Meanwhile , some recent works like Wu et al . ( 2019 ) ; Zhao et al . ( 2019 ) suggest important insights on whether it is sufficient to do domain adaptation by invariant representation and small empirical source risk , which shed light on exploring more effective alignment methods that are robust to common issues like different marginal label distributions . Another popular direction is modeling each domain explicitly using auxiliary network structures . Based on the hypothesis that explicitly modeling what is unique to each domain can improve a model ’ s ability to extract domain-invariant features , e.g. , Bousmalis et al . ( 2016 ) proposes feature representation by two components where domain similarities and shifts are modeled by a private component and a shared component separately . A completely two-stream network structure with no shared parameters is proposed in Rozantsev et al . ( 2018b ) , where auxiliary residual networks are trained to adapt the layer parameters of the source domain to the target domain . Chang et al . ( 2019 ) proposes attacking domain shifts by domain-specific batch normalization , which we believe is compatible with the proposed domain-specific filter decomposition for better performance . Another popular direction for domain adaptation is to remap the input data between the source and the target domain for domain adaptation Murez et al . ( 2017 ) ; Hu et al . ( 2018 ) ; Hoffman et al . ( 2018 ) , which is not included in the discussion since we are focusing on learning invariant feature space . Finally , learning invariance is of relevance beyond domain adaptation , e.g. , in the field of causal inference Bühlmann ( 2018 ) . 3 DOMAIN-ADAPTIVE FILTER DECOMPOSITION . A straightforward way to address domain shifts is to learn from multi-domain training data a single network as in Figure 1 ( a ) . However , the lack of explicitly modelling of individual domains often results in unnecessary information loss and performance degradation as discussed in Rozantsev et al . ( 2018b ) . Thus , we often simultaneously observe underfitting for domains with abundant training data , and overfitting for domains with limited training . In this section , we start with a simplistic pedagogical formulation , domain-adaptive layer branching as in Figure 1 ( b ) , where domain shifts are modeled by a respective branch of filters in a layer , one branch per domain . Each branch is learned only from domain-specific data , while non-branched layers are learned from data from all domains . We then propose to extend basic branching to bases-decomposed branching as in Figure 1 ( c ) , where domain characteristics are modeled by domain-specific bases , and shared decomposition coefficients are enforced to align cross-domain semantics . 3.1 PEDAGOGICAL BRANCHING FORMULATION . We start with the simple-minded branching formulation in Figure 1 ( b ) . To model the domainspecific characteristics , at the first several convolutional layers , we dedicate a separate branch to each domain . Domain shifts are modeled by an independent set of convolutional filters in the branch , trained respectively with errors propagated back from the loss functions of source and target domains . For supervised learning , the loss function is the cross-entropy for each domain . For unsupervised learning , the loss function for the target domain can be either the feature statistics loss or the adversarial loss . The remaining layers are shared across domains . We assume one target domain and one source domain in our discussion , while multiple domains are supported . Note that , though we adopt the source vs. target naming convention in the domain adaptation literature , we address here a general domain-invariant learning problem . Domain-adaptive branching is simple and straightforward , however , it has the following drawbacks : First , both the number of model parameters and computation are multiplied with the number of domains . Second , with limited target domain training data , we can experience overfitting in determining a large amount of parameters dedicated to that domain . Third , no constraints are enforced to encourage cross-domain shared semantics . We address these issues through layer branching with the proposed bases decomposition . 3.2 BASES-DECOMPOSED BRANCHING . To simultaneously counter domain shifts and enforce cross-domain shared semantics , we decompose each convolutional filter in a branched layer into domain-specific bases , and cross-domain shared coefficients , as illustrated in Figure 1 ( c ) . Filter Decomposition . A convolutional filter can be well approximated as a linear combination of a small set of basis elements Qiu et al . ( 2018 ) . In Figure 2 ( a ) , a convolutional filter W with a size of L×L×C ′×C is decomposed into a truncated expansion with pre-fixed bases in the spatial domain . The truncated bases ψ with a size L×L×K are predefined and remain fixed in training , whereK is a hyperparameter , usually a small number , e.g. , K = 5 , indicating the number of basis elements used in each layer . The trainable parameters are thus reduced to the decomposition coefficients a with a size of K × C ′ × C. With bases decomposition , a reduction rate of KL2 is achieved in both model complexity and computation . This parameter saving is not the only virtue of basis decomposition , such decomposition is natural to extract both invariance across domains and adaptation to specific domains as detailed next . In our approach , as shown in Figure 2 ( b ) , we decompose source and target domain filters over domain-specific bases , with decomposition coefficients shared across domains . Specifically , at each branched layer , the source domain filter Ws and target domain filter Wt of size L×L×C ′×C , are decomposed into ψs × a and ψt × a , where ψs and ψt with a size of L × L ×K are the domainadaptive bases for source and target domains , respectively ; and a ∈ RK×C′×C denotes the common decomposed coefficients shared across domains . Different from Qiu et al . ( 2018 ) , the domainadaptive bases are independently learned from the corresponding domain data to model domain shifts , and the shared decomposed coefficients are learned from the massive data from multiple domains . Note that thanks to this proposed structure , only a small amount of additional parameters is required here to model each additional domain , typically a few hundred . With the above domain-adaptive filter decomposition , at each branched layer , a regular convolution is now decomposed into two : First , a domain-specific basis convolves each individual input channel for domain shift “ correction. ” Second , the “ corrected ” output channels are weighted summed using domain-shared basis coefficients ( 1×1 convolution ) to promote common semantics . A toy example is presented in Figure A.1 for illustrating the intuition behind the reason why manipulating bases alone can address domain shifts . We generate target domain data by applying two trivial operations to source domain images : First , every 3×3 non-overlapping patch in each image is locally rotated by 90o . Then , images are negated by multiplying with −1 . Domain-invariant features are observed by manipulating bases alone . We will rigorously prove in Section 4 why such layer-wise “ correction ” aligns features across domains , and present real-world examples in the experiments . Parameters and Computation Reduction . Suppose that both input and output features have the same spatial resolution of W × W , in each forward pass in a regular convolutional layer , there are totally W 2 × C ′ × C × ( 2L2 + 1 ) flops for each domain . While in our model , each domain only introduces W 2 × C ′ × 2K ( L2 + C ) flops , where K is the number of basis elements . For parameters , there are totally D × C ′ × C × L2 parameters in a regular convolutional layer where D is the number of domains which is typically 2 in our case . In our model , each layer has only K× ( C ′×C+D×L2 ) parameters . Taking VGG-16 Simonyan & Zisserman ( 2014 ) as an example with an input size of 224 × 224 , a regular VGG-16 with branching , Fig 1 ( b ) and Rozantsev et al . ( 2018a ; b ) , requires adding 14.71M parameters and 15.38G flops in convolutional layers to handle each additional domain . With the here proposed method ( Fig 1 ( c ) ) , a VGG-16 only requires adding 702 parameters and 10.75G flops to handle one additional domain ( K=6 ) .

Summary: This paper introduces a way to decompose features for better domain adaptation via learning domain-invariant representations. The main advantage of the proposed approach is that by only introducing a few model parameters, the proposed approach could quickly adapt to new domains. Similar idea has been proposed in [1], where the authors propose to use domain-specific encoders to extract domain-invariant features. Compared with [1], the main novelty of this paper lies in a new feature decomposition of the traditional convolution layer.