Domain-Free Adversarial Splitting for Domain Generalization

Domain generalization is an approach that utilizes several source domains to train the learner to be generalizable to unseen target domain to tackle domain shift issue . It has drawn much attention in machine learning community . This paper aims to learn to generalize well to unseen target domain without relying on the knowledge of the number of source domains and domain labels . We unify adversarial training and meta-learning in a novel proposed Domain-Free Adversarial Splitting ( DFAS ) framework . In this framework , we model the domain generalization as a learning problem that enforces the learner to be able to generalize well for any train/val subsets splitting of the training dataset . To achieve this goal , we propose a min-max optimization problem which can be solved by an iterative adversarial training process . In each iteration , it adversarially splits the training dataset into train/val subsets to maximize domain shift between them using current learner , and then updates the learner on this splitting to be able to generalize well from train-subset to val-subset using meta-learning approach . Extensive experiments on three benchmark datasets under three different settings on the source and target domains show that our method achieves state-of-the-art results and confirm the effectiveness of our method by ablation study . We also derive a generalization error bound for theoretical understanding of our method . 1 INTRODUCTION . Deep learning approach has achieved great success in image recognition ( He et al. , 2016 ; Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ) . However , deep learning methods mostly succeed in the case that the training and test data are sampled from the same distribution ( i.e. , the i.i.d . assumption ) . However , this assumption is often violated in real-world applications since the equipments/environments that generate data are often different in training and test datasets . When there exists distribution difference ( domain shift ( Torralba & Efros , 2011 ) ) between training and test datasets , the performance of trained model , i.e. , learner , will significantly degrade . To tackle the domain shift issue , domain adaptation approach ( Pan & Yang , 2010 ; Daume III & Marcu , 2006 ; Huang et al. , 2007 ) learns a transferable learner from source domain to target domain . Domain adaptation methods align distributions of different domains either in feature space ( Long et al. , 2015 ; Ganin et al. , 2016 ) or in raw pixel space ( Hoffman et al. , 2018 ) , which relies on unlabeled data from target domain at training time . However , in many applications , it is unrealistic to access the unlabeled target data , therefore this prevents us to use domain adaptation approach in this setting , and motivates the research on the learning problem of domain generalization . Domain generalization ( DG ) approach ( Blanchard et al. , 2011 ; Muandet et al. , 2013 ) commonly uses several source domains to train a learner that can generalize to an unseen target domain . The underlying assumption is that there exists a latent domain invariant feature space across source domains and unseen target domain . To learn the domain invariant features , ( Muandet et al. , 2013 ; Ghifary et al. , 2015 ; Li et al. , 2018b ) explicitly align distributions of different source domains in feature space . ( Balaji et al. , 2018 ; Li et al. , 2019b ; 2018a ; Dou et al. , 2019 ) split source domains into meta-train and meta-test to simulate domain shift and train learner in a meta-learning approach . ( Shankar et al. , 2018 ; Carlucci et al. , 2019 ; Zhou et al. , 2020 ; Ryu et al. , 2020 ) augment images or features to train learner to enhance generalization capability . Conventional domain generalization methods assume that the domain labels are available . But in a more realistic scenario , the domain labels may be unknown ( Wang et al. , 2019 ) . To handle this domain-free setting , Carlucci et al . ( 2019 ) combines supervised learning and self-supervised learning to solve jigsaw puzzles of the training images . Matsuura & Harada ( 2020 ) divides samples into several latent domains via clustering and trains a domain invariant feature extractor via adversarial training . Huang et al . ( 2020 ) discards the dominant activated features , forcing the learner to activate remaining features that correlate with labels . Another line of works ( Volpi et al. , 2018 ; Qiao et al. , 2020 ) tackle the single source setting that the training set comprises a single domain , and the train and test data are from different domains . In this work , we focus on a general learning scenario of domain generalization as follows . First , we do not know the domain label of each data and do not assume that there are several domains in the training dataset . Second , we do not assume that the training and test data are from different domains ( e.g. , styles ) . However , the previous domain-free DG methods ( Matsuura & Harada , 2020 ) commonly evaluate on the datasets ( e.g. , PACS ) composed of several domains though they do not use domain labels in training . In our domain-free setting , we do not assume and know the domains in the training dataset , we therefore model domain generalization as a learning problem that the learner should be able to generalize well for any splitting of train/val subsets , i.e. , synthetic source/target domains , over the training dataset . This explicitly enforces that the trained learner should be generalizable for any possible domain shifts within the training dataset . To achieve this goal , we propose an adversarial splitting model that is a min-max optimization problem , due to the difficulty of enumerating all splittings . In this min-max problem , we adversarially split training dataset to train/val subsets by maximizing the domain shift between them based on the given learner , and then update learner by minimizing the prediction error on val-subset using meta-learning approach given the splitting . By optimizing this min-max problem , we enforce the learner to generalize well even in the worst-case splitting . We also investigate L2-normalization of features in our domain generalization method . It is surprisingly found that L2-normalization can improve performance of learner and mitigate gradient explosion in the meta-learning process of DG . We further theorectically analyze the underlying reasons for this finding . This proposed domain generalization approach is dubbed Domain-Free Adversarial Splitting , i.e. , DFAS . To verify the effectiveness of our method , we conduct extensive experiments on benchmark datasets of PACS , Office-Home and CIFAR-10 under different settings with multiple/single source domains . In experiments that the training data are from several source domains , our method achieves state-ofthe-art results on both PACS and Office-Home datasets . We also find that our method significantly outperforms baselines in experiments that the training data are from a single source domain on PACS and CIFAR-10 . We also confirm the effectiveness of our method by ablation study . Based on domain adaptation theory , we also derive an upper bound of the generalization error on unseen target domain . We analyze that the terms in this upper bound are implicitly minimized by our method . This theoretical analysis partially explains the success of our method . 2 RELATED WORKS . We summarize and compare with related domain generalization ( DG ) methods in two perspectives , i.e. , DG with domain labels and DG without domain labels . DG with domain labels . When the domain labels are available , there are three categories of methods for DG . First , ( Muandet et al. , 2013 ; Ghifary et al. , 2015 ; Li et al. , 2018b ; Piratla et al. , 2020 ) learn domain invariant features by aligning feature distributions or by common/specific feature decomposition . Second , ( Li et al. , 2019a ; Balaji et al. , 2018 ; Li et al. , 2019b ; 2018a ; Dou et al. , 2019 ; Du et al. , 2020a ; b ) are based on meta-learning approach that splits given source domains into meta-train and meta-test domains and trains learner in an episodic training paradigm . Third , ( Shankar et al. , 2018 ; Carlucci et al. , 2019 ; Zhou et al. , 2020 ; Wang et al. , 2020 ) augment fake domain data to train learner for enhancing generalization capability of learner . Our method may mostly relate to the above second category of methods . But differently , we consider the DG problem in domain-free setting and adversarially split training dataset to synthesize domain shift in a principled min-max optimization method , instead of using leave-one-domain-out splitting in these methods . DG without domain labels . When the domain label is unavailable , to enhance generalization ability of learner , Wang et al . ( 2019 ) extracts robust feature representation by projecting out superficial patterns like color and texture . Carlucci et al . ( 2019 ) proposes to solve jigsaw puzzles of the training images . Matsuura & Harada ( 2020 ) divides samples into several latent domains via clustering and learns domain invariant features via adversarial training of feature extractor and domain discriminator . Huang et al . ( 2020 ) discards the dominant activated features , forcing the learner to activate remaining features that correlate with labels . Volpi et al . ( 2018 ) and Qiao et al . ( 2020 ) propose adversarial data augmentation to tackle the setting that the training set comprises a single domain . In methodology , these methods either explicitly force the learner to extract robust features ( Wang et al. , 2019 ; Matsuura & Harada , 2020 ; Huang et al. , 2020 ) or augment new data to increase training data ( Carlucci et al. , 2019 ; Qiao et al. , 2020 ; Volpi et al. , 2018 ) . While our method is a novel meta-learning approach for DG by introducing adversarial splitting of training dataset during training , without relying on data/domain augmentation . 3 METHOD . In our setting , since we do not assume and know the domains in the training dataset , the training data could be independently sampled from several underlying source domains or just from a single source domain . We denote S = { ( xi , yi ) } Ni=1 as the training dataset . Our goal is to train the learner with S that can generalize well on an unseen target domain . In the following sections , we introduce details of our proposed model in Sect . 3.1 , followed by its optimization method in Sect . 3.2 . We also investigate L2-normalization for domain generalization in Sect . 3.3 . Theoretical analysis for our method is presented in Sect . 4 . Experimental results are reported in Sect . 5 . Sect . 6 concludes this paper . 3.1 DOMAIN-FREE ADVERSARIAL SPLITTING MODEL . As mentioned in Sect . 1 , we model DG as a learning problem that enforces the learner to be able to generalize well for any train/val subsets splitting of the training dataset . The learner is trained using meta-learning approach ( Finn et al. , 2017 ) . To formulate our idea mathematically , we first introduce some notations . We denote f as a function/learner ( f could be a deep neural network , e.g. , ResNet ( He et al. , 2016 ) ) that outputs classification score of the input image , l as the loss such as cross-entropy , St and Sv as the train-subset and val-subset respectively such that S = St ∪ Sv and St ∩ Sv = ∅ . The formulated optimization problem for domain generalization is min w 1 |Γξ| ∑ Sv∈Γξ L ( θ ( w ) ; Sv ) +R ( w ) s.t . θ ( w ) = arg min θ L ( θ ; St , w ) , St = S − Sv . ( 1 ) In Eq . ( 1 ) , Γξ = { Sv : Sv ⊂ S , |Sv| = ξ } is the set of all possible val-subsets of S with length of ξ , St = S − Sv is train-subset paired with each Sv , L ( θ ( w ) ; Sv ) = 1|Sv| ∑ ( x , y ) ∈Sv l ( fθ ( w ) ( x ) , y ) is the loss on Sv , where θ ( w ) is the parameters of f , L ( θ ; St , w ) is L ( θ ; St ) with θ initialized by w andR ( w ) is regularization term . In the optimization model of Eq . ( 1 ) , the parameter θ ( w ) of learner trained on St is treated as a function of the initial parameter w. To force the learner trained on St to generalize well on Sv , we directly minimize the loss L ( θ ( w ) ; Sv ) , dubbed generalization loss , on val-subset Sv , w.r.t . the parameter θ ( w ) trained on St . Solving Eq . ( 1 ) will force the learner to be able to generalize well from any train-subset to corresponding val-subset . Since |Γξ| may be extremely large , it is infeasible to enumerate all possible train/val splittings . Thus , we propose the following adversarial splitting model instead , min w max Sv∈Γξ L ( θ ( w ) ; Sv ) +R ( w ) s.t . θ ( w ) = arg min θ L ( θ ; St , w ) , St = S − Sv . ( 2 ) In the min-max problem of Eq . ( 2 ) , the train/val ( St/Sv ) splitting is optimized to maximize the generalization loss to increase the domain shift between train and val subsets by finding the hardest splitting to the learner . While w is optimized by minimizing the generalization loss of learner over the splitting . Solving this adversarial splitting optimization model in Eq . ( 2 ) enforces the learner to be generalizable even for the worst-case splitting . We therefore expect that the trained learner is robust to the domain shifts within the training dataset . For the regularization termR ( w ) , we set it to be the training loss on St ( i.e. , R ( w ) = L ( w ; St ) ) , which additionally constrains that the learner with parameter w should be effective on St ( Li et al. , 2018a ) . The effect of the hyper-parameter ξ will be discussed in Appendix A.2 . In conventional adversarial machine learning , adversarial training is imposed on adversarial samples and learner to increase robustness of the learner to adversarial corruption ( Goodfellow et al. , 2015 ) . While in our optimization model of Eq . ( 2 ) , adversarial training is conducted on data splitting and learner to force the learner to be robust to domain shift between train/val subsets . Our model bridges adversarial training and meta-learning . It is a general learning framework for domain generalization and is a complement to adversarial machine learning .

Summary: This paper focuses on domain generalization, targeting the challenging scenario where the training set might not include different sources; even under the presence of different sources, the problem formulation does not takes into account domain labels. The proposed solution is based on meta-learning, following the path drawn by Li et al. AAAI 2018; the Authors propose to adversarially split the training set in meta-train and meta-validation sets, and then update the current model in a direction that fosters good generalization performance on the meta-test. Results on standard benchmarks are encouraging.