Directional Domain Generalization

1 INTRODUCTION . Modern machine learning techniques have achieved unprecedented success over the past decades in various areas . However , one fundamental limitation of most existing techniques is that a model trained on one data set can not generalize well on another data set if it is sampled from a different distribution . Domain generalization ( DG ) aims to alleviate the prediction gap between the observed source domains and an unseen target domain by leveraging the knowledge extracted from multiple source tasks ( Ganin & Lempitsky , 2015 ; Arjovsky et al. , 2019 ; Li et al. , 2018b ) . Existing DG methods can be roughly categorized into three groups : data augmentation/generation , disentangled/domain-invariant feature learning , and meta-learning ( Wang et al. , 2021 ) . One intrinsic problem with these methods is that they treat all the domains equally and ignore the relationship between them , implicitly assuming that they are all sampled from a stationary environment . However , in many real-world applications , the data are usually collected sequentially and the learning tasks can vary in an evolving manner . For example , geological exploration is often carried out periodically and the distribution of data collected can change from year to year due to environmental changes . Medical data are also often collected with age or other indicators as intervals , and there is an evolving trend in the data of different groups . As a more concrete example , Fig . 1 ( a ) shows several instances from the rotated MNIST ( RMNIST ) data set , a widely used benchmark in the DG literature , where the digit images of each subsequent domain are rotated by 15◦ . Fig . 1 ( b ) reports the generalization performances of several state-of-the-art DG algorithms on the data set , from which it can be clearly observed that the performances drop when deploying the models on outer domains ( i.e. , tasks of 0 and 75 degrees ) . The results indicate that the algorithms ignore the evolving pattern between the domains properly . As a consequence , they are good at “ interpolation ” but not at “ extrapolation ” . In this paper , we formulate this learning scenario as progressive domain generalization ( PDG ) , which aims to capture and exploit the evolving patterns in the environment . In contrast to most existing DG methods , which produce models that are isotropic with respect to all the domains , PDG can generalize to a target domain along a specific direction by extracting and leveraging the relations between source tasks . Specifically , we develop a novel theoretical analysis that highlights the importance of modeling the relation between two consecutive tasks to extract the evolving pattern of the environment . Moreover , our analysis also suggests learning a globally consistent directional mapping function via meta-learning . Inspired the theoretical results , we slightly modify prototypical networks ( Snell et al. , 2017 ) and propose directional prototypical networks ( DPNets ) , a simple and efficient PDG algorithm that adapts to the environment shift and generalizes well on the evolving target domain . As a comparison , Fig . 1 ( c ) shows the performance improvement of DPNets over the other algorithms on RMNIST data set . It can be observed that the performance gap between DPNets and the other baseline algorithms has widened as the domain distance increases . More details can be found in Section 4 . Here , we would like to emphasize the key difference between PDG and domain adaptation in evolving domains ( Liu et al. , 2020 ; Wang et al. , 2020 ; Kim et al. , 2020 ) . While both learning paradigms aim to tackle the issue of evolving domain shifts , the latter still requires unlabeled instances from the target domain . In this sense , PDG is more challenging and existing theoretical and algorithmic results can not be applied to this problem directly . 2 RELATED WORK . Domain Generalization ( DG ) . Domain generalization aims to train a model which generalizes on all domains . Existing DG methods can be classified into three categories . The first and most popular category is representation learning , which focus on learning a common representation across domains . It can be achieved by domain-invariant representation learning ( Blanchard et al. , 2011 ; Ghifary et al. , 2015 ; Ganin & Lempitsky , 2015 ; Arjovsky et al. , 2019 ) and feature disentanglement ( Xu et al. , 2014 ; Ilse et al. , 2020 ) . The former focuses on aligning latent features across domains , and the later tries to distill domain-shared features . Secondly , data manipulation can also empower the model with generalization capability . Data manipulating techniques include data augmentation ( Yue et al. , 2019 ; Shankar et al. , 2018b ) , which usually extend the dataset by applying specific transformations on existing samples , and data generation ( Rahman et al. , 2019 ) , which often applies neural networks to generate new samples . Nguyen et al . ( 2021 ) convert DG to an infinite-dimensional constrained statistical learning problem under a natural model of data generation . The theoretically grounded method proposed in Nguyen et al . ( 2021 ) leverages generating model among domains to learn domain-invariant representation . The last part are meta-learning . As a widely applicable method , meta-learning framework ( Li et al. , 2018b ; Balaji et al. , 2018 ; Li et al. , 2019 ) is used to improve the generalizing capability by simulating the shift among domains . Apart from the above three categories , there are some other learning strategies that help to improve the generalization ability of the model . Mancini et al . ( 2018 ) tries to ensemble multiple models into a unified one which can generalize across domains . The DRO-based methods ( Rahimian & Mehrotra , 2019 ) which aim to learn a model at worst-case distribution scenario also match the target of DG well . Besides , gradient operation ( Huang et al. , 2020b ) , self-supervision ( Carlucci et al. , 2019 ) and random forest ( Ryu et al. , 2019 ) are also exploited to improve generalizing capability . Different from the existing DG methods that focus on learning one unified model for all domains , our approach tries to train prediction model for the target domain specifically by leveraging the evolving pattern among domains . Evolving Domain Adaptation ( EDA ) . Many previous work in domain adaptation area notice the evolving pattern of domains and leverage it to improve the performance in different settings . Liu et al . ( 2020 ) proposes a meta-adaptation framework which enables the learner to adapt from one single source domain to continually evolving target domains without forgetting . Kim et al . ( 2020 ) tries to adapt to multiple target domains sequentially without forgetting , while the most important difference is that there is no assumption about the evolving pattern between these domains . Kumar et al . ( 2020 ) focuses on adapting from source domain to the target domain with large shifts by leveraging the unlabeled intermediate samples . Wang et al . ( 2020 ) combines the traditional adversarial adaptation strategy with a novel regression discriminator that models the encoding-conditioned domain index distribution . Chen & Chao ( 2021 ) investigate how to discover the sequence of intermediate domains without index information then adapt to the final target . The experimental results and theoretical analysis demonstrate the value of leveraging index information when working on evolving domains . These studies fully demonstrate that leveraging evolving pattern between domains is beneficial and worth more exploration . However , there are two significant limitation in previous works . The first one is the requirements for accessing unlabeled data in the target domain . The second one is that all previous theoretical results are base on the assumption that distance between sequential domains is small , which would become vacuous as the environment evolves . This is contrary to the fact that more domains provide more evolving information which can help to improve the performance . In this paper , the evolving information is leveraged without accessing any samples from the target domain . Also , our theoretical results are based on proposed λ-consistency , a intuitive and realistic measurement of evolving level in the environments . 3 THEORETICAL ANALYSIS AND METHODOLOGY . Let { D1 ( x , y ) , D2 ( x , y ) , ... , Dm ( x , y ) } be m observed source domains sampled from an environment E where x ∈ X and y ∈ Y are , respectively , the data point and its corresponding label . The goal of DG is to learn a hypothesis h ∈ H so that it can have a low risk on an unseen but related target domain Dt : RDt ( h ) , E ( x , y ) ∼Dt [ ` ( h ( x ) , y ) ] where ` : Y×Y → R+ is a non-negative loss function , andH is a hypothesis class that mapsX to the set Y . In the setting of traditional DG , as there is no relation exploited between the source domains and the target domain , most existing techniques essentially either “ enlarge ” the input space X along all possible directions ( Volpi et al. , 2018 ; Shankar et al. , 2018a ; Qiao et al. , 2020 ) or learn a domaininvariant feature representation via domain alignment ( Li et al. , 2018c ; Arjovsky et al. , 2019 ; Zhao et al. , 2020 ) . In contrast , the objective of PDG is to generalize the model on Dt along a specific direction when there is underlying evolving pattern between the source domains and Dt = Dm+1 . 3.1 THEORETICAL MOTIVATIONS . In order to leverage the evolving pattern in E , it is reasonable to assume that such a pattern can be captured by a globally consistent mapping function g ∈ G : Dgi+1 , g ( Di ) in a way such that the synthetic domain Dgi+1 is close to Di+1 as much as possible , where G is the class of mapping functions . We first obtain the following bound of the risk on the target domain Dt with respect to Dgt , as shown in Lemma 1 . Lemma 1 . Let Dgt ( h ) = g ( Dm ) be the synthetic target domain , and suppose the loss function ` is bounded within an interval G : G = max ( ` ) −min ( ` ) . Then , for any h ∈ H , its target risk RDt ( h ) can be upper bounded by : RDt ( h ) ≤ RDgt ( h ) + G√ 2 √ dJS ( Dgt ||Dt ) , where dJS ( Dgt ||Dt ) is the Jensen-Shannon ( JS ) divergence between D g t and Dt ( Fuglede & Topsoe , 2004 ) . Remark 1 . To achieve a low risk on Dt , Lemma 1 suggests ( 1 ) learning h and g to minimize the risk over the synthetic domain Dgt and ( 2 ) learning g to minimize the JS divergence between D g t and Dt . While in practice RDgt ( h ) can be approximated by the empirical risk , Lemma 1 still can not provide any practical guidelines for learning g since Dt is unavailable . Moreover , note that Dgt can be replaced by g ( Di ) for any other source domain i in E and the bound still holds . Thus , Lemma 1 does not provide any theoretical insight into how to discover and leverage the evolving pattern in E . Intuitively , capturing the evolving pattern in E is hopeless if it varies arbitrarily . On the other hand , if the underlying pattern is consistent over domains , it is reasonable to assume that there exists g∗ ∈ G would perform consistently well over all the domain pairs . For example , given numbers 100 , 202 , 301 , one would expect that the numbers increase by around 100 and the next number will be around 400 , but it is challenging to guess the number if the first three numbers are −20 , 1300 , 4 . To formulate the this intuition , we first introduce the notion of consistency of an environment E below . Definition 1 ( Consistency ) . Let g∗ = arg ming∈G maxDi∈E dJS ( D g i ||Di ) be the ideal mapping function in the worst-case domain . Then , an evolving environment E is λ-consistent if the following holds : |dJS ( Dg ∗ i ||Di ) − dJS ( D g∗ j ||Dj ) | ≤ λ , ∀Di , Dj ∈ E . Note that λ does not characterize how fast E evolves , but if there exists a global mapping function that can model the evolving pattern consistently well in E . Given the definition of consistency , we can bound the target risk in terms of dJS ( Dgi ||Di ) in the source domains . Theorem 1 . Let { D1 , D2 , ... , Dm } be m observed source domains sampled sequentially from an evolving environment E , and Dt be the next unseen target domain : Dt = Dm+1 . Then , if E is λ-consistent , we have RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( √√√√ m∑ i=2 dJS ( Dg ∗ i ||Di ) + √ ( m− 1 ) λ ) . Remark 2 . ( 1 ) Theorem 1 highlights the role of the mapping function and λ-consistency in PDG . Given g∗ , the target risk RDt ( h ) can be upper bounded by in terms of loss on the synthetic target domain RDg∗t ( h ) , λ , and the JS divergence between Di and D g∗ i in all observed source domains . When g∗ can properly capture the evolving pattern of E , we can train the classifier h over the synthetic domainDg ∗ t generated fromDm and can still expect a low risk onDt . ( 2 ) λ is unobservable and is determined by E and G. Intuitively , a small λ suggests high predictability of E , which indicates that it is easier to predict the target domain Dt . On the other hand , a large λ indicates that there does not exist a global mapping function that captures the evolving pattern consistently well over domains . Consequently , generalization to the target domain could be challenging and we can not expect to learn a good hypothesis h on Dt . ( 3 ) In practice , g∗ is not given , but can be learned by minimizing dJS ( Dgi ||Di ) in source domains . Besides , aligning D g i and Di is usually achieved by representation learning : that is , learning g : X → Z to minimize dJS ( Dgi ||Di ) , ∀z ∈ Z . In addition , we can decomposeD ( x , y ) into marginal and conditional distributions to motivate more practical PDG algorithms . For example , when it is decomposed into class prior D ( y ) and semantic conditional distribution D ( x|y ) , we have the following Corollary . Corollary 1 . Following the assumptions of Theorem 1 , the target risk can be bounded by RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( √√√√ m∑ i=2 dJS ( Dg ∗ i ( y ) ||Di ( y ) ) ︸ ︷︷ ︸ I + √ ( m− 1 ) λ + √√√√ m∑ i=2 E y∼Dg ∗ i ( y ) dJS ( Dg ∗ i ( x|y ) ||Di ( x|y ) ) ︸ ︷︷ ︸ II + √√√√ m∑ i=2 Ey∼Di ( y ) dJS ( D g∗ i ( x|y ) ||Di ( x|y ) ) ︸ ︷︷ ︸ III ) . Remark 3 . To generalize well to Dt , Corollary 1 suggests that a good mapping function should capture both label shifts ( term I ) and semantic conditional distribution shifts ( terms II & III ) . If we further assume that the label distribution does not evolve over domains1 , we will have I = 0 and II = III , and the upper bound can be simplified as RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( 2 √√√√ m∑ i=2 Ey∼Di ( y ) dJS ( D g∗ i ( x|y ) ||Di ( x|y ) ) + √ ( m− 1 ) λ ) . Finally , we note two key theoretical differences between PDG and previous studies of DA in evolving environments ( Ben-David et al. , 2010 ; Liu et al. , 2020 ) . ( 1 ) In DA , the target risk is bounded in terms of source and target domains ( e.g. , H∆H-divergence ) , while our analysis relies on the distance between synthetic and real domains . ( 2 ) DA theories are built upon the assumption that there exists an ideal joint hypothesis that achieves a low combined error on both domains , while our assumption is the λ-consistency of E . These differences eventually lead to fundamentally different guidelines for PDG , as shown in Section 3.2 .

Summary: Summary:  The paper proposes a new domain generalization paradigm called evolving domain generalization, where the source domains and the target domain are not random but have an evolving pattern.  The paper provides theoretical results on the evolving domain generalization, which bounds the target domain test error with synthetic domains.  The paper proposes a new approach for evolving domain generalization, which learns two embedding functions to derive the synthetic domains and minimize the distances between the synthetic domains and the real domains within the hidden space  The paper conducts experiments on three domain generalization datasets including RMNIST, Portrait, and Cover Type.