Directional Domain Generalization

1 INTRODUCTION . Modern machine learning techniques have achieved unprecedented success over the past decades in various areas . However , one fundamental limitation of most existing techniques is that a model trained on one data set can not generalize well on another data set if it is sampled from a different distribution . Domain generalization ( DG ) aims to alleviate the prediction gap between the observed source domains and an unseen target domain by leveraging the knowledge extracted from multiple source tasks ( Ganin & Lempitsky , 2015 ; Arjovsky et al. , 2019 ; Li et al. , 2018b ) . Existing DG methods can be roughly categorized into three groups : data augmentation/generation , disentangled/domain-invariant feature learning , and meta-learning ( Wang et al. , 2021 ) . One intrinsic problem with these methods is that they treat all the domains equally and ignore the relationship between them , implicitly assuming that they are all sampled from a stationary environment . However , in many real-world applications , the data are usually collected sequentially and the learning tasks can vary in an evolving manner . For example , geological exploration is often carried out periodically and the distribution of data collected can change from year to year due to environmental changes . Medical data are also often collected with age or other indicators as intervals , and there is an evolving trend in the data of different groups . As a more concrete example , Fig . 1 ( a ) shows several instances from the rotated MNIST ( RMNIST ) data set , a widely used benchmark in the DG literature , where the digit images of each subsequent domain are rotated by 15◦ . Fig . 1 ( b ) reports the generalization performances of several state-of-the-art DG algorithms on the data set , from which it can be clearly observed that the performances drop when deploying the models on outer domains ( i.e. , tasks of 0 and 75 degrees ) . The results indicate that the algorithms ignore the evolving pattern between the domains properly . As a consequence , they are good at “ interpolation ” but not at “ extrapolation ” . In this paper , we formulate this learning scenario as progressive domain generalization ( PDG ) , which aims to capture and exploit the evolving patterns in the environment . In contrast to most existing DG methods , which produce models that are isotropic with respect to all the domains , PDG can generalize to a target domain along a specific direction by extracting and leveraging the relations between source tasks . Specifically , we develop a novel theoretical analysis that highlights the importance of modeling the relation between two consecutive tasks to extract the evolving pattern of the environment . Moreover , our analysis also suggests learning a globally consistent directional mapping function via meta-learning . Inspired the theoretical results , we slightly modify prototypical networks ( Snell et al. , 2017 ) and propose directional prototypical networks ( DPNets ) , a simple and efficient PDG algorithm that adapts to the environment shift and generalizes well on the evolving target domain . As a comparison , Fig . 1 ( c ) shows the performance improvement of DPNets over the other algorithms on RMNIST data set . It can be observed that the performance gap between DPNets and the other baseline algorithms has widened as the domain distance increases . More details can be found in Section 4 . Here , we would like to emphasize the key difference between PDG and domain adaptation in evolving domains ( Liu et al. , 2020 ; Wang et al. , 2020 ; Kim et al. , 2020 ) . While both learning paradigms aim to tackle the issue of evolving domain shifts , the latter still requires unlabeled instances from the target domain . In this sense , PDG is more challenging and existing theoretical and algorithmic results can not be applied to this problem directly . 2 RELATED WORK . Domain Generalization ( DG ) . Domain generalization aims to train a model which generalizes on all domains . Existing DG methods can be classified into three categories . The first and most popular category is representation learning , which focus on learning a common representation across domains . It can be achieved by domain-invariant representation learning ( Blanchard et al. , 2011 ; Ghifary et al. , 2015 ; Ganin & Lempitsky , 2015 ; Arjovsky et al. , 2019 ) and feature disentanglement ( Xu et al. , 2014 ; Ilse et al. , 2020 ) . The former focuses on aligning latent features across domains , and the later tries to distill domain-shared features . Secondly , data manipulation can also empower the model with generalization capability . Data manipulating techniques include data augmentation ( Yue et al. , 2019 ; Shankar et al. , 2018b ) , which usually extend the dataset by applying specific transformations on existing samples , and data generation ( Rahman et al. , 2019 ) , which often applies neural networks to generate new samples . Nguyen et al . ( 2021 ) convert DG to an infinite-dimensional constrained statistical learning problem under a natural model of data generation . The theoretically grounded method proposed in Nguyen et al . ( 2021 ) leverages generating model among domains to learn domain-invariant representation . The last part are meta-learning . As a widely applicable method , meta-learning framework ( Li et al. , 2018b ; Balaji et al. , 2018 ; Li et al. , 2019 ) is used to improve the generalizing capability by simulating the shift among domains . Apart from the above three categories , there are some other learning strategies that help to improve the generalization ability of the model . Mancini et al . ( 2018 ) tries to ensemble multiple models into a unified one which can generalize across domains . The DRO-based methods ( Rahimian & Mehrotra , 2019 ) which aim to learn a model at worst-case distribution scenario also match the target of DG well . Besides , gradient operation ( Huang et al. , 2020b ) , self-supervision ( Carlucci et al. , 2019 ) and random forest ( Ryu et al. , 2019 ) are also exploited to improve generalizing capability . Different from the existing DG methods that focus on learning one unified model for all domains , our approach tries to train prediction model for the target domain specifically by leveraging the evolving pattern among domains . Evolving Domain Adaptation ( EDA ) . Many previous work in domain adaptation area notice the evolving pattern of domains and leverage it to improve the performance in different settings . Liu et al . ( 2020 ) proposes a meta-adaptation framework which enables the learner to adapt from one single source domain to continually evolving target domains without forgetting . Kim et al . ( 2020 ) tries to adapt to multiple target domains sequentially without forgetting , while the most important difference is that there is no assumption about the evolving pattern between these domains . Kumar et al . ( 2020 ) focuses on adapting from source domain to the target domain with large shifts by leveraging the unlabeled intermediate samples . Wang et al . ( 2020 ) combines the traditional adversarial adaptation strategy with a novel regression discriminator that models the encoding-conditioned domain index distribution . Chen & Chao ( 2021 ) investigate how to discover the sequence of intermediate domains without index information then adapt to the final target . The experimental results and theoretical analysis demonstrate the value of leveraging index information when working on evolving domains . These studies fully demonstrate that leveraging evolving pattern between domains is beneficial and worth more exploration . However , there are two significant limitation in previous works . The first one is the requirements for accessing unlabeled data in the target domain . The second one is that all previous theoretical results are base on the assumption that distance between sequential domains is small , which would become vacuous as the environment evolves . This is contrary to the fact that more domains provide more evolving information which can help to improve the performance . In this paper , the evolving information is leveraged without accessing any samples from the target domain . Also , our theoretical results are based on proposed λ-consistency , a intuitive and realistic measurement of evolving level in the environments . 3 THEORETICAL ANALYSIS AND METHODOLOGY . Let { D1 ( x , y ) , D2 ( x , y ) , ... , Dm ( x , y ) } be m observed source domains sampled from an environment E where x ∈ X and y ∈ Y are , respectively , the data point and its corresponding label . The goal of DG is to learn a hypothesis h ∈ H so that it can have a low risk on an unseen but related target domain Dt : RDt ( h ) , E ( x , y ) ∼Dt [ ` ( h ( x ) , y ) ] where ` : Y×Y → R+ is a non-negative loss function , andH is a hypothesis class that mapsX to the set Y . In the setting of traditional DG , as there is no relation exploited between the source domains and the target domain , most existing techniques essentially either “ enlarge ” the input space X along all possible directions ( Volpi et al. , 2018 ; Shankar et al. , 2018a ; Qiao et al. , 2020 ) or learn a domaininvariant feature representation via domain alignment ( Li et al. , 2018c ; Arjovsky et al. , 2019 ; Zhao et al. , 2020 ) . In contrast , the objective of PDG is to generalize the model on Dt along a specific direction when there is underlying evolving pattern between the source domains and Dt = Dm+1 . 3.1 THEORETICAL MOTIVATIONS . In order to leverage the evolving pattern in E , it is reasonable to assume that such a pattern can be captured by a globally consistent mapping function g ∈ G : Dgi+1 , g ( Di ) in a way such that the synthetic domain Dgi+1 is close to Di+1 as much as possible , where G is the class of mapping functions . We first obtain the following bound of the risk on the target domain Dt with respect to Dgt , as shown in Lemma 1 . Lemma 1 . Let Dgt ( h ) = g ( Dm ) be the synthetic target domain , and suppose the loss function ` is bounded within an interval G : G = max ( ` ) −min ( ` ) . Then , for any h ∈ H , its target risk RDt ( h ) can be upper bounded by : RDt ( h ) ≤ RDgt ( h ) + G√ 2 √ dJS ( Dgt ||Dt ) , where dJS ( Dgt ||Dt ) is the Jensen-Shannon ( JS ) divergence between D g t and Dt ( Fuglede & Topsoe , 2004 ) . Remark 1 . To achieve a low risk on Dt , Lemma 1 suggests ( 1 ) learning h and g to minimize the risk over the synthetic domain Dgt and ( 2 ) learning g to minimize the JS divergence between D g t and Dt . While in practice RDgt ( h ) can be approximated by the empirical risk , Lemma 1 still can not provide any practical guidelines for learning g since Dt is unavailable . Moreover , note that Dgt can be replaced by g ( Di ) for any other source domain i in E and the bound still holds . Thus , Lemma 1 does not provide any theoretical insight into how to discover and leverage the evolving pattern in E . Intuitively , capturing the evolving pattern in E is hopeless if it varies arbitrarily . On the other hand , if the underlying pattern is consistent over domains , it is reasonable to assume that there exists g∗ ∈ G would perform consistently well over all the domain pairs . For example , given numbers 100 , 202 , 301 , one would expect that the numbers increase by around 100 and the next number will be around 400 , but it is challenging to guess the number if the first three numbers are −20 , 1300 , 4 . To formulate the this intuition , we first introduce the notion of consistency of an environment E below . Definition 1 ( Consistency ) . Let g∗ = arg ming∈G maxDi∈E dJS ( D g i ||Di ) be the ideal mapping function in the worst-case domain . Then , an evolving environment E is λ-consistent if the following holds : |dJS ( Dg ∗ i ||Di ) − dJS ( D g∗ j ||Dj ) | ≤ λ , ∀Di , Dj ∈ E . Note that λ does not characterize how fast E evolves , but if there exists a global mapping function that can model the evolving pattern consistently well in E . Given the definition of consistency , we can bound the target risk in terms of dJS ( Dgi ||Di ) in the source domains . Theorem 1 . Let { D1 , D2 , ... , Dm } be m observed source domains sampled sequentially from an evolving environment E , and Dt be the next unseen target domain : Dt = Dm+1 . Then , if E is λ-consistent , we have RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( √√√√ m∑ i=2 dJS ( Dg ∗ i ||Di ) + √ ( m− 1 ) λ ) . Remark 2 . ( 1 ) Theorem 1 highlights the role of the mapping function and λ-consistency in PDG . Given g∗ , the target risk RDt ( h ) can be upper bounded by in terms of loss on the synthetic target domain RDg∗t ( h ) , λ , and the JS divergence between Di and D g∗ i in all observed source domains . When g∗ can properly capture the evolving pattern of E , we can train the classifier h over the synthetic domainDg ∗ t generated fromDm and can still expect a low risk onDt . ( 2 ) λ is unobservable and is determined by E and G. Intuitively , a small λ suggests high predictability of E , which indicates that it is easier to predict the target domain Dt . On the other hand , a large λ indicates that there does not exist a global mapping function that captures the evolving pattern consistently well over domains . Consequently , generalization to the target domain could be challenging and we can not expect to learn a good hypothesis h on Dt . ( 3 ) In practice , g∗ is not given , but can be learned by minimizing dJS ( Dgi ||Di ) in source domains . Besides , aligning D g i and Di is usually achieved by representation learning : that is , learning g : X → Z to minimize dJS ( Dgi ||Di ) , ∀z ∈ Z . In addition , we can decomposeD ( x , y ) into marginal and conditional distributions to motivate more practical PDG algorithms . For example , when it is decomposed into class prior D ( y ) and semantic conditional distribution D ( x|y ) , we have the following Corollary . Corollary 1 . Following the assumptions of Theorem 1 , the target risk can be bounded by RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( √√√√ m∑ i=2 dJS ( Dg ∗ i ( y ) ||Di ( y ) ) ︸ ︷︷ ︸ I + √ ( m− 1 ) λ + √√√√ m∑ i=2 E y∼Dg ∗ i ( y ) dJS ( Dg ∗ i ( x|y ) ||Di ( x|y ) ) ︸ ︷︷ ︸ II + √√√√ m∑ i=2 Ey∼Di ( y ) dJS ( D g∗ i ( x|y ) ||Di ( x|y ) ) ︸ ︷︷ ︸ III ) . Remark 3 . To generalize well to Dt , Corollary 1 suggests that a good mapping function should capture both label shifts ( term I ) and semantic conditional distribution shifts ( terms II & III ) . If we further assume that the label distribution does not evolve over domains1 , we will have I = 0 and II = III , and the upper bound can be simplified as RDt ( h ) ≤ RDg∗t ( h ) + G√ 2 ( m− 1 ) ( 2 √√√√ m∑ i=2 Ey∼Di ( y ) dJS ( D g∗ i ( x|y ) ||Di ( x|y ) ) + √ ( m− 1 ) λ ) . Finally , we note two key theoretical differences between PDG and previous studies of DA in evolving environments ( Ben-David et al. , 2010 ; Liu et al. , 2020 ) . ( 1 ) In DA , the target risk is bounded in terms of source and target domains ( e.g. , H∆H-divergence ) , while our analysis relies on the distance between synthetic and real domains . ( 2 ) DA theories are built upon the assumption that there exists an ideal joint hypothesis that achieves a low combined error on both domains , while our assumption is the λ-consistency of E . These differences eventually lead to fundamentally different guidelines for PDG , as shown in Section 3.2 .

Summary: This paper tackles the issue of training a model to generalize to data from unseen domains when there is a natural progression between domain at training and test time (eg. temporal progression). With the assumption that there is a fixed underlying transformation "g" which underlies the evolution of the data distributions, they propose a training algorithm based on progressive net which simultaneously learns g as well as well as a strong classifier. The resulting model can be used on unseen domains at test-time, leveraging the learned transformation to extrapolate to the new domain.