Visual Representation Learning over Latent Domains

1 INTRODUCTION . Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades . They provide a testbed for developing new algorithms and comparing them to existing ones . However , datasets can also narrow down the focus of research into overspecialized solutions and impede developing a broader understanding of the world . In recent years this narrow scope of datasets has been widely questioned ( Torralba & Efros , 2011 ; Tommasi et al. , 2017 ; Recht et al. , 2019 ) and addressing some of these limitations has become a very active area of research . Two actively studied themes to investigate broader learning criteria are multi-domain learning ( Nam & Han , 2016 ; Bulat et al. , 2019 ; Schoenauer-Sebag et al. , 2019 ) and domain adaptation ( Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Hoffman et al. , 2018 ; Xu et al. , 2018 ; Peng et al. , 2019a ; Sun et al. , 2019b ) . While multi-domain techniques focus on learning a single model that can generalize over multiple domains , domain adaptation techniques aim to efficiently transfer the representations that are learned in one dataset to another . Related themes have also been studied in domain generalization ( Li et al. , 2018 ; 2019b ; a ; Gulrajani & Lopez-Paz , 2020 ) and continual learning ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Riemer et al. , 2019 ) , where the focus lies on learning representations that can generalize to unseen domains , and to preserve knowledge acquired from previously seen tasks , respectively . While there exists no canonical definition for what exactly a visual domain is , previous works in multi-domain learning assume that different subsets of data exist , with some defining characteristic that allows them to be separated from each another . Each subset , indexed by d = 1 , . . . , D , is assigned to a pre-defined visual domain , and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some pθ ( y|x , d ) . In some cases domains are intuitive and their annotation straightforward . Consider a problem where images have little visual relationship , for example joint learning of Omniglot handwritten symbols ( Lake et al. , 2015 ) and CIFAR-10 objects ( Krizhevsky & Hinton , 2009 ) . In this case , it is safe to assume that encoding an explicit domain-specific identifier into pθ is a good idea , and results in the multi-domain literature provide clear evidence that it is highly beneficial to do so ( Rebuffi et al. , 2018 ; Liu et al. , 2019a ; Guo et al. , 2019a ; Mancini et al. , 2020 ) . The assumption that domain labels are always available has been widely adopted in multi-domain learning ; however this assumption is not without difficulty . For one , unless the process of domain annotation is automated due to combining existing datasets as in e.g . Rebuffi et al . ( 2017 ) , their manual collection , curation , and domain labeling is very laborious . And even if adequate resources exist , it is often difficult to decide the optimal criteria for the annotation of d : some datasets contain sketches , paintings and real world images ( Li et al. , 2017 ) , others images captured during day or night ( Sultani et al. , 2018 ) . Automatically collected datasets ( Thomee et al. , 2016 ; Sun et al. , 2017 ) contain mixtures of low/high resolution images , taken with different cameras by amateurs/professionals . There is no obvious answer which of these should form their own distinct domain subset . Moreover , the work of Bouchacourt et al . ( 2018 ) considers semantic groupings of data : they show that when dividing data by subcategories , such as size , shape , etc. , and incorporating this information into the model , then this benefits performance . Should one therefore also encode the number of objects into domains , or their color , shape , and so on ? Given the relatively loose requirement that domains are supposed to be different while related in some sense ( Pan & Yang , 2009 ) , these examples hint at the difficulty of deciding whether domains are needed , and – if the answer to that is yes – what the optimal domain criteria are . And note that even if such assignments are made very carefully for some problem , nothing guarantees that they will transfer effectively to some other task . This paper carefully investigates this ambiguity and studies two central questions : 1 . Are domain labels always optimal for learning multi-domain representations ? 2 . Is it possible to learn models that generalize well over visually diverse domains without domain labels ? To study this problem , we introduce a new setting ( c.f . Fig . 1 ) in which models are learned over multiple domains without domain annotations — or latent domain learning for short . While latent domain learning is a highly practical research problem in the context of transfer learning , it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning . In particular , we find that the removal of domain associations leads to severe performance losses for standard architectures on most domains , due to imbalances in the underlying distribution and different difficulty levels of the associated domain-level tasks . We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning ( Rebuffi et al. , 2018 ; Chang et al. , 2018 ) , and find that their performance benefits do not extend to latent domain learning . To account for this lost performance , we formulate a novel method called sparse latent adaptation ( Section 3.2 ) which enables internal feature representations to dynamically adapt to instances from multiple domains in data , without requiring annotations for this . Furthermore , we show that latent domain learning naturally extends to several challenging real world tasks and single domain data , such as fairness problems ( Appendix F ) , and learning over imbalanced distributions ( Appendix G ) . 2 LATENT DOMAIN LEARNING . This section provides an overview over latent domain learning and contrasts it against other types of related learning problems , in particular multi-domain learning . 2.1 PROBLEM SETTING . When learning on multiple domains , the common assumption is that data is sampled i.i.d . from a mixture of distributions Pd with domain indices d = 1 , . . . , D. Together , they constitute the datagenerating distribution as P = ∑ d πdPd , where each domain is associated with a relative share πd = Nd/N , with N the total number of samples , and Nd those belonging to the d ’ th domain . In multi-domain learning , domain labels are available for all samples ( Nam & Han , 2016 ; Rebuffi et al. , 2017 ; 2018 ; Bulat et al. , 2019 ) , such that the overall data available for learning consists of DMD = { ( xi , di , yi ) } with i = 1 , . . . , N . In latent domain learning the information associating each sample xi with a domain di is not available . As such , domain-specific labels yi can not be inferred from sample-domain pairs ( xi , di ) and one is instead forced to learn a single model fθ over the latent domain dataset DLD = { ( xi , yi ) } . While latent domain learning can include mutually exclusive classes and disjoint label spaces Y1 ∪ · · · ∪ YD ( as in long-tailed recognition , see Appendix G ) , we mainly focus on the setting of shared label spaces , i.e . Yd = Yd′ . For example a dataset may contain images of dogs or elephants that can appear as either photos , paintings , or sketches . Latent domains have previously attracted interest in the context of domain adaptation , where the lack of annotations was recovered through hierarchical Hoffman et al . ( 2012 ) and kernel-based clustering ( Gong et al. , 2013 ) , via exemplar SVMs ( Xu et al. , 2014 ) , or by measuring mutual information ( Xiong et al. , 2014 ) . More recent work corrects batch statistics of domain adaptation layers using Gaussian mixtures ( Mancini et al. , 2018 ) , or studies the shift from some source domain to a target distribution that contains multiple latent domains ( Peng et al. , 2019b ; Matsuura & Harada , 2020 ) . Latent domain learning however differs fundamentally from these works : Table 1 contains a comparison to existing transfer learning settings . A common baseline in multi-domain learning is to finetune D models , one for each individual domain ( Rebuffi et al. , 2018 ; Liu et al. , 2019a ) . Doing so requires learning a large number of parameters and shares no parameters across domains , but can serve as a strong baseline to compare against . We show that in many cases , even when domains were carefully annotated , a dynamic latent domain approach can surpass the performance of such domain-supervised baselines , see Section 4 . 2.2 OBSERVED VS . UNIFORM ACCURACY . Consider a problem in which the data is sampled i.i.d . from P = πaPda + πbPdb , i.e . two hidden domains . Since domain labels are not available in latent domain learning , the best one can do is to treat all samples equally , and measure the observed accuracy : OAcc [ f ] = E ( xi , yi ) ∼P [ 1yf ( xi ) =yi ] , ( 1 ) where yf denotes the class assigned to sample xi by the model f , and yi its corresponding label for training . The OAcc has a problematic property : if P consists of two imbalanced domains such that πa ≥ πb , then the performance on da dominates it . For example if da has a 90 % overall share , and the model perfectly classifies this domain while obtaining 0 % accuracy on db , then OAcc would still assume 0.9 , hiding the underlying damage to domain db . This motivates alternative formulations for latent domain learning , to anticipate ( and account for ) imbalanced domains in real world data . If it is possible to identify some semantic domain labeling ( as typically included in multi-domain/domain adaptation benchmarks ) , one can compare performances across individual subgroups . This allows picking up on performance losses that just occur on some of them , and which traditional metrics ( such as OAcc ) fail to capture . Where this is possible , we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes : UAcc [ f ] = 1 D D∑ d=1 E ( xi , yi ) ∼Pd [ 1yf ( xi ) =yi ] . ( 2 ) Returning to the above example , a uniform measurement reflects the model ’ s lack of performance on db as UAcc = 0.5 . Once again note while ground-truth domain annotations are required in order to compute uniform accuracy , these are never used to train latent domain models . 3 METHODS . To enable robust learning in the new proposed setting , we formulate a novel module called sparse latent adaptation strategy which adaptively accounts for latent domains . Section 3.1 reviews adaptation strategies popular in the multi-domain context , which our method extends ( and generalizes ) .

Summary: In this paper, the authors proposed latent domain learning for adaptation. Experiments on multiple benchmark datasets show improved result. Several visualizations also illustrate the effectiveness of the proposed approach.