Visual Representation Learning over Latent Domains

1 INTRODUCTION . Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades . They provide a testbed for developing new algorithms and comparing them to existing ones . However , datasets can also narrow down the focus of research into overspecialized solutions and impede developing a broader understanding of the world . In recent years this narrow scope of datasets has been widely questioned ( Torralba & Efros , 2011 ; Tommasi et al. , 2017 ; Recht et al. , 2019 ) and addressing some of these limitations has become a very active area of research . Two actively studied themes to investigate broader learning criteria are multi-domain learning ( Nam & Han , 2016 ; Bulat et al. , 2019 ; Schoenauer-Sebag et al. , 2019 ) and domain adaptation ( Ganin et al. , 2016 ; Tzeng et al. , 2017 ; Hoffman et al. , 2018 ; Xu et al. , 2018 ; Peng et al. , 2019a ; Sun et al. , 2019b ) . While multi-domain techniques focus on learning a single model that can generalize over multiple domains , domain adaptation techniques aim to efficiently transfer the representations that are learned in one dataset to another . Related themes have also been studied in domain generalization ( Li et al. , 2018 ; 2019b ; a ; Gulrajani & Lopez-Paz , 2020 ) and continual learning ( Kirkpatrick et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Riemer et al. , 2019 ) , where the focus lies on learning representations that can generalize to unseen domains , and to preserve knowledge acquired from previously seen tasks , respectively . While there exists no canonical definition for what exactly a visual domain is , previous works in multi-domain learning assume that different subsets of data exist , with some defining characteristic that allows them to be separated from each another . Each subset , indexed by d = 1 , . . . , D , is assigned to a pre-defined visual domain , and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some pθ ( y|x , d ) . In some cases domains are intuitive and their annotation straightforward . Consider a problem where images have little visual relationship , for example joint learning of Omniglot handwritten symbols ( Lake et al. , 2015 ) and CIFAR-10 objects ( Krizhevsky & Hinton , 2009 ) . In this case , it is safe to assume that encoding an explicit domain-specific identifier into pθ is a good idea , and results in the multi-domain literature provide clear evidence that it is highly beneficial to do so ( Rebuffi et al. , 2018 ; Liu et al. , 2019a ; Guo et al. , 2019a ; Mancini et al. , 2020 ) . The assumption that domain labels are always available has been widely adopted in multi-domain learning ; however this assumption is not without difficulty . For one , unless the process of domain annotation is automated due to combining existing datasets as in e.g . Rebuffi et al . ( 2017 ) , their manual collection , curation , and domain labeling is very laborious . And even if adequate resources exist , it is often difficult to decide the optimal criteria for the annotation of d : some datasets contain sketches , paintings and real world images ( Li et al. , 2017 ) , others images captured during day or night ( Sultani et al. , 2018 ) . Automatically collected datasets ( Thomee et al. , 2016 ; Sun et al. , 2017 ) contain mixtures of low/high resolution images , taken with different cameras by amateurs/professionals . There is no obvious answer which of these should form their own distinct domain subset . Moreover , the work of Bouchacourt et al . ( 2018 ) considers semantic groupings of data : they show that when dividing data by subcategories , such as size , shape , etc. , and incorporating this information into the model , then this benefits performance . Should one therefore also encode the number of objects into domains , or their color , shape , and so on ? Given the relatively loose requirement that domains are supposed to be different while related in some sense ( Pan & Yang , 2009 ) , these examples hint at the difficulty of deciding whether domains are needed , and – if the answer to that is yes – what the optimal domain criteria are . And note that even if such assignments are made very carefully for some problem , nothing guarantees that they will transfer effectively to some other task . This paper carefully investigates this ambiguity and studies two central questions : 1 . Are domain labels always optimal for learning multi-domain representations ? 2 . Is it possible to learn models that generalize well over visually diverse domains without domain labels ? To study this problem , we introduce a new setting ( c.f . Fig . 1 ) in which models are learned over multiple domains without domain annotations — or latent domain learning for short . While latent domain learning is a highly practical research problem in the context of transfer learning , it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning . In particular , we find that the removal of domain associations leads to severe performance losses for standard architectures on most domains , due to imbalances in the underlying distribution and different difficulty levels of the associated domain-level tasks . We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning ( Rebuffi et al. , 2018 ; Chang et al. , 2018 ) , and find that their performance benefits do not extend to latent domain learning . To account for this lost performance , we formulate a novel method called sparse latent adaptation ( Section 3.2 ) which enables internal feature representations to dynamically adapt to instances from multiple domains in data , without requiring annotations for this . Furthermore , we show that latent domain learning naturally extends to several challenging real world tasks and single domain data , such as fairness problems ( Appendix F ) , and learning over imbalanced distributions ( Appendix G ) . 2 LATENT DOMAIN LEARNING . This section provides an overview over latent domain learning and contrasts it against other types of related learning problems , in particular multi-domain learning . 2.1 PROBLEM SETTING . When learning on multiple domains , the common assumption is that data is sampled i.i.d . from a mixture of distributions Pd with domain indices d = 1 , . . . , D. Together , they constitute the datagenerating distribution as P = ∑ d πdPd , where each domain is associated with a relative share πd = Nd/N , with N the total number of samples , and Nd those belonging to the d ’ th domain . In multi-domain learning , domain labels are available for all samples ( Nam & Han , 2016 ; Rebuffi et al. , 2017 ; 2018 ; Bulat et al. , 2019 ) , such that the overall data available for learning consists of DMD = { ( xi , di , yi ) } with i = 1 , . . . , N . In latent domain learning the information associating each sample xi with a domain di is not available . As such , domain-specific labels yi can not be inferred from sample-domain pairs ( xi , di ) and one is instead forced to learn a single model fθ over the latent domain dataset DLD = { ( xi , yi ) } . While latent domain learning can include mutually exclusive classes and disjoint label spaces Y1 ∪ · · · ∪ YD ( as in long-tailed recognition , see Appendix G ) , we mainly focus on the setting of shared label spaces , i.e . Yd = Yd′ . For example a dataset may contain images of dogs or elephants that can appear as either photos , paintings , or sketches . Latent domains have previously attracted interest in the context of domain adaptation , where the lack of annotations was recovered through hierarchical Hoffman et al . ( 2012 ) and kernel-based clustering ( Gong et al. , 2013 ) , via exemplar SVMs ( Xu et al. , 2014 ) , or by measuring mutual information ( Xiong et al. , 2014 ) . More recent work corrects batch statistics of domain adaptation layers using Gaussian mixtures ( Mancini et al. , 2018 ) , or studies the shift from some source domain to a target distribution that contains multiple latent domains ( Peng et al. , 2019b ; Matsuura & Harada , 2020 ) . Latent domain learning however differs fundamentally from these works : Table 1 contains a comparison to existing transfer learning settings . A common baseline in multi-domain learning is to finetune D models , one for each individual domain ( Rebuffi et al. , 2018 ; Liu et al. , 2019a ) . Doing so requires learning a large number of parameters and shares no parameters across domains , but can serve as a strong baseline to compare against . We show that in many cases , even when domains were carefully annotated , a dynamic latent domain approach can surpass the performance of such domain-supervised baselines , see Section 4 . 2.2 OBSERVED VS . UNIFORM ACCURACY . Consider a problem in which the data is sampled i.i.d . from P = πaPda + πbPdb , i.e . two hidden domains . Since domain labels are not available in latent domain learning , the best one can do is to treat all samples equally , and measure the observed accuracy : OAcc [ f ] = E ( xi , yi ) ∼P [ 1yf ( xi ) =yi ] , ( 1 ) where yf denotes the class assigned to sample xi by the model f , and yi its corresponding label for training . The OAcc has a problematic property : if P consists of two imbalanced domains such that πa ≥ πb , then the performance on da dominates it . For example if da has a 90 % overall share , and the model perfectly classifies this domain while obtaining 0 % accuracy on db , then OAcc would still assume 0.9 , hiding the underlying damage to domain db . This motivates alternative formulations for latent domain learning , to anticipate ( and account for ) imbalanced domains in real world data . If it is possible to identify some semantic domain labeling ( as typically included in multi-domain/domain adaptation benchmarks ) , one can compare performances across individual subgroups . This allows picking up on performance losses that just occur on some of them , and which traditional metrics ( such as OAcc ) fail to capture . Where this is possible , we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes : UAcc [ f ] = 1 D D∑ d=1 E ( xi , yi ) ∼Pd [ 1yf ( xi ) =yi ] . ( 2 ) Returning to the above example , a uniform measurement reflects the model ’ s lack of performance on db as UAcc = 0.5 . Once again note while ground-truth domain annotations are required in order to compute uniform accuracy , these are never used to train latent domain models . 3 METHODS . To enable robust learning in the new proposed setting , we formulate a novel module called sparse latent adaptation strategy which adaptively accounts for latent domains . Section 3.1 reviews adaptation strategies popular in the multi-domain context , which our method extends ( and generalizes ) .

Summary: The paper addresses the problem of learning a classification model on data from multiple domains, when explicit domain assignment for each data point is not provided.   To solve this problem, the paper proposes the data to be coming from 1 of K unknown (or latent) domains. A single 1x1 convolutional layer per latent domain is applied to feature maps in each residual layer of ResNet. A gating function (based on sparsemax) is proposed to decide which of the K convolutional layers will be applied. The entire network is learned jointly by minimizing the classification objective function.