Year: 2021

Abstract:
A fundamental shortcoming of deep neural networks is their specialization to a single task and domain. While multi-domain learning enables the learning of compact models that span multiple visual domains, these rely on the presence of domain labels, in turn requiring laborious curation of datasets. This paper proposes a less explored, but highly realistic new setting called latent domain learning: learning over data from different domains, without access to domain annotations. Experiments show that this setting is particularly challenging for standard models and existing multi-domain approaches, calling for new customized solutions: a sparse adaptation strategy is formulated which adaptively accounts for latent domains in data, and significantly enhances learning in such settings. Our method can be paired seamlessly with existing models, and boosts performance in conceptually related tasks, e.g. empirical fairness problems and long-tailed recognition.

heading: 1 INTRODUCTION
text: Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades. They provide a testbed for developing new algorithms and comparing them to existing ones. However, datasets can also narrow down the focus of research into overspecialized solutions and impede developing a broader understanding of the world.
In recent years this narrow scope of datasets has been widely questioned (Torralba & Efros, 2011; Tommasi et al., 2017; Recht et al., 2019) and addressing some of these limitations has become a very active area of research. Two actively studied themes to investigate broader learning criteria are multi-domain learning (Nam & Han, 2016; Bulat et al., 2019; Schoenauer-Sebag et al., 2019) and domain adaptation (Ganin et al., 2016; Tzeng et al., 2017; Hoffman et al., 2018; Xu et al., 2018; Peng et al., 2019a; Sun et al., 2019b). While multi-domain techniques focus on learning a single model that can generalize over multiple domains, domain adaptation techniques aim to efficiently transfer the representations that are learned in one dataset to another.
Related themes have also been studied in domain generalization (Li et al., 2018; 2019b;a; Gulrajani & Lopez-Paz, 2020) and continual learning (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017; Riemer et al., 2019), where the focus lies on learning representations that can generalize to unseen domains, and to preserve knowledge acquired from previously seen tasks, respectively.
While there exists no canonical definition for what exactly a visual domain is, previous works in multi-domain learning assume that different subsets of data exist, with some defining characteristic that allows them to be separated from each another. Each subset, indexed by d = 1, . . . , D, is assigned to a pre-defined visual domain, and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some pθ(y|x, d). In some cases domains are intuitive and their annotation straightforward. Consider a problem where images have little visual relationship, for example joint learning of Omniglot handwritten symbols (Lake et al., 2015) and CIFAR-10 objects (Krizhevsky & Hinton, 2009). In this case, it is safe to assume that encoding an explicit domain-specific identifier into pθ is a good idea, and results in the multi-domain literature provide clear evidence that it is highly beneficial to do so (Rebuffi et al., 2018; Liu et al., 2019a; Guo et al., 2019a; Mancini et al., 2020).
The assumption that domain labels are always available has been widely adopted in multi-domain learning; however this assumption is not without difficulty. For one, unless the process of domain
annotation is automated due to combining existing datasets as in e.g. Rebuffi et al. (2017), their manual collection, curation, and domain labeling is very laborious.
And even if adequate resources exist, it is often difficult to decide the optimal criteria for the annotation of d: some datasets contain sketches, paintings and real world images (Li et al., 2017), others images captured during day or night (Sultani et al., 2018). Automatically collected datasets (Thomee et al., 2016; Sun et al., 2017) contain mixtures of low/high resolution images, taken with different cameras by amateurs/professionals. There is no obvious answer which of these should form their own distinct domain subset.
Moreover, the work of Bouchacourt et al. (2018) considers semantic groupings of data: they show that when dividing data by subcategories, such as size, shape, etc., and incorporating this information into the model, then this benefits performance. Should one therefore also encode the number of objects into domains, or their color, shape, and so on?
Given the relatively loose requirement that domains are supposed to be different while related in some sense (Pan & Yang, 2009), these examples hint at the difficulty of deciding whether domains are needed, and – if the answer to that is yes – what the optimal domain criteria are. And note that even if such assignments are made very carefully for some problem, nothing guarantees that they will transfer effectively to some other task.
This paper carefully investigates this ambiguity and studies two central questions:
1. Are domain labels always optimal for learning multi-domain representations? 2. Is it possible to learn models that generalize well over visually diverse domains without
domain labels?
To study this problem, we introduce a new setting (c.f. Fig. 1) in which models are learned over multiple domains without domain annotations — or latent domain learning for short.
While latent domain learning is a highly practical research problem in the context of transfer learning, it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning. In particular, we find that the removal of domain associations leads to severe performance losses for standard architectures on most domains, due to imbalances in the underlying distribution and different difficulty levels of the associated domain-level tasks.
We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning (Rebuffi et al., 2018; Chang et al., 2018), and find that their performance benefits do not extend to latent domain learning. To account for this lost performance, we formulate a novel method called sparse latent adaptation (Section 3.2) which enables internal feature representations to dynamically adapt to instances from multiple domains in data, without requiring annotations for this. Furthermore, we show that latent domain learning naturally extends to several challenging real world tasks and single domain data, such as fairness problems (Appendix F), and learning over imbalanced distributions (Appendix G).

heading: 2 LATENT DOMAIN LEARNING
text: This section provides an overview over latent domain learning and contrasts it against other types of related learning problems, in particular multi-domain learning.

heading: 2.1 PROBLEM SETTING
text: When learning on multiple domains, the common assumption is that data is sampled i.i.d. from a mixture of distributions Pd with domain indices d = 1, . . . , D. Together, they constitute the datagenerating distribution as P = ∑ d πdPd, where each domain is associated with a relative share πd = Nd/N , with N the total number of samples, and Nd those belonging to the d’th domain. In multi-domain learning, domain labels are available for all samples (Nam & Han, 2016; Rebuffi et al., 2017; 2018; Bulat et al., 2019), such that the overall data available for learning consists of DMD = {(xi, di, yi)} with i = 1, . . . , N . In latent domain learning the information associating each sample xi with a domain di is not available. As such, domain-specific labels yi cannot be inferred from sample-domain pairs (xi, di) and one is instead forced to learn a single model fθ over the latent domain dataset DLD = {(xi, yi)}. While latent domain learning can include mutually exclusive classes and disjoint label spaces Y1 ∪ · · · ∪ YD (as in long-tailed recognition, see Appendix G), we mainly focus on the setting of shared label spaces, i.e. Yd = Yd′ . For example a dataset may contain images of dogs or elephants that can appear as either photos, paintings, or sketches.
Latent domains have previously attracted interest in the context of domain adaptation, where the lack of annotations was recovered through hierarchical Hoffman et al. (2012) and kernel-based clustering (Gong et al., 2013), via exemplar SVMs (Xu et al., 2014), or by measuring mutual information (Xiong et al., 2014). More recent work corrects batch statistics of domain adaptation layers using Gaussian mixtures (Mancini et al., 2018), or studies the shift from some source domain to a target distribution that contains multiple latent domains (Peng et al., 2019b; Matsuura & Harada, 2020). Latent domain learning however differs fundamentally from these works: Table 1 contains a comparison to existing transfer learning settings.
A common baseline in multi-domain learning is to finetune D models, one for each individual domain (Rebuffi et al., 2018; Liu et al., 2019a). Doing so requires learning a large number of parameters and shares no parameters across domains, but can serve as a strong baseline to compare against. We show that in many cases, even when domains were carefully annotated, a dynamic latent domain approach can surpass the performance of such domain-supervised baselines, see Section 4.

heading: 2.2 OBSERVED VS. UNIFORM ACCURACY
text: Consider a problem in which the data is sampled i.i.d. from P = πaPda + πbPdb , i.e. two hidden domains. Since domain labels are not available in latent domain learning, the best one can do is to treat all samples equally, and measure the observed accuracy:
OAcc[f ] = E(xi,yi)∼P[1yf(xi)=yi ], (1)
where yf denotes the class assigned to sample xi by the model f , and yi its corresponding label for training. The OAcc has a problematic property: if P consists of two imbalanced domains such that πa ≥ πb, then the performance on da dominates it. For example if da has a 90% overall share, and the model perfectly classifies this domain while obtaining 0% accuracy on db, then OAcc would still assume 0.9, hiding the underlying damage to domain db.
This motivates alternative formulations for latent domain learning, to anticipate (and account for) imbalanced domains in real world data. If it is possible to identify some semantic domain labeling (as typically included in multi-domain/domain adaptation benchmarks), one can compare performances across individual subgroups. This allows picking up on performance losses that just occur on some of them, and which traditional metrics (such as OAcc) fail to capture.
Where this is possible, we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes:
UAcc[f ] = 1
D D∑ d=1 E(xi,yi)∼Pd [1yf (xi)=yi ]. (2)
Returning to the above example, a uniform measurement reflects the model’s lack of performance on db as UAcc = 0.5. Once again note while ground-truth domain annotations are required in order to compute uniform accuracy, these are never used to train latent domain models.

heading: 3 METHODS
text: To enable robust learning in the new proposed setting, we formulate a novel module called sparse latent adaptation strategy which adaptively accounts for latent domains. Section 3.1 reviews adaptation strategies popular in the multi-domain context, which our method extends (and generalizes).

heading: 3.1 LATENT ADAPTATION
text: When domain labels d are available (not the case in latent domain learning) one strategy established by Rebuffi et al. (2017) is to modulate networks by constraining the layerwise transformation of residual networks (He et al., 2016) Φ(x) = x + f(x) to allow at most a linear change Vd per each domain from some pretrained mapping Φ0 (with f0 in every layer), whereby Φ(x)−Φ0(x) = Vdx. Note the slight abuse of notation here in letting x denote a feature map with channels C. Rearranging this yields:
Φ(x, d) = x+ f0(x) + D∑ d=1 gdVd(x), (3)
with a domain-supervised switch that assigns corrections to domains, i.e. gd = 1 for d associated with x and 0 otherwise. Each Vd is parametrized through 1x1 convolutions, and f0 denotes a shared 3x3 convolution obtained e.g. on ImageNet (Deng et al., 2009). This builds on the assumption that models with strong general-purpose representations require minimal changes to adapt to new tasks (Bilen & Vedaldi, 2017), making learning each Vd sufficient, while f0 remains as is. Such adaptation strategies have been successfully used in NLP (Stickland & Murray, 2019) and few shot (Li et al., 2021) settings to restrict the number of learnable parameters there.
In latent domain learning access to d is removed, resulting in two new challenges: we have no a priori information about the right number of corrections {Vd}, and we cannot use d to decide which one of these to apply.
To mitigate the lack of domain labels d, first we assume that input data is constituted by K latent distributions Pk. Second we propose to replace the switch gd with a learnable gating mechanism g1(x), . . . , gK(x) that assigns each sample x to latent domains as follows:
Φ(x) = x+ f0(x) + K∑ k=1 gk(x)Vk(x), (4)
The gates gk control which convolution is applied to which sample x, and correspond to a categorical variable over K categories, i.e. 0 ≤ gk ≤ 1 and ∑ k gk = 1. Note in particular how parametric dependency of Φ on d is removed. How to best choose K is discussed in more detail in Section 4.
While we motivate our latent domain module from learning over multiple domains, the main goal is not to recover the domain labels annotated in some datasets. When optimizing some loss (standard cross-entropy in the classification case), there is no guarantee that the learned Vk will correspond to an annotated visual domain and many additional factors (shape, pose, color, etc.) can enter them as
well. Latent domain models are simply optimized to produce the lowest training error, and in fact seldom recover ground-truth domains.
Note the broader concept presented here may in principle also be incorporated with other multi-task concepts (Perez et al., 2018; Guo et al., 2019a), adaptation strategies however stand out due to their methodological simplicity.
Different options exist for parametrizing the gating function g : X → G ⊆ RK . An ideal gating mechanism for latent domain learning would fulfill two seemingly incompatible properties: be able to filter latent domains in some layers (requiring a discrete gate), but also share parameters between related latent domains in other layers (smooth gates). The next section proposes how this can be resolved without requiring task relationships (Vandenhende et al., 2020) or outer optimization loops (Wu et al., 2018) through the use of sparseness.

heading: 3.2 SPARSE LATENT ADAPTERS (SLA)
text: We parameterize the gating function g with a small linear transformation W : C → RK that constitutes the pre-activation q=Wϕ(x) within the gates, where ϕ : X → C denotes an average pooling projection onto the channels.
Different options exist for activating q ∈ RK . A crucial choice is whether the activation should map to some discrete space G={0, 1}K or a continuous G=[0, 1]K in which the Vk are shared. We propose a different strategy that lets gates be smooth when appropriate, but a threshold τ allows for sparse (or discrete) outputs fτ (q) = [q − τ(q)]+ with [·]+ = max(0, ·). Crucially fτ can be solved in a differentiable manner (Martins & Astudillo, 2016) by sorting q1 ≥ · · · ≥ qK , solving k∗ = max{k | 1 + kqk > ∑ j≤k qj} and computing τ = [( ∑ j≤k∗ qj)− 1]/k∗.
Consider q = [0.1, 1.0, 0.5] for which sparse activation results in fτ (q) = [0.0, 0.75, 0.25] while softmax yields [0.202, 0.497, 0.301]. Sparse activation filters out q1, while sharing between q2 and q3. We may now define:
SLA(x) , x+ f0(x) + K∑ k=1 [ fτ ◦W ◦ ϕ(x) ] k Vk(x), (5)
where [·]k picks the k’th element of the gating sequence. To the best of our knowledge sparse activation strategies were never previously employed for expert models in computer vision and have so far been restricted to the NLP setting (Deng et al., 2017; Peters et al., 2019). Note SLA generalizes residual adaption (Rebuffi et al., 2017; 2018), which is recovered by setting K= 1.
While gating is subject to complex interactions such as negative transfer (Rosenbaum et al., 2019), our ablations in Table 5 clearly show that taking a sparse perspective – which allows the model to assume either continuous or discrete forms – outperforms the alternative of a priori fixing either smoothness through self-attention (Lin et al., 2017b) or discrete Gumbel-based sampling (Jang et al., 2016). In fact, this choice between discrete (Veit & Belongie, 2018; Guo et al., 2019b) and continuous mechanisms (Shazeer et al., 2017; Sun et al., 2019a; Wang et al., 2019) delineates previous work that employs differentiable gates.
It should be noted that a softmax-activated model can in principle also learn to suppress individual preactivation components by letting some qk go to −∞. This however requires either learning extra calibration parameters at every layer, defining a hard cutoff value (Shazeer et al., 2017) (thereby removing differentiability), or very large row-norms within the linear mapping W— a highly unlikely outcome given the several mechanisms found in state-of-the-art models (in particular weight decay, norm-penalties, or BN (Ioffe & Szegedy, 2015)) which act as direct counterforces to this.

heading: 4 EXPERIMENTS
text: We evaluate our proposed methods on three latent domain benchmarks: Office-Home, PACS, and DomainNet (c.f. Fig. 6, which shows example images from these benchmarks). Note the main goal here is not to compare to existing multi-domain or domain adaptation methods that these datasets were initially designed for, but to study our two central research questions: whether domain labels
are optimal for effectively learning over multiple domains, and whether we can learn such representations without domain labels.
We also examine a recent fairness benchmark (see Appendix F), and show that SLA significantly improves robustness under single domain long-tailed distributions (Appendix G). All experiments were implemented in PyTorch (Paszke et al., 2017), and code will be released alongside the final paper version.
Optimization In all experiments, we build our method on a ResNet26 model pretrained on ImageNet (Deng et al., 2009). In SLA only gates and corrections are learned, the backbone f0 remains fixed at its initial parameters which implicitly regularizes the model (Rebuffi et al., 2017). We train all models in exactly the same way by the same default hyperparameters: for 120 epochs using stochastic gradient descent (momentum parameter of 0.9), batch size of 128, weight decay of 10−4, and an initial learning rate of 0.1 (reduced by 1/10 at epochs 80, 100). We use official splits in each dataset, and report accuracies over five random initializations (variance is analyzed in Appendix E).
All experiments use standard augmentation techniques: random cropping and flipping, as well as normalization. Increasing the number of corrections K within SLA consistently increases performance, however K = 2 already represents a strong boost from the baseline of having no adapters.
Office-Home The underlying data contains a variety of objects classes (alarm clock, backpack, etc.) among four domains: art, clipart, product, and real world (Venkateswara et al., 2017). In Table 2 we show results for d-supervised multi-domain (MD) approaches: RA (Rebuffi et al., 2018), domain-adversarial learning (Ganin et al., 2016) and a baseline of 4×ResNet26, one for each domain. For latent domain (LD) baselines, we then learn a single ResNet26, this time as a latent domain model over all domains. Next, we couple SLA with the very same ResNet26.
Learning a single ResNet26 over latent domains with no access to d-labels significantly harms performance. This problem is not addressed by simply increasing the depth of the network: while accuracy improves slightly, a ResNet56 exhibits the same performance losses — in particular on the latent domains product (P) and real world (R).
While residual adaptation (RA) (Rebuffi et al., 2018) was shown to work extremely well in many multi-domain scenarios, performance here is sub-par, regardless of whether it accesses d (MD: one Vd per-domain) or not (LD). This likely results from linear modules being reserved for each d when using annotations, enabling no native cross-domain sharing of parameters. When d is hidden on the other hand, the model is forced to share a single linear adaptation module V between all four hidden domains, without the flexible gating we propose in SLA.
Learning annotations through latent domain clustering and coupling this with domain-adversarial gradient reversal as in MMLD (Matsuura & Harada, 2020) increases performance relative to its dannotated counterpart (Ganin et al., 2016). The increase is modest however, likely because enforcing
domain-invariance on the gradient level negatively impacts the model’s ability to discriminate between classes (Wang et al., 2020). Another related baseline is MLFN (Chang et al., 2018) which builds on ResNeXt (Xie et al., 2017) to define a latent-factor architecture that accounts for multimodality in data. Crucially where our method is fine-grained and shares convolutions at every layer, MLFN instead enables and disables entire network blocks, allowing us to outperform MLFN.
SLA outperforms all currently available latent domain models by a consistent margin, and increases UAcc by 12.79% relative to ResNet26. Best performance is obtained when K happens to equal D, with performance being reducing slightly from overfitting of larger domains for K > D.
PACS The second experiment examines performance on the PACS dataset (Li et al., 2017). Crucially PACS domains (art, cartoon, photo, sketch) differ more markedly from one another (c.f. examples in Fig. 6), hence constituting a demanding latent domain problem.
Even for more distinct domains as in PACS, results in Table 3 show that SLA improves over existing baselines. The largest gains occur on smaller domains (e.g. art), where standard models suppress underrepresented parts of the distribution (see additional discussion on imbalanced distributions in Appendix G). Our method again surpasses the accuracy of 4×ResNet26, while requiring a fraction of the total parameters (∼ 9.7 mil for K = 5 vs. ∼ 24.8 mil). The performance of SLA again continues to increase with larger K.
The performance increase from using a latent domain-adversarial approach (Matsuura & Harada, 2020) versus using domain-annotations (Ganin et al., 2016) confirms that learning domains alongside the rest of the network can be a better strategy than trusting in annotations. Our approach again improves over this, without requiring a clustering stage as in MMLD.
Results for k-means (we used D= 4 centers and clustered on both image and feature levels without noticeable changes in performance) and subsequent finetuning show that a two-stage strategy is suboptimal. This is not surprising since, similar to d-supervision via gd in Φ of eq. (3), clustering learns fixed switches that get used across all layers. In SLA we flexibly share or separate features individually at every layer (c.f. qualitative results in Fig. 3), synergizing only where appropriate.
DomainNet We also evaluate models on a large-scale benchmark called DomainNet (Peng et al., 2019a). This dataset contains 518 447 images from six domains (clipart, painting, photos, sketch,
infographics, and quickdraw), with a total of |Y| = 345 object classes. The optimization settings remain unchanged from those in previous sections.
Results are shown in Table 4. MLFN performs best on quickdraw, a domain that differs visibly from others (c.f. Fig. 6 for examples from each domain), and having entire network blocks dedicated to it seems to benefit performance. On all remaining domains, SLA (K = 2) outperforms existing models, regardless of whether they were designed specifically for multi-domain problems, such as RA, or whether they are much deeper/parameter-intensive (ResNet56).
Qualitative analysis We (i) compare global statistics of Office-Home and PACS domains as well as (ii) their per-layer treatment within SLA; (iii) analyze sparse gating, (iv) representations learned by SLA, and show that (v) our module shares between geometric properties (shape, pose, etc.).
i) Fig. 2: average cosine similarities of per-domain gating vectors g∈GL across l= 1, . . . , L layers of ResNet26 show that Office-Home domains differ less than those in PACS.
ii) Fig. 3: layerwise measurements of Corr[gl(x), gl(x′)] for x, x′ drawn from differing d 6= d′ for Office-Home. If inter-domain correlation is high, then similar corrections Vk are responsible for processing samples from two domains. Across top layers of the network there is little correlation, presumably as low-level information associated with each domain is processed independently. In the mid to bottom stages correlation increases: these layers are typically associated with higherorder features (Yosinski et al., 2014; Mahendran & Vedaldi, 2016; Asano et al., 2020), and since
label spaces are shared between latent domains, similar object-level features are required to classify objects into their respective categories.
iii) Fig. 4: sparse gates have the flexibility to either output singular activations (i.e. become fully discrete) or all non-zero values (a continuous gate). We measure the per-layer sparsity Ex∼Pd [K− ‖gl(x)‖0]/(K − 1) where ‖ · ‖0 counts values different from zero, finding sparsity of SLA to vary across model depth. Interestingly after each downsampling operation SLA tends to be relatively sparse, followed by a dense gate, then again a sparse one, and so forth. The model thus utilizes the extra flexibility resulting from sparse gates.
Due to PACS domains being relatively distinctive, the dataset is an interesting candidate for additional analysis in (iv) and (v) of how sparse adaptation handles the different ground-truth domains.
iv) Fig. 5 (left): gate vectors g ∈ GL for samples from all four domains in PACS visualized by their principal components. SLA exhibits an intuitive clustering of human-annotated PACS domains: visually similar art and photo (•,•) cluster together. The manifold describing sketches (•) is arguably more primitive than those of the other domains, and indeed only maps to a small region. Cartoon (•) lies somewhere between sketches and real world images. This matches intuition: a cartoon is, more or less, just a colored sketch.
Fig. 5 also highlights one sample that shows an elephant that SLA places among the cartoon (•) domain, but which has been assigned a ground-truth domain label of photo (•) in the PACS dataset. The ground-truth label seems to have been annotated in error, but different from approaches that use d-supervision, our SLA processes latent domains on-the-fly and is therefore not irritated by this.
v) Fig. 5 (right): pairs of samples with similar gates. This shows that latent domains are indicative of more than ground-truth domain labels and extend to geometric similarities: pose, color, etc. of the samples are visibly related. Compare in particular the poses of elephants (second row).

heading: 5 CONCLUSION
text: In this paper we explored two questions: (i) whether domain associations are required for learning effective models over multiple visual domains – as assumed by all existing work – and (ii) if manually curated domain labels are optimal for parametrizing visual representations.
As has been shown, the performance of existing models does degrade without domain labels, raising doubts about their suitability for realistic problems that involve diverse data sources. As a remedy, we proposed a novel adaptation strategy which reclaims (and often exceeds) lost accuracy on latent domains, benefiting several problems where some notion of a domain (but no annotation) exists.

heading: A DATASETS
text: Fig. 6 shows examples from the latent domain benchmarks evaluated in Section 4. The selected images have equivalent classes yd = yd′ ∈ Y (for example chair for Office-Home), but different domains (e.g. d = {art, clipart, product, real world}). These examples show that data from different domains often contain very different visual characteristics (compare e.g. photo vs. sketch for PACS), even when the object is the same. At the same time, other domains are more alike (e.g. art and photo), indicating that different amounts of sharing between per-domain parameters are required, which in SLA is facilitated by its gating mechanism.

heading: B RELATED WORK
text: Multi-domain learning relates most closely to our work. The state-of-the-art methods introduce small convolutional corrections in residual networks to account for individual domains (Rebuffi et al., 2017; 2018), which was recently extended to obtain efficient multi-task models for related language tasks Stickland & Murray (2019). Other work makes use of task-specific attention mechanisms (Liu et al., 2019a), attempts to scale task-specific losses (Kendall et al., 2018), or addresses tasks at the level of gradients (Chen et al., 2017). Crucially, these approaches all rely firmly on domain labels.
Our work is loosely related to learning universal representations (Bilen & Vedaldi, 2017), which was used as a guiding principle in designing more transferable models Tamaazousti et al. (2019). However, these works also assume the presence of domain labels. One recent line of work does not make this assumption, by normalizing data in separate batches to account for differences in feature distributions Deecke et al. (2019), or by accounting for latent semantic factors to match images Chang et al. (2018). As we show in our experiments (Section 4), latent domain learning however benefits from more customized solutions than these.
The proposed module gives rise to a differentiable dynamic network architecture, studied e.g. for reinforcement learning (Zoph & Le, 2016; Pham et al., 2018), Bayesian optimization (Kandasamy et al., 2018), or when adapting to new tasks (Mallya et al., 2018; Rosenfeld & Tsotsos, 2018). For such architectures, two components are commonly used: discrete Gumbel-based sampling (Jang et al., 2016), e.g. leveraged in dynamic computer vision architectures (Veit & Belongie, 2018; Sun et al., 2019a), or continuous self-attentive approaches (Lin et al., 2017b), which have been used successfully to scale models to large problem spaces (Shazeer et al., 2017; Wang et al., 2019).
From the perspective of algorithmic fairness, a desirable model property is to ensure consistent predictive equality across different identifiable subgroups in data (Zemel et al., 2013; Hardt et al., 2016; Fish et al., 2016). This relates to one of the goals in latent domain learning: to limit implicit model bias towards large domains, and improve robustness on small domains. Recent work explores connections between models and empirical fairness for visual recognition (Bagdasaryan et al., 2019; Hooker et al., 2020; Wang et al., 2020), different from our experiments however (see Appendix F) they focus their analysis on a setting in which annotations for protected attributes are available.

heading: C ABLATION
text: Replacing sparse gating within SLA registers a drop in performance, regardless of whether smooth or discrete mechanisms are used. Accuracies for soft and straight-through Gumbel-softmax sampling (Jang et al., 2016) were on par; we report straight-through sampling here.
We also ran experiments where we did not fix the residual backbone f0 but updated its parameters alongside the learning of SLA. In line with what Rebuffi et al. (2017) report, this lead to overfitting and performance dropped to UAcc = 73.53.

heading: D MEMORY REQUIREMENTS
text: In SLA every layer contains O(K|C| + K|C|2) parameters to parametrize gates and corrections Vk, respectively. This is however an extremely modest requirement, in particular because f0 stays fixed: while a ResNet26 contains ∼ 6.2 mil learnable parameters, even when setting K = 5 within SLA it has just ∼ 3.5 mil free parameters, and is a fraction of the number of parameters needed to parametrize four ResNet26 (around 24.8 mil parameters).
Note also that the complexity of solving sparse gates in SLA scales as O(K logK), a negligible increase given the small K required in our method.
E VARIATION OF RESULTS
Fig. 7 displays performance variations for OfficeHome (left) and PACS (right). The variance is observed to be relatively low across experiments, as we generally found SLA to be highly robust to optimize. Larger K consistently increase performance, at the cost of a slight increase in learnable parameters (see previous section). Performance gains from higher K eventually saturate, as more and more latent domains are covered by the model. Note the performance trends hold in terms of both uniform and observed accuracy.

heading: F FAIRNESS
text: Recent work elevated the role of small subgroups in data and examined model fairness on CelebA (Bagdasaryan et al., 2019; Wang et al., 2020; Hooker et al., 2020). Because such subgroups may be interpreted as constituting an individual latent domain component Pd, they are an interesting candidate to evaluate our purpose-built SLA on.
The benchmark contains different labeled attributes (e.g. “brown hair”, “glasses”), and is modified from the original dataset by hiding gender labels. Models are evaluated on all 39 remaining
Table 6: Average precision and bias amplification of SLA on the CelebA fair attribute recognition benchmark (Wang et al., 2020).
ResNet18 + SLA ResNet34 + SLA ResNet50 + SLA
mAP (↑) 71.76 73.22 (+1.46) 71.33 73.98 (+2.65) 74.52 75.03 (+0.51) BA (↓) 0.025 0.014 0.022 0.009 0.012 0.008
0.5 0.6 0.7 0.8 0.9 1.0 Skew
0
2
4
6 8 Ch an ge i n AP [ %]
Figure 8: Change in AP between ResNet18 and ResNet18-SLA for different gender skews in CelebA attributes.
attributes, which subsequently experience varying amounts of gender skew. Framed as a latent domain problem we have d={female,male}, but models have no access to this spurious information. The images used are the entire Aligned&Cropped subset (Liu et al., 2015) over which we finetune residual models, replacing only the fully-connected layer of the network. We use the optimization settings introduced in Section 4 for 70 epochs with reductions at epochs 30, 40, and 50, selecting the best model on the validation split. This experimental setup is identical to previous work on empirical fairness (Wang et al., 2020; Ramaswamy et al., 2020), which however – different from our work – focused on learning models that have access to the gender-attribute d.
We evaluate per-attribute accuracy using mean average precision (mAP) and report bias amplification (BA) (Zhao et al., 2017). This compares the propensity of a model to make positive predictions (i.e. f exceeds some threshold t+ ∈ [0, 1]) in the gender g∗y that appears most frequent within attribute y, compared to the true counted ratio of positive examples y+:
BA[f ] = Ex∼Px [1f(x)>t+|g∗y 1f(x)>t+ ] − Ey∼Py [1y=y+|g∗y 1y=y+ ] , (6)
where t+ is optimized for on the validation split. For example if 60% of male examples are wearing glasses but under the model this is raised to a total of 65%, then bias is amplified by BA = 0.05.
We report performance for ResNet18, ResNet34, and ResNet50 in Table 6 and compare this to the same model with SLA inserted (K = 2). SLA consistently raises both mAP and reduces bias, indicating that it relies less on spurious correlations in data to formulate its predictions.
In Fig. 8 we compare per-attribute skew toward either female or male (whichever is more frequent) to the gain in performance from ResNet18 to the same model but with SLA inserted. We observe a clear trend here, whereby SLA is able to raise performance the most in those attributes that experience the largest amounts of skew.

heading: G LONG-TAILED RECOGNITION
text: Standard models often experience difficulty when some classes are heavily underrepresented. This problem has recently been studied in long-tailed recognition (Liu et al., 2019b; Cao et al., 2019) with
benchmarks that modify CIFAR-10 and CIFAR-100 to an imbalanced version by dropping some classes (e.g. 6-10 for CIFAR-10) (Buda et al., 2018). The severity of the imbalance is described via the ratio ρ = nmax/nmin between the largest and smallest classes.
Long-tailed distributions may be viewed as containing an underrepresented latent component with π = 1/(1 + ρ), and previous results (c.f. Section 4) that fortified small latent domains within P motivate us to evaluate the imbalance setting more closely here.
Since our strategy is entirely model-based, it can be combined with the most recent state-of-theart (loss-based) techniques for long-tailed recognition: a label-distribution-aware margin loss with deferred reweighting (Cao et al., 2019), or reducing contributions from well-classified examples as in focal losses (Lin et al., 2017a). As Table 7 shows, adaptation via sparse gates acts as a regularizer on the underlying ResNet26, and consistently improves performance on long-tail benchmarks.

Review summaries:
In this paper, the authors proposed latent domain learning for adaptation. Experiments on multiple benchmark datasets show improved result. Several visualizations also illustrate the effectiveness of the proposed approach.

The paper addresses the problem of learning a classification model on data from multiple domains, when explicit domain assignment for each data point is not provided.   To solve this problem, the paper proposes the data to be coming from 1 of K unknown (or latent) domains. A single 1x1 convolutional layer per latent domain is applied to feature maps in each residual layer of ResNet. A gating function (based on sparsemax) is proposed to decide which of the K convolutional layers will be applied. The entire network is learned jointly by minimizing the classification objective function.

This paper introduces a new task called latent domain learning and proposes a baseline that extends an existing multi-domain learning method. The latent domain learning assumes that training data are sampled from different domains yet their domain labels are latent, and aims at learning models that generalize well to the domains. The proposed baseline deploys multiple parallel feature transform layers that are chosen on the fly through gating variables, with the hope that the gating variables learn to predict the latent domain label of input and choose feature transforms of the domain accordingly. This model demonstrates superior performance to existing multi-domain learning methods in the latent domain learning setting.   I recognize that latent domain learning is an interesting problem with great potential, and has many applications such as learning using web-crawled images. However, the proposed method looks limited in terms of novelty and the quality of writing is below the standard.