Visual Representation Learning over Latent Domains
Authors: Lucas Deecke
Timothy Hospedales
Hakan Bilen
Venue: ICLR 2022 Poster (2022)
Keywords: transfer learning
latent domains
computer vision

Abstract: A fundamental shortcoming of deep neural networks is their specialization to a single task and domain. While multi-domain learning enables the learning of compact models that span multiple visual domains, these rely on the presence of domain labels, in turn requiring laborious curation of datasets. This paper proposes a less explored, but highly realistic new setting called latent domain learning: learning over data from different domains, without access to domain annotations. Experiments show that this setting is challenging for standard models and existing multi-domain approaches, calling for new customized solutions: a sparse adaptation strategy is formulated which enhances performance by accounting for latent domains in data. Our method can be paired seamlessly with existing models, and benefits conceptually related tasks, e.g. empirical fairness problems and long-tailed recognition.

Published as a conference paper at ICLR 2022
VISUAL REPRESENTATION LEARNING OVER LATENT DOMAINS
Lucas Deecke, Timothy Hospedales & Hakan Bilen University of Edinburgh {1 deecke, hww.peaaess, h. acci
ABSTRACT
A fundamental shortcoming of deep neural networks is their specialization to single task and domain. While multi-domain learning enables the learning of compact models that span multiple visual domains, these rely the presence of domain labels, in turn requiring laborious curation of datasets. This paper pro- poses less explored, but highly realistic new setting called latent domain learn- ing: learning over data from different domains, without access to domain annota- tions. Experiments show that this setting challenging for standard models and existing multi-domain approaches, calling for new customized solutions: a sparse adaptation strategy formulated which enhances performance by accounting for latent domains in data. Our method can be paired seamlessly with existing mod- els, and benefits conceptually related tasks, e.g. empirical fairness problems and long-tailed recognition.
INTRODUCTION
Datasets have been a major driving force behind the rapid progress in computer vision research in the last two decades. They provide a testbed for developing new algorithms and comparing them to existing ones. However, datasets can also narrow down the focus research into overspecialized solutions and impede developing broader understanding of the world.
In recent years this narrow scope of datasets has been widely questioned (Torralba & Efros 201T Tommasi al. 2017] Recht et al.] 2019) and addressing some of these limitations has become very active area of research. Two actively studied themes to investigate broader learning criteria are multi-domain learning (Nam & Han 2016 Bulat et al., 2019 Schoenauer-Sebaar
al. 2019) and
domain adaptation (Ganin et al. [016]] Tzeng et al. 2017] Hoffman et al.) [0088] Xu et al. 2018
Peng et al. 2019a Sun al. (019b)). While multi-domain techniques focus on learning a single model that can generalize over multiple domains, domain adaptation techniques aim to efficiently
transfer the representations that are learned in one dataset another.
Related themes have also been studied in domain generalization (Li et al. 2018 2019b a Gulrajani & Lopezz-aaz 2020) and continual learning (Kirkpatrick al. 2017 Lopez-Paz Ranzato 2017
Riemer et al. 2019), where the focus lies on learning representations that can generalize to unseen domains, and to preserve knowledge acquired from previously seen tasks, respectively.
While there exists no canonical definition for what exactly visual domain is, previous works in multi-domain learning assume that different subsets of data exist, with some defining characteristic that allows them to be separated from each other. Each subset, indexed by d = 1, D, is as- signed to pre-defined visual domain, and vice-versa multi-domain methods then use such domain associations to parameterize their representations and learn some ps (ylzx, d).
In some cases domains are intuitive and their annotation straightforward. Consider problem where images have little visual relationship, for example joint learning of Omniglot handwritten symbols (Lake et al. 2015) and CIFAR-10 objects (Krizhevsky & Hinton 2009). In this case, it is safe to assume that encoding explicit domain-specifie identifier into pe is good idea, and results in the multi-domain literature provide clear evidence that is beneficial to do so (Rebuffi et al. 2018 Liu
al. 2019a Guo al. 2019a Mancini al. 2020

---

Published as a conference paper at ICLR 2022
Multi-Domain
Latent Domain
Seiessors
Lamp
Classes -
Dog
Horse
Dog
Art
Photo
Domains
Figure I: In multi-domain learning every sample has domain label. Latent domain learning studies how models may best be learned without this information.
The assumption that domain labels are always available has been widely adopted in multi-domain learning; however this assumption is not without difficulty. For one, unless the process of domain annotation is automated due to combining existing datasets as in e.g. |Rebufi al. (2017). their manual collection, curation, and domain labeling is very laborious.
And even if adequate resources exist, is often difficult to decide the optimal criteria for the anno- tation of d: some datasets contain sketches, paintings and real world images (L.i et al. (017)) others images captured during day or night (Sultani et al., 2018). Automatically collected datasets (Thomee) al. 2016; Sun et al. 2017) contain mixtures of low/high resolution images, taken with different cameras by umatturs/professionaal own distinct domain subset.
There is no obvious answer which of these should form their
Moreover, the work of Bouchacourt et al. (2018) considers semantic groupings of data: they show that when dividing data by subcategories, such size, shape, etc., and incorporating this information into the model, then this benefits performance. Should one therefore also encode the number of objects into domains, or their color, shape, and on?
Given the relatively loose requirement that domains are supposed to be different while related in some sense (Pan & Yang) 2009). these examples hint at the difficulty of deciding whether domains are needed, and - if the answer to that is yes what the optimal domain criteria are. And note that even such assignments are made very carefully for some problem, nothing guarantees that they will transfer effectively to some other task.
This paper carefully investigates this ambiguity and studies two central questions:
1. Are domain labels always optimal for learning
representations?
2. How can models best be learned that generalize well over visually diverse domains, without domain labels?
To study this problem, we introduce a new setting (c.f. Fig. 1 in which models are learned over multiple domains without domain annotations latent domain learning for short.
While latent domain learning is highly practical research problem in the context of transfer learn- ing, it poses multiple challenges that have not been previously investigated in connection with deep visual representation learning. In particular, we find that the removal of domain associations leads to performance losses for standard architectures due to imbalances in the underlying distribution and different difficulty levels of the associated domain-level tasks.
We carry out a rigorous quantitative analysis that includes concepts from multi-domain learning (Re- butfi et al. 2018 Chang al. 2018), and find that their performance benefits do not directly extend to latent domain learning. To account for this lost performance, we formulate a novel method called sparse latent adaptation (Section 3.22) which enables internal feature representations dynamically adapt to instances from multiple domains in data, without requiring annotations for this. Moreover, show that latent domain methods appear benefit single domain data and real world tasks, such as fairness problems (Appendix F), and long-tailed recognition (Appendix G).
2 LATENT DOMAIN LEARNING
This section provides an overview over latent domain learning and contrasts it against other types of related learning problems, in particular multi-domain learning.
2

---

Published as a conference paper at ICLR 2022
A comparison of latent domain learning versus unsupervised, latent source (Mancini al., 2018) and semi-supervised DA, domain generalization, and multi-domain learning. Sa denotes labeled dataset from the d' d'th domain, Pa are partially labeled, Us Ua unlabeled.
Table 1:
Setting
Domain Labels
Training Data
Evaluation Data
Unsupervised DA Latent Source DA
Yes
Si...., Sp, UD+1
ÜD+1
No
Smixture: UD+1
ÜD+1
Semi-supervised DA
Yes
S Sp, PD-1
UD+1
Domain Generalization
Yes
S Sp
UD+1
Multi-Domain Learning
Yes
S Sp
UI,...,Up
Latent Domain Learning
No
S mixture
UI,..., Un
2.1
PROBLEM SETTING
When learning multiple domains, the common assumption is that data is sampled i.i.d. from mixture of distributions P with domain indices d = 1..... D. Together, they constitute the data- generating distribution L Ea TelPa where each domain is associated with a relative share Td = =Na/N, with N the total number of samples, and N₂ those belonging to the d'th domain. In multi-domain learning, domain labels are available for all samples (Nam & Han 2016 Rebuffi
et al. 2017 2018 Bulat et al. 2019), such that the overall data available for learning consists of [1,,, d, y:) with
DMD
N.
In latent domain learning the information associating each sample x; with a domain d; is not avail- able. As such, domain-specific labels y; cannot be inferred from sample-domain pairs (Ia, de) and one is instead forced to learn a single model se over the latent domain dataset LLD = { M-) While latent domain learning can include mutually exclusive classes and disjoint label spaces Y11 U... U.D (as in long-tailed recognition, see Appendix G), we mainly focus on the setting of shared label spaces, i.e. Vd- Var For example a dataset may contain images of dogs or elephants that can appear as either photos, paintings, or sketches.
Latent domains have previously attracted interest in the context of domain adaptation, where the lack of annotations was recovered through hierarchical Hoffman et al. (2012) and kernel-based cluster- ing (Gong et al. (013), via exemplar SVMs (Xu et al.] or by measuring mutual information (Xiong et al.) 2014). More recent work corrects batch statistics of domain adaptation layers ing Gaussian mixtures (Mancini et al.. 2018), or studies the shift from some source domain to a target distribution that contains multiple latent domains (Peng et al.) 2019b] Matsuura & Harada 2020 Latent domain learning however differs fundamentally from these works: Table 1 contains a comparison existing transfer learning settings.
A common bascline in multi-domain learning is to finetune D models, one for each individual domain (Rebuffi al. [018 Liu al. 2019a). This requires learning large number of parameters and shares no parameters across domains, but can serve as a strong baseline to compare against. We show that in many cases, even when domains were carefully annotated, dynamic latent domain approach can surpass the performance of such domain-supervised baselines (see Section 4)
2.2 OBSERVED VS. UNIFORM ACCURACY
Consider a problem in which the data is sampled i.idd. from IP =.Paa + ToP dds i.e. two hidden domains. When domain labels are not available in the data, a standard strategy to treat all samples equally, and measure the observed accuracy:
OAce[f] = (yr
where y/ denotes the class assigned to sample x; by the model f. and y; its corresponding label for training. The OAcc has problematic property: if P consists of two imbalanced domains such that
Ta 2 To. then the performance on d, dominates it. For example if dj has 90% overall share, and the model perfectly classifies this domain while obtaining 0% accuracy on d,, then OAce would still assume 0.9, hiding the underlying damage to domain db-
3

---

Published as a conference paper at ICLR 2022
This motivates alternative formulations for latent domain learning, to anticipate (and account for) imbalanced domains in data. If it is possible to identify some semantic domain labeling (as typ- ically included in multi- dontnddmmmmin adaptation benchmarks), can compare performances across individual subgroups. This allows picking up on domain-specific performance losses which traditional metrics (such as OAcc) fail capture.
Where this is possible, we therefore propose to also measure latent domain performance in terms of uniform accuracy which decouples accuracies from relative ground-truth domain sizes:
1 D
Ufce[f]
Σ
Returning to the above example, uniform measurement reflects the model's lack of performance on d, as 0.5. Once again note while ground-truth domain annotations are required in order to compute uniform accuracy, these are never used to train latent domain models.
3 METHODS
To enable robust learning in the new proposed setting, we formulate a novel module called sparse latent adaptation which can adaptively account for latent domains. Section 3.1 reviews adaptation strategies popular in the multi-domain context, which our method extends (and generalizes).
3.1 LATENT ADAPTATION
When domain labels are available (not the case in latent domain learning) one strategy established by Rebuffi al. (2017) is to modulate networks by constraining the layerwise transformation of residual networks He et al. 2016) ((x)= = x + fz) to allow at most a linear change Va per each domain from some pretrained mapping by (with fu in every layer), whereby ((x) Øo(x)) Vaz.
Note the slight abuse of notation here in letting x denote feature map with channels C. Rearranging this yields:
(<r = =
+ Σ Valx),,
with a domain-supereee switch that assigns corrections to domains, i.e. 9a for d associated with x and 0 otherwise. Each Va is parametrized through 1xl convolutions, and fo denotes a shared 3x3 convolution obtained c.g. ImageNet (Deng al. 2009). This builds the assumption that representations require minimal changes to adapt to new tasks (Bilen & Vedaldi 2017), making learning each Va sufficient, while fo remains as is. Such adaptation strategies have been successfully used in few shot learning (Li et al. (022) and NLP (Stickland & Murray 2019) restrict the number of learnable parameters there.
models with strong general-purper
In latent domain learning access to is removed, resulting in two new challenges: we have no a priori information about the right number of corrections (Va), and cannot to decide which one of these to apply.
To mitigate the lack of domain labels d, first we assume that input data is constituted by K latent distributions PR. Second we propose to replace the switch ga with a learnable gating mechanism 91 (z). .........eee IK (x) that assigns each sample x latent domains as follows:
= = E gh (=))(()).
The gates gae control which convolution is applied to which sample x, and correspond to a categorical variable over K categories, i.c. 0 < St < and Σ. 9s = 1. Note in particular how parametric dependency of & on d is removed. How to best choose K is discussed in more detail in Section
While we motivate our latent domain module from learning over multiple domains, the main goal is not to recover the domain labels annotated in some datasets. When optimizing some loss (standard cross-entropy in the classification case), there is no guarantee that the learned Vn, will correspond to an annotated visual domain and many additional factors (shape, pose, color, etc.) can enter them as
4

---

Published as a conference paper at ICLR 2022
well. Latent domain models simply optimized to produce the lowest training error, and in fact seldom recover ground-trutt -oound-truth domains (c.f. Fig. 5. Note the broader concept presented here may in principle also be incorporated with other multi-task concepts (Perez et al. 2018 Guo et al. 2019a), adaptation strategies however stand out due to their methodological simplicity.
Different options exist for parametrizing the gating function g: X mechanism for latent domain learning would fulfill two seemingly incompatible properties: be able to filter latent domains some layers (requiring a discrete gate), but also share parameters between related latent domains in other layers (smooth gates). The next section proposes how this can be resolved without requiring task relationships (Vandenhende et al. 2020) or outer optimization loops (Wu et al. 2018) through the use of sparseness.
C R An ideal gating
3.2 SPARSE LATENT ADAPTERS (SLA)
We parameterize the gating function g with a small linear transformation W: C - RR that consti- tutes the pre-activation q W ((r) within the gates, where yp: X + C denotes average pooling projection onto the channels.
A crucial choice is whether the activation for E RK should map to a discrete space G = 10, or a continuous G = [0, 1]x in which the V& are shared.
We propose different strategy that lets gates be smooth when appropriate, but threshold 7 allows for sparse (or discrete) outputs fr(g) = 9 T with |++ = max(0, (0.-). -). Crucially fr can be
solved in a differentiable manner (Martins & Astudillo 2016) by sorting 91 2 9K. solving
k* = max (k: 1 + kqp. > Ejst t } and computing T jSk 45 1]
Consider q = 10.1, 1.0, 0.5] for which sparse activation results in (q) 10.0, 0.75, 0.25] while softmax yields [0.202, 0.497, 0.301]. Sparse activation filters out qı, while sharing between q2 and 43- We may now define:
K 2x x + Σ [ W w p(x)] Ve(x),,
where [:] picks the k'th element of the gating sequence. To the best of knowledge sparse activation strategies were never previously employed for expert models in computer vision and have so far been restricted the NLP setting (Deng et al. 2017; Peters et al.) 2019). Note SLA generalizes residual adaption (Rebuffi et al. 2017 2018). which is recovered by setting K=
While gating is subject complex interactions such negative transfer (Rosenbaum al. 2019), our ablations in Table 5 clearly show that taking a sparse perspective -which allows the model to assume either continuous or discrete forms outperforms the alternative of a priori fixing ei- ther smoothness through self-attention (Lin et al., 2017h), or discrete Gumbel-based sampling (Jang) al. (016)). Note this choice between discrete (Veit & Belongie [018 Guo et al.) (20Tbb) and con- tinuous mechanisms (Shazeer et al. [2017] Sun et al. 2019a Wang al. 2019) delineates previous
work that employs differentiable gates.
A softmax-activane model can in principle also learn to suppress individual preactivation compo- nents by letting some (k go to -o. This however requires either learning extra calibration parame- ters at every layer, defining a hard cutoff value (Shazeer et al. 2017) (thereby removing differentia- bility), or very large row-norms within the linear mapping W a highly unlikely outcome given the several mechanisms found in state-of-the-art models (in particular weight decay, norm-penalties, or BN (loffe & Szegedy 2015)) which act as direct counterforces to this.
4 EXPERIMENTS
We evaluate our proposed methods on three latent domain benchmarks: Office-Home, PACS, and DomainNet (c.f. Fig. 6 which shows example images from these benchmarks). The main goal here is not to compare to existing multi-domain or domain adaptation methods that these datasets were initially designed for, but to study two central research questions: whether domain labels useful for effectively learning over multiple domains, and whether one can learn such representations without domain labels.
5

---

Published as a conference paper at ICLR 2022
Table 2: Per-domain performance on Office-Home. Multi-domain (MD) baselines use domain an- notations, and latent domain (LD) models do not. Best overall performance underlined; best latent domain performance in bold.
Type
Params
A
C
P
R
OAcc
UAcc
Proportion Trl
15.57
28.01
28.45
27.95
RA (Rebuffi et al. 2018
MD
8.8 mil
48.05
76.12
80.74
67.78
70.73
68.17
Domain-Adv. (Ganin et al. 2016
MD
7.8 mil
55.14
72.85
81.98
68.81
71.57
69.70
4XResNN122
MD
24.8 mil
52.47
79.95
85.02
70.01
74.34
71.86
ResNet22
LD
6.2 mil
50.10
76.80
78.83
63.36
69.47
67.27
ResNet56
LD
14.0 mil
52.26
78.47
80.80
66.34
71.66
69.47
RA (Rebuffi et al. 2018
LD
6.9
58.44
79.15
81.55
72.13
74.65
72.82
MLFN (Chang al. 2018
LD
7.6 mil
50.72
78.81
81.36
64.56
71.18
68.86
MMLD Matsuura & Harada 2020
7.8
59.63
67.89
81.16
74.35
72.19
70.76
SLA
LDD
7i6
60.72
78.05
83.73
77.08
76.70
74.89
We also examine recent fairness benchmark (see Appendix and show that SLA improves robust- ness under single domain long-tailed distributions (Appendix G All experiments were implemented in PyTorch (Paszke et al. 2017
Optimization In all experiments, we couple our method with a ResNet26 model pretrained on downsized version of ImageNet that was used previous work by |Rebuffi et al. (2018). In SLA only gates and corrections are learned, the residual backbone fo remains fixed at its initial parameters, which implicitly regularizes the model (Rebuffi et al. 2017). Training is carried out for 120 epochs using stochastic gradient descent (momentum parameter of 0.9), batch size of 128, weight decay of 10 and an initial learning rate of 0.1 (reduced by 1/10 at epochs 80, 100).
All experiments follow the preprocessing Rebufii et al. (0117) 2018), alongside standard augmen- tations such as normalization, random cropping, etc. Accuracies are averaged over five seeds.
Increasing the number of corrections K within SLA results in small, consistent performance gains. As K 2 already represents a solid boost from the baseline of having no adapters, we focus on this result in the main part, and report results for higher K alongside variances in Appendix C
The underlying data contains a variety of objects classes (alarm clock, backpack, etc.) among four domains: art, clipart, product, and real world (Venkateswara et al., (017). In Ta- ble 2 we show results for d-supervised multi-domain (MD) approaches: RA (Rebufhi et al.) 2018)) domain-adversarial learning (Ganin et al., 2016) and a baseline of 4x Res22226, one for each do- main. For latent domain (LD) baselines, we then learn a single ResNet26, this time as a latent domain model over all domains. Next, we couple SLA with the very same ResNet26.
Office-Home
Learning single ResNet26 over latent domains with no access to d-labels significantly harms per- formance. This problem not addressed by simply increasing the depth of the network: while accuracy improves slightly, a ResNet56 exhibits the same performance losses - in particular on the latent domains product (P) and real world (R).
While residual adaptation (RA) (Rebulfi et al. 2018 was shown to work extremely well in many multi-domain scenarios, performance here is sub-par, regardless of whether it accesses d (MD: one Va per-domain) or not (LD). This likely results from linear modules being reserved for each d when using annotations, enabling no native cross-domain sharing of parameters. When is hidden on the other hand, the model is forced to share single linear adaptation module V between all four hidden domains, without the flexible gating we propose in SLA.
Learning annotations through latent domain clustering and coupling this with gradient reversal as in MMLD (Matsuura & Haradal 2020) increases performance relative to its d- annotated counterpart (Ganin et al. (016). The increase modest however, likely because enforcing on the gradient level negatively impacts the model's 's ability to discriminate be- tween classes (Wang et al. 2020 Another related baseline is MLFN (Chang et al., 2018) which builds on ResNeXt (Xie et al. 2017) to define a latent-factor architecture that accounts for multi-
"oode available gi tith.b. com/ wtt./o/l/laaatentotmmaiiiiaaaaaa

---

Published as a conference paper at ICLR 2022
Table Results on PACS, Best performance underlined, best latent domain performance bold.
Type
Params
A
CC
P
S
OAcc
UAcc
Proportion Tt
0.205
0.235
0.167
0.393
RA (Rehuffi ct al.
2018
MD
8.8 mil
85.14
92.05
94.50
94,30
91.93
91.50
Doman-Adv. Gamn el al. 2016
MD
7.8 mil
86.47
93.25
94.17
91.30
91.25
91.30
4x Nec2
MD
24.8 mil
88.41
95.53
94.34
95.71
93.94
93.50
k K-means+RA
LD
8.8 mil
83.84
92.10
94.75
93.01
91.21
90.93
ResNet26
LD
6.2 mil
85.27
94.55
93.85
94.98
92.70
92.16
ResNet56
LD
14.0 mil
86.96
94.34
95.15
95.34
93.36
92.95
RA (Rebufli et al. 2018
LD
6.9 mil
89.86
93.90
95.56
93.91
93.35
93.31
MLFN Chang a 2018
L.D
7.6 mil
78.38
91.29
88.19
92.95
88.78
87.70
MMLD Malsuura & Harada 2020
LD
89,93
92.26
96.25
94.34
93.20
93.27
SLA
LD
7.6
89.27
95.85
95.26
95.49
94.26
93.97
Table 4: Results DomainNet, best performances in bold.
clipart
infograph
painting
quickdraw
real
sketch
OAcc
UAce
ITd
0.082
0.088
0.123
0.294
0.295
0.118
ResNet26
66.46
27.99
51.70
67.46
66.58
56.95
60.47
56.19
ResNet56
69.08
29.55
53.85
68.61
68.60
58.42
62.19
58.01
RA (Rebufli 2018
68.23
29.34
$6.29
66.51
71.81
57.75
62.65
58.32
MLFN (Chang el al. 2018
68.20
24.64
50.39
69.85
65.78
56.27
60.54
55.85
SLA
69.46
30.14
57.36
67.97
72.89
58.83
63.83
59.44
modality in data. Crucially where method is fine-grained and shares convolutions every layer, MLFN instead enables and disables entire network blocks, allowing us to outperform it.
SLA outperforms the currently available latent domain models by consistent margin, and increases UAce by 12.79% relative to ResNet26. Best performance is obtained when K = D, with perfor- mance being reducing slightly from overfitting of larger domains for K D (see Appendix C
The second experiment examines performance on the PACS dataset (Li et al. 2017). Cru- cially PACS domains (art, cartoon, photo, sketch) differ more markedly from another (c.f. examples Fig. (). hence constituting an interesting latent domain problem.
PACS
Even for more distinct domains in PACS, results in Table that SLA improves existing baselines. The largest gains occur on smaller domains (e.g. art), where standard models suppress underrepresented parts of the distribution (see additional discussion on imbalanced distributions in Appendix G). Our method again surpasses the accuracy of 4x 4xResNe126, while requiring frac- tion of the total parameters (~ 9.7 mil for K = 5 vs. 24.8 mil). The performance of SLA again continues to increase with larger K (see Appendix
The performance increase from using latent domain-ad adversaiill approach (Matsuura & Harada 2020) versus using side the rest of the network can be better strategy than trusting annotations. Our approach again improves over this, without requiring clustering stage in MMLD.
(Ganin et al. 2016) confirms that learning domains along-
Results for k-means (using D= centers and clustered on the feature level) and subsequent finetun- ing show that two-stage strategy is suboptimal. This is not surprising since, similar to d-supervision via gd in & of eq. (3), clustering learns fixed switches that get used across all layers. contrast to this in SLA we fiexibly share separate features individually every layer (c.f. qualitative results in Fig. 3. synergizing only where appropriate.
DomainNet We also evaluate models large-scale benchmark called DomainNet (Peng al., 2019a). This dataset contains 518 447 images from six domains (clipart, painting, photos, sketch, infographics, and quickdraw), with total of |>l 345 object classes. The optimization settings remain unchanged from those in previous sections.
Results shown in Table 4 MLFN performs best quickdraw. domain that differs visibly from others fc.f. Fig. 6 for examples from each domain), and having entire network blocks dedicated to it seems to benefit performance. On all remaining domains, SLA outperforms existing models,

---

Published as a conference paper at ICLR 2022
Office-Home
PACS
SESURSTT mon surran -
Figure 2: Cosine similarity between SLA gates for Office-Home (left) and PACS (right). PACS domains are more dissimilar, but similarities exist, e.g. (A)rt and (P)hoto.
Art RPC
0.4
Chipart R
P
A
0.2
-0.8
Product
A
C
R
-0.2
Roal World
C A
2
6
10
14
18
22
Network rk depth
Figure 3: Layerwise correlation between SLA convolutions V Office-Home. Most correlations occur in the mid-to-late stages of the model.
Sparsity
0-0-0 0
14
18
22
Network depth
Figure 4: SLA sparsity (Office-Home), dotted lines show residual pooling transitions.
regardless of whether they were designed specifically for multi-domain problems, such as RA, or whether they are much deeper/parameter-intenss (ResNet56).
Qualitative analysis We (i) compare global statistics of Office-Home and PACS domains as well as (ii) their per-layer treatment within SLA; (iii) analyze sparse gating, (iv) representations learned by SLA, and show that (v) our module shares between geometric properties (shape, pose, etc.).
Fig. 2 average cosine similarities of per-domain gating vectors g g across / -= 1..... L layers of NesNet26 show that Office-Home domains differ less than those in PACS.
Fig. 3 layerwise measurements of Corr(gi (z), gi (x')) for xr, x'' drawn from differing d + d' for Office-Home. If inter-domain correlation is high, then similar corrections Ví are responsible for processing samples from two domains. Across top layers of the network there is little correlation, presumably as low-level information associated with each domain is processed independently. In the mid to bottom stages correlation increases: these layers are typically associated with higher- order features (Yosinski et al.) 2014 Mahendran & Vedaldi 2016 Asano al.) 2020), and since label spaces are shared between latent domains, similar object-level features are required to classify objects into their respective categories.
8

---

Published as a conference paper at ICLR 2022
Art Painting
Cartoon
Photo
Sketch
Figure 5: Left: PCA of samples represented by their SLA activation paths, colored by their ground- truth domain label assigned in PACS. SLA shares parameters between visually similar domains art and photo (+.*). while isolating sketch (+). The arrow highlights one sample that has been labeled photo in PACS. SLA categorizes it cartoon instead, more adequate assignment for this par- ticular image. Right: sample pairs from different domains (d; + ddy) with matching SL.A activations. Note their similar geometric properties.
úii) Fig. sparse gates have the flexibility to either output singular activations (i.e. become fully discrete) or all non-zero values (a continuous gate). We measure the per-layer sparsity E - 1) where || llo counts values different from zero, finding sparsity of SLA vary across model depth. Interestingly after each downsampling operation SLA tends to be relatively sparse, followed by a dense gate, then again a sparse one, and forth. The model thus utilizes the extra flexibility resulting from sparse gates.
Due to PACS domains being relatively distinctive, the dataset is an interesting candidate for addi- tional analysis in (iv) and (v) of how sparse adaptation handles the different ground ground-truth domains.
iv) Fig. (left): gate vectors g E G for samples from all four domains in PACS visualized by their principal components. SLA exhibits an intuitive clustering of human- PACS domains: vi- sually similar art and photo (+,²) cluster together. The manifold describing sketches (+) is arguably more primitive than those of the other domains, and indeed only maps to small region. Carlvon (+) lies somewhere between sketches and real world images. This matches intuition: a cartoon is, more or less, just colored sketch.
Fig. 5 also highlights one sample that shows an elephant that SLA places among the cartoon (+) domain, but which has been assigned a ground-truth domain label of photo (+) in the PACS dataset. The ground-truth label seems to have been annotated in error, but different from approaches that use d-supervision, our SLA processes latent domains on-the-fly and is therefore not irritated by this.
v) Fig. 5 (right): pairs of samples with similar gates. This shows that latent domains are indicative of more than ground-truth -ruth domain labels and extend to geometric similarities: pose, color, etc. of the samples are visibly related. Compare in particular the poses of elephants/dogs (second/third row).
5 CONCLUSION
In this paper explored two questions: (i) whether domain associations required for learn- ing effective models over multiple visual domains and (ii) how multi-domain models may best be learned without depending manually curated domain labels.
As has been shown, the performance of existing models does degrade without domain labels, raising doubts about their suitability for realistic problems that involve diverse data sources. As remedy, we proposed a novel adaptation strategy which reclaims (and often exceeds) lost accuracy on latent domains, benefiting several problems where some notion of domain (but annotation) exists.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
HB is supported by the EPSRC programme grant Visual ATP77028552// TH supported by EPSRC grant EP/R026173/1.
REFERENCES
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis self-supervision, or what can learn from a single image. International Conference on Learning Representations, 2020.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in Neural Information Processing Systems, 2019.
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text,
planktons, and cat breeds. arXiv preprint 1rXiv:1701.02255 2017.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI Conference on Artifi- cial Intelligence, 2018.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski, A systematic study of the class imbalance problem convolutional neural networks. Neural Networks, 106:249-259, 2018.
Adrian Bulat, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic. Incremental multi-domain learning with network latent tensor factorization. AAAI Conference on Artificial Intelligence, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-disrrbutiinnawwar margin loss. In Advances in Neural Information Processing Systems, 2019.
Xiaobin Chang, Timothy M Hospedales, and Tao Xiang. Multi-level factorisation net for person re-identification. In JEEE Conference Computer Vision and Paltern Recognition, 2018
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint
arXiv:1711.02233 022557, 2017.
Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In International Conference on Learning Representations, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. Image-to-markup generation with coarsc-to-fine attention. In International Conference on Machine Learning. 2017.
Benjamin Fish, Jeremy Kun, and Ádám D Lelkes. A confidence-based approach for balancing fairness and accuracy. SIAM International Conference on Data Mining, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1), 2016.
Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adaptation. In Advances in Neural Information Processing Systems, 2013.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 2020.
Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing, Depthwise convolution is all you need for learning multiple visual domains. In AAAI Conference Artificial Intelligence, 2019a.
10

---

Published as a conference paper at ICLR 2022
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. transfer learning through adaptive fine-tuning. In IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
SpotTune:
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for mul- tisource domain adaptation. In European Conference on Computer Vision, 2012.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. CyCADA: Cycle-consisten adversarial domain adaptation. In International Conference on Machine Learning, 2018.
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058, 2020.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. MIT Press, 1991.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint ArXiv: 144, 2016.
Michael I Jordan and Robert A Jacobs. Hicrarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181-214, 1994.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing- Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2018
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwiiska. al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3511-552 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human- concept learning through probabilistic program induction. Science, 350(6266), 2015.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In International Conference Computer Vision, 2017.
Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy Hospedales. Episodic training for domain generalization. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2019a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for sho- classification. arXiv preprint arXivv22103 13841, 2021.

---

Published as a conference paper at ICLR 2022
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales. Feature-critt networks for het- erogeneous domain generalization. In International Conference on Machine Learning. 2019b.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2017a.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In International Conference Learning Representations, 2017b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2019a.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision, 2015.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-22) 255, 2016.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul- tiple tasks by learning to mask weights. In European Conference on Computer Vision, 2018.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Boost- ing domain adaptation by discovering latent domains. IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Massimiliano Mancini, Elisa Ricci, Barbara Caputo, and Samuel Rota Bulo. Boosting binary masks for multi-domain learning through affine transformations. Machine Vision and Applications, (6):1-14, 2020.
Andre Martins and Ramon Astudillo. From softmax to sparsemax: sparse model of attention and multi-label classification. In International Conference on Machine Learning, 2016.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In AAAI Conference on Artificial Intelligence, 2020.
Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for vi- sual tracking. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-135) 2009.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In International Conference Computer Vision, 2019a.
Xingchao Peng. Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with dis- entangled representations. arXiv preprint 1904. 2019b.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. AAAI Conference on Artificial Intelligence, 2018.
Ben Peters, Vlad Niculae and André FT Martins. Sparse sequence-to-sequeeee models. In Annual Meeting of the Association for Computational Linguistics, 2019.
12

---

Published as a conference paper at ICLR 2022
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint wrXiv:/80.003266 2018.
Vikram V Ramaswamy, Sunnis SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de- bieabising. arXiv preprint arXiv:2012.01469, 2020.
S-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametrization of multi-domain deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning. 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representations, 2019.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation, arXiv preprint arXiv: vrXiv:/904. /t////444/227 /2774, 2019.
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transac- lions on Pattern Analysis and Machine Intelligence, 2018
Alice Schoenauer-Sebag, Louise Heinrich, Marc Schoenauer, Michele Sebag, Lani F Wu, and Steve Altschuler. Multi-domain adversarial learning. In International Conference on Learn- ing Representations, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference Learning Representations, 2017.
Asa Cooper Stickland and lain Murray. BERT and PALs:
Projected attention layers for efficient adaptation in multi-task learning. International Conference on Machine Learning, 2019.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 6479-6488, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In International Conference on Computer Vision, pp. 843-852, 2017.
Ximeng Sun, Rameswar Panda, and Rogerio Feris. AdaShare: Learning what share for efficient deep multi-task learning. arXiv preprint arXiv: 1xXi::911.11225. 2019a.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: 1825, 2019b.
Youssef Tamaazousti, Hervé Le Borgne, Céline Hudelot, Mohamed El Amine Seddik, and Mohamed Tamaazousti. I Learning more universal representations for transfer-learning. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2019.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communica- tions of the ACM, 59(2):64-75 73, 2016.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. deeper look at dataset bias. In Domain Adaptation in Computer Vision Applications, pp. 37-55. Springer, 2017.
Antonio Torrala and Alexei A Efros. Unbiased look at dataset bias. In IEEE Conference Computer Vision and Pattern Recognition, 2011.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEFE Conference Computer Vision and Pattern Recognition, 2017.
13

---

Published as a conference paper at ICLR 2022
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. British Machine Vision Conference, 2020.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Euro- pean Conference on Computer Vision, 2018.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object de- tection by domain attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry Davis, Kristen Grauman, and Rogerio Feris. BrockDrop: Dynamic inference paths in residual networks. In IEEE Confer- ence on Computer Vision and Pattern Recognition, 2018
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In IEEE Conference Computer Vision and Pattern Recog- nition, 2017.
Caiming Xiong, Scott McCloskey, Shao-Hang Hsieh, and Jason J Corso. Latent domains modeling for visual domain adaptation. In AAAI Conference on Artificial Intelligence, 2014.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Zheng Xu, Wen Li, Li Niu, and Dong Xu. Exploiting low-rank structure from latent domains for domain generalization. In European Conference on Computer Vision, 2014.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, 2013.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Empirical Methods in Natural Language Processing, 2017.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. Interna- tional Conference on Learning Representations, 2017.
APPENDIX
A DATASETS
The selected Fig. 6 shows examples from the latent domain benchmarks evaluated in Section 4 images have equivalent classes Ya = ya' y (for example chair for Office-Home), but different domains (e.g. d (art, clipart, product, real world)).
These examples show that data from different domains often contain very different visual charac- teristies (compare e.g. photo sketch for PACS), even when the object is the same. At the same time, other domains are more alike (e.g. art and photo), indicating that different amounts of sharing between per-domain parameters required, which in SLA is facilitated by its gating mechanism.
14

---

Published as a conference paper at ICLR 2022
Clipart
Real World
Office-Home
Sketch
PACS
Clipart
Infograph
Painking
Quickdraw
Real
Sketch
DomainNet
Figure 6: Datasets used for latent domain benchmarks: Office-Home, PACS, and DomainNet.
B RELATED WORK
Multi-domain learning relates most closely to work. The state-of-the-art methods introduce small convolutional corrections residual networks to account for individual domains (Rebuffi) et al. 2017: 2018), which was recently extended to obtain efficient multi-task models for related language tasks Stickland Murray (2019). Other work makes use of task-specific attention mech- anisms (Liu et al.]| (2019),, attempts scale task-specific losses (Kendall al. 2018), or addresses tasks at the level of gradients (Chen et al. 2017). Crucially, these approaches all rely firmly on domain labels.
Our work is loosely related to learning universal representations (Bilen & Vedaldi 2017) which was used guiding principle in designing more transferable models (Tamaazousti et al. 2019).
However, these works also assume the presence of domain labels. Multimodal learning does not make this assumption, and was shown benefit from accounting for latent semantic factors match images (Chang et al., 2018), or from normalizing data in separate groups (Deecke et al., 2019). As show our experiments (see Section latent domain learning however benefits from more customized solutions than these.
The proposed module gives rise differentiable dynamic network architecture, studied e.g. for reinforcement learning (Zoph & Le 2017 [ham et al., 2018 8). Bayesian optimization (Kandasamy jet al. 2018), or when adapting to new tasks (Mallya et al., 2018 Rosenfeld Tsotsos (008)). For such architectures, two components are commonly used: discrete Gumbel- sampling (Jang
cet al. 2016), c.g. leveraged in dynamic computer vision architectures (Veit Belongie 2018 Sun al. 2019a), continuous self-attentive approaches (Lin al.) 2017b). which have been used
successfully to scale expert models (Jacobs et al. 1991 Jordan Jacobs 1994) to large problem
spaces (Shazeer et al. 2017 Wang et (019).
From the perspective of algorithmic fairness, desirable model property is to ensure consistent predictive equality across different identifiable subgroups in data (Zemel al. 2013 Hardt al. 2016 Fish et al.] 2016). This relates to one of the goals in latent domain learning: to limit implicit model bias towards large domains, and improve robustness small domains. Recent work explores connections between models and empirical fairness for visual recognition (Bagdasaryan et al, 2019 Hooker et al. 2020 Wang et al., 2020), different from our experiments however (see Appendix they focus their analysis on setting in which annotations for protected attributes are available.
C VARIATION OF RESULTS
Fig. displays variances of accuracies recorded over ten random initializations Office-Home (left) and PACS (right). We generally found SLA to be robust different optimization settings, and result observed variances relatively low across experiments.
15

---

Published as a conference paper at ICLR 2022
UAco
0.96
UAoe
0.95
0.78
0.75
0.94
0.74
0.78
2
2
Figure 7: Variation of accuracies on Office-Home (left) and PACS (right).
Larger K brings an improvement of around 0.5-1 in performance at the expense of linear increase in learnable parameters (c.f. next section). While accuracy is improved by setting K 2, gains appear to saturate in line with previous observations around network width (Xie et al. 2017
D MEMORY REQUIREMENTS
In SLA every layer contains O(K[C) + K(C?) parameters to parametrize gates and corrections Vi. respectively. This is however an extremely modest requirement, in particular because fo stays fixed: while a ResNer26 contains ~ mil learnable parameters, even when setting K within SLA has just 3.5 mil free parameters, and is a fraction of the number of parameters needed to parametrize four ResNet26 (around 24.8 mil parameters).
Note also that the complexity of solving sparse gates in SLA scales O(K log K), negligible increase given the small K required in our method.
E ABLATION
Table 5: An ablation study for SLA. UAcc is shown (on Office-Home) with activations other than SparseMax [Martins & Astudillo 2016) used in this paper. For all variants we fix K 2.
Gating Mechanism
UAcc
SLA
74.89
Smooth (Lin al. 2017b
74.62
Discrete (Jag et al. 2016)
74.31
Replacing sparse gating within SI.A registers drop in performance, regardless whether smooth or discrete mechanisms are used. Accuracies for soft and straight-through Gumbel-softmax sam- pling (Jang et al. 2016) were on par; we report straight-through sampling here.
We also ran experiments where we did not fix the residual backbone fo but updated its parameters alongside the learning of SLA. In line with what Rebuffi et al. (2017) report, this lead to overfitting and performance dropped JAcc=73.55.
FAIRNESS
Recent work elevated the role small subgroups in data and examined model fairness on CelebA (Bagdasaryan al. 2019 Wang al. 2020: Hooker et al. 2020). Because such subgroups may be interpreted as constituting an individual latent domain component Pa, they are an interesting candidate to evaluate purpose-buil SLA on.
The benchmark contains different labeled attributes (e.g. "brown hair". "glasses"), and is modi- fied from the original dataset by hiding gender labels. Models are evaluated on all 39 remaining
16

---

Published as a conference paper at ICLR 2022
Table 6: Average precision and bias amplification of SLA the CelebA fair attribute recognition benchmark (Wang et 2020).
ResNet18
SLA
ResNet34
+ SLA
ResNet50
+ SLA
MAP (t)
71.76
73.22 (+1.46)
71.33
73.98 (+2.65)
74.52
75.03 (+0,51)
BA (L)
0.025
0.014
0.022
0.009
0.012
0.008
8
Change in AP [%]
6
4
2
0
0.5
0.6
0.7
0.8
0.9
1.0
Skew
Figure 8: Change AP between ResNet18 and ResNet18-Se S-LAA for different gender skews CelebA attributes.
attributes, which subsequently experience varying amounts of gender skew. Framed as latent do- main problem we have d= (female, male), but models have no access to this information.
The images used are the entire Aligned&Croppers subset (Liu et al. 2015) over which we finetune residual models, replacing only the fully-connected layer of the network. We use the optimization settings introduced in Section 4 for 70 epochs with reductions at epochs 30, 40, and 50, selecting the
best model on the validation split. This experimental setup is identical to previous work on empirical fairness (Wang et al., 2020 RRaaasamyy et al. 2020), which however different from work
focused on learning models that have access to the gender-attribute
We evaluate per-attribute accuracy using mean average precision (mAP) and report bias amplifica- tion (BA) (Zhao et al. (017). This compares the propensity of a model to make positive predictions (i.e. I exceeds threshold 4+ [0, 1J) in the gender dÿ that appears most frequent within attribute y, compared to the true counted ratio of positive examples y+:
1 [00124+107 Ey.P,
V=
BA[/] = Ez-r_
(6)
y=
where t. is optimized for on the validation split. For example 60% male examples are wearing glasses but under the model this is raised to total of 65%, then bias is amplified by =.00.
We report performance for ResNet18, ResNet34, and ResNet50 in Table 6 and compare this to the same model with SLA inserted. SLA consistently raises both mAP and reduces bias, indicating that relies less on spurious correlations data to formulate its predictions.
In Fig. 18 we compare per-attribute skew toward either female male (whichever is more frequent) to the gain in performance from ResNet18 to the same model but with SLA inserted. We observe a clear trend here, whereby SLA is able raise performance the most in those attributes that experience the largest amounts of skew.
G LONG-TAILED RECOGNITION
Standard models often experience difficulty when some classes are heavily underrepresented. This problem has recently been studied in long-tailed recognition (Liu et al. 2019h Cao et al. (019) with
17

---

Published as a conference paper at ICLR 2022
Table 7: Top-1 validation accuracy imbalanced CIFAR benchmarks (Buda et al. 2018). SLA consistently improves performance for standard ERM as well as existing long-tail approaches.
Imbalanced CIFAR-10
P
ERM
ERM-SLA
Focal
Focal-SLA
LDAM
LDAM-SLA
10
86.09
93.05 (+6,96) 96)
86.61
92.14 (+5.53)
91.08
92,49 (+1.41)
100
68.02
81.60 (+1..88)
67.44
78.82 (+11.38)
75.67
80.96 (+5.29)
Imbalanced
CIFAR-100
P
ERM
ERM-SLA
Focal
Focal-SLA
LDAM
LDAM-SLA
10
65.16
70.76 (+5.60)
64.56
70.60 (+6.04)
66.08
69.61
(+3.53)
100
45.44
48.46 (+3.02)
45.19
48.39 (+3.20)
51.25
55.06 (+3.81)
benchmarks that modify CIFAR-10 and CIFAR-100 to an imbalanced version by dropping some classes (e.g. 6-10 for CIFAR-10) (Buda al. 2018). The severity the imbalance described via the ratio Temax/ Mmin between the largest and smallest classes.
Long-tailed distributions may be viewed as containing an underrepresented latent component with 1/(1 + p). and previous results (c.f. Section 4) that fortified small latent domains within P motivate us to evaluate the imbalance setting more closely here.
Since our strategy is architecture-based it can be combined with the most recent state-of-the-art (loss-based) techniques long-tailed recognition: label-distribation-aaar margin loss with de- ferred reweighting (Cao al. 2019), reducing contributions from well-classified examples in focal losses (Lin et al. 2017a As Table shows, adaptation via sparse gates acts as a regularizer on the underlying ResNet26, and consistently improves performance long-tail benchmarks.
18


Reviews:
[{"review_id": "xRMPbqL85S", "reviewer": "Reviewer_Yiib", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\n1. A new and interesting problem setting with practical values:\nThe new task introduced in this paper, i.e., latent domain learning, looks like a tweak of existing problems such as multi-domain learning at first glance, but is interesting and has great potential as it could be practically useful when learning and testing models with data from heterogeneous domains (e.g., web images crawled by predefined search keywords). I also agree that the definition of visual domains is often vague and contrived, thus believe the motivations of latent domain learning make sense. \n\n2. Strong performance:\nThe proposed method outperforms multi-domain learning methods in the latent domain learning setting (although such a result is to be expected, the gap seems not large enough, and some scores of previous work look abnormal).\n\n\n[Weaknesses]\n\n1. Weak quality of writing:\nThe manuscript is overall readable but sometimes hard to follow, probably due to its weird terminology and expressions. Also,\n- U^tilde in Table 1 is not defined.\n- It is unclear why Eq. (2) is called “uniform” accuracy.\n- The description for Eq. (3) is overly complicated. \n- The operation denoted by stars is not defined appropriately, one may guess it stands for correlation though.\n\n2. Limited novelty:\nThe proposed model LSA looks like an extension of the residual adapter (Rebuffi et al., 2018), dubbed RA in the paper. Both RA and LSA adopt and modify the well-known residual connection by adding domain-specific feature transforms. The main difference between them is that to choose appropriate feature transforms RA utilizes domain labels explicitly while LSA estimates the latent domain label of input through gating variables. The use of gating variables could be counted as a contribution, but they are not a new idea as they have been widely used in other fields of machine learning.\nAlso, I would note that some objections against RA in the paper look invalid and accordingly the contribution of this paper should be undervalued. As far as I understand the residual adaptation method (Rebuffi et al., 2018) aims at designing models that maximize parameter sharing across different domains; only a small number of parameters of their residual adaptation modules are learned in a domain-specific manner while the majority of their parameters are shared across domains. \n\n3. Experiments:\nCompared to RA, the improvement by the proposed method does not look sufficiently large. Also, the records of RA are weird: I wonder why its score is degraded when using domain labels, and how it is trained without domain labels in the latent domain learning setting.\n", "summary_of_the_paper": "This paper introduces a new task called latent domain learning and proposes a baseline that extends an existing multi-domain learning method. The latent domain learning assumes that training data are sampled from different domains yet their domain labels are latent, and aims at learning models that generalize well to the domains. The proposed baseline deploys multiple parallel feature transform layers that are chosen on the fly through gating variables, with the hope that the gating variables learn to predict the latent domain label of input and choose feature transforms of the domain accordingly. This model demonstrates superior performance to existing multi-domain learning methods in the latent domain learning setting. \n\nI recognize that latent domain learning is an interesting problem with great potential, and has many applications such as learning using web-crawled images. However, the proposed method looks limited in terms of novelty and the quality of writing is below the standard.\n", "summary_of_the_review": "The new problem setting, latent domain learning, looks interesting and practically useful. However, the baseline model is limited in terms of novelty, its performance is not impressive (decent though), and the paper misleads readers about the important previous work (i.e., RA). \nThe paper has both pros and cons, but I would value novelty and correctness more, thus leaning towards rejection.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "uV16b80CkvH", "reviewer": "Reviewer_cM2p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed latent domain learning makes sense and technically sound to me. It also has great potential in real world applications.\n2. Extensive experiments and sufficient analysis validated the approach empirically.\n3. Writing is good and easy to follow.\n\nWeakness:\n1. It seems that the optimal value of hyperparameter K is different for different datasets. Is there any thorough methodology to pick a good value instead of using K from 2 to 5 or just 2 as experimented in the current draft?\n2. For the results, it was averaged over 5 random initialization, what is the variance for each experiment? It is not sufficient enough to use the mean itself for comparison as there might be a very large variance that indicates the model is not robust.\n", "summary_of_the_paper": "In this paper, the authors proposed latent domain learning for adaptation. Experiments on multiple benchmark datasets show improved result. Several visualizations also illustrate the effectiveness of the proposed approach.", "summary_of_the_review": "Based on the above analysis, I am leaning towards acceptance for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iwBwvX9B-Jb", "reviewer": "Reviewer_dZE1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, easy to follow and the motivations and contributions are clear.  Multiple datasets are used for evaluation against multiple SOTA approaches.\n\nSLA is referenced before defining it in 3.1\n\nI am confused on how g_k is learned in Eq 4.  How does it learn domain related gating without domain labels?  Figure 5 shows that in PACS some domains seem to be learned (eg sketch) while others may not have been learned.  Could this suggest that sketch images required additional parameterization to increase performance due to task difficulty? It is mentioned that the goal is not to recover domain labels, but simply adding additional parameterization may be similarly beneficial in single domain data.  Have the authors tested whether the performance improvement extends to single domain settings?  The results are interesting and I feel the paper could benefit from a deeper analysis on what is being learned by the \"domain\" gates.  If they are not domain specific is this really multi-domain learning?. Also, how are the number of gates (K) chosen?\n\nI would prefer a bit more examination on the results and what is being learned as well as comparisons to only learning the gating parameters and training the whole model together with the gates.", "summary_of_the_paper": "The authors present the latent domain learning aproach to multi-domain learning without domain annotations by introducing sparse latent adaptation (SLA) for learning sparse domain gates without domain labels.", "summary_of_the_review": "The paper is well motivated, clear and well evaluated with strong results.  It could benefit from a deeper analysis on what is being learned, how that contributes to different data settings and if novel aspects of datasets can be learned from this approach (eg does the model identify previously missed domains).  Overall a good paper that could be better with more detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jqw9xJLc6Z2", "reviewer": "Reviewer_LPa6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. The paper is well-written, easy to follow and understand.\n2. The paper claims to propose a new setting for learning from multiple domains, where domain labels are unknown. This is an important problem since labeling domains can be challenging and/or expensive in real-world applications. While I am not extremely familiar with the field, according to the paper, prior works assume the knowledge of domain associated with a data point.\n3. The idea of using sparse gating mechanism to allow for multiple transformations makes sense since it is possible that transformations learned across various domains are useful for predicting image category.\n4. Qualitative and quantitative experiments seem to corroborate that authors' proposed method is learning a sensible domain representation.\n\n**Weaknesses**\n\n1. The experimental setup used in the paper, as well as the baselines are not properly justified. I specifically don't understand\n    - Why is the ResNet backbone kept frozen, and not fine-tuned along with rest of the parameters?\n    - Why do the authors use same parameters across different methods/models? A fair way to compare across methods would be to perform cross-validation across range of hyper-parameters and report the performance on the test set.\n2. It is a bit surprising to me that all the proposed multi-domain approaches match perform worse when compared to latent-domain? I'd imagine that a model having a domain information for each datapoint would set an upper bound on the performance when compared to methods with no domain information. Is the presence of domain information hurting for the model performance? \n3. Although the paper states that the reported numbers are averaged over five random initializations, the standard errors are not reported\n4. I would like to hear authors' take on connection between self-supervised learning approaches, and given latent domain learning. The SSL methods learn image representations using self-similarity. In an extreme case where each image is its own domain, SSL might be suited to perform representation learning on such multi-domain data.\n\n\n**Minor points**\n\n- In Table 1, $\\tilde{U}_{D+1}$ is undefined. I presume it is validation or test data from $D+1$ domain.\n- In Section 3.1, using a linear shift in feature map of ResNets for mapping domains (to a \"canonical domain\"?) is not properly motivated. I assume this strategy has been used in prior works in language but it would be useful to motivate it in the context of domain adaptation problem.\n- In Section 3.2, some of the symbols (eg. $\\tau$, $q$) are introduced without proper definition. While I was able to understand them by going back and forth between current paper and referred paper, it would be helpful if the authors clarify the notations.\n- Section 2.2 is a relatively minor detail given that the datasets used in the paper are not highly imbalanced and both observed accuracy and uniform accuracy are highly correlated to each other. Authors can consider moving this detail to the experiments section.\n\n", "summary_of_the_paper": "The paper addresses the problem of learning a classification model on data from multiple domains, when explicit domain assignment for each data point is not provided. \n\nTo solve this problem, the paper proposes the data to be coming from 1 of K unknown (or latent) domains. A single 1x1 convolutional layer per latent domain is applied to feature maps in each residual layer of ResNet. A gating function (based on sparsemax) is proposed to decide which of the K convolutional layers will be applied. The entire network is learned jointly by minimizing the classification objective function.", "summary_of_the_review": "Overall my opinion is that the paper explores an interesting problem of learning latent domains present in data, when no explicit domain information is provided. I feel the experiments performed in the paper can be improved a bit. I've scored the paper accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The problem setting studied in this paper is an extension of the problem setting of multi-domain learning, where domain information is missing in training. This is an interesting and practical problem setting. However, regarding technical novelty, the contributions are relatively limited. Specifically, first, the overall idea is an extension of an existing multi-domain learning method [Rebuffi et al. 2017] by replacing the domain indicators with learnable gates. Second, the idea of introducing learnable gates is borrowed from some previous works. Third, the sparse activation is also based on an existing work of sparsemax. Though the authors claim sparsemax or sparse activation was only used in the NLP domain, this does not increase the technical novelty by applying sparsemax to the CV domain. 

To be fair, the combination of the aforementioned techniques to solve the so-called latent domain learning problem looks reasonable, while the technical contribution is not significant. Overall, I feel slightly positive about this work and recommend a weak acceptance (it could be considered for publication only if there is space).
Decision: Accept (Poster)