Reducing Implicit Bias in Latent Domain Learning
Authors: Lucas Deecke
Timothy Hospedales
Hakan Bilen
Venue:  (2021)
Keywords: Latent Domain Learning
CNN Architectures

Abstract: A fundamental shortcoming of deep neural networks is their specialization to a single task and domain. While recent techniques in multi-domain learning enable the learning of more domain-agnostic features, their success relies firmly on the presence of domain labels, typically requiring manual annotation and careful curation of datasets. Here we focus on latent domain learning, a highly realistic, yet less explored scenario: learning from data from different domains, without access to domain annotations. This is a particularly challenging problem, since standard models exhibit an implicit bias toward learning only the large domains in data, while disregarding smaller ones. To address this issue, we propose dynamic residual adapters that adaptively account for latent domains, and weighted domain transfer – a novel augmentation strategy designed specifically for this setting. Our techniques are evaluated on image classification tasks containing multiple unannotated domains, and we demonstrate they enhance performance, in particular, on the smallest of these.

Under review as a conference paper at ICLR 2021
REDUCING IMPLICIT BIAS IN LATENT DOMAIN LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A fundamental shortcoming of deep neural networks is their specialization to single task and domain. While recent techniques multi-domain learning enable the learning of more domain-agnostic features, their success relies firmly on the presence domain labels, typically requiring manual annotation and careful cu- ration of datasets. Here focus on latent domain learning. highly realistic, yet less explored scenario: learning from data from different domains, without access domain annotations. This is particularly challenging problem, since standard models exhibit an implicit bias toward learning only the large domains in data, while disregarding smaller ones. To address this issue, propose dy- namic residual adapters that adaptively account for latent domains, and weighted domain transfer novel augmentation strategy designed specifically for this setting. Our techniques are evaluated on image classification tasks containing multiple unannotated domains, and we demonstrate they enhance performance, particular, on the smallest of these.
INTRODUCTION
While the performance of deep learning has surpassed that of humans in range of tasks (He et al., 2016 [Silver et al.] 2017). machine learning models perform best when the learning objective narrowly defined. Practical realities however often require the learning of joint models over seman- tically different examples. In this case, best performances are usually obtained by fitting a collection of models, with each model solving individual subproblem. This is somewhat disappointing see- ing how humans and other biological systems are capable of flexibly adapting to a large number of scenarios (Kaiser al. 2017).
Past solutions that address this problem tend to fall into some category of multi-domain learning (Nam & Han 2016 Bulat al. 2019 Schoenauer-Seer -Seuuueer--eggg-ae et al., 2019). In this setting. models learned over diverse datasets each associated with an underlying distribution. Multi-domain learning however relies firmly the availability of domain annotations, for example to control domain- specific architectural elements (Rebuffi et al. 2017 2018 Liu et al.) 2019 Guo et al. 2019).
Reliance on domain annotations is limited to the multi-domain scenario, their presence is also required in domain adaptation where models transfer between related tasks (Ganin et al.) 2016
Tzeng et al. 2017, [ootmmn al. 2018; Xu al. 2018 Peng al., 2019a; Sun et al. (019b))
continual learning (Kirkpatrick et al. 2017 Lopez- Paz & Ranzato 2017 Riemer et al. 2019), meta learning over multiple tasks (Finn al.
| 2017
20188)) or the generalization previously
unseen domains (Li et al. 201 8b 2019b Guirajani & Lopez- Paz 2020).
The above approaches have established the notion that the presence of domain labels improves gen- eralization. In the real world however, these can often be difficult expensive to obtain. Consider images that were scraped from the web. Most image datasets such as Pascal VOC (Li et al. 20) or ImageNet (Deng et al. 2209) already rely on expensive manual filtering and the removal of differ- ent looking images. Existing multi-domain approaches require that the scraped images are further annotated for the mixture of content types they will contain, such real world images or studio photos (Saenko et al., 2010), clipart or sketches (Li et al. 2017). This can be an expensive process, moreover is not clear which variations (indoor/outdoor, urban/rural, etc.) should be grouped.

---

Under review as a conference paper at ICLR 2021
Relative Performance [%)]
10
ReeNee28
Ohrs
5
-10
-15
-20
5.2%
15.6%%
8.0%
2.8%
12.2%
10.0%
40.6%
1.5%
3.0%
Arcral
Diwc100
Dtd
Clarb
Onniget
Udf101
Vog Nogg-Fooowe
Figure 1: Changes in performance without access to domain labels, relative the multi-domain
baseline of learning an individual baseline on each dataset (sizes in %).. For ResNer26 (*), perfor- mance losses the smallest domains (yellow) significant. Our proposed solutions (+) recover large portion of the gap to full domain-supervison. Best viewed in color.
Here we consider the task of learning in absence of domain labels, or latent domain learning. This scenario encompasses any task where we have inadequate resources to assign domain labels to all data, but have reason to believe that such partitioning of the data would in principle make sense. And as our experiments show, even when domain labels already exist, there is no guarantee that these are optimal for given model. In this paper, therefore argue and demonstrate that learning implicit domain assignments, end-to-end alongside the rest the network, the superior option.
In Figure M we display accuracy for latent domain learning by jointly learning single model over datasets from the Visual Decathlon benchmark that contains images from distinct domains with mu- tually exclusive classes. Per-domain performance is measured relative to that of 9x models learned individually on each domain a common baseline in the multi-domain setting (Rebuffi et al. 2018 Liu et al. 2019). This highlights the central challenge of latent domain learning: significant loss of performance for the joint model (+) on small domains. While the performance drop is not significant on large domains, relative accuracy reduced by 15-20% on Aircraft and Ucf101, for example. Latent domain learning therefore requires customized solutions, because standard models have an implicit bias through which they disregard the smallest domains in data.
The mechanisms we propose throughout this paper (shown in *) help overcome this: dynamic resid- ual adapters (Section 3.3). which we couple with weighted domain transfer (Section 3.4) obtain robust performance on small domains, without trading performance on larger ones. Our proposed solutions can incorporated and trained seamlessly with existing architectures, and are able sur- pass the performance of domain-supervised approaches that have access to human-annotated domain labels (Section (..).. Moreover, qualitative analysis demonstrates that they partition latent domains in highly intuitive way (Figures and
RELATED WORK
Multi-domain learning relates most closely to our paper. The state-of-the-art introduces small con- volutional corrections in residual networks to account for individual domains (Rebuffi et al., 2017] 2018). Sticlland & Murray (2019) extend this approach to obtain efficient multi-task models for re- lated language tasks. Other recent work makes use task- ssecifpcc attention mechanisms (Liu al.) 2019). attempts to scale task-specifi losses (Kendall et al. 2018), or addresses tasks at the level of gradients (Chen et al. 2017). Crucially, these approaches rely firmly on domain labels.
A lack of domain labels has previously attracted interest in domain adaptation, Hoffman et al. (2012) use hierarchical clustering to uncover latent domains, other work investigates the use of kernel based clustering (Gong et al. 2013). via exemplar SVMs (Xu et al. 2014),, or mutual information (Xiong et al. 2014). Different from these works, we propose tackling latent domains in an end-to-end fashion, which no longer requires clustering ansatz. In another line of work Mancini et al.) (2018) estimate batch statistics of domain adaptation layers with Gaussian mixture models using only few
domain labels. [Png et al. (019h) study the shift from some source domain to target distribution
2

---

Under review as a conference paper at ICLR 2021
that contains multiple latent domains. In our setting, there is no significant shift between source and target distributions, instead the focus lies on learning parameter efficient models that generalize well across multiple latent domains.
Furthermore, our work is loosely related to learning universal representations (Bilen & Vedaldi 2017), which Tamaazousti al. 2019) as guiding principle in designing more transferable models. However, these works also assume the presence domain labels. Multi-modal learning does not make this assumption: Deecke et al. (2018) normalize data in separate batches to account for differences in feature distributions, while Chang et al. (2018) propose an architecture that ac- counts for latent semantic factors to match images. As we show in our experiments, latent domain learning benefits from more customized solutions than these.
Our module gives rise a differentiable dynamic network architecture, studied e.g. in the context of reinforcement learning (Zoph & Le] 2016 Pham et al. 2018). Bayesian optimization (Kandasamy et at. 2018), when adapting to new tasks (Mallya et al. Rosenfeld Tsstsss] 2018). or in uni- versal object detection (Wang et al. 2019). For such architectures, two components are commonly used: Gumbel-softmax sampling (Jang et al. 2016), e.g. leveraged in dynamic computer vision ar- chitectures (Veit & Belongie 2018: Sun et al. 2019a), or mixtures of experts (Jacobs et al., 1991.
Jordan & Jacobs
1994), used to scale models to large problem spaces (Shazeer et al. 2017).
From the perspective of algorithmic fairness, desirable property for models is to ensure consistent predictive equality across different identifiable subgroups in the data (Zemel et al.
[ 2013 Hardt ct al.
2016 Fish et al.; 2016; Corbett- 2016/Corbett- Davies et al.) (017). This relates to the central goal in latent domain learning: to limit the implicit model bias toward large domains, and improve robustness on small domains. [Wang al. (2020) explore this connection for visual recognition problems, different from
our work they focus their analysis on a setting in which domain annotations are available.
METHOD
3.1 PROBLEM SETTING
In multi-domain learning Nam & Han 2016, Rebuth et al. 2017 2018 Bulat et al. 2019 Schoenauer-Sete Sebag al. 2019). it assumed that data is sampled i.i.d. from some mixture of do- main distributions Pa with domain labels = 1, I...., D. Together, they constitute the underlying distribution TP = Ea nalPa, where each domain is associated with relative share Rd Na/N, with N' the total number of samples, and N₂ those belonging to the d'th domain. In multi-domain learning, the domain label d is always available.
......,
While the two are closely related, in the broader multi-task scenario (He et al. 2017 Kokkinos 2017 Vandenhende al. 2020) the nature of underlying tasks t I,..., is inherently differ- ent, and learning on each task distribution P, is associated with an individual loss function L, (for multi-domain example, one task may be object classification, the other semantic segmentation). learning on the other hand, all the L, are associated with an equivalent problem type, but can vary substantially depending on domain membership This is also the case for latent domain learning, with the important distinction that learning occurs over non-annotated domains.
In latent domain learning the information that associates each sample with a domain is no longer available. As such, we cannot infer domain-specific labels ya from sample-domain pairs (æt d) and are instead forced to learn a single model Sa over the entire P. We study two potential cases for latent domain learning. The first setting (Section 4.1 includes mutually exclusive classes (as in Visual Decathlon) and a disjoint label space Vi U J..... VD that encompasses all domains. This is challenging setting. models do not have priori information about each domain's 's label space Dd. and may therefore assign samples to false ones. For the second one (Section 4.2). we focus on setting in which label spaces are shared (for example, elephants can appear as photo painting) and Vd+ = ya for all domains.
A standard baseline in multi-domain learning is to finetune D individual models, one for each do- main (Rebuffi et al. 2018: Liu et al, 2019). Doing so requires learning a large number of parameters
In addition, in multi-task learning samples are often associated with a set of task labels yi..... U7. In multi-domain learning, each sample has distinct label ya with mutually exclusive Y' for each domain d.
3

---

Under review as a conference paper at ICLR 2021
and shares no parameters across domains, but serve upper guide performance. We show that in some cases, even when domain annotations were assigned carefully, a latent domain approach can surpass the performance of such strong
baselines, Section 4.2
3.2 RESIDUAL ADAPTATION
Residual adaptation builds on the observation that features from large pretraining tasks are univer- sally applicable (Bilen & Vedaldi 2017). and require only small modifications to correct for domain- specific differences. |Rebuthi et al. (2017) use this insight to extend the layer-wise transformation of the widely adopted residual architecture (He al. 2016):
+ fa())) ha.d(
where So denotes the main convolutions of the residual network (with parameters 0), and ha,d are small domain-specific convolutional corrections, i.e. dim(a) < dim While several concepts can in principle be incorporated with latent domains (Perez et al. 2018 Park et al. 2019), residual adaptation stands out due to its methodological simplicity and is a well-established method in the multi-domain literature.
In this work access to is removed, resulting in two new challenges: we have no a priori information about the right number of corrections {ha.d). and we cannot use d to decide which one of these to apply. Throughout the next section, we present an alternative strategy of inspecting æ and choosing relevant corrections ha the fly.
3.3 DYNAMIC RESIDUAL ADAPTERS
While there is access to domain labels d in latent domain learning, we still assume IP is constituted by several domain distributions Pa. To account for these in unsuperiedd fashion, we propose the use of dynamic residual adapters (DRA): each incoming sample x: is first processed by K corrections ) k==.....K which we parametrize with light-weight 1x1 convolutions. Next, gating mechanism Jb. responsible for weighing each correction hag under which æ is then transformed. In the l'th layer of the network, the subsequent feature representation computes as
K
DRA Φx) _ x So (x) +
Σ
que (a)) (x))
with Itx (x) the k'th component of the I'th gating function. For an illustration, see Figure While we motivate DRA from learning on latent domains, there no guarantee that each gate will corre- spond to single domain and many additional factors (shape, pose, color, etc.) may enter the gate assignments as well.
We parametrize the gating units using self-attention mechanism (Lin et al. 2017). The only learn- able parameters are a small linear transformation W: C -> TRKK s.t. g(z) = Softmax (WT ((x)) e). Here ((x) is a projection onto the channel dimension C. that averages out height and width (aver- age pooling) and ~ (0, Me) a channel-wise noise that encourages gate exploration (Shazeer categorical distribution over K categories, and Ee S How to choose is discussed in more detail in Section 4.1
let al. 2017 The gating mechanism corresponds
Le.
0
< 9k
While many other parametrizations of the gating function 9ù possible, self-attention enables signments and thus allows for the weighted combination of different hay- Though discrete assign- ments also be enforced through Suumbel-Softmax sampling (Jang et al.] 2016), found this approach to be too restrictive and underperforming. We compare different gate parametrizations in Table of the Appendix.
3.4 WEIGHTED DOMAIN TRANSFER
The central challenge in latent domain learning is the tendency of models to overaccount for large domains in IP. Besides accounting for this via dynamic residual adapters, we introduce an augmen- tation technique that encourages information exchange between latent domains.
Exploration noise is fixed E.g 10 Yi, zero otherwise.
4

---

Under review as a conference paper at ICLR 2021
We motivated by the following example: assume two classes (say, cats and dogs) each with two latent domains (sketches and photos). Ideally, we would want to encourage the model to learn a domain--gmostic representation of from which it may infer y, invariant of its latent domain d. We achieve this here by augmenting æ with the style information of a second sample x', drawn at random from P (so potentially, but not necessarily crossing domains).
A central challenge in latent domain learning is that we need to limit the transfer of information between highly similar samples, as this may restrict the model's access to discriminative features. For example, within the same domain we can often associate particular visual information with individual classes, e.g. aircraft tend to appear in front of blue sky, while dogs appear in natural scenes. Conceptually speaking. if we transfer between these two samples too aggressively, we may change the background scene of the dog to sky, whereby the model will learn it can no longer correlate sky and aireraft. To stop this from happening, our methodology limits the transfer between samples that are too similar, but transfers when the visual information them appears different.
In weighted domain transfer (WDT) randomly pair each sample o; with a second sample æ; from the mini batch. After average pooling samples onto channel dimensions s.t. (Mi, 142) E C, we compute their pairwise similarity in terms of the Bhattacharyya distance 6 (4--Hy)Tor (Hu
14)) + 0.5 Indett a /dct a Here
(a; + 3/2 denotes the average of the channel-wise variances associated with each sample. Note distances here scalar values, i.e. ô E I...
Before exchanging information, distances have to constrained to suitable range. To achieve this, we normalize them with their associated mean /48 = E..e/v 6./n (where denotes the batch size) and standard deviation Jo. squash them with a sigmoid S. and finally scale them with an exchange strength 17, i.c. 5n. n)S((), uss//as) This yields single exchange strength d [0, n] for each sample in the mini batch, which WDT uses to transfer information from æ. to 2;;
T ( pb)
n.w az,,
+ (1 )..
a;
In practice, this combines the statistics of æ; and æy with dynamic strength 5np depending on their pairwise similarity, Crucially for latent domain learning we do not want to transfer between samples that are very similar, so if their pairwise distance is small, then dng 2 0 + I.,eew RJ z; Figure 2 (right) shows that 6.; can vary substantially, and does so depending on the domain associated with
In line with style exchange approaches (Gatys et al., 2016;; Huang & Belongie 2017). we find that applying WDT in early layers of the network is the most effective (see Table 7). Further abla- tions in Tables 5o that WDT is stable under various choices of and significantly outperforms a competitive augmentation strategy MixUp (Zhang et al.) 2017), which does not consider the simi- larity between samples and less suited for latent domains.
EXPERIMENTS
We consider two experimental settings to evaluate our proposed approaches: Visual Decathlon for traditional multi-domain setting but without access to domain labels (Section 4.1) and dataset that contains equivalent classes for multiple latent domains (Section 4.2). While not the focus of our
paper, DRA also increases performance for single datasets and sigmiiccntly improves robustness to class imbalance, see Tables and in the Appendix. All experiments were implemented in Py PoToch (Paszke al. 2017), and code will be made available alongside the camera ready version.
4.1 LATENT VISUAL DECATHLON
The first experiment combines nine datasets from the Visual Decathlon challenge (Rebuffi et al.
2017) that contain a variety of different images, with mutually exclusive labels, i.c. [ Es [V Note the goal here not to compare the performance of existing multi-domain approaches that
'hiile closely related, this metric has some improved stability properties relative to the Mahalanobis dis- tance, which can he tricky optimize in practice.
'For Visual Decathlon, this gives rise to a 1128-dimensional label space. This implies that in latent domain learning models may erroncously classify samples from one domain (say, SVHN to another (e.g. Omniglot). In the multi-domain setting domain labels and their association individual label spaces Y'a prevent this.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Performances of 9 ResNet26 learned individually on each domain versus that of latent domain models: ResNet26, ResNet56 and our DRA+WDT with (K, 7) = (2, 0.2). For domain- supervision (9x ResNet26), label space sizes vary between 2 (Daimler Pedestrian) and 1623 (Om- niglot). In latent domain learning, there is single label space over all domains 3 2128. Overall best performance underlined, best latent domain models bold. Yellow: smallest domains.
Latent d
Airc.
C-100
Doim.
Dtd
Gtsrb
Omn.
Svhn
Ucf101
Vag-F.
W-Acc.
0,052
0.156
0.09
0.028
0.122
0.1
0.406
0.016
0.03
9xResNet26
X
39.48
77.96
99.95
38.19
87.62
95.12
73.00
87.01
ResNe126
70.71
99.49
33.67
99.87
87.80
94.64
58.25
60.39
84.73
ResNet56
34.62
71.63
99,52
34.79
99,90
87.72
95.12
60.66
57.55
85.22
DRA+WDT (K = 2)
40.83
77.55
99.22
40.37
99.80
86.58
94.14
65.47
72.16
86.58
DRA+WDT iK =3)
39.36
78.27
99.10
41.97
86.35
94.21
63.83
71.96
86.62
DRA+WDT (K =4)
36.81
77.97
98,91
40.48
99.77
87.17
94.21
65.47
70.69
86.46
DRA+WDT (K' 55
40.38
78.64
99.03
42.29
99.77
86.60
93.94
63.93
72.94
86.68
Visual Decathlon was designed for, but to show that deep networks struggle with learning small latent domains when domain annotations are provided.
For latent domain learning, domain labels are never used in training, and only serve to posteriori decompose the weighted accuracy (denoted as W-Acc) reported in tables into domain-conditional metrics.
Optimization Initial ResNet parameters were obtained from ImageNet (Deng et al. 2009 For dynamic residual adapters, only gates and corrections are learned, the ResNet26 backbone remains fixed at its initial parameters. As Rebuffi et al. (2018) showed, this implicitly regularizes the net- work, while also benefiting performance (c.f. our ablation in Table[5). The exact same optimization routine is applied across all experiments: we trained for 120 epochs using stochastic gradient de- scent (momentum parameter of 0.9), batch size of 128, weight decay of 10-4, and initial learning rate of 0.1 (reduced by 1/10 at epochs 80, 100). We use the official splits for Visual Decathlon, and report average accuracies over five random initializations.
All experiments use standard augmentation techniques: random cropping and flipping, as well as normalization. For WDT, we experimented with different exchange amounts A range of values improve over having no augmentation, best results were obtained for n 0.2, see Table 6 When increasing the number of residual adapters this also increases performance, however K 2 already represents a good trade-off between learnable parameters and increase in modeling power.
Results In T we show results for domain supervised baseline of 9xResNet26, one individ- ual model for each domain d. We then learn a single ResNet26, this time as a latent domain model over all nine domains. Next, we couple DRA+WDT with the very same ResNet26. For further comparison, we also include significantly deeper ResNet56.
Learning single ResNet26 over latent domains with no access to labels significantly harms perfor- mance, with accuracy dropping significantly on the smaller domains. This problem is not addressed by simply increasing the depth of the network: while weighted accuracy improves slightly, ResNet56 exhibits the same implicit bias against small domains, leaking performance there. DRA+WDT close a large portion of the gap to domain- supervisu and on four domains (Aircraft, Cifar-100 Dtd, Vgg- Flowers) even surpasses the performance of the domain-superrised baseline.
Qualitative analysis To understand better how DRA processes samples from different latent do- mains, is helpful to inspect which gates activate across the entirety of the network. For a given sam- ple, we follow its activation path across the T--layered network as (ar) & (gez ...)..... I (x:)). When two samples have similar paths, this means they share similar residual corrections hov. For simplicity, we set K = 2 in our analysis, s.t. V (x) = - (z(x).
We collected 26-dimensional gate activation paths for samples from all domains in Visual Decathlon, and visualize their principal components in Figure (left). This reveals a highly intuitive clustering of domains: Svhn (+) and Gtsrb (+) both show real-world, yet strictly defined content (numbers and street signs). CIFAR- 100 (+) is known to exhibit a large amount variation, and correspondingly maps relatively large region compare this to Omniglot (+) which is visually much simpler

---

Under review as a conference paper at ICLR 2021
Aitroft
Did
Syin
Cilc- 100
Gler
Udf101
Daimler Pad.
Orriiggtt
Vop-Frowar
WDT Exchange Strength
Figure 2: Left: PCA over gate activation paths. DRA partitions the data into separate latent domains in semantically meaningful way: Omniglot (+) occupies distinct region and also differs visually from the other datasets. Svhn (+) and Gtsrb (+) cluster together- both datasets contain real-world, but well-defined symbols (house numbers, street signs). Right: the effective distance 8, that governs the amount of exchange in WDT for each latent domain. Omniglot shares little information with other domains, Dtd (+) shares more actively.
dataset, and occupies more primitive manifold. Real-world datasets such Ucf101 (+), Aireraft (+) and Dtd (+) share another common region.
These observations align with the exchange strength 6, for WDT recorded in Figure (right). There little average exchange for Omniglot (+). Information exchange for Dtd (+) on the other hand is substantial, in correspondence with its gate activations that place it in the close proximity of other latent domains.
4.2 LATENT PACS
The second experiment examines performance dataset called PACS (Li et al., 2017). stand- ing for its four constituting domains (art painting, cartoon, photo, sketch). Each domain contains samples of equivalent classes ("elephant", "giraffe", etc.). The domains are unbalanced (see za in Table 2) We reserved 20% of samples for evaluation, leaving the remainder for training; we use random splits of the entire dataset, as the official splits are balanced for domains, whereas in the la- tent domain setting we assume no a priori knowledge of domain memberships. We make no changes the optimization described in Section 4.1
Results Table 2 shows that dynamic residual adapters improve considerably over the latent domain baseline of a joint ResNet26, with the largest gains on smaller domains (91.67% versus 85.27% on art painting). Notably, DRA surpasses the weighted accuracy of 4x domain supervie ResNet26, while requiring much smaller number of learnable parameters (3.5mil versus 24.8mil). Perfor- mance again increases with larger K DRA.
To the best of our knowledge, latent domain learning has not been targeted through customized deep learning architectures. related baseline MLFN (Chang et al. 2018), which builds on ResNeXt (iie et al., 2017) to define latent-factor architecture that accounts for multi-modality in data. Crucially, where we share small convolutional corrections at every layer, MLFN instead enables and disables entire network blocks, allowing to outperform MLFN.
While domain- -suprviiu residual adapters (Rebuffi et al. 2018 have been shown to work extremely well in the multi-domas scenario, their performance here is sub-par. This is likely because they exhibit no cross-domain sharing of parameters, which DRA does natively. As k-means (using D = 4 centers) with subsequent finetuning shows, obtaining fixed assignments across the depth of the network is suboptimal. In DRA we learn gates in an end-to-end fashion and flexibly share or separate features every layer, thereby obtaining domain assignments that synergize with the overall model.
Memory requirements Adding DRA to the residual network results in additional memory quirements, since every layer requires O(K CI + K|C(2) learnable parameters to parametrize gates and corrections, respectively. As outline in Table 2 this is extremely modest increase, in par-

---

Under review as a conference paper at ICLR 2021
Table 2: Results on the PACS dataset. Shown are performances for ResNet26, MLFN, a domain- supervised ensemble of 4x ResNet26, and our DRA. Third column lists the number of parameters that have be learned in each approach. Best overall performance underlined, best latent domain performance bold.
Latent d
Param. [+]]
Art Painting
Cartoon
Photo
Sketch
W-Acc.
Ti
0.205
0.235
0.167
0.393
RA (Rebuffi al., 2018)
X
2.6 mil
85.14
92.05
94.50
94.30
91.93
4x ResNet26
X
24.8 mil
88.41
95.53
94.34
95.71
93.94
4x ResNet56
55.9 mil
87.53
94.34
96.44
96.08
93.98
k-meuns+RA
V
5.2 mil
82.92
91.52
90.65
93.76
90.49
ResNet26
/
6.2 mil
85.27
94.55
93.85
94.98
92.70
ResNe156
/
14.0 mil
86.96
94.34
95.15
95.34
93.35
MLEN (Chang al., 2018)
/
7.6 mil
78.38
91.29
88.19
92.95
88.78
DRA+WDT K
1.4 mil
90.46
94.77
97.41
94.73
94.31
DRA+WDT K
2.1 mil
90.10
94.88
97.09
95.16
94.38
DRA+WDT K=
2.8 mil
91.67
95.42
96.60
95.10
94.72
DRA+WDT (K=
3.5 mil
90.10
95.86
97.41
95.40
94.76
Arthhaningg
Carton
Phob
WDT Exchenge Strength
Figure 3: Left: PCA of samples represented by their T-dimensional activation paths. DRA shares parameters between visually similar domains arf painting and photo (+,*), while isolating skeich (+). The arrow highlights one sample that has been labeled a photo in PACS. DRA categorizes it as cartoon instead, a more adequate assignment for this particular elephant. Middle: WDT exchange for different domains, sketch (+) is particularly inactive. Right: samples from different domains with similar gate activations R.
ticular only small subset of the parameters are learned (gates and residual corrections), while the convolutions of the residual backbone remain fixed.
Qualitative analysis As Figure 3 (left) shows, DRA exhibits an intuitive clustering of PACS do- mains: visually similar domains painting and photo (*,*) cluster together. The manifold describ- ing sketches (*) is arguably more primitive than those the other domains, and indeed only maps to small region. Cartoon (+) lies somewhere between sketches and real world imagery. This matches well with our intuition: a cartoon is, more or less, just a colored sketch. We highlight one particular elephant that is placed amongst the carloon domain, but has been assigned ground-truth domain label of photo in the PACS dataset. The ground-truth label was assigned in error, but different from approaches that use superviiu,erriiionn. dynamic residual adapters are irritated by this. Ex- change strengths in Figure 3 (middle) again confirms that WDT is more active for latent domains such art painting and photo (*.*) that share parameters with each other.
As we demonstrate through samples from PACS in Figure (right), similar activation paths V are indicafive of additional visual similarities, not just domain membership: pose, color edges of the shown samples are all closely related, compare in particular the poses of the elephants (second row).
Table 3: Accuracies for ResNet26, RA (Rebuffi et al. 2018 and our DRA.
ResNet26
RA Rebuffi et al.
2018
DRA
CIFAR-10
95.20 77.85
95.80 81.01
96.32
CIFAR-100
82.18
8

---

Under review as a conference paper at ICLR 2021
Table 4: Performance of ResNet26, ResNet56 and DRA on an unbalanced variant of CIFAR-10 where 75% of examples from classes 1-5 have been removed.
Sumple rutio
0.25
1.0
2
4
5
7
9
10
W-Ace.
Unif--Ac..
ResNer266
91.59
95.16
82.22
70.34
88.88 88.88
93.28
98.53
97.28
97.23
96.89
94.44
91.14
ResNer56
90.03
93.30
84.97
74.25
87.55
93.96
98.33
97.83
97.81
97.45
94.86
91.65
DRA
93.43
96.39
91.09
79.96
93.59
95.59
99.15
98.45
98.65
97.10
96.45
94.44
4.3 SINGLE DATASETS
Dynamic residual adapters require no domain annotations, and can therefore be used for learning on single datasets. Table 3 contains test accuracies on CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton 2009) for DRA (K = 2). We compare this to standard finetuning of the residual backbone as well residual adapters Rebufii et al. (2018). DRA can be inserted seamlessly into the network, and we make no changes to the optimization outlined in Section 4.1
On both datasets, dynamic residual adapters outperform traditional finetuning and residual adapters. We use no WDT here, which is specifically designed for the latent domain setting, but show that DRA can be used as general purpose module increase performance on benchmark datasets. Note the larger performance gap on CIFAR-100. In all likelihood DRA has special advantage there, as CIFAR- 100 known to have small modes, which can be identified as small latent domains.
4.4 IMBALANCED DATA
The benefits of DRA are not restricted the case of learning over multiple potentially unbalanced domains, but also extend to learning problems in which some classes are significantly underrepre- sented, i.c. the marginals p(y) = [x p(y x p(()) are heavily skewed. We construct the following experiment for CIFAR-10 10: samples that belong to one of the first five classes (airplane, automo- bile, bird, cat, deer) are excluded from the dataset u < 0.25 for u ~ ([, 1], so that around 75% of them removed. We learn models this dataset using the standard optimization routine outlined in Section 4.1 Results are shown in Table
Measured in uniform accuracy (which weighs performance equivalently for all classes), ResNet26 drops from 95.20% (c.f. Table 3 to 14%, because the model similar to what was observed for latent domains suppresses the underrepresented classes 1-5. Opposed to this, DRA performs as well as it does on standard CIFAR- 10 classjfication. In other words, in the same way DRA raises the robustness for small latent domains, also enhances models have better performance on object classes for which less examples exist in the dataset.
We also display performance for ResNet56 (which has around 40% more parameters than DRA). Its poor performance highlights that DRA addresses a fundamental issue that off- fffeth deep learning models exhibit when dealing with imbalanced distributions, and which cannot be resolved by simply learning a deeper network.
CONCLUSION
Recent work in multi-domain learning has been chiefly focused on a setting where domain anno- tations are assumed to be routinely available. As this requires careful curation of datasets, in real world scenarios this assumption often of limited merit. Dynamic residual adapters inject adaptiv- ity into networks, preventing them from overfitting the largest domains in distributions, failure mode of traditional models that is exposed in latent domain learning.
Not only does our approach close large amount of the performance gap to domain-supervised solutions, but in some scenarios even when domains have been assigned very carefully by human annotators exceeds their performance. In the future, hope to extend the mechanisms described here to the regime of extremely small domains, and examine additional research questions that arise there, e.g. whether they can help defend against adversarial attacks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv: 1rXXi:1701..7275: 2017.
Adrian Bulat, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic. Incremental learning with network latent tensor factorization. AAAI Conference on Artificial Intelligence, 2019.
Xiaobin Chang, Timothy M Hospedales, and Tao Xiang. Multi-level factorisation net for person re-identification. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint arXiv:171..0225.. 2017.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional net- works for text classification. arXiv preprint arXiv:1606.01781, 2016.
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic deci- sion making and the cost of fairness. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2017.
Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In International Conference on Learning Representations, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 2017.
Benjamin Fish, Jeremy Kun, and Ádám D) Lelkes. A confidence-based approach for balancing fairness and accuracy. In SIAM International Conference on Data Mining, 2016.
Yaroslav Ganin, Bvgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learming Research, 17(1), 2016.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adaptation. In Advances in Neural Information Processing Systems, 2013.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing. Depthwise convolution is all you need for learning multiple visual domains. In AAAI Conference on Artificial Intelligence, volume 33, 2019.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Judy Hoffman, Brian Kulis, Trevor Darrell and Kate Saenko. Discovering latent domains for mul- tisource domain adaptation. In European Conference Computer Vision, 2012.
10

---

Under review as a conference paper at ICLR 2021
Judy Hoffiman, Eric Tzeng, Taesung Park, Jun- Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. CyCADA: Cycle-consisten adversarial domain adaptation. In International Conference Machine Learning, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In International Conference on Computer Vision, 2017.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. MIT Press, 1991.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterizzatoon with gumbel-softtax. arXiv preprint arXiv: 161 140, 2016.
Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181-214, 1994.
Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In IEEE Conference on Computer Vision and Pattern Recog- nilion, 2018
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang. Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. Cell, 17(55:1122-1133 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinkke et al. Overcom- ing catastrophic forgetting neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-332 2017.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In International Conference on Computer Vision, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI Conference Artificial Intelligence, 2018a.
Li Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2019a
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In IEEE Conference Computer Vision and Pattern Recognition, 2018b.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales. Feature-critic networks for het- erogeneous domain generalization. In International Conference on Machine Learning, 2019b.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In Proc. ICLR, 2017.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.

---

Under review as a conference paper at ICLR 2021
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul- tiple tasks by learning to mask weights. In European Conferencee on Computer Vision, 2018.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Boost- ing domain adaptation by discovering latent domains. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for vi- sual tracking. In JEEE Conference on Computer Vision and Puttern Recognition, 2016.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaption normalization. In IEEE Conference on Computer Vision and Pattern Recogni- lion, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurlPS Autodiff Workshop, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. International Conference on Computer Vision, 2019a.
Xingchao Peng. Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with dis- entangled representations. arXiv preprint vrXivv/904.111 12347, 2019b.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with general conditioning layer. AAAI Conference on Artificial Intelligence, 2018.
Hicu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 002.0026...3... 2018.
S-A. Rebuffi, H. Bilen, and Vedaldi. Efficient parametrization of multi-domain deep neural networks. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representattinss, 2019.
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 2018.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference Computer Vision, 2010.
Alice Schoenauer-Setee Louise Heinrich, Mare Schoenauer, Michele Sebag, Lani Wu, and Steve J Altschuler. Multi-domain adversarial learning. In International Conference on Learn- ing Representations, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gatd garsell-gated mixture-of-experts. layer. In International Conference Learning Representations, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, (5507777))354--333 2017.
Asa Cooper Stickland and lain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. International Conference Machine Learning, 2019.
12

---

Under review as a conference paper at ICLR 2021
Ximeng Sun, Rameswar Panda, and Rogerio Feris. AdaShare: Learning what to share for efficient deep multi-task learning. arXiv preprint arXiv:1911.1242: 2019a.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv: 17825, 2019b.
Youssef Tamaazzostti, Hervé Le Borgne, Céline Hudelot, Mohamed EI Amine Seddik, and Mohamed Tamaazousti. Learning more universal representations for transfer-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5: 180161, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. arXiv preprint arXiv:2004. 13379, 2020.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Euro- pean Conference on Computer Vision, 2018.
Xudong Wang. Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object de- tection by domain attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In IEEE Conference Computer Vision and Pattern Recog- nition, 2017.
Caiming Xiong, Scott McCloskey, Shao-Hang Hsich, and Jason J Corso. Latent domains modeling for visual domain adaptation. In AAAI Conference on Artificial Intelligence, 2014.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. In IEEE Conference Computer Vision and Pattern Recognition, 2018
Zheng Xu, Wen Li, Li Niu, and Dong Xu. Exploiting low-rank structure from latent domains for domain generalization. In European Conference on Computer Vision, 2014.
Jiancheng Yang, Rui Shi, and Bingbing Ni. MedMNIST classification Decathlon: lightweight AutoML benchmark for medical image analysis. arXiv preprint arXiv:2010. 14925, 2020.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, 2013.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David I Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171000412 2017.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arX1v:6611.01578, 2016.
13

---

Under review as a conference paper at ICLR 2021
(b)
(c)
h1
Figure 4: Graphical models for (a) multi-domain learning, (b) latent domain learning. (c) A ResNet block, equipped with dynamic residual adapter (K" 2). Incoming samples pass down three streams: an identity function, transformation via a large convolution f, as well as an evaluation by expert gates g, which dynamically assigns (dashed arrows) small corrections hl and h2.
Algorithm Single iteration of DRA and WDT.
Input: Parameters (n, K), sample pairs {(0,, 3:)). Fixed model parameters 0, learnable ax, loss L: y x y + R2o, learning rate Rzo-
for each I; do
F₁ - WDT(2,,2y:1) draw zj uniformly at random
1/4 fajo K)
f, (Vii v/)
end for
- a- -Vaa-VVall E.4
Return: Latent domain model Saje: X y
A
APPENDIX
ILLUSTRATIONS
Figure 4 visualizes the difference between which domain labels d remain hidden from the model. In (c) we visualize the routing within our proposed dynamic residual adapters.
(a) multi-domain learning and (b) latent domain learning.
ALGORITHM
In Algorithm we outline pseudo-code for DRA and WDT. Note DRA be implemented with standard models in seamless way, and only modifies its internal structure. The resulting fae can be therefore trained in end-to-end fashion with any loss function through standard empirical risk minimization.
ABLATION
Our ablations show that latent domain learning benefits from both the addition of multiple dynamic residual adapters (i.e. K 1) as well as WDT augmentation between feature maps of samples. Each row in Table 5 coorrespodd to changing single parameter design element for DRA or WDT, and compared to the baseline configuration used in Table
Removing the dynamic element of DRA by fixing a single residual adapter (K 1) registers significant performance loss relative to K when not fixing parameters 0 of the ResNet convolutions, this leads to problems with overfitting. Replacing mixtures of experts with Gumbel- sampling negatively impacted performance, as smooth interpolations between residual corrections beneficial for sharing information between sampling were on par,
In line with what Rebuffi et al. (2017) report,
latent domains. Performances for soft and straight-through and report through sampling here.
14

---

Under review as a conference paper at ICLR 2021
Table 5: An ablation study on Visual Decathlon. First row contains DRA+WDT with default pa- rameters (K, 17) (2, 0.2). We remove each of our proposed solutions, and also replace it with an alternative parametrization: for DRA, this means that we remove the dynamic element and single (K 1) residual adapter in each layer. Next we set K = 2. but replace the gating mechanism with Gumbel-Softmax sampling. Besides the complete removal of WDT the fourth row, we also measure performance with MixUp as an alternative augmentation strategy. We also show accuracy when the residual network that we couple DRA+WDT with is not fixed at its pretrained parameters, regularization strategy initially proposed in Rebufii al. (2017) that we also employ here.
Fixed 0
WDT
MixUp
DRA
DRA-Gumbel
W-Acc.
ours
86.58
no gates (K=1)
85.65
(-1.07%)
Gumbel gates
/
X
85.76
(-0.95%)
no WDT
/
X
V
86.38
(-0.23%)
MixUp
X
/
83.47
(-3.59%)
unfix ResNet26
85.35
(-1.42%)
no DRA, no NDT
X
X
84.73
(-2.14%)
Cifar-130
Glarb
Ucf101
Daimier Ped.
Omnig lot
Vag Flowers
Figure 5: PCA over ResNet26. Only Omniglot is separated from other latent domains.
Having no augmentation from WDT (17 0) does not impact weighted accuracy much (-0.23%), but drastically reduces performance on small domains: accuracy drops from 40.83% 37.50% for Aircraft (-8. (6%,,, as well as for Dtd (-2.23%), Ucf101 (-3.91%), and Vgg-Flowers (-2.99%).
MixUp Zhang et 2017 an alternative augmentation that interpolates between samples, is not equally well suited for latent domain learning.
We include additional qualitative analysis in Figure by ResNet26 without domain annotations. The model clearly separates Omniglot due to its low- level differences from other domains, but otherwise does not account for latent domain structure.
This shows the embedding space learned
TUNING WDT
Results for additional choices 7) are shown in Table 6 most values for r) increase performance, when augmenting too little this however prevents sufficient amount of exchange, whereas too strong an augmentation impairs the relationship between latent domains and their associated labels.
Table 6: Weighted accuracies for WDT under different exchange strengths for Visual Decathlon
Rebuffi et al. 2017) and PACS (Li et al. 2017).
77
0.05
0.1
0.2
0.3
0.4
0.5
Visual Decathlon
86.33
86.43
86.58
86.47
86.57
86.41
PACS
93.67
93.32
94.31
94.06
94.01
93.71
15

---

Under review as a conference paper at ICLR 2021
Table Performance of WDT when inserted different depths of ResNet26.
Position Before block1
Before b blok2
Before block3
Before FC
W-Ace.
86.58
84.28
80.07
35.45
Table Sentiment classification accuracy for underrepresented latent domains.
d Virgin
American
United
Delta
Southwest
US Airw.
W-Ace.
π
0.034
0,188
0.261
0.152
0.165
0.200
73.05
82.01
77.58
72.44
75.13
84.84
78.53
Ratio
0.25
1.0
62.17
80.96
74.92
71.87
75.60
84.12
77.11
Moreover, WDT should be placed early in the network, as transferring information through augmen- tation at later stages of the model negatively affects performance, see Table 7
SENTIMENT ANALYSIS
We construct an experiment on language data to showcase that latent domain suppression is not restricted the realm of images. For this, we make use of a publicly available dataset that contains sentiment scores over Tweets addressed at publicly maintained airline accounts. The dataset is available at this jurl
In this scenario, we identify d with different carriers, e.g. Virgin, US Airways, etc. The experimental setup closely follows that Section 4.4; first, we train latent domain model on all available data and record its performance. Next, the first three hidden domains are subsampled (25% are kept, after which sentiment models are trained anew on the remaining data. We use a simple model here (ridge regression over tf ifdidf vectors) the focus is to show that latent domains suppressed in NLP problems- jjust as for images.
Accuracies are shown in Table The largest drop in performance is registered on the smallest domain (Virgin) for a relative change in accuracy of -1.9%% At the same time, performance is stable larger domains. As in previous experiments, comparatively small drop in weighted accuracy (-1.8%) hides the severity with which small domains are suppressed by the model.
Table 9: Topic classification on AG News.
Topic
Business
Science
Sports
World
W-Ace.
Unif.-Acc.
0.1
0.1
0.4
0.4
VDCNN
85.7
85.2
90.4
96.8
92.0
89.5
DRA
87.9
85.6
91.5
97.1
92.8
90.5
TOPIC CLASSIFICATION
For our topic classification trials, we make use of VDCNN language models that resemble residual networks (Conneau et al. 2016). Due to their residual property, these can easily be extended with DRA. The main difference here is that convolutions are one-dimensional in VDCNN (applied to text data) versus ResNet (for images), a difference that is straightforward to account for in DRA.
Experiments are carried out over AG News which is widely used scientific NLP publications To test model's robustness against underrepresented modes, we randomly drop 75% of examples from the "Business" and "Science" topic category. The results shown Table Aublegeee compare the 9-layer VDCNN to the exact same model extended with our DRA (K 2) at every layer. Models are optimized via SGD and 0.9-mmmetuum for 100 epochs, with learning rate halvings each 10 epochs.
SSee for instance paperswi thcode. com/ sota tataa//tettcccasee firifiiaaa averview of models applied AG News.
for an
16

---

Under review as a conference paper at ICLR 2021
Table 10: Accuracies for ResNet18 and DRA on three medical image benchmarks.
ResNet18
DRA
train
rest
train
Test
DermaMNIST
89.39
72.07
94.64
73.22
+1.60%
OCTMNIST
99.77
73.10
99.77
75.00
+2.60%
RetinaMNIST
71.76
49.50
84.82
53.00
+7.07%
MEDICAL IMAGE ANALYSIS
We evaluate DRA on images from the medical domain. As baseline we employ ResNet18, 8, and we follow the optimization settings (SGD, 0.9-momentum) outlined Yang et al. (2020).
We evaluate models on (i) DerMaMNIST, a 7-way classification problem that is constructed from 2018); (ii) OCTMNIST contains optical coherence tomography images showing 4 retinal diseases based on Kermany et al. (2018); (iii RetinaMNIST containing retina fundus images from DeDD (https: //ssbi. deepár
HAM10000 (Tschandl
html).
We make no modifications to the benchmark data and use it is, training both ResNet18 and the exact same model with DRA (K 2) added. The results shown in Table 10 highlight that DRA provides a robust increase in test accuracy, finding that is in line with previous results that indicated enhances performance over multi-modal distributions (c.f. Table 3).
17


Reviews:
[{"review_id": "qJ1d_DXvR5A", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------ Update after discussion with authors ---------\n\nI would like to thanks the author for their efforts by adding additional experiments, which surely enhances the significance of the proposed approach. Based on these, I increased my score to 5.\n\nI have re-checked the final revised version, I think the current version still *requires proper organizations and justifications*. For example, the added experiments still talked about the accuracy, the in-depth analysis seems lacking. I think a substantial revision of the paper in terms of structure, idea presentation, and analysis is still needed. Based on this, my final score is 5.\n\n-----------------------------------\nSummary: \n\nThis paper studied how to learn a neural network with multiple domains without knowing the exact domain label (by merging all the domains as a large domain). Then they proposed dynamic residual adapters and weighted domain transfer to address this issue. The empirical results showed its practical benefits.\n\n------------------------------------------------------\n\nOverall review \n\nPros:\n\n[1] This paper is well-motivated. I like the analyzed scenario and I think it can have a strong practical utility. \n\n[2] The high level of proposed ideas is technically sound. \n\nCons:\n\n[1] The submitted version seems to be a preliminary version with many missing and unclear elements.\n\n[2] As an **empirical** paper, the experimental results are not sufficient for ICLR.\n\n[3] Some technical details need better justifications and discussions.\n\nBased on these, I recommend a rejection at this time but encourage a major revision for resubmission.\n\n----------------------------------------------------\n\nDetailed explanations\n\n[1] Missing elements\n\n[a] I am rather confused and unclear about the whole learning procedure. It seems the author used DRA in the residual module. However, the role of WDT is unclear. What is the global training loss in the proposed approach? WDT is a part of the loss or used for analyzing the problem?  I would like to see a pseudocode/protocol for the whole algorithm or a clear network structure for illustrating the idea in Sec 3.3-3.4.\n\n[b] The mathematical notations defined in this paper are presented oddly (particularly in sec 3) for example:\n\n[b1] Equation (1), $\\alpha$ and $\\theta$ are scalars or vectors? what is meaning for $|\\alpha|\\ll |\\theta|$? I guess it is $\\text{dim}(\\alpha)$ but it makes me rather confused.\n\n[b2] The same problem for eq(2) and $\\epsilon$\n \n[b3] In WDT, the same problem for $\\delta$, $\\delta_i$ and $\\delta_j$\n\nThese confusions make it more difficult to understand the approach.\n\n[2] The empirical results\n\nThe current empirical results only compare MLFN, which is not sufficient. \n\nI noticed the author claimed, “Note the goal here is not to compare to the performance of existing multi-domain approaches that Visual Decathlon was designed for, but to show that deep networks struggle with learning small latent domains when no domain annotations are provided.”\n\nI agree with this opinion if the paper aims to only analyze this scenario (generally from a theoretical perspective).  These kinds of experiments are sufficient. \n\nBy contrast, the current version aims at **proposing a new empirical approach for the real-world practice**, which is not sufficient. I would like to see a **strong practical result** either outperforming the recent baselines or applying in many real-world problems.\n\n[3] Technical details\n\n[a] I suggest not using the term “domain labels” since it can be confusing to label $y$ information in the unsupervised domain adaptation. I think “domain index” or “task index” are better choices.\n\n[b] The visualization of $\\delta$ sounds interesting but I can not understand the meaning. A better explanation is expected. \n\n[c] Fig (1),(2) why PCA visualization? Why not Tsne?\n\n[d] The benefits of self-attention are unclear. More analysis (not numerical accuracy) is expected.\n\n--------------------------------------------\n\nSuggestions \n\nI suggest a major revision on the proposed approach, empirical results, and more analysis (not accuracy) on the benefits of the idea.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LN5mj9TH940", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes dynamic residual adapters designed to adaptively account for latent domains, and weighted domain transfer. This framework injects adaptivity into networks, preventing them from overfitting to the largest domains in distributions, a failure mode of traditional models that are exposed in latent domain learning. The approach closes a large amount of the performance gap to domain-supervised solutions. \n\nThis paper is well written. The motivation of this paper is clear and the proposed framework does break the limitation of the existing deep learning methods. Below I present some suggestions, which hopefully can help the authors improve their study:\n\nThe authors do not describe any processing they have done of the data. This should be clearly included in the methods section. More experimental details and insightful discussions should be provided. I suggest the authors repeat the benchmarking using a selection of datasets more similar to those used in current studies.\nThe benchmarking results are insufficiently described in the text and can only really be seen in the tables/figures. The authors should explain the results in more detail in the text and include their interpretation of the results.\nSection 3. In this part, the authors introduce the algorithm. Since there are many formulas in this section, additional explanations on the learning procedure would help understand the proposed method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kl_QO5GxJqm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a method for latent domain learning, where input data come from different domains and the domain labels are unknown. The proposed method consists of two parts: dynamic residual adapter and weighted domain transfer. The dynamic residual adapter acts as a mixture of expert layer. And the weighted domain transfer which augments the dataset by interpolating between different input pairs. Empirical results show that when combined together, the proposed method perform better than training a regular model.\n\nPros:\n1. Latent domain discovery is a very interesting topic. \n2. Empirical results show that the method brings improvement to minority domains.\n\nCons:\n1. Maybe I missed something, but I don't there are new insights in the paper. The proposed dynamic residual adapter is just an instance of MoE [1] with adapters, which I think should be a baseline in the experiments.\n2. \"Section 3.4 Weighted Domain Transfer\" is not well-motivated and very confusing. Here you want to interpolate between x_i and x_j. But why do you compute the difference between the input x_i and the feature \\mu_i in equation 3? Are they comparable with each other? What is the goal that you want to achieve here?\n\nOther comments:\n1. I think the introduction describes the problem too much, leaving it no space to expand your idea and intuition. For example, you start describing your idea at the very last paragraph.\n2. When creating the augmented examples, can you leverage the gate information that you produced from the MoE?\n3. It will be more helpful to understand the DRA component if you can provide PCA over the original activations.\n\n\n[1] OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER", "extra_scores": {}, "extra_text": {}}, {"review_id": "5j3JXy0LiYh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summarize what the paper claims to contribute. Be positive and generous.\nThe paper claims to contribute a new method *dynamic residual adapters (DRA)* coupled with *weighted domain transfer (WDT)* to tackle *latent domain learning.* The proposed method improves the model performance against small domain datasets without hurting the model performance against large domain datasets in two different settings:\n  (1) multi-domain setting (10 different tasks with 10 different domain per task)\n  (2) multi-style setting (1 task with 4 different styles)\nImpressive empirical results! I especially enjoyed reading PCA graphs and its qualitative analyses.\n\n2. List strong and weak points of the paper. Be as comprehensive as possible.\n  (1) Strengths\n    a. Exhaustive empirical analyses. (ablation tests and tests with varying K value)\n    b. Qualitative analyses backed by graphs like PCA and example images. \n    c. Impressive accuracy improvements.\n    d. Intuitive theoretical explanation. Cool idea to use a gate to make RA dynamic.\n  (2) Weaknesses\n    a. Skipped the math that yields the equation (3) in the section 3. Would be nice if the steps are attached as an Appendix.\n    b. In Table 1, there is a data domain size metric: $\\pi_{d}$ . Can you add how this is computed? Also, how small is the smallest data in sheer number of examples?\n    c. Table 1 has RestNet56, but Table 2 doesn't. Why did you make this choice of experiment design?\n    d. With PACS dataset, you have experimented with the model (k-means+RA). I didn't quite understand the model setup and the motivation. In my understanding, the model learned the latent domain labels via k-means. And, then, based on this pseudo domain labels, the model is fine-tuned with Residual Adapter applied. Did I understand the model correctly? What is the motivation of doing this? I am not sure if this is a fair comparison between the RA and the proposed DRA+WDT methods. It seems rather a comparison between DRA+WDT and K-Means.\n    e. In Figure 3, the paper says \"Middle: WDT exchange or different domains, sketch is particularly inactive.\" I had a bit of difficulty parsing what \"inactive\" means here because the \"Middle\" figure is about \"WDT Exchange Strength\" and because \"sketch\" has the largest strength. Can you explain what \"inactive\" here means?\n    f. Based on Table 3, the positive effect of WDT is not strong. I wonder if WDT is necessary.\n    g. It seems to me that the strength of the proposed method is much more evident in the second problem type (PACS) where the task is the same across different domains. In the first problem type (Visual Decathlon), the DRA+WDT's performance boost is not consistent across different domains. I see that DRA+WDT hurts the performance compared to the baseline ResNet26/56 on a few different domains, such as Daim., Gtsrb, Omn., and Svhn. $\\pi_{d}$ values of the domains of PACS are greater than those of Visual Decathlon, excluding svhn. Why do you think that is? When should one use or not use DRA+WDT in order to avoid hurting the model performance?\n    h. Minor formatting issues. See 6 below.\n\n3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.\nAccept because of the impressive performance of the proposed method against PACS (single task, multi domain) with clear visual analyses. Meantime, 2.(2).g requires more explanation to make the paper's claim stronger.\n\n4. Provide supporting arguments for your recommendation.\nSee 2.(1) Strengths, 3, and 2.(2).g above.\n\n5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \nSee 2.(2) Weaknesses above.\n\n6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n  (1) The indentation of Table 5 seems to be inconsistent with the rest of the tables in the paper.\n  (2) In Figure 3, the paper mentions $P_{k}$ without the denotation explained explicitly in any of the main body of the text. I had to re-read the paper to find a footnote 5 to finally understand what this denotation meant. It would be good to briefly explain this denotation in the same description of Figure 3.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While all reviewers agree that the topic is interesting and the work has merit, several issues have been pointed out, especially by R1 and R3, that indicate that the work is not  ready for acceptance at this stage. the authors are strongly encouraged to continue to work on this topic, taking into account the feedback received.
Decision: Reject