Multi-Domain Long-Tailed Learning by Augmenting Disentangled Representations
Authors: Huaxiu Yao
Xinyu Yang
Allan Zhou
Chelsea Finn
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-domain long-tailed learning
balanced representation augmentation
out-of-distribution robustness

Abstract: There is an inescapable long-tailed class-imbalance issue in many real-world classification problems. Existing long-tailed classification methods focus on the single-domain setting, where all examples are drawn from the same distribution. However, real-world scenarios often involve multiple domains with distinct imbalanced class distributions. We study this multi-domain long-tailed learning problem and aim to produce a model that generalizes well across all classes and domains. Towards that goal, we introduce TALLY, which produces invariant predictors by balanced augmenting hidden representations over domains and classes. Built upon a proposed selective balanced sampling strategy, TALLY achieves this by mixing the semantic representation of one example with the domain-associated nuisances of another, producing a new representation for use as data augmentation. To improve the disentanglement of semantic representations, TALLY further utilizes a domain-invariant class prototype that averages out domain-specific effects. We evaluate TALLY on four long-tailed variants of classical domain generalization benchmarks and two real-world imbalanced multi-domain datasets. The results indicate that TALLY consistently outperforms other state-of-the-art methods in both subpopulation shift and domain shift.

Under review as a conference paper at ICLR 2023
MULTI-DOMAIN LONG-TAILED LEARNING BY AUG- MENTING DISENTANGLED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
There is an inescapable long-tailed class-imbalance issue in many real-world classification problems. Existing long-tailed classification methods focus on the single-domain setting, where all examples are drawn from the same distribution. However, real-world scenarios often involve multiple domains with distinct imbal- anced class distributions. We study this multi-domain long-tailed learning problem and aim to produce a model that generalizes well across all classes and domains. Towards that goal, we introduce TALLY, which produces invariant predictors by balanced augmenting hidden representations over domains and classes. Built upon proposed selective balanced sampling strategy, TALLY achieves this by mixing nuisances
the semantic representation of one example with the domain-associnn of another, producing new representation for use as data augmentation. To im- prove the disentanglement of semantic representations, TALLY further utilizes a Nomain-invarient class prototype that averages out domain-speciffe effects. We evaluate TALLY on four long-tailed variants of classical domain generalization benchmarks and two real-world imbalanced multi-domain datasets. The results indicate that TALLY consistently outperforms other state-of-the-an methods in both subpopulation shift and domain shift.
INTRODUCTION
Deep classification models can struggle when the num- ber of examples per class varies dramatically (Beery et al. 2020 /Zhang et al. 2021). This long-tailed setting practice, such wildlife recogni- tion (Beery et al. (020). Classifiers tend to be biased towards majority classes and perform poorly on class- balanced test distributions, i.e. when there is shift in the label distribution between training and test. Ex- isting approaches address the long-tailed problem by modifying the data sampling strategy (Chawla et al. 2002 Zhang & Pfister 2021). adjusting the loss func- tion for different classses (Cao et al. Hong et al.) 2021), augmenting minority classes (Chou et al. 2020 [Zhon et al. (021). Unlike these works, which focus on single-domain long-tailed learning, we study multi-domain long-tailed learning, where each domain has its own long-tailed distribution. Take wildlife recognition as an example (Figure D images of wildlife are collected from various locations, and the distribution over species (classes) at each location typically imbalanced and the class distribution also varies between locations.
Location
Location
Location
Train
arises frequently
Location
Cres Melo
Shift
Figure 1: Illustration of imbalanced class distributions across domains in iWildCam, wildlife recognition benchmark (Beery
et al. 2020). Both subpopulation shift and domain shift settings are illustrated.
In multi- multi-domain tailed classification, the classifiers need to handle distribution shift amidst class imbalance. Here, we focus on two types of distribution shift: subpopulation shift and domain shift. In subpopulation shift, we train a model on data from multiple domains and evaluate the model on a test set with balanced domain-class pairs. In the wildlife recognition example, species are often concentrated at only a few locations, creating spurious correlation between the label (species) and the domain (location). A machine learning model trained on the entire population may fail on the test set when this correlation does not hold anymore. In domain shift, expect the trained model to generalize well to completely new test domains. For example, in wildlife recognition, we train a model on data from fixed set of training locations and then deploy the model to new test locations.

---

Under review as a conference paper at ICLR 2023
Prior long-tailed classification methods work well in single-domain settings, but may perform poorly when the test data is from underrepresented domains or novel domains. Meanwhile, invariant learning approaches alleviate cross-domain performance gaps by learning representations or predictors that are invariant across different domains (Arjovsky et al. 2019 Li et al. 2018). Yet, these approaches are mostly evaluated in llassbbalanced settings, where models must be trained plenty of examples from each class even if augmentation strategies are applied (Yao et al., (022) see a detailed discussion in Appendix B With multi-domain long-tailed data, learning class-unbiased domain-invariant model is not trivial since the imbalance can exist within domain or across domains. We aim to address these challenges in this work, leading to novel method named TALLY (mulTi-domAin Long-tailed learning with bal bancd representation reassemblY).
TALLY empowers augmentation to balance examples over domains and classes by decomposing and reassembling example pairs, combining the class-relevant semantic information of one example with the Nomain-associater nuisances of another [Zhou et al.) (2022). Specifically, TALLY first decouples the representation of each example into semantic information and nuisances with instance normalization. To further mitigate the effects of nuisances, we first average out domain information examples of the same class and construct class prototype representations. Each semantic representation is then linearly interpolated with a corresponding class prototype, leading to the prototype-enhancee semantic representation. The domain-assoocitee factors are similarly interpolated with class-agnostic domain factors to improve training stability and remove noise. Finally, TALLY produces augmented representations to benefit the training process by reassembling the prototype- enhanced semantic representation and Nomain-associatee nuisances among examples. To further achieve balanced augmentation, we propose a selective balanced sampling strategy draw example pairs for augmentation. Concretely, for each pair, the label of one example is uniformly sampled from all classes and the domain of another example is uniformly sampled from all domains. In this way, TALLY encourages the model to learn a class-unbiased invariant predictor.
In summary, our major contributions are: we investigate and formalize an important yet less explored problem multi-domain long-tailed learning, and propose an effective augmentation algorithm called TALLY to simultancously address the class-imbalance issue and Iearn domain-invariaant predictors. We empirically demonstrate the effectiveness of TALLY under subpopulation shift and domain shift. We observe that TALLY outperforms both prior single-domain long-tailed learning and domain- invariant learning approaches, with 5.18% error decrease over all datasets. Furthermore, TALLY is capable of capturing stronger invariant predictors compared with prior invariant learning approaches.
2 FORMULATIONS AND PRELIMINARIES
Long-Tailed Learning. In this paper, investigate the setting where one predicts the class label
y E C based on the input feature x E X, where C = {1...., C). Given a machine learning model f parameterized by parameter 0 and loss function f, empirical risk minimization (ERM) trains such
model by minimizing average loss over all training examples as
min F (x))
(1)
which works well when the label distribution is approximately uniform. In long-tailes learning, however, the label distribution is long- long-tailed, where small proportion of classes have massive labels and the rest of classes are associated with a few examples. Assume ((,,, y:)) is a training set sampled from training distribution and the number of examples for each class is (m1.-.., nc.), where >e nhe N. In long-tailed learning, all classes are sorted according to cardinality (i.c., ni < nc) and the imbalance ratio p is defined as p n1/nc > Note that same definitions are used in the test set ((.., yz) ) Under the blass-imbalanced training distribution, vanilla ERM model tends to perform poorly on minority classes, but we expect the model can perform consistently well on all classes. Hence we typically assume the test distribution is class-balanced (i.e., p = =))
long-tailed learning is a natural extension of classical long-tailed learning, where the overall data distribution is drawn from a set of domains
Domti- Imbalanced Learning.
D f1, .., D} and each domain d is associated with a class-imbalanced dataset {(x,, V., d) N drawn from domain-speen specifcc distribution pa. Following (Albuquerque et al. 2019 Koh et al. 2021). ). both training and test distribution
be formulated as mixture distribution over domain space D, Ep= v/a pdr The corresponding training and test domains are D (d D D' > 0} and D [d D 0), respectively, where n/ and n/ represent the
ie., p'r = >== nia par and p
2

---

Under review as a conference paper at ICLR 2023
Overall Pipeline
Representation Augmentation
Cr wi ye- Uni(C)
(gp, yy.d
d (inii))
Nuisances
Class-Prolo
Proto-Fabaneed
Selective
a(a))
Balanced Sampling
1-4
2
Pre-Layers
Pre-Layers
Samantic =(k+
A
Representation Disentanglement
Representation Reassembly
(5.3). 4j)
Representation Augmentation
Naisemoes ((,)
Aa 1-Aa
(G).
(b,51)
(A,
51
Semantic z(sy)
Pross-Fahancol Nuisances
Post-Layers
Domain-Prata Mag
Figure 2: An illustration of TALLY. Left: the overall approach produces augmented representations from a pair of examples x; and v.y. At a chosen layer, mixes their semantic and nuisance information to create augmented representation 3. Right: In detail, the augmentation step disentangles hidden representations 8; and 8 into separate semantic and nuisance factors. interpolates these with comain-invarian class-invariant prototypes (respectively) for more robust disentanglement. Finally, combines the semantic information from 8; with the nuisance information from to create 8,.
mixture probability. For each domain d, we define the number of training examples in each class as (r (mi.d...., sorted by cardinality. The imbalance ratio p²" is extended to domain-level ratio as t" nccaann..d During test time, we consider two kinds of test distributions, corresponding to two categories of distribution shifts subpopulation shift and domain shift. In subpopulation shift, the test domains have been observed during training time, but the test distribution is baass-balanced and domain- mamain-balanced.. i.c., D C D' and {n/² 1/|D'>|[vd e D In domain shift, the test domains are disjoint from the training domains, i.e., D n 0.
MULTI-DOMAIN LONG-TAILED LEARNING WITH BALANCED REPRESENTATION REASSEMBLY
To improve robustness in multi-domain long-tailed learning, we would like method that can learn class- unbiased Nomain-invariant representations. To accomplish this, we introduce TALLY to do balanced augmentation over classes and domains. The key idea motivating TALLY is that every example decomposed into class-relevant semantic information and domain-associaan assoccated nuisances that should be ignored by ideal classifier. Here, following (Zhou et al. 2022], nuisances is defined as "class- agnostic" transformations that apply similarly to all classes, such as image style and background changes in image classification. As outlined in Figure [ TALLY assumes that domain- associated nuisance information can be transferred among examples. It leverages this to perform augmentation by transferring domain- domain-specific nuisance factors between classes with novel selective balanced sampling strategy. In practice, TALLY first disentangles examples into latent semantic and nuisance factors. Then, we introduce a domain-agnostie prototype representation for each class in order to eliminate the nuisance information, whereby the semantic representation of each example is linearly interpolated with the corresponding prototype representation. Finally, TALLY reassembles the semantic and nuisance factors of different examples to produce augmented representations.
be
3.1 REPRESENTATION DISENTANGLEMENT AND REASSEMBLY
As described above, TALLY reassembles augmented examples from pairs of examples by com- bining the semantic representation of one with the domain-related nuisance factors of the other. Motivated by style transfer (Huang Belongie 2017). we use instance normalization (Instan- ceNorm (llynnov et al. 2016)) perform the required disentanglement of semantic and nuisance information. Concretely, given an example (x,, y. d) we denote the hidden representation at layer (z) E RCx where C, H, and W denote channel, height, and width dimensions, respectively, Ignoring affine parameters, InstanceNorm normalizes the example as:
2(x) InstanceNorm(e) -[(ss) a(s))
where 2(8), C
(2)
3

---

Under review as a conference paper at ICLR 2023
where (-) are the mean and standard deviation computed across the spatial dimensions of
w s|: h, W| a(8) = HW
pls) HW
(s|: h w/ - (()))²
(3)
Following Huang& Belongie 2017). treat the normalized example z(8) the semantic represen- tation, and regard (a(s) and (8)) as the domain-associated nuisances. Notice that we adopt warm start strategy of running vanilla ERM for the first few epochs to ensure reliable disentanglement.
After decoupling representations, we produce an augmented representation from pair of examples (2i, Mk, d.) and (æ,, Vy. d;) by swapping semantic representations and aosain-associated nuisances:
= a(s;)
$;
n(م.)) + (*3)),, ū = yi-
(4)
(s:)
Since the semantic content of the augmented representation á is from example (I,,, Vis de), we label our augmented example with ÿ = y:- By reassembling disentangled representations, we can augment representations for minority domains or minority classes.
3.2 SELECTIVE BALANCED SAMPLING
In the process of representation disentanglement and re- assembly, finding a suitable strategy of sampling exam- ples from the training distribution is crucial to solving the domain-class imbalance problem. In single-domain long-tailed learning, up-sampling examples from mi- nority classes classical yet effective method. In multi-domain long-tailed learning, the most straightfor ward extension is up- upmmamlin examples from minority domain-class groups, which is named balanced sam- pling here. practice, for each example (I+; Ve, d.), the label ys and domain d, uniformly sampled joint uniform distribution over all domain-class combi- nations, i.e., (ye, d.) Uniform(C, D).
Dezame Class
Paa Clpar))
Art
(Campeac Clpuri
Gooup
Sours Grop
B Traniing Faaeple
15
10
Talaaced Samplinn
1/
1304
Sumpligg
Figure 3: Illustration of selective balanced sampling. Transition probabilities between different pairs are visualized. Detailed dis- cussion is in Appendix A.I
However, to transfer the knowledge between different domain-class groups in TALLY, using such a sampling strategy may overemphasize the importance of minority domain-class groups. In Figure 3 we illustrate three domain-class groups from OfficeHome--T, which is a long-tailed variant of OfficeHome dataset (Venkateswara et al.) 2017). To augment minority groups (e.g., fan-clipart pair), balanced sampling tends to repeatedly draw examples from the same minority group. We do not expect this because of two reasons: first, it limits the sample diversity in knowledge transfer; second, as shown in Figure minority groups typically perform worse than majority groups, which may make the knowledge transfer less reliable. Hence, we propose a selective balanced sampling strategy in TALLY. Concretely, for a pair of examples (I,, Vi. d,) and (z,, Mj, d,), the label y, of example is uniformly sampled from all classes (y, ~ Uniform(C)) and the domain d, of example j is uniformly sampled from all domains (dy ~ Uniform(D)). The illustration verifies that selective balanced sampling has a higher chance of diversifying the sample selection in transferring domain and class information.
3.3 PROTOTYPE-GUIDED INVARIANT LEARNING
Since the semantic representation 2ss) (Eqn. 2 should contain only class-relevant information, should ideally be domain-invariant However, per-instance statistics can be noisy and instance normalization may not perfectly disentangle the semantic information from the relate-related nuisances. To improve robustness, we can "average out" domain information over many examples of the same class from different domains. However, merely averaging over examples would remove the diversity that distinguishes different examples of the same class. We balance diversity and Nomain-invariant by interpolating 2s)) with the corresponding class prototype representation. We define the class prototype representation re the average semantic representation over examples
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 TALLY Training Process
Require: learning rates fy, start epochs To: prototype momentum y; model fo(-) with hidden
representation (-) at layer r; Dataset D = {(r, y/ d)
1: Initialize domain-agnostic prototypes {r. JemI and class-agnostic statistics { (0) (a) )
2: Train fo with ERM for To
3: for t = Ty to T do
4: y/s ~ Uniform(C), dj ~ Uniform(D)
Randomly sample domains and classes
5:
(21, (D h (2), d,) (D"r/d= d,}
6:
(s1,8;) - (f)),
> Compute hidden representations
7:
z(si) InstanceNorm(ss
Disentangle semantic factor (Eqn.
8:
x(s..) xe2ss.) +1
P Enhance semantic factor (Eqn. 6
9:
(''(,,), + {a(u(s)), (1- (1)))
>. Enhance nuisances (Eqn.
10:
(3' - ((((s))(())++(() 8:)
Generate augmented example (Eqn.
11:
Train augmented example (Eqn. 10
12:
Estimate the current prototypes and feature statistics (re)ema. [(vd, ma))
13:
for = to C do
14:
re (++1) + pr (1 - r)rr
D Update domain-agnostic prototypes
15:
for d = 1 to Do
16:
(u)
(t+1) o 1(+1)) + 7u1 (0) (1 (us, vs)
Update class-agnostic statistics
belonging to class regardless of domain:
8; ((:)) ne 1=1 (s:))
To
Th (s:)
(5)
For each example (I., y, d) with y obtain the prototype-enhanced semantic representation by linearly interpolating z(s) with the corresponding class prototype Te:
<s(s) ).=(s) + (1 Ae)re.
(6)
where Ao Beta (ce) is the interpolation coefficient. By applying this class prototype-based interpolation strategy, we are capable capturing invariant knowledge and keeping the diversity of instance-level semantic representation when swapping information.
We also desire that the disentangled μu(s) and (s)) (Eqn. 2) contain only Nomain-relatee nuisance information. However, for similar reasons with 2(s). they may still contain some class-related semantic information which we would like to remove by "averaging out." In this case, we remove semantic information by averaging over examples from different classes within the same domain:
Uid
Σ (ss)),
1 nid
(3)),
nid
where n4 represents the number of training examples in domain d. Then, for each example, we linearly interpolate its domain-associater nuisances with the above class-agnostic nuisances as:
p (s)) = >ap(s) (1 Aa)ua, o' Asa(s) + (1 Aaava,
(8)
where the interpolation ratio is Xa Beta(a,, aa). In practice, we update the class prototype and domain-agnostic nuisances ua and va with momentum updating, where we denote the values of re. uas va at epoch L. as (1) uz and (()) respectively.
By replacing the original semantic representation and domain-associated nuisances in Eqn. with the prototype-guide guided ones, we obtain the enhanced augmented representation as follows:
== a' (s)))= (ss))=((s)) (x;) p (+z), D' Vi-
Finally, replace the original training data with the augmented ones and reformulate the optimization process in Eqn. as:
min Eg be
(10)
where fC-" represents the post-layers after layer r. It is also worthwhile to point out that TALLY be incorporated into any kinds of class-imbalanced losses (e.g., Focal, LDAM). We summarize the overall framework of TALLY in Algorithm
5

---

Under review as a conference paper at ICLR 2023
Table Results of subpopulation shifts and domain shifts on synthetic data. Domain-class balanced accuracy is reported. See full table with standard deviation in Appendix D.3 We bold the best results and underline the second best results. OH-LT and DN-LT represent OfficeHome-LI and DomainNet-LT, respectively.
Subpopulation Shift OH-LT
Domain Shift
VL
PACS LT
DN-LT
VLCS-LT
PACS-LT LT
OH-LT
DN-LT
ERM
73.33%
90.40%
61.07%
44.33%
67.62%
76.27%
51.95%
33.21%
Focal
74.83%
90.44%
62.57%
47.35%
69.38%
75.29%
54.03%
35.23%
LDAM
73.83%
90.91%
63.57%
46.71%
69.41%
77.53%
53.60%
34.42%
CRT
73.83%
89.17%
61.92%
47.37%
65.67%
73.82%
53.62%
36. 36%
MISLAS
71.83%
90.99%
61.38%
49.15%
68.64%
77.94%
52.86%
36.18%
Remix
74.16%
90.83%
61.59%
47.56%
67.71%
75.25%
$1.43%
35.14%
RIDE
74.33%
90.48%
63.27%
47.51
69.29%
77.41%
53.75%
35.26%
PaCo
73.67%
91.31%
63.03%
48.26%
69.05%
76.79%
53.98%
35.76%
IRM
50.50%
65.24%
45.48%
35.57%
48.32%
52.60%
42.34%
28.19%.
GroupDRO
72.50%
89.80%
59.79%
43,86%
69.18%
76.75%
51.12%
32.54%
CORAL
71.67%
88.22%
59.10%
43.92%
66.54%
75.62%
50.74%
33.44%
LISA
74.67%
90.08%
57.39%
43.17%
66.42%
74.47%
48.22%
34.99%
MixStyle
74.30%
91.55%
62.26%
43.59%
67.75%
79.78%
52.47%
33.71%
DDG
73.00%
89.60%
58.80%
44.46%
68.38%
75.97%
51.07%
33.94%
BODA
74.83%
91.03%
62.79%
47.61%
69.63%
78.81%
53.32%
35.85%
TALLY (ours)
76.83%
92.38%
67.00%
50.15%
70.60
81.55%
55.69 %
36.45%
EXPERIMENTS
In this section, we conduct extensive experiments to answer the following questions: Q1: How does TALLY perform relative to prior invariant learning and single-domain long-tailed learning approaches under subpopulation shift and domain shift? Q2: Since it is straightforward to combine invariant learning with imbalanced data strategies, how does TALLY compare with such combinations? Q3: What affect does incorporating the prototype representation (Eqn. (9) have, in comparison with naive representation swapping (Eqn. 4 Q4: Can TALLY produce models with greater domain invariance?
We compare TALLY two categories of algorithms. The first category includes single-domain long-tailed learning methods such as Focal (Lin et al.] (017)). LDAM (Cao et al. 2019), CRT (Kang
et al. 2020 MiSLAS (Zhong al.) (0021) RIDE (two experts) (Wang al. 2020a), PaCo (Cui
et al. 2021I. and Remix (Chou et al.) 2020). The second category includes approaches for improving
robustness distribution shift: IRM (Arjovsky et al. 2019), GroupDRO (Sagawa et al. 2020).
LISA Yao et al.) 2022). MixStyle (Zhou et al. (020b). DDG (Zhang et al. 2022). and BODA (Yang al. 2022), where BODA is work studying dolti-domain long-tailes learning by adding regularizer on domain-class pairs. Follow Yang et al. (2022), we use a ResNet-50 for all algorithms, and detail the All hyperparameters are selected vrossvaliddatio.
baselines and evaluation metrics in Appendix C
4.1 EVALUATION ON LONG-TAILED VARIANTS OF DOMAIN GENERALIZATION BENCHMARKS
Datasets. Many standard -omaaingennrrllizatoo benchmarks are not long-tailed, while standard cabbllancd--lassificatiin datasets tend be a single-domain. We curate four multi-domain
long-tailed datasets by modifying four existing
benchmarks: VLCS (Fang 2013), PACS (Li al. 2017), OfficeHome (Venkateswara al. 2017), and DomainNet (Peng
al.
et
al. 2019). We modify the prior datasets by removing training examples so that each domain has a
longetailed label distribution (overall imbalance ratio: 50) and call the resulting datasets VLCS-LT.
PACS-LT, OfficeHome-LTT, and Domain See Appendix D.I for more details.
Evaluation Protocol. We evaluate performance under both subpopulation shift and domain shift. In subpopulation shift, the test set is balanced across both domains and classes, which means that each domain-class pair contains the same number of test examples. In domain shift, we use the classical domain generalization setting (Zhang et al. 2022). More specifically, alternately use one domain as the test domain, and the rest as the training domains. Results are averaged over all combinations. Appendix D.I detail the statisties and training class distribution for each multi-doman long- tailed dataset. The hyperparameters a and a, in the Beta distribution are set to 0.5 and the warm start epoch Tp is set to 7. We list all hyperparameters in Appendix D.2

---

Under review as a conference paper at ICLR 2023
Results. The overall performance of TALLY and prior methods for tackling subpopulation shift is reported in Table (left). For subpopulation shift, we report the average performance over all domains and the full results are presented in Table of Appendix D.3 The following key observations be made according to T We first observe that most single-domain re-weighting approaches (e.g., Focal, L.AAM) consistently outperform multi-domain learning approaches (e.g., GroupDRO, CORAL) in most scenarios, indicating that imbalances in classes are probably more detrimental than imbalances in domains. This observation is not surprising, since all domains are observed in during training and testing in subpopluation shift problems. Even so, TALLY consistently outperforms all methods with 7. 29% error decreasing, verifying its effectiveness in improving the robustness to subpopulation shifts. It is particularly noteworthy that TALLY shows superior performance compared with BODA- - an invariant learning approach deal with multi-domain long-tailed learning. The results indicate that designing regularizers that are suitable for datasets from diverse domains can be challenging. Instead, balanced augmentation are capable of improving the robustness by transferring domain nuisances between examples.
In addition, Figure T shows performance bro- ken down by class size for OfficeHome-LT and DomainNet-LT,, where we split all classes into five levels according to their cardinality. We compare TALLY with ERM, and four strongest baselines (LDAM. CRT, MiSLAS, BODA). The results show that TALLY's performance improve- ments arise from larger improvements on smaller classes rather than performance improvements across the board, hence indicating that it is partic- ularly well-suited for class-imbalanced problems.
90%
75%
Acennyy
20%5
55%
50%
30%
XL
M
X5
15%
Cloos
Clocs
(a): OfficeHome-LT
(b): DettannNttLLT
Figure 4: Performance w.r.t. Class Size. We split all classes into five levels. XL and XS represent the largest and smallest classes, respectively.
Results of Domain Shifts. Table T (right) shows the domain shift results. We first find that ERM works relatively well compared to invariant learning approaches in most cases. This is expected since we evaluate the performance on unseen domains and similar observations have been reported from prior domain shift benchmarks (Koh al., 2021 Second, single-domain long-tailed learning methods boost the performance of ERM in most cases, showing that class-imbalance is still an important issue in domain shift. Even so, with the subpopulation shift setting, TALLY consistently outperforms prior approaches, indicating its efficacy in enhancing robustness to domain shift. Finally, also provide the comparison the standard version in Appendix Gnd TALLY also achieves comparable results compared with state-of-the-ar methods.
4.2 EVALUATION ON NATURALLY IMBALANCED MULTI- DATA
Datasets and Evaluation Protocol. To further evaluate TALLY and prior methods, we study two multi-domain datasets that are naturally imbal- anced: Terra Incognita (TerraInc) (Beery et al., 2018) and iWildCam (Beery et al., 2020), both of which aim to classify wildlife across differ- collected
Table 2: Results of domain shifts on real-world data. We report the full results in Appendix
Terraine
iWildCam Macro Fl
Macro F1
Acc
Acc
ERM
42.35%
54.81%
32.0%
69.0%
ent camera traps. The examples from different camera traps (domains) and the observed frequency of each species is naturally uneven, leading to an imbalanced class distribu- tion. More details of these datasets and class distribution are described in Appendix [ Here, we consider generalization new camera traps, i.c. to new domains. In TerraInc, the number of training, validation and test domains are 10, 5, 5, respectively. For iWildCam, we follow the same training, validation, and test splits as used the WILDS benchmark (Koh et al.) 2021). To bet- ter capture performance on rare species, we use macro Fl score the primary evaluation metric following Koh et al.) (202T). but we also report average accuracy. We list all hyperparameters in Appendix E.2
Focal
43.54%
56.62%
33.2%
74.7%
LDAM
44.29%
57.22%
32.7%
75.2%
CRT
43.09%
58.27%
32.5%
67.3%
MiSLAS
40.68%
52.96%
30.5%
59.8%
Remix
43.72%
58.40%
28.4%
65.8%
RIDE
44.03%
57.89%
32.8%
70.1%
PaCo
43.40%
57.35%
31.9%
72.6%
IRM
31.17%
49.27%
15.1%
59.8%
GroupDRO
42.22%
56.43%
23.9%
72.7%
CORAL
45,43%
58.10%
32.8%
73.3%
LISA
39.27%
54.92%
27.6%
64.9%
MixStyle
44.73%
57.55%
32.4%
74.9%
DDG
40.47%
53.61%
29.8%
69.7%
BODA
44.47%
57.52%
32.9%
70.5%
TALLY (ours)
46.23%
59.89
34.4
73.4%
7

---

Under review as a conference paper at ICLR 2023
Results. We report the results over all test domains in Table 2 The conclusions largely consistent with the results from Sec. Sec. where TALLY consistently improves the performance over all baselines and enhances the robustness of multi momti- long-tait oangtailed learning. Additionally, data interpolation based invariant learning approaches (e.g.. LISA) hurt performance compared with ERM. This is not a surprise because examples from large classes dominate the interpolation process, which essentially exacerbate the class imbalance (See Appendix [ for more discussion). The superiority of TALLY over prior augmentation techniques is further evidence of the effectiveness of balanced augmentation.
4.3 CAN WE SIMPLY COMBINE INVARIANT LEARNING APPROACHES WITH LONG-TAILED LEARNNIN TECHNIQUES?
Relaivve Saegweennt oves Viuila Mobad
To further understand the performance gains TALLY, we investigate whether combining existing invariant learning and long-tailed learning approaches can tackle multi-domain long-tain tailed distribution shifts. Specifically, we incorporate four up-weighting or up-sampling ap- proaches (UW, Focal, LDAM, CRT) with two represen- tative invariant learning methods (CORAL, MixStyle). We report the relative improvement of each combina- tion over the vanilla methods in Figure 6 Here, we use Officehome-I and DomainNet-La to evaluate subpop- ulation shift and Terrainc and iWildCam to evaluate do- main shift (Appendix) F-3)) performance. We see that ap- plying loss up-weighting or up-sampling approaches on performant invariant learning approaches does improve their performance, as evidenced by Figure 5 Nonethe- less, the consistent improvements from TALLY indicates the importance of considering domain-class pair information to achieve balanced augmentation.
-UW -LDAM
CRT
Misinle
MisStyle
Offechlonz-L11
Figure 5: and variants of two domain generalization approaches (CORAL, MixStyle), where replace the losses of them with class re-weighting or re-sampling ones. More results are reported in Appendix [3
Comparison between TALLY
4.4 ANALYSIS
How do prototypes benefit invariant learning? We analyze the effects of prototypes in alleviat- ing domain-associated nuisances. Specifically, we compare TALLY with three variants: (1) with- out using any prototype information (None); (2) only applying class prototype (C Only): (3) only applying class-agnostic nuisances (D Only). We report the results in Figure 6 (full results in Ap- pendix F.4). We observe that adding class proto- type does improve the performance. The class- agnostics domain factors also benefits the perfor- mance to some extent. In summary, TALLY out- performs its variants, demonstrating the effective- ness of prototype representation in mitigating domain-associated nuisances.
New Catylyy
posty TALLY
New: Coibyty
Do 124LLY
44%
500
34%
(a) Subpopulation Shift
(b) Domain Shift
Figure 6: Analysis of prototype-guidee invariant learning. C Only and D Only represent only using class prototype representation or class- agnostic domain factors, respectively.
Does TALLY lead to stronger domain invariance? We analyze and compare the domain invariance of classifiers trained by ERM, TALLY, and other in- variant learning approaches. Following (Yang et al. 2022; Yao et al., 2022). we measure the lack of do- the accuracy of domain prediction (Iace) and as the pairwise divergence of unscaled log- its (IRz). Specifically, for the accuracy of domain top of
Table 3: Invariance Analysis of TALLY. OH- and DN-LT represents Officehome- and DomainNet-LT, respectively.
OH-LT
DN-LT
Model
main invariance
Laen J
1a 4
Lace +
ler 1
ERM
46.35%
2.030
70.00%
4.852
MixStyle
44.42%
2.169
67.11%
5.661
prediction, we perform logistic regression the unscaled logits to predict the domain. For the pair- wise divergence, use kernel density estimation to estimate the probability density function P(A-d)) of logits from domain-class pair (c, d' and calculate the KL divergence of the distribution of logits from different pairs. Formally, Tg is defined as Tui
CORAL
42.21%
1.248
66.79%
4.593
BODA
40.10%
2.052
65.15%
6.810
TALLY
39.52%
1.179
63.80%
3,956
" Enge KI.
8

---

Under review as a conference paper at ICLR 2023
We report the results of Officehome-LI LT and DomainNet-LT in Table B Smaller lace and I&st values indicate more invariant representations with respect to the labels. The results show that TALLY does lead to greater domain-invariaal compared to prior invariant learning approaches (e.g., BODA).
Analysis of Sampling Strategies. Finally, we compare the pro- posed selective balanced sampling in TALLY with domain-class balanced sampling. As discussed in Sec. [.2] for an example pair (Ii, Vi. di) and (ry Mý. dj), selective balanced sampling gets ~ Uniform(C) and d, ~ Uniform(D) while traditional bal- anced sampling get (y,, (w,, d,) ~ Uniform((. D). The results of subpopulation shifts in OfficeHome-LL DomainNet-LT and of domain shifts (Macro-FI) in Terralne, iWildCam are reported in Table (see Appendix F5 for full results), indicating the effectiveness of selective balanced sampling in transferring knowledge over domains and classes.
Table 4: Comparison between sampling strategies.
OH-LT
DN-LT
Balanced Sampling (SeeeiSeleectie))
65.03%
49. 39.35%
67.00%
50.15%
Terralnc
iWildCam
Balanced Sampling
44.79% 46.23%
33.1%
TALLY(Sclocive))
34.4%
5 RELATED WORK
Long-Tailed Learning. Training a well-performed machine learning model on class-imbalanced data has been widely studied and a lot of approaches have been proposed, including over-sampling minority classes or under- samplig majority classes (Chawla al. 2002 Estabrooks et al. 2004: Kang et al.)
2020; et al. [2008] Zhang & Pfister, 2021). 1,, adjusting loss functions or logits for different classes
during training (Cao et al.) 2019 Cui et al. 2019 Hong al. all]| [202]] Jamal et al. 2020 Lin et 2017
2019|
transferring knowledge from head classes to tail classes (Wang et al.] 2017 Liu et al. 2020; Yin et al.
2019 Zhou et al. 2022). directly augmenting tail classes (Chou et al. [2020] Kang et al. [220)] Zhong
ct al. 2021 1, and ensembling models with different sampling or loss weighting strategies (Xiang
ct al. 2020; Zhou et al., (020a). Unlike single-domain imbalanced learning, Yang et al.] (2022)
targets the multi-domain imbalanced learning scenario by encouraging invariant representation
learning with a domain- calibrated regularizer. However, BODA focuses on subpopulation shift with the imbalanced distribution for each domain, while the overall distribution among all classes are relatively balanced. TALLY instead studies more kinds of distribution shifts with conceptually different direction alleviate domain assoociaed nuisances via balanced augmentation. It relaxes the explicit constraint on internal representations and leads to stronger empirical performance.
Domain Generalization and Out-of- De--strttbbuion Robustness. To improve out-of-distributio robustness, one line of works aims to learn domain-invariant representations by 1) minimizing the
discrepancy of feature representations across all training domains (Li al. 2018 Sun & Saenko
2016 Tzeng et al. 2014: Zhou et al. 20200); 2) leveraging domain augmentation methods to generate
more training domains and improve the consistency among domains (Shu et al. 2021 Wang et al..
2020b Xu et al.) 2020: Yan et al., 2020: Yue et al. 2019, Zhou et al. 2020c); ; 3) disentangling
feature representations to semantic and domain-varying ones and minimizing the semantic differences across training domains (Robey et al. 2021 Zhang et al. 2022). Another line of works focuses on learning invariant predictors with regularizers, including minimizing the variances of risks across domains (Krueger et al. (002)) encouraging a predictor that performs well over all domains (Ahuja et al. 2021 Arjovsky et al. 2019: Guo et al. 2021: Khezeli et al.) (021)). Apart from explicitly involving regularizers, data augmentation
another promising approach for learning invariant predictors (Yao et al. 2022) Zhou et al. 20206). Unlike previous augmentation methods that require sufficient training examples for each class to learn invariance (see detailed discussion in Appendix B issue in domain generalization and employs a domain-balance augmentation strategy to learn class-unbiased invariant representation.
TALLY tackles the class-imbalang
6 CONCLUSION
In this paper we investigate multi-domain imbalanced learning, natural extension of classical single- domain imbalanced learning. We propose a novel balanced augmentation algorithm called TALLY to achieve robust imbalanced learning that can overcome distribution shifts. To generate more examples, TALLY introduces a prototype enhanced disentanglement procedure for separating semantic and nuisance information. TALLY then mixes the enhanced semantic and information among examples. The results on four synthetic and two real-world datasets demonstrate its effectiveness over existing imbalanced classification and invariant learning techniques. In the future, we plan to conduct theoretical studies to better understand how TALLY works.
nuisance

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. 2021.
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint 1eXiv:9911008000 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 0000000999, 2019.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European conference computer vision (ECCV), pp. 456-473, 2018.
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildeam 2020 competition dataset. arXiv preprint arXiv:2004.1034 10340, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abeel-distribution-awaae margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference on Computer Vision, pp. 95-110. Springer, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Blass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-33) 36,, 2004.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference Computer Vision pp. 1657-1664, 2013.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2021.
Ruocheng Guo, Pengchuan Zhang, Hao Liu, and Emre Kiciman. Out-of-distribution prediction with invariant risk minimization: The limitation and an effective fix. arXiv preprint arXiv:2101.0773 2021.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailes visual recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Muhammad Abdullah Jamal, Matthew Brown, Ming- Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pp. 7610-7619, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. 2020.
10

---

Under review as a conference paper at ICLR 2023
Kia Khezeli, Arno Blaas, Frank Soboczenski, Nicholas Chia, and John Kalantari. On invariance penalties for risk minimization. arXiv preprint arXiv:2106.09777, 2021.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMIR, 2021.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp- 5815-5826. PMLR, 2021.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp- 5542-5550, 2017.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2970-2979, 2020.
Xu- Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cyhernetics, Part B (Cybernetics), 39(2): 539-550, 2008
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1416-1415, 2019.
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210-20229, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
Yuge Shi, Jeffrey Seely, Philip HS Torr, Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint 1:Xxi:2104.09933: 2021.
Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain- augmented meta-leaming In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0624-9633, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450. Springer, 2016.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1rXiv:1607.08022, 2016.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5018-5022 2017.
11

---

Under review as a conference paper at ICLR 2023
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by aware experts. arXiv preprint arXiv:2010.0/809, 2020a.
routing diverse distribution-avve -awaut
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in Neural Information Processing Systems, 30, 2017.
Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3622-3626. IEEE, 2020b.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 34, pp. 6502-6509, 2020.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.000 .00677, 2020.
Yuzhe Yang. Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, generalization and beyond. arXiv preprint wrXiv:2203.095/1 13, 2022.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of- diitr---ittiittiion robustness via selective augmentation. arXiv preprint arXiv:2201.00299, 2022.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5704-5715 2019.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boging Gong, Domain randomization and pyramid consistency: Simulation-to-reat generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100-2116 10, 2019.
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. CVPR, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 2018.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv: 10.00596, 2021.
Zizhao Zhang and Tomas Pfister. Learning fast sample re- re-weighting without reward data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 725-734, 2021.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recogni- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
Allan Zhou, Fahim Tajwar, Alexander Robey, Tom Knowles, George J Pappas, Hamed Hassani, and Chelsea Finn. Do deep networks transfer invariances across classes? 2022.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brance network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9719-9728. 2020a.
Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa. Domain generaliza- tion with optimal transport and metric learning. arXiv preprint urXiv:2007. 2020b.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13025-13032, 2020c.
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL INFORMATION FOR TALLY
A.1 ADDITIONAL DISCUSSION OF SELECTIVE BALANCED SAMPLING
In this section, we detail our explanation about Figure and show why selective balanced sampling is a better strategy in TALLY. In Figure 3 there are three domain-class groups: (Fan, Clipart,, (Fan, Art), (Computer, Clipart) and the number of examples for each group is 1, 13, 83, respectively. We specifically focus on augmenting examples from the minority group (i.e., (Computer, Clipart) group). In this case, the samples (I,, yr. d.) and ( Vj+ d,) need to contain class semantic information (i.e., Fan) and domain information (i.e., Clipart) in the representation augmentation module, respectively.
The balanced sampling under the multi-domain long-tailed classification problem to up-sample examples from minority domain-class groups. Concretely, the label Yi and domain de for each example (I,; Vi, d,) are jointly sampled from Uniform(C, D). If we employ original balanced sampling in the representation augmentation module of TALLY, then (yi, d.) Uniform D), dz) Uniform (C, D). To augment the (Fan, Clipart) group, the class semantic information from (24, Ve, d;) has 1/2 probability to be obtained from the original (Fan, Clipart) group, and 1/2 probability to be obtained from the (Fan, Art) group. Similarly, the domain information has a 1/2 probability to be obtained from the original (Fan, Clipart) group, and 1/2 probability to be obtained from the (Computer, Clipart) group. Thus, examples from the minority group (i.e., Fan, Clipart) will be repeatedly sampled during the augmentation process and this is what we do not expect.
Instead, for selective balanced sampling, the label y; of example i is uniformly sampled from all classes (i.e.. yi Uniform(C)), and the domain dj or example j uniformly sampled from all domains (i.e.. d, Uniform(D)) Thus, to augment the (Fan, Clipart) group, the class semantic information from example (z,, Yi: d.) has a 1/14 probability to be obtained from the original (Fan, Clipart) group and 13/14 probability to be obtained from the (Fan, Art) group because we do not consider domain information sampling example (xi, yt: de). Similarly, for selective balanced sampling, the domain information has a 1/84 probability to be obtained from the original (Fan, Clipart) group, and 83/84 probability to be obtained from the (Computer, Clipart) group.
To sum up, using selective balanced sampling can provide more diverse and effective knowledge transfer.
A.2 ALGORITHM OF THE TESTING STAGE OF TALLY
In this section, summarize the testing stage of TALLY in Alg. 2 It is worthwhile notice that the representation augmentation module is only used during the training stage.
Algorithm 2 TALLY Testing Process
Require: model fe- (.) with learned parameter 6": Test Dataset Dts
1: Feed all testing examples {(z;, yi, Hi to model fe- (-) and get the predicted label of each example (2r, y, d) as ý fo-
2: Evaluate and report the performance based on the predicted values and the groundtruth.
B ADDITIONAL DISCUSSION OF RELATED WORKS
In this section, we provide an additional discussion of related works. Specifically, we would like to point out why data interpolation-basee based domain generalization approaches LISA (Yao et al.) 2022), mixup) can not benefit the performance when encountering the long-tailed distribution. Take LISA as an example, we adopt intra-label LISA in this paper, which is more suitable for domain shift as mentioned in (Yao et al.) 2022). Intra-label LISA learns comain-invarian predictors by interpolating examples with the same label but from different domains, which can probably aggravate the label imbalance issue. We provide a simple example to explain why LISA changes the label distribution. Assume we have two classes and two domains, the ratio of training examples between four domain-class pairs in training set is: (y1, dz) (y1, d2) (V2, d1): (22, d,) 100 200 80 5. The label imbalance ratio is 300 : 85 = 3.52. To apply LISA, we can essentially have 100 x 200 = 20000 example pairs in class 1 (11), and 80 x 5 400 example pairs in class (yz), which roughly
LISA
13

---

Under review as a conference paper at ICLR 2023
leads to new label imbalance ratio 20000: 400 50 > 3.52. The comparison between the number of example pairs for interpolation can not precisely reflect the training label imbalance ratio in LISA, but it shows that LISA changes the label distribution to some extent.
C ADDITIONAL DETAILS OF GENERAL EXPERIMENTAL SETUPS
C.I DETAILED BASELINE DESCRIPTIONS
In this paper, we compare TALLY with two types approaches: long-tailed classification methods and invariant learning approaches. We detail these methods here:
Long-tailed Classification Methods. We compare TALLY with Focal Lin al. (017)) p. LDAM Cao ct al. (2019), CRT Kang et al. (2020). MiSLAS Zhong et al. (202I). and Remix Chou al. 2020).
Here, Focal and LDAM up- up-weight the loss for minority classes. CRT uses up- up-sampling strategy to fine-tune the classifier. MiSLAS and Remix modify the vanilla mixup Zhang et al. (2018) and make it suitable to long-tailed distribution.
Invariant Learning. We further compare TALLY with invariant learning approaches, i.e. IRM Ar jovsky et al. (2019), GroupDRO Sagawa et al. (2020), LISA Yao et al.] (2022), MixStyle ZZhu et al. (2020b), DDG DDG et al. (2022), and BODA Yang et al. (2022). IRM learns invariant predictors that perform well across different domains. GroupDRO optimizes the worst-domain loss. LISA cancels out domain-associated information by mixing examples with the same label but different domains. MixStyle decomposes the feature representation into content information and style information. It then mixes the style information and generates new examples. Unlike MixStyle, TALLY generates examples of minority classes or domains, and uses prototypes to improve the model robustness, which more suitable for long-tailed multi-domain learning. DDG uses an extra network disentangle original examples and generate more. Finally, BODA is concurrent work for long-tailed multi- domain learning with an explicit regularizer. Unlike BODA, TALLY studies a conceptually different direction to cancel out domain domain-associater associated nuisances by domain-class balanced augmentation, leading to stronger empirical performance.
C.2 DETAILED EVALUATION METRICS
In this section, we detail our evaluation metrics. In synthetic data, to emphasize the performance on minority classes and domains, we use domain-class balanced accuracy as the evaluation metrie, where the number of test examples is the same across every class within every test domain. In real- world data, follow Koh al. (20TT) we use Macro Fl over all classes and average accuracy to evaluate the performance, where the Macro Fl is served as the primary metric to emphasize the performance on rare classes.
D ADDITIONAL RESULTS OF SYNTHETIC DATA
D.1 DETAILED DATASET DESCRIPTION
VLCS-LT contains examples from 4 different domains, including Caltech101, LabelMe, SUN09, VOC2007. To create the long-tailed class distribution, we modify the original dataset by removing training examples. The dataset contains 5 classes with 6,361 images of dimension (224, 224, 3). The long-tailed training distribution is visualized Figure 7a In subpopulation shift, the number of examples of each class per domain for validation and testing is 5, 10, respectively.
PACS-LT includes 3,097 images collected from 4 domains (Art painting, Cartoon, Photo. Sketch) and 7 classes. Similar to VLCS-LT, we construct PACS-LT with long-tailed training distribution illustrated in Figure 7b The validation set size and test set size of each class per domain in subpopulation shift are and 30 respectively.
OfficeHome-LT is built upon the original OfficeHome dataset, including 3280 images of 65 classes collected from four domains Art, Clipart, Product, Real. The long-tailed training distribution is
14

---

Under review as a conference paper at ICLR 2023
shown in Figure 7c and the number of examples for each class per domain in validation and test sets are 4, 8, respectively.
DomainNet-LT. Similar to the other three datasets, DomainNet-LT covers 173,200 examples from Sketch, Infograph, Painting, Quickdraw, Real, Clipart. There 345 classes in DomainNet-LT. In subpopulation shift, the number of examples of each class per domain is 3, 6, respectively. We illustrate the long-tailed training distribution in Figure
Number of Instances
Cemehbua Labelit VOCGE8?
Number Instances
1400
Cottem
Number Instances
200
Art
Clipart
Product
1900
(a) VLCS-LT
(b) LT
(c) OfficeHome-LI
Sketch
Number of Instances
Infograph
1600
Painting
Quickdraw
Real
Clipart
(d) DomainNet-LT
Figure 7: Long-tailed training distributions for all synthetic datasets. Here, the x-axis represents sorted class indices.
D.2 DETAILED EXPERIMENTAL. SETUPS AND HYPERPARAMETERS
In this section, detail how split the training and test set in synthetic datasets under both subpopulation shifts and domain shifts. subpopulation shift, the training distribution for each domain is long-tailed distribution and the test domain-class balanced distribution, i.e., the number of examples in each domain-class pair is the same. In terms of domain shifts in synthetic datasets, following (Peng et al. 2019 Shi et al. (02T), hold one domain the testing domain and the rest domains as training. All baselines and TALLY use the same evaluation protocol.
We list the hyperparameters in Table for the above four synthetic datasets.
Table 5: Hyperparameters for experiments synthetic data.
Hyperparameters
VLCS-LT
PACS-LT
OfficeHome-LL
DomainNet-LT
Learning Rate
le-5
lc-5
3e-5
3e-5
Weight Decay
lc-6
le-6
le-6
le-6
Batch Size
18
18
18
18
Epochs
15
15
15
15
Steps
200
500
500
1000
Warm Start Epochs
? in feat. estimation
0.8
0.8
0.8
0.8
class prototype mixup parameter
0.2
0.5
0.5
0.5
domain prototype mixup parameter aza
0.2
0.5
0.5
0.5
15

---

Under review as a conference paper at ICLR 2023
D.3 FULL RESULTS
The full results of subpopulation shift are reported in 6 In domain shift, we report the results of each domain for VLCS-LT, PACS-LT, Hofice and Domain in Table 7 8 respectively. In the domain shift scenario of VLCS-LT, though TALLY only performs best in VOC2007 (VLCS), the results of TALLY is relatively more stable compared other approaches. leading to the best averaged performance.
9
10
Table Full results of subpopulation shifts on long-tailed variants of domain generalization bench- marks. The standard deviation is computed across three seeds.
VLCS-LT
PACS-LT
OfficeHoome-LT
DomainNeetLLT
ERM
73.33
0.76%
90.40 + 0.88%
61.07 + 0.73%
44.33 0.14%
Focal
74.83 1 0.29%
90.44 1 0.06%
62.57 1. 0.50%
47.35 0.09%
LDAM
73.83 1.04%
90.91 + 0.15%
63.57 0.08%
46.71 0.33%
CRT
73.83 - 1.89%
89.17 + 1.47%
61.92 + 0.54%
47.37 + 0.83%
MiSLAS
71.83 + 1.25%
90.99 + 0.90%
61.38 + 0. 0.19%
49.15 + 0.69%
Remix
74.16 + 0.76%
90.83 + 0.77%
61.59 4 0.44%
47.56 + 0.25%
Avg-
RIDE
74.33 + 0.76%
90.48 + 0.32%
63.27 + 0.54%
47.51 0.83%
PaCo
73.67 + 0.28%
91.31 + 0.58%
63.03 + 0.92%
48.26 0.22%
IRM
50.50 8.18%
65.24 + 7.57%
45.48 + 4.30%
35.57 5.76%
GroupDRO
72.50 + 0.50%
89.80 + 0.70%
59.79 + 0.43%
43.86 0.33%
CORAL
71.67 + 0.28%
88.22 + 0.67%
59.10 + 0.20%
43.92 + 0.36%
LISA
74.67 + 0.76%
90.08 1 0.45%
S7.39 1 0.59%
43.17 1 0.53%
MixStyle
74.30 + 1.04%
91.55 1 0.25%
62.26 1 0.22%
43.59 1 0.57%
DDG
73.00 1.63%
89.60 1 0.40%
58.80 0.57%
44.46 0.06%
BODA
74,83 - 1.84%
91.03 + 0.31% %
62.79 + 0.45%
47.61 + 0.04%
TALLY (ours)
76.83
1.04%
92.38
0.26 %
67.00 1L
50.15
0.46%
ERM
52.67 + 2.31%
83.81 1: 2.43%
54.48 + 0.89%
25.36 + 0.63%
Focal
52.67 +: 1.15%
84.44 1 0.81 %
56.41 1: 1.34%
27.68 1: 0.13%
LDAM
51.33 1:33:::2 2.31%
85.24 1 1.03%
58.07 1 0.82%
27.23 1 0.26%
CRT
52,00 0.00%
83.02 1 1.12%
55,51 0.79%
27.55 0.49%
MiSLAS
52.00 3.46%
86.03 + 0.90%
52.82 + 0.39%
29.42 + 0.15%
Remix
51.33 + 3.05%
86.98 + 0.59%
53.85 + 0.54%
28.13 + 0.99%
Worst
RIDE
53.50 2.31%
85.71 + 0.82%
54.04 + 0.28%
28.16 + 0.60%
PaCo
53.83 = 0.94%
87.14 + 0.46%
56.15 + 0.71%
27.36 + 0.52%
IRM
32.63 + 7.03%
59.38 + 5.93%
40.58 + 4.42%
20.48 + 3.71%
GroupDRO
51.33 1.15%
83.02 + 0.59%
54.04 + 0.30%
25.02 0.73%
CORAI.
49.33 + 1.15%
81.59 + 0.81%
53.53 + 0.60%
24.50 0.68%
LISA
53.33 + 1.15%
83.01 + 0.81% %
49.04 + 0.40%
24.05 + 0.48%
MixStyle
54.00 + 2.00%
86.98 + 0.98%
55.19 + 1.10%
22.65 + 0.22%
DDG
51.33 1 0.94%
82.70 1 2.38%
51.99 1 0.55%
24.35 1: 0.20%
BODA
54.00 2.83%
85.08 1 1.37%
55.70 1 0.50%
26.94
0.44%
TALLY (ours)
56.00 2.00%
89.21
0.22%
60.45 + 0.09%
29.55 + 0.19%
E ADDITIONAL RESULTS REAL-WORLD DATA
E. DETAILED DATASET DESCRIPTION
TerraIne. Building upon the original Terra Incognita Beery et al. (2018), we select images from 10 classes and split the entire dataset to training, validation and test domains, which includes images from 38,042, 6,783, 7,303 camera traps, respectively.
iWildCam is wildlife recognition datasets. It is multi-class species classification. where the training data are collected from 243 domains and the test data includes images from 164 domains. We follow Koh et al. (2021) split the data and construct training, validation and test sets.
16

---

Under review as a conference paper at ICLR 2023
Table Domain shift results on VLCS-LT.
Caltech101
LabelMe
SUN09
VOC2007
Avg
ERM
92.39 + 0.35%
47.74 + 1.13%
59.79 + 2.70%
70.55 + 1,51 %
67.62%
Focal
97.12 i 1.06% %
48.83 + 0.38%
58.66 1: 2.31% %
72.91 1: 1.51%
69.38%
LDAM
95.55 1 1.65%
47.61 + 1.12%
61.34 1: 3.02%
73.17 1: 1.33%
69.41%
CRT
92.39 1.82%
47.74 = 1.52%
55.10 + 2.61%
67.45 1.54%
65.67%
MiSLAS
95.24 1.51%
47.00 = 0.99%
56.03 = 1.29%
76,28 + 1.66%
68.64 68.64%
Remix
92.66 + 1.42%
48.77 + 1.31%
57.98 = 2.62%
71.45 + 0.87%
67.71%
RIDE
95.70 + 1.24%
48.13 + 0.58%
59.62 + 1.53%
73.72 + 1.19%
69.29%
PaCo
96.11 + 0.98%
49.48 + 1.01%
58.85 + 2.04%
71.76 + 1.52%
69.05
IRM
74.10 = 2.67%
37.07 + 1.87%
34.33 + 1.77%
47.78 + 3.59%
48.32%
GroupDRO
93.79 1.01%
49.63 + 1.09%
62.25 1.89 %
71.06 0.55%
69.18%
CORAL
93.94 + 1.53%
48.29 + 1.08%
56.12 + 1.84%
67.82 + 1.43%
66.54%
LISA
90.28 + 0.68%
48.51 + 1.58%
58.82 + 2.41 %
68.09 + 1.53%
66.42%
MixStylc
96.58 + 0.84%
48.15 + 1.20%
58.82 + 1.94%
68.09 + 1.98%
67.75%
DDG
95.46 1.19%
50.42 + 1.45%
57.44 + 2.07%
70.21 + 1.33%
68.38%
BODA
95.60 1.37%
51.42 1.31% %
59.93 1: 1.97%
71.57 1.18%
69.63%
TALLY (ours)
95.22 + 0.92%
50.07 + 1.17%
60.13 + 2.17%
76.98 + 0.57 %
70.60%
Table 8: Domain shift results PACS-LT.
Art painting
Cartoon
Photo
Sketch
Avg
ERM
80.41 + 1.21%
70.21 + 1.14%
94,46 + 0.19%
60.00 + 5.04%
76.27%
Focal
80.92 1 0.51 %
69.58 1 0.64%
93.81 1: 0.80%
56.83 1: 2.04%
75.29%
LDAM
81.82 1: 1.14%
71.64 1 0.66%
95.34 1: 0.32%
61.30 1: 4.83%
77.53%
CRT
78.14 1 0.99%
67.17 + 0.73%
94.33 +: 0.78%
55.62 1 6.57%
73.82%
MiSLAS
81.31 0.49%
71.15 = 0.28%
93.51 = 1.40%
65.78 2.13%
77.94%
Remix
82.79 1.21%
69.10 + 1.13%
92.09 = 1.01%
57.00 + 4.13%
75.25%
RIDE
81.57 + 0.57%
71.64 + 0.42%
94.69 = 0.49%
61.74 + 2.52%
77.41%
PaCo
80.86 + 0.88%
71.29 + 0.28%
95.45 + 0.85%
59.55 + 3.02%
76.79%
IRM
51.87 + 4.93%
50.27 + 5.77%
69.11 + 4.43%
39.13 + 9.65%
52.60%
GroupDRO
80.20 + 0.57%
70.61 + 1.41%
94.58 + 0.90%
61.61 + 1.48%
76.75%
CORAL
77.60 + 0.79%
68.19 + 0.73%
93.88 0.40%
62.82 2.67%
75.62%
LISA
81.09 + = 0.61%
65.68 + 0.87%
94.40 + 0.33%
56.69 + 1.99%
74.47%
MixStyle
83.45 + 0.90%
72.84 + 0.59%
95.20 + 0.49%
67.61 + 0.83%
79.78%
DDG
79.67 + 0.77%
68.30 + 0.34%
94.72 + 0.50%
61.20 + 0.82%
75.97%
BODA
81.13 1 0.59%
72.03 1 0.65%
95.73 1 0.56%
66.34 1 1.55%
78.81%
TALLY (ours)
85,86 0.40%
74.20 1 0.30%
96.56 1 0.20 %
69.58 1 0.62%
81.55%
E.2 DETAILED EXPERIMENTAL. SETUPS AND HYPERPARAMETERS
In this section, detail how we split the training and test set in real-world datasets with domain shifts. Specifically, we follow Koh et al.) (202I) and use the same split as they did for iWildCam, where bunch of locations is selected for testing and the rest
used for training. For Terraine, we adopt the same strategy to split training and test domains since Terrainc also focuses on wildlife recognition and the data distribution is similar WWildCam. All baselines and TALLY use the same evaluation protocol.
We list the hyperparameters in Table 11 for both TerraIne and iWildCam datasets.
E.3 FULL RESULTS
The full results Real-world Data reported in Table 12
17

---

Under review as a conference paper at ICLR 2023
Table 9: Domain shift results OfficeHome-LT,
Art
Clipart
Product
Real
Avg
ERM
45.20 + 0.73%
41.94 + 0.17%
59.21 + 0.44%
61.44 + 0.27%
51.95%
Focal
47.06 1 0.24%
43.29 1 0.71%
62.34 1 0.16%
63.45 1 0.19%
54.03%
LDAM
47.08 0.37%
42.89 0.18%
61.48 0.55%
62.93 1 0.24%
53.60%
CRT
47.17 + 0.26%
42.62 0.55%
61.37 0.12%
63.31 = 0.25%
53.62%
MISLAS
45.22 + 0.52%
41.36 0.09%
62.28 0.49%
62.56 = 0.25%
52.86%
Remix
44.26 + 0.49%
39.18 = 0.34%
60.70 0.28%
61.58 = 0.42%
51.43%
RIDE
47.15 4: 0.38%
43,00 + 0.49%
61.50 0.19%
63.36 + 0.13%
53.75 %
PaCo
48.06 + 0.40%
42.61 0.83%
62.26 + 0.34%
62.98 + 0.27%
53.98
IRM
33.55 + 4.21%
34.34 3.74%
49.54 5.30%
51.95 + 4.64%
42.34%
GroupDRO
44.62 0.51%
41.84 0.68%
58.40 0.43%
59.63 = 0.53%
51.12%
CORAL
43.93 0.56%
42.71 0.59%
56.91 0.45%
59.40 + 0.94%
50.74%
LISA
41.80 + 0.36%
36.96 + 0.45%
56.51 + 0.16%
57.62 + 0.39%
48.22%
MixStylc
45.11 + 0.18%
45,52 = 0.20%
58.32 + 0.64%
60.92 + 0.22%
52.47%
DDG
43.89 + 0.39%
42.79 0.91%
57.92 0.15%
59.69 + 0.30%
51.07%
BODA
47.08 0.25%
44.38 0.77%
0.26%
62.25 1 0.10%
53.32%
TALLY (ours)
49.79 + 0.76%
44.22 0.45%
63.02 0.52%
65.71 + 0.26%
55.69 %
Table 10: Domain shift results on DomainNet-LT.
Sketch
Infograph
Painting
Quickdraw
Real
Clipart
Avg
ERM
39.22 0.24%
18.96 + 0.37%
34.71
+ 0.49%
10,70 = 0.06%
50.87 + 0.43%
44,83
0,25%
33.21%
Fical
41.01 + 0.61%
19.99 + 0.14%
36.42 + 0.45%
10.29 = 0.23%
55.63
0.56%
48.09 +: 0.72%
35.23%
19.06
36.32
11.38 0.29%
52.91
0.52%
CKT
40.78 0.35%
20.41 = 0.42%
39.01
+ 0.35%
11.41 = 0.17%
55.26 + 0.69%
50,00
0.83%
36.14%
MISLAS
41.34 0.39%
19.89 + 0.25%
39.85 + 0.56%
11.00 = 0.13%
$5.50 + 0. 0.36%
49.49
0.29%
16%18%
Remix
40.01 0.51%
19.17 0.46%
38.93
0.32%
11.39 0.20%
53.43
0.60%
47.94
0.71%
35.14%
RIDE
41.12 + 0.57%
19.23 + 0.34%
37.29 + 0.44%
11.21 = 0.35%
$4.00 1: 0.63%
48.72 1: 0.30%
35 25.266%
0.25%
39.11
IRM
34,65 0.75%
15.41
1.26%
0.40%
40.83
1.25%
28.195
GroupDRO
38.47 +: 0.27%
18.63 +: 0.07%
34.23 E 0.15%
10.26 :: 0.35%
50.80 1: 0.47%
42,85 1: 0.63%
32.54%
CORAL
+:
19.30 0.33%
35.15 0.70%
51.05 0.28%
0.38%
LISA
40.75 0.46%
18.47 1 0.14%
37.99
1. 0.19%
9,98 0.09%
54.33 1: 0.49%
48.42 0.42%
34.99%
MixStyle
40.99 0.60%
18.64 0.32%
35.86
+ 0.39%
11.03 + 0.1%%
50.26 +: 0.53%
45.49
0.84%
33.71%
DDG
0.58%
0.34%
35.61 0.63%
11.39 0.29%
50.93
0.41%
45.95
0.47%
33.94%
BODA
41.93 + 0.45%
20.65 + 0.58%
37.98 1 0.27%
11.02 = 0.23%
$5.22 1 0.65%
48.26
0.33%
35.85%
TALLY (ours)
42.66 = 0.32%
19.26 + 0.09%
40.49 = 0.34%
11.15 = 0.21%
$4.79 + 0.62%
50.36 0.41 %
36.45%
Table 12: Full Results of Domain Shifts on Real-world Data.
TerraInc
iWildCam
Macro F1
Acc
Macro F1
Acc
ERM
42.35
1.25%
54.81
+.0833%
32.0 1.5%
69.0 0.4%
Focal
43.54 0.81%
56.62 1.49%
33.2 1.2%
74.7
1.9%
LDAM
44.29 1.41%
57.22 0.92%
32.7 = 0.9%
75.2 = 2.0%
CRT
43.09 +: 0.79%
58.27 + 1.35%
32.5 = 1.8%
67.3 + 1.3%
MiSLAS
40.68 + 1.33%
52.96 2.58%
30.5 = 1.1%
59.8 2.8%
Remix
43.72 + 1.87%
58.40 2.57%
28.4 0.8%
65.8 1.6%
RIDE
44.03 1.42%
57.89
1.46%
32.8 0.5%
70.1 1.9%
PaCo
43.40 + 1.03%
57.35 + 0.89%
31.9 = 0.2%
72.6 + 2.5%
IRM
31.17 3.52%
49,27 5.22%
15.1 1: 4.9%
59.8 3.7%
GrouDDRO
42.22 + 0.87%
56.43 + 1.63%
23.9 = 2.1%
72.7 + 2.0%
CORAL
45.43 + 0.92%
58.10 1.38%
32.8 0.1%
73.3 4.3%
LISA
39,27 0.69%
54,92 1.04%
27.6 1.2%
64.9 2.2%
MixStyle
44.73 0.99%
57.55 2.05%
32.4 1.1%
74.9
2.7%
DDG
40.47 + 1.93%
53.61 1.71%
29.8 + 0.2%
69.7 +: 2.3%
BODA
44.47 + 0.84%
57.52 + 1.13%
32.9 0.3%
70.5 + 2.3%
TALLY (ours)
46.23 1 0.56%
59.89 = 1.32%
34.4 + 0.4 %
73.4 + 1.8%
18

---

Under review as a conference paper at ICLR 2023
Table 11: Hyperparameters for experiments on real-worke data.
Hyperparameters
Terralne
iWildCam
Learning Rate
3e-5
3c-5
Weight Decay
le-6
0
Batch Size
18
16
Epochs
15
Steps
1000
1000
Warm Start Epochs
7
7
in feat. estimation
0.8
0.8
class prototype mixup parameter de
0.5
0.5
domain prototype mixup parameter atd
0.5
0.5
F ADDITIONAL RESULTS OF ANALYSIS
In this section, we conduct two additional analysis to better understand how TALLY works. Then, we provide additional results of the analysis in the main paper.
F.1 FININGUNING ERM ON BALANCED DATASETS
In this section, we conducted an additional experiment to compare TALLY with a simpler approach training ERM imbalanced dataset and finetuning it on the corresponding balanced dataset. Here, we adopt three variants of balanced datasets for fine-tuning: (1) class-balanced dataset, where the number of examples the same across different classes; (2) Noomin--allacee dataset, where the number of examples is the same across different domains; (3) domain-class balanced dataset, where the number of examples
same across each domain-class group. The results are reported in Table [3] where the performance under subpopulation shift in OfficeHome-LT and DomainNet-LiI and the performance under domain shift in TerraInc and iWildCam are reported. According to the results, observe that TALLY still outperforms all variants of balanced finetuning, indicating its effectiveness in addressing multi-domain long-tailed learning problem by augmenting disentangled representations.
Table 13: Performance comparison between TALLY and balanced finetuning. FT means fine-tuning. Here, worst performance means the class-balanced worst-domain accuracy.
OfficeHome-LD
DomainNet-LD
Subpopulation shift
Avg.
Worst
Avg.
Worst
Class-ball bassnbalanced Domain-balanced FT
63.41 0.32%
58.41 0.64%
47.23 0.83%
27.65 + 0.05%
61.19 = 0.79%
54.29
0.55%
44.32 1 0.28%
24.37 + 0.13%
Domain-class wlaaa-lllsss---eeeeee FT
62.84 = 0.28%
57.88 1 0.82%
47.07 1 0.41%
27.16 + 0.22%
TALLY
67.00 + 0.47 %
60.45 + 0.09%
50.15 + 0.46%
29.55 + 0.19%
TerraIne
iWildCam
Domain shift
Macro F1
Acc
Macro F1
Acc
Class-balanced FT
44.36 + 0.45%
57.85 1.91%
32.5 + 0.3%
72.2 + 2.1%
Domain-balanced FT
43.92 + 0.77%
58.12 + 0.73%
32.9 + 0.8%
73.7 + 0.7%
comain-clas--aalneee FT
44.21 = 0.31%
59.13 + 1.24%
32.1 + 0.5%
71.9 + 1.8%
TALLY
46.23 + 0.56%
59.89 + 1.32%
34.4 0.4%
73.4 + 1.8%
F.2
COMPATIBILITY ANALYSIS OF TALLY
In this section, we analyze the compatibility of TALLY. Since TALLY only augmenting disentangled representations during the training stage,
easily incorporate TALLY with other long-tail tailed learning approaches. Specifically, we incorporate TALLY with PaCo and RIDE in this analysis and report the results in Table 14 We observe that incorporating TALLY significantly improves the
19

---

Under review as a conference paper at ICLR 2023
performance over the vanilla PaCo and RIDE. Nevertheless, the original TALLY already showed competitive performance compared with TALLY+PaCo and TALLY+RIDE.
Table 14: Compatibility Analysis of TALLY. Worst performance represents the class-balanced worst-domain accuracy.
OfficeHome-LT
DomainNet-LT
Subpopulation shift
Avg.
Worst
Avg.
Worst
Vanilla TALLY
67.00 + 0.47%
60.45 + 0.09%
50.15 + %
29.55 + 0.19%
63.27 = 0.54%
54.04 + 0.28%
47.51
0.83%
28.16 + 0.60%
RIDE
+TALLY
66.20 = 0.25%
60.38 0.37%
49.82 0.78%
28.72
10%
63.03 + 0.92%
56.15 + 0.71%
48.26
0.22%
27.36 + 0.52%
PaCo
+TALLY
67.30 0.22%
60.62 +
50.01 0.19%
29.70
0.29
TerraInc
iWildCam
Domain shift
Macro
Acc
Macro F1
Acc
Vanilla TALLY
46.23 + 0.56%
59.89
1.32%
34.4 0.4%
73.4
1.8%
44.03 + 1.42%
57.89 + 1.46%
32.8 0.5%
70.1 + 1.9%
RIDE
+TALLY
45.76 = 0.38%
59.41
1.24%
33.4 0.3%
73.7
1.4%
43.40 + 1.03%
57.35
0.89%
31.9
0.2%
72.6 2.5%
PaCo
+TALLY
46.46 + 0.21 %
59.19 + 0.82%
33.9 0.7%
72.6 2.1%
F.3 FULL RESULTS COMBINING BOIGETAILED LEARNING AND INVARIANT LEARNING APPROACHES
In Figure & and Table 15 report the full results of combining long-tailed learning and invariant
learning approaches, where Figure g shows the results of domain shifts and Table 15 listed all results with standard deviation over three seeds.
Relative Improvements over Vanilla Method
+UW
HCRT
+Focal
TALLY
+LDAM
1%
0%
CORAL
MixStyle
CORAL
MixStyle
Terrainc
iWildCam
Figure 8: Domain shift results (Macro FI) of the comparison between TALLY and variants two domain generalization approaches (CORAL, MixStyle), where replace the losses of them with class re-weighting or re-sampling ones.
F.4 FULL RESULTS OF THE EFFECT OF PROTOTYPES
We report the full results of the prototype analysis in Table 16
20

---

Under review as a conference paper at ICLR 2023
Table 15: Full results of the comparison between TALLY and variants of two representative domain generalization approaches (CORAL, MixStyle), Worst means bass-bbllanced worst domain accuracy subpopulation shift.
OfficeHome-LT
DomainNet-LT
Subpopulation shift
Avg.
Worst
Avg.
Worst
59.10 + 0.20%
53.53 + 0.60%
43.92 + 0.36%
24.50 + 0.68%
+UW
60.91 + 1.40%
55.32 + 2.14%
46.26 0.48%
26.74 + 0.15%
CORAL
+Focal
61.72 + 0.93%
55.26 1.60%
46.39
0.31%
27.13 + 0.95%
+LDAM
61.09 + 1.52%
56.22 + 1.45%
46.69 + 0.32%
26.58 + 0.62%
+CRT
62.61 +: 0.68%
56.41 +: 0.51%
47.68 + 0.53%
27.60 + 0.37%
62.26 + 0.22%
55.19 + 1.10%
43.59 + 0.57%
22.65 + 0.22%
+UW
63.02 + 0.69%
57.50 + 2.11%
46.30 + 0.35%
25.53 + 0.48%
MixStyle
+Focal
63.69 + 0.32%
57.56 1 1.40%
46.18 1 0.36%
26.51 1 0.88%
+LDAM
62.77 0.78%
57.50
1.50%
46.32 0.35%
25.31 0.48%
+CRT
63.20 1 0.29%
55.77 1 0.63%
48.11 + 0.29%
27.95 1 0.72%
TALLY
67.00 + 0.47
60.45 + 0.09%
50.15 0.46%
29.55
0.19%
Terrllne
iWildCam
Domain shift
Macro
Acc
Macro F1
Acc
45.43 0.92%
58.10 1.38%
32.8 0.1%
73.3 4.3%
+UW
45.67 1 0.73%
59.73 1: 1.53%
32.6 1 0.7%
73.0 1 2.9%
CORAL
+Focal
45.27 + 0.88%
59.54 + 1.71%
32.9 + 0.8%
74,9 + 3.2%
+LDAM
45.45 + 1.13%
59.50 + 2.04%
33.4 0.5%
74.3 = 1.6%
+CRT
45.72 + 1.59%
59.47 2.32%
33.2 0.6%
73.2
5.3%
44.73 + 0.99%
$7.55 + 2.05%
32.4 1.1%
74.9 + 2.7%
+UW
45.25 0.67%
58.65 1.39%
32.7 0.9%
72.0
1.3%
MixStyle
+Focal
45.04 + 1.22%
58.15 + 2.18%
32.8 1 0.5%
74.0 + 2.8%
+LDAM
44.74 + 1.03%
58.66 + 1.76%
32.7 + 1.3%
77.1 + 3.0%
+CRT
44.97 + 1.83%
58.38 3.11%
33.1 + 0.7%
71.7 + 3.3%
TALLY
46.23 + 0.56%
59.89 + 1.32%
34.4 1 0.4%
73.4 + 1.8%
Table 16: Full results of the analysis of prototype-guided invariant learning. C Only and D Only represent only using class prototype representation or class-agnostic domain factors, respectively. Worst means class-balanced worst domain accuracy in subpopulation shift.
OfficeHomeeLT
DomainNet-LT
Subpopulation shift
Avg.
Worst
Avg.
Worst
None
66.19 + 0.34%
58.72 + 0.89%
48.78
0.43%
27.21 + 0.12%
C Only
66.54 + 0. 14%
59.55 + 0.55%
49.45
0.27%
27.50 + 0.38%
D Only
66.23 1 0.24%
58.98 1 1.18%
49.09
0.54%
27.50
1 0.46%
TALLY
67.00 1 0.47%
60.45 + 0.09%
50.15 + 0.46%
29.55 1 0.19%
Terrainc
iWildCam
Domain shift
Macro
Acc
Macro F1
Acc
None
45,30 + 0.64%
58.02 + 1.26%
32,9 0.8%
72.6 1: 2.9%
C Only
45.86 1 0.81%
59.06 + 1.46%
33.9 0.5%
74.6 1: 2.8%
Only
45.47 1 0.57%
58.22 +: 1.84%
33.2 1.1%
72.3 1 1.5%
TALLY
46.23 + 0.56%
59.89 = 1.32%
34.4 + 0.4%
73.4 + 1.8%
F.5 FULL RESULTS THE ANALYSIS OF SAMPLING STRATEGIES
In Table 4
report the full results of the analysis of different sampling strategies.
21

---

Under review as a conference paper at ICLR 2023
Table 17:
Full results of comparison between sampling strategies. Worst means class-balanced worst domain accuracy in subpopulation shift.
OfficeHome-IIT
DomainNet-L.T
Subpopulation shift
Avg.
Worst
Avg.
Worst
Balanced Sampling
65.03 1 0.91%
58.33 0.24%
49.35 0.21%
27.97 0.25%
TALLY(Selcctive))
67.00 0.47%
60.45
0.09%
50.15
0.46%
29.55
0.19%
TerraInc
iWildCam
Domain shift
Macro F1
Acc
Macro F1
Acc
Balanced Sampling
44.79 + 0.62%
57.78 = 0.36%
33.1 0.4%
71.6 + 1.8%
TALLL(Selccctvve)
46.23 + 0.56%
59.89 1.32%
34.4 + 0.4 %
73.4 + 1.8%
G RESULTS ON STANDARD DOMAIN GENERALIZATION BENCHMARKS
In this section, we present the additional comparison on standard domain generalization benchmarks. Notice that the data distributions in these standard benchmarks not long-tailed, which is thus not our focus in this paper. The goal is to compare our approach with other domain generalization methods. In Table 21 present results four standard benchmarks: VLCS, PACS, OfficeHome, DomainNet, respectively. Results for all algorithms except TALLY are directly copied from Gulrajani Lopez-Pazz (202T) and Yang et al. (2022). Table 22 we summarize all results and show the comparison between different approaches. According to the results, TALLY can achieve comparable performance compared with state-of-the-an domain generalization approaches.
Table 18: Comparison on the standard VLCS benchmark.
Caltech101
LabelMe
SUN09
VOC2007
Avg
ERM
97.7 0.4
64.3 0.9
73.4
74.6 1.3
77.5
IRM
98.6
64.9
0.6
0.9
78.5
GroupDRO
0.3
63.4
0.
7.7
76.7
Mixup
9.6
64.8 1.0
0.5
7..3
77.4
MLDG
97.4
6.7
1.
1.0
77.2
CORAL
0.1
6..1
0.3
1.2
78.8
MMD
0.1
6..0
0.2
7.3
77.5
DANN
99.0 0.3
1.
0.3
0.6
78.6
CDANN
9.3
6.21
7.87
1.5
77.5
MTL
9.
6.33
7.7
7..3
77.2
SagNet
9.4
6.5
1.
0.5
77.8
ARM
9.2
6.76
1.2
7.6
77.6
VREx
98.4 0.3
6..41:44
7.41
1.3
78.3
RSC
0.1
6.75
1.2
75.6 0.8
77.1
BODA
08.1 0.3
0.45
74.3 0.3
78.0 0.6
78.5
TALLY (ours)
0.5
67.2 1: 1.1
73.8
79.2 1 0.9
78.8
22

---

Under review as a conference paper at ICLR 2023
Table 19: Comparison on the standard PACS benchmark.
Art painting
Cartoon
Photo
Sketch
Avg
ERM
84.7
80.8
97.2 +
79.3 1.0
85.5
IRM
84.8 + 1.3
76.4 + 1.1
96.7 + 0.6
76.1
+ 1.0
83.5
GroupDRO
83.5 0.9
79.1 0.6
96.7 = 0.3
78.3
: 2.0
84,4
Mixup
86.1 1 0.5
78.9 0.8
97.6 0.1
75.8 1: 1.8
84.6
MLDG
85.5
1.7
0.3
76.6
84.9
CORAL
88.3
0.5
97.5
78.8 - 1.3
86.2
MMD
8.4
7.94
96.6
76.5
+ 0.5
84,6
DANN
8.84
7.8
73.5 0.4
83.6
CDANN
84
0.9
96.8 0.3
73.5 1 0.6
82.6
MTL
0.8
7..1
96.4 0.8
77.3 4 1.8
84.6
SagNet
87.4 1.0
8..
97.1 + 0.1
80.0 + 0.4
86.3
ARM
86.8
7.5
9.3
79.3 + 1.2
85.1
VREx
86.0
0.6
0.5
77.7 + 1.7
84.9
RSC
0.8
97.6 1.8
78.2
1.2
BODA
88.2 9
81.7 0.3
0.2
80.2 + 0.3
86.9
TALLY (ours)
89.5 0.8
81.2 + 0.7
97.0 +
81.7 0.9
87.4
Table 20: Comparison on the standard OfficeHome benchmark.
Art
Clipart
Product
Real
Avg
ERM
0.7
52.4
0.3
75.8
0.1
+
66.5
IRM
4
52.2 +: 1.6
72.1
2.9
74.0 + 2.5
64.3
GroupDRO
6.77
52.7
12.7:00 1.0
75.0
76.0 0.7
66.0
Mixup
6.84
0.6
76.9
78.3
68.1
MLDG
6.9
0.6
1.2
7.4
66.8
CORAL
6.4
5.54
0.1
78.4
68.7
MMD
6.2
0.3
0.1
77.4
66.3
DANN
1.39
0.3
0.7
76.9
65.9
CDANN
1.4
2.44
7.94
7.8
65.8
MTL
6.7
52.4
1.49
76.8 1.44
66.4
SagNet
6.2
54.8
1.
783
68.1
ARM
58.9
0.5
7.11
75.2
64.8
VREx
0.9
5.90
0.1
7
66.4
RSC
60.7
0.3
1.1
75.1
65.5
BODA
+
55.4 0.3
77.1 0.1
+
69.3
TALLY (ours)
64.2 0.5
55.1 0.8
78.0
69.1
Table 21: Comparison on the standard DomainNet benchmark.
Sketch
Infograph
Painting
Quickdraw
Real
Clipart
Avg
ERM
+ 0.4
18.8
0.3
46.7 0.3
12.2
58.1
40,9
IRM
42.3 + 3.1
15.0 + 1.5
38.3 + 4.3
10.9 1 0.5
48.2 + 5.2
48.5 +: 2.8
33.9
GrouDDRO
40.1 0.6
33.8 + 0.5
9.3 0.3
51.6 0.4
4
33.3
Mixup
4
0.5
44.3
12.5
5
0.3
39.2
MIDG
50.2
0.3
0.7
0.3
59.6
59.1
CORAL
50.1 0.6
0.2
46.6 0.3
1..4
59.8
0.2
5..2
41.5
MMD
28.9
4..0
26.8
16.8110 11.3
8.7
32.7
42.1+13.3
23.4
DANN
46.8 0.6
0.1
4
1.
55.5 0.4
53.1+
38.3
CDANN
45.9
0.3
43.7
1.7
56.2
0.4
38.3
MTL
49.2
0.4
46.0
12.5 0.1
5.3
57.9
40.6
SagNet
48.8 0.2
0.2
45.3
12.7
58.1
57.7
40.3
ARM
43.5 0.4
16.3
40.9 10.9111
9.1
5.44
49.7
35.5
VREx
1
1..0
35.8
10.9 0.9
49.6
47.3
RSC
47.8 0.9
1.3
44.4 1.6
12.2
55.7 1 0.7
55.0 1.2
38,9
BODA
51.3 1 0.3
20.5 1 0.7
48.0
1 0.1
13.8
0.6
60.6 1 0.4
62.1 1. 0.4
42.7
TALLY (ours)
50.5 1 0.2
19.7 1 0.1
47.7 . 0.6
14.1 J.
0.3
60.0 . 0.2
60.1 J. 0.5
42.0
23

---

Under review as a conference paper at ICLR 2023
Table 22: Domain shift results over all four benchmarks.
VLCCS
PACS
OfficeHome
DomainNet
Avg
ERM
77.5 + 0.4
85.5 + 0.2
66.5 = 0.3
40.9 + 0.1
67.6
IRM
78.5 1.510.5 0.5
83.5 0.8
=.2
33,9 1: 2.8
65.1
GroupDRO
0.6
84.4 0.8
66.0
65.1
Mixup
0.6
14.6-1.66
68.1
39.2 0.1
67.3
MLDG
77.2 0.4
84.9 1.0
41.2
67.5
CORAL
78.8 0.6
86.2
68.7 0.3
4..
68.8
MMD
77.3
8.
66.3
9.5
63.0
DANN
0.4
83.6 0.4
65.9 =
38.3 0.1
CDANN
0.1
82.6 0.9
65.8 = 1.3
38.3
66.3
MTL.
7
84.6 0.5
66.4 = 0.5
4.1
67.2
SagNet
77.8
86.3 0.2
68.1 = 0.1
0.1
68.1
ARM
77.6 0.3
85.1
64.8
0.2
65.8
VREx
84.9 0.2
66.4
33.6
65.8
RSC
77.1 0.5
85.2 0,9
65.5
38.9 0.5
66.7
BODA
78.5 - 0.3
86.9 + 0.4
69.3 0.1
42.7 + 0.1
69.4
TALLY (ours)
78.8 0.4
87.4 1 0.2
69.1 - 0.4
42.0 1 0.1
69.3
24


Reviews:
[{"review_id": "vj8uJAWU23", "reviewer": "Reviewer_WENZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the multi-domain long-tailed recognition problem.         \n(1) An augmentation strategy is proposed for this goal:     \nI. For each time, uniformly sample an image across all classes and uniformly sample an image across all domains.     \nII. Each image representation is disentangled by class-relevant features and domain-relevant features.     \nIII. Combining the semantic representation of one with the domain-related nuisance factors of the other to form the augmented representation.    \n\n(2) Experiments are conducted on several long-tailed datasets with subpopulation shift and domain shift. Clear improvements are observed.\n", "summary_of_the_review": "This paper proposes an augmentation strategy to address multi-domain long-tailed recognition. \nIf more advanced long-tailed methods, like PaCo and RIDE, are explored (as comparison baselines or whether the proposed method is complementary to them), the experimental results will be more convincing.", "strength_and_weaknesses": "Strength:    \n(1) The paper writes clear and is easy to follow.    \n(2) The method is simple and elegant.    \n(3) Experiments on different benchmarks show good improvements over baselines.    \n(4) Sound ablations are conducted.  \n\nWeakness:     \n(1) Some long-tailed methods are selected as baselines, like CRT, and MiSLAS. However, state-of-the-art long-tailed methods can be transferred to this multi-domain long-tailed recognition task, like PaCo [1], and RIDE [2].    \n\n(2) Would the representation augmentation module be removed at inference?    \n\n(3)  Regular augmentations like RandAug can also improve performance. Can the proposed TALLY work with them?\n\n(4) Can the proposed TALLY  work with long-tailed methods like PaCo [1], and RIDE[2]?\n\n\n[1] Parametric Contrastive Learning. ICCV 2021.\n[2] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts. ICLR 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "hVIM7-wXEc", "reviewer": "Reviewer_nEAQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-domain long-tailed recognition method to simultaneously address the two types of distribution shift: subpopulation shift and domain shift. The basic idea is sample augmentation where new samples are generated by representation disentanglement for tailed class. The experimental results on synthesis and real-world multi-domain long-tailed data show its effectiveness.", "summary_of_the_review": "the proposed method is effective and interesting, but some details are missed.", "strength_and_weaknesses": "Strengths: \n--the paper focus on a general problem: the multi-domain long-tailed learning.\n\n--The motivation is clear. The solution for multi-domain long-tailed learning is data augmentation by representation disentanglement, where any sample is synthesized with class-relevant semantic information and domain-associated nuisances. \n\n--This paper is well written.\n\nWeaknesses:\n\n--The novelty is limited. Representation disentangle and date augmentation are the common approaches for long-tailed recognition, and no novel approach is proposed for the specific issue of multi-domain long-tailed recognition.\n\n--Some details are questioned. For example, why the representation of each examples can be decoupled into semantic information and nuisances, and the domain information is contained in nuisances? \n\n--The construction of prototype is too simple to ineffective. I think the updating of prototypes is needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8tE6OlvvzP_", "reviewer": "Reviewer_ijLa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a balanced sampling data augmentation technique, for addressing the multi-domain and long-tailed learning problems in which the number of samples in different classes is imbalanced and potential domain shift. First, the class-specific and domain-specific representations are decoupled by instance-normalization (Ulyanov et al., 2016),  thus new domain-class pairs can be formulated by a swap operation similar to style transfer (Huang and Belongie, 2017). Moreover, prototype vectors are computed and mixup techniques are used to make the class and domain representations more stable and diverse. The proposed approach is compared against single-domain long-tailed learning methods and domain adaptation or generalization methods and evaluated under two settings. (1) when all classes in test data are balanced (2) when test data is from an unseen domain. Performance improvements are demonstrated especially on synthetic datasets.  The ablation study demonstrated that the proposed approach works better on small-size classes and can learn more domain-invariant features.  ", "summary_of_the_review": "This paper presents an interesting combination of ideas for the multi-domain long-tailed setting and achieves improved results over many baselines. However, the sampling scheme is unclear, and I am not sure whether the evaluation protocol is fair. I’ll vote for borderline reject. I will re-evaluate based on the authors' responses.\n\n######\nAfter rebuttal: Thank you for the response and additional experiments during the rebuttal. Most of my questions have been clarified. I have updated my score to reflect this. Two additional comments: \n\n- Selective balanced sampling: it seems the main difference to balanced sampling is whether the sampling of class and domain representations are done separately and in proportion to the sample size. The proposed method can avoid oversampling from the minority class, but the current text description and the transition probabilities make things overcomplicated.  \n\n- Domain adaptation baselines: Fig 5, CORAL should be an unsupervised domain adaptation method since it uses unlabeled data from the target domain. This should be clarified - as the proposed method does not even need to use any data or labels from the target domain. \n ", "strength_and_weaknesses": "The paper studies the long-tailed learning problem in the multi-domain setting. Unlike traditional methods that can only upsample or downsample using data from the same domain, the proposed approach is able to borrow representations cross-domain, so the model can see data of more variety with data augmentation. To achieve this, this paper combines a few effective techniques, including instance normalization, moment matching, prototype, and mixup. Experiment results show improvements against many baseline methods on both synthetic and real-world datasets. \n\nThe difference between the selective balanced sampling strategy and balanced sampling is unclear. I read Section 3.2 and “Analysis of sampling strategies” but still confusing. Figure 3 does not do a good job on explain the difference either.  What is the difference between y, d ~ Uniform(C, D) vs y~ Uniform(C) and d ~Uniform(D)? Can you explain more about the transition probabilities in Figure 3?\n\nFor the evaluation protocol, it is unclear what exactly is held-out? It seems in the long-tailed learning setting, all the classes and domains (except for one) are seen. It is very important to make sure there is no information leakage from training to test sets, and that the comparisons are fair. \n\nThe paper considered a wide range of baselines, including both domain adaptation and domain generalization methods. It would be helpful to explain the underlying assumptions of different methods, such as whether labels or data from the target domain is needed. Is equation (4) possible for a new unseen domain?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8sNB2lfDas", "reviewer": "Reviewer_yuqM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on multi-domain long-tail classification, where each domain has its own long-tailed distribution. To solve this problem, the authors propose a novel balanced augment method TALLY, which separates and reassembles class-relevant semantic information and domain-associated nuisance information. Specifically, first, the representation of each instance is divided into semantic representation and nuisances. Then the semantic information and domain-associated factors are separately enhanced using argument techniques to improve stability and reduce noise. Finally, the semantic information and nuisances are reassembled to generate new instances for training. Several experiments on four synthetic multi-domain long-tailed datasets and two real-world datasets indicate that the proposed methods improve the overall performance for subpopulation shift and domain shift compared to two types of baselines.\n", "summary_of_the_review": "This paper identifies a critical problem, multi-domain long-tail learning, and proposes a novel TALLY method to address this problem with reasonable motivation and some novelty. The experimental results on synthetic and real-world datasets illustrate the effectiveness of the proposed method.\n", "strength_and_weaknesses": "This paper focuses on a less-studied problem and defines the problem formally. With an example on the real-world dataset and some discussion, the authors illustrate why it is not trivial to explore the problem. Unlike single-domain long-tail classification, the authors figure out that multi-domains exist with their own long-tail distributions, and previous approaches may exacerbate class imbalance in this scenario.\nFor the implementation details of the proposed method, the authors give good motivation. For example, in proposing selective balanced sampling, the authors illustrate by figure the drawbacks of balanced sampling from all domain-class groups and how selective balanced sampling solves this problem.\nThe proposed approach is relatively novel. Although some of the techniques are derived from existing work, such as the representation disentanglement and reassembly section, the authors provide some new technical insights.\nThe experimental results broadly validate the authors' claim that TALLY is more effective than the existing single-domain imbalanced classification and domain-invariant learning methods under subpopulation shift and domain shift tasks. In addition, sufficient experiments are carried out to demonstrate the importance of each component in TALLY.\nHowever, my main concerns are some details in the model and experimental part of this paper. Here are the detailed comments:\n[Presentation] The paper's writing and statements are easy to follow. However, I have some minor suggestions. (1) Since one of the main contributions of the paper is to formalize the multi-domain long-tailed learning problem, clearer notation should be given to help future works better understand the problem and better use these notations. For example, the meanings of some symbols are explained but remain complex, considering the different superscripts and subscripts. (2) Also, some definitions should be better given in more obvious places, such as class-agnostic statistics u_d^{(0)} and v_d^{(0)}.\nExperimental results demonstrate the effectiveness of the proposed method, but some theoretical analysis can be given to shed light on the algorithm designs of TALLY and inspire future work.\n[Experiments] On domain generalization benchmarks, overall performance is used to measure the performance of subpopulation shift and domain shift; on naturally imbalanced multi-domain data, macro F1 score and average accuracy are used. However, I am not sure of the reason for using these measures. Considering that we are exploring the multi-domain long-tailed problem, should some evaluation metrics that measure tail performance be considered?\n[Reproducibility] Although the critical parts of the method are carefully clarified with the pseudo algorithm, the code of this paper is not given, which limits the reproducibility of this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject