Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations
Authors: Yuli Slavutsky
Yuval Benjamini
Venue: NeurIPS 2024 poster (2024)
Keywords: Zero-Shot Learning
Distribution Shift
Out of Distribution Generalization
Robust Representation Learning
Primary area: evaluation
TL;DR: This work tackles the challenge of learning data representations robust to class distribution shifts in zero-shot learning, by constructing synthetic data environments and harnessing out-of-distribution generalization techniques.

Abstract: Zero-shot learning methods typically assume that the new, unseen classes encountered during deployment come from the same distribution as the the classes in the training set. However, real-world scenarios often involve class distribution shifts (e.g., in age or gender for person identification), posing challenges for zero-shot classifiers that rely on learned representations from training classes. In this work, we propose and analyze a model that assumes that the attribute responsible for the shift is unknown in advance. We show that in this setting, standard training may lead to non-robust representations. To mitigate this, we develop an algorithm for learning robust representations in which (a) synthetic data environments are constructed via hierarchical sampling, and (b) environment balancing penalization, inspired by out-of-distribution problems, is applied. We show that our algorithm improves generalization to diverse class distributions in both simulations and experiments on real-world datasets.

Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations
Yuli Slavutsky
Yuval Benjamini
Department of Statistics and Data Science
Department of Statistics and Data Science
The Hebrew University of Jerusalem
The Hebrew University of Jerusalem
Jerusalem, Israel yuli. slawusskymmaillll...........aa.. http:sakymmmma............ ac..llc.ll
Jerusalem, Israel
yuval. benj amin @mail. huji. huji.ac. ac
Abstract
Zero-shot learning methods typically assume that the new, unseen classes encoun- tered during deployment come from the same distribution the the classes in the training set. However, real-world scenarios often involve class distribution shifts (e.g., in age gender for person identification), posing challenges for zero-shot classifiers that rely on learned representations from training classes. In this work, we propose and analyze model that assumes that the attribute responsible for the shift is unknown in advance. We show that in this setting, standard training may lead non-robust representations. To mitigate this, we develop an algo- rithm for learning robust representations in which (a) synthetic data environments are constructed via hierarchical sampling, and (b) environment balancing penal- ization, inspired by out-of distriibbttio problems, is applied. We show that our algorithm improves generalization diverse class distributions in both simulations and experiments on real-worll datasets.
Introduction
Zero-shot learning systems [14 27 are designed to classify instances of new, previously unseen classes at deployment, a scenario known open-world classification. These systems are widely applied in extreme multi-class applications, such as face or voice recognition [19] for matching observations of the same individual, and more generally, for learning data representations (2].
Class distribution shifts typically refer to changes in the prevalence of fixed set of classes between training and testing. In zero-shot learning, however, different challenge arises: the appearance of entirely new classes at test time. This raises a critical question are these new classes drawn from the same distribution the training classes? Most zero-shot methods assume that they are, an assumption that not only shapes the design test sets [57] [6] but also plays an explicit role in assessing the generalization capabilities of zero-shot classifiers
559 48
In practice, training classes are often chosen based on convenience and accessibility during data collection. Even when data is carefully collected, the distribution of classes may shift over time, leading to a different distribution. For instance, this could occur when a face recognition system is deployed in building located in neighborhood undergoing demographic changes.
Class distribution shifts pose significant challenges to zero-shot classifiers, since they rely on learning data representations from the training classes to distinguish new, unseen ones. Typically, these classifiers are trained by minimizing the loss on the training set to effectively separate the training classes. However, this approach may result in poor performance when confronted with data from distributions that differ significantly from the class distribution in the training data. Notably, in person re-identification, this concern gained attention from fairesss perspective with respect to gender
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

15 23 age 15 B3 50 and racial [39 [4] bias. In all these studies the variable (i.e., gender, age, race) expected to cause the distribution shift was known advance.
In contrast, in real-world scenarios, the attribute responsible for a future distribution shift is usually unknown during training. In such cases, existing approaches based collecting balanced datasets re-weighting training examples [54] 41 53 are inapplicable. Furthermore, while class distribution shifts have been extensively studied in the standard setting of supervised learning (see Appendix (A) previous research assumed a closed-world setting that does not account for new classes at test time. Instead, it only addressed changes in the prevalence of fixed classes between training and testing. Consequently, class distribution shifts in zero-shot learning remain largely unaddressed.
In this paper we first address these limitations by examining the effects of class distribution shifts on constrastive zero-shot learning, by proposing and analyzing a parametric model (33) We identify conditions where minimizing loss in this model leads to representations that perform poorly when distribution shift has occurred.
We then use the insights gained from this model to present our second contribution ($4 an algorithm for learning representations that are robust against class distribution shifts zero-shot classifica- tion. In our proposed approach, artificial data environments with diverse attribute distributions are constructed using hierarchical subsampling, and an environment balancing criterion inspired by out-of- distiiiiitiiitiinnn (OOD) methods is applied. We assess method's effectiveness in both simulations and experiments on real-world datasets, demonstrating its enhanced robustness in $
1.1 Problem Setup
Let (2, c) be labeled set of training data points 2 and classes c C, such that a is the class of 2;.
In this work, we focus verification algorithms that enable open-world classification by determining whether two data points Zij = (zi, 2j) belong to the same class. For instance, in person re- identification the task is to identify whether two data points (e.g., face images voice recordings) belong to the same person. We denote this by Yıy. where Yij = if c, = cj and Yij = 0 otherwise. When the identity of each data point in the pair is not important, single index is used for simplicity, namely (Ix, Jle)
We assume that each class e is characterized by some attribute A. We further assume that the training classes are sampled from Pc(cc, the test classes are sampled according to Qc(c), and the two distributions differ solely due to shift in the distribution of an attribute A:
Pc(c) / P(la) da, Qc(c)= f ((a) da.
(1)
Importantly, assume that the attribute A is unknown, and that both during training and testing, data points 2 E 2 for each class are sampled according to (z]c() For instance, revisit the person identification example where each person is a class. If the attribute is binary (e.g., aj is blond and 02 is dark-haired), then P(C)A = a1) represents the distribution of people with blond hair, and P(CIA a2) of individuals with other hair colors. The training classes might be predominantly sampled from the blond population P(A = 01) = Pr. = 0.8, while test classes are predominantly sampled from Q(4 a1) Pre 0.1.
We focus on verification techniques based on deep metric learning methods (for surveys see 43 St] such contrastive-leaming [17. Siamese neural networks 24), triplet networks (00) and other
more recent variations 35 56 58). These methods learn a representation function that maps data
points to representation space Z + E, so that examples from the same class are close (in
predefined distance function d(, -), while those from different classes are farther apart.
We assume that g is a neural network trained by optimizing a meee--etric-learning loss, such as the contrastive loss [17]
8 (Zt, Zy Migi dg) dl (z,, 2z)) + (1 - yis) max (0, m dg 21)²²
(2)
where m 2 predefined margin, and d, (zs, 2,) = d(g(z;), g(2))) is the distance between the representations of the datapoints 24, 2j. In theoretical analysis, we examine the no-hinge

---

contrastive loss (see Appendix B for additional details):
(21, 23. Biy: dy) == di (2,, 23) + (1 (1-9:t) Viy) (m dg (21: 2,))
To evaluate the class separation capability of a representation, we treat the distances between representations, dg(2;, 2n), as classification scores. Following common practice in the field (e.g.. 47 22 we use the area under the receiver operating characteristic curve (AUC) to evaluate the representation, enabling threshold aggossicc assessment:
AUC(g) = (z,,(zi, 2,) < dg (Zu, 2v) Mij 1, yuv 0).
Our goal is to learn a representation g that is robust to class attribute shifts. That is, such that for an unknown shifted distribution Q A. the performance Egq, [AUC(g)] does not significantly deteriorate compared to the performance obtained on the training distribution PA.
2 Background on Environment Balancing Methods in OOD Generalization
The field of OOD generalization gained attention since the work of Peters et al. 36 37,, which deals with closed-world classification where training data is gathered from multiple environments Emain- In this setting it is assumed that in each environment e E Emain examples share the same joint distribution Pê,z (c, z), but across environments the joint distribution changes, often due to variations in (22 (2(c). A well-known example [1] involving the classification of images of cows and camels demonstrates how an algorithm relying on background cues during training (e-g.. cows in green pastures, camels in deserts) performs poorly on new images of cows with sandy backgrounds.
Several approaches that rely on access to diverse training environments were proposed to identify stable relations between the data point and its class c. Examples of such stable relations include choosing causal variables using statistical tests [42) leveraging conditional independence induced by the common causal mechanism [9] and using multi-nnvironment calibration as a surrogate for OOD performance [52]
Most relevant to our work are methods that aim to balance the loss over multiple environments. These methods consider representation g go that is neural network parameterized by 0 trained to optimize an objective of the form
min (ge) XR(ges Ewain)
where (ge) is the empirical loss obtained on the environment e, Erain is the set of all training environments, R is regularization term designed balance performance over multiple environments, and À is a regularization factor balancing the tradeoff between the empirical risk minimization (ERM) term and the balance penalty. Below, we describe three such methods, which we will refer later in the paper.
Invariant risk minimization (IRM) presented in (. aims to find data representations ge such that the optimal classifier w top of the data representation w go is shared across all environments. Therefore, the authors proposed minimizing the sum of environment losses l" (w (ooge) go) over all training environments such that w E arg min ["(w' go) for all E Euain However, since this objective is too difficult to optimize, relaxed version was also proposed, taking the form of Equation 5 with a
penalty that measures how close w is to minimizing 8 (woog)) Ri (ge) = || \ (w ((o-go)) go)
Note that for loss functions for which optimal classifiers can be expressed conditional expectations, the original IRM objective is equivalent to the requirement that for all environments e, E Eusin- Erg, (<lg()) = h] = Epg' [clg(z) = h], where Pú. 2 and Pe are the joint data distributions in the respective environments.
Calibration Loss Over Environments (CLOvE) presented in [52] leverages the equivalence above to establish a link between multi-environment calibration and invariance for binary predictors (c E {0, 1}). The proposed regularizer is based on the maximun mean calibration error (MMCE) [26] Let s 2 + [0, 1] be classification score function applied on the representation g. and
3

---

s; max s g(2), 1 1 so g(;)) be the confidence on the i-th data point. Denote the correctness as b; [[[c - s,1 < 11 and let R x R + be a universal kemel. Let Z" denote the training data in the environment e. The authors proposed using the MMCE as penalty in an objective that takes the form of Equation with RMMCE (8, go)
- Ea, -8: 8i) (b3 82) K (81,8)).
m
Variance Risk Extrapolation (VarREx) proposed by Krueger et al. 25 based on the observation that reducing differences in loss (risk) across training domains can reduce a model's sensitivity to a wide range of distribution shifts. The authors found that using the variance of losses regularizer is stabler and more effective compared to other penalties. Therefore. they propose the following regularization term for training environments: RVaaRTs (go; Emiin) Var (f*1 (go),.... e (go)).
While simple and intuitive, this approach assumes that losses across different environments accurately reflect the classifier's 's performance. However, as discussed in $ this is often not true for deep metric learning losses, where significant changes in loss may correspond to only minor variations in performance.
3 Parametric Model of Class Distribution Shifts in Zero-Shot Learning
In this section, we introduce a parametric model of class distribution shifts. Our model shows that in zero-shot learning, even if the conditional distribution of data given the class P(zlc) remains the same between training and testing, a shift in the class distribution from P(c) to Q(c) can cause poor performance on newly encountered classes sampled from the shifted distribution ((c).
Assume that for all classes, the data points E IRd are sampled from z/e;; ~ N (c,, Ez), where Σ, V/= - Ia. and Ia the identity matrix. Let the attribute indicate the type of class c, with two possible types: a1 and 0g. Assume that the classes c, are drawn according to G ~ N (0, Ea) for a E (a1,02). a. Finally, assume that in training, ai is the majority type with P(a1) = Prr > 0.5, whereas at test time, (2 is the majority type with ((a1) = Pie << 0.5.
We construct the model such that differences between the training class distribution P(c) and the test distribution Q(c) stem solely from a shift in the mixing probabilities of an unknown attribute A (see Equation D. Therefore, we define M.
diagonal matrix with replicates of three distinct values on its diagonal: Vo, y+ V", Let 0 < リー < Vs S 1/ < vt. Then, in the coordinates corresponding to Po and v data points from different classes are well separated, whereas in the coordinates corresponding to v they are not. Assume the coordinates corresponding to 1/D are shared by both types, but w and v are swapped:
do
Σx, = diag (o ....,. vo. vt
vt D ,...,
Eng diag (Po....,
vt
An illustration with one replicate of each value is shown in Figure [
The following proposition shows that if the number of dimensions dN that allow good separation for classes of type a1 is relatively similar to the number of dimensions d2 that enable good separation for classes of type 02, specifically if by (p, V2, Mo, U1, 22) 4,+2
d2+2 < h,, (p, Uz, Mo, M1; M2), then the
optimal solution for the training distribution prioritizes the components (features) corresponding to - for classes of type a1- Thus, the prioritized features allow good separation for classes from the majority type in training, but offer poor separation for the shifted test distribution, where most classes are of type 02. Note that if d2 is large, when combined, the corresponding components may still provide reasonable separation. We define h and h, in Equation 35 and provide the proof of Proposition
D in Appendix B.2
Proposition 1. Consider a weight representation g(2) Wz, where W E dxdiis diagonal matrix, and the squared Euclidean distance d, (Ze, 2g) = I|W (z+ 21) Let W diag(w*)
arg miny [((r)rdg))] Denote w1² J/ Edi Ek-dot1 wi and wy² th Ed Ek ut² Then, for
all p > 1/2 and V2 Vo, v1, 12, d1, d2 satisfying hy (p, Ue, vo, 22) < d.t2 holds that d di w.
dr
< hu (p, 22, 2o, U1, v2) it

---

0500
X0X0000
1.0
=1
5.0% 5.6. -0.4
de
-200
1.0
10,
2.
Illustration of the parametric model. Classes of each type are best sep- arated along specific axes: classes of type a1 along the red axis (z(1) and classes of type along the green axis (z(2)), On
Figure 1:
6.5
0.
axis 20) both types can be separated but not as effectively optimal axes.
Figure 2: Optimal weights. Top row: dg is fixed, d1 and d2 vary. Middle and bottom rows: da, d1, d2 fixed. Middle: "ofu- varies. Bottom: "0/u varies.
their respective
Note that the conditions outlined in Proposition I are sufficient but not necessary. Accordingly, in Appendix [.3 we provide the complete analytical solution for w" that minimizes the expected loss
((..... : dg) for the weight representation (2) 2, using the squared Euclidean distance. According Proposition| 1 larger d2 values favor for better aggregated separation. Increasing vo/vt leads to increased differences between wi² and wz?, and vice versa for 1o/2".
These relationships in the optimal solution The top row shows that when dj = d2 10 dimensions favoring classes of type aj are prioritized for p 0.5, while those favoring type d2 are prioritized for p 0.5. When d1 10 while d2 dimensions favoring type d1 are prioritized for all values of p. and vice versa when d2 is significantly larger than d,. The middle and the bottom further explore the d, d2 case, showing how differences in separability between shared dimensions (uD) and type-favoring dimensions impact weight allocation.
illustrated in Figure 2 showcasing different scenarios.
Since components corresponding to yt for classes of type aj align with nー for classes of type a2. the optimal representation for the training distribution results in poor separation for the shifted test distribution. Therefore, a robust representation should prioritize dimensions that provide effective separation for both class types, corresponding Pu-
This aligns with a common principle in the OOD generalization field, where robust representations are those that rely on features shared across environments (see $ This principle is often referred to as invariance.
Proposed Approach
Motivated by our analysis of the parametric model, we propose new approach for tackling class distribution shifts in zero-shot learning. Our approach revolves around two key ideas: (i) during training, different mixtures of the attribute A can be produced by sampling small subsets of the classes, forming artificial environments, and (ii) penalizing for differences performance across these environments is likely to increase robustness to the class mixture encountered at test time.
4.1 Synthetic Environments
Standard ERM training involves sampling pairs of data points (2g, 25) uniformly at random from all
N. classes available during training. However, as discussed in ] this prone to overfitting to attribute distribution of the training data. Since the identity of the attribute is unknown, weighted sampling (and similar approaches) cannot be used to create environments with different attribute mixtures.
5

---

Yet, our goal is to design artificial environments with diverse compositions of the (unknown) attribute of interest. To do so, leverage the variability in small samples: while class subsets of similar size to No maintain attribute mixtures similar to the overall training set, smaller subsets with l & No classes are likely to exhibit distinct attribute mixtures. Therefore, we propose creating multiple environments, composed of examples from few sampled classes.
This results in hierarchical sampling scheme for the data pairs: first, sample subset of k classes, S = {C1,,... ..... Ck.. Then, for each e E S sample 2r pairs of data points as follows: r pairs from within the class c, {24: = c}. uniformly random (positive pairs); and negative pairs, where one point is sampled uniformly at random from and the other from all other data points in S, { * C, E S
Across multiple class subsets 5 = (S....., Sn), this hierarchical sampling results in diverse mixtures of any unknown attribute (see Figure Bp. In particular, in some of the class subsets, classes from the overall minority type constitute the majority in the environment.
F
Sample subsets of classes
CAE E
A
D
C
D
F
B
A
D
B
Sample pairs data points
Figure 3: Illustration of the proposed hierarchical sampling. Top: N. = classes, with 2 minority- type classes D. F (in purple). Middle: synthetic environments formed by sampling small (k 3) class subsets; 1/5 of the environments, minority-type classes become the majority constituting 2/3 of the classes. Bottom: sampling r = 1 positive and negative pairs for each class in the environment.
4.2 Environment Balancing Algorithm for Class Distribution Shifts
Our goal is to learn data representations that will allow separation between classes without knowing which attribute is expected to change and how significantly. Therefore, we require the learned data representation to perform similarly well on all mixtures obtained on the synthetic environments.
To achieve this, inspired by OOD performance balancing methods (see objective:
$2). we optimize penalized
min
St (go) XR (S,,,,... S.)
where RSS...... Sx)) is any balancing term between the constructed synthetic environments.
Note that computing (SI...... Sn) often involves evaluating some value on each environment separately. For general balancing term, we denote the value in the I-th environment as f(S)) and accordingly express R (S,,... (S,,..., S.) (J(S),, ... f(S,)), where / represents the corresponding aggregation function. Our approact for balancing performance across synthetic environments of class subsets, outlined in Algorithm
4.3 Balancing Performance Instead of Loss
In multiple OOD penalties (e.g., IRM and VarREx), f represents the loss in each environment, which, in deep metric learning algorithms, is based on distance. This presents a challenge in zero-
Herre, for simplicity create balanced environments, but different proportions of positive examples can be considered instead.
For notation simplicity we assume that the unpenalized training loss is applied to pairs of data points (Wig, Mo) = ((zi, 23), Ae.=ey).. ). but it can easily be adapted for any tuple size (e.g.. triplets).

---

Algorithm Robust Zero-Shot Representation
Labeled data D = {zz, Ci ) number of synthetic environments n, number of classes within subset k, number pairs per class 2r, neural network 9(: loss E, distance function regularization functions f. f. initial weights db, number of training iterations T, learning rate Output: Learned representation g(: 0r)
Input:
Compute unique classes C {cl NN.)
for t = 1 to T do
for - 1 to do
( From each class in s( sample positive and negative data pairs, Denote the set by D Compute [(S(")).
Sample classes from C* without replacement: s/0= = (o)
Compute average unpenalized loss over (Im ym) E D(:
E00 = 2rk
E2rt
((ams Fm)
end for
Compute RO = R (1 "i......!"" -f
f(s!))
Update network parameters performing a gradient descent step:
g(t) g(t Ve ( Ei= z R(()) end for Return: g(:: 0r)
shot verification, where sampled tuples often include numerous easy negative examples, leading to performance plateau early in the learning process, although the distances themselves still exhibit considerable variations. Strategies like selecting the most difficult tuples [18] were proposed to address this issue, however these methods have been found to generate noisy gradients and loss values
34
We therefore propose to balance performance directly instead of relying on the losses in the training environments. Denote the set of negative pairs in synthetic environment by D9 = {Ii = (2,, 2z)
C; Cj E St, Viy = 0} and the set of positive pairs by DJ = {zi (2,, 2y) Ca. Cj E St, Viy = 1}. An unbiased estimator of the AUC on given synthetic environment S is given by
AUC (St:d))
1
ΣΣ Iaj
1 (dg()))) < dy (zww)
(7)
DPI
for Jaj E DJ and Tue Df. Since this estimator is non non-differentiabee and therefore cannot be used in meddientddecentt-aae optimization, we use soft-AUC as an approximation
AUC (g;;
ΣΣ og (F - dg (x1z))
(8)
DD
Jiy Rue
where sigmoid (t) Ite- approximates the step function. Note that when B co, 0g converges pointwise to the step function. Consequently, we propose the penalty:
RVarAUC (S,,..., ga) = Var (AUC
AUC. (S d))
(9)
4.4 How Many Environments Are Needed?
The proposed hierarchical sampling scheme allows for the construction of many synthetic environ- ments with various attribute mixtures, influenced by the number of classes in each environment. As shown in the analysis below, this ensures that with high probability there will be at least one environment with a pair of minority type classes, thereby supporting learning to separate negative pairs within the minority type.
In each training iteration, we consider n class subsets (environments) of size Our goal is to achieve robustness to all attribute values that are associated with at least Pmin E (0, 1) of the training classes. Note that Pmin specified by the practitioner without knowledge of the true attribute that may cause the shift or its true prevalence p in the training set.
7

---

We compute the number of synthetic environments n, such that with high probability of (1 a). Si...., Sn will include least one subset with least two classes associated with a (otherwise none of the subsets would contain negative pairs with the attribute a). Denote the probability of given subset not to contain any class associated with a by du ( ( M (()) and the probability
of given subset contain exactly such class by Φ1 = No ( g (/()) Therefore, the Ne
k-
required number of environments needed to ensure that at least two minority-tyl classes appear together in the same environment is
log (a)
12 ~2 log (Øo + Φ1)
(10)
Note that that n is typically much smaller than (%
5 Empirical Results
Our method enhances standard training with two components: a hierarchical sampling scheme and a balancing term for synthetic environments. To the best of our knowledge, this is the first work addressing OOD generalization for class distribution in zero-shot learning. We therefore benchmark our algorithm against the ERM baseline (uniform random sampling with an unpenal- ized score) and hierarchical sampling baseline (hierarchical sampling with unpenalized score). Additionally, we tested standard regularization techniques including dropout and the L2 norm, which did not yield notable improvements in the distribution shift scenario, and are therefore not shown.
0.95
0.90
To ensure a comprehensive comparison, in addi- tion the proposed VarAUC penalty, we evalu- ate variants of our algorithm in which the IRM, CLOvE, and VarREx penalties are used instead. While we show that VarAUC consistently outper- forms other penalties, the crucial improvement lies in its performance compared to the ERM baseline: application of existing OOD penalties is enabled by the construction of synthetic en- vironments our algorithm. As discussed in Appendix C this construction facilitates the for- mulation of class distribution shifts in zero-shot learning within the OOD setting.
0.75
0.70
0.65
ERM
IRM
VarREx
Hierarchical
CLOVE
VarAUC
0.60
100
200
300
400
500
600
700
Data Points (pairs)
Figure 4: Average AUC over 10 simulation repeti- tions for majority attribute proportion p 0.9 in training (and 0.1 in test). Solid lines: distribution- shift. Dashed lines: in-distribution. Our method improves robustness for shifts, without compro- mising training distribution results.
In all of the experiments performed, we trained the network with contrastive loss (Equation 2 and the (1
normalized cosine distance: (2121,2))
g(21) (22))) The specific setups are detailed
To(ai) [=]]]
below (additional details can be found in Appendix FF), and code to reproduce our results is available
htttps:/ggthhub.comm www.oo///ll/eeoo////i LaaprrnShhoooooooootteeeee DepbuussRerrseetationnss
5.1 Sinulations: Revisiting the Parametric Model
We now revisit the parametric model presented in $3 To increase the complexity of the problem, we add dimensions where classes from both types are not well separated. That is, Eg includes additional dimensions set zero.
Setup We used 68 subsets in each training iteration, each consisting of two classes. This cor- responds choosing Pmin 0.1 (desired sensitivity, regardless of the true unknown parameter E {0.05, 0.1, 0.33), with a low a value of 0.5, resulting in the construction of fewer environments according to Equation 10 For each class, we sampled 2r- 10 pairs of data points. The representation
was defined as g(z) wz for w E d 3 Here we focus on the case of p 16, Vz vn = 1,
A linear representation is chosen facilitate analysis of the learned representation space.
8

---

V- = 001, 0.1, = 2, do 5, = 10. The results for additional representation sizes noise ratios == and varying proportions of positive and negative examples are presented in Appendix D.I To assess the importance assigned to each dimension, we examine weight values relative to other
Σ
weights: Importance,
(11)
Results In Figure we examine the learned representation. The analysis indicates that ERM prioritizes dimensions 5-15, providing good separation for d1, the dominant type in training, but leading to poor separation after the shift. ERM assigns low weights dimensions benefi- cial for both types (0-5) and those suitable for 02 (15-25). In contrast, with the two seriance-basel penalties, assigns the lowest weights to dimensions corresponding to 01 and higher weights to shared dimensions and those that effectively separate 02 classes.
0.06
algorithm, particularly
0.05
Importance
0.03
0.02
In Figure the learning progress is depicted for 0.9 (a similar analysis for 0.95 and 0.7 can be found in Appendix DD. Performance on the same distribution the training data is similar for ERM and our algorithm, suggesting that applying algorithm does not negatively impact performance when no distribution shift occurs. However, when there distribution shift our algorithm achieves much better results. The VarREx penalty achieves high AUC values more quickly than the VarAUC penalty, but the VarAUC penalty attains higher overall accuracy. IRM shows noisier convergence, since it is applied directly on the gradients, which have been shown to be noisy in contrastive learning due to high variance in data-pair samples 34. Means and standard deviations are reported in Appendix [D. as well as the results for additional data dimensions, positive proportions, and variance ratios.
0.00
ERM
IRM
VarREx
Hierarchical
CLOVE
VarAUC
Figure 5: Average feature importance for p 0.9. 10 repetitions. Our VarAUC penalty favors shared features (blocks and 3), while deprioritizing ma- jority features (block 2). All methods assign low weight to noise features (block 4).
5.2 Experiments on Real Data
Experiment 1- Species Recognition We used
15.0%
the ETHEC dataset (11] which contains 47.978
10.0%
5.0%
butterfly images from six families and 561 species (example of the images are provided in
0.0%
550%
Appendix D). We filtered out species with less
10.0%
than five images and focused on images butter-
-1.00
flies from the Lycaenidae and Nymphalidac fam- ilies. In the training set, 10% of the species were from the Nymphalidae family, while at test time, 90% of the species were from the Nymphalidae family. For each class we sampled 2r 20
Hier.
IRM
CLOvE VarREx VarAUC
15.0%
10.0%
5.0%
0.0%
pairs.
-5.0%
10.0%
Experiment 2 Face Recognition We used the CelebA dataset [30 which contains 202.599 im- ages of 177 celebrities. We filtered out peo- ple for which the dataset contains less than three images. Following Vinyals et al. [51), we im- plemented g as convolutional neural network which has four modules with 3 x 3 convolutions and filters, followed by batch normalization,
-15.0%
Hier.
IRM CLOvE
VarREx VarAUC
Figure 6: Average percentage changes of our method compared ERM across 10 repetitions are shown for the ETHEC (top) and CelebA (bot- tom) datasets. Error bars represent + one std-dev.
ReLU activation, 2 2 max-pooling, and fully connected layer of size 32. We used the attribute blond hair for the class distribution shift: for training, mainly sampled people without blond hair (95%), while at test time, most people (95%)

---

had blond hair. Each training iteration had 150 synthetic environments of two classes and 2r 20 data points per class.
We trained the models on 200 synthetic environments at time, each of two classes. We implemented g fully connected neural network with layers of sizes 128, 64, 32 and 16, and ReLU activations between them.
Experimental Results As can be seen in Figure 6 while all versions of our algorithm show some improvement over ERM, the best results are achieved with the VarAUC penalty (exact means and standard deviations reported in Table in Appendix [D). One-sided paired t-tests show that the improvement over ERM achieved by our algorithm with the VarAUC penalty is statistically significant, with p-values of 0.04 on both datasets; p-values for other penalties are reported in Table 4 All p-values were adjusted with FDR [ correction.
In Appendix [ we also provide additional analysis confirming that the main improvement of our algorithm over the ERM baseline stems from improved performance on negative minority pairs.
Discussion
In this study, we examined class distribution shifts in zero-shot learning, with focus shifts induced by unknown attributes. Such shifts pose significant challenges in zero-shot learning where new classes emerge in testing, causing standard techniques trained via ERM to fail on shifted class distributions, even when the conditional distribution of the data given class remains the same.
Previous research (see Appendix A assumes closed-world classification or known cause, making these methods unsuitable zero-shot learning or shifts caused by unknown attributes. response, introduced framework and the first algorithm to address class distribution shifts in zero-shot learning using OOD environment balancing methods.
In the causal terminology of closed-worid OOD generalization, our framework employs synthetic environments intervene on attribute mixtures by sampling small class subsets, thereby manipulating the class distribution. This facilitates the creation of diverse environments with varied attribute mix- tures, enhancing the distinction between negative examples. A further comparison our framework with OOD environment balancing methods is provided in Appendix C Additionally, our proposed VarAUC penalty, designed for metric losses, enhances the separation of negative examples.
Our results demonstrate improvements compared to the ERM baseline on shifted distributions, without compromising performance on unshifted distributions, enabling the learning of more robust representations for zero-shot tasks and ensuring reliable performance.
While the proposed framework is general, current experiments address shifts in binary attribute. We defer exploration of additional scenarios, such as those involving shifts in multiple correlated attributes, future work. An additional promising direction for future work is the consideration of shifts where the responsible attribute strongly correlated with additional attributes or covariates. This opens up possibilities to explore structured constructions of synthetic environments that leverage such correlations.
10

---

References
[1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 2019.
[2] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis- Louis-Philippe Morency. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 59 66. IEEE, 2018
[3] Aharon Bal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
[4] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series (Methodological)) 57(1):289-333 300, 1995.
[5] Lacey Best-Rowden and Anil K Jain. Longitudinal study of automatic face recognition. IEEE transactions on pattern analysis and machine intelligence, 40(1): 2017.
[6] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International conference on machine learning, pages 872-881. PMLR, 2019.
[7] Toon Calders and Szymon Jaroszewicz. Efficient optimization for classification. In European conference on principles of data mining and knowledge discovery, pages 42-53. Springer, 2007.
[8] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distribution-awaae margin loss. Advances in neural information processing systems, 32, 2019.
[9] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In Interna- tional Conference on Machine Learning, pages 1448-1458. PMLR, 2020.
[10] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[11] Ankit Dhall. Eth entomological collection (ethec) dataset [palearctic macrolepidoptera, spring 2019]. 2019.
[12] John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3):1378-1400)) 2021.
[13] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946-969, 2021.
[14] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611 2006.
[15] Patrick Grother, Mei Ngan, Kayee Hanaoka, al. Ongoing face recognition vendor test (frvt) part 3: Demographic effects. Nat. Inst. Stand, Technol. Gaithersburg, MA, USA, Rep. NISTIR, 8280, 2019.
[16] Zhangxuan Gu, Siyuan Zhou, Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1921-1929, 2020.
[17] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR' (CVPR'06), volume 2, pages 1735-1742. IEEE, 2006.
[18) Alexander Hermans, Lucas Beyer, and Bastian Leibe. defense of the triplet loss for person re-identification. arXiv preprint arXiv: 1703::1703.77722 2017.

---

[19] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737. 2017.
[20] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International workshop on similarity- based pattern recognition, pages 84-92. Springer, 2015.
[21] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Pazz Simple data balancing achieves competitive vorst-group-accuree Reasoning, pages 336-351. PMLR, 2022.
Conference on Causal Learning and
[22] Aparna Joshi, Xavier Suau Cuadros, Nivedha Sivakumar, Luca Zappella, and Nicholas Apostoloff. Fair sa: Sensitivity analysis for faimess in face recognition. In Algorithmic fairness through the lens of causality and robustness workshop, pages 40-58. PMIR, 2022.
[23] Brendan F Klare, Mark J Burge, Joshua C Klontz, Richard W Vorder Bruegge, and Anil K. Jain. Face recognition performance: Role of demographic information. IEEE Transactions on information forensics and security, 7(6): 1789-1801111 2012.
[24] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, al. Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning (ICMI) deep learning workshop, volume 2, 2015.
[25] David Krueger, Ethan Caballero, Zhang, Remi Le Priol, and Aaron Courville. Out-of-disrribution generalization via risk extrap- olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
[26] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neu- ral networks from kernel mean embeddings. Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 2805-2814. PMLR, 10-15 Jul 2018. URL httpp://proceedinggs mlr grress/v0//kumrr html
[27] Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume page 3, 2008.
[28] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980, 2017.
[29] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
[30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
[31] Arakaparampil M Mathai and Serge B Provost. Quadratic forms in random variables: theory and applications. (No Title), 1992.
[32] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations, 2020.
[33] Dana Michalski, Sau Yee Yiu, and Chris Malec, The impact of age and threshold variation on facial recognition algorithm performance using images of children. In 2018 International Conference on Biometrics (ICB), pages 217-224. IEFE, 2018
[34] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In Computer VESV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020. Proceedings, Part XXV 16, pages 68 681-699. Springer, 2020.
12

---

[35] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. Proceedings of the IEEE conference on computer vision and paltern recognition, pages 4004-4012, 2016.
[36] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947-1012, 2016.
[37] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda tions and learning algorithms. The MIT Press, 2017.
[38] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Focus on the common good: Group distributional robustness follows. In International Conference on Learning Representations,
2021.
[39] Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAT/ACM Conference on Al, Ethics, and Society, pages 429-435, 2019.
[40] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
[41] Joseph P Robinson, Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. Face recognition: too bias, or not too bias? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-1, 2020.
[42] Mateo Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309 ((1):1309-1142, 2018.
[43] Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjor Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference on Machine Learning, pages 8242-8252. PMLR, 2020.
[44] Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: a simple procedure. Neural computation, 14(1):21-41, 2002.
[45] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
[46] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likeling likelihood function. Journal of statistical planning and inference, 90(2):227-244 2000.
[47] Tomáš Sixta, Julio CS Jacques Junior, Pau Buch-CCrdona, Eduard Vazquez, and Sergio Escalera. Fairface challenge eccv 2020: Analyzing bias in face recognition. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 463-481. Springer, 2020.
[48] Yuli Slavutsky and Yuval Benjamini. Predicting classification accuracy when adding new unob- served classes. In International Conference on Learning Representations, ICLR, Conference Track Proceedings, 2021.
[49] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[50] Nisha Srinivas, Karl Ricanek, Dana Michalski, David S Bolme, and Michael King. Face recognition algorithm bias: Performance differences images of children and adults. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[51] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
13

---

[52] Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out- dfmainnin generalization. Advances in neural information processing systems, 34:2215-2223 -227,, 2021.
[53] Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness-aware reinforcement learning. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 9322-932 1231,, 2020.
[54] Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In Proceedings of the ieee/cvf international conference on computer vision, pages 692-702, 2019.
[55] Jiaheng Wei, Harikrishna Narasimhan, Ehsan Amid, Wen-Sheng Chu, Yang Liu, and Ab- hishek Kumar. Distributionally robust post-hoc classifiers under prior shifts, In International Conference on Learning Representations (ICLR), 2023.
[56] Chao-Yure Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE international conference on computer vision, pages 2840-2848, 2017.
[57] Yongqin Xian, Subhabrata Choudhury, Yang Hc, Bernt Schicle, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2019.
[58] Tongtong Yuan, Weihong Deng, Jian Tang, Yinan Tang, and Binghui Chen. Signal-to-noise ratio: A robust distance metric for deep metric learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4815-4824, 2019.
[59] Charles Zheng, Rakesh Achanta, and Yuval Benjamini. Extrapolating expected accuracies for large multi-class problems. The Journal of Machine Learning Research, 19(1):2009-2638 2018.
[60] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia, Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16489-16498, 2021.
14

---

A Related Work on Class Distribution Shifts in Clrld Settings
In class-imbalanee learning [8 10, 8 it is assumed that some classes are more dominant in training.
while in deployment this no longer the case. Therefore, solutions classically include data or loss
re- re-weighting [46] 6 40 32 and calibration of the classification score [44 60) A popular framework
for addressing class distribut ion shifts is distributionally robust optimization (DRO) [ [3 12 (55),
where instead of assuming a specific probability distribution, a set range of possible distributions considered, and optimization is performed to achieve the best results on the worst-case distribution. A special case known as group DRO [45] 38) involves group variable that introduces discriminatory patterns among classes within specific groups. The framework
address this includes methods that assume that the classifier does not have access to the group information, and therefore propose re-we re-weighting high loss examples [29]. and data sub-sampling to balance classes and groups [21]. Nevertheless, the methods mentioned above rely on the training and test class sets being identical, making them unsuitable for direct application in zero-shot learning scenarios.
B Analysis of the Parametric Model
B.1 Derivation of the Loss
We begin by revisiting the parametric model introduced in 3 Let 2 ~ (c, Eg), where Σg Vela. 0 < Vs E R, and La is the dimensional identity matrix. Classes e₁ drawn according to Gaussian distribution ~ N (0, M.) corresponding to their type E {ai, 42). Here, we use simpler (although less intuitive) notation for the values of the diagonal matrices Eg:
do
Ea, diag 100 Vo:..., Vo, .1....; 11, 22, .......... ,2)
Eeg diag (o...., 40 12, 12, Viy..., V1
where 0 < 1/2 < 1/8 < 10 < 11.
We consider weight representations g(2) Wz, where W is diagonal matrix with diagonal
W E IRd.
Since Σg is of full rank, it suffices to consider the no-hinge version of the contrastive loss, that is
(zt, Zz. Migi dg) yıj ||W (=: =1),1* + (1 (1ss)))
Mis) (m W (z: =
(12)
where dg (zi, 2,) = 1lg (z, 2,)11 = |W (z, - 21) (I) denotes the Euclidean norm For a balanced sample of positive and negative examples, the expected loss is given by
E 8 (2i, 2j. Wiji dg)
1
Em.,=1
[IW (z: - =+))1
2
2 Ey, =0 m² 2m |IW (z+ - -1))11 + IIW (z+ - 2))11]]
(13)
To calculate the expression above, we begin by proving the following lemma:
Lemma 1. Let μ E Rdd be random variable and let tlp ~ (N, E). If 0 (constant), then
1. E ||4]1* = 21r( + (2))
If μ ~ (O, E,), then
2. IE || = tr()) + tr(2,,,
3. IE =|
2 (r
4 (222)) + tr²(2') + +2tr(z) tr( + 2tr + t2²()),)
'Squared distance is selected for its simplicity in computing the expected value of even powers of the Euclidean norm of Gaussian variables.
15

---

Proof. For any random variable u E R such that a ~ MMu, Ex), and any symmetric matrix A,
we have
Eulu [u]] - tr(AEu) + Mu Apl:
(14)
Eu Eu Aul² 2T ((AEw)²) 4p ApT + (tr(AE)) + p. An,)
(15)
(see, for example, Thm. 3.2b.2 in 31
First, letting Mou = 0, En = E and A = La in 15 we get
E ||||** B[ 21r(E3) (E).
(16)
Now, assume that j ~ (0, Ep). From 14 we get E, ||al12 = Ep ju ud] = tr(Ex), and thus
E ||41 Ey Eej []] E,, [tr(Z) + n² n] tr(E) + tr(Ew).
(17)
Similarly, from 15 we have
[|[||| = E 1 [ 8/2 [0] = 2 + F Ep] tr +rrr(E) + 2tr(( Eap 11/112 + Ey |Ial]
(18)
By substituting A = E 14 we get E, Ja "x] tr tr(2>2,). ,) and from 15 we have
En ||]l411 2 tr ( + tr (E,))
(19)
Therefore.
E ||/ = 21=(2u1(()))))) (ur(EE, 22()) 21r(()) (E,)) +
(20)
Note that W (2: - 22) ~ N (N, E), with μ W (c: c2) and 2v, WTW W.
If yıy = 1, then 2, and are from the same class, meaning that c; = and thus = 0. Therefore,
by Lemma | we have
| (z. =,))1= =2 2rr (x.2) + tr² (Σ) Byg
= 4v² (WTWW? 2)
4v tr² (WTW)
w + 2v
Σ
w
(21)
However, for pairs from different classes, that is, when yig random variable distributed according to N (0, Ex), where
O. the mean μ is itself Gaussian
WT (2E,,) W
Ca Gj are both of type a1
Σp
WT (2 W
Ci Gj are both of type d2
(22)
WT (Ea, + Eaz) W
C, Gj are of different types.
Therefore, by Lemma
we have
Em, =0 |IW (z: 2z =1 = E [tr (2ip) + tr (Σ)] = Ey,, (tr (2nh)) + tr (2))
= p² tr (2W Ea,, W) + (1- - (1-p)² 2r tr TEW
Eng W)
+pp(1 p) tr (WT (Ea, + Eaz) W) + tr (Σ)
do
dotdi
d
(un + Uz) = 2
Σ
/ + (ar + Uz)
Σ
w + (02 + vz)
Σ
w
iml
#1
(d
+1
(23)
16

---

where
01 = +'v 92 + p(1 0) (v) 12) pn + (1 p) V2
0g (1 + p(1 p) (N + 1/2 pvz (1 - p) VI: p011))p1pp))p0ppp --1--) (v (11+1))) +12)²
61 =
+ p(1
8 = = + 2 p) p(1 p) (4 + +2)
(24)
By Lemma 33 we have
Ey =0 ||1 (z+ 2z)l1* By, =0 2r (2:) + 4 (2 + + (xx tr (Ex,)
+2 (E7) (tr (x, |
=2)r(>²) + 4 و ttr -لمادد | + tr (2)) + 2 =o,
+ 2 Byy |t ( الشاح + My =o (اnد))
(25)
where
Ey,,= [ir (EEm))) =2 tr W W [2p2 Ea,, W + 2(1 - u) Eaz W
+ 2p(1 - p) (Des + Eag) W])
do
do+d1
=4/;
Vu
W
4
+ a1
Σ i
W. + 02
W
(26)
i=1
do+1
do
Ey-=0 [tr (Ep)) =p² tr (2W Eaj W) + +(1 (11pp) p) tr (2W Ea, W)
+ 2p(1 p) tr (W (Eu. + Doz) W)
do
dotd, Σ
=2
40
an
wh + 02
Σ
2
br
(27)
i=1
d=
+d
and so
d
do
do+dd
tr (Σ) Ey, [tr (Ex)) = 4v:
Σ
w
w
+
2 +
(28)
i=1
i=
i= do +1
Eg, [tr (2:2)] =p² tr ((WW En, W)2) +(1 (1- (2W ()2)
+ 2p(1 - p) tr (WWT) (Ea, + Enz) W)2)
do
do+d1
-2
2
w B
Σ wi! + 32
Σ
w!
(29)
1=1
i=
+1
and similarly
do Σ
2
dg
2
d
Ey ttr² (2in)] =2 2n
wi
3
Σ
Wi
2
+
B2
Σ
wi
i=1
ii
ii
+1
dg
da
do
+ 4
w
w + 470,2
w
1=1
i= 1
1=1
dodo+ +1
dg +dd1
+ 471,2
W
(30)
i=do+1 1
i= d +1
17

---

where we denote for short
70,1 = pou + (1 - + p(1 plv0 (n + U2)
70,2 = piuous (1)))²m001 + p(1 p)u0 (11 + 12),
1, = p²v + (1 - u) + 70(1 p) (vr + tz2)²
(31)
Finally, due to symmetry, at the optimal solution we have
0 s do U1 do + do d1
(32)
U2
do + dl + + d,
and by combining these results, we get
E (y.
down (8v² 8v:40 + 4v0 + 20)
+ dl diut (8v2 8 + + 3,d,)
+ dzuš (8v2 8v 232 + 32d2)
2dou (v) + v2) - ud (a1 + V2))
22222 (ay + vzs)
1
+ 4v² (dou) d d dz
2 m
+ 4u, (dou) + d u dzuz)
[uodou a aj of azd2uă]
+ 40 uzu + do 2d d2.
(33)
B.2 Analysis of the Optimal Solution (Proof of Proposition
Proposition I shows that when d and d2 are relatively similar, the optimal solution on the training distribution, assigns more weight to components with high variance the training data than to those with high variance in the shifted test distribution.
We begin by defining the required condition dj and d2. Denote
11 2v + 21-01 31
4/2 = 20 + 2v=02 83
7o1 = 4v² + 2v: (a1 + vo) + 20o,,1
7o2 = Av? + 2v: (02 + up) + 27,,2
712 = 4v² + 2v (a1 + a2) + 271,2.
(34)
Then for B,y values as in equations
24 and 31
we define
11 (a2 + U2) 2/2 (01 + vz)'
1/1 1702
hy (p, Uz, No. M1, v1,
bn (p, U2, vo, (1,2) = 1/22 1/01
(35)
and the corresponding condition
d2 +
hy (p, Uz. Mo, v2
< ha (p. Uz. Mu, (,, 12)
(36)
d1 +
While this condition sufficient, not necessary. Values of p. Mz, Pp, V1, 1/2 and di, d2 that satisfy 35 provide an example requiring only simple analysis, without a full characterization of the optimal solution, for the failure of optimization over the training distribution. However, such failures can occur for additional parameter values, and the full characterization is provided in Appendix B.3
18

---

Proof. Without loss of generality assume m=1. Then.
dE ē
(2), 2gz Viyi do)
2doug (8v3 + Suzu0 + 4e0 + 2v)
Oug
2do (vo + vz)
+ +douz (doug + d1 duu + d2uz)
4dgu= (vodnug + a1 viddun, azd2uÿ)
4do (doug + duis + d2už)
+ 470, + 2702d
(37)
and by setting the partial derivative to we get
2n6 (2 + do) (2v² + 20zu0 + v²) =2d u (2v² + μ= (o) + vo) + %0.1)
+2d2uz [2v² μ= (a2 + vo) + 70.2) (vD + vz).
(38)
Therefore,
2 + - "unddu - 22222
ug (2 + do) (2v² + 2v 40 v7)
(39)
and similarly
of = (a1 + v/2) - 701 d 722222
(40)
22 + d1) (2v? + 20, 31)
2 = (a2 +1) - n2 m2djut
(41)
2 d2) (2v² + 2v,02 32)
Hence,
(2 d2) (2v² + 22,02 + Bb) d1 (a1 + 1/2) do m221 d2
d - dgug
22,0; + (1) (2v? + 2v + 82) (2 + 2223+ + de (a2 + vz) do2d2do4 112d1 dzu?
(12+
(42)
(2 2 + (2,
2v, + 32)
Denoting
& = 22 + de) (2v² + 20 + 31) (2v, + 2v,02 + B2)
= 22 + d,) (2 + d2) 1//
have
1 [
1 (2 d2) 222722
uf
(2 + (d) dimn =2 1-
+ 1 (2 d2) 2/ (an + v=))
(2 + d1) th (az + vz)
doou } (2 (2+ d2) 12
(2 + d2) $21101
and therefore
1
(2 + d2) 12712 -1)
di du d2u2 =d2
1
& (2 (2+d) di) 11
1
(2 + d2) the (a1 + +v²) (2+ d1) 1/2 (az + vz)
+ 2(2 + di) (2 + d2) $1422
1 1- (2 dz) 1/1711
1
(2 dh) 11762
(2 + d2) 1/2
dd
2(2 + d1) (2 d2) du
1 1- //2 (2 + di 2/m2
1- (2 d1) 117712
(43)
19

---

Denote
À = (2+ + d1) [dousm12h - (az + 1/2) 1 + dou 702
(2 + d2) [22ud12222 (a1 + 12) th2 do 20
(44)
and thus
duuî - dzu6 (2 + d1) (2 d2) 0182 (2 d1) 111122
A.
(45)
Note that for di, d2 such that
(2 d1) (2 d2) 1/2
d
d1+2
422
(2 d2) (01 + v2) 2/2 (2 d,) (a2 + v2) >1
d
Wi (aatte)
V2 (aitvs)
(2 + d1) 4/ 1/2 (2 + d2) 2/2 > 0
V1 n02
21+2
we have A
Since
202 Mo1
this reduces the last two conditions and therefore, in particular for
1/ (ag + V2))
d2 + 2
1/ 7/2
(46)
1/2 (a1 + 12))
di
2 7/1
have A 0. Additionally, note that
1 1 (2 d1) 01 <
712
2(2 d2) 1/2 m2
(2 (2+d) 41
(47)
22 222 + 12122
2 + d2) 1/2
and thus 1 - (2 + d1) dime > 0ff
1 1112 2 the
d2 + 2
(48)
Combining these conditions reduces to
1 (a2 + 12)
d2 + 2
t/ 1722
(49)
U (ai Vv)
di +
1/2 7/01
and therefore, for Ms, Mo, M1, M2, d1, d2 satisfying
Φ (a2 vz)
d2 +2 2
1/1 1/02
<
(50)
th (a1 vz)
dj +
1/2 7/1
we have diuţ - dz
"Similarly, the condition obtained for - } (2+ d1) iima < 0 and A 0 2h (2+d)) < de <
al. (a2) (2 d1), which cannot be achieved since az
(oi+t))
20

---

B.3 Explicit Expression for the Optimal Representation
In order to derive the optimal representation, we differentiate the expected loss with respect to the squared values in the diagonal of W, that is, wf:
a
tr (22)
(51)
2 (w?)
θ
d
j tr² (E) 2
Σ j=1
(52)
Suzuouf Su.aiuf,
1 dn dg 1 +1 do fo+d+1ii d
0
By= [tr (2121,))
(53)
0 (w??
Sy.azu,
0 tr.))) E (tr c
(54)
0 (w?)
do+di
4v2
2v0
Σ j=1
w + (al + +b)
Σ
w + (a2 vo)
Σ
wj
isis do
j=
+d
dg
dotdi
d
4v=
(vo + 01)
Σ
u/ + 201
Σ
w/ + (02 + 01)
do +1 d1ssisd do+0 d1
j=1
j=do+1
dddo+
du
dp+di
d
4//
(1o + a2)
Σ w3 + (ai + a2)
Σ ur + 2a2
Σ
w
dg + +1
1=1
1=
+d++
(55)
1 isis do dn + 1 S < do + di dg + d1 +lsisd d
0
E B [tr (Σ/)
48ww
(56)
0 (w²?)
432w?
0
Ey=u [tr² [Σn)]
(57)
0 (w?)
dg
dotd;
d Σ
81
Σ
w + 8%
Σ
w/ + +8,0
m
1 i do
/=1
j=d
jmdg +d1+1
dp
8 j=1
u + 431 Σ
w/ + +8,12
Σ
uz dg + +lsis +
(58)
j=d+
de
dp
d Σ
8
Σ
w + 871,2
Σ
? + 4,3
m3 do + +
j
j= +d
Combining these results, we get for 1 <is do
d
0- = 0 (w?)
7 (z 2j, Vigi d() =
2. 2.8 u + 81 Σ wp
m (2 (vD + vz)l
2
5=1
+ W/ + +4 Σ
w 2 + Su ?
/=1
dg
do
+ 4vz
210
2
(a)
vo) Σ
wz (a2 vo)
10
Ø=1
j=d++1
j=do+d1+1
du+di
+ 888 wi
+
1
81 Σ
2
i 2
+ 8
Σ
w 2
+ 8
w 2
2
j=1
j +1
j + +1
21

---

for do +lsis do + d1
0
O1 )
8 (Zes 2z, Migi dg)
2 2.8v.uu uj
+ 81 Σ wj
m (ai + vz)]
+ 8V w + 4v 2
Wj + 2- Su
j=l
do Σ
do
+ 422
(vo + a1)
+ 201
Σ
uj u + (az +
mn
5=1
j=d++1
+=do+d +1
dg
dp
d
+ 4,3 w/
+
1
870,1
]j + 4
Σ
wj + 1,,
/=1
j=do+1 1
j dd
and similarly for do + dd + sisd
02 = ②
d Σ w
{ 2j. Viji dg)
2. 8v u
+ 8v 2
m (22 + vz)
(w
2
1=1
+ 8v2
+ 4v² Σ
w 2
+ 8
j=l
do
do
+ 4vz
(4) + (2))
Σ
w + (0) + a2)
Σ
wz + 202
j=1
j=du+1
di +1
do
do
d
1
+ 432 +
87
w + 8
w + 482
j 1
j t11
Thus, we can write for the symmetric solution
d 2m (N) + v.) G + u² + ug
(59)
a 2m (an v2) + ugG u vgG
(60)
0 2m (a2 + v2)) + 22. 22 2²
(61)
where
Gy,0 16v² Sv 16 Svd SuI + 40
Go.1 Su2d1 + 4v, (a) + vo) dl dd
G, Sn-dı + 4v; (a2 + un) d2 + 4/
G1,0 = su-do + 4v; (un aa) dg + 470,1
G1,1 = 16x sv + 16v,0; + av + 48 + 43,d
G122 = Su2d2 + 4v, (o2 + 0,) d2 + 2
G2,0 = Swndn + 4v, (no a + 47
G2,1 Srnd, + 4v, (a: + a2) d1 471
22.2 16 Su 16 + S2ga2d2 4,9 + 4,2d22
Therefore, the optimal representation given by the solution to the following set of linear equations:
4
1o V/3
2m G-1
an + V2
(62)
Q2 + Uz
22

---

Table 1: Simulation results. For each mixture ratio we report the mean AUC and the standard deviation across 10 repetitions of the experiment. Results are reported for in-distribution scenario (Pc), and class distribution shift (Qc). Best result is marked in bold.
-= 0.05
p p= 0.1
0.3
ERM
0.948110.013
0.94870.007
0.913
Hier
0,945:10.013
0.949
0.917-00011
In Distribution
IRM
0.9451
0.947+1.009
0.909=100188
CLOvE
0,944:10008
0.949=0.001
0.911=0.020
VarREx
0,949=00013
0.948=00000
0.910=0.022
VarAUC
0,950:10.017
0.947
0.912#0.022
ERM
0.701
0.801
0.883=0.014
Hier
0.727 + 0.009
0.810 0.014
0.882 0.020
Distribution Shift
IRM
0.724+0.017
0.806+0.019
0.880=0.023
CLOvE
0.745+0.020
0.807#0.018
0.878=0.020
VarREx
0.729:#0.005
0.811#0.018
0.880=0.026
VarAUC
0.767
0.838+0.019
0.881 #0.024
where
Go.o
Ga,1
Ga,2
G
Gı,0
G1,1
G1,2
(63)
G2,0
G2
G2,2
C Comparison to OOD Environment Balancing Methods
Previous methods the field of OOD generalization (see $2 exhibit several key differences compared to our setting: (i) They address closed- classification, whereas in zero-shot learning, new classes are encountered. (ii) The presumed shift is typically in the conditional distribution of the data given the class (c.g., the background given the class being cow or camel), whereas we consider shifts in the class distribution P(c). (iii) Existing methods often assume that training data comes from various data environments, providing explicit information about how the distribution might shift, while we assume the attribute A causing the shift is unknown.
Despite these differences, in this work we recast class distribution shifts in zero-shot learning into environment balancing OOD setting, by making the following observations. First, when posed as verification methods, zero-shot classifiers in fact perform a binary (closed-world) classification task, predicting whether pair of data points zij (zi, 2j) belong to the same class Vij Le,=cy-
Note that the distribution of possible pairs maj = (zi, 2;) given the label Vij changes with variations in class attribute probabilities, and therefore across synthetic environments S. Thus, in this formulation the shift occurs in the conditional distribution of the data given the class p(r,ş) (Wr)).
Another distinction lies in data availability: in the setting of closed-world OOD environment balanc- ing methods, a main drawback is the challenge of securing a sufficient number of diverse training environments. This is essential to ensure that representation performing well on observed environ- ments, will likely perform similarly on unobserved ones. In contrast, our framework allows for the construction of many synthetic environments via sampling.
D
Additional Empirical Results
D.1 Additional Simulation Results
Simulations
Exact mean and standard deviations matching Figure are provided table
AUC progress during training iterations and feature importance results for the majority class propor-
tion of p = were shown the main text. Here, we provide analogous results for p = 0.05 and 0.3. These are summarized in Figure g
23

---

0.95
0.95
0:90
0.90
AUE
0.75
0.75
0.70
0.70
0.60
0.60
0.55
0.55
VarAUC
CLOVE
VarAuc
V1 2, 02
V1 4, Vy 0.1
0.95
0.90
0.90
0.85
0.85
AVC
0.75
0.70
0.70
0.65
0.65
0.55
0.50
0.50
ERM
IRM
CLOVE
VarREx
VarAUC
ERM
IRM
CLOVE
VarREx VarAUC
30% negative examples
70% negative examples
0.95
0.85
0.85
ALC
0.80
0.80
0.75
0.65
0.65
0.60
0.60
ERM
IRM
CLOVÉ
VarREx
VarAUC
ERM
IRM
CLOVE
VarREx
VarAuC
Figure Additional simulation results. Top row: Additional dimensions of the representation. Middle row: additional rations of the attribute variances, Bottom row: unbalanced sets of positive and negative examples. Bars show mean AUC values on the across 5 repetitions of the experiment, whiskers show + standard deviation.
For p 0.05 the convergence results similar those obtained for p 0.1 under distribution- shift the two variance based methods show significantly better results compared to other approaches. Our algorithm with the VarREx penalty achieves high AUC values more quickly than the VarAUC penalty, but the VarAUC penalty attains higher accuracy overall. The CLOvE penalty achieves improvement over ERM, but smaller compared the variance based methods. IRM converges the same AUC ERM. In contrast, on in-distribution data all methods perform well.
For p 0.3 the distribution shift is milder and therefore ERM performs very well (0.902 AUC achieved on distribution shift scenario compared to 0.932 on in-distribution setting). Therefore encouragement of similar performance across different data subsets does not benefit the learning process. Slightly better result is achieved with VarREx penalty (0.911).
The analysis of feature importance for 0.05 yields results similar those for 0.1. At p 0.3 the analysis remains mostly unchanged, except that VarREx assigns higher importance to features corresponding ro (0-5) compared VarAUC, while in more extreme distribution shifts VarAUC assigns higher importance to the shared features.
D.2 Additional Representation Sizes, Noise Ratios and Positive Proportions
In $5.1 we explored varying values of p setting where + 2, 0.1 (=- 20). We now tocus on the case of 0.1 and examine additional representation sizes and noise ratios (gt (10,40)) Additionally, examine the original setting where p 16 and + 2, -ー 0.1, with varying proportions positive and negative examples.
The results in Figure 7 show that in all the additional settings our methods provides statistically significant improvement over the baseline. FDR adjusted p-values for multiple comparisons are provided in Table
24

---

Table 2: FDR adjusted p-values for the results reported in Figure
Experiment
IRM
CLOvE
VarREx
VarAUC
0.7339
0.0112
0.0003
0.0001
p
0.8552
0.0005
0.0003
0.0001
V1 = 2, 1/2 = 0.2
0.9995
0.9995
0.0002
<0.0001
V1 4, 1/2 0.1
0.9989
0.9971
0.0041
0,0041
30% negative
0.9939
0.9939
<0.0001
<0.0001
70% negative
1.0
1.0
<0.0001
0.0002
Experiments In Table we provide the means and standard deviations for the experiments detailed in [5.2 Additionally, Table 4 presents the adjusted p-values for assessing the performance increase over the ERM baseline achieved by our algorithm with the explored penalties.
Table 3: Experimental results. Mean and standard deviation of AUC values over 5 repetitions are reported for in distribution scenario (Pc), and class distribution shift (Qc). Best result is marked in bold.
IN DISTRIBUTION
DISTRIBUTION SHIFT
ERM
0.826 + 0.001
0.666 0.001
IRM
0.843 + 0.009
0.659 0.087
CELEBA
CLOVE
0.853 0.002
0.677 0.012
VARREX
0.834 0.002
0.676 0.004
VARAUC
0.836 0.002
0.697 1 0.027
ERM
0.869 + 0.004
0.786 0.030
IRM
0.879 + 0.004
0.795 0.034
ЕTHEC
CLOVE
0.888 0.004
0.800 0.040
VARREX
0.877 0.007
0.805 0.033
VARAUC
0.872 0.004
0.838 0.049
Table Adjusted p-values for one-sided paired t-tests for testing the improvements over the ERM baseline.
CELEBA
ETHEC
HIERARCHICAL
0,0154
0.7677
IRM
0.6117
0.1290
CLOVE
0.0119
0.0383
VARREX
0.0001
0,0383
VARAUC
0.0058
0.0383
D.3 Analysis of Loss Values
Here we present analysis of the unpenalized loss after convergence in both real-data experiments. We performed separate analyses on pairs of data points from the dominant type during training (majority), and those from the other type (minority). Additionally, we separated positive pairs (y = 1) and negative pairs (y = 0). Figure 9 displays histograms illustrating the differences between losses on the training set obtained with the representation learned using ERM (gERM). and those obtained using our algorithm with VarAUC penalty (gvarAuc):
Diff,j ((raj, Voy; dgmaw) ((zaj. Viji d.gware).
Positive values of the differences correspond to higher losses for ERM.
In both experiments, when examining negative pairs from the minority group, shown in the top-left histograms, most of the observed differences are positive. This indicates that the ERM losses for these pairs higher compared to the losses obtained for the representation trained with the VarAUC
25

---

penalty. The disparities are smaller for the other three groups: majority negative pairs, minority negative pairs, and minority positive pairs. Among these groups, ERM performs better on positive pairs
0.06
0.9
0.05
AUC
0.8
0.7
0.02
0.01
0.6
Hierarchical
CLOVE
VarAUC
100
200
300
400
500
600
700
EAM
IRM CLOVE
VarREx
Points (pairs)
Hierarchical
0.05
0,90
0.85
Imgortance
0.03
0.02
0.70
0.65
ERM
IRM
VerREx
0.60
300
500
ERM
VarREx
Data Points (pairs)
Hierarchical
CLOvE
VarAUC
Figure 8: Additional Simulation Results. Top row: p= 0.05, Bottom row: 0.3. Left: Average AUC progress over 10 repetitions of the simulation. Solid lines correspond to performance on test data (distribution shift scenario), dashed lines show performance on data sampled from the same distribution as training data (in-distribution scenario). Right: Average feature importance results over 10 repetitions.
26

---

[e] CelabA Dataset
STHIC Dataset
Blord Omenoiis bringg
Not skand Emsuto by in tralringg)
Pyyrphuliiaa Ontrorily auring)
Iaceeridee Imaaelle intraingg
0.5
532
0.00
3:2
0.00-
0.3
0.3
-01
0.3
-33
5.
-02
-03
0.8
0.1
0.20
3.5
01 6.15-
3.0
0.35
0.2
2..05
3.00 0:03
-2.1 0.7
-0.555-0 025 2.00 5005
5.000
-5.80-3025 0551 0225 1.000
Figure 9: Analysis of Loss Differences. Histograms of differences between ERM and our algorithm with VarAUC penalty are shown for two experiments in separate sub-figures: (a) CelebA dataset, (b) ETHEC dataset. The top rows show differences for negative pairs (y = 0), bottom ones show differences for positive pairs (y 1). In each sub-figure the left column corresponds to the minority type and right one to the majority. dotted black line marks difference of Positive values correspond to higher losses for ERM.
27

---

E
Datasets
Treim
Test
Figure 10: Sample Images from the CelebA Dataset. Top: random sample of the training data with 95% non-blond people. Bottom: a random sample of the test data with 95% blond people.
Pluto
lile
Nic
Epiphron
Ficla
Didyma
Achine
Aristaeuss
Didyma
Thersamon
Thai
tolus
Virgaureae
Trappi
Alcon
M
Figure 11: Sample Images from the ETHEC Dataset. Top: a sample of the the training data species of the Lycaenidae family and 1 from the Nymphalidae family. Bottom: a sample of the test data where the proportion of the families is reversed. Nymphalidae species names are marked bold.
F Implementation Details
A link to a permanent repository with code to reproduce our results is included in the main text. The data-related parameters of our experiments described in the main text. In all our experiments we used margin of m 0.5 for the contrastive loss and Adam (Kingma Ba, 2014) optimizer to train all models.
For the CLOvE penalty we used Laplacian kernel k(r,r') e -r-r'l with width of 0.4 as originally suggested by Kumar et al. (2018).
For optimization of the VarAUC objective we disregard the finite sample correction N in the implementation since n is very small compared N.. In practice, we minimize the standard deviation instead of the variance in both variance based penalties, and the hyperparameters reported accordingly.
In our scenario where the attribute of interest is unknown, we generated synthetic attribute for hyperparameter selection using Principle Components (PC). We ranked examples based on their first PC component values, classifying the top 10% as positive and the rest as negative. Hyperparameters all methods were chosen via grid-search in single experiment repetition, ensuring robustness against this synthetic attribute. Notably, the experiments themselves did not involve the PC attribute; instead, they focused on dimension swapping in simulations and attributes like hair color or species family in CelebA and ETHEC experiments.
The grid search produced almost identical hyperparameters for all three p values. We observed that performance converged to the same value when employing hyperparameters derived from cross- validation for one P value, those selected for another. Therefore, for simplicity we repeated simulations using the same hyperparameters, determined based on the grid search results for 0.1 (the intermediate parameter value). Similarly, minimal differences in optimal learning rates were observed among the methods within experiment and therefore a shared learning rate was used for each experiment. To emphasize the improvement OOD methods over the ERM baseline, we used the learning rate optimized for the ERM method. Large differences were observed in optimal
28

---

regularization factors, and therefore these parameters (as well as method-specific parameters) were shared. All hyper-parameters are reported in Table
All models were initialized with identical weights, and trained on identical data splits.
All the code in this work was implemented in Python 3.10. We used the TensorFlow 2.13 and TensorFlow Addons 0.21 packages. For evaluation we used the auc function from scikit- learn 1.2. The CelebA dataset was loaded through TensorFlow Datasets 4.9 and pandas 1.5 was used to process the ETHEC dataset. Statistical tests were performed using ttest_rel and false wwwodddssvvveycccoooooooo Matplotlib 3.7.
functions from scipy.stats 1.11.4. All figures were generated using
The IRM implementation was adapted from the source code of the paper, available at https://gt tiggtb........ com//fcceookreseerchh a aaaa
We ran all experiments on a single A100 cloud GPU. For simulations, each full repetition of the experiment (comparing all methods) required on average 2.06 hours. Each repetition on the ETHEC dataset took 7.38 hours on average, and on the CelebA dataset 11.52 hours.
Table 5: Hyper Parameters.
ERM
IRM
CLOVE
VARREX
VARAUC
LEARNING RATE REGULARIZZTIIN FACTOR À
0.01
0.01
0.01
0.01
0.01
SIMULATIONS
0.01
0.05
3.0
1.5
NETWORK WEIGHT REGULARIZER
0.01
LEARNING RATE
10
10-5
10-5
10
10 5
CELEBA
REGULARIZATION FACTOR À
0.1
0.085
0.01
0.2
NETWORK WEIGHT REGULARIZER
0.01
LEARNING RATE
10-3
10-3
10
10-3
10 10-3
ETHEC
REGULARIZATION FACTOR À
0.02
0.05
0.1
0.2
NETWORK WEIGHT REGULARIZER
0.01
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: Both abstract and the introduction (last 2 paragraphs) accurately state the main contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Main limitations are discussed in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification: All assumptions are clearly stated and full proofs to the theoretical claims appear Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the details are provided in Section $ and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: The datasets are publicly available and code implementing all our results is submitted with the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides ottp://sseoooeeeemessssooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wttgg/uddeo/oouuuueeeeeee ssinoolicyy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We specify all hyperparameters and training details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform statistical testing to provide significance of our results and report FDR adjusted p- p-values. For all reported results we include either error bars the main text, or when other visualizations are chosen we report means and standard deviations in Appendix [
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All resources including GPU information and run times are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the wwtpherwwpppppppppieeeeepc.cpeee
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms with the provided code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of learning robust data representations. It is not tied any particular applications and therefore we do an immediate risk for negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
33

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The only asset used is the IRM implementation. The corresponding paper is cited and explicitly mention this in Appendix F while providing also reference for the code itself.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve human subjects and did not use crowd sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects and did not use crowd sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "fi36Cw92lV", "reviewer": "Reviewer_LBt4", "summary": "The paper treats the problem of learning models for zero-shot open-world classification settings (open-world meaning previously unseen classes might appear at test time) that are robust to distribution shifts.\n\nThe proposed approach consists of two stages. In the first stage, synthetic environments $S_i$ are sampled from the training data following a hierarchical sampling approach, where first classes and then data pairs according to sampled classes are sampled. \nIn the second stage, the model is updated to minimise a loss composed of standard ERM and the variance over environment AUC scores.\n\nThe benefits of the method are demonstrated on synthetic data, CelebA, and ETHEC (where also on the latter two a distribution shift is introduced synthetically).", "questions": "- How was the number of synthetic environments in Experiments 1 and 2 on real data chosen? What values of $\\alpha$ from Section 4.4 do they correspond to?", "limitations": "Limitations are briefly discussed in the Discussion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach to generate synthetic environments through hierarchical sampling seems neat and novel (even though the idea of generating synthetic environments for learning robust models is not novel, see weaknesses)\n- Adjusting the performance metric in the variance regularisation term for zero-shot verification, using AUC on embedding distances instead of loss (like in VaRex) seems like a nice way of avoiding performance plateaus and enables better performance in the conducted experiments\n- In the experiments, the proposed method shows significant performance gains over ERM\n- It is good to see a theoretical derivation of the necessary number of sampled environments to achieve a minimum number of examples from each class in at least one environment during the hierarchical sampling with a certain probability (Section 4.4). I believe this result should be made more prominent in form of a Proposition with a proof (in the appendix).", "weaknesses": "- Lack of baselines wrt generation of synthetic environments: The idea of generating synthetic environments to learn models that are robust to distribution shift is not new, and as such the proposed approach should have been compared to existing methods for this. For example, it would be interesting to see how the approach proposed by the authors compares to the approach of the 'Environment Inference for Invariant Learning' paper by Creager et al. (2021).\n- The real data experiments are still semi-synthetic in the sense that the distribution shift is introduced synthetically (and is quite stark). I do understand that finding a dataset that has a stark enough distribution shift of one attribute inherently in it is hard or even impossible, and the synthetic shifts are good for highlighting the potential merits of the proposed approach. However, what is missing is a report on the performance of the proposed approach (in comparison to baselines) on the unshifted train and test sets of CelebA and ETHEC, to ensure that there is no performance trade-off. \n\nMinor:\n- Figure 2 is never referred to in the text - as a result it is unclear what its purpose is.\n- It would be helpful to refer to the result of Section 4.4 already in Section 4.1, when it is claimed that 'hierarchical sampling results in diverse mixtures of any unknown attribute' and that 'smaller subsets with $k < N_c$ classes are likely to exhibit distinct attribute distributions' Section 4.4. backs up these claims. \n- L 235 typo at the end of the line -> (10)\n- Line 151: $h$ needs to be defined before referring to it in an equation, or immediately after the equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and questions. Below we address the raised weaknesses and questions.\n\n**Weaknesses:**\n\n1. *Comparison with Creager et al. (2021):*  \nThank you for referring us to Creager et al. (2021). We found their work very interesting and will cite it in our related work.  \nThe main difference between our construction of environments and theirs is that Creager et al. (2021) infer the worst-case environments for a fixed classifier (e.g., trained via ERM). In contrast, we consider a shift in an unknown class attribute, which is not necessarily the worst-case and may be uncorrelated or misaligned with the worst-case scenario.  \nHowever, in our synthetic data simulations (but not real-data experiments), we demonstrated our method on the worst-case shift. Therefore, following your suggestion, we compared the performance of our method using our environments versus those of Creager et al. (2021).   \n*The results showed no statistically significant improvement over ERM when using Creager et al.'s environments. We included the results in Figure 1 in the rebuttal figures file.*  \nAnalyzing the results, we discovered that in the context of contrastive learning, Creager et al.'s  assigned environments were almost random, as the optimized soft-assignment q barely changed from the random initialization. We attribute this to Creager et al.'s method being based on the IRM objective, which directly applies to gradients that are known to be noisy in contrastive learning (as discussed at the end of Chapter 5.1). This finding aligns with our broader results, which show that applying the IRM penalty, even on our environments where other penalties provide improvement, does not yield significant improvement over ERM.\n\n2. *Performance on unshifted distributions:*  \nThe performance of the proposed approach on the unshifted train and test sets of CelebA and ETHEC you mentioned is indeed included in figure 4 for the simulations and in table 3 in the Appendix and indeed shows no negative effect on unshifted distribution performance. We agree it might be preferable to move it to the main article.\n\n**Question:**\nIn both experiments, we set the minimal $\\rho$ to 0.15. Therefore, the number of synthetic environments in Experiment 1 (CelebA, with 450 classes after filtering) and Experiment 2 (ETHEC, with 117 classes after filtering) corresponds to $\\alpha=0.05$ and $\\alpha=0.09$, respectively. We will specify this in the manuscript as well.\n\n**Minor:**\n1. Figure 2: Thank you for pointing this out, we will add an explanation in the manuscript. Figure 2 shows that the optimal weights of dimensions corresponding to a given type are larger for higher proportions of that type, and that the discrepancy between the weights increases as the ratio of the type variances grows.\n2. Location of Section Section 4.4: We accept the recommendation to move Section 4.1 immediately after Section 4.4 and thank the reviewer for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VzTKY37jaG", "reviewer": "Reviewer_X2Y7", "summary": "Zero-shot learning classifiers face the challenge of distribution shifts, where the distribution of new classes differs significantly from that of the training data. In this paper, the authors introduce a novel algorithm to address this problem by creating robust representations through hierarchical sampling and environment balancing penalization. \n\nExperimental results also demonstrate a performance increase compared to the baseline ERM model on several real-world datasets.", "questions": "Please see the Weakness section above.\n\nAdditional questions:\n\n1. Although the authors mention how to calculate the number of environments, it's better to include an ablation study to test the performance with different numbers of environments.", "limitations": "The authors acknowledge the limitations in their paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to understand.\n2. The paper proposes a new model that enables handling unknown attributes for distribution shifts and addresses new classes at test time.\n3. The method is tested through both simulations and real-world experiments.", "weaknesses": "1. Some parameters need to be clearly defined, for example, $\\rho_{tr}$, $\\rho_{te}$, and $y_{uv}$ in Eq (4).\n2. The proposed method creates multiple environments and computes penalties across them. What is the computational complexity? It's also beneficial to discuss the time complexity of Algorithm 1.\n3. Figure 5 and Figure 6 do not straightforwardly show the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. Below we address the raised weaknesses and questions:\n\n**Weaknesses:**  \n1. *Definition of parameters:*  \n$\\rho_{tr}$ and $\\rho_{te}$ correspond to the the proportion of type $a_1$ classes in train and test sets correspondingly and are defined in lines 139-140, but we will make their definition more clear. $y_{uv}$ is simply the label for the pair of datapoints indexed by u and v: we need two sets of indices since Eq. 4 treats pairs from the same class ($y_{ij}=1$) and those from different classes ($y_{uv}=0$) separately.\n2. *Complexity:*  \nAll OOD approaches from the considered family (see Section 2) involve optimization across multiple environments. Therefore, the complexity of each step includes: (i) the complexity of generating the environments, (ii) the original complexity of a single iteration over the representation neural network multiplied by the number of environments, and (iii) the computation of the aggregated penalty.  \nComponent (ii) is shared among all approaches from the considered family, while (iii) is negligible compared to training networks, usually involving simple operations like calculating the mean or variance of scalars. Thus, differences between the methods may arise due to (i). However, the additional training time due to the proposed hierarchical sampling, even when compared to training a simple linear representation, is also negligible since hierarchical sampling can be performed offline before training, as shown in the code provided in the supplementary material. We will include a note about this in the manuscript.  \nFor example, in the species recognition task, standard sampling takes 1.03 seconds, a naive implementation of hierarchical sampling takes 16.6 seconds, and training the representation takes 1 hour and 6 minutes. Thus, the additional time due to hierarchical sampling is less than 0.4% and similar results (all less than 0.5%) are achieved in all our experiments.\n3. *Exact performances:*\nSimulation performance is shown in Figure 4, with exact performances reported in Table 1 in the appendix. For real-data experiments, exact performances (and corresponding p-values) are provided in Tables 3 and 4 in the appendix.  \nFigures 5 and 6 highlight other important aspects: the learned feature importance in the simulations, demonstrating that our method relies on the intended features, and the performance of our method compared to ERM (i.e., the improvement over ERM).\n\n**Question:** Unlike the standard OOD setting, our environments are synthetically generated via sampling, resulting in many more environments (Nc over k) than specified in Equation 10. Therefore, there is no reason to use fewer environments, and using more can only enhance the performance of our method. However, we will include an analysis in the appendix that examines performance as a function of the number of environments used for both ERM and our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0a2jNyQoau", "reviewer": "Reviewer_sFfJ", "summary": "This paper proposes a robust representation learning method that could assume the shift between seen classes and unseen classes.", "questions": "See weakness", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Good presentation and sound method.", "weaknesses": "Lack the experiments on the most popular benchmark of zero-shot learning [1] and comparison to some SOTAs, e.g. [2][3].\n\n[1] Zero-shot learning-the good, the bad and the ugly[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n[2] Rebalanced zero-shot learning[J]. IEEE Transactions on Image Processing, 2023.\n[3] Transzero: Attribute-guided transformer for zero-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and the provided references. \n\nWe did not use the datasets mentioned in your references since they either (i) do not have labeled attributes (e.g., the SUN dataset), (ii) the provided attributes correlate with the data-point label such that shifts in them do not affect ERM, making interventions unnecessary (e.g., the CUBS dataset), or (iii) include too few classes (e.g., the AWA2 dataset with only 50 animal classes).\n\n\nTherefore, we performed the real-data experiments on the  CelebA dataset (which is one of the most popular zero-shor benchmarks), and the ETHEC dataset, since both include a large number of classes and labeled attributes in addition to the primary labels (e.g., butterfly family in ETHEC and hair color in CelebA, in addition to species and person identity, respectively).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nPWaBc6n7v", "reviewer": "Reviewer_YoYU", "summary": "This paper first investigates the effect of class distribution changes on comparative zero-sample learning by proposing and analysing a class distribution shifts parameter model, leading to the idea that loss minimisation leads to poor performance of representations over the class distribution shifts. Based on this finding, the authors utilise hierarchical sub-sampling and OOD environment balancing methods to obtain robust representations and address the poor performance caused by class distribution changes in zero-sample learning, and experimentally validate the effectiveness of the methods.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1- This paper studies the distribution bias problem caused by challenging unknown attributes in zero-shot learning and proposes an effective solution, which is important and innovative for solving the distribution bias problem in zero-shot learning.\n\n2- The structure is clear. It enables the reader to quickly follow the research ideas and understand the content of each section.\n\n3- Figures and tables are clear and accurate. The figures and tables in this paper are concise and clear, effectively support the ideas or conclusions, and enable the reader to grasp the critical information quickly.\n\n4- Comparison and ablation studies are comprehensive. The authors demonstrated the superiority of their method through many experiments and analysed various factors.", "weaknesses": "1- Authors should describe their proposed soft-AUC trends in detail and analyse the penalty to help readers understand how they play a role.\n\n2- In Experiment 1, the authors used the attribute blonde hair to shift the class distribution, but we know that some people may be hairless, so can the attribute gender be used to shift the class distribution?\n\n3- The language of this paper needs to be scrutinized and improved. For example, redundant phrases such as \"with respect to\" (line 32) should be avoided. In addition, there are some grammatical errors that need to be improved, such as \"assumes\" in line 333 should be changed to \"assume\", and \"leverages\" in line 350 should be changed to \"average\".\n\n4- WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear, such as “OOD”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Soft-AUC:**\nThe use of soft-AUC instead of the standard AUC score is intended to make the complete loss (including the penalty) differentiable, thus enabling gradient-based optimization. As mentioned in lines 218-219, the soft-AUC pointwise converges to the standard AUC score as $\\beta$ approaches infinity.  \nFor a fixed $\\beta$, define $f(t_{1},t_{2})=I[t_{1}<t_{2}]$ \nand $g_{\\beta}(t_{1},t_{2})=\\frac{1}{1+e^{-\\beta(t_{2}-t_{1})}}$.\nDenoting $t=t_{2}-t_{1}$ we have \n$$\\left\\Vert f(t_{1},t_{2})-g_{\\beta}(t_{1},t_{2})\\right\\Vert ^{2}=\\int_{0}^{\\infty}\\left(1-\\frac{1}{1+e^{-\\beta t}}\\right)^{2}dt+\\intop_{-\\infty}^{0}\\left(0-\\frac{1}{1+e^{-\\beta t}}\\right)^{2}dt=\\frac{1}{\\beta}\\left[2\\log2-1\\right].$$\nWe will include this calculation in the appendix, along with a graph illustrating the differences, which we have added as Figure 2 in the rebuttal figures file.\n2. **Gender attribute for the CelebA dataset:**  \nIn principle, the male/female attribute could be used instead. However, on the CelebA dataset, representations that distinguish well between male individuals also distinguish well between female ones, and vice versa, for a reasonable representation network. Thus, shifts in gender do not call for interventions on the CelebA dataset, as all methods would be equivalent to ERM.  \nThis is not the case for the attribute of being blond. Our experiments show that the representation learned via ERM on mainly non-blond people (including hairless individuals, as this binary indication distinguishes blond people from all others) fails to separate blond people effectively. \n\nWe thank the reviewer for pointing out the typos in 3 and 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their efforts in reviewing our paper. We address the concerns raised by each reviewer separately. We attach here the rebuttal figures file, which contains the additional figure referenced in our individual responses.

Meta-review: The paper proposes a method for making zero-shot classifiers robust to a particular class of distributions shifts, which the shift is localized in some unknown latent attribute variable. The paper is well-written and the authors have taken a great care in defining the setup precisely. It seems to be a reasonable extension of classical methods in the supervised classification setting that mitigates against label distribution shifts, where each synthetic environment often corresponds to a single class.

The reviewers share an unanimous agreement in favor of accepting the paper, and I agree with that assessment. 

I encourage the authors to address all of the comments raised in the reviews.
Decision: Accept (poster)