HYPO: Hyperspherical Out-Of-Distribution Generalization
Authors: Haoyue Bai
Yifei Ming
Julian Katz-Samuels
Yixuan Li
Venue: ICLR 2024 poster (2024)
Keywords: OOD Generalization
Domain Generalization
Hypersphere
Primary area: general machine learning (i.e., none of the above)

Abstract: Out-of-distribution (OOD) generalization is critical for machine learning models deployed in the real world. However, achieving this can be fundamentally challenging, as it requires the ability to learn invariant features across different domains or environments. In this paper, we propose a novel framework HYPO (HYPerspherical OOD generalization) that provably learns domain-invariant representations in a hyperspherical space. In particular, our hyperspherical learning algorithm is guided by intra-class variation and inter-class separation principles—ensuring that features from the same class (across different training domains) are closely aligned with their class prototypes, while different class prototypes are maximally separated. We further provide theoretical justifications on how our prototypical learning objective improves the OOD generalization bound. Through extensive experiments on challenging OOD benchmarks, we demonstrate that our approach outperforms competitive baselines and achieves superior performance. Code is available at https://github.com/deeplearning-wisc/hypo.

Published as a conference paper at ICLR 2024
HYPO: HYPERSPHERICAL OUT-OF-DISTRIBUTION GENERALIZATION
Haoyue Bai'; Yifei Ming'; Julian Katz-Samuels Yixuan Li'
Department of Computer Sciences, University of Wisconsin-Madison" Amazon²
(ba i ihayuue,,, alvinming, sharonli) @css @cs wisc.edu, jkatzsamuels$@ma com
ABSTRACT
Out-of-distribution (OOD) generalization is critical for machine learning models de- ployed in the real world. However, achieving this can be fundamentally challenging, requires the ability learn invariant features across different domains or envi- ronments. In this paper, we propose a novel framework HYPO (HYPerspherical OOD generalization) that provably learns domain-invarien representations in a hyperspherical space. In particular, our hyperspherical learning algorithm is guided by intra-class variation and inter-class separation principles ensiuiins-e that features from the same class (across different training domains) are closely aligned with their class prototypes, while different class prototypes are maximally separated. We further provide theoretical justifications on how our prototypical learning ob- jective improves the OOD generalization bound. Through extensive experiments challenging OOD benchmarks, demonstrate that approach outperforms competitive baselines and achieves superior performance. Code available at https:// / hwgggttthh../////gg///g com detp//erlppeeeeneeeeeeeee srisc/hyp..
INTRODUCTION
Deploying machine learning models in real-world settings presents critical challenge of generalizing under distributional shifts. These shifts are common due to mismatches between the training and test data distributions. For instance, in autonomous driving, model trained on in-distribution (ID) data collected under sunny weather conditions is expected to perform well in out-of of-issiiiiittoionnn (OOD) scenarios, such as rain or snow. This underscores the importance the OOD generalization problem, which involves learning a predictor that can generalize across all possible environments, despite being trained on a finite subset of training environments.
A plethora of OOD generalization algorithms has been developed in recent years (Zhou et al., 2022), where a central theme to learn nonain-invariant gepresentations-feateee meaningful across different environments (domains) and can generalize the unseen test environment. Recently, Ye et al. (2021) theoretically showed that the OOD generalization error can be bounded in terms of intra-class variation and inter-class separation. Intra-class variation measures the stability of representations across different environments, while inter-class separation assesses the dispersion of features among different classes. Ideally, features should display low variation and high separation, in order to generalize well to OOD data (formally described in Section 3). Despite the theoretical analysis, research question remains open in the field:
that are consistent and
RQ: How to design a practical learning algorithm that directly achieves these two properties, and what theoretical guarantees can the algorithm offer?
To address the question, this paper presents a learning framework HYPO (HYPerspherical OOD generalization), which provably learns domain-invariant invariant representations in the hyperspherical space with unit norm (Section 4). Our key idea to promote low variation (aligning representation across
Equua contribution. Correspondence to Yifei Ming and Yixuan Li This work is related to the author's position Amazon.

---

Published as a conference paper at ICLR 2024
domains for every class) and high separation (separating prototypes across different classes). In particular, the learning objective shapes the embeddings such that samples from the same class (across all training environments) gravitate towards their corresponding class prototype, while different class prototypes are maximally separated. The two losses in our objective function can be viewed as optimizing the key properties of intra-class variation and inter-class separation, respectively. Since samples are encouraged to have a small distance with respect to their class prototypes, the resulting embedding geometry can have a small distribution discrepancy across domains and benefits OOD generalization. Geometrically, we show that our loss function can be understood through the lens of maximum likelihood estimation under the classic von Hises-Fisher distribution.
Empirical contribution. Empirically, we demonstrate strong OOD generalization performance by extensively evaluating HYPO on common benchmarks (Section 5). On the CIFAR-10 10 (ID) vs. CIFAR- 10-Corruption (OOD) task, HYPO substantially improves the OOD generalization accuracy on challenging cases such as Gaussian noise, from 78.09% to 85.21%. Furthermore, we establish superior performance on popular domain generalization benchmarks, including PACS, Office-Homee VLCS, etc. For example, we achieve 88.0% accuracy PACS which outperforms the best loss- based method by %.. This improvement is non-trivial using standard stochastic gradient descent optimization. When coupling our loss with specialized optimization SWAD (Cha al., 2021), the accuracy is further increased to 89%. We provide visualization and quantitative analysis to verify that features learned by HYPO indeed achieve low intra-class variation and high inter-class separation.
Theoretical insight. We provide theoretical justification for how HYPO can guarantee improved OOD generalization, supporting our empirical findings. Our theory complements Ye et al. (2021), which does not provide a loss for optimizing the intra-class variation or inter-class separation. Thus, key contribution of this paper is to provide a crucial link between provable understanding and practical algorithm for 00D generalization in the hypersphere. In particular, our Theorem 6.1 shows that when the model is trained with our loss function, we can upper bound intra-class variation, key quantity to bound OOD generalization error. For a learnable OOD generalization task, the upper bound on generalization error is determined by the variation estimate the training environments, which is effectively reduced by our loss function under sufficient sample size and expressiveness of the neural network.
PROBLEM SETUP
We consider a multi-class classification task that involves a pair of random variables (X, Y) over
instances E X C _d and corresponding labels y E y := {1, 2, ..., ,.. The joint distribution of X and is unknown and represented by IP xxx.. The goal is to learn a predictor function, f ; - RC, that can accurately predict the label y for input x, where (x. v) Fxy.
Unlike in standard supervised learning tasks, the out-of-distribution (OOD) generalization problem is challenged by the fact that one cannot sample directly from Pxy. Instead, we can only sample (X, Y) under limited environmental conditions, each of which corrupts or varies the data differently. For example, in autonomous driving, these environmental conditions may represent different weathering conditions such as snow, rain, etc. We formalize this notion of environmental variations with a set of environments or domains Eau- Sample pairs (Xe, Y*) are randomly drawn from environment e. In practice, we may only have samples from a finite subset of available environments Eyail C E Given Egvall, the goal to learn a predictor ff that can generalize across all possible environments. The problem is stated formally below.
Definition 2.1 (OOD Generalization), Let Eavail C. Eall be a set of training environments, and assume that for each environment E Evaall we have a dataset D = {[xx, ;-> sampled i.i.d. from an unknown distribution yxy. The goal of OOD generalization is to find classifier f*, using the data from the datasets D=, that minimizes the worst-case risk over the entire entiry of environments Emn:
min max Ep,, ((((X)) JEF eEEa
where F is hypothesis space and ((-.-) is the loss function.
The problem is challenging since we do not have access to data from domains outside Eyvii- In particular, the task is commonly referred to multi-source domain generalization when |Ewvit] 1.
2

---

Published as a conference paper at ICLR 2024
3 MOTIVATION OF ALGORITHM DESIGN
Our work is motivated by the theoretical findings in Ye et al. (2021), which shows that the OOD generalization performance can be bounded in terms of intra-class variation and inter-class separation with respect to various environments. The formal definitions are given as follows.
Definition 3.1 (Intra-class variation). The variation feature & across a domain set & is
V(d, 8) = max sup yEy e,''EE
((P()) P(O" lu))
where p(P, ; Q) is a symmetric distance (e.g. Wasserstein distance, total variation, Hellinger distance) between distributions, and (p v) denotes the class-conditional distribution for features of samples in environment e.
Definition 3.2 (Inter-class separation'), ). The separation feature &b across domain set is
I)(0,8)= = C(C 1)
Σ wHV" D.Y EY
minn(((("1)))E(6)))))
CEE
The intra-class variation V(o, E) measures the stability of feature & over the domains in & and the inter-class separation I(Ø, E) captures the ability of & distinguish different labels. Ideally, features should display high separation and low variation.
Definition 3.3. The OOD generalization error of classifier f is defined as follows:
err(J) max Ep, ((((X)),Y") -max eEEmat
Epxy (((()))YY)
eEE.i
which is bounded by the variation estimate on Evviil with the following theorem.
Theorem 3.1 (OOD error upper bound, informal (Ye et al., 2021)). Suppose the loss function El:;-) is bounded by [0, B). For a learnable OOD generalization problem with sufficient inter-class separation. the OOD generalization error err(f) can be upper bounded by
err S ((VrUP Eavail)
(a-d)²
for some O, and youp (h, Eavail) SUP V (3 Enval) is the inter-class variation, h(()) Rd is the feature vector, and B is vector in unit hypersphere Sd-1 {B Rd ||31|2 1). and is classifier based on normalized feature h.
Remarks. The Theorem above suggests that both low intra-class variation and high inter-class separation are desirable properties for theoretically grounded OOD generalization. Note that in the full formal Theorem (see Appendix C), maintaining the inter-class separation is necessary for the learnability of the OOD generalization problem (Def. C.2). In other words, when the learned embeddings exhibit high inter-class separation, the problem becomes learnable. In this context, bounding intra-class variation becomes crucial for reducing the OOD generalization error.
Despite the theoretical underpinnings, it remains unknown to the field how to design a practical learning algorithm that directly achieves these two properties, and what theoretical guarantees can the algorithm offer. This motivates our work.
To reduce the OOD generalization error, our key motivation is to design hyperspheri- cal learning algorithm that directly promotes low variation (aligning representation across domains for every class) and high separation (separating prototypes across different classes).
4 METHOD
Following the motivation Section 3, we now introduce the details of the learning algorithm HYPO (HYPerspherical 0OD generalization), which is designed to promote domain invariant representations
Referred "Informativeness" in Ye al. (2021).
3

---

Published as a conference paper at ICLR 2024
in the hyperspherical space. The key idea is to shape the hyperspherical embedding space that samples from the same class (across all training environments Evvii) are closely aligned with the corresponding class prototype. Since all points are encouraged to have small distance with respect to the class prototypes, the resulting embedding geometry can have a small distribution discrepancy across domains and hence benefits OOD generalization. In what follows, we first introduce the learning objective (Section 4.1), and then we discuss the geometrical interpretation of the loss and embedding (Section 4.2). We will provide theoretical justification for HYPO in Section 6, which leads to a provably smaller intra-class variation, key quantity to bound OOD generalization error.
4.1 HYPERSPHERICAL LEARNING FOR OOD GENERALIZATION
Loss function. The learning algorithm motivated to directly optimize the two criteria: intra-class variation and inter-class separation. At high level, HYPO aims to learn embeddings for each sample the training environments by maintaining class prototype vector μe IR for each class E 21,2, ..., C}. To optimize for low variation, the loss encourages the feature embedding of sample to be close to its class prototype. To optimize for high separation, the loss encourages different class prototypes to be far apart from each other.
we consider a deep neural network h x' Rd that maps an input E X to a feature embedding ă := h(x). The loss operates the normalized feature embedding := 2/ / The normalized embeddings are also referred to as hyperspherical embeddings, since they are on a unit hypersphere, denoted as Sd-1 {z Rd
Specifically,
||z||2 1}. The loss is formalized as follows:
C =
1 N BE E 1=1
log Ej= exp (of U3/T)
exp (z. Mc(i/T)
+
C log
C
1
Σ jaJjY
exp
C
Cor variation
1 separation
where N is the number of samples, is the temperature, is the normalized feature embedding. and μ4, is the prototype embedding for class c. While hyperspherical learning algorithms have been studied in other context (Mettes al., 2019; Khosla et al., 2020; Ming al., 2023), none of the prior works explored its provable connection to domain generalization, which is our distinct contribution. We will theoretically show in Section 6 that minimizing our loss function effectively reduces intra-class variation, a key quantity to bound OOD generalization error.
The training objective in Equation 5 can be efficiently optimized end-to-end. During training, an important step is to estimate the class prototype μe for each class c E {1,2, C}. The class- conditional prototypes can be updated in 2020):
aonooentiia-moving-vvraag manner (EMA) (Li et al.,
(5) where the prototype μac for class is updated during training as the moving average of all embeddings with label c, and denotes the normalized embedding of samples of class An end-to-end pseudo algorithm is summarized in Appendix A.
ee = Normalize(ayu:: + (1 - a))z, Ve {1, .1.
Class prediction. In testing, classification is conducted by identifying the closest class prototype: ý argmax.etc] fe(x), where fx( z He and z
h(x)
the normalized feature embedding.
[
4.2 GEOMETRICAL INTERPRETATION OF LOSS AND EMBEDDING
Geometrically, the loss function above can interpreted as learning embeddings located on the surface of a unit hypersphere. The hyperspherical embeddings can be modeled by the von Mises- Fisher (vMF) distribution, a well-known distribution in directional statistics (Jupp & Mardia, 2009). For unit vector e dd in class c, the probability density function is defined as
p(z y = c) Za(ss) exp(1141 z),
(6)
where mec E d denotes the mean direction of the class c, к 2 0 denotes the concentration of the distribution around ph. and (k) denotes the normalization factor. A larger indicates higher concentration around the class center. In the extreme case of = 0, the samples are distributed uniformly on the hypersphere.
4

---

Published as a conference paper at ICLR 2024
Under this probabilistic model, embedding is signed to the class c with the following probability
Zu(K) exp(KH
ply = e 2; {n, (3)g0))
Σj=1 Za(n)) exp(xtu
expl1e 2/7)
(7)
Ej=repp(11 2/T)
where 1/Kx denotes temperature parameter.
Maximum likelihood view. Notably, minimizing the first term in loss (ef. Eq. is equivalent per- forming maximum likelihood estimation under the vMF distribution:
Figure 1: Illustration of hyperspherical embed-
dings. Images are from PACS (li et al., 2017).
N argmax, II Pls Xsi (M, where (xi, E
U D eEEmi
where is the index of sample, j is the index of the class, and N is the size of the training set. In effect, this loss encourages each ID sample have high probability assigned the correct class in the mixtures of the vMF distributions.
EXPERIMENTS
In this section, we show that HYPO achieves strong OOD generalization performance in practice, establishing competitive performance on several benchmarks. In what follows, describe the experimental setup in Section 5.1, followed by main results and analysis in Section 5.2.
5.1 EXPERIMENTAL. SETUP
Datasets. Following the common benchmarks in literature, we use CIFAR-10 (Krizhevsky et al., 2009) as the in-distribution data. We use CIFAR-IO-C (Hendrycks & Dietterich, 2019)
as OOD data, with 19 different common corruption applied to CIFAR-10. In addition to CIFAR-10, we conduct experiments on popular benchmarks including PACS (Li et al., 2017), Office-Home (Gulrajani & Lopez-Paz, 2020), and VLCS (Gulrajani & Lopez-Paz, 2020) val- idate the generalization performance. PACS contains 4 domains/envirommenss (photo, art painting, cartoon, sketch) with classes (dog, e elephant, gi garaf ra fe, guitar, horse, house, person). Office-Home comprises four different domains: art, clipart, product, and real. Results on additional OOD datasets Terra Incognita (Gulrajani & Lopez-Paz, 2020), and ImageNet can be found in Appendix F and Appendix G.
Evaluation metrics. We report the following two metrics: (1) classification accuracy (ID Acc.) for ID generalization, and (2) OOD classification accuracy (OOD Acc.) for OOD generalization.
Experimental details. In our main experiments, we use ResNet-18 for CIFAR-I0 and ResNet-50 for PACS, Office-Home, and VLCS. For these datasets, we use stochastic gradient descent with momentum 0.9, and weight decay 10-4 For CIFAR-10 10, we train the model from scratch for 500 and cosine scheduling, with a batch size of 512. Following common practice for contrastive losses (Chen et al., 2020; Khosla et al., 2020; Yao et al., 2022), we use an MLP projection head with one hidden layer to obtain features. The embedding (output) dimension is 128 for the projection head. We set the default temperature update factor a as 0.95. For PACS, Office-Home, and VLCS, we follow the common practice and initialize the network using ImageNet pre-trained weights. We fine-tune the network for 50 epochs. The embedding dimension 512 for the projection head. We adopt the leave-one-domain-our evaluation protocol and use the training domain validation for model selection (Gulrajani & Lopez-Paz, 2020), where the validation set is pooled from all training domains. Details on other hyperparameters are in Appendix D.
epochs using an initial learning rate
0.1 and the prototype
5

---

Published as a conference paper at ICLR 2024
Algorithm
PACS
Office-Home
VLCS
Average Acc. (%)
ERM (Vapnik, 1999)
85.5
67.6
77.5
76.7
CORAL (Sun & Saenku, 2016)
86.2
68.7
78.8
77.9
DANN (Ganin et al., 2016)
83.7
65.9
78.6
76.1
MIDG (Li al., 2018a)
84.9
66.8
77.2
76.3
CDANN al., 2018c)
82.6
77.5
MMD (Li al., 2018b)
84.7
66.4
77.5
76.2
IRM (Arjovsky et al.. 2019)
83.5
64.3
78.6
75.5
GroupDRO (Sagawa et al., 2020)
84.4
66.0
76.7
75.7
I-Mixup (Wang et al., 2020; Xu et al., 2020; Yan et al., 2020)
84.6
68.1
77.4
76.7
RSC (Huang et al., 2020)
85.2
65.5
77.1
75.9
ARM (Zhang al., 2021)
85.1
64.8
77.6
75.8
MTL (Blanchand al., 2021)
84,6
66.4
77.2
76.1
VREx (Krueger ct al., 2021)
84.9
66.4
78.3
76.5
Mixstyle (Zhou et al., 2021)
85.2
60.4
77.9
74.5
SelfReg (Kim el al., 2021)
85.6
67.9
77.8
77.1
SagNet (Nam et al., 2021)
86.3
68.1
77.8
77.4
GVRT (Min et al., 2022)
85.1
70.1
79.0
78.1
VNE (Kim et al., 2023)
86.9
65.9
78.1
77.0
HYPO (Ours)
88.0.g044
71.7.10.3
78.2
79.3
Table Comparison with domain generalization methods on the PACS, Office-Home, and VLCS. All methods are trained on ResNet-50. The model selection is based on a training domain validation set. To isolate the effect of loss functions, all methods are optimized using standard SGD. We report the average and std of our method. denotes the rounded standard error.
5.2 MAIN RESULTS AND ANALYSIS
HYPO excels on common corruption benchmarks. As shown in Figure HYPO achieves consistent improvement over the ERM baseline (trained with cross-entropy loss), on a variety of common corruptions. Our evaluation includes different corruptions including Gaussian noise, Snow, JPEG compression, Shot noise, Zoom blur, etc. The model is trained on CIFAR-10, without seeing any type of corruption data. In particular, our method brings significant improvement for challenging cases such as Gaussian noise, enhancing OOD accuracy from 78.09% to 85.21% (-7.12%). Complete results on all different corruption types are Appendix E.
ERM
Ours
75
Gausstian Noto
Snow
UPEB Comp
Shot Noisa
Zoon Blur
Frest
Motion Bar
impoito Hoiso
Preeiaee
Speakd Noie
Elnno Transform
Satario
Avg
Figure 2: Our method HYPO significantly improves the OOD generalization performance compared to ERM
on various OOD datascts w.r.t. CIFAR-10 (ID). Full results can he seen in Appendix E.
HYPO establishes competitive performance on popular benchmarks. Our method delivers superior results in the popular domain generalization tasks, as shown in Table 1. HYPO outperforms an extensive collection of common OOD generalization baselines on popular domain generalization datasets, including PACS, Office-Home VLCS. For instance, on PACS, HYPO improves the best loss-based method by 1.1%. Notably, this enhancement is non-trivial since we are not relying on specialized optimization algorithms such as SWAD (Cha et al., 2021). Later in our ablation, we show that coupling HYPO with SWAD can further boost the OOD generalization performance, establishing superior performance this challenging task.
With multiple training domains, we observe that it desirable to emphasize hard negative pairs when optimizing the inter-class separation. As depicted in Figure 3, the embeddings of negative pairs from the same domain but different classes (such as dog and elephant in art painting) can be quite close on the hypersphere. Therefore, it is more informative to separate such hard negative pairs. This can be enforced by a simple modification to the denominator of our variation loss (Eq. 11 in Appendix D), which we adopt for multi-source domain generalization tasks.

---

Published as a conference paper at ICLR 2024
Algorithm
Art painting
Cartoon
Photo
Sketch
Average Acc. (%)
PCL w/ SGD (Yao al., 2022)
88.0
78.8
98.1
80.3
86.3
HYPO w/ SGD (Ours)
87.2
82.3
98.0
84.5
88.0
PCL w/ SWAD (Yao al., 2022)
90.2
83.9
98.1
82.6
88.7
HYPO w/ SWAD (Ours)
90.5
84.6
97.7
83.2
89.0
Table 2: Results for comparing PCL and HYPO with SGD-based and SWAD-basee optimizations the PACS
benchmark. (*The performance reported in the original PCL paper Table is implicitly based on SWAD).
Dog protolype
Relations to PCL. PCL (Yao al., 2022) adapts proxy- based contrastive learning framework for domain generaliza- tion. We highlight several notable distinctions from ours: (1) While PCL offers no theoretical insights, HYPO is guided by theory. We provide formal theoretical justification that our method reduces intra-class variation which is essential to bounding OOD generalization error (see Section 6); (2) Our loss function formulation is different and can be rigorously interpreted as shaping vMF distributions of hyperspherical em- beddings (see Section 4.2), whereas PCL can not; (3) Unlike PCL (86.3% w/o SWAD), HYPO is able to achieve competitive performance (88.0%) without heavy reliance on special opti- mization SWAD (Cha et al., 2021), a dense and overfit-aware stochastic weight sampling (Izmailov al., 2018) strategy for OOD generalization. As shown in Table 2, we also conduct experiments in conjunction with SWAD. Compared PCI., HYPO achieves superior performance with 89% accuracy, which further demonstrates its advantage.
Elephant prototype
hard negative pairs
Figure 3: Illustration of hard negative pairs which share the domain (art painting) but have different class labels.
Visualization of embedding. Figure shows the UMAP (MeInnes et al., 2018) visualization of feature embeddings for ERM (left) vs. HYPO (right). The embeddings are extracted from models trained on PACS. The red, orange, and green points are from the in in-distribution, corresponding to painting (A), photo (P), and sketch (S) domains. The violet points are from the unseen OOD domain cartoon (C). There are two salient observations: (1) for any given class, the embeddings across domains Eall become significantly more aligned (and invariant) using our method compared to the ERM baseline. This directly verifies the low variation (ef. Equation 2) of our learned embedding. (2) The embeddings are well separated across different classes, and distributed more uniformly in the space than ERM. which verifies the high inter-class separation (cf. Equation 3) of our method. Overall, our observations well support the efficacy HYPO.
Guitar
A (tain))
A Ibrin! thair)
(rain)
S (tain)
House
Oraini (les0
Elephant
Elephant
Girafte
Dog
Horse
Person
Horse
Person
Dog
Girafin
Guitar
(a) ERM (high variation)
(b) HYPO (low variation)
Figure 4: UMAP (McInnes et al., 2018) visualization of the features when the model trained with CE vs. HYPO for PACS. The red, orange, and green points are from the in-distribution, which denote art painting (A), photo (P), and sketch (S). The violet points are from the unseen OOD domain cartoon (C).
Quantitative verification of intra-class variation. We provide empirical verification on intra-class variation in Figure 5, where the model is trained on PACS. We measure the intra-class variation with Sinkhorn divergence (entropy regularized Wasserstein distance). The horizontal axis (0)-(6) denotes
7

---

Published as a conference paper at ICLR 2024
ERM
Ours
114 /9)
9:15
0.22
19A: PP)
(4. 53
6:14
0.21
0.19
0.30
5:18
0.14
0.22
(14) 15
6:08
0.00
0.01
0:26
2:05
0.11
0.06
Sinkhom
5)
(P:
Figure 5: Intra-class variation for ERM (left) HYPO (right) PACS. For each class y, measure the Sinkhorn Divergence between the embeddings of each pair of domains. Our method results significantly lower intra-class variation across different pairs of training domains compared to ERM.
different classes, and the vertical axis denotes different pairs of training domains ('P', 'A', 'S'). Darker color indicates lower Sinkhorn divergence. We can see that method results in significantly lower intra-class variation compared to ERM, which aligns with our theoretical insights in Section 6.
Additional ablation studies. Due to space constraints, we defer additional experiments and ablations to the Appendix, including (I) results on other tasks from DomainBed (Appendix F); (2) results on large-scale benchmarks such ImageNer-100 (Appendix G); (3) ablation of different loss terms (Appendix H); (4) an analysis on the effect of and a (Appendix I).
WHY HYPO IMPROVES OUT-OF-DISTRIBEUS
GENERALIZATION?
In this section, we provide a formal justification of the loss function. Our main Theorem 6.1 gives a provable understanding of how the learning objective effectively reduces the variation estimate 1ssp Evait), thus directly reducing the OOD generalization error according to Theorem 3.1. For simplicity, we assume т
and denote the prototype vectors f1:.... MC E Sd-1 Let H. C {h: X + }d-1} denote the function class induced by the neural network.
Theorem 6.1 (Variation upper bound using HYPO). When samples aligned with class prototypes such that N/ Σz= M 1 8,
eforr some E (0, 1), then 38 E (0, 1), with probability at least
In(2/5) 1/6 + (Ep) E ysup (h, Eavai)) < 0(e1/3 +
N o,z
&N sup KEH
N
N
h(x;) where zj [h(x;)] " 01...., ON are Rademacher random variables and O(-) suppresses depen- dence on constants and |Evvill..
Implications. In Theorem 6.1, we can see that the upper bound consists of three factors: the optimization error, the Rademacher complexity of the given neural network, and the estimation error which becomes close to 0 as the number of samples N increases. Importantly, the term reflects how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have N EE= H ([j 2 1 1-e),, which is directly minimized by our proposed loss in Equation The above Theorem implies that when we train the model with the HYPO loss, we can effectively upper bound the intra-class variation, key term for bounding OOD generation performance by Theorem 3.1. I. In Section H, we provide empirical verification of our bound by estimating é, which is indeed close for models trained with HYPO loss. We defer proof details to Appendix C.
Necessity inter-class separation loss. We further present a theoretical analysis in Appendix explaining how our loss promotes inter-class separation, which is necessary ensure the learnability of the OOD gencralization problem. We provide a brief summary in Appendix and discuss the notion of OOD learnability, and would like refer readers to Ye et al. (2021) for in-depth and formal treatment. Empirically, to verify the impact of inter-class separation, we conducted an ablation study in Appendix H, where we compare the OOD performance of our method (with separation loss) vs. our method (without separation loss). We observe that incorporating separation loss indeed achieves stronger OOD generalization performance, echoing the theory.
8

---

Published as a conference paper at ICLR 2024
7 RELATED WORKS
Out-of-distribution generalization. OOD generalization is an important problem when the training and test data are sampled from different distributions. Compared to domain adaptation (Daume III & Marcu, 2006; Ben-David et al., 2010; Tzeng et al., 2017; Kang et al., 2019; Wang et al., 2022c), OOD generalization is more challenging (Blanchard et al., 201 20 Muandet et al., 2013; Gulrajani & Lopez-Paz, 2020; Bai et al., 2021b; Zhou et al., 2021; Koh et al., 2021; 1; Bai et al., 202 Wang et al., 2022b; Ye et al., 2022; Cha al., 2022; Bai et al., 2023; Kim et al., 2023; Guo et al., 2023; Dai et al., 2023; Tong et al., 2023), which aims to generalize to unseen distributions without any sample from the target domain. In partieular, A popular direction is to extract domain-invarianl feature representation. Prior works show that the invariant features from training domains can help discover invariance on target domains for linear models (Peters al., 2016; Rojas-Carulla al.. 2018). IRM (Arjovsky et al., 2019) and its variants (Ahuja et al., 2020; Krueger et al., 2021) aim to find invariant representation from different training domains via an invariant risk regularizer. Mahajan et al. (2021) propose a causal matching- based algorithm for domain generalization. Other lines of works have explored the problem from various perspectives such as causal discovery (Chang et al., 2020), distributional robustness (Sagawa et al., 2020; Zhou et al., 2020), model ensembles (Chen et al., 2023b; Rame et al., 2023), and test-time adaptation (Park et al., 2023; Chen et al., 2023a). In this paper, we focus on improving OOD generalization via hyperspherical learning, and provide new theoretical analysis the generalization error.
Theory for OOD generalization. Although the problem has attracted great interest, theoretical understanding of desirable conditions for OOD generalization under-explored. Generalization to arbitrary OOD is impossible since the test distribution is unknown (Blanchard et al., 2011; Muandet et al., 2013). Numerous general distance measures exist for defining a set of test domains around the training domain, such KL divergence (Joyce, 2011), MMD (Gretton et al., 2006), and EMD (Rubner et al., 1998). Based on these measures, some prior works focus on analyzing the OOD generalization error bound. For instance, Albuquerque et al. (2019) obtain risk bound for linear combinations of training domains. Ye et al. (2021) provide OOD generalization error bounds based on the notation of variation. In this work, we provide a hyperspherical learning algorithm that provably reduces the variation, thereby improving OOD generalization both theoretically and empirically.
Contrastive learning for domain generalization Contrastive learning methods have been widely explored in different learning tasks. For example, Wang & Isola (2020) analyze the relation between the alignment and uniformity properties on the hypersphere for unsupervised learning. while we focus on supervised learning with domain shift. Tapaswi et al. (2019) investigates a contrastive metric learning approach for hyperspherical embeddings in video face clustering, which differs from our objective of OOD generalization. Von Kügelgen et al. (2021) provide theoretical justification for self-supervised learning with data augmentations. Recently, contrastive losses have been adopted for OOD generalization. For example, CIGA (Chen et al., 2022) captures the invariance of graphs to enable OOD generalization for graph data. CNC (Zhang et al., 2022) is specifically designed for learning representations robust to spurious correlation by inferring pseudo-group labels and performing supervised contrastive learning. SelfReg (Kim et al., 2021) proposes a self-supervised contrastive regularization for domain generalization with non-hyperspherical embeddings, while focus on hyperspherical features with theoretically grounded loss formulations.
CONCLUSION
In this paper, we present theoretically justified algorithm for OOD generalization via hyperspherical learning. HYPO facilitates learning nomain--nvvriann representations the hyperspherical space. Specifically. we encourage low variation via aligning features across domains for each class and promote high separation by separating prototypes across different classes. Theoretically, we provide provable understanding of how our loss function reduces the OOD generalization error. Minimizing our learning objective can reduce the variation estimates, which determine the general upper bound on the generalization error of learnable OOD generalization task. Empirically, HYPO achieves superior performance compared to competitive OOD generalization baselines. We hope our work can inspire future research on OOD generalization and provable understanding.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
The authors would like to thank ICLR anonymous reviewers for their helpful feedback. The work supported by the AFOSR Young Investigator Program under award number 1495500-23-101888 National Science Foundation (NSF) Award No. IIS-2237037 & IIS-2331669, and Office of Naval Research under grant number N00014-23-1-2643.
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference on Machine Learning, pp. 145-155, 2020.
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint 1eXiv:/9/1.0000 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.02899 2019.
Haoyue Bai, Rui Sun, Langing Hong, Fengwei Zhou, Nanyang Ye, Han-Jia Ye, S-H Gary Chan, and Zhenguo Li. Decaug: Out-of distribution generalization via decomposed feature representation and semantic augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6705 6713, 20
Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Ye, S-H Gary Chan, and Zhenguo Li. Nas-ood: Neural architecture search for out-of-distribution generalization. In Proceedings of the IEEFJCVF International Conference Computer Vision, pp. 8320-8329, 2021b.
Haoyue Bai, Ceyuan Yang, Yinghao Xu, S-H Gary Chan, and Bolei Zhou. Improving out-of- distribution robustness classifiers via generative interpolation. arXiv preprint 1rX1v22307.12219, 2023.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151-175, 2010.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems, volume 24, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1): 46-100, 2021.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22415 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In European Conference on Computer Vision, pp. 440 440-457 457, 2022.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International 2020.
Conference on Machine Learning, pp.
Liang Chen, Yong Zhang. Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24172 221182, 2023a.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
10

---

Published as a conference paper at ICLR 2024
Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning, pp. 4623-4640. PMLR, 2023b.
Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distributie generalization on graphs. Advances in Neural Information Processing Systems, pp. 22131-22148, 2022.
Rui Dai, Yonggang Zhang, Zhen Fang, Bo Han, and Xinmei Tian. Moderately distributional exploration for domain generalization. In International Conference Machine Learning. 2023.
Hal Daume III and Daniel Marcu. Domain adaptation for statistical classifiers. Journal of Artificial Intelligence Research, 26:101-126, 2006.
Cian Fastwood, Alexander Robey, Shashank Singh, Julius Von Kügelgen, Hamed Hassani, George Pappas, and Berhhard Schölkopf. Probable domain generalization via quantile risk minimization. Advances in Neural Information Processing Systems, 35:17340-17358 2022.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2996-0300, 2016.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problee Advances in Neural Information Processing Systems, 19, 2006.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020.
Yaming Guo, Kai Guo, Xiaofeng Cao, Tieru Wu, and Yi Chang- Out-of-distribution generalization of federated learning via implicit invariant relationships. International Conference on Machine Learning, 2023.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pp. 124 140, 2020.
Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimiza- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16175-16185, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Uncertainty Artificial Intelligence, 2018.
James Joyce. Kullback-leible divergence. In International Encyclopedia of Statistical Science, pp. 720-722. 2011.
P.E. Jupp and K.V. Mardia. Directional Statistics. Wiley Series in Probability and Statistics. 2009. ISBN 9780470317815.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 1899-4902, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systemx, volume 33, pp. 18661-18673, 2020.
11

---

Published as a conference paper at ICLR 2024
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: Self Supe-supe contrastive regularization for domain generalization. In IEEE International Conference on Com- puter Vision, pp. 9619-9628, 2021.
Jaeill Kim, Suhyun Kang, Duhun Hwang, Jungwook Shin, and Wonjong Rhee. Vne: An effective
method for improving deep representation by manipulating eigenvalue distribution. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3799-3810, 2023.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. et al. Wilds: A benchmark of wi-the-wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
David Krueger, Ethan Caballero, Joeri- Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion. In International Conference on Machine Learning, pp. 5815-5826, 2021.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference on Computer Vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI Conference on Artificial Intelligence, volume 32, 201 8a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018b.
Junnan Li, Caiming Xiong, and Steven Hoi. Mopro: Webly supervised learning with momentum prototypes. In International Conference Learning Representations, 2020.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In European Conference on Computer Vision, pp. 624-639, 2018c.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pp. 7313-7324. PMLR, 2021.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018
Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in Neural Information Processing Systems, 32, 2019.
Seonwoo Min, Nokyung Park, Siwon Kim, Seunghyun Park, and Jinkyu Kim. Grounding visual representations with texts for domain generalization. In European Conference Computer Vision, pp. 37-53. Springer, 2022.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In International Conference on Learning Representations, 2023.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Hyconseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8690-8699, 2021.
Jungwuk Park, Dong-Jun Han, Soyeong Kim, and Jackyun Moon. Test-time style shifting: Handling arbitrary styles in domain generalization. In International Conference Machine Learning, 2023.
12

---

Published as a conference paper at ICLR 2024
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Sociery, pP- 947-1012,
Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model ratatouille: Recycling diverse models for outoof-distribution generalization. International Conference on Machine Learning, 2023.
Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):109--332, 2018.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas, A metric for distributions with applications to image databases. In International Conference on Computer Vision, pp. 59-66, 1998.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, pp. 443-450, 2016.
Mátyás Sustik, Joel A Tropp, Inderjit Dhillon, and Robert Heath Jr. the existence of equiangular tight frames. Linear Algebra and its applications, 426(2-3)x619-638 2007.
Makarand Tapaswi, Mare T Law, and Sanja Fidler. Video face clustering with unknown number of clusters. In Proceedings of the IEEE/VV International Conference on Computer Vision, pp. 5027-5036, 2019.
Peifeng Tong, Wu Su, He Li, Jialin Ding, Zhan Haoxiang, and Song Chen. Distribution free domain generalization. In International Conference on Machine Learning, pp. 34369-34378. PMLR, 2023.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176. 2017.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE Transactions Neural Networks, 10(5):988-999) 999998 1999.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 14:16451116467, 2021.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao Pico: Contrastive label disambiguation for partial label learning. In International Conference Learning Representations, 2022a.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yigiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: survey on domain generalization. IEEE Transactions Knowledge and Data Engineering, 2022b.
Rongguang Wang, Pratik Chaudhari, and Christos Davatzikos. Embracing the disharmony in medical imaging: A simple and effective framework for domain adaptation. Medical Image Analysis, 76: 102309, 2022c.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939, 2020.
Yufei Wang. Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In IEEÉ International Conference on Acoustics, Speech and Signal Processing, pp. 6622-662, 2020.
13

---

Published as a conference paper at ICLR 2024
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence, pp. 6502-6509, 2020.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprini urXiv:2001.00677 2020.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In JEEE Conference on Computer Vision and Puttern Recognition, pp. 7097-7107, 2022.
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of duitr--distri generalization. Advances in Neural Information Processing Systems, 222223599223333 2021.
Nanyang Ye, Kaican Li. Haoyue Bai, Rumpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of out-of-distribution generalization. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7947-7958, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Marvin Mengxin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt domain shift. In Advances in Neural Information Processing Systems, 2021.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Re. Correct-n- contrast: a contrastive approach for improving robustness to spurious correlations. International Conference on Machine Learning, pp. 26484-26516 2022.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. European Conference on Computer Vision, pp. 561-578, 2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
14

---

Published as a conference paper at ICLR 2024
A PSEUDO ALGORITHM
The training scheme of HYPO is shown below. We jointly optimize for (1) low variation, by encouraging the feature embedding samples to be close to their class prototypes; and (2) high separation, by encouraging different class prototypes to be far apart from each other.
Algorithm 1: Hyperspherical
Generalization
Imput: Training dataset D, deep neural network encoder h, class prototypes p.. temperature т ..... ... do for iter = 1,2,.... do sample a mini- obtain augmented batch B = {&,, Whe by applying two random augmentations to x; E B Vi .I,2.....6)
for epoch
(x,, Vi 1
for x; do
/ obtain norma lized embedding
7. = h(x;), 2, 2/1
// update class protooyyes
Pe := Normalize(ape + (1 a)z,), e (1,2,..,,C}
// calculate the loss for low variation exp(1fT man/r)
Lvar = = Eneen !D' log Ee expee (s/rr)
// calculate the loss for high separation 1 Ceep = Ein log c-1 Ejj,jey exp (H 12/T)
10
// calculate overall loss
11
L= Lvar Coop
// update the network weights
12
update the weights the deep neural network
B BROADER IMPACTS
Our work facilitates the theoretical understanding of OOD generalization through prototypical learning, which encourages low variation and high separation in the hyperspherical space. In Section 5.2, we qualitatively and quantitatively verify the low intra-class variation of the learned embeddings and we discuss in Section 6 that the variation estimate determines the general upper bound on the generalization error for a learnable OOD generalization task. This provable framework may serve as foothold that can be useful for future OOD generalization research via representation learning.
From a practical viewpoint, our research can directly impact many real applications, when deploying machine learning models in the real world. Out-of-distribution generalization is fundamental problem and is commonly encountered when building reliable ML systems in the industry. Our empirical results show that our approach achieves consistent improvement over the baseline wide range of tasks. Overall, our work has both theoretical and practical impacts.
C THEORETICAL ANALYSIS
Notations. We first set up notations for theoretical analysis. Recall that P denotes the joint dis- tribution of X, Y in domain The label set 3 21, ,C}. For input its feature embedding. Let IP" denote the marginal distribution of in domain e with class y. Similarly, pEy denotes the marginal distribution of Z in domain with class y. Let E == E for abbreviation. As we do not consider the existence of spurious correlation in this work. it is natural to assume that domains and classes are uniformly distributed: Px = EC Eey P... We specify the
-((x) (x)((x)]
15

---

Published as a conference paper at ICLR 2024
distance metric to be the Wasserstein-1 distance ie., Wi(-.) and define all notions of variation under such distance.
Next, proceed with several lemmas that particularly useful to prove main theorem.
Lemma C.I. With probability at least
h(x)
1
h(x:) [|((x)]l2 S ESNEN N
h(x;)
In(2/6)
-B(x,e) Me
[1(x)]l2
+
N
N
Bo
ON
sup kEH
0 s c(t)
|1((x))12 1 + 3v
N
where B3 is universal constant and , N are Rademacher variables.
Proof. By Cauchy-Schwarz inequality.
h(x:) Pd [1(x))]l2
h(x;)
1 |
[11(xx)11212
Define G = (1x())) , : h M0. Let S (uI...., un) ~ PN where u,
x;
and N is the
Me(ss)
sample size. The Rademacher complexity of is
Rx( = ESSPN
sup (ug( gEg t=1
We can apply the standard Rademacher complexity bound (Theorem 26.5 in Shalev-Shwartz and Ben-David) to G, then we have that,
h(x)
1
h(x:)
N o.g (ug( + +BV
ln(2/8)
-B(x.c)-Pxyle Me [
+
N
[1(xx)]12
S ES-AN
N
E n
sup gEg
N
1=1
h(x;)
In(2/8)
= EsSPN
E
ON sup kER
[[(x;)]]
+ B
N
i=1
where 3 is universal positive constant.
Remark 1.
The above lemma indicates that when samples are sufficiently aligned with their class h(x.)
prototypes on the hyperspherical feature space, i.e., 1 Σi=1 Ma(a) A(xx))l2 2 - ffr some small
constant > 0, we can upper bound -Eix,c) Mo hxx) This result will be useful to prove Thm 6.1.
[[(x)]]
Lemma C.2. Suppose Pzy M z 2 1 7. Then. for all Eusin and y [C], we have that
E Exprr u z 2 -CEy.
Proof. Fix e' Euain and y E [C]. Then,
1 1- 7 S B(axc) ui
Exapzez by
CE
e wE(C!
z T My +
Σ
Exwzyoz T Hyy
CE zwpee
CE
(f (CM(())))))
CE-1
CF Z R py
CE
where the last line holds by Iz T Hdl < and we also used the assumption that the domains and classes are uniformly distributed. Rearranging the terms, we have
- CE. < E E v Mg
16

---

Published as a conference paper at ICLR 2024
Lemma C.3. Fix y [C] and e Euain- Fix 17 0. 1f Bampa My CEy, then
P2 (1 Myll2 2 11) - 2CE2
17
Proof. Note that
1/ M4y 11 = + 14 112 2z 1by
2z Myy
Taking the expectation on both sides and applying the hypothesis, we have that
ERwPz ||z Me 12 S 2C
Applying Chebyschev's inequality to 1|z - Myll2 we have that
Var(Iz - Hull2)
PZ (1|2 Mull2 n) 2
172
E Bumpzy (1lz 1yl122
112
2CEy
1)2
Lemma C.4. Fix y E [C), Fix e, Enain- Suppose Eu Me A1--EY, Fix E S Let P denote the distribution of v Ze and Q denote the distribution v Zet. Then,
(i(P,Q) S 10(CE))
where Wi (P, is the Wassersisten-1 distance.
Proof. Consider the dual formulation of Wasserstein-1 distance:
W(P.Q)
sup 1
Expee [f(v x)] E xw [f(vT [x]
where Ilfp denotes the Lipschitz norm. Let > There exists fu such that
W(P,Q) Eynag [So(v T 2)] E Ee [/o(Vo[V [2)]] + k.
We assume that without loss of generality fo(Hs]] v) O. Define f'(-) fol-) - folu, v). Then, note that (' v) 0 and
Enppp 10 (v 2) 2 > [f'(('' x)] = Empgyo [oo(v 2)] z v [So(v 2)] + 0) ('1
= En-pyo [fo(v 2)] E Enper v [o((vz)))
proving the claim.
Now define B = {u Sdl1 :||u Myll2 S n). Then, we have
Ezwesy [fo(v" 2)] - ze ifo(v T 2)] = Ez-egy (fo(v z)1(z e B)| E E
2 [fo(v 2)1 e BY]
+ Egwrgo [fo(v 2)1(2 & BH Ba ffo 2)1 & B)
E
Note that if z E B, then by Ilfll <
[o(v 2) - -o((v (0,)) < vv - Myy)
< ||v ||z 14yl2
n
17

---

Published as a conference paper at ICLR 2024
Therefore, [fo(v*z)) z)1 $ 77 and we have that
Ex ( 2)1(z e B}] E anr Parrgslful 2)1(z (f)
BY] s [1fz BHI + E Bn e B)))
2m-
Now, note that m If(u T v)1 $ (repeat the argument from above but use ||u Hyll2 2. Then.
Exwez [fo(v 2)1(z & BH| - E 2 2 & BY| s 2 (1f2 & B}] + E ampe [1fz & BH]
2)1{2
SCEy
where in the last line, we used the hypothesis and Lemma C.3. Thus, by combining the above, have that
SCEY
W(P, Q) 2m
+ K.
172
Choosing (/3 have that
W(P, Q) & 10(CE)) + k.
Since was arbitrary, we can let it go to 0, obtaining the result.
Next, ready to prove our main results. For completeness, state the theorem here.
Theorem C.1 (Variation upper bound (Thm 4.1)). Suppose samples are aligned with class prototypes
such that tV Ei= Hegy 2 e some E (0, 1), where 2, bxxx)]la probability at least 8.
h(x;) Then 30 (0, 1), with
N o Mc(i))) 1/3 +
ln(2/6)
(hu Eavait s 0(e1//3 + (Epl E
N ) )100),
sup hCH
i=1
where 01,..., ON are Rademacher random variables and O() suppresses dependence on constants and (Ewait)-
Proof of Theorem 6.1. Suppose 1/ E M (e Lemma C.I, with probability at least 6, we have
M
A(x;: TMx.ll
Then, by
M(x)
N h(x,)
ln(2/5)
N
)(x,))
-Exx,e) Me
[Txx))l12
S ES-Pwl N
E
sup hEH
O.MM) n(t)
[1,(x;)]] 11
1+1
N
pe(
[
i=1
h(x,)
In(2/5)
Es-ewl ON
NEW
sup
o())
[1((x,)]
1l|
+ 3
N
-
where 01, ON denote Rademacher random variables and / is universal positive constant. Define 7 ESNPx I Eo suPNEH EN O.H
R(x,) (xx) Bv [n(2/8))
Then, have
(z,z) Me - 7.
Then, by Lemma C.2, for e Evain and E (C),
Ex " u, 2 2 1 - 1-C.y.
Let > 0 and such that
shu (h, Eurin) = sup V(v Evain) S V(vo h Evain) +
vESd
18

---

Published as a conference paper at ICLR 2024
Let Qud" denote the distribution of vi in domain under class y. From Lemma C.4, have that
Wi d:" - 10(CE))
for y E [C] and e, Errinn.
We have that
sup vegu-1
V(v h, Eusin)
sup V(v h, Euaia)
vegú-1
sax Wi (Q Q )) +
0,6'
$ 10(CE)) 1/3 + a.
Noting that a was arbitrary, we may send it to 0 yielding
V(v h, Euain) & 10(CEt)
sup vESU
Now, using the inequality that for a, b, c 2 0, (a bb+b++))++ c) s 1/3 + b1/3 + c//3, we have that
1 yuu (h, Erraio) 50(e2/3 (ESwPN I Boi sup
h(x;) 1)1/3
In(2/0)
o ( [1(x;)]12
3(
2//)
N
Remark 2. As our loss promotes alignment of sample embeddings with their class prototypes on the hyperspherical space, the above Theorem implies that when such alignment holds, we can upper bound the intra-clan class variation with three main factors: the optimization error e, the Rademacher complexity of the given neural network, and the estimation error
(n(2/8) N 11/6
C.1 EXTENSION: FROM Low VARIATION TO Low OOD GENERALIZATION ERROR
Ye al. (2021) provide OOD generalization error bounds based on the notation of variation. Therefore, bounding intra-class variation is critical to bound OOD generalization error. For completeness, we reinstate the main results in Ye et al. (2021) below, which provide both OOD generalization error upper and lower bounds based on the variation w.r.t. the training domains. Interested readers shall refer to Ye et al. (2021) for more details and illustrations.
Definition C.1 (Expansion Function (Ye et al., 2021)). We say function R U (0} -
RR U {0, } is an expansion function, iff the following properties hold: 1) s(-) is monotoni- cally increasing and s(x) x, Vx 0; li s(x) s(0) 00.
As it is impossible to generalize to arbitrary distribution, characterizing the relation between Evaiil and Ean is essential to formalize OOD generalization. Based on the notion of expansion function, the learnabllity of OOD generalization is defined as follows:
Definition C.2 (O0D-Learaabiie) (Ye et al., 2021)). Let & be the feature space and p be distance metric on distributions. We say an OOD generalization problem from Enyail to Ean is learnable if there exists expansion function s(.) and 8 > O, such that: for all á &² satisfying I,(ś), Envii) > 8, we have s(V s(Vp(ó, Eavai)) 2 Vp Ean). If such s(-) and 8 exist, we further call this problem (s(-) 6)- learnable.
For learnable OOD generalization problems, the following two theorems characterize OOD error upper and lower bounds based on variation.
Theorem C.2 (OOD Error Upper Bound (Ye et al., 2021)). Suppose we have learned a classifier with loss function ((.-)) such that Ve Emi and Vy Y, Phe (hly) E (2 h(-)) e Bd denotes the feature extractor. Denote the characteristic function of random variable he l as
2 referred feature h theoretical analysis.
19

---

Published as a conference paper at ICLR 2024
Phojy (tly) (itt, h²) HY = y). Assume the hypothetical space F satisfies the following
regularity conditions that 3a, M1. M2 > 0,Vf E F F,ye Ve E &all y E
JhereP Pholy (hly) - M1
JEER (U S M2.
and
(8)
If (Eavail, Ean) is (s(.), Emai)) learnable under with Total Variation p², then we have
err(f) S 50 (1 Eh, (ata)a
(9)
where (-) depends on d, C, M1, M2.
Theorem C.3 (OOD Error Lower Bound (Ye et al., 2021)). Consider 0-1 loss: E(ú. 9) = 1(g # y).
For any 5 O and any expansion function satisfying 1) 8' (0) & limz-o+ s(a)=a(U) (1, +00); 2) exists k > 1, > 0, s.1. kr S. 8(x) < +00,0 x E [0, t). there exists a constant Co and an 0OD generalization problem (Evvil, Eal) that is (s(+), ol-earnable under linear feature space w.r.t
symmetric KL-divergence s.t. Ve E [0, 11 the optimal classifier f satisfying youp Evvail will have the OOD generalization error lower bounded by
err(f) 2 Co s(VRp (h, Emvai)))
(10)
D ADDITIONAL EXPERIMENTAL DETAILS
Software and hardware. Our method is implemented with PyTorch 1.10. All experiments conducted on NVIDIA GeForce RTX 2080 Ti GPUs for small to medium batch sizes and NVIDIA A100 and RTX A6000 GPUs for large batch sizes.
Architecture. In our experiments, we use ResNet- 18 for CIFAR-10, ResNet-34 for ImageNet- 100, ResNet-50 for PACS, VLCSS, Office-Home and Terra Incognita. Following common practice in prior works (Khosla et al., 2020), we use non-linear MLP projection head to obtain features in our experiments. The embedding dimension 128 of the projection head for ImageNet- The projection head dimension is 512 for PACS, VLCS, Office-Home, and Terra Incognita.
Additional implementation details. In our experiments, we follow the common practice that initializing the network with ImageNet pre-trained weights for PACS, VLCS, Office-Home, and Terra Incognita. We then fine-tune the network for 50 epochs. For the large-scale experiments on ImageNet- 100, we fine-tune ImageNet pre-trained ResNet-34 with our method for 10 epochs for computational efficiency. We set the temperature т 0.1, prototype update factor a = 0.95 as the default value. We use stochastic gradient descent with momentum 0.9, and weight decay 10-1 The search distribution in our experiments for the learning rate hyperparameter is: Ir E (0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001, 0.00005). The search space for the batch size bs E (32, 64). The loss weight À for balancing
loss function (C XL + sepp) is selected from E {1.0, 2.0, 4.03. For multi-source domain generalization, hard negatives can be incorporated by simple modification to the denominator of the variation loss:
cxp (z,T Mo(i)/T)
Lvar
Σ log
(11)
j= exp (oTTu/T) Eラn1 I1 + Vi, 6; = =e) exp (zT 21/TT
eEE 1=1
Details of datasets. We provide a detailed description of the datasets used in this work:
CIFAR-10 (Krizhevsky et al., 2009) is consist of 60, 000 color images with 10 classes. The training set has 50, 000 images and the test set has 10, 000 images.
ImageNet-1000 is composed by randomly sampled 100 categories from ImageNet- K. dataset contains the following classes: n01498041, n01514859, n01 582220, n01608432, n01616318, n01687978, n01776313, nO1 n01806567, n01833805, n01882714, n01910747, n01944390, n01985128, n02007558, m02071294,
For distribution Q with prohahility density function p. q. p(P, Q) 1 f. (p(>) ((=))dz.
20

---

Published as a conference paper at ICLR 2024
n02085620, n02114855, n02123045, o20.0044450222...222222200020204
n02326662, m02226429, n02279972, n02317335, n02326432, n02342885, 222442288...22200000022299992990 00022666000.02222909992222223339 10239454, m02403003.
n02422699, n02442845, n02444819, n02480855, n025
n02744221,....772211,,,,,,, n02687172, (00000007.222220200000 a02701002.
n02814860, n02840245, m02730930, m02769748, m02782093, m02787622, m02793495, m02799071, n02802426, n022949,,,,
n02906734, n02948072, m02980441, n02999410, n03014705, m03028079, n03032252, n03125729, n03160309,
n03179701, n03220513, n03249569, n03291819, n03384352, n03388043, n03450230, n03481172, n03594734,
n03594945, n03627232, n03642806, n03649909, n03661043, n03676483, n03724870, n03733281, n03759954,
n03761084, m03773504, n03804744, n03916031, n03938244, n04004767, n04026417, n04090263, n04133789,
n04153751, n04296562, m04330267. n04371774, n04404412, n04465501, n04485082, n04507155, n04536866,
n04579432, m04606251, n07714990, n07745940.
CTEAR-IO-C generated based on previous literature (Hendrycks & Dietterich, 2019), applying different corruptions on CIFAR-10 data. The corruption types include gaussian noise, zoom blur, impulse noise, defocus blur, snow, brightness, contrast, elastic transform, fog, frost, gaussian blur, glass blur, JEPG compression, motion blur, pixelate, saturate, shot noise, spatter, and speckle noise.
ImageNet- 100-C algorithmically generated with Gaussian noise based on (Hendrycks Dietterich, 2019) for the ImageNet-100 100 dataset.
PACS (Li et al., 2017) is commonly used in OoD generalization. This dataset contains 9, 991 examples of resolution 224 x 224 and four domains with different image styles, namely photo, art painting, cartoon, and sketch with seven categories.
VLCS (Gulrajani & Lopez-Paz, 2020) comprises four domains including Caltech101, LabelMe, SUN09, and VOC2007. It contains 10, 729 examples of resolution 224 224 and classes.
Office-Home (Gulrajani & Lopez-Paz, 2020) contains four different domains: art, clipart, product, and real. This dataset comprises 15, 588 examples of resolution 224 x 224 and 65 classes.
Terra Incognita (Gulrajani & Lopez-Paz, 2020) comprises images of wild animals taken by cameras at four different locations: location 000, location38, location43, and location46. This dataset contains 24, 788 examples of resolution 224 224 and 10 classes.
E DETAILED RESULTS ON CIFAR-10 10
In this section, we provide complete results of the different corruption types on CIFAR-IO. In Table 3, we evaluate HYPO under various common corruptions. Results suggest that HYPO achieves consistent improvement over the ERM baseline for all 19 different corruptions. We also compare our loss (HYPO) with more recent competitive algorithms: EQRM (Eastwood al., 2022) and SharpDRO (Huang et al., 2023), on the CIFAR dataset (Gaussian noise). The results on ResNet- 18 are presented Table
Table 3: Main results for verifying OOD generalization performance on the 19 different covariate shifts datasets. We train CIFAR-10 ID, using CIFAR-10-C as the OOD test dataset. Acc. denotes the accuracy on the OOD test set.
Method
Corruptions
Ace.
Corruptions
Acc.
Corruptions
Acc.
Corruptions Acc.
CE
Gaussian noise
78.09
Zoom blur
88.47
Impulse noise
83.60
Defocus blur
94,85
HYPO (Ours)
Gaussian noise
85.21
Zoom blur
93.28
Impulse noise
87.54
Defocus blur
94.90
CE
Snow
90.19
Brightness
94.83
Contrast
94.11
Elastic transform
90.36
HYPO (Ours)
Snow
91.10
Brightness
94.87
Contrast
94.53
Elastic transform
91.61
CE
Fog
94.45
Frost
90.33
Gaussian blur
94.85
Glass blur
$6.99
HYPO (Ours)
Fog
94.57
Frost
92.28
Gaussian blur
94.91
Glass blur
63.66
CE
JEPG compression
86.95
Motion hlur
90.69
Pixelate
92.67
Saturate
92.86
HYPO (Ours)
JEPG compression
89.24
Motion blur
93.07
Pixelate
93.95
Saturate
93.66
CE
Shot noise
85.86
Spatter Spatter
92.20
Speckle noise
85.66
Average
88.32
HYPO (Ours)
Shot noise
89.87
92.46
Speckle noise
89.94
Average
90.56
21

---

Published as a conference paper at ICLR 2024
Algorithm
Art painting
Cartoon
Photo
Sketch
Average Acc, (%)
IRM (Arjovsky et al., 2019)
84.8
76.4
96.7
76.1
83.5
DANN (Ganin et al. 2016)
86.4
77.4
97.3
73.5
83.7
CDANN (Li et al., 2018c)
84,6
75.5
96.8
73.5
82.6
GroupDRO (Segawa
2020)
83.5
79.1
96.7
78.3
84.4
MTL (Blanchard 2021)
87.5
77.1
96.4
77.3
84.6
I-Mixup (Wang
2020; al,, 2020; Yan al., 2020)
78,9
97.6
MMD ct 2018h)
2018h)
86.1
79.4
96.6
76.5
84.7
VREx (Krueger
2021)
86.0
79.1
96.9
77.7
84,9
MI.D 2018a)
K5.5
80.1
97.4
76.6
84.9
ARM (Zhang et al., 2021)
86.8
76.8
97.4
79.3
85.1
RSC (Huang
2020)
85.4
79.7
97.6
78.2
85.2
Mixstyle (Zhou et al., 2021)
86.8
79.0
96,6
78.5
85.2
ERM (Vapnik. 1999)
84.7
80.8
97.2
79.3
85.5
CORAL (Sun & Sacnko, 2016)
88.3
80.0
97.5
78.8
86.2
SagNet (Nam et al., 2021)
87.4
80.7
97.1
80.0
86.3
SelfReg (Kim
2021)
87.9
79.4
96.8
78.3
85.6
GVRT Min et al. (2022)
87.9
78.4
98.2
75.7
85.1
(Kim al., 2023)
79.9
96.7
86.9
HYPO (Ours)
87.2
82.3
98.0
84.5
88.0
Table 4: Comparison with state-of-the-art methods the PACS benchmark. All methods are trained on ResNet-50. The model selection is based training domain validation set. To isolate the effect of loss functions, all methods are optimized using standard SGD. "Results based on retraining of PCL with SGD using official implementation. PCL with SWAD optimization further compared in Table 2. We run HYPO times and report the average and std. -r denotes the standard error, rounded to the first decimal point.
Algorithun
Clipart
Product
World
Average (%)
IRM (Arjovsky et 2019)
58.9
52.2
72.1
74.0
64.3
DANN (Ganin et a0l, 2016)
59.9
53.0
73.6
76.9
65.9
CDANN (Li et 2018c)
61.5
50.4
74.4
76.6
65.7
GroupDRO (Sagawa et 2020)
60.4
52.7
75.0
76.0
66.0
MTL (Blanchard et 2021)
61.5
52.4
74.9
76.8
66.4
I-Mixup (Wang et al., 2020: Xu al., 2020; Yan et al., 2020)
62.4
54,8
76.9
78.3
68.1
MMD (Li
60.4
77.4
66.4
VREx (Krueger et al.. 2021)
60.7
$3.0
75.3
76.6
66.4
(Li al., 2018a)
61.5
77.5
ARM (Zhang et al., 2021)
58.9
51.0
74.1
75.2
64,8
RSC (Huang al., 2020)
60.7
51.4
74.8
75.1
65.5
Mixstyle (Zhou et 2021)
$1.1
53.2
68.2
69.2
60.4
ERM (Vapnik, 1999)
63.1
51.9
77.2
78.1
67.6
CORAL (Sun & Saenko, 2016)
65.3
54.4
76.5
78.4
68.7
SagNet (Nam et al., 2021)
63.4
54,8
75.8
78.3
68.1
SelfReg (Kim al. 2021)
63.6
53.1
78.1
67.9
GVRT Min et (2022)
66.3
55.8
78.2
80.4
70.1
VNE (Kim et al., 2023)
60.4
74.7
65.9
HYPO (Ours)
68.3
57.9
79.0
81.4
71.7
Table 5: Comparison with state-of-the ahett-rrtt methods on the Office-Home benchmark. All methods are trained on ResNer-50. The model selection based on training domain validation To isolate the effect of loss functions, all methods are optimized using standard SGD.
F ADDITIONAL EVALUATIONS ON OTHER OOD GENERALIZATION TASKS
In this section, we provide detailed results on more OOD generalization benchmarks, including Office-Home (Table 5), VLCS (Table 6), and Terra Incognita (Table 7). We observe that our approach achieves strong performance on these benchmarks. We compare our method with collection of OOD generalization baselines such as IRM (Arjovsky et al., 2019). DANN (Ganin et al., 2016), CDANN (Li et al., 2018c), GroupDRO (Sagawa et al., 2020), MTL (Blanchard et al., 2021), I-Mi xup (Zhang et al., 2018), MMD (Li et al., (018)), VREx (Krueger al., 2021), MLDG (Li et al., 2018a), ARM (Zhang et al., 2021), 12,1, RSC (Huang al., 2020), Mixstyle (Zhou al., 2021), ERM (Vapnik, 1999), CORAL (Sun & Saenko, 2016), SagNet (Nam et al., 2021), Se SelfReg (Kim et al., 2021), GVRT Min et al. (2022), VNE (Kim al., 2023). These methods and optimized using standard SGD. On the Office- Home, our method achieves an improved OOD generalization performance of 1.6% compared to competitive baseline (Sun & Saenko, 2016).
all loss- based
We also conduct experiments coupling with SWAD and achieve superior performance on OOD gener- alization. As shown in Table 8, Table 9, Table 10, our method consistently establish superior results
22

---

Published as a conference paper at ICLR 2024
Algorithm
Caltech101
LabelMe
SUN09
VOC2007
Average Acc. (%)
IRM (Arjovsky al.,
98.6
64,9
73.4
77.3
78.6
DANN (Ganin et al., 2016)
99.0
65.1
73.1
71.2
78.6
CDANN al., 2018c)
97.1
65.1
70.7
77.1
77.5
GroupDRO (Sagawa 2020)
97.3
63.4
69.5
76.7
76.7
MTL 202)
97.8
64.3
71.5
75.3
77.2
I-Mixup (Wang al., 2020; Xu et al., 2020, Yan et al., 2020)
98.3
64.8
72.1
74.3
77.4
MMD (Li 2018b)
97.7
64.0
72.8
75.3
77.5
VREX (Krueger et 2021)
98.4
64.4
74.1
76.2
78.3
MLDG (Li al., 2018a)
97.4
65.2
71.0
75.3
77.2
ARM (2hang al., 2021)
98.7
63.6
71.3
76.7
77.6
RSC (Huang
2020)
97.9
62.5
72.3
75.6
77.1
Mixstyle (Zhou
2021)
72.6
75.7
77.9
ERM (Vapnik, 1999)
97.7
64.3
73.4
74.6
77.5
CORAL (Sun Saenko, 2016)
73.4
77.5
SagNet (Nam al., 2021)
97.9
64.5
71.4
77.5
77.8
SelfRey (Kim
2021)
73.1
76.2
GVRT Min et al. (2022)
98.8
64.0
75.2
77.9
79.0
VNE et al., 2023)
70.4
78.4
HYPO (Ours)
98.1
63.3
73.1
76.3
78.2
Table 6: Comparison with state-of-the-ee methods on the VLCS benchmark. All methods are trained on ResNet-50. The model selection is based training domain validation set. To isolate the effect of loss functions, all methods are optimized using standard SGD.
Algorithm
Location100
Location38
Location4.3
Location46
Average (%)
IRM (Arjovsky et al., 2019)
S4,6
39.8
56.2
39.6
47.6
DANN (Ganin sl. 2016)
51.1
40.6
57.4
37.7
46.7
2018c)
GroupDRO (Sagawn al., 2020)
41.2
38.6
56.7
36.4
43.2
MTL (Blanchard 2021)
49.3
39,6
$5.6
37.8
45.6
I-Mixup (Wang
2020)
MMD (Li 201
41.9
34,8
57.0
35.2
42.2
(Kraggr al., 2021)
48.2
41,7
56.8
38.7
46.4
MLDG ct.,
ARM (Zhang ct al., 2021)
49.3
38.3
55.8
38.7
45.5
RSC (Hunng al., 2020)
50.2
39,2
56.3
40.8
46.6
Mixstyle (Zhou
2021)
RRM 1999)
49.8
42.1
569
35.7
46.1
CORAL (Son Sacako, 2016)
51.6
42.2
57.0
39.8
47.7
SugNet
SelfReg (Kim al., 2021)
48.8
41.3
57.3
40.6
47.0
GVRT Min al. (2022)
53.9
41.8
58.2
38.0
48.0
HYPO (Ours)
58.8
46.6
58.7
42.7
31.7
Table Comparison with state-of offthe--rr methods on the Terra Incognita benchmark. All methods are trained on ResNet-50. The model selection is based on training domain validation set. To isolate effect loss functions, all methods are optimized using standard SGD.
Algorithm
Art
Clipart
Product
Real World
Average Acc. (%)
SWAD (Cha et al., 2021)
66.1
57.7
78.4
80.2
70.6
PCL+SWAD (Yao ul., 2022)
67.3
59.9
78.7
80.7
71.6
VNE+SWAD (Kim al., 2023)
66.6
58.6
78.9
80.5
71.1
HYPO+SWAD (Ours)
68.4
61.3
81.8
82.4
73.5
Table 8: Results with SWAD-based optimization on the Office-Home benchmark.
Algorithm
Caltech101
LabelMe
SUN09
VOC2007
Average Acc. (%)
SWAD (Cha et al., 2021)
98.8
63.3
75.3
79.2
79.1
PCL+SWAD (Yao et al., 2022)
95.8
65.4
74.3
76.2
77.9
VNE+SWAD (Kim et al., 2023)
99.2
63.7
74.4
81.6
79.7
HYPO+SWAD (Ours)
98.9
67.8
74.3
77.7
79.7
Table 9: Rresults with SWAD-based optimization on the VLCS benchmark.
on different benchmarks including VLCS, Office-Home, Terra Incognita, showing the effectiveness of our method via hyperspherical learning.
23

---

Published as a conference paper at ICLR 2024
Algorithm
Location100
Location38
Location43
Location46
Average Acc. (%)
SWAD (Cha et al., 2021)
55.4
44.9
59.7
39.9
50.0
PCL+SWAD (Yau et al., 2022)
58.7
46.3
60.0
43.6
52.1
VNE+SWAD (Kim et al., 2023)
59.9
45.5
59.6
41.9
51.7
HYPO+SWAD (Ours)
56.8
61.3
54.0
53.2
56.3
Table 10: Results with SWAD-based optimization on the Terra Incognita benchmark.
G EXPERIMENTS ON IMAGENET-IOO AND IMAGENET-100-
In this section, provide additional large-scale results the ImageNet benchmark. We use ImageNet- 100 as the in-distribution data and use ImageNet-1000 100-C with Gaussian noise as OOD data in the experiments. In Figure observe method improves OOD accuracy compared to the ERM baseline.
H ABLATION OF DIFFERENT LOsS TERMS
90
ERM Ours
88.1
Ablations on separation loss. In Table 11, we demon- strate the effectiveness of the first loss term (variation) empirically, We compare the OOD performance of our method (with separation loss) vs. our method (without sep- aration loss). We observe our method without separation loss term still achieve strong OOD accuracy accuracy-average 87.2% on the PACS dataset. This ablation study indicates the first term (variation) of method plays more im- portant role in practice, which aligns with our theoretical analysis in Section 6 and Appendix C.
85
81.0
80
79.1
average
75
ImageNet-100-C ImageNet-100
Figure Experiments on ImageNet-
(I)) vs. ImageNet-100-C (OOD).
Algorithm
Art painting
Cartoon
Photo
Sketch
Average Acc. (%)
Ours (w/o separation loss) Ours (w separation loss)
86.2
81.2
97.8
83.6
87.2
87.2
82.3
98.0
84.5
88.0
Table 1I: Ablations on separation loss term.
Ablations on hard negative pairs. To verify that hard negative pairs help multiple training domains, conduct ablation by comparing ours (with hard negative pairs) ours (without hard negative pairs). We can see in Table 12 that our method with hard negative pairs improves the average OOD performance by 0.4% the PACS dataset. Therefore, empirically demonstrate that emphasizing hard negative pairs leads to better performance for multi- domain generalization tasks.
Algorithm
Art painting
Cartoon
Photo
Sketch
Average Acc.
Ours (w/o hard negative pairs) Ours (w hard negative pairs)
87.8
82.9
98.2
81.4
87.6
87.2
82.3
98.0
84,5
88.0
Table 12: Ablation on hard negative pairs. OOD generalization performance on the PACS dataset.
Comparing EMA update and learnable prototype. We conduct ablation study the prototype update rule. Specifically, we compare our method with
sawtiiaal-moviinvvarreeee (EMA) (Li et al., 2020; Wang et al., 2022a; Ming et al., 2023) prototype update versus learnable prototypes (LP). The results on PACS are summarized in Table 13. We observe our method with EMA achieves better average OOD accuracy 88.00 88.0% compared to learnable prototype update rules 86. 7%. We empirically verify EMA-style method is a suitable prototype updating rule to facilitate gradient-based prototype update in practice.
Quantitative verification of the factor in Theorem 6.1. We calculate the average intra-class variation over data from all environments N E
1 25 (Theorem 6.1) models trained with
24

---

Published as a conference paper at ICLR 2024
Algorithm
Art painting
Cartoon
Photo
Sketch
Average Acc. (%)
Ours (LP)
80.7
97.5
80.7
86.7
Ours (EMA)
87.2
82.3
84.5
88.0
Table 13: Ablation on prototype update rules. Comparing EMA update and learnable prototype (LP) on the PACS benchmark.
N Ej= 2c)) We evaluated PACS, VLCS, and OfficeHome and summarized the results in Table 14. We observe that training with HYPO significantly reduces the average intra-class variation, resulting
НУРO. Then we obtain é := 1 1
small epsilon (é < 0.1) practice. This suggests that the first term O(eà) in Theorem 6.1 indeed small for models trained with HYPO.
Dataset
PACS
0.06 0.08
VL.CS
OfficeHome
0.09
Table 14: Empirical verification of intra-class variation in Theorem 6.1.
Method
OOD Acc. (%)
EQRM (Eastwood et al., 2022)
77.06
SharpDRO (Huang et al., 2023)
81.61
HYPO (ours)
85,21
Table 15: Comparison with more recent competitive baselines. Models are trained on CIFAR-10 using ResNet-18 and tested CIFAR10-C (Gaussian noise).
ANALYZING THE EFFECT OF T AND Q
In Figure 7a, we present the OOD generalization performance by adjusting the prototype update factor a. The results are averaged over four domains on the PACS dataset. We observe the generalization performance is competitive across a wide range of a. particular, our method achieves the best performance when 0.95 on the PACS dataset with an average of 88.0% OOD accuracy.
We show in Figure 7b the OOD generalization performance by varying the temperature parameter т. The results are averaged over four different domains PACS. We observe a relative smaller results in stronger OOD performance while too large T (c.g., 0.9) would lead to degraded performance.
90
90
85
88
80
75
Acc
86
82
0.1
0.5
0.8
0.95
0.98
0.2
0.3
0.4
0.5
0.9
(a) Moving average a
(b) Temperature т
Figure 7: Ablation on (a) prototype update discount factor and (b) temperature 7. The results averaged over four domains on the PACS dataset.
25

---

Published as a conference paper at ICLR 2024
J THEORETICAL INSIGHTS ON INTER-CLASS SEPARATION
To gain theoretical insights into inter-class separation, we focus on the learned prototype embeddings of the separation loss with simplified setting where we directly optimize the embedding vectors.
Definition J.1. (Simplex ETF (Sustik et al., 2007)). A set of vectors {p:) in Rd forms a simplex Equiangular Tight Frame (ETF) if | = for vi E [C] and μi HJ = -1/(C 1) for vi #i j.
Next, we will characterize the optimal solution for the separation loss defined as:
C
1
Sep
log
exp (p p3/T)
Σ
log Cue (1)
separation
Lemma J.1. (Optimal solution of the separation loss) Assume the number of classes C S d frep is minimized when the learned class prototypes {11r) form simplex ETF.
Proof.
Lep (2)
exp
(n, Mg/T)
(12)
C
jøi,=1
2 exp
Σ u Mn/T
(13)
C ifi.j=l
M. 11 n. ui
exp
(14)
T(C -1
H. M- T(C (1C-1))
exp
(15)
where we define μ = Ee bo and (13) follows Jensen's 's inequality. Therefore, we have
C
Cup
log Seep (2)
C
1=1
1
H: 1
log exp
C
7(C-
1=1
Σ
(uT k 1-1) 1)
TC(C
1)
CCC 7(C -1)
suffices to consider the following optimization problem,
minimize L1 = pe P
subject to || =1 Vi [C]
where 41 == (4) T>
p.)
Di Ejy; M4 My +
However, the problem is non-convex. We first consider convex relaxation and show that the optimal solution to the original problem is the same as the convex problem below,
C
minimize C2
Σ
Σ
pj
i=
subject
|mill 51 Vi e [C]
26

---

Published as a conference paper at ICLR 2024
Note that the optimal solution L] C2. Next, obtain the Lagrangian form:
C(p pC, , )0) Σ Σ pfp + Σ AA 1)
where A; are Lagrange multipliers. Taking the gradient of the Lagrangian with respect to μ and setting it zero, have:
Oc
2
Σ
pi
2A =
Opk
if
Simplifying the equation, we have:
14 (1 A
Therefore, the optimal solution satisfies that (1) either all feature vectors are co-linear (i.e.
for some vector R Ak [C]) (2) the sum μ Ei= ,4 0. The Karush Kuhn-Tucke
(KKT) conditions are:
1x 2
Yk
A (1 1
1) = 0
AA
d 2 0
Vk
[ || S
Ak
C form a simplex ETF, n μ 1+ My ps When the learned class prototypes {p:]
1 1- 0. Therefore, we have = 0, A = 1, | and KKT conditions are satisfied. Particularly, | 1 means that all vectors also optimal for the original problem LT. The solution is optimal for Leo as Jensen's 's inequality (13) becomes equality when (m)) form simplex ETF. The above analysis provides insights why Cuop promotes inter-class separation.
the unit hypersphere and thus the solution is
27


Reviews:
[{"review_id": "z6SZLWi3WN", "reviewer": "Reviewer_HW2k", "summary": "This paper proposes a new loss function tailored to the out-of-distribution problem, where generalisation of the algorithm is required across multiple (and sometimes unseen) environments. Inspired by prior theoretical work, the authors devise an algorithm that encourages samples with the same label to be learnt by features that are as stable as possible across environments, while at the same time encouraging embeddings to look very dissimilar for data points with different labels. They achieve this by embedding points on the sphere and introducing class centroids per label (shared among environments), where points are encouraged to lie close to their corresponding centroid, and centroids themselves are pushed apart. The authors derive a theoretical guarantee for their algorithm and demonstrate its empirical success on CIFAR10-C, PACS and similar.", "questions": "1. I think Equation (5) has some typos, shouldn’t the embedding $z_i$ also depend on the environment $e$, i.e. $z_i^e$? If one interprets this equation “literally” you would be summing over the same $z_i$ over and over again.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well-written which made it (mostly) easy to follow as well as a pleasure to read for me. \n2. The suggested loss function is very intuitive and I like the geometric interpretation the authors provide in terms of the Mises-Fisher model. The visualisation in Fig 4 is also very neat. Empirical performance is also very strong across the different explored tasks.", "weaknesses": "1. I struggle to see how Theorem 5.1 connects back to the proposed loss function. From Theorem 3.1 we know that $\\nu^{\\text{sup}}$ serves as an upper bound to the OOD error, and then Theorem 5.1 in-turn provides an upper bound for $\\nu^{\\text{sup}}$ in terms of the  Rademacher complexity and some additive constants. Which term here is the loss trying to minimise here? The Rademacher complexity is over any $\\sigma_i$, so its sign has nothing to do with the true labels. I don’t see how the developed loss would encourage to minimise this quantity. It’s also a worst-case bound in terms of the hypothesis $h$, so again I don’t see how that could be minimised. I hope the authors can elaborate on this connection.\n2. The CIFAR10-C results look strong but only naive ERM is provided as a baseline. How does the approach fair against more specialised algorithms. I don’t expect this novel approach to be state-of-the-art but it would be nice to know where it stands among more modern algorithms.\n3. While the authors do compare against [1], I think the paper would benefit from a more in-depth comparison of the two losses. I’m also a bit confused as to why the results of [1] are not reported in Table 1 but only in a separate ablation in Table 2. Could the authors clarify this?\n\n\\\n\\\n[1] Yao et al, Pcl: Proxy-based contrastive learning for domain generalization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "scL2oRt9Rc", "reviewer": "Reviewer_W1ps", "summary": "This paper introduces a pracical algorithm for achieving provable out-of-distribution (OOD) generalization. The proposed approach is motivated by recent theoretical work that decomposes OOD generalization into two measurable quantities: intra-class variation and inter-class separation. This paper designs a training objective (and representation space) where these terms can be optimized to achieve low OOD generalization error.\n\nSpecifically, the proposed method learns representations for each data point that lie on a hypersphere. The goal is to encourage data points belonging to the same class to lie close together on the hypersphere (in terms of cosine distance) but to have the centroids of each class lie far apart. This approach itself is not particularly novel, as several learning methods have previously been proposed that utilize hyperspherical embeddings. But this paper is the first to provide a theoretical justification angled at OOD generalization.\n\nThe paper provides a formal theoretical proof that bounds the OOD generalization error via a standard PAC-like learning bound. The proof leans on the prior theoretical results that motivated this work.", "questions": "- I'd appreciate it if the authors could explain the motivation behind Equation 6 a little more.  Is the primary goal to improve on the computational efficiency of computing the average across all training data points? Or is there another benefit to adopting an exponential moving average? This also ties loosely into my next question.\n\n- How strong is the assumption that the samples are aligned? Intuitively, the intra-class variation measures how much the features vary across environments for a single class. The alignment assumption is an assumption over all of the training data in the available environments. Consequently, Theorem 5.1 consists of a term that depends on epsilon, and a generalization term that (intuitively) describes generalization to the unavailable environments. I think it would be more valuable if one could show that the alignment assumption is satisfied by reducing the training loss directly, bringing the result more in line with typical PAC generalization bounds.\n\n- The epsilon factor could potentially make the bound very loose if it is too close to 1. Given that this value is easy to compute, I would be curious to know what epsilon looks like for some of the models trained in the experiments.\n\n- Ye et al. provide a specialized result for linear models (Theorem 4.2 in their work). I see this as a justification that the theoretical framework can be realized by some model. However, in the present work, it is unclear whether the vMF distribution can satisfy the regularity conditions for some choice of environment distribution(s). In other words, how do we know that the OOD generalization bounds can actually be computed for the choice of model used?\n\n\nMinor comments:\n\n- In the introduction, I'd recommend replacing the four lines of citations with a survey paper, for example [1]. The full list of references could be included in the related work, or even as an extended discussion of related work in the supplementary material.\n- [2] is another reference that explores a contrastive metric learning approach for hyperspherical embeddings. The goal here is not to do OOD generalization, but the algorithm is modestly similar.\n- In proof of Theorem 5.1, \"at last $1 - \\delta$\" -> \"at least $1-\\delta$\".\n- It would be nice if Table 1 were sorted by ascending average accuracy.\n\n\n[1]: Domain Generalization: A Survey, Zhou et al.\n[2]: Video Face Clustering with Unknown Number of Clusters, Tapaswi et al. ICCV 2019", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper proposed a simple algorithm that is easy to implement. The loss terms can be computed efficiently and are easy to mini-batch for SGD. The authors provide a clear description of the algorithm and even include pseudo-code. It would be easy to reproduce the proposed method.\n\nThe paper is well-written and easy to follow. Motivation is laid out clearly and the paper accurately describes its contributions relative to prior work. I was able to find all of the information that I wanted while reading the paper either within the main text or the appendices.\n\nI see the primary contribution of this work to be the formal theoretical guarantee on the generalization performance of the proposed method. The theoretical results presented in this work are environment agnostic in the sense that they only depend on the environments through the ability to fit the training data effectively and reduce the intra-class variation. This is a valuable contribution.\n\nThe empirical results are relatively thorough and compare HYPO (the proposed method) against a wide range of baseline methods across several tasks. The results show that HYPO performs well consistently, and is on average the best OOD classifier.\n\nI liked the simple theoretical exploration in Appendix J. This was a valuable inclusion that helped to give some intuition for the class separation loss component.", "weaknesses": "The paper lacks quantitative verification of the theoretical result. I think that this would be a valuable contribution to help give an idea of how tight/vacuous the bound is. I am mostly curious about the $\\epsilon$ term that appears in Theorem 5.1 and can be easily computed in practice.\n\nThe theoretical result shown gives a bound on the intra-class variation. This is a useful component of producing an OOD generalization bound, but it is not sufficient by itself. The results in Ye et al. require some regularity conditions that depend on the distribution over the learned representations --- this is difficult to compute in this case. From my point of view, the theoretical results in this paper provide a strong intuition for the success of the method but have not yet been demonstrated to produce a tractable OOD generalization bound.\n\nSpurious correlations are ignored in this work, though are one of the more challenging aspects of OOD generalization in practice. However, I think that this is a reasonable compromise to make at this stage.\n\nI feel that the novelty is slightly limited here. The proposed learning algorithm is a form of prototypical learning on a hypersphere. The specific loss is, to my knowledge, novel but is made up of fairly standard components. The theoretical results are novel and interesting, but are essentially an instantiation of results from prior work. Indeed, the contribution of the training loss to the generalization error is largely captured in an assumption within the theoretical statement. I do consider the overall novelty of this paper to be sufficient for me to recommend acceptance, but it has affected my overall judgment so I am including this as a weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oB6RJNMRrW", "reviewer": "Reviewer_kMBq", "summary": "This paper delves into the challenge of out-of-distribution (OOD) generalization. Building upon previous research, it introduces the HYPO learning algorithm aimed at reducing intra-class variation while increasing inter-class separation. Notably, the paper establishes a connection between the loss function and the von Mises-Fisher (vMF) distribution. Subsequently, it provides a generalization upper bound of variation. These set HYPO apart from an existing work PCL. Extensive experimentation on OOD benchmarks showcases the superior performance of the HYPO algorithm.", "questions": "See weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and well-organized.\n- The problem studied in this paper is interesting and important.\n- The authors have provided a clear discussion of the relation to previous work, PCL.", "weaknesses": "1. The theoretical result appears to have limitations. \n- Although Theorem 5.1 provides insights into the upper bound of generalization variation, it does not conclusively demonstrate the superiority of the proposed method or loss, since the theorem directly assumes that the variation can be optimized to a small value under the proposed loss, i.e., $\\frac{1}{N}\\sum_j\\mu_{c(j)}^T z_j\\ge 1-\\varepsilon$. If one were to substitute an alternative loss, such as changing the prototype to another sample within the same class (e.g., employing the SupCon loss) or directly using PCL's loss, it would also yield a generalization bound. Consequently, the question arises: How can we establish that the proposed loss is indeed superior, provably?\n- Theorem 5.1 cannot be valid unless we explicitly specify the distribution distance  $\\rho$.\n- Theorem 5.1 does not account for the influence of inter-class separation, a key aspect that this paper seeks to enhance through the second term in loss eq. (5). I notice that in Ye et al (2021)'s Theorem 4.1, function O(.) also depends on additional factors beyond just the variation.\n\n2. Training Loss.\n- Since prototypes $\\mu_i$ are updated in an EMA manner, it's worth noting that the second term in eq. (5) will not generate a gradient for $h$. Consequently, the second term of the loss becomes devoid of meaning.\n\n3. The idea is quite straightforward and shares many similarities with proxy-based contrastive learning methods. Is there any additional insight that I might have overlooked?\n\n4. The empirical improvements appear to be marginal, as indicated by the data in Tables 1 and 2.\n\nOverall, I think the theoretical contribution and empirical enhancements appear to have room for further development and strengthening.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)