Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler
Authors: Kunyu Peng
Di Wen
Kailun Yang
Ao Luo
Yufan Chen
Jia Fu
M. Saquib Sarfraz
Alina Roitberg
Rainer Stiefelhagen
Venue: NeurIPS 2024 poster (2024)
Keywords: open set domain generalization
image classification
Primary area: machine_vision

Abstract: In Open-Set Domain Generalization (OSDG), the model is exposed to both new variations of data appearance (domains) and open-set conditions, where both known and novel categories are present at test time. The challenges of this task arise from the dual need to generalize across diverse domains and accurately quantify category novelty, which is critical for applications in dynamic environments. Recently, meta-learning techniques have demonstrated superior results in OSDG, effectively orchestrating the meta-train and -test tasks by employing varied random categories and predefined domain partition strategies. These approaches prioritize a well-designed training schedule over traditional methods that focus primarily on data augmentation and the enhancement of discriminative feature learning. 
The prevailing meta-learning models in OSDG typically utilize a predefined sequential domain scheduler to structure data partitions. However, a crucial aspect that remains inadequately explored is the influence brought by strategies of domain schedulers during training. 
In this paper, we observe that an adaptive domain scheduler benefits more in OSDG compared with prefixed sequential and random domain schedulers. We propose the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS) to achieve an adaptive domain scheduler. This method strategically sequences domains by assessing their reliabilities in utilizing a follower network, trained with confidence scores learned in an evidential manner, regularized by max rebiasing discrepancy, and optimized in a bilevel manner. We verify our approach on three OSDG benchmarks, i.e., PACS, DigitsDG, and OfficeHome. The results show that our method substantially improves OSDG performance and achieves more discriminative embeddings for both the seen and unseen categories, underscoring the advantage of a judicious domain scheduler for the generalizability to unseen domains and unseen categories. The source code is publicly available at https://github.com/KPeng9510/EBiL-HaDS.

Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler
Kunyu Peng', Di Wen', Kailun Yang?; Ao Luo Yufan Chen', Jia Fu M. Saquib Sarfraz Alina Roitherg', Rainer Stiefelhagen' Karlsruhe Institute of Technology HHunan University Waseda University 'KTH Royal Institute of Technology RISE Research Institutes of Sweden Mercedes Benez Tech Innovation University of Stuttgart
Abstract
In Open-Set Seen-Settii Domain Generalization (OSDG), the model is exposed to both new variations of data appearance (domains) and open-set conditions, where both known and novel categories
present test time. The challenges of this task arise from the dual need to generalize across diverse domains and accurately quantify category novelty, which is critical for applications in dynamic environments. Recently, meta-learning techniques have demonstrated superior results in OSDG, effectively orchestrating the meta-train and -test tasks by employing varied random categories and predefined domain partition strategies. These approaches prioritize a well- designed training schedule augmentation and the enhancement of discriminative feature learning. The prevail- ing meta-learning models in OSDG typically utilize predefined sequential domain scheduler to structure data partitions. However, crucial aspect that remains inade- quately explored is the influence brought by strategies of domain schedulers during training. In this paper, we observe that an adaptive domain scheduler benefits more OSDG compared with prefixed sequential and random domain schedulers. We propose the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDSS) to achieve an adaptive domain scheduler. This method strategically sequences domains by assessing their reliabilities in utilizing a follower network, trained with confidence scores learned in an evidential manner, regularized by max rebi- asing discrepancy, and optimized in a bi-level manner. We verify approach on three OSDG benchmarks, i.e., PACS, DigitsDG, and OfficeHome. The results show that our method substantially improves OSDG performance and achieves more discriminative embeddings for both the seen and unseen categories, under- scoring the advantage of judicious domain scheduler for the generalizability unseen domains and unseen categories. The source code is publicly available at https: / / thub. /aoom//Peggg55 0/ O/EBiLL HaDS.
traditional methods that focus primarily data
Introduction
Open-Set Domain Generalization (OSDG) is challenging task where the model is exposed to both: domain shift and category shift. Recent OSDG works often take a meta-learning approach [54, 46] which simulates different cross-domain learning tasks during training. These methods conventionally use predefined sequential domain scheduler to create meta-train and meta-test domains within each minibatch. But is fixing the meta-learning domain schedule a priori the best way to go? As step to explore this, our work investigates the new idea of adaptive domain scheduler. which dynamically adjusts the training order based on ongoing model performance and domain difficulty.
"Correponneenc:: kailun.yang hnu.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

OSDG is critical for many real-world applications with changing conditions, ranging from health- care [33] and security [5] to autonomous driving [19]. Despite the remarkable success of deep learning, the recognition quality often deteriorates when facing out-of out-of-distribution samples. This problem is amplified OSDG settings, where the model faces a dual challenge of identifying and rejecting unseen categories, e-g., by delivering low confidence score in such cases [27, 46] while simultaneously generalizing well to unseen data appearances (domain shift). Historically. research efforts in OSDG have predominantly focused on the latter, developing methods to adapt models to varying domain conditions. Strategies to improve domain generalization include the use of Generative Adversarial Networks (GANs) [4], contrastive learning [65]. and metrie learning [27]. MLDG [46] for the first time proposed to use meta-learning to handle OSDG tasks. However, all these works follow the OSDG protocols where different source domains preserve different distributions of known categories, which diverges the domain gaps for different categories. This divergence makes the challenge more inclined towards the domain generalization aspect. Wang et al. [54] revised these benchmarks for OSDG, standardizing the category distribution across source domains to achieve a more balanced evaluation of both domain generalization and open set recognition challenges. This revision includes established open-set recognition methods such as Adversarial Reciprocal Points Learning (ARPL) [8]. Recently, Wang al. [54] introduced an effective meta-learning approach named MEDIC, featuring a binary classification and a predefined sequential domain scheduler for the data partition during meta-train and -test stages.
However, these existing meta- peatallerning----e brannigg-based OSDG approaches, i.e., MEIDC [54] and MLDG [46], do not consider how the order in which domains are presented during training affects model gener- alization. We believe this overlooks the potential to dynamically adapt the domain scheduler used for data partition based certain criteria, such domain difficulty, which could result in a more targeted training strategy and, therefore, better outcomes. In this paper, we observe that different ordering strategies for domain presentation used for data partition during the meta-training and testing phases lead to significant variations in OSDG performance, emphasizing the critical role of domain scheduling in optimizing model generalization.
To bridge this gap, we introduce a new training strategy named the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which allows dynamically adjusting the order of domain presenta- tion during data partitioning in the meta-training and -testing phases. The key idea of our method is to quantify domain reliability, defined as the aggregated confidence of the model on the samples across unseen domains, which will then be used as the main criterion for the data partition. To assess the domain reliability, incorporate secondary follower network to assess the domain reliability alongside the primary network. This allows for prioritizing the optimization of meta-learning on less reliable domains, facilitating an adaptive domain scheduler-based data partitioning. This follower network is trained using bi-level optimization, which involves a hierarchical setup where the solution to a lower-level optimization problem (evaluating domain reliability) serves as a constraint in an upper-level problem (meta-learning objective). Optimization of the follower network is guided by confidence scores generated through our proposed max rebiased evidential learning method, which adjusts the confidence by amplifying the differences between the decision boundaries of different classes. As a result, the follower network can better quantify the reliability of each domain based on how distinct and consistent the classification boundaries are, improving the ability to generalize to unseen domains. EBiL--HaDS enhances cross-domain generalizability and differentiation of seen and unseen classes by prioritizing training on less reliable domains through adaptive domain scheduling.
Our experiments demonstrate the effectiveness of domain scheduling via EBiL-HaDS on three established datasets: PACS [31], DigitsDG [63], and OfficeHome [51], which span a variety of image classification tasks. Results demonstrate that EBiL-HaDS significantly improves model generalizability in open-set scenarios, enhancing domain generalization and the model's 's ability to distinguish between known and unknown categories in new domains. This performance surpasses that of both random and standard sequential domain scheduling methods during training, underscoring EBiL-HaDS's potential to advance current OSDG capabilities in deep learning.
2 Related Work
We simultaneously address two challenges: domain generalization and open-set recognition. Domain generalization is task that expects model to generalize well to unseen domains while leveraging

---

multiple seen domains for training [48, 52]. Open-set recognition, on the other hand, aims to reject unseen categories at test-time, e.g., by delivering low confidence scores in such cases [16].
Domain generalization methods usually alleviate the domain gap with techniques such as data augmentation [55, 40, 65, 18, 64, 34, 35], contrastive learning [57, 24, 28, 43], domain adversarial learning [15]. domain-speciin normalization [44], and GANs-based methods [10, 32]. For open-set recognition, common approaches include logits calibration [3, 42], evidential learning [59, 53, 62, 2], reconstruction-basee approaches [58, 22], GANs-based methods [29], and reciprocal point-based approaches [8, 9].
A considerable cluster of research utilizes source domains that encompass diverse categories for train- ing, highlighted in [14, 47, 4, 8, 36, 61], where each category poses unique domain generalization challenges. The primary focus of these methodologies is to improve domain generalizability, and they tend to allocate less attention to the complexities associated with open-set scenarios. In this setting, the categories involved in each source domain may not be the same. ODG-Net proposed by Bose et al. [4] leverages GAN to synthesize data from the merged training domains to improve cross-domain generalizability. SWAD proposed by Chen et al. [8] uses models averaged across various training epochs. Katsumata er al. [27] propose use metric learning to get discriminative embedding space which benefits the open-set domain generalization. Wang et al. [54] introduce a new MEDIC model together with a new formalization of the open-set domain generalization protocols, where the source domains share the same categories defined as seen. This benchmark definition balances the impact of the model's 's open-set recognition and domain generalization performance in evaluation, which is adopted in our work.
Newer OSDG approaches show great promise of meta-learning strategies for improving cross- domain generalization [54, 46]. Yet, these works mainly utilize a fixed, sequential scheduling of source domains during training [54, 46]. Existing works in curriculum learning indicate that using a specific training order at the instance level can benefit the model performance on various tasks [56, 26, 17, 37, 20, 41, 28, 45], However, existing curriculum learning approaches usually operate at the instance level (scheduling individual dataset instances within standard training) and are not designed for OSDG tasks, while we focus on domain-based scheduling by quantifying the domain difficulty in meta-learning. The influence of domain scheduling in the OSDG task remains unexplored. This paper, for the first time, examines the effects of guiding the meta-learning process with an adaptive domain scheduler, named HBiL-HHDDS, which achieves data partition based on domain reliability measure estimated by a follower network, trained in a bi-level manner with the supervision from the confidence score optimized by novel max rebiased discrepancy evidential learning.
3 Method
The most challenging domain is chosen to perform data partitioning for the meta-task reservation during meta-learning. In our proposed EBiL-HaDS, we first utilize max rebiased discrepancy evidential learning (Sec. 3.1) to achieve more reliable confidence acquisition, which is subsequently used as the supervision for the reliability prediction of the follower network. During the training stage, our method makes use of two networks with identical architectures: one serves as the main feature extraction network, and the other functions as the follower network, aiming to assess the domain reliability. The follower network is optimized in bi-level manner alongside the main network (Sec. 3.2). Hardest domain selection is accomplished by aggregating votes for samples from each domain for the randomly selected reserved classes using the follower network (Sec. 3.3).
To optimize domain scheduling, we first define the term domain reliability, as the degree to which data from a domain consistently aids in improving the model's 's accuracy and generalizability across unseen domains. An important step is therefore to adaptively rate the domain reliability during training. To achieve this, we employ two parallel networks: the main network used for feature extraction, and the follower network, which assesses the reliability of different domains based on the refined confidence metrics. This follower network plays a central role in our adaptive domain scheduling strategy: it employs a voting process to identify and select the most challenging domains those that exhibit the least reliability according to its assessments. After selecting the hardest domain, the data divided into two sets. One set includes data from more challenging domains outside the reserved classes and data from more reliable domains within the reserved classes, which together form the
3

---

meta- veta-training set. The complementary partitions of the meta-training set are used as the meta-testing Both networks are simultaneously optimized through bi-level training approach, meaning that the outcome of lower-level optimization problem (evaluating domain reliability) is constraint in an upper-level problem (meta-learning objective). The entire pipeline during training when using the proposed EBiL--HaDS is depicted in Alg. 1.
3.1 Max Rebiased Discrepancy Evidential Learning
The domain scheduler we propose leverages follower network to ascertain reliability measurements, based on reliability evaluations conducted before the start of each epoch for all souree domains. As such, confidence calibration is crucial for the main network's ' functionality. Evidential learning, which has been extensively applied across various domains such as action recognition [62] and image classification [25], effectively calibrates these confidence predictions. However, a notable limitation of evidential learning is its propensity for overfitting, leading to suboptimal performance [11].
To address these challenges, we propose to regularize the evidential learning by novel rebiased discrepancy maximization, which employed for the confidence calibration of the main network to encourage diverse decision boundaries. This method involves training dual decision-making heads designed to exhibit rebiased maximized discrepancies. The aim is to foster the development of both informative and dependable decision- maaonn capabilities within the leveraged deep learning model. Let denote batch of data used in training, Ma denote the feature extraction backbone, Rs, and Ra, denote the two rebiased layers, and A. denote the Gaussian kernel to reproduce the Hilbert space. We first calculate the max rebiased discrepancy regularization by Eq. 1,
RRB(x;0) = Σ (M (E(1.2)
Re, (M, - 2 E| (Ma(x)) Rg, (M&())))
(1) We aim to maximize the above loss function to achieve the maximum discrepancy between the em- beddings extracted from the two rebiased layers. This maximization encourages the learned evidence from the two layers to diverge from each other, thereby capturing open-set domain generalization cues from two different perspectives. Deep evidential learning is then applied the conventional classification head, providing an additional constraint to achieve more reliable confidence calibration, as described in Eq. 2.
e
LIBEly, x;0) = Σ (E(1.2)
ly (log S; log( Ro, (Ma(x)). + +1)]] - Rxx(x;)) O) (2)
where S. SC=EC (Dir(p) Re, (Ma + denotes the strength of a Dirichlet distribution, ye is the one-hot annotation of sample x from class p is the predicted probability. The two rebiased layers are engineered to capture distinct evidence by employing max discrepancy regularization. By averaging the logits produced by the two prediction heads on the top of the two rebiased layers for the conventional classification on the seen categories, we can harvest the final estimated confidence score. This score is subsequently utilized supervise the follower network, elaborated in the following subsections.
3.2 Follower Network for Reliability Learning
To establish an adaptive domain scheduler for the OSDG task, the most straightforward approach would involve training network to directly predict the sequence in which domains employed during the training phase for sample selection. However, this method does not facilitate gradient computation, thereby preventing the direct optimization of the scheduler network.
In this work, we propose an alternative method where follower network is trained to assess the reliability of each sample, utilizing predicted confidence scores derived from max discrepancy evidential learning as supervision. Throughout the training process, we employ samples from various domains to collectively assess reliability. Additionally, we utilize a follower network, denoted as Mj, which mirrors the architecture of the main network, with classification heads replaced by one regression head. θ indicates all the parameters in the main network, including the parameters from

---

Algorithm Training with Evidential Bi-Level Hardest Domain Scheduler.
Require:
Known domains D; Known classes C; backbone Mai two rebiased layers Ra, and Rag:
two heads H2 and Ház follower scheduling network Ma: weighted cross entropy WC E; mean squared eπror MSE.
while not converged do
2: Randomly select two known classes C₁2 c:;
3: Get the hardest domain d" using M3 by Eq. 5; Select two domains from di, d, <- 2,,d,,-. )D/(d);:
4: Sample data Su S2g = {xlex,del|At: E j]], (xlex,eelccr E e;), {d;
5: Construct meta-train set by Qm--train = S2 Uh:
6: Meta-train:
7: for x in Im-ttran do:
8:
Extract rebiased embeddings fi = Rg, (x)) and f2 = Rg,
9:
Obtain the max rebiased discrepancy evidential learning loss LIBE (x) using fi and f;;
10:
Follower learning LREG (x) = M (Mx(( EnEtL Con (fx)));
11:
Obtain classification loss LcLs(x) = (, (WCE(Hg, (Ma (M)x))), w +
Mx(x), where y indicates the classification annotation; end for
12:
13:
Lm- Lm--train -xxeSmm trein (Lors(( (x) + LREG(X) LRRE(X)). Backpropagation and pa-
rameter update for the whole network: Meta-test:
14:
Sample data 2g: 52 {xlev,delles lou C,, de E {d (xlendallee E C/tes, cj du {d, d, J. Construct meta-test set Sm = S% USE
Obtain loss for meta-test Lm-test + ExE(Amm=uee) (LCLs(x) + LREG(X) + +o++LLEE LRBE(x));
Back propagation and parameter update using T.all Tim test + I'm -trann
18: end while
the backbone, rebiased layers, and heads. We aim to solve the optimization task, as shown in Eq. 3.
-- arg min (M)( Ma-(x)) subject 8 arg min (M)( Mx()), (3)
θ
where w' indicates the instance wise reliability which serves as the weight for each instance during the loss calculation. Substituting the best response function (*(O) = arg ming (MM Mx)))) provides single-level problem, as shown in Eq. 4.
+ = arg min Im (9, 8 (e)),
(4)
where L, denotes classification loss (LCLs) and LABE- Ly denotes the regression loss (LREg).
3.3 Hardest Domain Scheduler during Training
We illustrate the details of the training procedure by the proposed domain scheduler in Alg. We adopt the meta-training framework outlined by MLDG [46], integrating our proposed domain scheduler facilitate the data partition of meta-tasks. In this approach, optimization is achieved using both the meta-train and -test sets, characterized by distinct data distributions. For each domain present the training dataset, we sample a batch that encompasses the reserved categories. Subsequently, we identify the most challenging domain by determining which domain exhibits the lowest reliability under the selected seen categories. This procedure is accomplished by the calculation the expected reliability as in Eq. 5.
(x((c (cd))) (Ma(x)
d' = arg min (fwald E D)), Wa = min
cxp
1
* (0.1 + o
(5)
ccC
No
where d' denotes the estimated hardest domain. N and C- denote the number of samples from domain d and the number of selected known categories at the start one epoch. D denotes the known domains used during the training procedure. x (c.4) indicates the i-th sample from class and domain d. 7a indicates the schedule frequency for domain the past training period, which considers the balance of different domains.
5

---

Table 1: Results (%) of PACS on ResNet18 [21]. The open-set ratio is 6:1.
Photo (P)
(A) H H-sooce
Carioon (C)
Sketcch (S)
Arg
Method
Acc
H-scare
OSCR
Asc
OSCR
Acc
H-seore
OSCR
Acc
H H-score
OSCR
Acc
H-score
OSCR
OpenSlax [3]
95, 36.56
92.48
$3.68
69.61
76.61
64.36
70.89
50.67
82.19
09.25
BRM (50)
96.04
93:40
96.11
84.18
70.54
71.89
72.63
62.80
62:57
70.44
55.81
51.75
82.07
70.64
70.83
ARPL|S]
94,83
95.06
94.63
83.93
07.88
08.82
78.56
02.9%
65.30
74.34
61.20
59.80
82.91
7L.78
72.14
MMLD [19)
94,83
88.30
92.94
84.43
68.83
69.43
77.11
64.21
65.36
75.14
67.70
64.69
82.88
71.38
73.11
RSC (23)
94.43
88.37
91.38
83.36
70.27
73.55
78.09
65.13
66.15
77.16
52.95
02.31
83.26
09.19
73.35
DAML [45]
91.44
80.37
82.53
83.11
72.05
71.25
79.11
66.26
66.46
82.97
72.63
73.71
84.16
72.95
73.00
MixStyle [65]
95.22
82.02
$8.59
36.18
70.62
72.57
78.92
03.25
63.81
10.34
71.90
72.07
85.17
71.94
74.35
Selliee (28)
95.72
89.34
92.26
86.24
72.45
73.37
S1177
65:75
66.38
78.30
67.106
05.66
85.26
72.05
74.53
MLDG [30]
04.99
01.48
93.70
84.3 64.12
09.52
72.15
78:45
61.59
04:32
79.99
69.67
68. 60
84.30
73.05
74.69
MVDX (60]
04.43
74.07
88.07
87.62
71
75.05
81.18
63.95
66.34
82.41
73.55
83
96.41
70.89
75.82
OeG-NNt [4]
93.54
89.39
89.76
85.74
72.36
73.41
$1.59
07.04
67.99
79.89
61.57
67.
85.19
72.59
74,66
MEDIC [54]
94,83
83.68
90.30
86.20
69.35
74.16
$1.94
63.26
67.43
81.84
60.60
70.85
96.20
71.47
75.09
MEDIC-Bel [54]
94,83
89.49
92.40
86.20
73.82
75.58
81.94
66.26
69.04
81.84
74.37
74.52
86.20
75.98
77.89
HBiL.HDDS (ours)
05.80
01.54
94.62
87.24
71.87
74.15
82.96
68.55
71:62
X3.21
74.399
74.50
87.31
76.71
78.72
HBiL. Juds (ours)
95.80
93.30
94.42
87.24
75.66
77.19
82.98
07.57
72:22
X3.21
78.29
77.52
87.31
78,66
59.34
Table 2: Results (%) of PACS on ResNer50 [21]. The open-set ratio 6:1.
Photo (P)
(r)
Cartoon (C)
Sketch
Arg H-sutre
Method
Asc
OSCR
H-sooee
OSCR
OSCR
OSCR
OpenMax [3]
97.58
93.09
38.37
73.91
$4.38
68:25
80.07
68.06
$7.60
75.82
ARPL
97.09
38.24
80:32
67.19
75.08
70.01
69:47
86.52
78.74
MIRO
94.85
92.32
88.51
79.01
42.58
63.05
H2.22
70.61
MLDG
96.77
95.85
87.99
77.16
79.93
83.45
08.74
H2.25
72.27
78.73
HRM [50)
65.79
70.10
12
CIRI. [3X]
87.75
77.4 44
68.82
84.35
MicShyle [65]
05.57
68.00
84.
73.41
79.119
CrosMaathh [67]
97
56
81
MVDG
05.00
85.02
71.05
ODG [4)
96.53
96.93
95.58
65.22
75.60
93.86
64.32
71.20
94.80
77.55
77.38
85.61
75.51
MEDIC-ele
06.37
05.80
84.67
86
75.85
84.61
75.80
76.79
06
MEDIC-ack (54)
85.81
77.39
84.61
HRil-HalD--ch
97.82
03.58
02.31
80:05
84.15
75.65
85.91
76.57
90.89
BBil-HalD--ldds
92.31
86.17
78.34
85.91
4 Experiments
4.1 Implementation Details
All the experiments use PyToch 2.0 and one NVIDIA A100 GPU. We set the upper limit of the training step as le and use SGD optimizer, where the learning rate (Ir) is set as 1e 1-3 and batch size is chosen as 16. The weights of LeLS, LREG, and LBBE are chosen as 1.0, 1e-4, and 5e-4, Lr decay is 1! and conducted at Se meta-training step. The worker number is 4 and y is 2 For ResNet18 [21] and ResNet50 [21], each rebiased layer is constructed using residual convolutional block. For ConvNet [66], convolutional layers are utilized. Apart from the conventional classification head (cls), we also utilize binary classification head (bels) as in MEDIC [54]. The training time of our method is 1h on PACS (ResNet18 [21]). 1.2h on PACS (ResNet50 [21]). 20min on DigitsDG (ConwNet [66]), 2h on OfficeHome (ResNet18 [21]). The parameter ablation is provided in the appendix.
4.2 Datasets and Metries
We adopt the open-set protocols provided by MEDIC [54], wherein the training set of each domain, shares the same categories. Three benchmarks are involved. PACS [31] comprises 4 distinct domains, i.e., photo, art-painting, cartoon, and sketch, totaling 9, 991 images. 7 classes are contained in this dataset for each domain. Digits-DG [63] aggregates 4 standard digit recognition dataset, i.e., Mnist, Mnist-m, SVHN, and SYN. Office-Home [51] includes 15, 500 images across 65 classes from 4 domains, i.e., art, clipart, product, and real-world Since MEDIC [54] did not provide detailed benchmark on OfficeHome, we construct the whole benchmark using several outstanding baselines and our approach, where the last 30 categories following the alphabet order are chosen as unseen categories. The leave-one-domain-onl DG setting adopted. Close-set accuracy (acc), H-score, and OSCR serve metrics following [54], where OSCR is the primary metric for OSDG.

---

Table 3:
Results (%) of PACS on 55 [21]. The open-set ratio is 6:1.
Photo (P)
Art H-score
Cartoon (C)
Sketch (S)
Aag
Method
Ace
H-score
OSCR
Acc
USCR
Asc
H H-scuce
OSCR
Asc
H-score
USCR
Adc
H-acore
OSCR
ARPL
94.35
85.45
16.74
59.81
71.27
78.53
$3.91
69.75 05.75
72.05
77.53
52.70
66.68
77.53
69.51
76.01
MLDG
96.20
91.07
94.64
59.81
77.05
82.19
83.36
73.66
74.03
62.89
64.30
72.9%
88.19
76.67
30.96
SWAD
95.64
34.82
59.74
86.30
73.85
75.91
78.49
70.18
08.41
76.92
75.33
03.35
86.34
76.05
74.35
ODXI-Net
95.88
89.11
91.85
80.62
90.65
62.48
85.35
70.37
73.66
79.30
77.00
72.22
87.49
79.38
80,05
MEDIC-cl
94.67
49.54
76.95
80.37
73.26
77:79
86.59
68.49
74.82
85.81
56,14
78.83
89.01
61.86
77.11
MEDIC-hale
94.67
72.88
$1.30
80.37
74.92
76.70
86.59
71.46
75.17
85.81
$8,80
78.32
89.11
69.52
78.37
FRI Haat--lh (ours)
97.90
01.66
09-62
92.06
81.52
85:43
87.21
76.61
78.19
87.08
81.13
60.21
91.06
82:73
85.11
ERiL HallS-hllleelleee (ours)
97.90
94,34
97.39
92.116
82,00
85.94
87.21
76.62
80.15
87.00
18.57
K1.52
91.06
85.38
86,25
Table 4: Results (%) of PACS on ViT base model [13] (patch size 16 and image size 224). The
open-set ratio is 6:1.
Photo (P)
Art (A)
Cartoon (C)
Sketchh (8)
Avg H-score
Method
Ace
H-spoee
USCR
Ace
H-spoee
USCR
Ace
H-spoee
OSCR
Aee
H-score
OSCC
Ase
OSCC
ARPI.
99.19
95.31
95.61
90.49
85. 4G
$8.59
81.58
72.17
73.34
63.01
29.33
50.50
$3.64
70:57
77.78
MLDG
99.19
95.40
98.88
91.87
82.46
89.47
80.56
69:62
74.19
61. 63.66
411.79
43.83
53.52
72.07
76.61
SWAD
98.55
93.19
07.62
90.81
81.
89.52
83.24
73.03
76.59
57.89
35.83
41.66
$2.62
70.55
76.30
ODG-Ner
97.58
96.24
05.25
90.49
83.32
87.50
82:36
08:66
75.30
62.59
43.59
50.20
83:26
72.05
77.09
MEDIC-cls
99:03
05.33
98:22
92.06
R3:27
87:46
85.60
69:79
75.37
68
41.95
56. 56
96. 28
72.59
79.40
MEDIC-hele
90:03
96.04
97.55
92.06
82.68
87.73
85.62
69:15
76.30
68.40
39.60
55.00
86.
71.87
79.
RbiL-HaaDSS-te (ours)
99.52
97.30
99:11
94.68
85 10
92.10 10
89.22
74.31
72.76
69.49
48.34
55.35
$8.23
75.53
$1.09
EBiL.-HGDDS-ee (ours)
00.52
96.97
99.18
94.68
88.31
92.28
89.22
73.91
77.96
69.49
48.09
$6.78
88.23
76.81
$1.55
Table 5: Results (%) of Digits-DG on ConvNet [66]. The open-set ratio is 6:4.
MNIST
MNIST-M
SVHN
SYN
Avg
Method
Azc
OSCK
OSCR
OSCR
Auc
11 1l-ccoe
OSCR
Acc
11 1:scoce
OSCK
OpenMax
97.33
52.03
71.00
57:26
72.00
49. -6.46
54.33
54.78
MisStyll [65]
97.65
73.25
89:36
74.50
59.30
56.95
69.35
$3.34
45.43
85.06
69:22
61.50
65.05
ERM [50]
97:47
71.00
53.92
54.37
09.56
51.57
61.31
97.75
69:77
55.08
71.7%
56.18
65.39
MLDG [50)
97.83
80:36
94:78
71.11
46.84
55.17
73.64
86.08
63.35
70.34
82.
68
SWAD [6]
97.71
84.44
92.65
73.09
53.35
55.94
76.08
59.18
56.25
87.95
51.27
69.03
83:71
62.06
58.47
ODGGNNe (4)
96.86
71:34
90.93
72.92
55.47
56.96
69:05
55.74
51.55
85.42
67.67
68.12
81.26
63.31
66:00
MEDIC-dl [54]
97.89
67:37
95.17
71.14
48.44
55.37
76.00
51. 20
55.58
88.11
64 00
73.62
83.78
57.98
20.19
MEDIC- bels [54]
97.89
83.20
95.81
71.14
60.98
58.25
76.00
58.77
57.60
88.11
62.24
72.91
83:28
66.30
71.15
HaDD-HaDS-sck
99.50
87:40
97.49
74.28
56.58
60.86
80.33
61.27
62. 54
93.97
73.95
78:14
87.02
70.27
75.83
EDGL-IlaDS-be bels
99.50
91.63
97.58
74.28
60.33 72
59.39
80.33
62.23
63.88
93.97
69.9
79.28
87.02
71.13
75.87
Table 6: Results (%) of OfficeHome on ResNet18 [21]. The open-set ratio is 35:30.
Clipart H-score
World H-score
Product H-scoce
Aog Hiscore
Method
Azc
H-score
USCK
Acc
USCK
Auc
USCR
Acc
USCR
OpenMaxx [:]
10.48
72:92
MiaStyle [65]
62:81
53.91
50.71
52.46
42:27
81.16
67.77 77770
67.95
76.29
62.51
68. 18.18
57.41
ERM [50]
66.30
57.39
54.36
59.50
46.81
47.84
84.50
69.99
74.00
80.81
67.40
67.44
72:30
50.40
51.04
ARPI_ [5]
60:06
50.34
25.68
54.82
45.72
43.21
76.24
62:04
61.73
75.30
62.47
60.19
66.61
55.14
52.
MLDG [30]
66.56
52.45
$5.
$8.85
$3.09
47.69
80.00
70.66
70.02
75.02
66.10 16
63.49
70.13
60.59
59.08
SWAD [o]
59.12
$3.05
47.57
57.37
45.78
47.28
78.38
66.43
65.48
76.50
64.29
03322
07.54
$7.39
58.95
ODD-Neo [4]
64.
$4.97
50.64
61.06
$2.36
48.33
83.93
70.04
71.34
79.07
65.47
65.49
72.04
50.69
$8.95
MEDIC-ds [54]
66.8
55.78
55.85
61.14
54.31
48.51
85.03
71.16
73.15
80.69
67.7
68:09
73.42
62.32
61
MEDIC bels (54)
66.81
51.76
56.21
61.14
53.28
85.00
70.61
74.08
80.69
67.70
67.17
73:42
61.61
57.01
70:25
74.69
63:45
EDGL- HaDS
68.15
53.57
52.12
74.20
75.64
$1.61
71.62
74.69
4.3 Analysis of the Experimental Results on Three OSDG Benchmarks
We first validate the performances of our approach on the three well-established benchmarks for the OSDG task, e.g., PACS, DigitsDG, and OfficeHomee In Table 1, we use ResNet18 [21] as the feature extraction backbone which is pre-trained on ImageNet21 K [12] for the PACS benchmark. Compared with the state-of-the-an methods, i.e., MEDIC and ODG-Net, the proposed EBiL-HaDS achieves promising performance improvements for all the domain generalization splits. On averaged metrics across all of these DG splits, our approach delivers performance improvements by 1.11% of close-set set accuracy, 2.68% of H-score, and 2.45% of OSCR for the binary classification head (bcls), and consistent performance improvements can be found in the conventional classification head (cls). Through using EBiL-HaDS to achieve a more reasonable domain scheduler during the training, we observe that on the most challenging domain generalization split, i.e., Cartoon as unseen domain, EBiL--HaDS delivers the most performance benefits. The core strength of EBiL-HHaDS is its adaptive domain scheduling, optimized through a bi-level manner to achieve maximum rebiased discrepancy evidential learning. This ensures comprehensive and discriminative data partitions during
7

---

Table 7: Module ablation of the DigitsDG ConvNet [66].
Hoad
DGS
RBB
MNNST H-score
MNIST-M Hi-sooe
SVINN H-score
SYN
Avg
Acc
USCK
Ace
USCR
Acc
USCK
Acc
H-score
OSCR
Acc
Hiscore
OSCK
cbs
97.59
67.37
96.17
71.34
48.44
55.37
76.00
51.20
55.58
55.11
64.90)
73.62
63.28
57.98
70.19
bels
97.80
83,30
95.81
71.14
60.98
56.78
76.00
58.77
57.60
68.11
62.24
72.91
83.28
66.30
71,15
cle
99.17
79,09
95.37
21.78
60.66
57.05
18.58
60.34
59.52
91.28
70.05
76.65
85.20
67.54
72.15
bels
99.17
80.52
96.08
71.78
55,79
58. 10
78.52
61.51
60.50 50
91.28
72.31
74.55
85.20
67.35
72.41
cle
99.14
79.47
14
72.06
59.19
56 76
77.61
56.65
58.37
91.11
72.28
72.78
$4.95
66.96
70.76
99.14
22.00
73.53
cls
/
99.50
99.50
97.58
17.
74.28
74.25
60.86
50.33
50.33
62.25
61.27
75.77
75.82
75.87
75.83
training, enhancing generalization to unseen domains, which is observed from the above experimental analyses. Significant performance gains in challenging DG splits, such the unseen Cartoon domain, demonstrate its effectiveness in handling extreme domain shifts. Consistent metric improvements highlight EBiLL-HaDS's versatility across various OSDG challenges.
Further experiments on a different backbone, i.e. ResNet50 [21], are delivered in Table 2, where our method contributes 1. 1.08%. 0.94%, and 1.27% performance improvements of close-set accuracy, H-score, and OSCR for binary classification head and consistent performance improvements for the conventional classification head. EBiL-HaDS contributes more performance improvements when we compare the experimental results on ResNet18 with ResNet50 [21] for the PACS dataset, which illustrates that the EBiL-HaDS is more helpful in alleviating the generalizability issue of model- preserving light- network structure since network with small size is hard to optimize and obtain the generalizable capabilities on challenging unseen domains.
We further conduct ablation on model architecture on ResNet152 and VIT base model [13], where our proposed method is compared with the MEDIC and other challenging baselines, i.e., ARPL, MLDG, SWAD, and ODG-Net. From Table and Table we can observe an obvious OSDG performance improvement by increasing the complexity of the leveraged feature extraction backbone. However, when we compare Table 3 and Table 2, some baseline approaches trained with ResNet152 backbone even show performance decay on the major evaluation metric OSCR. This observation demonstrates that most OSDG methods face the overfitting issue when using a very large backbone, which is a critical issue for open-set challenging to recognize samples from unseen categories, especially in an unseen domain. The MEDIC-bels approach shows 6.48% performance degradation on OSCR when replace the backbone from ResNet50 to ResNet152. Using the proposed BHiL-HaDS to achieve more reasonable task reservation in meta-learning procedure. BHiL-HaDS bels delivers 86. 25% in terms of OSCR ResNet152 [21] backbone, where the OSCR performance of BHiL-HaDS -ells ResNe150 is 86.12%
This observation shows that our proposed adaptive domain scheduler can make the meta-learning effective on large complex models by reserving reasonable task for model optimization. Additional experiments on the ViT base model [13] (patch size 16 and window size 224) are provided in Table 4, where we observe that our proposed method can deliver consistent performance gains on the transformer architecture.
This observation is further validated by the experimental results on DigitsDG where a smaller network structure, i.e., ConvNet [66], is used, as shown in Table 5. Our method contributes 3.74%, 4.8 4.83%, and 4.72% performance improvements of close-set accuracy, Hcsree, and OSCR for binary classification head and 3. 74%, 12.29%, and 5.64% performance improvements of close-set accuracy, H-score, and OSCR for the conventional classification head. Consistent performance improvements are shown in Table 6 on the OfficeHome. We further observe that using EBiL--HaDy the optimized model can contribute a distinct separation between confidence scores of the model on the unseen categories and seen categories in the test unseen domain, showing the benefits of reasonable domain scheduler for OSDG.
4.4 Analysis the Ablation Experiments
We deliver the ablation experiments Table We first remove the LAFE by directly supervising the follower network using the confidence score provided by SoftMax supervised by cross-entropy loss for classification, where the results are shown in the second part of Table 7 (w/o RBE). Compared with this ablation, our method achieves 1.82%, 5. 5.24%, and 3.46% performance improvements of
8

---

Table 8: Comparison with different domain schedulers DigitsDG with open-set ratio 6:4.
MNNSTT Hoscore
MNIST-M
SVIIN H--0000
SYN H-score
Arg Hiscos
Method
Ace
OSCK
Ace
H4-s00ce
OSCR
Acc
OSCR
Acc
OSCR
Auc
OSCK
SespenialSSitol-ls
97.89
67.37
96.17
71.14
48.44
55.37
76.00
51.20
55.55
55.11
64.30
73.62
63.28
57.98
70.19
SequernallSchee..e bels
97.89
85.20
95.83
71.14
00.98
58.25
76.00
55.77
$7.60
55.11
02.24
72.91
83.28
06.30
71.15
Kandom-cls els
98.39
52.93
94.221
70.92
52.70
52.43
77.92
59.65
$7.95
58.33
44.11
75.06
K3.89
52.35
70.06
Random-bels bell
96.39
75.67
94.22
70.92
57.33
54.87
77.92
57.54
61.00
88.33
68.3 34
74.81
63.89
64.20
71.24
EBit HaDS--ls
99.50
87.40
57:49
74.28
56.58
60.86
60.33
61.37
62.84
90,07
75.82
78.14
87.02
70.27
75.81
EBit BBiL-HaDSS HaDDS-hhs
99.50
91.63
97.58
74.28
60.72
59.39
80.33
02.23
63.88
93.97
75.77
79.25
87.02
72.59
75.87
(a) ODGNet
(b) MEDIC-els
(c) MEDIC-bels
(d) Ours-cls
(e) Ours- bels
Figure 1: Comparison of open-set confidence using ResNet18 [21] on PACS. Photo is the unseen domain. We use red and blue colors to denote unseen and seen categories.
(a) MEDIC photo
(b) Ours photo
MEDIC
(d) Ours
Figure 2: The visualization of the embeddings through TSNE [49] for PACS on ResNet18 [21].
EGatt #Det BOwe
Acc-SA1
(b) OSCR-SA1
Acc-SA2
(d) OSCR-SA2
Figure 3: Ablation of different open-set ratios on DigitsDG dataset by ConvNet [66] backbone, where indicates Split Ablation. Regarding the splits, Case (denoted by red color) indicates using 8, 9, 10 as unseen categories. In (a) and (b), Case 2 (denoted by blue color) and Case (denoted by gray color) indicate that using 1, 2, and 2, 3, 4, as unseen categories. (c) and (d), Case 2 and Case 3 indicate using 7, 8, 9 and 8, 9 as unseen categories, respectively.
close-set accuracy, H-score, and OSCR for binary classification head and 1.82%, 2.73%, and 3.68% performance improvements of close accuracy, H-score, and OSCR for conventional classification head. The significant OSDG performance improvements highlight the importance of the confidence score learned by the max rebiased discrepancy evidential learning supervising the follower network. ensuring the promising reliability prediction. Then we use sequential scheduler and keep the LABE in the third part of Table (w/o DGS), where our approach outperforms this variant by 2.04%. 3. 3.92%, and 3.89% of close accuracy, H-score, and OSCR for binary classification head and 2. 2.04%. 3.31%, and 5.07% of these metrics for conventional classification head. This observation shows the importance of the proposed domain scheduler for the OSDG task and highlights the effect of using meta-learning trained with reasonable data partition. Both ablations show better OSDG performances compared with MEDIC, confirming the benefit of each component. We further deliver

---

more ablations, e.g., the benefits brought by the max discrepancy regularization term, comparison with recent curriculum learning approaches, and ablation of the rebiased layers in the appendix.
4.5 Comparison of Different Domain Schedulers for OSDG Task
We present several comparison experiments in Table 8 to demonstrate the efficacy of various domain schedulers when applying meta-learning to the OSDG task. We compare our proposed EBDL--HaDS with both the sequential domain scheduler and the random domain scheduler. The sequential domain scheduler selects domains in fixed order for batch data partitioning, while the random scheduler assigns domains randomly. Results show that EBiL- HaDS significantly outperforms both the random and sequential domain schedulers. Specifically, EBDL achieves performance improvements of 3. 3.13%. 8. 8.39%, and 4. 4.63% in closed accuracy, H-score, and OSCR for the binary classification head, and 3. 131. 17.92%, and 5. 6.77% in these metries for the conventional classification head compared to the random scheduler. This ablation demonstrates that scheduler enables the model to converge to more optimal region, which outperforms both predefined fixed-order (sequential) and maximally random (random) schedulers, underscoring the importance of well-designed domain scheduler in meta-learning for the OSDG. More ablations on domain schedulers are supplemented in the appendix.
4.6 Analysis of the TSNE Visualizations of the Latent Space
In Figure 2, we deliver the TSNE [49] visualization of the latent space of MEDIC and our approach on the OSDG splits, i.e., photo and art as unseen domains. Unseen and seen categories are denoted by red and other colors. we observe that the model trained by our method delivers more compact cluster for each category and the unseen category is more separable regarding the decision boundary in the latent space. Our method's ' ability to improve the generalizability of the model is particularly noteworthy. The well-structured latent space facilitates better transfer learning capabilities, allowing the model to adapt more efficiently to new, unseen categories. This characteristic is especially beneficial in dynamic environments where the data distribution can change over time. In essence, the effectiveness of our approach in achieving more discriminative and generalizable latent space be directly linked to the sophisticated data partitioning achieved through EBiL--HDDS. This demonstrates the profound influence that carefully designed domain schedulers can have the overall performance of deep learning models, emphasizing the need for thoughtful consideration in their implementation.
4.7 Ablation of the Open-Set Ratios and the Number of Unseen Categories
We first conduct the ablation towards different unseen categories with a predefined open-set ratio in Figure 3a and Figure 3b of close-set accuracy and the OSCR for open-set evaluation, where the performance of the ODG-NET, binary classification head, and conventional classification head of MEDIC method and our method are presented. We then conduct the ablation towards different numbers of unseen categories in Figure 3c and Figure 3d of close-set accuracy and the OSCR for open-set evaluation, where the performance of the ODG-NET, binary classification head, and classifi- cation head of MEDIC method and our method are presented. From the experimental results and comparisons, we can find consistent performance improvements, indicating the high generalizability of our approach across different open-set ratios and unseen category settings.
Conclusion
In this study, we introduce the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS) for the OSDG task. EBiL-HaDS is designed to create an adaptive domain scheduler that dynamically adjusts to varying domain difficulties. Extensive experiments
diverse image recognition tasks across three OSDG benchmarks demonstrate that our proposed solution generates more discriminative embeddings. Additionally, it significantly enhances the performance of state-of-the-an techniques in OSDG, showcasing its efficacy and potential for broader applications for deep learning models.
Limitations and Societal Impacts. EBiL-HaDS positively impacts society by enhancing model awareness of outtof-distribution categories in unseen domains, leading to more reliable decisions. emphasizes the importance of domain scheduling in OSDG. However, the method may still result in misclassification and biased predictions, potentially causing negative effects. EBiL--aDDS relies on source domains with unified categories and has far only been tested on image classification.
10

---

Acknowledgements
The project served to prepare the SFB 1574 Circular Factory for the Perpetual Product (project ID:: 471 471687386), approved by the German Research Foundation (DFG, German Research Foundation) with a start date of April 2024. This work was also partially supported by the SmartAge project sponsored by the Carl Zeiss Stiftung (P2019-01-003; 2021-2026). This work was performed on the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden- Württemberg and by the Federal Ministry of Education and Research. The authors also acknowledge support by the state of Baden-Württemen Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG. This project is also supported by the National Natural Science Foundation of China under Grant No. 62473139, Lastly, the authors thank for the support of Dr. Sepideh Pashami, the Swedish Innovation Agency VINNOVA, the Digital Futures.
References
[1] Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization the unseen, logic reasoning and degree curriculum. In ICML, 2023.
[2] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In ICCV, 2021.
[3] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR, 2016.
[4] Shirsha Bose, Ankit Jha, Hitesh Kandala, and Biplab Banerjee. Beyond boundaries: A novel data-augmentationn discourse for open domain generalization. TMLR, 2023.
[5] Pau Panareda Busto, Ahsan lqbal, and Juergen Gall. Open set domain adaptation for image and action recognition. TPAMI, 2018
[6] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho. Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In NeurlPS, 2021.
[7] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. In ECCV, 2022.
[8] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. TPAMI, 2022.
[9] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In ECCV, 2020.
[10] Keyu Chen, Di Zhuang, and J. Morris Chang. Discriminative adversarial domain generalization with meta-learning based cross-domain validation. Neurocomputing, 2022.
[11] Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Am Ann Heng. Uncertainty estimation by fisher iiatmatioo-bbsed evidential deep learning. In ICML, 2023.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Bo Fu, Zhangjic Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In ECCV, 2020.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Dovaainadversarial training of neural networks. JMLR, 2016.

---

[16] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recogni- tion: survey. TPAMI, 2021.
[17]
Tiantian Gong, Kaixiang Chen, Liyan Zhang, and Junsheng Wang. Debiased contrastive curriculum learning for progressive generalizable person re-identification. TCVST, 2023.
[18] Jintao Guo, Na Wang, Lei Qi, and Yinghuan Shi. Aloft: A lightweight mlp-like architecture with dynamic low- low-frequency transform for domain generalization. In CVPR, 2023.
[19] Xiaoqing Guo, Jie Liu, Tongliang Liu, and Yixuan Yuan. Simt: Handling open-set noise for domain adaptive semantic segmentation. In CVPR, 2022.
[20] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In ICML, 2019.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Hongzhi Huang. Yu Wang, Qinghua Hu, and Ming-Ming Cheng. Class-specific semantic reconstruction for open set recogmition. TPAMI, 2023.
[23] Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross- domain generalization. ECCV, 2020.
[24] Seogkyu Jcon, Kibeom Hong, Pilhyeon Lee, Jewook Lee, and Hyeran Byun. Feature stylization and domain-aware contrastive learning for domain generalization. In MM, 2021.
[25] Fengcheng Ji, Wenzhi Zhao, Qiao Wang, William J. Emery, Rui Peng, Yuanbin Man, Guogiang Wang, and Kun Jia. Spectral-spatial evidential learning network for open-set hyperspectral image classification. TGRS, 2024.
[26] Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Samarasekera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. In CVPR, 2023.
[27] Kai Katsumata, Ikki Kishida, Ayako Amma, and Hideki Nakayama. Open-set domain general- ization via metric learning. In ICIP, 2021.
[28] Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In ICCV, 2021.
[29] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In CVPR, 2021.
[30] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning generalize: Meta- learning for domain generalization. In AAAI, 2018.
[31] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. ICCV, 2017.
[32] Fan Li, Yanxiang Chen, Haiyang Liu, Zuxing Zhao Yuanzhi Yao, and Xin Liao. Vocoder detection of spoofing speech based gan fingerprints and domain generalization. TOMM, 2024.
[33] Haoliang Li, YuFei Wang, Renjie Wan, Shigi Wang, Tie-Qiang Li, and Alex Kot. Domain gen- eralization for medical imaging classification with linear-dependency regularization. NeurIPS, 2020.
[34] Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In CVPR, 2021.
[35] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M. Hospedales. simple feature augmentation for domain generalization. In ICCV, 2021.
12

---

[36] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization conditional invariant adversarial networks. ECCV, 2018.
[37] Kang Liao, Lang Nie, Chunyu Lin, Zishuo Zheng, and Yao Zhao. Recreenet: Rectangling rectified side-angle images by thin-plate spline model and dof-based curriculum learning, In ICCV, 2023.
[38] Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In CVPR, 2022.
[39] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In AAAI, 2020.
[40] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, 2021.
[41] Vaasudev Narayanan, Aniket Anand Deshmukh, Ürün Dogan, and Vineeth N. Balasubramanian. On challenges in unsupervised domain generalization. In NeurlPSW, 2022.
[42] Kunyu Peng. Cheng Yin. Junwei Zheng, Ruiping Liu, David Schneider, Jiaming Zhang, Kailun Yang, M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Navigating open set scenarios for skeleton-based action recognition. In AAAI, 2024.
[43] Mohamed Ragab, Zhenghua Chen, Wenyu Zhang. Emadeldeen Eldele, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Conditional contrastive domain generalization for fault diagnosis. TIM, 2022.
[44] Seonguk Sco, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In ECCV, 2020.
[45] Yang Shu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Transferable curriculum for weakly-supervised domain adaptation. In AAAI, 2019.
[46] Yang Shu, Zhangjie Cao, Chenyu Wang. Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmented meta-learning. In CVPR, 2021.
[47] Mainak Singha, Ankit Jha, Shirsha Bose, Ashwin Nair, Moloud Abdar, and Biplab Banerjee. Unknown prompt, the only lacuna: Unveiling clip's potential for open domain generalization. arXiv preprint arXiv:2404.007/00 10, 2024.
[48] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
[49] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
[50] Vladimir Vapnik. Statistical learning theory, 1998.
[51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[52] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generaliza- tion. TKDE, 2023.
[53] Ruofan Wang, Rui-Wei Zhao, Xiaobo Zhang, and Rui Feng. Towards evidential and class separable open object detection. In AAAI, 2024.
[54] Xiran Wang, Jian Zhang, Lei Qi, and Yinghuan Shi. Generalizable decision boundaries: Dualistic meta-leamnng for open set domain generalization. In ICCV, 2023.
[55] Yufei Wang, Haoliang Li, and Alex C. Kot. Heterogeneous domain generalization via domain mixup. In ICASSP, 2020.
[56] Yulin Wang, Yang Yue, Rui Lu. Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficientrrain: Exploring generalized curriculum learning for training visual backbones. In ICCV, 2023.
13

---

[57] Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In CVPR, 2022.
[58] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Clasifiiation-reconsttttiioo learning for open-set recognition. In CVPR, 2019.
[59] Yang Yu, Danruo Deng, Furui Liu, Qi Dou, Yueming Jin, Guangyong Chen, and Pheng Ann Heng. Anedl: Adaptive negative evidential deep learning for open-set semi semi-superrised learning. In AAAI, 2024.
[60] Jian Zhang, Lei Qi. Yinghuan Shi, and Yang Gao. Mvdg: A unified multi-view framework for domain generalization. In ECCV, 2022.
[61] Chao Zhao and Weiming Shen. Adaptive open set domain generalization network: Learning to diagnose unknown faults under unknown working conditions. Reliability Engineering System Safety, 2022.
[62] Chen Zhao, Dawei Du, Anthony Hoogs, and Christopher Funk. Open set action recognition via multi-label evidential learning. CVPR, 2023.
[63] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Deep aavaiiaadversaaial image generation for domain generalisation. In AAAI, 2020.
[64] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang, Learning to generate novel domains for domain generalization. In ECCV, 2020.
[65]
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICIR, 2020.
[66]
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. TIP, 2021.
[67] Ronghang Zhu and Sheng Li. Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization. In ICLR, 2021.
14

---

Appendix
binary classification A Further Illustration of the Evaluation Methods and Protocols
We follow the same protocol according to the MEDIC approach [54]. For PACS dataset which is used in Table I and Table 2, we use the open-set ratio 6 1, where the elephant, horse, giraffe, dog, guitar, house are selected as seen categories and person is selected as the unseen category. For the DigitsDG dataset, we leverage the open-set ratio as 6 : 4, where digits 0, 1, 2, 4, are used as seen categories while digits 6, 7, 8, 9 are selected as unseen categories, in Table 5. For the OfficeHome dataset, Mop, Mouse, Mug, Notebook, Oven, Pan, PaperClip, Pen, Pencil, PostitNotes, Printer, PushPin, Radio, Refrigerator, Ruler, Scissors, Screwdriver, Shelf, Sink, Sneakers, Soda, Speaker, Spoon, TV, Table, Telephone, ToothBrush, Toys, TrashCan, Webcam are chosen as the 30 unseen categories. The Acc indicates the close-set accuracy measured on the seen categories to assess the correctness of the classification. The H-score and OSCR are measurements for the open-set recognition which are widely used in the OSDG field. Since the H-score relies on a predefined threshold derived from the source domain validation set to separate the seen categories and unseen categories, it is regarded as secondary metric in our evaluation. MEDIC proposes OSCR for the OSDG evaluation where no predefined threshold is required, which used as our primary evaluation metric.
Regarding the calculation of the H-Score, we first have a threshold ratio À separate the samples coming from seen and unseen classes. When the predicted confidence score is below X, we regard the corresponding samples as unseen category. Then, we calculate the accuracy for all the samples regarded as seen categories according to their corresponding seen labels, which can be denoted as Acc.. The accuracy calculation of the unseen categories is conducted manner, where the label for the samples from the seen category is annotated as and the label for the samples from the unseen category annotated as O. Then the accuracy for the unseen evaluation can be denoted as Acc., The H-score is calculated as follows,
2 * Acca * Acck
H Hscore
(6)
Acc. + Acck
OSCR is a combination of the accuracy and the AUROC via a moving threshold to measure the quality of the confidence score prediction for the OSDG task. Different from the AUROC, OSDG only calculates the samples that are correctly predicted using such moving threshold, which is a combination of the calculation manner from the H-score and AUROC.
B More Ablations of the Proposed Method
B.1 Ablation of the RBE
We deliver the ablation results of schedulers with different loss function components on the Digits- DG dataset with an open-set ratio of 6:4 for close-set accuracy, H-score, and OSCR of both the conventional and binary classification heads under different dataset partitions (Mnist, Mnist-m, SVHN, SYN). The experiments are conducted by removing the whole LABE component or removing only the regularization term REB from the EBiL-HaDS method. The results show that EBiL-HaDS performs better than the other two variations on almost all the dataset partitions except the H-score of Mnist-m, as shown in Table 9, demonstrating the effect of our LRBE and the importance of the regularization term RRB in the whole LABE component.
Comparisons between EBiL-HaDS and the method without the CRBE illustrate the whole improve- ment of our deep evidential learning component with regularization term RRB. EBiL-HaDS has in average 2.04%. 3.31%, 5.07% performance increase of the conventional classification head and 2.04%, 3.92%, 3.89% increase of the binary classification head for close-set set accuracy, H-score, and OSCR, respectively. Differences between the models without LABE and without RAB demonstrate the effect of our deep evidential learning itself. The performance with deep evidential learning improves 0.66%, 0.43%, 2.47% with the conventional classification head and 0.66%, 0.73%, 0.83% with the binary classification head for close-set accuracy, H-score, and OSCR in average. On the other hand, EBBiL-HaDS has superior results than the model without regularization term RAB by 1.38% 2.88% 2.60% and 1.38%, 3. 3.19% 3.06% for close-set accuracy, H-score, OSCR with the
15

---

conventional and binary classification heads averagely. These experiments showcase the significance of regularization term RRB which contributes more than 50% performance improvement in the whole CIBE component on all 3 metrics with either conventional or binary classification head.
Table 9: Comparison with different domain schedulers on Digits-Do DGG with open-set ratio 6:4 using ConvNet [66] (Best in Bold).
MNIST H scoce
MNIST-M 11cor
SVHN H:score
SYN H-score
Avg H-scoce
Method
Acc
OSCR
Acc
OSCR
Acc
OSCR
Acc
OSCR
Ace
OSCK
wio RBE (cls)
99114
79.47
95.14
72.05
59.19
36.76
77.61
56.88
58.37
91.11
72.28
72:78
84.99
06.96
70.76
win RRF (bclk)
99.14
$1.03
96.03
72.06
66.89
$7.50
77.61
59.21
59.32
91.11
73.55
73.96
84.98
68.67
71.98
wio RB (cls) wio RB (hels)
99.19
62.72
94.67
73.89
56.47
60.78
78.17
56.54
59.54
91.31
23.81
77.92
53.64
67.39
73:23
99.19
84.89
96.13
73.89
60.03
58.75
78.17
61.49
60.20
91.31
71.17
76.17
85.64
62:40
72.81
EBaL HaDS cls BBil-HaDS--ell
99.50
87.40
97.49
74.28
56.58
60.86
80.33
61.27
62.84
93.97
75.82
78.14
57.02
70.27
75.83
74.28
02.23
75.77
87.02
75.87
B.2 Ablation of the Hyperparameters
We visualize the impact of various hyperparameter configurations on the DigitsDG dataset for close-set accuracy, well as the OSCR of both the conventional and binary classification heads under different dataset partitions (Mnist, Mnist-m, SVHN, SYN as unseen domains), as illustrated in Figure 7, Figure 8, and Figure 9. Figure 7 presents an ablation study focusing on the influences, and demonstrates that varying a from 2e 1 to 2e slightly impacts model performance, with certain settings yielding optimal results. For instance, lower values of a generally enhance the OSCR of binary classification head, particularly in more complex datasets like SVHN, suggesting that o is important for achieving a balance between robustness and accuracy in domain generalization scenarios.
In Figure 8, adjustments to the loss weight of LIFg ranging from 1e to le are analyzed, which evident that the loss weight of LREG with l yields the most optimal results across various metrics. Conversely, excessively lower loss weight of LREG appears to have a detrimental effect, potentially leading to overfitting or diminishing the model's ability to generalize effectively across different domains. Besides, Figure 9 explores the ablation of the loss weight of LRBE via adjusting LABE from 5e -1 to 5e -4 It shows a direct effect on both accuracy and OSCR, with lower weights generally improving performance, particularly for challenging datasets like SVHN, which suggests that LABE plays a critical role in the evidential learning framework, significantly influencing the the loss
model's ' ability in open-set conditions under unseen domains. Tuning of hyperparameters weights of LREG and LABE are significant for optimizing the performance of domain generalization models. Optimal settings remarkably enhance both the accuracy of meta-training and the OSCR of the classification heads, particularly under complex and challenging dataset conditions like SVHN as the unseen domain.
B.3 Ablation of the Rebiased Layers
In this subsection, we provide the ablation of the layer number used for the rebiased operation as shown in Table 10. In this ablation, 1-1 layer, 2-1 layer, and 2-2 layer indicate that the Re, and Reg are both constructed by 1 convolutional layer, constructed by 1 and 2 convolutional layers, and both constructed by 2 convolutional layers, with unified kernel size 3. We first observe that all of the ablation experiments outperform the baseline MEDIC in terms of the averaged OSDG performance, indicating that with rebiased setting our method can achieve overall OSDG performance improvements regardless of the layer constructions. Delving deeper into the ablation comparison, find that different convolutional layers to construct the rebiased head can achieve the best performance. This 2-1 layer setting is thereby adopted in other experiments.
B.4
Comparison with the Self- Gelertene Reliability-Base Domain Scheduler and Easy-to-Hard Domain Scheduler
We deliver the comparison among the predefined domain scheduler [54], the self-generated reliability- based domain scheduler (denoted as SDGS), the easier domain scheduler (denoted as EDS), and our domain scheduler Table 11. The self-generated reliability- domain scheduler indicates that do not rely on the follower network to achieve the reliability assessment while the confidence
16

---

Table 10: Ablation of the rebiased layer number Digits-DG with open-set ratio 6:4 using Con- vNet [66] (Best in Bold).
MNIST H-score
MNIST-M H-score
SVHN H-acome
SYN H-soore
Avg H-score
Method
Apc
OSCR
OSCR
Ace
OSCC
OSSC
Aoc
OSSR
MEDIC (cla)
97.89
62.57
96.17
71.14
48.44
55.37
76.00
51.20
55.58
58.11
04.90
73.02
83.22
57.95
70.15
MEDIC (bells)
97.89
83.20
95.81
71.14
60.98
58.28
76.00
58.77
$7.60
58.11
62.24
72.91
83.25
66.30
71.15
1-1 layer (cla)
99.11
87.30
94.12
72.86
56.50
59.29
78.64
61.09
61.45
92.58
74.30
78.12
85.80
09.97
73.25
1-1 layer (hels)
99.11
91.35
96.92
72.86
60.20
61.10
78.64
62.16
60.67
92.55
75.05
79.78
85.80
72.19
74.62
2.1 layer (cls)
99.50
87.40
97.49
74.25
56.55
60.86
88.33
61.27
62.84
93.97
75.82
78.14
87,02
70.27
75.83
2-1 layer
99.50
74.28
60.72
59.39
98.33
75.77
87,02
72.59
75.87
2.2 bayers (cla) layers (bels)
56.57
59:33
74.54
79.33
86.39
99.08
90.12
96.79
73.32
60.11
59.59
79.14
59.58
60,04
72.74
77.48
36.39
70.64
score from the main network is utilized for the domain scheduling. This comparison is designed to showcase the importance of the follower network used for the domain reliability assessment. The easier domain scheduler indicates that use the domain with the highest reliability accomplish the data partition during the meta-learning.
Through using the follower network, we observe that our method outperforms SDGS by 2.19%, 2.81%, 81, and 3.29% of Acc, H-score, and OSCR for binary classification head and 2. 2.1%, 3.42%, and 4.05% of Acc, H-score, and OSCR for conventional classification head. Consistent performance benefits of our method can be observed when we compare the results of the EDS with ours, indicating the importance of using the hardest domain scheduler in the OSDG task when meta-leaming is involved. Compared with the predefined domain scheduler from MEDIC, SDGS and EDS outperform it obviously, indicating the importance of using an adaptive domain scheduler during the meta-learning procedure for the OSDG task.
Table 11: Comparison of self-generated reliability- based domain scheduler, the Easy2Hard scheduler, and ours on Digits-DG with open-set ratio 6:4 using ConvNet [66] (Best in Bold).
MNIST sosco
MNIST-M [[-sore
SVHN 11-score
SYN
Avg. [[-score
Method
Acc
OSCR
OSCR
Acc
11sccue
OSCR
MEDIC (sls)
97.89
67.37
96:17
71.14
55.37
51.20
73.62
83.28
70.19
MEDIC (bels)
97.89
83:20
95.81
71.14
60.98
58.28
76.00
58.77
57.00
$8.11
62:24
72.91
83.28
66.30
71.15
SDGS (cls)
98.97
$5.99
95.67
71.17
50.86
$6.05
76.72
57.55
57.78
92.47
72.95
77.61
81.83
66.85
71.78
SDGS (bels)
98.97
58.79
96.62
71.17
57.01
54,18
76.72
57.89
59.05
92.47
75.44
80.46
84.83
09.78
72,58
EDS (clo)
98.89
76.21
95:10
72.17
53.81
57,45
76.86
58.07
58.95
90.72
69.40
76.45
84.66
61.37
72.07
EDS (bels)
98.89
72.17
39.15
76.36
90.72
84.66
09:23
21.78
Ours (cls)
56.55
16
93.97
70:27
75.83
Durs (bel)
99.50
91.63
97.58
74.38
60:72
59.39
62.23
63.88
93.97
75.77
87.02
72.59
75.87
B.5
Comparison with Other Curriculum Learning Approaches
We implement two recent curriculum learning approaches, ie., the training paradigms proposed by Wang et al. [56] and Abbe al. [1] into the OSDG task, where the comparison of the experimental results are delivered in Table 12. We first observe that using different training paradigms can benefit OSDG. Compared with the MEDIC baseline, the approach proposed by Abbe el al. [1] achieves performance improvements of 2.44%, 3.49%, and 0.92% and 2.44%, 2.04%, and 1.09% of Acc, H-score, and OSCR for the conventional elassification head and the binary classification head. The approach proposed by Wang et al. [56] also delivers promising comparable results with the MEDIC baseline. Furthermore, since our approach specifically designed for the OSDG task, our approach achieves performance improvements of 1.30%, 8. 80%, and 4.72% and 1.30% 4.25%, and 3.63% in Ace, H-score, and OSCR for the conventional classification head and the binary classification head compared with the approach proposed by Abbe et al. [1], illustrating the superior performance of our domain scheduler based data partition in the OSDG task.
C More Details of the Open-Set Ratios and Splits Ablations
We visualize the impact of different strategies for splitting known and unknown classes on the DigitsDG dataset, specifically examining the effects on close-set accuracy and OSCR of both conventional and binary classification heads. The analysis covers various dataset partitions, including Mnist, Mnist-m, SYN, and SVHN, depicted in Figure 4 and Figure 5. Figure 4 presents an
17

---

Table 12: Comparison with other curriculum learning methods on Digits-DG with open-set ratio 6:4 using ConvNet [66] (Best in Bold).
Method
Ase
OSCR
OSCR
OSCR
MEDIC (cls)
50.
67:57
96.
48.45
15.37
76.00
51.20
64.90
13.62
K3.28
57.08
70.19
MEDIC (bxaa)
97.59
$3.20
95.81
71.14
60.99
58.28
76.00
58.77
57.60
98.11
62.24
72.91
63.25
66.30
71.15
Wing at 150) (cls)
96.92
$7.80
94.34
72.22
$8.27
57.99
74.14
52.87
59.92
90.03
02.48
09.35
54.85
62.96
70:48
Wang oll [56] Ondaa
98.02
$9.42
94.05
72.22
50.13
58.22
78.4
50.20
59.46
90.03
06.43
70.67
54.83
66.78
70.83
Aboe at ids)
99:22
78.06
94.79
71.92
50.16
55.55
79.81
$2.80
19.55
91.92
64.87
74.33
55.72
61,47
71.11
Afbic ol. (bcle)
38:37
71.92
60.06
$7.96
73.34
68.34
Ous
61.27
78.16
70.27
Ous (bcls)
91.63
97.55
74.38
00.72
59.39
62.33
75.77
79.28
73.59
ablation study focusing on the selection of unknown classes with a 6:4 ratio. Despite the suboptimal dataset partitioning leading declines in both close-set accuracy and OSCR, the superiority of our method remains largely unaffected by the choice of unknown classes. It consistently achieves the highest close-set accuracy and OSCR across most domains within the DigitsDG dataset, while also maintaining competitive performance in the remaining domains. This demonstrates the model's proficiency distinguishing and recognizing known classes within the training set, as well as its capability of managing unseen classes, thereby highlighting its robustness in open-set environments. Furthermore, Figure illustrates an ablation study examining various open-set ratios (7:3 and 8:2). By analyzing the impact of various open-set ratios in this ablation, we show that our method effectively mitigates saturation in close-set accuracy, maintaining robust generalization capabilities of OSDG. However, the imbalance between known and unknown samples diminishes the model's s ability to differentiate unknown classes, resulting in a general decrease in OSCR. Despite this challenge, our method consistently achieves the highest OSCR across different open-set ratios compared with the baselines. Notably, in the complex and challenging SVHN domain, our OSCR exceeds the baseline MEDIC-bels by 5. 5.08% the 8:2 ratio. This finding underscores our model's 's exceptional performance in detecting unknown classes and accurately classifying known ones.
D Analysis of the Performances during Training
We visualize the accuracy changes during training every 100 epochs for the meta-learning process, reporting both validation accuracy and test accuracy, as shown in Figure 6. Implementing our domain scheduler introduces significant improvements the model's training dynamics. Notably, the valida- tion accuracy curve appears smoother compared to the MEDIC baseline, as illustrated in Figure 6a and Figure 6c. At the early stages of meta-learning for OSDG, our domain scheduler demonstrates superior initial training performance, with test set accuracy surpassing that of MEDIC [54]. These findings suggest that customizing the training schedule match distinct domain partitions within the data can substantially enhance both training efficiency and overall performance. Our approach capitalizes on specialized domain knowledge, allowing the training algorithm to better adapt to varying data characteristics, ultimately optimizing the model's 's performance outcomes.
18

---

100
97.8997.6497.66 77.6697.6488 08.3699.88
ODGNet
MEDiCel
MEDK-bdd
Ours-o's
Ours-bels
91.1991.19
77.3377.8377.83
79.4479.4
75
632260..22..8800 07.
63.36
60
(a) Close-set accuracy DigitsDG when choose 0, 1, unknown classes.
100
97.89 25 25.8.7588.77
OOGNet
MEDIC-4l
MEDIC
Ours-ds
Ours-bol
63.2503.25
90
83.53 83.53
80
27.3977.1477.34
68.4768477
65
miist
meist_m
whh
(b) Close-set set accuracy on DigitsDG when we choose 2, 3, 4, as unknown classes.
100
94.8694.84
ODGNet
MEDICell
MEDIC-bds
Ours-o's
Ours-bels
92.74
86.93
78.9
80.93
90
60 62.13 63 59 64.85
60
$2.6952.07 53.8354.17
60
(c) OSCR on DigitsDG when we choose 0, 1, 2, as unknown classes.
100
ODGNet
MEDIC-cl
MEDIC
88.76
30
81.52 81.5282.51 82.51
78.9
69 94
58.8 60.79 62.99
58.12 57.14
80
46.46 48.52 53.24 54.35
50
mnist
moist.m m
2/n
2yhn
(d) OSCR on DigitsDG when we choose 2, 3, 4, 5 as unknown classes.
Figure 4: Experimental details for the ablation of different splits on 6:4 ratio on DigitsDG dataset. (Supplementary figure)
19

---

100
97.8197.71991
000Net
MEDIC-S
MEDIC-bch
Ours-ds
Qurs-bds
91,939193
90
86.02
76.5
76.3 76.3
71.247.2477.3377335
73,45
65.96
50
40
miist.m
(a) Close-set accuracy DigitsDG when choose 7, 8, 9 as unknown classes.
100
97.9999999999999. 97.949
ODGNet
MEDIC-ds
MEDIC
Oursic's
Ours
85.9
83.56 83.56
75.71
70
68.33 68.376877668888 68.25 68.25
50
50
mnist
moist.m m
2/n
20hn
(b) Close-set accuracy on DigitsDG when we choose 8, 9 unknown classes.
CDGNet
MEDIC
MEDiC.ock
Ours
Oursibds
92.33
92.3
90
78.52
74.73 76.34
70
58.9
39.71
58.01 90.04
56.74
55.66
54.61
$1.77
mnist
mnist.
svhn
OSCR DigitsDG when choose
9 unknown classes.
100
OOGNet
MEDIC-ds
MEDiC-bols
Ours-dss
Ours bdl
91.54 90,90.000 $2.03 92.22 27
90
76.09
74.15
70
70.49
68.91
60.67 61.07
53.87 54.9355.585..25
46.07
meist
meistmm
svhn
(d) OSCR DigitsDG when we choose 8. unknown classes.
Figure 5: Experimental details for the ablation of different open-set ratios DigitsDG dataset. (Supplementary figure)
20

---

- GUAA
- -MICCNE
100
40
10
(a) Val and test accuracy MnistDG, where Mnist is (b) Val accuracy MnistDG, where Mnist-m chosen as the unseen domain.
chosen as the unseen domain.
10
10
(c) accuracy on MnistDG, where SYN is (d) test accuracy on MnistDG, where SVHN is chosen as the unseen domain.
chosen as the unseen domain.
Figure 6: Val and aceuracy on MnistDG, where the validation accuracy of MEDIC and our approach are indicated by lines in blue and orange colors, and the test accuracy of MEDIC and our approach are indicated by lines in gray and yellow colors. The horizontal axis indicates the evaluation step with stepsize 100 during the meta-learning procedure.
(a) Accuracy
(b) OSCR (cls)
(c) OSCR (bcls)
Figure 7: Ablation for the a, where the horizontal axis indicates the ablation cases. Case 1, 2, 3, 4, 5,
and indicate 2e-1 2e -3-3, 2e -4 2e 2e-5 and -6 The experiments are conducted DigitsDG using a ConvNet architecture.
21

---

Minche SPN -H-
Accuracy
(b) OSCR (cls)
(c) OSCR (bels)
Figure 8: Ablation for the loss weight of the LREG, where the horizontal axis indicates the ablation cases. Case 1, 2, 3, and indicate le -3 le- le-5, and le-6 The experiments are conducted on DigitsDG using ConvNet architecture.
- So. ---0
Accuracy
(b) OSCR (cls)
110
(c) OSCR (bcls)
Figure 9: Ablation for the loss weight of the LNBE, where the horizontal axis indicates the ablation cases. Case 1, 2, 3, and 4 indicate 5e-1, 52- 5 and 54-4 The experiments are conducted on DigitsDG using ConvNet architecture.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The authors confirm that the main claim provided in the abstract and the introduction is verified through experiments and can demonstrate the contribution and the scope this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: The authors confirm that the limitation of the proposed approach is discussed in the conclusion section alongside the negative and positive society impacts.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
The authors confirm that this submission does not contain theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The authors confirm that detailed algorithm description is provided in our main paper alongside with the implementation details. The code will be made publicly available to the community upon decision.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code link is provided in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines pottps://nisee cc/
pub 1 / gu des CCodeSuomm ss onPol cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips.... cc/
cccppcllpccccccccccccccc.ecccccl public quiiees! CodeS ubmi i ceiiinooliiee oollylic) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The authors confirm that the detailed training scripts are illustrated in the appendix. The training details, e.g., optimizer, scheduler, and hyperparameters, are delivered in the implementation details section in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The authors confirm that this submission does not provide error bars in our main experiments. However, ablation experiments include cross backbone generalizability. cross split tests and cross open-set ratio tests, which can demonstrate the statistic significance of the proposed approach shown in Figure in main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The authors confirm that the details of the computational resources provided in the implement details section in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neeur/e http:/pp...c.p..cectttctttaaeaae
Answer: [Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics and confirmed for all the content.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors confirm that the potential positive and negative society impacts are discussed in the conclusion section.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The authors confirm that this submission poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The authors confirm that the original papers that produced the code package dataset are correctly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, paperswa thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
The authors confirm that this submission does not contain new dataset. Pro- tocols are demonstrated in the supplementary materials and code will be made publicly available upon decision.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The authors confirm that this submission does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The authors confirm that this submission does not involve crowdsourcing nor research with human subjects.
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Jq8s7V1PbW", "reviewer": "Reviewer_DFZJ", "summary": "The paper presents an adaptive domain scheduler with ability to adjust the training order dynamically according to model’s current performance and domain difficulty to address OSDG, short for Open-Set Domain Generalization (OSDG), task where the model is exposed to domain shift and category shift.", "questions": "1.\tCitation in line 91、93 and so on, citation order is not standard. \n2.\tLine 331~474, The reference format is incorrect, citation of journals needs to be marked with issues, pages and years", "limitations": "Experiments show that the method proposed in this paper may still lead to misclassification and biased content. This may cause useless even false predication , causing critical negative influence in popularizing. Additionally, EBiL-HaDS is significantly relied on source domains with unified categories and its using scene is strictly limited, it has only been tested on image classification tasks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tEffectiveness of addressing limitation of current study. The paper focus on OSDG task and pointed out the problem of existing meta-learning-based OSDG approaches that they didn’t consider influence that the order where domains are presented brings to the process of model generalization. To address this major shortness, the author proposed a novel training strategy named EBiL-HaDS which emphasis the influence of training order and can adjust the order of domain presentation dynamically and automatically.\n2.\tAbundance of references. The research and experiment is based on abundant and various related work. Summary of existing work especially about domain generalization and open-set recognition is abundant and enough to support his research. With accurate overview and clear awareness of critical shortness of existing study, the paper is revolutionary and meaningful in addressing major problem.  \n3.\tSufficient and reliable Experiment. The paper conducted several experiments on extensive datasets ,all of which showed dramatically improvement. This showed effectiveness of the proposed method.", "weaknesses": "1.\tSeveral mistakes in format such as missing indent before paragraphs and size of pictures, and the formula in between line 161 and line 162. \n2.\tNo open resource code and data. Inconvenience to reproduce and follow the experiment. Also unable to exam the effectiveness of the result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the motivation behind our work and the effort put into the review. We will address the mentioned formatting and citation issues in our final version submission. The source code has been shared in the official comment to the AC, including a sample for result reproduction via an anonymous GitHub link. We hope the provided source code will enhance the contribution of our paper. The included model weight is for the PACS dataset, with the cartoon domain as the target and ResNet18 as the feature extraction backbone. The code will be made publicly available in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJLklDU04c", "reviewer": "Reviewer_QWTs", "summary": "In this paper, an observation is proposed, that an adaptive domain scheduler benefits more in OSDG compared with prefixed sequential and random domain schedulers. A follower network is proposed to strategically sequences domains by assessing their reliabilities, which is trained with confidence scores learned in an evidential manner and optimized in a bi-level manner. Experiments show the superior OSDG performance and ability to get more discriminative embeddings for both the seen and unseen categories.", "questions": "If the comparison is conducted on models with a large number of parameters, like ResNet152 or ViT-based models, would the advantage in classification performance be leveled off?", "limitations": "The experiments only conducted on classification tasks.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The influence of domain scheduling in the OSDG task remains unexplored. This paper examines the effects of guiding the meta-learning process with an adaptive domain scheduler. A domain reliability measure method is proposed by an follower network.\n3. The proposed method conducts comprehensive and discriminative data partitions during training, enhancing generalization to unseen domains.\n4. Experiments on OSDG tasks show the superior performance.", "weaknesses": "The experiments conducted only on ResNet50 ResNet18 and ConvNet. The number of parameters in those backbone are limited. If the comparison is conducted on models with a large number of parameters, like ResNet152 or ViT-based models, would the advantage in classification performance be leveled off?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the motivation and contribution of our proposed method. We have conducted an additional 48 experiments using the ResNet152 and ViT base models. The results, detailed below, compare our proposed method with MEDIC and other challenging baselines such as ARPL, MLDG, SWAD, and ODG-Net. From Tables 1 and 2 in our main paper, we observe a significant OSDG performance improvement when increasing the complexity of the feature extraction backbone. However, when comparing Table A (rebuttal) with Table 2 (main paper), some baseline approaches trained with the ResNet152 backbone show performance decay on the major evaluation metric, OSCR.\n\nThis observation indicates that most OSDG methods face overfitting issues with larger backbones, which is particularly problematic for recognizing samples from unseen categories in unseen domains. Specifically, the MEDIC-bcls approach shows a 6.48% performance degradation on OSCR when switching from ResNet50 to ResNet152. In contrast, using the proposed BHiL-HaDS for more reasonable task reservation in the meta-learning procedure, BHiL-HaDS-bcls achieves 86.25% OSCR with the ResNet152 backbone, compared to 86.12% OSCR with the ResNet50 backbone. This demonstrates that our adaptive domain scheduler can effectively optimize meta-learning for large complex models by reserving appropriate tasks for model optimization.\n\n**Table A: Ablation for ResNet152 backbone**\n| Method            | Photo (P)  Acc | Photo (P) H-score | Photo (P) OSCR | Art (A) Acc | Art (A) H-score | Art (A) OSCR | Cartoon (C) Acc | Cartoon (C) H-score | Cartoon (C) OSCR | Sketch (S) Acc | Sketch (S) H-score | Sketch (S) OSCR | Avg Acc | Avg H-score | Avg OSCR |\n|-----------------------|---------------|-------------------|---------------|-------------|-----------------|--------------|-----------------|---------------------|-----------------|----------------|---------------------|-----------------|----------|--------------|-----------|\n|ARPL\t|94.35|85.45|86.74|89.81|71.27|78.53|83.91\t|69.75|72.08|77.53\t|52.70|66.68|77.53\t|69.81\t|76.01\t|\n|MLDG\t|96.20|91.07|94.64|89.81|77.65|82.19|83.86\t|73.66|74.03|82.89\t|64.30|72.98|88.19\t|76.67\t|80.96|\n|SWAD\t|95.64|84.82|89.74\t|86.30|73.86|75.91|78.49|70.18|68.41|76.92|75.33|63.35|84.34\t|76.05\t|74.35\t|\n|ODG-Net|95.88|89.11|91.85|89.62|80.65|82.48|85.15|70.37|73.66|79.30|77.00|72.22|87.49\t|79.28\t|80.05\t|\n|MEDIC-cls|94.67\t|49.54|76.98|89.37|73.26|77.79|86.59|68.49|74.82|85.81|56.14|78.83\t|89.11\t|61.86|77.11|\n|MEDIC-bcls|94.67|72.88|81.30|89.37|74.92|78.70|86.59|71.46|75.17|85.81|58.80|78.32\t|89.11\t|69.52\t|78.37|\n|ours-cls|**97.90**\t|91.66|96.62|**92.06**|81.52|85.43|**87.21**|76.61|78.19|**87.08**|81.13\t|80.21|**91.06**|82.73\t|85.11|\n|ours-bcls|**97.90**|**94.34**\t|**97.39**|**92.06**|**82.00**\t|**85.94**\t|**87.21**\t|**76.62**\t|**80.15**\t|**87.08**\t|**88.57**\t|**81.52**\t|**91.06**|**85.38**|**86.25**|\n\n\n\nAdditional experiments on the ViT-base model (patch size 16 and window size 224) are presented in Table B of this response. We observe that our proposed method consistently delivers performance gains on the transformer architecture. These ablation experiments and discussions will be included in our final paper.\n\n\n**Table B: Ablation for ViT base backbone**\n| Method          | Photo (P)  Acc | Photo (P) H-score | Photo (P) OSCR | Art (A) Acc | Art (A) H-score | Art (A) OSCR | Cartoon (C) Acc | Cartoon (C) H-score | Cartoon (C) OSCR | Sketch (S) Acc | Sketch (S) H-score | Sketch (S) OSCR | Avg Acc | Avg H-score | Avg OSCR |\n|-----------------------|---------------|-------------------|---------------|-------------|-----------------|--------------|-----------------|---------------------|-----------------|----------------|---------------------|-----------------|----------|--------------|-----------|\n| ARPL | 99.19| 95.31| 98.61| 90.49 | 85.46 | 88.59| 81.88| 72.17| 73.34| 63.01| 29.33| 50.59| 83.64| 70.57| 77.78|\n| MLDG| 99.19| 95.40| 98.88| 91.87| 82.46| 89.47| 80.56| 69.62| 74.19| 61.66| 40.79| 43.88| 83.32 | 72.07 | 76.61|\n| SWAD| 98.55| 93.19| 97.62| 90.81| 81.34| 88.52| 83.24 | 73.03| 76.59| 57.89| 35.83| 41.68 | 82.62 | 70.85       | 76.10     |\n|ODG-Net|97.58\t|96.24\t|95.23\t|90.49\t|83.32\t|87.90\t|82.36\t|68.66\t|75.80\t|62.59\t|43.59\t|50.22\t|83.26\t|72.95\t|77.29\t|\n|MEDIC-cls\t|99.03\t|95.33\t|98.22\t|92.06\t|83.27\t|87.46\t|85.62\t|69.79\t|75.37\t|68.40\t|41.95\t|56.56\t|86.28\t|72.59\t|79.40\t|\n|MEDIC-bcls|99.03\t|96.04\t|97.55\t|92.06\t|82.68\t|87.73\t|85.62\t|69.15\t|76.80\t|68.40\t|39.60\t|55.92\t|86.28\t|71.87\t|79.50\t|\n|ours-cls|**99.52**\t|**97.30**|99.11|**94.68**|86.10\t|92.10\t|**89.22**\t|**74.31**\t|77.76\t|**69.49**\t|44.34\t|55.37\t|**88.23**|75.53\t|81.09\t|\n|ours-bcls|**99.52**\t|96.91\t|**99.18**\t|**94.68**|**88.31**\t|**92.28**\t|**89.22**\t|73.91\t|**77.95**\t|**69.49**\t|**48.09**\t|**56.78**\t|**88.23**|**76.81**|**81.55**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ntjKua6JiU", "reviewer": "Reviewer_euSj", "summary": "The paper addresses the challenges of Open-Set Domain Generalization (OSDG) and introduces the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which adaptively sequences domains based on their reliability, assessed through a follower network. The authors verify EBiL-HaDS on three benchmarks: PACS, DigitsDG, and OfficeHome, demonstrating substantial performance improvements in OSDG.", "questions": "What is the performance under ViT architecture?", "limitations": "The authors discuss the limitation of potential leading to misclassification and biased content, potentially causing false predictions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. EBiL-HaDS dynamically adjusts the domain training sequence, which is a novel idea compared to the fixed schedulers used in previous works.\n2. Comprehensive experiments and ablations on PACS, DigitsDG, and OfficeHome demonstrate the effectiveness of the proposed approach\n3. The paper is very well written and easy to follow", "weaknesses": "1. The proposed framework is too complex and hard to reproduce. The source code is also not provided.\n2. The authors only use CNN backbones. More ablations on ViT backbone should be added, as it demonstrates strong domain generalization performances compared with CNN [A, B]\n\n[A] Li et al., Sparse Mixture-of-Experts are Domain Generalizable Learners, ICLR 2023\n\n[B] Zheng et al., Prompt Vision Transformer for Domain Generalization, arxiv 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the motivation and contribution of the proposed method. \n\n**W1:** The anonymous link to the source code has been submitted in the official comment to AC which can be helpful for the results reproduction. The source code will be released publicly in the final version.\n\n\n**W2:** We have conducted additional 24 experiments on the ViT-base model according to your suggestion in Table A of this response. The results can be found below, where our proposed method is compared with the MEDIC and other challenging baselines. We can observe that our proposed method achieves state-of-the-art performances when we use ViT-base model as the feature extraction backbone. \n\nCompared with the MEDIC method, our EBil-HaDS achieves 1.69% and 2.05% performance improvements in terms of OSCR for standard classification head and binary classification head, demonstrating the capability of the proposed adaptive domain scheduler on the generalization to different feature extraction backbones. The corresponding experiments and discussion will be added to the final version of our paper.\n\n\n**Table A: Ablation for ViT base backbone**\n| Method          | Photo (P)  Acc | Photo (P) H-score | Photo (P) OSCR | Art (A) Acc | Art (A) H-score | Art (A) OSCR | Cartoon (C) Acc | Cartoon (C) H-score | Cartoon (C) OSCR | Sketch (S) Acc | Sketch (S) H-score | Sketch (S) OSCR | Avg Acc | Avg H-score | Avg OSCR |\n|-----------------------|---------------|-------------------|---------------|-------------|-----------------|--------------|-----------------|---------------------|-----------------|----------------|---------------------|-----------------|----------|--------------|-----------|\n| ARPL | 99.19| 95.31| 98.61| 90.49 | 85.46 | 88.59| 81.88| 72.17| 73.34| 63.01| 29.33| 50.59| 83.64| 70.57| 77.78|\n| MLDG| 99.19| 95.40| 98.88| 91.87| 82.46| 89.47| 80.56| 69.62| 74.19| 61.66| 40.79| 43.88| 83.32 | 72.07 | 76.61|\n| SWAD| 98.55| 93.19| 97.62| 90.81| 81.34| 88.52| 83.24 | 73.03| 76.59| 57.89| 35.83| 41.68 | 82.62 | 70.85       | 76.10     |\n|ODG-Net|97.58\t|96.24\t|95.23\t|90.49\t|83.32\t|87.90\t|82.36\t|68.66\t|75.80\t|62.59\t|43.59\t|50.22\t|83.26\t|72.95\t|77.29\t|\n|MEDIC-cls\t|99.03\t|95.33\t|98.22\t|92.06\t|83.27\t|87.46\t|85.62\t|69.79\t|75.37\t|68.40\t|41.95\t|56.56\t|86.28\t|72.59\t|79.40\t|\n|MEDIC-bcls|99.03\t|96.04\t|97.55\t|92.06\t|82.68\t|87.73\t|85.62\t|69.15\t|76.80\t|68.40\t|39.60\t|55.92\t|86.28\t|71.87\t|79.50\t|\n|ours-cls|**99.52**\t|**97.30**|99.11|**94.68**|86.10\t|92.10\t|**89.22**\t|**74.31**\t|77.76\t|**69.49**\t|44.34\t|55.37\t|**88.23**|75.53\t|81.09\t|\n|ours-bcls|**99.52**\t|96.91\t|**99.18**\t|**94.68**|**88.31**\t|**92.28**\t|**89.22**\t|73.91\t|**77.95**\t|**69.49**\t|**48.09**\t|**56.78**\t|**88.23**|**76.81**|**81.55**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Elk0JIS9ds", "reviewer": "Reviewer_tsHR", "summary": "In this paper, the authors identify a new problem in open-set domain generalization, proposing that dynamically adapting the domain scheduler used for data partitioning based on specific criteria could lead to a more targeted training strategy and improved outcomes. They introduce a novel training strategy named the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which combines max rebiased discrepancy evidential learning with meta-learning to create an adaptive domain scheduler.", "questions": "Discussed in the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors focus on a new and interesting problem that dynamically adapt the domain scheduler used for data partition may improve open-set domain generalization performance.\n2. Extensive experiments verify the efficiency of the proposed adaptive domain scheduler.", "weaknesses": "1. The motivation for considering the training order of domains is unclear. I believe that if there were temporal correlation between domains, considering the training order would be reasonable. However, in this research setting and the datasets used in the experiments, there is no temporal correlation. Therefore, why should the training order be considered?\n\n2. In my opinion, this paper primarily makes some improvements to the MEDIC model. However, compared to MEDIC, the proposed method does not show significant performance improvement on some tasks.\n\n3. Dynamically adapting the domain scheduler used for data partitioning based on certain criteria can be seen as a trick rather than a major contribution.\n\n4. The key idea of EBiL-HaDS is to quantify domain reliability. I suggest that the authors provide a more detailed definition of domain reliability and demonstrate that domain reliability is existing in real-world scenarios with a significant impact on generalization.\n\n5. What is the motivation for applying max rebiased discrepancy evidential learning to achieve more reliable confidence acquisition? Are there any particular advantages to this approach? The authors should compare this approach with other similar techniques.\n\n6. There are still some grammatical mistakes and typos in this paper. Please proofread the paper carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the review effort from Reviewer tsHR and provide the point-to-point responses as follows. All the explanations will be included into our final paper.\n\n\n**W1**: Training order scheduler is very important in deep learning for some specific techniques, e.g., meta learning and curriculum learning. In curriculum learning, researchers propose the instance scheduler to decide the next training samples to pursue a better optimization of the performance of the deep learning model, where the samples do not contain temporal correlation, as demonstrated in works, [a] and [b]. \nMeta-learning aims at reserving different training tasks for model optimization, where the training order can guide the model to different optimization situations. Choosing different tasks/samples for training at different stages can result in obvious performance differences in the models. \nMEDIC and MLDG propose to use meta learning techniques for OSDG, where meta learning shows strong OSDG capability when the meta training and meta testing are conducted on different source domains. However, both of them rely on sequential source domain order when setting up the meta tasks. Our experiments turn out that sequential domain scheduler is not the optimal way for domain scheduling in the meta learning of the OSDG task. In this work, we for the first time propose an adaptive domain scheduler in a bi-level optimization manner, which shows strong OSDG capability across different backbones and datasets. \n\nWe conducted the ablation in Table 6 of our main paper to demonstrate the performance gains brought by our proposed adaptive domain scheduler, when we compare our proposed domain scheduler method with random domain scheduler and sequential domain scheduler.\n\n[a] Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficienttrain: Exploring generalized curriculum learning for training visual backbones. In ICCV, 2023\n\n[b] Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In ICML, 2023\n\n\n**W2**: In our paper we for the first time propose an adaptive evidential bi-level optimized domain scheduler for the meta learning used in the OSDG task, where the existing works, i.e., MLDG and MEDIC, rely on sequential domain scheduler. Our approaches achieve consistent state-of-the-art performances across 3 different DG benchmarks where 4 different domain generalization settings are explored. In the response to the Reviewer QWTs, we provide more ablations on larger backbones, i.e., ResNet152 and ViT-base, where obvious OSCR gains are observed. All of these experimental results showcase the strong capability of our proposed method towards open-set domain generalization challenge.\n\nIn Table 9 of the appendix, we conducted ablation experiments to justify the effectiveness of our model with more variants of domain scheduler for meta learning on OSDG task.\n\n\n**W3**: We clarify that the adaptive domain scheduler is our major contribution, which is proposed by us in this work to estimate the uncertainty level of different source domains during the meta-learning procedure using an evidentially bi-level optimized follower network. Unlike curriculum learning (e.g., [a] and [b]), which focuses on scheduling the training order of different samples or batches, we for the first time propose the adaptive domain scheduler to achieve adaptive reservation of different source domains for meta-learning approaches within the realm of OSDG task. We demonstrate that the proposed domain scheduler is much more powerful compared with other curriculum learning strategies as proved by the experimental results in Table 10 in the appendix. \n\n\n**W4**:  Domain reliability is based on the statistic mean of the uncertainty estimation of the samples from one specific domain, estimated by the follower network. The prediction generated by the bi-level optimized follower network is used as uncertainty quantification of the corresponding input sample, since this follower network is optimized by the evidentially learned uncertainty score from the main network.\nFrom our experiments, we observe that the  data partition based on the most unreliable domain, where the uncertainty estimation of this source domain from follower network is the highest, benefit the OSDG performances. In Table 9 of the appendix, we provide further ablation between the domain scheduler achieved by bi-level optimized follower and the uncertainty estimation of the main network. Our method shows better performances.\n\n\n**W5**: Evidential learning is a well-established technique to pursue better uncertainty estimation of the deep learning methods. However, as mentioned in related work [c], evidential learning faces with challenge of overfitting. In this work, we propose to use max rebiased discrepancy to force the two rebiased layers to learn different cues to overcome this challenge, where the max rebiased discrepancy serves as additional regularization for the evidential learning of the main network. After the supervision of the max rebiased discrepancy evidential loss, the main network can supervise the follower network well by providing more reliable estimated uncertainty.\n\nIn Table 7 in the appendix, we conducted experiments to illustrate the benefits brought by the max rebiased discrepancy regularization term, where the discussion of this ablation is demonstrated in Section B.1 in our appendix. Our observation turns out that by using the proposed max rebiased discrepancy regularization, the follower network can be optimized better and more reliable in the domain reliability quantification, thereby the proposed method can achieve higher OSDG performances.\n\n[c] Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. Uncertainty estimation by fisher information-based evidential deep learning. In ICML, 2023.\n\n**W6**: The corresponding typos will be changed in our final version, thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: While the initial recommendations were divergent, all four reviewers reached a positive consensus (three borderline accept and an accept) after the post-rebuttal discussions. The AC therefore recommends accepting the paper and asks the authors to include their discussions with the reviewers in the final manuscript.
Decision: Accept (poster)