Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler

Authors: kunyu  peng
di  wen
kailun  yang
ao  luo
yufan  chen
jia  fu
m  saquib sarfraz
alina  roitberg
rainer  stiefelhagen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:53:19

Abstract: In Open-Set Domain Generalization (OSDG), the model is exposed to both new variations of data appearance (domains) and open-set conditions, where both known and novel categories are present at test time. The challenges of this task arise from the dual need to generalize across diverse domains and accurately quantify category novelty, which is critical for applications in dynamic environments. Recently, meta-learning techniques have demonstrated superior results in OSDG, effectively orchestrating the meta-train and -test tasks by employing varied random categories and predefined domain partition strategies. These approaches prioritize a well-designed training schedule over traditional methods that focus primarily on data augmentation and the enhancement of discriminative feature learning. 
The prevailing meta-learning models in OSDG typically utilize a predefined sequential domain scheduler to structure data partitions. However, a crucial aspect that remains inadequately explored is the influence brought by strategies of domain schedulers during training. 
In this paper, we observe that an adaptive domain scheduler benefits more in OSDG compared with prefixed sequential and random domain schedulers. We propose the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS) to achieve an adaptive domain scheduler. This method strategically sequences domains by assessing their reliabilities in utilizing a follower network, trained with confidence scores learned in an evidential manner, regularized by max rebiasing discrepancy, and optimized in a bilevel manner. We verify our approach on three OSDG benchmarks, i.e., PACS, DigitsDG, and OfficeHome. The results show that our method substantially improves OSDG performance and achieves more discriminative embeddings for both the seen and unseen categories, underscoring the advantage of a judicious domain scheduler for the generalizability to unseen domains and unseen categories. The source code is publicly available at https://github.com/KPeng9510/EBiL-HaDS.

Main paper: 
 introduction: 
 Open-Set Domain Generalization (OSDG) is a challenging task where the model is exposed to both: domain shift and category shift. Recent OSDG works often take a meta-learning approach [54, 46] which simulates different cross-domain learning tasks during training. These methods conventionally use a predefined sequential domain scheduler to create meta-train and meta-test domains within each minibatch. But is fixing the meta-learning domain schedule a priori the best way to go? As a step to explore this, our work investigates the new idea of adaptive domain scheduler, which dynamically adjusts the training order based on ongoing model performance and domain difficulty.OSDG is critical for many real-world applications with changing conditions, ranging from healthcare [33] and security [5] to autonomous driving [19] . Despite the remarkable success of deep learning, the recognition quality often deteriorates when facing out-of-distribution samples. This problem is amplified in OSDG settings, where the model faces a dual challenge of identifying and rejecting unseen categories, e.g., by delivering low confidence score in such cases [27, 46] while simultaneously generalizing well to unseen data appearances (domain shift). Historically, research efforts in OSDG have predominantly focused on the latter, developing methods to adapt models to varying domain conditions. Strategies to improve domain generalization include the use of Generative Adversarial Networks (GANs) [4] , contrastive learning [65] , and metric learning [27] . MLDG [46] for the first time proposed to use meta-learning to handle OSDG tasks. However, all these works follow the OSDG protocols where different source domains preserve different distributions of known categories, which diverges the domain gaps for different categories. This divergence makes the challenge more inclined towards the domain generalization aspect. Wang et al. [54] revised these benchmarks for OSDG, standardizing the category distribution across source domains to achieve a more balanced evaluation of both domain generalization and open set recognition challenges. This revision includes established open-set recognition methods such as Adversarial Reciprocal Points Learning (ARPL) [8] . Recently, Wang et al. [54] introduced an effective meta-learning approach named MEDIC, featuring a binary classification and a predefined sequential domain scheduler for the data partition during meta-train and -test stages.However, these existing meta-learning-based OSDG approaches, i.e., MEIDC [54] and MLDG [46] , do not consider how the order in which domains are presented during training affects model generalization. We believe this overlooks the potential to dynamically adapt the domain scheduler used for data partition based on certain criteria, such as domain difficulty, which could result in a more targeted training strategy and, therefore, better outcomes. In this paper, we observe that different ordering strategies for domain presentation used for data partition during the meta-training and testing phases lead to significant variations in OSDG performance, emphasizing the critical role of domain scheduling in optimizing model generalization.To bridge this gap, we introduce a new training strategy named the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which allows dynamically adjusting the order of domain presentation during data partitioning in the meta-training and -testing phases. The key idea of our method is to quantify domain reliability, defined as the aggregated confidence of the model on the samples across unseen domains, which will then be used as the main criterion for the data partition. To assess the domain reliability, we incorporate a secondary follower network to assess the domain reliability alongside the primary network. This allows for prioritizing the optimization of meta-learning on less reliable domains, facilitating an adaptive domain scheduler-based data partitioning. This follower network is trained using bi-level optimization, which involves a hierarchical setup where the solution to a lower-level optimization problem (evaluating domain reliability) serves as a constraint in an upper-level problem (meta-learning objective). Optimization of the follower network is guided by confidence scores generated through our proposed max rebiased evidential learning method, which adjusts the confidence by amplifying the differences between the decision boundaries of different classes. As a result, the follower network can better quantify the reliability of each domain based on how distinct and consistent the classification boundaries are, improving the ability to generalize to unseen domains. EBiL-HaDS enhances cross-domain generalizability and differentiation of seen and unseen classes by prioritizing training on less reliable domains through adaptive domain scheduling.Our experiments demonstrate the effectiveness of domain scheduling via EBiL-HaDS on three established datasets: PACS [31] , DigitsDG [63] , and OfficeHome [51] , which span a variety of image classification tasks. Results demonstrate that EBiL-HaDS significantly improves model generalizability in open-set scenarios, enhancing domain generalization and the model's ability to distinguish between known and unknown categories in new domains. This performance surpasses that of both random and standard sequential domain scheduling methods during training, underscoring EBiL-HaDS's potential to advance current OSDG capabilities in deep learning. 
related work: 
 We simultaneously address two challenges: domain generalization and open-set recognition. Domain generalization is a task that expects a model to generalize well to unseen domains while leveraging multiple seen domains for training [48, 52] . Open-set recognition, on the other hand, aims to reject unseen categories at test-time, e.g., by delivering low confidence scores in such cases [16] . Domain generalization methods usually alleviate the domain gap with techniques such as data augmentation [55, 40, 65, 18, 64, 34, 35] , contrastive learning [57, 24, 28, 43] , domain adversarial learning [15] , domain-specific normalization [44] , and GANs-based methods [10, 32] . For open-set recognition, common approaches include logits calibration [3, 42] , evidential learning [59, 53, 62, 2] , reconstruction-based approaches [58, 22] , GANs-based methods [29] , and reciprocal point-based approaches [8, 9] .A considerable cluster of research utilizes source domains that encompass diverse categories for training, as highlighted in [14, 47, 4, 8, 36, 61] , where each category poses unique domain generalization challenges. The primary focus of these methodologies is to improve domain generalizability, and they tend to allocate less attention to the complexities associated with open-set scenarios. In this setting, the categories involved in each source domain may not be the same. ODG-Net proposed by Bose et al. [4] leverages GAN to synthesize data from the merged training domains to improve cross-domain generalizability. SWAD proposed by Chen et al. [8] uses models averaged across various training epochs. Katsumata et al. [27] propose to use metric learning to get discriminative embedding space which benefits the open-set domain generalization. Wang et al. [54] introduce a new MEDIC model together with a new formalization of the open-set domain generalization protocols, where the source domains share the same categories defined as seen. This benchmark definition balances the impact of the model's open-set recognition and domain generalization performance in evaluation, which is adopted in our work.Newer OSDG approaches show great promise of meta-learning strategies for improving crossdomain generalization [54, 46] . Yet, these works mainly utilize a fixed, sequential scheduling of source domains during training [54, 46] . Existing works in curriculum learning indicate that using a specific training order at the instance level can benefit the model performance on various tasks [56, 26, 17, 37, 20, 41, 28, 45] . However, existing curriculum learning approaches usually operate at the instance level (scheduling individual dataset instances within standard training) and are not designed for OSDG tasks, while we focus on domain-based scheduling by quantifying the domain difficulty in meta-learning. The influence of domain scheduling in the OSDG task remains unexplored. This paper, for the first time, examines the effects of guiding the meta-learning process with an adaptive domain scheduler, named EBiL-HaDS, which achieves data partition based on a domain reliability measure estimated by a follower network, trained in a bi-level manner with the supervision from the confidence score optimized by a novel max rebiased discrepancy evidential learning.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The authors confirm that the limitation of the proposed approach is discussed in the conclusion section alongside the negative and positive society impacts. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The most challenging domain is chosen to perform data partitioning for the meta-task reservation during meta-learning. In our proposed EBiL-HaDS, we first utilize max rebiased discrepancy evidential learning (Sec. 3.1) to achieve more reliable confidence acquisition, which is subsequently used as the supervision for the reliability prediction of the follower network. During the training stage, our method makes use of two networks with identical architectures: one serves as the main feature extraction network, and the other functions as the follower network, aiming to assess the domain reliability. The follower network is optimized in a bi-level manner alongside the main network (Sec. 3.2). Hardest domain selection is accomplished by aggregating votes for samples from each domain for the randomly selected reserved classes using the follower network (Sec. 3.3).To optimize domain scheduling, we first define the term domain reliability, as the degree to which data from a domain consistently aids in improving the model's accuracy and generalizability across unseen domains. An important step is therefore to adaptively rate the domain reliability during training. To achieve this, we employ two parallel networks: the main network used for feature extraction, and the follower network, which assesses the reliability of different domains based on the refined confidence metrics. This follower network plays a central role in our adaptive domain scheduling strategy: it employs a voting process to identify and select the most challenging domains -those that exhibit the least reliability according to its assessments. After selecting the hardest domain, the data is divided into two sets. One set includes data from more challenging domains outside the reserved classes and data from more reliable domains within the reserved classes, which together form the meta-training set. The complementary partitions of the meta-training set are used as the meta-testing set. Both networks are simultaneously optimized through a bi-level training approach, meaning that the outcome of a lower-level optimization problem (evaluating domain reliability) is a constraint in an upper-level problem (meta-learning objective). The entire pipeline during training when using the proposed EBiL-HaDS is depicted in Alg. 1.The domain scheduler we propose leverages a follower network to ascertain reliability measurements, based on reliability evaluations conducted before the start of each epoch for all source domains. As such, confidence calibration is crucial for the main network's functionality. Evidential learning, which has been extensively applied across various domains such as action recognition [62] and image classification [25] , effectively calibrates these confidence predictions. However, a notable limitation of evidential learning is its propensity for overfitting, leading to suboptimal performance [11] .To address these challenges, we propose to regularize the evidential learning by novel rebiased discrepancy maximization, which is employed for the confidence calibration of the main network to encourage diverse decision boundaries. This method involves training dual decision-making heads designed to exhibit rebiased maximized discrepancies. The aim is to foster the development of both informative and dependable decision-making capabilities within the leveraged deep learning model. Let x denote a batch of data used in training, M α denote the feature extraction backbone, R θ1 and R θ2 denote the two rebiased layers, and K denote the Gaussian kernel to reproduce the Hilbert space. We first calculate the max rebiased discrepancy regularization by Eq. 1,R RB (x; Θ) = i∈{1,2} E [K(R θi (M α (x)), R θi (M α (x)))] -2 * E[K(R θ1 (M α (x)), R θ2 (M α (x)))].(1) We aim to maximize the above loss function to achieve the maximum discrepancy between the embeddings extracted from the two rebiased layers. This maximization encourages the learned evidence from the two layers to diverge from each other, thereby capturing open-set domain generalization cues from two different perspectives. Deep evidential learning is then applied to the conventional classification head, providing an additional constraint to achieve more reliable confidence calibration, as described in Eq. 2.L RBE (y, x; Θ) = i∈{1,2} C c=1 [y c (log S i -log(R θi (M α (x)) c + 1))] -R RB (x; Θ) (2)where S i = C c=1 (Dir(p|R θi (M α (x)) c + 1) denotes the strength of a Dirichlet distribution, y c is the one-hot annotation of sample x from class c, p is the predicted probability. The two rebiased layers are engineered to capture distinct evidence by employing max discrepancy regularization. By averaging the logits produced by the two prediction heads on the top of the two rebiased layers for the conventional classification on the seen categories, we can harvest the final estimated confidence score. This score is subsequently utilized to supervise the follower network, as elaborated in the following subsections.To establish an adaptive domain scheduler for the OSDG task, the most straightforward approach would involve training a network to directly predict the sequence in which domains are employed during the training phase for sample selection. However, this method does not facilitate gradient computation, thereby preventing the direct optimization of the scheduler network.In this work, we propose an alternative method where a follower network is trained to assess the reliability of each sample, utilizing predicted confidence scores derived from max discrepancy evidential learning as supervision. Throughout the training process, we employ samples from various domains to collectively assess reliability. Additionally, we utilize a follower network, denoted as M β , which mirrors the architecture of the main network, but with classification heads replaced by one regression head. Θ indicates all the parameters in the main network, including the parameters from Randomly select two known classes c i , c j ← C; Get the hardest domain d * using M β by Eq. 5; Select two domains from d i , d j ← D/{d * }; Meta-train:4: Sample data Ω a , Ω b = {x [c k ,d k ] |k ∈ [i, j]}, {x [c k ,d k ] |c k ∈ {C/{c i , c j }, d k ∈ {d * }};7:for x in Ω m-train do;8: Extract rebiased embeddings f 1 = R θ1 (M α (x)) and f 2 = R θ2 (M α (x)); 9:Obtain the max rebiased discrepancy evidential learning loss L RBE (x) using f 1 and f 2 ; 10:Follower learning L REG (x) = M SE(M β (x), 1 2 k∈{1,2} Conf (H Φ k (f k ))); 11: Obtain classification loss L CLS (x) = k∈{1,2} (W CE(H Φ k (M α (x))), y, ω)), ω ← M β (x), where y indicates the classification annotation; L m-train ← x∈Ωm-train (L CLS (x) + L REG (x) + L RBE (x)). Backpropagation and parameter update for the whole network; 14:Meta-test:15: Sample data Ω * a , Ω * b = {x [c k ,d k ] |c k ∈ {c i , c j }, d k ∈ {d * }}, {x [c k ,d k ] |c k ∈ C/{c i , c j }, d k ∈ {d i , d j }}. Construct meta-test set Ω m-test = Ω * a ∪ Ω * b ; 16:Obtain loss for meta-testL m-test ← x∈{Ωm-test} (L CLS (x) + L REG (x) + L RBE (x)); 17:Back propagation and parameter update using L all = L m-test + L m-train . 18: end while the backbone, rebiased layers, and heads. We aim to solve the optimization task, as shown in Eq. 3.EQUATIONwhere ω * indicates the instance-wise reliability which serves as the weight for each instance during the loss calculation. Substituting the best response function β * (Θ) = arg min β L f (M Θ (x), M β (x)) provides a single-level problem, as shown in Eq. 4.EQUATIONwhere L m denotes classification loss (L CLS ) and L RBE . L f denotes the regression loss (L REG ).We illustrate the details of the training procedure by the proposed domain scheduler in Alg. 1. We adopt the meta-training framework outlined by MLDG [46] , integrating our proposed domain scheduler to facilitate the data partition of meta-tasks. In this approach, optimization is achieved using both the meta-train and -test sets, characterized by distinct data distributions. For each domain present in the training dataset, we sample a batch that encompasses the reserved categories. Subsequently, we identify the most challenging domain by determining which domain exhibits the lowest reliability under the selected seen categories. This procedure is accomplished by the calculation of the expected reliability as in Eq. 5. 4 ExperimentsEQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: The authors confirm that the details of the computational resources are provided in the implement details section in the main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
implementation details: 
 All the experiments use PyToch 2.0 and one NVIDIA A100 GPU. We set the upper limit of the training step as 1e4 and use SGD optimizer, where the learning rate (lr) is set as 1e -3 and batch size is chosen as 16. The weights of L CLS , L REG , and L RBE are chosen as 1.0, 1e -4 , and 5e -4 . Lr decay is 1e -1 and conducted at 8e 3 meta-training step. The worker number is 4 and γ is 2e -5 . For ResNet18 [21] and ResNet50 [21] , each rebiased layer is constructed using a residual convolutional block. For ConvNet [66] , convolutional layers are utilized. Apart from the conventional classification head (cls), we also utilize a binary classification head (bcls) as in MEDIC [54] . The training time of our method is 1h on PACS (ResNet18 [21] ), 1.2h on PACS (ResNet50 [21] ), 20min on DigitsDG (ConvNet [66] ), 2h on OfficeHome (ResNet18 [21] ). The parameter ablation is provided in the appendix.We adopt the open-set protocols provided by MEDIC [54] We first validate the performances of our approach on the three well-established benchmarks for the OSDG task, e.g., PACS, DigitsDG, and OfficeHome. In Table 1 , we use ResNet18 [21] as the feature extraction backbone which is pre-trained on ImageNet21K [12] for the PACS benchmark.Compared with the state-of-the-art methods, i.e., MEDIC and ODG-Net, the proposed EBiL-HaDS achieves promising performance improvements for all the domain generalization splits. On averaged metrics across all of these DG splits, our approach delivers performance improvements by 1.11% of close-set accuracy, 2.68% of H-score, and 2.45% of OSCR for the binary classification head (bcls), and consistent performance improvements can be found in the conventional classification head (cls). Through using EBiL-HaDS to achieve a more reasonable domain scheduler during the training, we observe that on the most challenging domain generalization split, i.e., Cartoon as unseen domain, EBiL-HaDS delivers the most performance benefits. The core strength of EBiL-HaDS is its adaptive domain scheduling, optimized through a bi-level manner to achieve maximum rebiased discrepancy evidential learning. This ensures comprehensive and discriminative data partitions during Further experiments on a different backbone, i.e., ResNet50 [21] , are delivered in Table 2 , where our method contributes 1.08%, 0.94%, and 1.27% performance improvements of close-set accuracy, H-score, and OSCR for binary classification head and consistent performance improvements for the conventional classification head. EBiL-HaDS contributes more performance improvements when we compare the experimental results on ResNet18 with ResNet50 [21] for the PACS dataset, which illustrates that the EBiL-HaDS is more helpful in alleviating the generalizability issue of modelpreserving light-weight network structure since network with small size is hard to optimize and obtain the generalizable capabilities on challenging unseen domains.We further conduct ablation on model architecture on ResNet152 and ViT base model [13] , where our proposed method is compared with the MEDIC and other challenging baselines, i.e., ARPL, MLDG, SWAD, and ODG-Net. From Table 2 and Table 1 we can observe an obvious OSDG performance improvement by increasing the complexity of the leveraged feature extraction backbone. However, when we compare Table 3 and Table 2 , some baseline approaches trained with ResNet152 backbone even show performance decay on the major evaluation metric OSCR. This observation demonstrates that most OSDG methods face the overfitting issue when using a very large backbone, which is a critical issue for open-set challenging to recognize samples from unseen categories, especially in an unseen domain. The MEDIC-bcls approach shows 6.48% performance degradation on OSCR when we replace the backbone from ResNet50 to ResNet152. Using the proposed BHiL-HaDS to achieve a more reasonable task reservation in meta-learning procedure, BHiL-HaDS-bcls delivers 86.25% in terms of OSCR on ResNet152 [21] backbone, where the OSCR performance of BHiL-HaDS-bcls on ResNet50 is 86.12%.This observation shows that our proposed adaptive domain scheduler can make the meta-learning effective on large complex models by reserving reasonable task for model optimization. Additional experiments on the ViT base model [13] (patch size 16 and window size 224) are provided in Table 4 , where we observe that our proposed method can deliver consistent performance gains on the transformer architecture.This observation is further validated by the experimental results on DigitsDG where a smaller network structure, i.e., ConvNet [66] , is used, as shown in Table 5 . Our method contributes 3.74%, 4.83%, and 4.72% performance improvements of close-set accuracy, H-score, and OSCR for binary classification head and 3.74%, 12.29%, and 5.64% performance improvements of close-set accuracy, H-score, and OSCR for the conventional classification head. Consistent performance improvements are shown in Table 6 on the OfficeHome. We further observe that using EBiL-HaDS, the optimized model can contribute a distinct separation between confidence scores of the model on the unseen categories and seen categories in the test unseen domain, showing the benefits of a reasonable domain scheduler for OSDG.We deliver the ablation experiments in close-set accuracy, H-score, and OSCR for binary classification head and 1.82%, 2.73%, and 3.68% performance improvements of close accuracy, H-score, and OSCR for conventional classification head. The significant OSDG performance improvements highlight the importance of the confidence score learned by the max rebiased discrepancy evidential learning in supervising the follower network, ensuring the promising reliability prediction. Then we use a sequential scheduler and keep the L RBE in the third part of Table 7 (w/o DGS), where our approach outperforms this variant by 2.04%, 3.92%, and 3.89% of close accuracy, H-score, and OSCR for binary classification head and 2.04%, 3.31%, and 5.07% of these metrics for conventional classification head. This observation shows the importance of the proposed domain scheduler for the OSDG task and highlights the effect of using meta-learning trained with a reasonable data partition. Both ablations show better OSDG performances compared with MEDIC, confirming the benefit of each component. We further deliver more ablations, e.g., the benefits brought by the max discrepancy regularization term, comparison with recent curriculum learning approaches, and ablation of the rebiased layers in the appendix.We present several comparison experiments in Table 8 to demonstrate the efficacy of various domain schedulers when applying meta-learning to the OSDG task. We compare our proposed EBiL-HaDS with both the sequential domain scheduler and the random domain scheduler. The sequential domain scheduler selects domains in a fixed order for batch data partitioning, while the random scheduler assigns domains randomly. Results show that EBiL-HaDS significantly outperforms both the random and sequential domain schedulers. Specifically, EBiL-HaDS achieves performance improvements of 3.13%, 8.39%, and 4.63% in closed accuracy, H-score, and OSCR for the binary classification head, and 3.13%, 17.92%, and 5.77% in these metrics for the conventional classification head compared to the random scheduler. This ablation demonstrates that our scheduler enables the model to converge to a more optimal region, which outperforms both predefined fixed-order (sequential) and maximally random (random) schedulers, underscoring the importance of a well-designed domain scheduler in meta-learning for the OSDG. More ablations on domain schedulers are supplemented in the appendix.In Figure 2 , we deliver the TSNE [49] visualization of the latent space of MEDIC and our approach on the OSDG splits, i.e., photo and art as unseen domains. Unseen and seen categories are denoted by red and other colors. we observe that the model trained by our method delivers a more compact cluster for each category and the unseen category is more separable regarding the decision boundary in the latent space. Our method's ability to improve the generalizability of the model is particularly noteworthy. The well-structured latent space facilitates better transfer learning capabilities, allowing the model to adapt more efficiently to new, unseen categories. This characteristic is especially beneficial in dynamic environments where the data distribution can change over time. In essence, the effectiveness of our approach in achieving a more discriminative and generalizable latent space can be directly linked to the sophisticated data partitioning achieved through EBiL-HaDS. This demonstrates the profound influence that carefully designed domain schedulers can have on the overall performance of deep learning models, emphasizing the need for thoughtful consideration in their implementation.We first conduct the ablation towards different unseen categories with a predefined open-set ratio in Figure 3a and Figure 3b of close-set accuracy and the OSCR for open-set evaluation, where the performance of the ODG-NET, binary classification head, and conventional classification head of MEDIC method and our method are presented. We then conduct the ablation towards different numbers of unseen categories in Figure 3c and Figure 3d of close-set accuracy and the OSCR for open-set evaluation, where the performance of the ODG-NET, binary classification head, and classification head of MEDIC method and our method are presented. From the experimental results and comparisons, we can find consistent performance improvements, indicating the high generalizability of our approach across different open-set ratios and unseen category settings. 
conclusion: 
 In this study, we introduce the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS) for the OSDG task. EBiL-HaDS is designed to create an adaptive domain scheduler that dynamically adjusts to varying domain difficulties. Extensive experiments on diverse image recognition tasks across three OSDG benchmarks demonstrate that our proposed solution generates more discriminative embeddings. Additionally, it significantly enhances the performance of state-of-the-art techniques in OSDG, showcasing its efficacy and potential for broader applications for deep learning models.Limitations and Societal Impacts. EBiL-HaDS positively impacts society by enhancing model awareness of out-of-distribution categories in unseen domains, leading to more reliable decisions. It emphasizes the importance of domain scheduling in OSDG. However, the method may still result in misclassification and biased predictions, potentially causing negative effects. EBiL-HaDS relies on source domains with unified categories and has so far only been tested on image classification.conventional and binary classification heads averagely. These experiments showcase the significance of regularization term R RB , which contributes more than 50% performance improvement in the whole L RBE component on all 3 metrics with either conventional or binary classification head.  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The authors confirm that the potential positive and negative society impacts are discussed in the conclusion section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The authors confirm that this submission poses no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The authors confirm that the original papers that produced the code package or dataset are correctly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We visualize the impact of various hyperparameter configurations on the DigitsDG dataset for close-set accuracy, as well as the OSCR of both the conventional and binary classification heads under different dataset partitions (Mnist, Mnist-m, SVHN, SYN as unseen domains), as illustrated in Figure 7 , Figure 8 , and Figure 9 . Figure 7 presents an ablation study focusing on the σ influences, and demonstrates that varying σ from 2e -1 to 2e -6 slightly impacts model performance, with certain settings yielding optimal results. For instance, lower values of σ generally enhance the OSCR of binary classification head, particularly in more complex datasets like SVHN, suggesting that σ is important for achieving a balance between robustness and accuracy in domain generalization scenarios.In Figure 8 , adjustments to the loss weight of L REG ranging from 1e -3 to 1e -6 are analyzed, which is evident that the loss weight of L REG with 1e -4 yields the most optimal results across various metrics. Conversely, excessively lower loss weight of L REG appears to have a detrimental effect, potentially leading to overfitting or diminishing the model's ability to generalize effectively across different domains. Besides, Figure 9 explores the ablation of the loss weight of L RBE via adjusting L RBE from 5e -1 to 5e -4 . It shows a direct effect on both accuracy and OSCR, with lower weights generally improving performance, particularly for challenging datasets like SVHN, which suggests that L RBE plays a critical role in the evidential learning framework, significantly influencing the model's ability in open-set conditions under unseen domains. Tuning of hyperparameters σ, the loss weights of L REG and L RBE are significant for optimizing the performance of domain generalization models. Optimal settings remarkably enhance both the accuracy of meta-training and the OSCR of the classification heads, particularly under complex and challenging dataset conditions like SVHN as the unseen domain.In this subsection, we provide the ablation of the layer number used for the rebiased operation as shown in Table 10 . In this ablation, 1-1 layer, 2-1 layer, and 2-2 layer indicate that the R θ1 and R θ2 are both constructed by 1 convolutional layer, constructed by 1 and 2 convolutional layers, and both constructed by 2 convolutional layers, with unified kernel size 3. We first observe that all of the ablation experiments outperform the baseline MEDIC in terms of the averaged OSDG performance, indicating that with rebiased setting our method can achieve overall OSDG performance improvements regardless of the layer constructions. Delving deeper into the ablation comparison, we find that different convolutional layers to construct the rebiased head can achieve the best performance. This 2-1 layer setting is thereby adopted in other experiments.We deliver the comparison among the predefined domain scheduler [54] , the self-generated reliabilitybased domain scheduler (denoted as SDGS), the easier domain scheduler (denoted as EDS), and our domain scheduler in Table 11 . The self-generated reliability-based domain scheduler indicates that we do not rely on the follower network to achieve the reliability assessment while the confidence score from the main network is utilized for the domain scheduling. This comparison is designed to showcase the importance of the follower network used for the domain reliability assessment. The easier domain scheduler indicates that we use the domain with the highest reliability to accomplish the data partition during the meta-learning.Through using the follower network, we observe that our method outperforms SDGS by 2.19%, 2.81%, and 3.29% of Acc, H-score, and OSCR for binary classification head and 2.19%, 3.42%, and 4.05% of Acc, H-score, and OSCR for conventional classification head. Consistent performance benefits of our method can be observed when we compare the results of the EDS with ours, indicating the importance of using the hardest domain scheduler in the OSDG task when meta-learning is involved. Compared with the predefined domain scheduler from MEDIC, SDGS and EDS outperform it obviously, indicating the importance of using an adaptive domain scheduler during the meta-learning procedure for the OSDG task. We implement two recent curriculum learning approaches, i.e., the training paradigms proposed by Wang et al. [56] and Abbe et al. [1] into the OSDG task, where the comparison of the experimental results are delivered in Table 12 . We first observe that using different training paradigms can benefit OSDG. Compared with the MEDIC baseline, the approach proposed by Abbe et al. [1] achieves performance improvements of 2.44%, 3.49%, and 0.92% and 2.44%, 2.04%, and 1.09% of Acc, H-score, and OSCR for the conventional classification head and the binary classification head. The approach proposed by Wang et al. [56] also delivers promising comparable results with the MEDIC baseline. Furthermore, since our approach is specifically designed for the OSDG task, our approach achieves performance improvements of 1.30%, 8.80%, and 4.72% and 1.30%, 4.25%, and 3.63% in Acc, H-score, and OSCR for the conventional classification head and the binary classification head compared with the approach proposed by Abbe et al. [1] , illustrating the superior performance of our domain scheduler based data partition in the OSDG task.We visualize the impact of different strategies for splitting known and unknown classes on the DigitsDG dataset, specifically examining the effects on close-set accuracy and OSCR of both conventional and binary classification heads. The analysis covers various dataset partitions, including Mnist, Mnist-m, SYN, and SVHN, as depicted in Figure 4 and Figure 5 . Figure 4 presents an ablation study focusing on the selection of unknown classes with a 6:4 ratio. Despite the suboptimal dataset partitioning leading to declines in both close-set accuracy and OSCR, the superiority of our method remains largely unaffected by the choice of unknown classes. It consistently achieves the highest close-set accuracy and OSCR across most domains within the DigitsDG dataset, while also maintaining competitive performance in the remaining domains. This demonstrates the model's proficiency in distinguishing and recognizing known classes within the training set, as well as its capability of managing unseen classes, thereby highlighting its robustness in open-set environments. Furthermore, Figure 5 illustrates an ablation study examining various open-set ratios (7:3 and 8:2). By analyzing the impact of various open-set ratios in this ablation, we show that our method effectively mitigates saturation in close-set accuracy, maintaining robust generalization capabilities of OSDG. However, the imbalance between known and unknown samples diminishes the model's ability to differentiate unknown classes, resulting in a general decrease in OSCR. Despite this challenge, our method consistently achieves the highest OSCR across different open-set ratios compared with the baselines. Notably, in the complex and challenging SVHN domain, our OSCR exceeds the baseline MEDIC-bcls by 5.08% at the 8:2 ratio. This finding underscores our model's exceptional performance in detecting unknown classes and accurately classifying known ones.We visualize the accuracy changes during training every 100 epochs for the meta-learning process, reporting both validation accuracy and test accuracy, as shown in Figure 6 . Implementing our domain scheduler introduces significant improvements in the model's training dynamics. Notably, the validation accuracy curve appears smoother compared to the MEDIC baseline, as illustrated in Figure 6a and Figure 6c . At the early stages of meta-learning for OSDG, our domain scheduler demonstrates superior initial training performance, with test set accuracy surpassing that of MEDIC [54] . These findings suggest that customizing the training schedule to match distinct domain partitions within the data can substantially enhance both training efficiency and overall performance. Our approach capitalizes on specialized domain knowledge, allowing the training algorithm to better adapt to varying data characteristics, ultimately optimizing the model's performance outcomes. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The authors have reviewed the NeurIPS Code of Ethics and confirmed for all of the content.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).domains, i.e., art, clipart, product, and real-world. Since MEDIC[54] did not provide a detailed benchmark on OfficeHome, we construct the whole benchmark using several outstanding baselines and our approach, where the last 30 categories following the alphabet order are chosen as unseen categories. The leave-one-domain-out DG setting is adopted. Close-set accuracy (acc), H-score, and OSCR serve as metrics following[54], where OSCR is the primary metric for OSDG. 


Decision: While the initial recommendations were divergent, all four reviewers reached a positive consensus (three borderline accept and an accept) after the post-rebuttal discussions. The AC therefore recommends accepting the paper and asks the authors to include their discussions with the reviewers in the final manuscript.

Reviews: review: paper_summary: paper_summary: The paper presents an adaptive domain scheduler with ability to adjust the training order dynamically according to model’s current performance and domain difficulty to address OSDG, short for Open-Set Domain Generalization (OSDG), task where the model is exposed to domain shift and category shift.
strength_weakness: strength_weakness: 1.	Several mistakes in format such as missing indent before paragraphs and size of pictures, and the formula in between line 161 and line 162. 
2.	No open resource code and data. Inconvenience to reproduce and follow the experiment. Also unable to exam the effectiveness of the result.
questions: questions: 1.	Citation in line 91、93 and so on, citation order is not standard. 
2.	Line 331~474, The reference format is incorrect, citation of journals needs to be marked with issues, pages and years
limitations: limitations: Experiments show that the method proposed in this paper may still lead to misclassification and biased content. This may cause useless even false predication , causing critical negative influence in popularizing. Additionally, EBiL-HaDS is significantly relied on source domains with unified categories and its using scene is strictly limited, it has only been tested on image classification tasks.
paper_summary: paper_summary: In this paper, an observation is proposed, that an adaptive domain scheduler benefits more in OSDG compared with prefixed sequential and random domain schedulers. A follower network is proposed to strategically sequences domains by assessing their reliabilities, which is trained with confidence scores learned in an evidential manner and optimized in a bi-level manner. Experiments show the superior OSDG performance and ability to get more discriminative embeddings for both the seen and unseen categories.
strength_weakness: strength_weakness: The experiments conducted only on ResNet50 ResNet18 and ConvNet. The number of parameters in those backbone are limited. If the comparison is conducted on models with a large number of parameters, like ResNet152 or ViT-based models, would the advantage in classification performance be leveled off?
questions: questions: If the comparison is conducted on models with a large number of parameters, like ResNet152 or ViT-based models, would the advantage in classification performance be leveled off?
limitations: limitations: The experiments only conducted on classification tasks.
paper_summary: paper_summary: The paper addresses the challenges of Open-Set Domain Generalization (OSDG) and introduces the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which adaptively sequences domains based on their reliability, assessed through a follower network. The authors verify EBiL-HaDS on three benchmarks: PACS, DigitsDG, and OfficeHome, demonstrating substantial performance improvements in OSDG.
strength_weakness: strength_weakness: 1. The proposed framework is too complex and hard to reproduce. The source code is also not provided.
2. The authors only use CNN backbones. More ablations on ViT backbone should be added, as it demonstrates strong domain generalization performances compared with CNN [A, B]

[A] Li et al., Sparse Mixture-of-Experts are Domain Generalizable Learners, ICLR 2023

[B] Zheng et al., Prompt Vision Transformer for Domain Generalization, arxiv 2022
questions: questions: What is the performance under ViT architecture?
limitations: limitations: The authors discuss the limitation of potential leading to misclassification and biased content, potentially causing false predictions.
paper_summary: paper_summary: In this paper, the authors identify a new problem in open-set domain generalization, proposing that dynamically adapting the domain scheduler used for data partitioning based on specific criteria could lead to a more targeted training strategy and improved outcomes. They introduce a novel training strategy named the Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS), which combines max rebiased discrepancy evidential learning with meta-learning to create an adaptive domain scheduler.
strength_weakness: strength_weakness: 1. The motivation for considering the training order of domains is unclear. I believe that if there were temporal correlation between domains, considering the training order would be reasonable. However, in this research setting and the datasets used in the experiments, there is no temporal correlation. Therefore, why should the training order be considered?

2. In my opinion, this paper primarily makes some improvements to the MEDIC model. However, compared to MEDIC, the proposed method does not show significant performance improvement on some tasks.

3. Dynamically adapting the domain scheduler used for data partitioning based on certain criteria can be seen as a trick rather than a major contribution.

4. The key idea of EBiL-HaDS is to quantify domain reliability. I suggest that the authors provide a more detailed definition of domain reliability and demonstrate that domain reliability is existing in real-world scenarios with a significant impact on generalization.

5. What is the motivation for applying max rebiased discrepancy evidential learning to achieve more reliable confidence acquisition? Are there any particular advantages to this approach? The authors should compare this approach with other similar techniques.

6. There are still some grammatical mistakes and typos in this paper. Please proofread the paper carefully.
questions: questions: Discussed in the weaknesses.
limitations: limitations: Yes.

Comments: title: To Reviewer euSj
To Reviewer tsHR
To Reviewer tsHR
To Reviewer euSj
To Reviewer QWTs
Anonymous code
To author
Question to AC regarding the url code link and response to the reviewer
To Author
comment: Dear Reviewer euSj,

Thank you very much for your response and decision, the authors are very glad to hear your decision. We will include all the experiments and source code link in our final version.

Thank you.

Best,

The authors
Thank the authors for their rebuttal. Most of my concerns have been well-addressed and I thus increased my score to 5.
Dear Reviewer tsHR,

Thank you for your fast feedback. The authors are pleased with your decision and will conduct all your suggested modifications in the final version.

Best,

The authors.
Thanks for the authors' detailed response and additional experiments. I will increase the score to 5.
Dear Reviewer tsHR,

Thank you very much for your review effort and questions. 

The authors would like to ask if you have any further questions on the rebuttal reply, since the author reviewer discussion period will end soon.


The point-to-point response of each question in the weakness part is provided in the rebuttal response. In case there is anything unclear please feel free to let us know.

Thank you.

Best,

The authors.
Dear Reviewer euSj,


Thank you very much for your comments and your contribution to the review procedure. 

The authors would like to ask if you have any question on the rebuttal, since the discussion period between the authors and the reviewers will end soon.

We provided the source code link and the additional 24 ablation experiments on ViT-Base according to your instruction in the rebuttal. Please let us know if you have any further questions.

Thank you.

Best,

The authors.
Dear Reviewer QWTs,

Thank you for your effort during the review procedure. 

Since the author-reviewer discussion will be end soon, the authors would like to ask if you have any questions on the rebuttal. We have provided 48 experiments to build the benchmarks on ViT-base and ResNet152 backbone to illustrate the efficacy of our proposed method according to your question. 

Thank you.

Best,

The authors.
Dear reviewers, 

I have checked the anonymity of the source code, which is okay for me. The anonymous GitHub link provided by the authors is available at https://anonymous.4open.science/r/EBiL-HaDS-623E. 

Best
Thank you, I would like to increase my score to 7, and I look forward to the author's future work on DG.
Dear Reviewer DFZJ, dear AC,

@Reviewer DFZJ, Thank you for your fast response and we are very glad to hear that you will increase your score. Thank you for the effort of review. We will ask the AC to make the code available to the reviewers. 

@AC, could you please make the source code link we submitted in the official comment to AC available to the reviewers? Thank you. 

According to the rebuttal instruction we have provided the anonymous link of code in the official message to the AC, where the reviewers are not chosen as readers.

Thank you for your help.

Best regards,

The authors.
I can't find the url of the code, and if the code is available, I would like to increase my score.