Clustering-Based Validation Splits for Model Selection under Domain Shift
Authors: Andrea Napoli
Paul White
Venue: Accepted by TMLR ()

Abstract: This paper considers the problem of model selection under domain shift. Motivated by principles from distributionally robust optimisation and domain adaptation theory, it is proposed that the training-validation split should maximise the distribution mismatch between the two sets. By adopting the maximum mean discrepancy (MMD) as the measure of mismatch, it is shown that the partitioning problem reduces to kernel k-means clustering. A constrained clustering algorithm, which leverages linear programming to control the size, label, and (optionally) group distributions of the splits, is presented. The algorithm does not require additional metadata, and comes with convergence guarantees. In experiments, the technique consistently outperforms alternative splitting strategies across a range of datasets and training algorithms, for both domain generalisation and unsupervised domain adaptation tasks. Analysis also shows the MMD between the training and validation sets to be well-correlated with test domain accuracy, further substantiating the validity of this approach.

Published in Transactions on Machine Learning Research (08/2025)
Clustering-Based Validation Splits for Model Selection under Domain Shift
Andrea Napoli & Paul White
fon1g18, P.R. Whitee
Institute of Sound and Vibration Research
University of Southampton, UK
Reviewed on OpenReview: https: openreview. net/ forum? id= Q69200Wt:
Abstract
This paper considers the problem of model selection under domain shift. Motivated by principles from distributionally robust optimisation and domain adaptation theory, it is proposed that the tween the two sets. By adopting the maximum mean discrepancy (MMD) as the measure of mismatch, it is shown that the partitioning problem reduces to kernel k-means cluster- ing. A constrained clustering algorithm, which leverages linear programming to control the size, label, and (optionally) group distributions of the splits, presented. The algorithm does not require additional metadata, and comes with convergence guarantecs. In experi- ments, the technique consistently outperforms alternative splitting stralegies across a range of datasets and training algorithms, for both domain generalisation and unsupervised do- main adaptation tasks. Analysis also shows the MMD between the training and validation sets to be well-correlated with test domain accuracy, further substantiating the validity of this approach.
split should maximise the distribution mismatch be-
1 Introduction
The ability for models to maintain high performance data lying outside their training distribution, known domain generalisation (DG), is crucial to the widespread deployment of artificial intelligence. Although exlensive research has been conducted lowards developing more generalisable training algorithms Guulaajani & Lopez-Pazz 2021), significantly less focus has been given to increasing the robustness of the model selection process, despite being as intogral part of the learning problem, and indcod, just as susceptible to distribution shifts, as the fitting of the model itself.
As with model parameters, hyperparameters chosen based on in-distribution (ID) performance lack opti- mality guarantecs out-of-distribution (OOD) test data. Metadata- based dataset splitting, which creates OOD validation sets distinct from both the training and test data, is commonly used in this scenario, and has empirically been shown to encourage the selection of more generalisable hyperparameters (Koh et al. 2021). This paper is motivated by the principles of distributionally robust optimisation (DRO) (Rahimian & Mehro- tra 2019), which aims to optimise for worst-case performance within some uncertainty set of distributions. This suggests that the validation set should indeed be marimally domain shifted from the training set, while still retaining relevant distribution. It proposod that a worst-case datasct split which maximises domain mismatch would balance these two aims, and further encourage the selection of robust hyperparameters.
Among other considerations, the measure of mismatch should be such that the partitioning problem be tractably solved. To this end, it is noted that performing kernel k-mcans clustering is equivalent to maximising the empirical maximum mean discrepancy (MMD) between clusters (weighted by cluster size). Thus, this paper proposes to perform a validation split based on kernel k-means, and presents a modified clustering algorithm for this purpose. Specifically, constraints are introduced the cluster assignment step control the holdout fraction (i.e., the cluster sizes), and to preserve class and (optionally) domain/grouu /group dist distributions; this is then formulated and solved linear program, providing convergence guarantees not

---

Published in Transactions on Machine Learning Research (08/2025)
present in prior work. In addition, the Nyström method for low-rank approximation is employed to make the algorithm tractable for large datasets.
The proposed method provides a model selection strategy based on OOD performance that does not require additional metadata, which is not always available. It is also argued that the method is able to capture more nuances in real-world data than can be described by a single domain variable, which is another limitation of iatadata-bassee splits. For example, in tumour identification, domain shifts can be caused by variations in sample preparation methods patient populations. Or, in wildlife monitoring (both acoustic and visual), these can be due to differences in environmental weather conditions, data collection equipment, or new unseen events (Koh et al. (021). However, all this information is unlikely to be described in the available metadata, which may only state the hospital of origin or the recording location, respectively. In such cases, splitting the data along lines closer to the underlying cause of the shift, as determined by the clustering algorithm, rather than a more loosely correlated proxy, may permit a more informed model selection which can result in hyperparamcters better suited to the nature of the shift.
This paper considers the two most common paradigms for learning under domain shift: domain generalisation (DG) and unsupervised domain adaptation (UDA). Both paradigms assume the data can be grouped into "domains" which share certain characteristics, and that each domain is associated with different data distribution. Given multiple source domains, DG seeks to learn domain-invriant feature representations such that the model becomes robust to future (unknown) changes in the data. In contrast, UDA assumes unlabelled data from a specific target domain also available during training. Although DG operates in a stricter setting, it also requires fewer assumptions about the target data for example, UDA methods often assume minimal label shift., but this can have a significant detrimental effect on performance (Napoli & White, 2024). UDA is also less stable than DG, as it is susceptible to phenomena such catastrophic forgetting, negative transfer, or overfitting on the target data. However, DG requires that the causal mechanisms behind the domain shift remain stable, that is, that the test domain is drawn from the same meta-distribution as the source domains. As this is difficult to ascertain in practice, test domain performance is also harder to predict. In contrast, lest domain accuracy in UDA is directly coupled to domain alignment quality, given the theoretical link between the two (Redko et al. 2022
In summary, this paper contributes the following:
Description of constrained clustering algorithm based on kernel k-means which can be used to perform training-validation split in applications expected to involve domain shifts.
Comparison of this approach with existing validation splitting strategies for a range of datasets and algorithms, in both DG and UDA settings.
Analysis of the relationship between test domain accuracy and the MMD between the training and validation sets.
2 Prior work
Gulrajani & Lopez-Paz (2021) reviewed 3 criteria for model selection in a DG setting: ID accuracy (on a randomly held-out subset of each training domain); OOD accuracy on an additional domain held-out using metadata (referred to as leave-ne-domain-outt); and test-distribution accuracy on a held-out subset of the test domain (referred to as the oracle critcrion), which can be used to provide an upper bound on performance. Where multiple validation domains are given, DRO-style treatment of model selection which considers only the worst- perrt-perfoming domain is studied by /Sagaaa et al. (2019) Gao et al. (0023 Pfohl et al. h although this again relies on the availability of metadata.
In addition to validation accuracy, it has been suggested that a model's stability to distribution shifts should also be explicitly considered. Prior work has quantified stability in terms of the expected calibration error (the average deviation between accuracy and confidence) (Wald et al.) [021]; ; the MMD between features
from different domains (Lyu et al. 2023); or the average variation of each feature between domains (Ye al.
2021
2

---

Published in Transactions on Machine Learning Research (08/2025)
It has also been proposed to induce domain shift between the training and validation sets by performing mix-up augmentation on the held-out data (Luu et al. 2023). This links to a range of general robustness benchmarks, where the evaluation sets have been subjected to various synthetic transformations. For exam-
ple, visual corruptions and perturbations (Hendrycks & Dictterich 2019), stylisation (Geirhos et al. 2018
the addition of spurious cues (Li et al.] 2022), and adversarial raltration (Hendrycks et al., 2019) have all
been employod; a more complete review of this approach is given in Koh et al. (2021).
Non-random data splits have previously been used to produce domain-shifted evaluation sets, but prior work has investigated these in model selection context. Sogaard et al. (021) proposed an adversarial split- ting hcuristic based on k-ncarest neighbours and the Wasserstein distance between term count distributions of text documents, but this is not generally applicable to features in IR", and no attempt was made to control the label distribution. Adversarial data splits have also been used in meta-learning to improve DG (Gu et al. 2023 Wang et al. 2024). Recently, Napoli & White (2025) proposed a data selection algorithm which can be used choose a validation set closer to the test domain in the UDA sctting. However, this is not applicable if the split needs to be decided before the Lest domain is accessible, e-g., for test-time adaplation.
Finally, with approach most similar to ours, Wecker et al. (2020) proposed an evaluation split based on k-means clustering, with constraints to control for cluster size and label distribution. Our paper develops on this work in a number of ways. Firstly, the motivation of Wecker et al.] (2020) is merely to create more challenging data splits, and is not applied to improving model selection nor theoretically connected to distributional robustness. Secondly, Weekerr et al. (2020) impose the constraints using a greedy strategy, which are susceptible to local minima. In contrast, we solve constrained assignment problem using linear programming, which avoids this effect and results in better quality solutions. We also generalise the label- wise constraints to domains for multi-domain data. Finally, our theoretical analysis motivates kernelised version of this algorithm, thereby exploiting the relation between the kernelised objective (which we link to the MMD) and validation accuracy.
The theoretical relation between the MMD and kernel k-means clustering has also been derived in other contexts. For example, |Frnçç et al. (2017) obtain similar results from the viewpoint of generalised energy statistics, while Ohl et al. (2024) use the relation to improve training of unsupervised decision trees.
3 Preliminaries and notation
Let S = V1. di).... (In, Un, d } be the development set consisting of laput-label-domain triplets over X'x xx>xDD x DS. Similarly, let E. be the evaluation set over X' x x'x3'x x DE, where DEDDE = 0 (i.e., the development and evaluation domains are disjoint), and let V, DS and DE be finite sets. For ease of notation, subscripts are used on sets to simplify sel-builder notation, in two ways. Firstly, "slices" of a set are denoted using capitals, for example:
Sx (x (a, d) SS
SYD ((y d) (I, d) S}.
Additionally, a predicate can be specified to restrict the set to samples satisfying a condition. For example, to denote only inputs associated with specific class y':
Sx.Y=y = fo: (z. d) E ESAV-
In short, the goal of DG is to use S to produce model 0 0: X + y that performs well on E. 0 comprises featuriser OF X F and label classifier Oc F 37, such that 0 Oc Or. In order to tune hyper- parameters, S must be partitioned into training and validation sets, T and V respectively. A mumber of models θ = {01..... ) are trained on T using different hyperparameters: the "best' model is then selected, according to
0-
arg min R(0, V). lee
(1)
where R(0, V) is the error of 0 V, and 0* is evaluated E.
3

---

Published in Transactions on Machine Learning Research (08/2025)
3.1 Theoretical motivation
DRO (Rahimian & Mehrrotra 2019: Sagawa et al. 2019) is a well-established and theorelically grounded paradigm for obtaining more generalisable solutions to optimisation problems. Instead of reducing overall error, DRO adopts a minimax strategy which minimises the worst-case error over some uncertainty set Q. By defining Q as the possible (suitable) partitions of S, the model selection problem becomes
min max R(O,V) BEO (T.
(2)
This could be solved directly via an expensive alternating optimisation process (Gu ct al. 2023). However, we propose instead to find arg maxt,v R(0, V), the worst-case partition in the sense of R, in a single step by solving a surrogate problem based on the well-known relationship between R(0, V) and the mismatch between T and V, which has been fundamental to domain adaptation theory (Ben-David al. 2006 2010). By using the MMD measure this mismatch, the partitioning problem can be efficiently solved through clustering, as will be shown in Section 4 The theoretical relation between MMD(T, V) and R(0, V) is given by the generalisation error bound (Redko et al. 2022 Theorem 36)
R(0, V) S T) + MMD V) +
where À is term depending on the capacity of the hypothesis space and the combined performance of an ideal model on both T and V. A linear correlation has also been observed empirically (Napoli & White, 2025 Although maximising the MMD is not guaranteed to maximise this bound due to the dependence of R(0, 7) on the partitioning, the empirical analysis in Section 5.5 shows that this term remains fairly stable in practice.
Equation (3) is quite general and not strictly limited to covariate shift. However, the robustness of the bound relies on the ability of the model to simultaneously perform well on both and V (through A); and this can be affected by the presence of conditional shifts. There are also assumptions made about both the (joint) data distributions and the hypothesis class of models of these- and these ultimately define the reproducing kernel Hilbert space in which the MMD in the bound is measured. Thus, by symmetry, the choice of kernel shapes the underlying assumptions about the model and data, and ultimately determines whether the MMD being optimised is predictive of the discrepancy between the training and validation losses.
refer to Redko et al. (2022) for formal statements
4 Method
This section defines the constraints needed to ensure an appropriate split, formulates the partitioning prob- lem, and describes an algorithm to solve the optimisation.
T and V should be of sizes determined by a user-defined holdout fraction h satisfying 0 < h < 1, and have equivalent class distributions. This can be achieved by constraining the size of each label group in V to h times the size of the corresponding group in S:
VV==gl = h Sy=gl. Ag E y.
(4)
This constraint ensures that there are sufficient examples from each class both and V to properly train and validate the models. Without it, the data may have a tendency to cluster by class, which may result in certain classes being placed entirely in one set.
It may also be necessary desirable to control domain distributions. For example, certain training algo- rithms may require that the domains in T be uniformly represented to avoid overfitting: controlling the distributions of validation groups has also been suggested to reduce noise in the hyperparameter luning process (Sagawa et al.) 2019). Finally, in the multi-task learning setting (i.e., some or all of the domains cor- respond to different tasks, with separate validation metries), the domain split-ratio should also be controlled: this both prevents bias towards a specific task, and also ensures that each task has sufficient representation in the validation set to reliably estimate the corresponding validation metric. In these cases, the constraints
4

---

Published in Transactions on Machine Learning Research (08/2025)
should be taken over all (y, d) pairs instead:
[ = |
yx
(5)
For the remainder of this section, the latter set of constraints 5 are assumed, although 4 can easily be substituted if desired (as relaxing the constraints will increase the clustering objective), or if domain labels are unavailable.
The objective is to maximise the discrepancy between and V, measured using the MMD. Given the potentially high dimensionality of X, doing this directly in the input space can be impractical. Therefore, it is proposed to instead measure the MMT) between the distributions of feature sets Or [Tx] and Or [Vx], where Or[-] denotes the image under 8y. Morcover, these intermediate features are those most highly correlated with the model's s outpuls, meaning this is the representation space for which the MMD is most predictive of classification error. For further intuition, consider that a large MMD in this space implies domain shift that the model is not invariant to (and thus is more likely to cause problems) hence why this is also the representation space normally targeted by feature alignment methods (Gulrajani & Lopez-Pas 2021) For models where no intermediate representations are available, the input representations can be used directly, or an alternative feature extractor or dimensionality reduction technique can be used. Prior knowledge on the expected nature of the domain shift could also be incorporated into the clustering features; detailed discussion this is given in Appendix A
Assune OF [[x] and 0F [Vx] are samples from distributions Pr, Pv over F. A positive-definite kernel K K:F. x F + R induces a unique reproducing kernel Hilbert space (RKHS) H on F, along with a mapping Φ F - H. The empirical mean map of Pr (and analogously for Pv) in H given by
PPr
Σ
$()).
(6)
[Tx
[EXx(Tx]
The MMD can then be estimated as the distance between means of samples embedded H:
MMD (Pr, = 1|pr, Mav I
(7)
The partitioning problem can now be formulated as
arg max [Apr, prv lll subject to (5)
(8)
T.V
This is equivalent to performing kernel k-means clustering with k = (i.e., using cluster for each of T and V), subject to the same constraints:
arg min (T,, V) subject to 15 TV
(9)
where V) = SSq(Tx) + SSq(Vx) is the standard kernel k-means objective function and
SSq(Tx)= = Σ llx(f) - 1
(10)
fE0n[Tx]
is the sum of squared deviations of set of points from their centroid (in feature space).
Theorem 1. Problems and are equivalent.
Proof sketch. The equivalence can be derived by applying an ANOVA sum-of-squares decomposition, fol- lowed by a substitution based the polarisation identity. The resulting identity is
|Sxl (SSq(5x) (T,
|pp, pev l1
(11)
[Tx
[Vx|
For a full derivation, see Appendix As Sx is constant with respect to the cluster allocations, and the cluster sizes are lixed by the constraint, these terms can all be dropped from the objective function. Hence, and (9) are equivalent.
Problem 9
be solved by applying variation of Lloyd's algorithm Chitta et al. 2014
5

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 1
Constrained kernel k-mmans clustering
Given an initial set of assignments, alternate between 2 steps until convergence or max iterations reached:
1. Distance Update (maximisation). Compute the distance matrix D E I from each point to the centroid of each cluster using the kernel trick. For large datasets, use the Nyström method (Chitta et al., 2014) to reduce the complexity of the kernel computations from O(n²) to O(gn), by computing only a randomly selected submatrix of the full kernel, of size q x n.
2. Constrained Assignment (expectation). Compute the one-hot cluster assignment matrix U E (0, 10.12 that assigns each point to exactly one of the two clusters. U is the solution to the binary lincar program (LP):
arg min EUsD
(12)
1
subject to
Uig E {0,
for iE fl, e [1,2]
(13)
Ui Ut=1 =1
j for iE fl......
(14)
Σ Uy round (h ((y,D)==l) for gEy D j= = or 2. ())=
(15)
14h ensures that each point is assigned to only one cluster. The disjunctive constraint 15 enforces and indicates that j can take value of either 2. The disjunction arises as (5) is independent of the cluster indices, i.e., it does not matter which index is designated as the validation set. Which option has lower cost depends on the initialisation of the centroids. As there are only 2 clusters, the easiest way approach this is simply to solve 2 LPs, one for cach value of j. and then select the lower-cost solution.
The constraints satisfy Hoffman's sufficient conditions for total unimodularity (Heller & Tompkins 1956) (in particular, can be seen that (14) and (15) form two disjoint sets of constraints, and every element of is referencod at most once in cach sct). The consoquence is that the LP will always have integer solutions, without having to enforce them explicitly. This means the binary constraint [1 be relaxed to
0 U. < for i (1......n) j (1,2)
(16)
and the problem can be solved without integer constraints. To enforce soft constraints, by the inequality round (h (().)))))
15 can be replaced
<
Σ
Un < round (h (1 + Ty) (( for g ge x DS; j or 2,
(I(Y.
(17)
where Ty is the relative tolerance for constraint associated to g-
Proposition 2. Algorithm converges to locally optimal partitioning in fonite ramber of iterations.
Proof sketch. Note that (T,, V) is bounded below by 0 and is also non-increasing, since both the centroid updates and cluster assignments are (or can be interpreted as) optimisation problems which share the same objective function as (gb. can be solved globally at each iteration, and do not. violate any of the coistsaints. Note also that only a finite number partitionings are possible, meaning (T, V) can only decrease a finite number of times. Therefore, convergence in finite time is guaranteed. For a full proof, see Bradley et al. 22000).
Although Proposition provides a trivial upper bound for the number of iterations required until convergence, Lloyd's algorithm is known to be fast in practice, and the runtime is often considered to be linear in the

---

Published in Transactions on Machine Learning Research (08/2025)
number of datapoints based on empirical observations (Cordeiro de Amorim & Makarenkov 2023 For further theoretical analysis on the number of iterations required, see, e.g., Arthur et al. 2009 Arthur & Vassilvitskii (0066).
4.1 Kernel choice
As noted in Section 3.1 the choice of kernel implics certain assumptions about the data. In some applications (e.g.: physical modelling), known feature constraints naturally guide the kernel choice. In our case, however, only relatively weak geometric assumptions can be made. The kernel should also be characteristic to ensure the MMD can distinguish between any two distributions (Sriperunbudur et al. 2200)). However, we note that, unlike kernel methods which make use of the kernel distances directly, kernel k-means clustering only compares belween kernel distances when assigning dalapoints, which would argue makes the solution less sensitive to kernel choice kernel hyperparamcters. Additionally, the cluster size constraints restricts the solution space, which further desensitises the solution to kernel choice.
For the experiments in the subsequent section, we opt to use a Gaussian radial basis function (RBF) kernel x(x,,) v) -->l==-v|" which is a convenient and popular choice for MMD estimation (Sriperumbudur et al. 2009). Perhaps more relevantly, we also note that the correlation between the MMD (with an RBF kernel) and V) has been supported by empirical observations (Napoli & White 2025).
5 Experiments
The benefits of any new model selection method can only be verified when the oracle criterion suggests there is "room for improvement" over a basic random split, i.c., there a performance gap between the two. Thus, the experiments described in this section are set up to reflect this scenario (the limitations of this are discussed further in Section G). For example, it is noticed that UDA tends to exhibit a larger gap than DG, and this is especially pronounced (perhaps unsurprisingly) for adversarial algorithms, which tend to be more sensitive to hyperparameter choices.
Two batches of experiments are run, to reflect both the UDA and DG settings. Each batch comprises an identical training setup applied to different datasets. For the DG experiments, models are trained using the CORAL algorithm (Sun & Saenko 2016), with the clustering performed using constraints 4 For the UDA experiments, the DANN algorithm (Ganin et al. 2015) used to adapt to an additional, unlabelled subset of test domain samples, as well as to align the training domains each other. As DANN was observed be more sensitive to domain imbalances, the validation split is set to preserve domain distributions, i.e., using constraints
In this work, we fix the RBF bandwidth to y 1. To assess the sensitivity of our results to the choice of kernel, we also test a linear kernel <(x, y) = x" which is equivalent to linear k-means clustering.
All feature extractors are finetuned on the entirety S before the features are computed for the clustering, regardless of pretraining. Experiments are conducted using the DomainBed framework (Gulrajani & Lopez- Paz 2021 This means all-but-one of the domains are placed in the development set, and the remaining "evaluation" domain is randomly split into UDA set (for adaptation, unused for the DG experiments), and an independent test set used to determine final accuracy values. Every domain is tested 3 times for reproducibility, each time with a different random seed for model initialisation, hyperparameter search and other stochastic variables. The reported accuracy values are averages over all domains and repeats. Further training and hyperparameter search details are given in Appendix C unless otherwise stated, the remaining details all follow the Domainbed default options. The Gurobi Optimizer (Gurobi Optimization L.I.C. (023) is used solve the LPs. In total, the experiments involve training 5,160 models, requiring around 100 GPU-days of computation.
5.1 Datasets
The datasets represent a range of domain shift problems encompassing both image and audio classification tasks. In addition to the covariate shifts which occur across all datasets, the two ecological datasets (Hump-
7

---

Published in Transactions on Machine Learning Research (08/2025)
backs and Terralncognita) also contain significant conditional shifts due to the open-set and sometimes annotator-depender nature of wildlife monitoring data. With the exception of Camelyon17, the datasets are all small enough that the entire kernel matrix can be computed. So, Camelyon17 is the only dataset for which the Nyström method is applied.
Camelyon17-WIN WILDS (Bándi et al., 2019; Koh et al. 2021 tumour detection in tissue samples across 5 hospitals, 2 classes and 455,954 samples. In keeping with the WILDS setup, the model in this case is trained from scratch rather than using pretrained weights. However, note that these results still cannot be compared directly with results from WILDS, as the DomainBed setup does not match exactly. License: CCO.
Humpbacks (Napoli & White 2023 detection of humpback whale vocalisations across 4 recording locations, 2 classes and 43,385 samples. This is the only dataset not to use the ResNet-18 architecture, and instead uses a custom CNN architecture and acoustic front-end described in Appendix C
License: Proprietary.
SVIRO (Dias Da Cruz al., 2020) classification of vehicle rear seat occupancy across 10 car models, classes and balanced subset of 24,500 samples of the original dataset. License: CC BY-NC-SA 4.0.
VLCS (Fang al. 2013 object classification across image datasets, 5 classes and 10,729 samples. License: unknown.
PACS [Li et al. 2017 object classification across 4 image styles (photos, art, cartoons, and sketches), 7 classes and 9,991 samples. License: unknown.
Terra Incognita (Beery et al. 2018) classification of wild animals across 4 camera trap locations, 10 classes and 24,788 samples. License: CDL.A-Permise 1.0.
5.2 Results
In total, 6 model selection methods are compared. These are: the random split; the leave-one-domain- out split; the test domain (oracle) validation set; a random split followed by mix-up augmentation on the validation set [Lu et al. 2023]; and cluster-base splits using linear and kernel k-means clustering. The random and handom-plus-ning class-balanced for all methods.
splits are stratified by domain. The validation and test accuracics are
The results are shown in Table D along with standard errors (the standard errors are as computed by DomainBed, and capture variability in the overall experimental run, including random seeds and across domains). In the following sections, a 95% confidence level is used when verifying whether two values have statistically significant difference, which corresponds to non-overlapping confidence intervals of 1.96 times the standard error (assuming normally distributed errors).
Table 1: Average test domain accuracies for all datasets and model selection criteria.
experiments
UDA experiments
Raw average
Normalised
Split type
Camelyon
Humpbacks
SVIRO
VLCS
PACS
TerraInc
average
Random
L.0
76.4 +
98.1 1 0.2
2.9
80.3 =
2.9
74.7
1
0.0
Leave-one-ont
85.6 + 1.0
77.3 + 1.7
98.6 0.0
71.3
- 3,5
83.7
0.6
37.3
= 2.5
75.6
- 0.8
28.2 11.7
Linear k-means
87.2
0.2
78.0
+ 1.8
98.1 0.0
76.9
1 0.1
82.7 = 0.2
43.0
2.0
77.7
+ 0.5
55.2
+ 5.9
RBB lemmaans
87.3
0.7
78.3
98.6 0.2
75.2
0.8
82.3
40.1
2.1
77.0
0.4
46.9
7.6
Mix-up
85.1 4 0.1
76.1
- 1.2
98.2 2 0.1
73.8
4 1.7
80.5 = 0.6
37.3
- 2.9
75.2
- 0.6
10.4
+ 8.9
Oracle
0.6
85.4
99.1
0.5
84.4
1.0
80.1
100.0 5.1
The range of possible performance improvement differs by dataset, determined by the gap between the random split and oracle criterion; accuracy values should be considered relative to this scale when averaging across datasets. Therefore, a column of average normalised values is also shown, where each dataset is shifted and scaled such that the random split has a value of 0 and the oracle a value of 100.
Where the validation set comprises multiple domains, model selection is based on average validation accuracy across these domains, as is the DomainBed default. Results based on worst-domain accuracy are also shown in A Appendix [) In theory, worst-domain accuracy should provide additional layer of distributional robustness
8

---

Published in Transactions on Machine Learning Research (08/2025)
However, no significant difference in test accuracy overall is observed if worst-case validation accuracy is used, although the standard error does increase. It is possible that the uncertainty set 2 = D has too few degrees of freedom for DRO to confer any meaningful robustness in this case.
with respect to the domain labels (Sagawa al. 2019
On average, the cluster-based splits provide net absolute accuracy gain of around percentage points compared to the random split, and percentage points gain compared leave-one-domain out validation. In relative terms, clustering is observed to close around 50% of the gap between the random split and oracle criterion, compared to 28% for leave-one-domainoott validation and 10% for mix-up. Overall, performance is slightly higher using the linear kernel than with the RBF kernel, although this is within margin of error. It is possible that the latter may be improved with more careful luning of y. It is noted that both clustering methods are also within margin of error leave-one ceeeoooone-d-----eeeeeee validation.
5.3 Ablation study on VLCS
An ablation study conducted on the VT.CS dataset shown Table 2 This shows the effects of finetuning the feature extractor before clustering, the Nyström approximation, and the different constraint sets 4 and
5 For this datasct, use of the Nyström approximation, as well as additional finetuning of 0r, are not observed have significant effects on test accuracy. For the linear kernel, clustering with constraints (4) performs significantly lower than using constraints (5) however, for the RBF kernel, this difference is not observed.
Table 2: Ablation study VLCS.
Kernel/ Split type
Constraint groups
Full/
Finetuning 0p
Accuracy (%)
Nyström kernel
Linear
True
E D
Nyström
76.4 0.4
RBF
True
g E
D
Nyström
76.2 0.6
Linear
True
g E g E g C
Full
72.4
RBF
True
Full
75.1
Linear
True
D 9 E x D E V X D
Full
RBF
True
Full
75.2 =
Linear
False
Full
75.6 + 0.8
RBF
False
$ E VEYAN D
Full
76.4 + 0.5
Mix-up
True
N/A
N/A
73.8 + 1.7
Slave-one-domain-o0
True
N/A
N/A
71.3 = 3.5
Random
True
N/A
N/A
70.7 + 2.9
Oracle
True
N/A
N/A
77.6 + 0.5
5.4 MMD analysis
As stated in Section 1, the motivation for cluster-based splits is the notion that increasing the MMD between the training and validation sets increases test domain accuracy. To provide empirical support for this, these two variables are plotted against each other in Figure D for each dataset. As for the clustering, an RBF kernel is used with = 1. Again, the Nyström method is used to estimate the MMD for the Camelyon17 dataset due to its size. Each subplot in Figure D shows a different dataset, and each point in a subplot represents one of the 5 split types (not including the oracle), averaged across the domains and 3 repeats. Standard errors were sufficiently small (i.c., at least order of magnitude smaller than the data range for all subplots) that error bars were not included. The correlation coefficients and associated p-values are also shown. As only monotonic associations are being tested for, Spearman's 's rank correlation p is used.
The leave-one-domain-ont method often produced large outlier values for the MMD, which can be seen in Figure g for all datasets other than Camelyon17 and Humpbacks. The reason for these outliers is unclear. Nonetheless, the correlation for all datasets, with the exception PACS, is positive.
The correlation is clearest for Camelyon17, possibly bocause the dataset large enough that a low-noise estimate of the MMD is possible. The results for this dataset are given in tabular form in Table along with additional ablation study comparing the different clustering constraints (4) and (5) Tt. can be seen that
9

---

Published in Transactions on Machine Learning Research (08/2025)
Camelyon17
Humpbacks
SVIRO
98.6
78
87
77.5
86
85
1=10
000016
76.5
0.00833
- 0.9
98.2
p
p=
Test accuracy
76
Split type
0.2
0.2
0.4
Random
x10*
Leave one
inear k-means
VLCS
PACS
Terra Incognita
k-means
84
Mix-up
76
p
0.233
83
p 0.0833
74
82
40
72
81
p==00.6
38
p m.035
70
x10
x10
x10
MMD(Train, Val)
Figure
The MMD between the training and validation sets versus test domain accuracy, by dataset.
Table 3: The MMD between the training and validation sets versus test domain accuracy by split type for Camelyon17.
Split type (Constraint)
Accuracy (%)
MMD x1000
Random
0.0003
Mix-up
1
5.4
Leave-one-tomain-ont
+
14.3 0.8
Linear k-means
86.5 + 1.3
97.0 6.2
RBF kernel k-means
85.6 + 1.3
183.1 + 2.0
Linear k-means
87.2 + 0.2
108.3 t 2.6
RBF kernel k-means
87.3 + 0.7
206.0 = 3.4
the additional constraint on the clustering (i.e., taking E y D) reduces the optimised objective function value (as would be expected), and this also corresponds to a reduction in test domain model accuracy.
The evidence this section supports the proposition that the validation split should be allempting maximise the MMD between the training and validation sets, and that this is more effective than using the same metadata-baad splitting rule the test split, the leave-one-domsin-out split intends do.
5.5 Assessing the effect of the changing training distribution
As evidenced in 33, the use of non-random data splitting introduces a confounding variable to the exper- iments: as both the training and validation distributions are dependent on the split, it is possible the test accuracy being influenced by the model training, as well as the validation. To support the claim that cluster-based splitting results in more generalisable model selection, it is necessary to decouple the effects of these interventions, and show that is indeed the improved model validation, and not the training, that is improving test accuracy.
As non-random data splitting inherently changes the training distribution, the effect of the model selection cannot be isolated from that of the training. However, the inverse that is, varying the training distribution by changing the split type, but keeping the validation set constant can be achieved, if the oracle selection criterion is used.
If the test accuracy remains constant across split types, this would be sufficient verify the hypothesis that the performance improvements due to the model selection, without having to make any additional
10

---

Published in Transactions on Machine Learning Research (08/2025)
assumptions. If the accuracy is lower than the random split, the hypothesis can still be confirmed (which then implies the model selection is additionally compensating for this reduction), as long as the effects of training and validation robustness on test accuracy are assumed to be additive (i.e., any interaction effects are minor). If anything, the non-random splits would be expected to underperform, since coverage of the overall data distribution by the training set being reduced and this would be expected to be detrimental to generalisation power. The results are given Table
Table 4: Test accuracy using the oracle criterion, for training sets induced by the different split types.
DG experiments
UDA experiments
Split type
Camelyon17
Humpbacks
SVIRO
VLCS
PACS
TerraInc
Average
Random
88.3 0.6
85.4 + 1.5
99.1 + 0.1
77.6
0.5
84.4 0.4
45.8 1.0
80.1 + 0.3
Leave-one-oomain-oot
85.4 = 0.4
82.1 + L.3
99.0 - 0.1
73.8
+ 0.3
82.2
- 0.5
38.9
- 122
76.9 + 0.3
Linear k-means
88.0 = 0.3
85.4 = 0.8
99.1 = 0.1
77.0
+ 0.3
85.7
= 0.3
47.1 0.9
80.4 + 0.2
RBF kernel k-means
88.6 + 0.6
85.0 + 0.7
99.2 1 0.1
77.9
1: 0.5
85.7
+ 0.3
44.5
i 0.6
80.2 i 0.2
Mix-up
88.3 + 0.6
85.6 + 1.6
99.2 1 0.1
77.3 1: 0.5
84.4 + 0.5
44.6 1 0.9
79.9 1 0.3
These results show that performing a cluster-based split has significant effect the generalisation power of a model when the hyperparameters are being chosen via the oracle criterion. Therefore, it can be concluded that the performance improvements of cluster-based splitting seen in Table 1 come entirely from the model selection, and not from the training. On the other hand, test accuracy for the leave-nne-domain-our split does significantly reduce. This finding may help to explain why metadata-basee splits have been found to underperform random splits in some works (Gulrajani & Lopez-Par (02T)): the increased robust of OOD validation is not enough to counterbalance the reduced robustness of training on fewer domains.
5.6 Convergence behaviour of Algorithm 1
As mentioned previously, the proposed split method similar in principle to the one of Wecker al. (2020) One important inplementational distinction the manner by which the constraints are enforced. We have claimed that the greedy approach of Wecker et al. (2020) is susceptible to local minima and thus tends to converge to inferior solutions. To support this claim, this section presents an experiment to compare the quality of the optima attained by the two methods. We use the official implementation of [eeker et al. 2020,, which uses linear k-means with label-only constraints (4), and configure our method similarly for fair comparison. For each domain in the VLCS dataset, we run both algorithms 20 times with different random initialisations, and that our method has significantly less spread than the one of Wecker al. (2020) (around 1000 times less for Domain 3, 100 times less for Domains 1 and 4, and 1.8 times less for Domain 2), and the mean losses also tend to be lower. These findings suggest that the greedy method is indeed getting trapped in local minima.
0.5. The attained clustering objectives (T, V) are shown Figure 2 It can be seen
Wecker et Ours
x10
x
x10
x10 4
5.08
T
9.35
3.54
5.06
O
1.094
Loss
5.04
9.3
O
3.52
1.092
5.02
9.25
3.5
1.09
Figure 2: The attained loss of our clustering algorithm that of Wecker et al. (2020), on the Humpbacks dataset, by domain.
5.7 Assessing the effect of the number of classes
As can be inferred from the problem formulation, a large number of classes may limit the effectiveness of the method. Specifically, with more classes, the partitioning problem is less flexible due to the increased
11

---

Published in Transactions on Machine Learning Research (08/2025)
number of clustering constraints, which reduces the upper bound the MMD which can be achieved. For example, in an extreme case with only two examples per class, random split (stratified by class) and a clustering-based based split would be equivalent. Thus, this section presents an experiment to analyse the effect of the number of classes on MMD(T,V) V). Specifically, using a Humpbacks subset of 8,000 examples, we generale synthetic class labels using another round of kernel k-means clustering, with a constraint. to ensure oqual class sizes. Then, we perform a kernel k-means data split using the synthotic labels, and the label-only constraints (4). Table 5 shows how the MMDs change as the nunber of classes varies between 1 and 500. We also report the MMDs for two of the baselines: the random split and the metadata-basee split.
Table 5: The MMDs of the kernel k-means clustering data split by number of classes, and two baselines.
Number of classes/ Baselines
MMD(T.V V)
MMD(T.E) E)
MMD(V,E) F)
0.221 = 0.016
0.152 = 0.042
0.148
0.040
2
0.187 + 0.007
0.154 = 0.048
0.129 = 0.048
5
0.177 = 0.007
0.121 = 0.037
0.157
0.034
10
0.161 = 0.006
0.132 = 0.020
0.138 0.021
20
0.159 + 0.005
0.132 = 0.021
0.137 0.019
50
0.145 + 0.002
0.137 = 0.031
0.125
0.031
100
0.138 = 0.004
0.119 = 0.019
0.140 = 0.017
500
0.129 = 0.001
0.123 0.027
0.130 0.027
Random
0.001 + 0.000
0.095
0.064
0.095
0.064
Leave-one-domain-onr
0.052 + 0.034
0.053 = 0.036
0.135 = 0.087
To determine a threshold for the maximum number of classes for which a chustering- based split is still valid, we can consider the range for which MMD(T, V) is higher than MMD(T, E) and MMD(V, E), since this is the assumption made by DRO. We consider both MMD(T, E) and MMD(V, E) since the data split is symmetric (it does not matter which way around T and V are labelled). Note that MMD(T, F)) and MMD(V, F) also tend to reduce with MMD(T, V) due to the triangle inequality. In this case, it can be seen that this threshold occurs at around 100 classes. Although, it is noted that even for larger class numbers, the clustering- based split still yields significantly higher MMD(T, V) than both baselines.
6 Limitations
As with all robust optimisation, "no free lunch" theorem (Wolpert & Macready 1997) applies to model selection as well: using a worst-case split loses the guarantce that the selected model will be optimal in the nominal (ID) case. no domain shift is expected, an ID validation set is clearly preferable; in general, robust strategies may be inappropriate the uncertainty minimal, so the nature of the problem should be considered prior to choosing a methodology. Additionally, if unlabelled test domain data is available at split time, this should also be factored in when choosing the validation set (e.g., using the method of Napoli & White (2025)). However, this is not applicable if the hyperparameters need to be decided before the test domain is accessible, e.g., for test-time adaptation, and our experiments show that the worst-case split is still valid in the UDA case. The motivations and considerations for using worst-case optimisation are discussed thoroughly in Sagawa et al. 2019 It is also noted that in the nominal case, the favourable training distribution will be the dominating factor on test accuracy compared to the model selection, and so sub-optimality of the selection method will be less of an issue.
On evaluation, using 7 1 for the RBF kernel bandwidth was not appropriate choice, especially in the high-dimensional case. A more practical option would be to use 7 = 1/dr, where dp is the dimensionality of the clustering features (Liu et al. 2020
Dataset and training algorithm choices for these experiments were biased towards combinations with larger performance gaps between the random split and oracle critcria. This was necessary to properly validate the method: it would be impossible to see any significant signs of improvement the random split and oracle criterion were within margin of error. Although this does inevitably limit the scope of the experiments, it
12

---

Published in Transactions on Machine Learning Research (08/2025)
by no means invalidates the results: cluster-based splits do outperform random and metadata-based splits, where such on improvement. is possible.
As stated in Section | major advantage of the clustering- based validation split is that it is not dependent on domain metadata. However, even with the domain constraint removed or with irtradata-independeer training algorithms (e.g,, empirical risk minimisation), the specific experimental setup in this paper precludes the ability to test this method in truly metadata-free settings. Due to the fundamental design of the DomainBed framework, domain metadata can still be leveraged (both implicitly and explicitly) through several avenues. For example, models are still trained on Noomain-balancee minibatches of data, and validation accuracy is averaged over the validation domains. The datascts themselves may also be unrcalistically domain-balanced to begin with. However, the authors believe that these influences are mild enough that the overall trends observed in these experiments can reasonably be expected to hold in metadata- treee settings as well.
The experiments would ideally be repeated across range of model architectures to reflect the differences in generalisation power of aager/newer models. However, it was necessary to restrict the experiments in this paper to single, smaller model (ResNet-18) due to the high computational costs involved in developing and comparing model selection criteria.
7 Conclusion
This paper presented a method for model selection under domain shift, where the baiiiing-valdationn split is performed using a constrained kernel k-means clustering algorithm. In addition lo outperforming traditional methods, this approach is grounded by an observed strong correlation between the MMD between the training and validation sets, and test domain accuracy.
The algorithm is not parameter-free; future work could include data-driven method for selecting these parameters, for example with additional layer of meta-tuning mneta-tunng. The algorithm also trivially be extended to k-fold cross-validation by increasing the number of clusters. Finally, an investigation into the use of prior knowledge-informed feature extraction (see Appendix A would be of interest.
8 Acknowledgements
This work was supported by grants from BAE Systems and the Engineering and Physical Sciences Re- search Council. The authors acknowledge the use of the IRIDIS High Performance Computing Facility, and associated support services at the University of Southampton, the completion of this work.
References
David Arthur and Sergei Vassilvitskii. How slow is the k-means method? Proceedings of the Annuaa Symposium Computational Geometry, pp. 144-153, 2006. doi: 10.1114/1137856.1 1377117776611377888 URL https:://d.. acm. org/doi/
pdf/10. 11t.111111/178666 1111//1177785661118888888 1137880
David Arthur, Bodo Manthey, and Heiko Röglin. k-Means has Polynomial Smoothed Complexity. Proceedings Avouual IEEE Sympusium on Foundations of Compuler Science, FOCS, pp. 405 414, 4 2009. ISSN 02725428. doi: 10. D0.109/FOOCS.209.11 URL lattps://arxxt www.eriwwvooggg.///oeeeea org/pdt/ 1113
Péter Bándi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Svitlana Zinger, Keisuke Fukuta, Daisuke Komura, Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng, Jeppe Thagaard, Anders B. Dahl, Huangjing Lin, Hao Chen, Ludwig Jacobsson, Martin Hedlund, Melih Çetin, Eren Halici, Hunter Jackson, Richard Chen, Fabian Both, Jörg Franke, Heidi Kusters-Vant Vandevelde, Willem Vreuls, Peter Bult, Bram Van Ginneken, Jeroen Van Der Laak, and Geert Litjens. From Delection of Individual Metastases Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challeuge. IEEE transactions on medical imaging, 38(2):550 560, 2 2019. ISSN 1558-254X. doi: 10. 10.1109/TM1.200 TTM1/ 28670....000.. URL https::
7/pubmed. ncbi. www..nh...aammmmmmmmmmmmmaaaa..a gov3301660//
13

---

Published in Transactions on Machine Learning Research (08/2025)
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in Terra Incognita. ECCV, 11220 LNCS:472 489, 7 2018. ISSN 16113349. URL https://aaxii Sttp:://araiv.orr/abb/////e
Shai Ben-David, John Blitzer, Koby Crammerr and Fernando Pereira. Analysis of Represenlations for Domain Adaptation. NewrIPS, 19, 2006.
Shai Ben-David, John Blitzer, Koby Crammerr, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A
theory of learning from different domains, Machine Learning, 79(1-1)151 10 2010. 18SN 15730565. doi: 10. URL
1007//1009--000555555-//111
P Bradley, Bennett, and A Demíriz. Constrained K-Means Clustering. Technical Report MSR-
TR-2000-65, Microsoft Research, 5 2000. URL httpp://ww..ncrroofft..co
Radha Chitta, Rong Jin, Timothy C. Havens, and Anil K. Jain. Scalable Kernel Clustering: Approximate Kernel k-means, arXiv, 2 2014. URL
art.///b//4222.33499 3849v1
Renato Cordeiro de Amorim and Vladimir Makarenkov. On k-means iterations and Gaussian clusters. Neu-
rocomputing, 553: 126:17,547, 10 2023. ISSN 0925-2312. doi: 10. 1016/..
URL https:
Steve Dias Da Cruz, Oliver Wasenmüller, Hans-Peter Beise, Thomas Stifter, and Didier Stricker. SVTRO: Synthetic Vehicle Interior Rear Seat Occupancy Dataset and Benchmark. In JEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013. doi: 10.1109/ICCV CC000///CV.0011.20.. 2003.208..
Guilherme França, Maria L. Rizzo, and Joshua T. Vogelstein. Kernel G-Groups via Hartigan's Method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 411224:441144425 2017. doi: 10.1109/TPAMI. 2020.2998120. URL http::/arxiv..o/ wrg./bggg/a..................... awt//ab/...11/////// 11100 09859
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain--Advesaarial Training of Neural Networks. JMLR, 2015. ISSN 21916594.
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimolo, and Percy Liang. Out-of-Distribution Robustness via Targeted Augmentations. ICML, 2023.
Robert Geirhos, Claudio Michaelis, Felix A. Wichmann, Patricia Rubisch, Matthias Bethge, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. TCLR, 2018. URL htR.ttttstt///// http://////xxxv..///a/oa///
12231v3
Xiang Gu, Jian Sun, and Zongben Xu, Adversarial data splitting for domain generalization. Science China Information Sciences, 521110 2023. ISSN 1869-1919. doi: 0..1007/111443--22--8855 URL httttpp:///o...rg///..ot 61 611432-022-3857-B
Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. ICLR, 2021.
Gurobi Optimization LLC. Gurobi Optimizer Reference Manual, 2023. URL htttps://uus.gurrtt .cm
1. Heller and C.B. Tompkins, An Extension of Theorem of Dantzig's. Linear Inequalities and Related Systems, 1956.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR, 3 2019. URL https://arzziv.org//ae
12261v1
Dan Hendrycks,, Kevin Zhao, Steven Basart, Jacob Sleinhardt, and Dawn Song. Natural Adversarial Examples. Proceedings of the IEEE Compuler Sociely Conference Compuler Vision and Pallern Recognition, pp. 15257 15266, 7 2019. ISSN 10636919. doi: 2w.110//CVPRR4443.22200...0 URL 07174v4
wwt.oogg/////.eoeooooooooo abg/9979
14

---

Published in Transactions on Machine Learning Research (08/2025)
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasumaga, Richard Lanas Phillips, Irema Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Aushul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the- Wild Distribution Shifts. ICML, 2021.
Da Li, Yongxin Yang, Yi Zhe Song, and Timothy M. Hospedales. Deeper, Broader and Artier Domain Generalization. Proceedings of the IEEE International Conference on Computer Vision, pp. 5543 5551, 10 2017. ISSN 15505499. doi: 10.1109 Www11999/CV.... /01////C..211...1 2017.591. URL https://arxii.. org.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others. CVPR, 12 2022. URL
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and D. J. Sutherland. Learning Deep Kernels for Non-Parametric Two-Sample Tests. ICML, pp. 6272 6282, 2 2020.
Wang Lu, Jindong Wang, Yidong Wang, and Xing Xie. Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution. Proceedings of The KDD'23 Workshop on Causal Discovery, Prediction and Decision, 218:75-97, 11 2023. URL https::/pprcceeddngg/ mlr. www.eprrsss/vvv press/v218/1u23.. html
Boyang Lyu, Thuan Nguyen, Matthias Scheutz, Prakash Ishwar, and Shuchin Aeron. A principled approach to model validation in domain generalization. TCASSP, 2023.
Toshihiko Matsuura and Tatsuya Harada. Domain Generalization Using a Mixture of Multiple Latent Domains. 11 2020. ISSN 2159-5399. doi: /at///aaaaaa........ URL httpps://axxiv.org///
AAAL pp.
1911.07661v2
Andrea Napoli and Paul White. Unsupervised Domain Adaptation for the DataseDataset Detection of Humpback Whale Calls. DCASE, 2023.
Andrea Napoli and Paul White. Diversity- Based Sampling for Imbalanced Domain Adaptation. EUSIPCO, 2024.
Andrea Napoli and Paul White. Unsupervised Domain Adaptation Via Data Pruning. ICASSP, 2025. doi: 10.1109/
ICASSP11960.2000 202...2002....109900 URL hctpp://arrxv..or///a//////// 12076
Louis Ohl, Pierre-Alexandi Mattei, Mickaël Leclerog, Arnaud Droit, and Frédéric Precioso. Kernel KMeans clustering splits for end-to-end unsupervised decision trees arXiv, 2 2024. URL httpe://arxiv. http:///wwi....r///////2//222 12232
Stephen R. Pfohl, Haoran Zhang, Yizhe Xu, Agata Foryciarz, Marzyeh Ghassemi, and Nigam H. Shah. A comparison of approaches to improve worst-case predictive model performance over patient subpopulations. Scientific Reports, 12(1))1-18, 2 2022. ISSN 2042-2322. doi: 10.1038/s41598--000 9411599100222-0767777 URL httpps://ww. www.:w/.atuue...mm cow/coo//aaaaa articces/ 941668-022-07167-7
Hamed Rahimian and Sanjay Mehrotra. Distributionnally Robust Optinization: A Review. arXin, 3, 2019. doi: 10.5802/ojmn.15. URL http:://aaxiv.or////// wwtp://////...//////33
levgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR, 11 2019. URL https://aarzit http://b//111....98
Anders Sogaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. We Need To Talk About Random Splits. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: doi: (0.18653/11/2221.ea main. 156. URL https:://ccaantholloe org/
Main Volume, pp. 1823-1832, 4 2021.
1001.eac1-main.155
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Gert R Lanckriet, and Bernhard Schölkopf. Kernel Choice and Classifiability for RKHS Embeddings of Probability Distributions. Advances in Neural Information Processing Systems, 22, 2009.
Baochen Sun and Kate Saenko. Deep CORAL: Correlation Aligament for Deep Domain Adaptation. ECCV, 9915 L.NCS:433445 450, 7 2016. ISSN 16113349. URL hTt.ttttst:////t http:///wwwa..rrre//////////////
15

---

Published in Transactions on Machine Learning Research (08/2025)
Yoar Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On Calibration and Out-of-Domain Generalization. NewrIPS, 34:2215 2227, 12 2021.
Da Wang, Lin Li, Wei Wei, Qixian Yu, Jianye Hao, and Jiye Liang. Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting. ICML, pp- 50822-50839, 7 2024. ISSN 2640-3498. URL https://
proceedings mlr. (aress/2355/aang4a)] htmm
Hanna Wecker, Annemarie Friedrich, and Heike Adel. ClusterDataSplit: Exploring challenging clustering-based data splits for model performance evaluation. Proceedings of Evaluation and Comparison of NLP Systems, 2020.
David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE Transactions on Buolutionary Computation, 1(1):77 82, 1997. ISSN 1089778X. doi: 10.1109/1235.85588
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards Theoretical Framework of Out-of-Distriuution Generalization. NeurIPS, 11 2021.
16

---

Published in Transactions on Machine Learning Research (08/2025)
A Prior Knowledge-Informed Feature Extraction
Clusterin--bseed splits may be rendered more effective the features used for clustering are informed by domain knowledge on the underlying causes of the domain shift. In this section, we discuss various appli- cations where domain knowledge may be available and propose relevant features which could leverage this information.
In visual tasks involving stylistic shifts, style features Matsuura & Harada 2020
CLIP-derived tokens
can explicitly captare these differences. In histopathology, tumour detection is affected by staining variations across tissue slides, so colour- based features such as hue histograms or colour deconvolution can be used. In wildlife monitoring, shifts due to habitat or camera placement can be isolated by using segmentation to extract backgrounds. Visual changes such as lighting or colour tone shifts can be captured through metrics like brightness, contrast, and colour temperature. In cases where image resolution or fidelity varies, edge density, compression downsampling artefacts, or image quality scores may provide useful indicators. Audio tasks can suffer from variations in recording equipment conditions; features such spectral descriptors, microphone gain, and SNR estimates can help characterise these differences. Or, for speech recognition, speaker differences can be captured using accent classifiers, phonetic parameters, or word rate.
B Derivation of equation
11
Based on the law of total variance, a fundamental theorem in ANOVA that the total sum of squares of a set of points can be decomposed into the sum of squares within each cluster and the sum of squares of each cluster centroid to the overall centroid (weighted by cluster size):
SSq(Sx) = SSq(Tx) + (Vx) + [Txlllm,, - MBs 12 + [VxIllee -P
Next, using the fact that
[Txlp, |Vx|urr
Hps
[x + +VV
have
1M I
Me,
(4p 112c
PPT
[Txl + +
12
Mer (1Txl |Vx1) | HPr Iyev
2
[Tx + Vxx
H
VXI(UPT v
[Txl + [xx]
VV 12
|| - Me
(1Zxl + [x
Similarly,
|x
|PT 14p 12 ([Tx + [xx]2 1I 1
Substituting and simplifying:
Vx + Vx
SSq(Sx) = SSq(Txx) SSq(Vx)
|| MFv 112
[[xx] [Vx))²
|Tx||VXl
SSq(Tx) + SSq(Vx)+ +
like, - -ee -Meel
[Txl + [Vx
Finally, rearranging:
[xxl + |Vx|
|ppp 11
(SSq(Sx) - SSQq(Tx) - -SSq(Vx)),
[Tx [V
required.
17

---

Published in Transactions on Machine Learning Research (08/202))
C
Additional training and hyperparameter details
Table 6: General parameter values and training details for the experiments.
Experimental parameter
Value
Hyperparameter random search size
10
Number of trials
Holdout fraction
0.2
UDA holdout fraction
0.5
Number of training steps
3000
Gaussian kernel bandwidth
Finetuning iterations before split Nyström subset size (if applicable) Architocture
3000
2000
ResNet-18
Class balanced
True
Table 7: Classification pipeline of the Humpbacks dataset. This follows the pipeline in Napoli & White
2023
Step number
Step detail
Acoustic front-end
Resample to 10 kHz
Mel-scale filter bank with 64 filters
Short-time Fourier transform with 100 ms FFT window, 50% overlap
Per-channel energy normalisation
Split into 3.92 (128 pixel) analysis frames with 50% overlap
CNN
Conv2D (nodcs=16, kernel=3x3, stride=2, activation=ReLU) Conv2D (nodes=16; 16, kernel=3x3, stride=2, aclivation=RRelU)
Conv2D (nodes=16, 16, kernel=3x3, stride=2, activation=RRLU)
4
Conv2D (nodes=16 16, kernel-3x3, stride-2, activation-RelL)
5
Global average-pooling 2D
6
Fully-comnected layer
D
Worst-case accuracy validation
Table 8: Model selection using validation accuracy of the worst performing domain. Note that this does not apply to the leave-one-domain-ont and oracle criteria, so these values are unchanged from Table [
experiments
UDA experiments
Split type
Camelyon
Humpbacks
SVIRO
VLCS
PACS
TerraInc
Raw average
Normalised average
Random
76.7
98.2
3.0
75.0
0.0
Leave-one-out
85.6
1.0
77.3
+ 1.7
98.6 + 0.0
71.3
+ 3.5
83.7
= 0.6
37.3
+ 2.5
75.6
+ 0.8
23.3 + 12.1
Linear k-means
87.1
0.2
77.8
+ 1.6
98.6 0.1 0.1
73.7
2.0
83.8
0.6
42.3
1.9
77.2
+ 0.5
49.8
8.8
RBF k-means
87.5
0.7
78.2
+
98.5
76.8
0.6
83.9
38.8
2.5
77.3
0.5
46.8
+ 10.9
Mix-up
83.9 + 0.8
76.9
+ 1.5
98.1
72.9
1.5
81.0 = 0.5
38.3 - 2.7
75.2
- 0.6
2.3
10.7
Oracle
18


Reviews:
[{"review_id": "Qpt92H2Z4J", "reviewer": "Reviewer_UUVW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- While a significant body of work has been done on robust inference under domain shift, hyperparameter selection has beens seemingly quite overlooked, and I think that this paper somehow helps filling this gap (I am not an expert of the domain so I may be missing some references).\n\n- The general story of maximising the MMD to be more robust, and using $k$-means to that end is quite convincing, and well-narrated.\n\n- The experiments are very convincing, I particularly enjoyed the attention to detail in the ablation study and in the attempts at assessing the effects of training versus validation (Section 2.5). I do believe that a simple baseline (stratified splitting) could make them even more convincing though (see weaknesses).\n\n# Weaknesses\n\nMy main concerns are related to the discussion of prior work:\n1. Most importantly, the authors mention that their technique is very similar to the one of Wecker et al. (2020), but the differences and similarities between the two approaches are not clear enough. A detailed discussion (potentially with small experiments, and potentially in the Appendix) of the different motivations/algorithms/contributions of the two papers would be very beneficial. \n2. Less importantly, the main theoretical contribution (the link between MMD and $k$-means) is not really new, and related results were obtained (without the same constraints) by França et al. (2020) and Ohl et al. (2024). Here again, discussing these results would improve the paper.\n\nAs mentioned earlier, I really enjoyed the experiments, but have the small following comment about them:\n\n3. A very natural baseline would be to use stratified splits (to satisfy the same constraints as $k$-means). This can be done using, e.g. the model selection module of Scikit-learn.\n\n# Minor points\n\n- In the MMD experiments, why are there only $5$ datapoints? I thought experiments were repeated?\n\n# Very minor points\n\n- Some (admittedly very well-known) acronyms are not defined: AI, ERM\n- The authors use Theorem 36 from Redko et al. (2020). The paper was updated on arXiv in 2022, but fortunately, Theorem 36 is still Theorem 36. It would be nice to cite the last version (and more importantly to check if it is updated again).\n\n# References mentioned in the paper\n\n- Redko et al., A survey on domain adaptation theory: learning bounds and theoretical guarantees, arXiv preprint 2022\n- Wecker et al., ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation,  Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, 2020\n\n# Additional references\n\n- França et al., Kernel k-Groups via Hartigan’s Method, IEEE TPAMI, 2020\n- Ohl et al., Kernel KMeans clustering splits for end-to-end unsupervised decision trees, arXiv preprint 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of model selection under domain shift. More precisely, they assume that the data at hand comes from several domains, and that the model will be evaluated on yet another domain. The authors argue that standard random training/validation splits will lead to selecting hyperparameters that are not robust to the shift, and suggest using a split based on constrained kernel $k$-means.\n\nThe main motivation for using kernel $k$-means is that this will maximise the maximum mean discrepancy (MMD) between the training and validation sets. The constraints on the kernel $k$-means are here to make sure that the training and validation sets have the same labels/domains proportions.\n\nTheir experiments on several datasets indicate that the $k$-means splits allow for selecting better hyperparameters than standard random splits, as well as other splitting strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe it is very important to act upon the points 1. and 2. above. Adding the baseline of point 3. is less pressing, but would improve the paper.", "broader_impact_concerns": "I do not have particular concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p6Ja3MdjUD", "reviewer": "Reviewer_oGTt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The writing of the paper is fairly clear and easy to follow.\n2. The proposed method is simple but theoretically well motivated for domain adaptation tasks.\n3. The paper provide detailed ablation studies for various scenarios.\n\nWeaknesses\n1. It assumes there are one cluster center for each T and V. How realistic assumption is that? Can the authors provide any argument for this?\n2. Can Algorithm1 be extended to k > 2 case, which seems more realistic given that each of T and V consists of multiple classes, and may consist of multiple domains? If not, why?\n3. I haven’t done any math but at the first glance, it seems like the objective of k > 2 is upper-bounded by $\\Psi(T, V)$ (with modification for Eq.(3) accordingly). Would that justify a variant Algorithm1 with k > 2?  \n4. Every experiment result between RBF kernel k-means and Leave-one-domain-out is within one standard deviation\n5. Limited experiments for larger-scale benchmark datasets such as DomainNet. I wonder  how good the proposed method is over baselines on datasets with a large number of classes like DomainNet given that only one cluster centroid is assumed for each of T and V. \n6. How meaningful is MMD(T, V) in Figure1? According to the figure, MMD $< 5 \\times 10^{-4}$, which seems fairly small to say there is a meaningful distinction between T and V from one algorithm over another.\n7. In addition, can the authors also provide MMD(T, E) and MMD(V, E) along with MMD(T, V) to check if large MMD(T, V) potentially helps better generalizing on E?\n8. I am not sure how meaningful the p-values are for Spearman's rank correlation given there are only 5 samples. If it is, can the authors provide a justification?\n9. The authors said \"The algorithm .... and comes with convergence guarantees.\" But I do not see convergence guarantees in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses model selection under domain shift, covering both unsupervised domain adaptation (UDA) and domain generalization (DG) scenarios. Motivated by principles from distributionally robust optimization (DRO) and domain adaptation theory, the authors propose a novel method for splitting the training data into training (T) and validation (V) sets. The key idea is to maximize the distributional discrepancy between T and V using Maximum Mean Discrepancy (MMD) as a divergence measure. They show that this objective is equivalent to performing kernel k-means clustering with k=2, where the two cluster centroids correspond to the T and V sets. To implement this, the paper introduces a constrained clustering algorithm that uses linear programming to control the size of the splits. It empirically shows that the proposed method outperforms existing splitting strategies across various datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Sections are not divided properly. I suggest to have Section 2. Prior work, Section 3. Preliminary (Theoretical motivation as a subsection), Section 4. Methodology\n2. It would be better if there are equation numbers.\n3. Suggest to use the capital $Y$ instead of $y$ for a restricted set e.g., $S_{X, y=y’} \\rightarrow S_{X, Y=y’} $.\n4. I expect to have answers for weakness 1, 2, 5-7, in the revision.", "broader_impact_concerns": "I do not have any broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CSwBY8txcI", "reviewer": "Reviewer_VAvi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. Clear language. Clear claims. Good illustrations. The idea is simple to follow. The authors address several concerns, such as empirical efficacy and computational needs. The method is general and widely applicable. It is interesting from both a ML perspective, and a statistical one. I appreciate that you report the GPU runtime for training. Well done!\n\nThe concerns I have with the paper are mostly about motivation and discussion. These points are listed below.\n\n1. Equation 1 is provided and suggest that if MMD(T,V) is small, the validation loss will have a low upper bound, and provide little insight in robustness. It is not clear to the reader if this is valid under only some domain generalization problems (e.g. covariate shift) or much more generally. The reference Redko 2020 is new to me, and from a cursory read it was not obvious if it is only valid in covariate shift, or if it is valid more broadly. Equation 1 also depends on the kernel choice. A simple way to increase the MMD without altering the partition (T,V) is to use a different kernel, but obviously we will not get better predictors by changing the bandwidth in the RBF kernel and keeping the partition (T,V). That the influence of kernel choice is not discussed here or otherplaces in the paper is a shortcoming.\n2. In section 1.4, the motivation for balanced classes is too brief. Enforcing balanced classes tends to give overly optimistic results. Why should we have class balance when optimizing for robustness with respect to domain shift?\n3. On the middle of the page 4, the sentence “Moreover, the highest-level feature activations (those closest to the end of the network) are those most highly correlated with the network’s outputs […]” is the first and only mention of networks. Does the method only work with neural nets? If so, this must be said clearly. If not, just remove this confusing remark.\n4. In proposition 2 you state the algorithm converges in finite number of steps. What is the time complexity? Not providing this makes the result feel weak.\n5. In section 2 you start to contrast UDA and DG, and say that you design different experiments for the two settings. However, you never properly discussed and contrasted them in the introduction,  so it is hard for the reader to fully follow this part.\n6. It is surprising that the related works section promotes Søgaard 2020 as a similar method, but you don’t include that in the benchmarking experiment. Why?\n7. I think that the scientifically weakest part of the paper is Figure 2 and the accompanying significance test for rank correlation on pooled data. Since the patterns in Figure 1 are reasonably clear, I don’t see the point of Figure 2. The pooling after normalizing MMD seems statistically improper, so the computed $\\rho = 0.63$ feels very shaky. The impression I get from the writing as it is now, is that you pooled the data to get higher statistical power and smaller p-value. That seems like the wrong thing to do.\n8. Finally, you mention in the conclusion that prior knowledge is important. I think there should be a benchmark where prior knowledge is used in the train-val split. You would strengthen the paper by expanding this discussion, or even include that as a benchmark.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors has developed a method to make train/validation splits that --when used for model selection -- yields better performance out-of-distribution compared to baselines. They start idea that a model that performs well out-of-distribution should be one that performs minimax optimal over some set of evaluation distributions. They argue that one should find a validation set that is as dissimilar to the training set as possible in order to make the the selected model as robust as possible. They quantify the dissimilarity between train and validation, denoted MMD(T,V), develop a linear program that maximizes MMD(T,V), provide evidence about its effectiveness for model selection, and corroborates the heuristic that large MMD(T,V) improves Test performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain equation 1 better. Discuss it in relation to choice of kernel. Please also discuss choice of kernels in other parts of the article. In other settings, such as Covariate Shift by Kernel Mean Matching (Gretton et al 2008), the resuts are highly sensitive the kernel choice. Are your results also sensitive to kernel choice?\n2. Discuss UDA/DG in introduction. Define the terms properly. How are they similar and different? I find this important.\n3. Add discussion on class balance.\n4. Fix the discussion on neural nets (see above). Not critical, but helps readability.\n5. In section 1.4 it becomes clear that the target domain $\\mathcal{Y}$ must be a finite set, i.e. classification problems. State this in the notation section 1.2 instead. Same for the domain $\\mathcal D^S$ and $\\mathcal D^E$ .\n6. Before equation (5), you say ““This is equivalent to performing kernel k-means clustering”.  The reader would be helped much be a clear statement that k=2 (for T and V) at this place in the text. This is not critical but a good service to the reader.\n6. Discuss runtime in relation to proposition 2. This is not critical, but would make the paper more appealing to others who might consider using the method.\n7. Remove Figure 2 and the $\\rho$ in the abstract. I believe that would improve the scientific standard of the article.\n\nTwo typographic points:\n- Remove the abbreviations DG, DRO and UDA from the abstract. They are only defined and never reused.\n- In equation (3), use $\\mathcal{D}^S$, not $\\mathcal D$. This seems to be a typo.", "broader_impact_concerns": "I do not think this work requires a broader impact statement. The compute use is reasonably low. The work seem to provide less fragile machine learning models, which I can only consider a positive impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)