Stratify or Die: Rethinking Data Splits in Image Segmentation
Authors: Naga Venkata Sai Jitin Jami
Thomas Altstidl
Jonas Mueller
Jindong Li
Dario Zanca
Bjoern Eskofier
Heike Leutheuser
Venue: NeurIPS 2025 poster ()
Keywords: segmentation
stratification
genetic algorithm
cross-validation
Primary area: general_machine_learning
TL;DR: We introduce two novel stratification methods, IPS and WDES, to improve dataset splitting in image segmentation, enhancing model evaluation and cross-validation stability.

Abstract: Random splitting of datasets in image segmentation often leads to unrepresentative test sets, resulting in biased evaluations and poor model generalization. While stratified sampling has proven effective for addressing label distribution imbalance in classification tasks, extending these ideas to segmentation remains challenging due to the multi-label structure and class imbalance typically present in such data. Building on existing stratification concepts, we introduce Iterative Pixel Stratification (IPS), a straightforward, label-aware sampling method tailored for segmentation tasks. Additionally, we present Wasserstein-Driven Evolutionary Stratification (WDES), a novel genetic algorithm designed to minimize the Wasserstein distance, thereby optimizing the similarity of label distributions across dataset splits. We prove that WDES is globally optimal given enough generations. Using newly proposed statistical heterogeneity metrics, we evaluate both methods against random sampling and find that WDES consistently produces more representative splits. Applying WDES across diverse segmentation tasks, including street scenes, medical imaging, and satellite imagery, leads to lower performance variance and improved model evaluation. Our results also highlight the particular value of WDES in handling small, imbalanced, and low-diversity datasets, where conventional splitting strategies are most prone to bias.

Stratify or Die: Rethinking Data Splits in Image Segmentation
Naga Venkata Sai Jitin Jamil
Thomas Altstidl'
Jonas Leo Mueller
Jindong Li'
Dario Zanca'
Björn Eskofier!
Heike Leutheuser
Machine Learning and Data Analytics Lab, FAU Erlangen-Nürnberg Germany Ambient Assisted Living and Medical Assistance Systems, Department of Computer Science,
University of Bayreuth, Germany
{jitin. thomas altstdd,, llo. mueller, , jindong. liigg.lijee @iuu......... de {dario. zanca, bjoern. eskofier heike. leutheuser Bewerruunniiuuiuueee
Abstract
Random splitting of datasets in image segmentation often leads to unrepresentative test sets, resulting in biased evaluations and poor model generalization. While stratified sampling has proven effective addressing label distribution imbalance in classification tasks, extending these ideas to segmentation remains challeng- ing due to the multi-label structure and class imbalance typically present in such data. Building existing stratification concepts, we introduce Iterative Pixel Stratification (IPS), a straightforward, label-aware sampling method tailored for segmentation tasks. Additionally, we present Wasserstein-Driven Evolutionary Stratification (WDES), novel genetic algorithm designed to minimize the Wasser- stein distance, thereby optimizing the similarity of label distributions across dataset splits. We prove that WDES is globally optimal given enough generations. Using newly proposed statistical heterogeneity metrics, evaluate both methods against random sampling and find that WDES consistently produces more representative splits. Applying WDES across diverse segmentation tasks, including street scenes, medical imaging, and satellite imagery, leads to lower performance variance and im- proved model evaluation. Our results also highlight the particular value of WDES in handling small, imbalanced, and low-diversity datasets, where conventional splitting strategies most prone bias.
Introduction
Image segmentation involves the assignment of label each pixel in image, with applications in diverse fields such as autonomous driving (TH5), medical imaging [610], and satellite image analysis [1T-15]. In supervised learning setting, splitting the dataset into disjoint subsets for training and validation is common practice. However, following a random splitting strategy can result in class distribution shifts across subsets compared the original dataset. This can lead to biased evaluations, as the test set is typically smaller than the training set and more affected by this phenomenon. Typically, such issues are addressed through stratification to guarantee a consistent distribution of class labels across subsets [16] but this is not straightforward for image segmentation datasets [7421]. The fundamental unit, an image, is not sample with a single class but sample with pixels belonging to many classes. Stratification strategies for similarly complex multi-label datasets [22] could provide interesting solutions that can applied segmentation datasets. Strategies relevant to multi-label datasets have been proposed and investigated in various forms ranging from iterative strategics (23/255, greedy search 26 27]. and genetic algorithms 2hesThese methods do not dircctly apply to the problem of segmentation, multi-label datasets have the complexity of multiple positive
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Original Image
Mask
Fold
Fold
Fold 4
Fold
Ideal
Split
Random Spii
Figure 1: Representative examples of Plane images from PascalVOC [17. The illustration shows an ideal split with balanced class distribution across folds which is not ensured by random splitting. Such balance is essential to avoid bias and ensure reliable model evaluation.
classes for each sample but do not have the added complexity of proportional representation of these classes in the sample. Segmentation datasets present a unique challenge this regard, complicating the task of stratification.
Segmentation datasets are often small due high annotation costs and exhibit imbalanced class distributions due to label diversity. As dataset size decreases, inconsistent class distributions in subsets increase, hindering generalization [29]. This issue is critical in imbalanced datasets, where some classes may be absent from splits, making performance metrics like Fl-score and IoU unmeasurable. Such missing data undermines the reliability of model evaluation, especially K-fold cross-validation 30 where unrepresentative subsets affect performance variance.
Motivated by these challenges. we propose and investigate two stratification strategies: Iterative Pixel Stratification (IPS) and Wasserstein-Dree Driven Evolutionary Stratification (WDES These methods are designed assign samples subsets, folds. for a train-test split scenario (two folds) K-fold tross-validation scenario (K folds), taking into account the label composition of the sample. IPS extends the Iterative Stratification (IS) algorithm proposed literature (3), assigning samples to folds using a greedy approach to meet predefined class presence requirements measured by the number of pixels each fold. prioritizes assigning samples containing rare classes to folds, progressively addressing those with increasing ubiquity. In contrast, WDES employs evolutionary algorithm akin to Evosplit (28), where a population of potential fold assignments (individuals) evaluated simultaneously. The fittest individuals undergo selective crossover and mutations for further refinement. The fundamental distinction between these two methods lies in their approach to fold assignment and evaluation. The iterative nature of IPS confines it to sequential decision-makin process and is limited to a single chain of iterations. Conversely, the genetic algorithm in WDES evaluates multiple solutions independently by maintaining population, enabling it to explore broader solution space. The flexibility WDES allows us to choose a better objective function to be minimized. We choose the Wasserstein distance between the class distribution in the fold and the overall dataset as it indicates the similarity between these distributions. Hence, minimizing the distance reduces dissimilarity between the subsets and the dataset.
evaluate IPS, WDES, and random splitting across five benchmark segmentation datasets. Our evaluation begins by analyzing the consistency of folds generated through repeated experiments. We
'Implementation available at: httt:///tthhoooooooootaaaataattt

---

further assess the impact of targeted stratification versus random splitting by examining the standard deviation of accuracy, Fi-score, and Intersection over Union (IoU) across 10-fold cross-validation. A lower deviation indicates a more reliable assessment of model performance. Empirical results show that WDES consistently achieves the highest quality splits. contrast, IPS under-performs relative to WDES, suggesting that sequential strategy may be suboptimal for segmentation tasks. Notably, WDES also demonstrates better consistency over the other two for evaluating model performance on low-entropy datasets, where class distributions are highly imbalanced or concentrated. Overall, our findings highlight the importance stratification tailored to structured outputs and advocate for the adoption of principled splitting methods in segmentation tasks.
2 Related Work
Existing methodologies for stratifying complex datasets trace their origins to Iterative Stratification (IS) by (23). In their work, the authors addressed the intricacies of multi-label datasets, proposing a greedy algorithm that iteratively assigns samples to folds based on the required frequency of positive labels within each fold. Fold quality is evaluated by comparing the Label Distribution (LD) metric. ID the difference between the proportion of samples belonging to a given class within a fold and the corresponding proportion in the overall dataset
The partitioning method based on stratified random sampling (PMBSRS) by [24] employs a similar iterative approach but incorporates pre-sorting step. This method clusters samples with similar label sets before assigning them equally to folds, thereby enhancing the homogeneity of label distribution across folds. Building upon the work of (23), authors of [25] introduced Second Order Iterative Stratification (SOIS), which considers second-ordes label relationships. Instead of focusing on individual labels, SOIS accounts for label pairs to determine the demand within each fold, thereby capturing more complex interactions between labels.
Other strategies approach stratification by starting from an initial assignment and iteratively refining it. The Stratified Sampling (SS) method proposed by (26) pre-assigns samples to folds and evaluates them based on deviations from the ideal label distribution. Samples contributing to the highest deviation subset are shuffled between folds to minimize discrepancies. This approach was designed with large-scale XML datasets in mind [3 A similar split-and-shufft approach was adopted by 27 but from a label-centric perspective. They developed a scoring mechanism to assess fold quality relative to each class, shuffling samples associated with the class exhibiting the poorest quality before recalculating scores. This method demonstrated effectiveness on large Gene Ontology datasets.
Finally, Evosplit, proposed by [28). leverages a genetic algorithm to address the challenges outlined by (3 and 255 This method generates population of fold assignments, evaluates their fitness, performs crossover and mutation, and iterates over generations to arrive at an optimal solution. Fitness determined using Label Distribution (LD) from (23] and Label Pair Distribution (LPD) from (25)
While these methods offer valuable strategies for stratifying multi-label datasets, they do not directly extend to image segmentation. In segmentation, each sample contains dense array of wixel-wise labels rather than a simple set of associated classes, and the proportion of pixels per class varies significantly across images. This makes stratification more complex: it is not enough ensure the presence of a class in a fold; one must also consider how extensively each class is represented at the pixel level. Therefore, segmennatioon-speeiff stratification requires reformulating both the problem and the evaluation criteria. In the next section, we formalize the segmentation stratification task and introduce two new approaches, Iterative Pixel Stratification (IPS) and Wasserstein- Deiserstein-Drrvvn Evolutionary Stratification (WDES), which aim to create representative folds by explicitly accounting for pixel-level label distributions.
3 Stratification algorithms
Consider an image segmentation dataset D = Yn 1 input images and yn the corresponding label masks. Let the total number of pixels across all samples be P. and the number of semantic classes be C. The objective is to partition D into K disjoint subsets, denoted S = (S', (S'......S! 5k) according to target proportion vector (r'..... ...., r*). ,)), where n specifies the desired proportion of samples in fold k.
with N samples, where x,, denotes the
3

---

Finding such partition that preserves the global class distribution across folds is NP-Hard because can be reduced to a multi-war way partition problem [32.. is analogous to dividing multi-set of weighted elements into K subsets with balanced sums; except here, the "weights" are pixel counts per class and the balance across multiple classes simultaneously
Let No be the number of samples in the dataset that contain class and N the required number of such samples in fold k. Similarly, let l" be the total number of pixels in sample n, and Pa the number of pixels of class с in sample n. The total number of pixels of class e in the dataset is Po. and the desired number in fold & is P These quantities are central to the evaluation of fold quality and to guiding the stratification process.
3.1 Similarity measures
We first introduce similarity measures calculated on the subsets to assess the quality of the stratified folds S. These statistical properties of the subsets aim to quantify: (1) how well each fold adhere to the target sample proportions defined by r; (2) how closely the class-wise pixel proportions in each subset match those in the full dataset; (3) the overall similarity between fold-level and global label distributions using distributional distance.
Sample Distribution (SD) With this measure, we aim to assess the deviation of the number of samples in each subset from the required number of samples as defined by the proportion vector For example, if dataset of 10 samples to be equally divided but the subsets contain and samples respectively, the value SD is 1. It is calculated as the average across folds of the deviations from the desired number of samples in each subset. A low SD indicates that the number of samples per fold closely matches the intended proportions.
SD
| - N*
(I)
K
where 2 = ph . is the expected number of samples in fold k.
Pixel Label Distribution (PLD) This measure evaluates how closely the proportion of pixels of each class within each subset matches the proportion of that class in the entire dataset. For each class c, it calculates the absolute difference between the ratio of pixels of that label in each subset k and the ratio of pixels of that label in the whole dataset D. then averages these results across all labels. This measure is inspired by Label Distribution (LD) in [3] Lower PLD values indicate better preservation of class proportions the pixel level.
1 PLD
pk P. -
Pe P-P.
C
K
An additional similarity measure examining class co-occurrence relationships is provided in Ap- pendix H. where we extend this evaluation to capture second-order interactions between classes using the Pixel Label Pair Distribution (PLPD).
Label Wasserstein Distance (LWD) This measure uses the Wasserstein Distance to quantify the dissimilarity between the cumulative class distributions in each fold and that of the entire dataset. By calculating this distance for each subset and averaging the results, LWD provides an indication of how closely the pixel-level class distributions in the subsets resemble that of the original dataset. Let F. and bl denote the cumulative pixel distributions up to class e in the dataset and in fold k respectively:
F.
P.
Fk
pk
i=1

---

Then the Label Wasserstein Distance is defined as:
C ME
1 K k=1
LWD
Fe F*1
To complement the Wasserstein-based similarity assessment, Appendix Heanndixx[]pesseee presents further similarity assessments using alternative distributional measures, namely the Linear-kernel Maximum Mean Discrepancy (L-MMD) and the Linear Energy Distance (L-ED).
3.2 Iterative Pixel Stratification (IPS)
The motivation behind Iterative Pixel Stratification (IPS) stems from Iterative Stratification (IS) [23]. which aims to distribute samples with specific positive labels proportionally across folds. This distribution is guided by two key properties within each fold: the number of samples required and the number of positive label instances required. To achieve this, IS starts by identifying the rarest label among the unassigned samples, determines which fold most requires positive instances of it and assigns samples that have positive instances of this label to that fold. In cases where multiple folds have equal demand, the fold with the fewest total samples prioritized. Once a sample is assigned to a fold, it is removed from the pool of unassigned samples, and the demand for all positive labels associated with that sample reduced in the fold. This iterative process continues, with the demand for specific positive labels decreasing samples are assigned. The process concludes once all samples have been allocated, typically after iterations.
IPS differs from IS in one key way: the label demand in folds is determined by the number of pixels required, not by the positive instances of that label. This also means that when a sample is assigned to fold, the desire for all labels reduced by the number of pixels of every label that sample. As result, IPS emphasizes achieving pixel-level balance over maintaining sample count proportionality. Note that IPS is implemented by adapting Iterative Stratification without additional optimizations thereby serving as a straightforward baseline. The pseudo-code for IPS algorithm is presented in Appendix B An alternative iterative stratification approach is presented in Appendix G
3.3 Wasserstein-Drivve Evolutionary Stratification (WDES)
WDES formulates the stratification problem as an optimization task and solves it using genetic algorithm. Each candidate solution, or individual, represents an assignment of samples to folds. Initially. a population of individuals generated by randomly assigning samples folds according to the target proportion vector r. Each gene (element) in an individual encodes the fold assignment of a specific sample.
The evolution of individuals is driven by crossover and mutation, First, we evaluate fitness using LWD) (Eq. 3 and perform tournament selection [33. Crossover occurs with fixed probability, exchanging gene segments between individuals. A correction step follows to maintain target proportions across folds, reassigning samples if necessary. Mutation swaps assignments of randomly selected pairs, preserving fold proportions and adding diversity. This cycle repeats for a set number of generations, with the fittest individual (i.e., with the lowest LWD) selected as the final stratification.
We use the LWD as the fitness function to measure class distribution similarity. The proportionality requirement is managed through the initial population and crossover design, ensuring the genetic algorithm optimizes class distribution while maintaining proportionality across folds for balanced stratification.
3.4
On the convergence of stratification algorithms
For large multi-label datasets, as the dataset size grows, random splits asymptotically preserve the marginal label frequencies in each fold due to the law of large numbers. We provided mathematical proof this property in Appendix IA.I This property makes random stratification a practical choice for very large datasets, where computational efficiency and simplicity are preferable.
For smaller datasets, where N finite, WDES provides strong guarantees of approaching the optimal stratification the number of generations and population size increase.
5

---

Proposition 3.1 (Convergence Rate of WDES to Empirical Optimum), Let FM.G G C F denote the set of stratifications explored by the WDES algorithm after G generations with a population size M. Then the expected suboptimality of the final stratification SwDes FMM.G satisfies:
E [L(WDE))) LFM So
MG
where Lissa denotes the optimal Wasserstein score in the empirical subset.
Proof. See Appendix A.2
The above results shows that the expected quality of the WDES stratification improves rate of O(1/MG), meaning that as the number of generations and population size increase, the WDES algorithm converges toward the global optimum. This makes WDES a particularly effective choice for small datasets, where the algorithm can afford the additional computational cost required to reach an optimal stratification, providing guarantees of high-quality splits the search progresses.
4 Experiments
We conduct comparative analysis of three stratification strategies: a) random sampling using KFold from the trikit-leand library [34), b) Iterative Pixel Stratification, and c) Wasserstein-Driver Evolutionary Stratification implemented using the deap library [35
To achieve this, we start by testing these algorithms on the datasets outlined in Section 4.1 to calculate the statistical similarity measures (described in Section 3.1). For all the algorithms, we consider the number of pixels belonging to each class the provided mask of every sample to guide the stratification procedure. The applicable parameters used for WDES are outlined in Appendix [ We perform stratification with datasets randomly shuffled five times and average the calculated similarity measures.
Following this, we perform 10-fold cross-validation tests with random splitting and the targeted stratification strategies. For every fold, we train UNet 6 with resnet34 encoder from (36] for 50 epochs to perform segmentation on the images. The training process employs a learning rate of 2e-4 and utilizes the Adam optimizer with DiceLoss. For PascalVOC, CELoss is used instead, with learning rate of 1e-4 and trained for 100 epochs. All experiments are conducted on a single Nvidia A 100 graphics card with 40GB of VRAM, without distributed training, ensuring consistent and comparable results across the different stratification methods. We evaluate the stratification methods by measuring the variance in model performance across folds, to assess whether targeted stratification provides more stable and reliable evaluations.
4.1 Datasets
We select datasets spanning four major application domains: autonomous driving/street scenes, medical images, satellite imagery, and general-purposs datasets. For autonomous driving, we use Cityscapes [21] and Cam Vid [18). In the medical domain, we consider EndoVis201 8, robotic scene segmentation dataset from MICCAI 2018 19. For satellite imagery, we use the LoveDA dataset 220. which focuses on land-cover segmentation in urban and rural locations. For general-purpose segmentation tasks, we include PascalVOC 2012 [17].
To characterize the complexity of each dataset, we define several key properties. Class Cardinality (CC) refers to the average number of classes present in a sample. Class Ubiquity (CU) captures the average number of samples in which each class appears, indicating how frequently each class occurs across the dataset. Average Imbalance Ratio (AIR) the mean of class-wise imbalance ratios, where the imbalance ratio for given class is the ratio of pixels in the most frequent class to those in the class under consideration. Entropy quantifies the diversity in class proportions; higher entropy implies a more balanced distribution of classes, whereas lower entropy indicates imbalance. These metrics provide a nuanced understanding of the datasets' structural diversity and segmentation challenges. The properties of all datasets are summarized in Table sorted by entropy.

---

Table 1: Datasets and properties
Dataset
N Images
N Classes
CC
CU
AIR
Entropy
PascalVOC 11
2,913
21
2.48
344
79
1.82
Cam Vid [88
701
32
17.44
370
7.56E+06
1.94
VidoVis [9]
2,235
12
6
1,117
1,492
2.38
LoveDA 20
2,522
8
4.97
1568
4.5
2.61
CityScapes 21
2,974
35
16.6
1,412
467
3.17
Table 2: Comparison of stratification methods using statistical similarity measures across datasets. WDES yields folds that better reflect the overall distribution, as indicated by PLD and LWD. The advantage narrows for datasets with higher class cardinality and lower ubiquity.
SD
PLD ex10
LWD
Dataset
Method
Mean
Std
Mean
Std
Mean
Std
Random
0.42
0.00
955
137
75.0
2.62
PascalVOC
IPS
10.28
2.46
733
244
53.3
1.20
WDES
0.42
0.00
456
77.9
51.4
1.63
Random
0.00
0.00
63.4
3.73
183
5.08
Camvid
IPS
6.68
1.18
76.6
5.30
209
8.44
WDES
0.00
0.00
36.7
4.47
138
7.84
Random
0.50
0.00
609
81.4
661
35.5
EndoVis
IPS
11.42
1.05
769
12.2
764
75.2
WDES
0.50
0.00
208
23.8
399
10.4
Random
0.32
0.00
1110
109
1080
88.4
LoveDA
IPS
8.06
1.15
1090
207
1100
87.4
WDES
0.32
0.00
377
31.4
635
16.0
Random
0.50
0.00
126
15.4
304
16.5
Cityscapes
IPS
11.54
2.69
148
17.4
324
22.2
WDES
0.50
0.00
67
3.44
217
5.17
Random
1
2.4
2.2
Ranking
IPS
2
2.6
2.8
WDES
1
GEna6vis
0.6
0.6
PLD
0.5
citysrapes
0.5
Chyccapes
ComMid 16
CamVil
10
12
14
400
600
800
1000
1200
1400
1600
0.40
LoveDA & @EndoVis
0.40
Ensowis
LoveDA
LWDD
0.35
PascavoC
Cityscapes
PascaMoC
Chyscapes
0.25
CamMid
0.25
Camamin
12
14
400
600
800
1000
1200
1400
1800
Cardinality
Ubiquity
Figure 2: Relative improvement of WDES over random splitting in PLD and LWD. Performance gap decreases with increasing class cardinality, but WDES maintains an edge in datasets with high class ubiquity.

---

Table 3: Average segmentation performance of UNet across stratification methods in 10-fold cross- validation. Standard deviations in Accuracy, FI, and IoU are used assess consistency. WDES performs best in low-entropy datasets, while random splitting becomes competitive as entropy increases.
Accuracy
IoU
Dataset
Method
Mean
Std (x10-3)
Mean
Std (x10->)
Mean
Std (x10
Random
0.76
20.6
0.58
32.5
0.44
34.3
PascalVOC
IPS
0.76
14.5
0.58
30.3
0.44
31.6
WDES
0.75
11.0
0.57
24.0
0.43
24.2
Random
0.94
0.68
0.91
1.16
0.89
1.17
Camvid
IPS
0.94
0.74
0.91
1.35
0.89
1.27
WDES
0.94
0.67
0.91
1.09
0.89
1.06
Random
0.94
19.4
0.86
27.0
0.80
28.1
EndoVis
IPS
0.94
14.1
0,87
29.3
0.79
30.7
WDES
0.94
13.0
0.85
17.9
0.79
18.9
Random
0.88
6.63
0.80
6.39
0.69
8.38
LoveDA
IPS 0.88
7.39
0.80
9.29
0.69
10.7
WDES
0.88
7.49
0.80
9.86
0.69
11.0
Random 0.79
11.2
0.61
9.12
0.50
8.64
Cityscapes
IPS 0.79
10.3
0.61
14.9
0.50
15.2
WDES
0.79
11.2
0.61
14.4
0.50
12.8
Random
2.4
1.8
1.8
Ranking
IPS
2
2.6
2.6
WDES
1.6
1.6
1.6
5 Results and Discussion
5.1 Distribution of Labels and Samples
Table presents the statistical similarity measures calculated for the subsets generated across the five datasets. For each measure and dataset, we report the mean over five runs, highlighting the best-performing method in bold. An overall average rank is also computed for each method by assigning rank 3 to the method with the lowest mean and rank to the one with the highest.
Our findings indicate that WDES consistently achieves superior performance in terms of PLD and LWD across all datasets, underscoring effectiveness preserving label proportions and label distributions within the stratified subsets. Although IPS explicitly designed to address proportionality in class presence, WDES outperforms not only in LWD (which it directly optimizes) but also in PLD. This reinforces the perspective that stratification for image segmentation tasks benefits more from minimizing distributional dissimilarity than from simply ensuring class proportions.
WDES also exhibits favorable SD scores, which follow directly from its design. Because WDES explicitly enforces predefined sample proportions across folds, it naturally results in balanced sample counts. This behavior is similarly observed in random splitting, which allocates samples proportionally across folds without replacement.
Another noteworthy pattern emerges in relation to dataset complexity. As class cardinality increases, the performance gap between WDES and random splitting diminishes for both PLD and LWD. This trend suggests that when more classes are present per sample, the benefits of targeted stratification decrease due to the inherent balancing effect of high cardinality. In contrast, as class ubiquity increases, indicating that classes are more commonly present across the dataset, the superiority of WDES becomes more pronounced. This expected, as higher ubiquity leads to more overlap between classes and samples, thereby amplifying the benefits of optinization-based stratification. These trends are illustrated in Figure [] which presents the changes in PL.D and LWD differences relative to class cardinality and ubiquity using trend lines.

---

Accurecy
Rervertt
Camind
Chycass
Prevent
Camod
Chyccaae
Phyatenc
Camod
nelaaing Friray
Rchouing Feroyy
scnocin Sroeey
Figure 3: Stratification method rankings by performance variance across datasets. WDES ranks highest in low-entropy settings; random splitting improves with higher entropy, reducing the benefit of targeted stratification.
Table 4: Average segmentation performance of UNet for underrepresented classes (bicycle, boat, and polled plant) in PascalVOC under 10-fold cross-validation. Standard deviations in Accuracy, F1 Score, and IoU are reported to assess consistency. WDES yields the most stable results across all three rare classes, indicating improved fold balance for low-frequency categories.
Accuracy
F1
loU
Class
Method
Mean
Std
Mean
Std
Mean
Std
Random
0.62
0,04
0.31
0.08
0.19
0.06
Bicycle
IPS
0.62
0.04
0.31
0.09
0.19
0.06
WDES
0.62
0.03
0.31
0.07
0.19
0.05
Random
0.72
0.05
0.53
0.09
0.37
0.08
Boat
IPS
0.70
0.06
0.49
0.14
0.33
0.12
WDES
0.72
0.04
0.55
0.08
0.38
0.08
Random
0.71
0.08
0.47
0.16
0.32
0.14
Potted Plant
IPS
0.71
0,09
0.51
0.16
0.36
0.14
WDES
0.70
0.07
0.47
0.12
0.31
0.10
5.2 Variance of Performance
Table 3 presents the results training a UNet model on each dataset using 10-fold cross- voos-validation. For each fold, evaluate performance using Accuracy, Fl Score, and Intersection over Union (IoU), and report the average across the ten experiments. To highlight the robustness of each stratification strategy, identify the method with the lowest standard deviation for each metrie. Rankings assigned from to 3, with 1 indicating the lowest variance in performance and thus better fold consistency.
Our analysis shows that WDES achieves the best average rank standard deviation across Accuracy, F1 Score, and loU. This indicates that WDES produces splits that are more consistent with each other and better reflect the overall data distribution, leading to more stable model evaluation. While cannot distinguish whether the reduced variance stems from the training test set, the smaller size of the test set suggests is more affected by sampling variability. Thus, the improvements in test set consistency are likely the main driver of the reduced standard deviation. This makes the average cross-vaus coss-valldation score under WDES a more reliable estimate of model's 's true generalization performance. We leave empirical validation of this to future work.
To further examine whether these improvements extend to underrepresented categories, we analyzed the segmentation performance for three rare classes in PoCc bicycle, boat, and potted plant As shown Table WDES consistently yields lower standard deviation in Accuracy, Score, and IoU for these classes compared to random and IPS-based splits. This mirrors the global trend

---

observed in our main results, suggesting that WDES not only improves stability across folds overall but also enhances evaluation consistency for rare classes in low-entropy datasets.
As we move from low-entropy to high-entropy datasets, where label distributions are inherently more balanced and diverse, the advantage of WDES diminishes. The performance of random splitting improves and can even outperform stratified approaches. In these cases, the class distributions are already relatively uniform across samples, making targeted stratification less critical. This trend is reflected in the increasing rank of random splitting and the decline in WDES performance with increasing entropy, as illustrated in Figure B
While IPS was designed to be label-aware stratification method, it ranks lowest on average across accuracy, F1, and IoU deviation. A potential reason lies in its single-pass, greedy allocation strategy, which reduces label demands a fold as soon as a sample is assigned. This mechanism introduces path dependency that prevents the algorithm from reassessing earlier decisions accounting for broader distributional goals. Furthermore, although IPS considers pixel-level proportions, its strategy is primarily focused on satisfying per-label pixel quotas without evaluating the resulting inter-label distribution in each fold. As result, by prioritizing rare labels first and assigning samples based solely on immediate label demand, it may overlook more nuanced strategies for dividing the dataset, such as balancing co-occurring label groups, preserving contextual diversity, or accounting for interdependencies between frequent and infrequent classes.
Conclusion and Limitations
In this paper, we introduced WDES, stratification method for image segmentation tasks based genetic algorithm that minimizes the Wasserstein distance between label distributions. We provide theoretical guarantee (Appendix A.2) that WDES converges to globally optimal stratification given sufficient generations. We empirically compared its performance against iterative (IPS) and random sampling across several datasets and evaluation criteria. WDES outperforms both iterative and random methods, achieving the best average rank in accuracy, Fl score, and IoU variance in segmentation tasks, particularly for low-entropy datasets. Conversely, random stratification shows lower variance in high-entropy datasets, indicating that the benefits of targeted stratification diminish as class distributions become more uniform.
We emphasize the importance of stratification in cross-validation and train-test splits for image segmentation, particularly to avoid underrepresenting certain test sets and biasing the evaluation. This issue especially critical for rare or underrepresented classes and when working with small datasets. Because final performance scores in cross-valiattion of fold-wise means, the effects of imbalanced splits are not mitigated by the repeated nature of the process. While WDES improves upon simpler strategies, it is not universal solution. Its reliance on minimizing Wasserstein distance assumes that distributional similarity alone ensures balanced splits, potentially overlooking other factors such spatial structure class co-occurrence. Future work should explore alternative similarity measures and more comprehensive modeling of dataset composition improve stratification in segmentation tasks.
typically computed the mean
10

---

References
[1] Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D Collins, Ekin D Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation. In Computer Vision-Eoe ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part IX 16, pages 695-714. Springer, 2020.
[2] Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv preprint 20X.::2005.10821., 2020.
[3] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang. al. Deep high resouuesotttion representation learning for visual recognition. IEEE transactions pattern analysis and machine intelligence, 43(10): -364,, 2020.
[4] Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 752-761. IEEE, 2023.
[5] Serdar Erişen. Sernetforrme: Segmentation by efficient-resnet with attention-boosting gates and attention-fusion networks. In 2024 IEEE International Conference Computer Vision and Machine Intelligence (CVMI), pages 1-6. IEEE, 2024.
[6] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICC MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[7] Debesh Jha, Michael Riegler, Dag Johansen, Pål Halvorsen, and Håvard D Johansen. Doubleu- net: A deep convolutional neural network for medical image segmentation. In 2020 JEEE 33rd International symposium on computer-based medical systems (CBMS), pages 558-564. IEEE, 2020.
[8] Razvan-Gabris Dumitru, Darius Peteleaza, and Catalin Craciun. Using duck-net for polyp image segmentation. Scientific reports, 13(1):9803, 2023.
[9] Meng Wei, Charlie Budd, Luis C Garcia-Peraza Herrzza, Reuben Dorent, Miaojing Shi, and Tom Vercauteren. Segmatch: A semi- sumi-super learning method for surgical instrument segmentation. arXiv preprint arXiv:2308.0522. 052332 2023.
[10] Ioannis A Vezakis, Konstantinos Georgas, Dimitrios Fotiadis, and George K Matsopoulos. Effisegnet: Gastrointestinal polyp segmentation through a pre-trained efficientnet- based net- work with simplified decoder. In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 1-4. IEEE, 2024.
[11] Libo Wang, Rui Li, Chenxi Duan, Ce Zhang, Xiaoliang Meng, and Shenghui Fang. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
[12] Taisei Hanyu, Kashu Yamazaki, Minh Tran, Roy A McCann, Haitao Liao, Chase Rainwater, Meredith Adkins, Jackson Cothren, and Ngan Le. Aerialformer: Multi-resolution transformer for aerial image segmentation. Remote Sensing, 16(16):2930, 2024.
[13] Gyutae Hwang, Jiwoo Jeong, and Sang Jun Lee. Sfa-net: Semantic feature adjustment network for remote sensing image segmentation. Remote Sensing, 16(17):3278, 2024.
[14] Ivica Dimitrovski, Vlatko Spasev, Suzana Loshkovska, and Ivan Kitanovski. U-net ensemble for enhanced semantic segmentation in remote sensing imagery. Remote Sensing, 16(12):2077,
2024.
11

---

[15] Libo Wang, Rui Li, Ce Zhang, Shenghui Fang, Chenxi Duan. Xiaoliang Meng, and Peter M Atkinson. Unetformer: unet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery. ISPRS Journal of Photogrammelry and Remote Sensing, 190: 196-214, 2022.
[16] Max Kuhn, Kjell Johnson, et al. Applied predictive modeling, volume 26. Springer, 2013.
[17] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.paseal. Sewwork.rrgchallleess/OOOCCC / http:///222/wwwwohooooooee 2/workshop/index html, 2012.
[18] Gabriel Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern recognition letters, 30(2):88-97, 2009.
[19] Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammaddi Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge. arXiv preprint arXiv:2001.11103 2020.
[20] Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. Loveda: remote sensing land-cover dataset for domain adaptive semantic segmentation. In Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL
www.rps.cc/pppppeepppeee haapppprrrfll____ www.ffileaaaaeeee hilee//pper///// 2021 220//f11|44720000000000000111113 Paper round2. pdf
[21] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[22] Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. Mining multi-label data. Data mining and knowledge discovery handbook, pages 667-685, 2010.
[23] Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis Vlahavas. On the stratification of multi-label data. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011. Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pages 145-158. Springer, 2011.
[24] Francisco Charte, Antonio Rivera, María José del Jesus, and Francisco Herrera. On the impact of dataset complexity and sampling strategy in multilabel classifiers performance. In Hybrid Artificial Intelligent Systems: 11th International Conference, HAIS 2016, Seville, Spain, April 18-20, 2016, Proceedings 11, pages 500-511. Springer, Spriger. 2016.
[25] Piotr Szymański and Tomasz Kajdanowiez. A network perspective stratification of multi- label data. In First International Workshop on Learning with Imbalanced Domains: Theory and Applications, pages 22-35. PMLR, 2017.
[26] Maximillian Merrillees and Lan Du. Stratified sampling for extreme multi-label data. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 334-345. Springer, 2021.
[27] Henri Tiittanen, Liisa Holm, and Petri Törönen. Novel split quality measures for stratified multilabel cross validation with application large and sparse gene ontology datasets. arXiv preprint arXiv:2109.01423 2021.
[28] Francisco Florez-Revuelt.. Evosplit: An evolutionary approach to split multi-label data set into disjoint subsets. Applied Sciences, 11(6)22823, 2021.
[29] Annette M Molinaro, Richard Simon, and Ruth M Pfeiffer. Prediction emor estimation: a comparison of resampling methods. Bioinformatics, 21(15):3301-3307 2005.
[30] Ji-Hyun Kim. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational statistics & data analysis,
2009.
12

---

[31] K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. The extreme classification repository: Multi- datasets and code, 2016. URL http::/manikvantt htmll
[32] Richard E Korf. Multi-way number partitioning. In IJCAI, volume 9, pages 538-543, 2009.
[33]
Brad L Miller, David E Goldberg, et al. Genetic algorithms, tourmament selection, and the effects of noise. Complex systems, 9(3):193-211 112, 1995.
[34] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[35] Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Mare Parizeau, and Christian Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171-2175, jul 2012.
[36] Pavel lakubovskii. Segmentation models pytorch.
https://github.comm/ltii/ qubvel/
segmentation_modes coctoommmmels..eoeee pytorch
pyt_modess..tttrchh 2019.
[37] G. Rudolph. Convergence of evolutionary algorithms in general search spaces. In Proceedings of IEEE International Conference on Evolutionary Computation, pages 50-54, 1996. doi:
10.1109/ICEC. NA1109//CEC.1199.55222 1996.542332.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims made in the abstract and introduction, namely a stratification algorithm for image segmentation datasets, is supported by the experiments conducted in section and results outlined in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our methods and findings are mentioned in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: The paper offers theorems in the Appendix A
Assumptions are clearly stated in the statements. Theorem statements mentioned in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information model architecture, training procedures, and experimental setup in Section 4 The datasets used in our experiments are publicly available. Additionally, we provide the algorithm and parameters of the proposed methods for easy reproducibility in the Appnedix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We used exclusively publicly available data for training and testing. We also provide reference implementation allowing for easy reproducibility of our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Sublic/guides httpe//ggiiieee/eeeubbbisiseee CoodeSuumisssonoolli..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //iips.ccppbbll/
guwdes/ Comm// cetisbbmissiooni ssioonolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification; We provide detailed information about dataset used and the experimental setup. Additional information about model hyper-parameters and the optimizer can be found in Section 4 We also provide a reference implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviation across five random seeds for statistical similarity measures and across ten folds for segmentation performance metrics. These error bars reflect variability due to different train/test splits and random initializations, which are central to our main claims. All reported deviations correspond to one standard deviation. The calculation done using standard statistical functions from Python libraries such as NumPy. This approach captures the expected variance across repeated runs and ensures the reported model performance is not the result of single experiments. The tables question are table 2 and table
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computational resources outlined in section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///euuuip..pee http///pbbccctthcccciieee.....le
Answer: [Yes]
Justification: We have followed the NeurIPS Code of Ethics to the best of our ability and knowledge.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [Yes]
Justification: The work improves evaluation reliability in image segmentation tasks, espe- cially in critical domains like medical imaging. While primarily methodological, its misuse could enable biased performance reporting if stratification selectively applied.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The proposed methods and findings pose minimal risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit and cite the original owners of assets (PascalVOC, Cam Endo Vis. LoveDA and Cityscapes) used this paper. LoveDA and Cityscapes explicitly state in its license that these datasets are freely available for research or academic work. For the PascalVOC, Cam Vid and Endo Vis datasets, no original license is available. The model implementation from smp is under MIT License.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com camdsataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for the proposed algorithm is submitted as part of supplementary material with the necessary documentation. The license for the assets is mentioned wherever applicable. The assets have been anonymized.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
20

---

A Convergence of Stratification Algorithms
This section presents results the convergence behavior of stratification methods. The first shows that random splits of large multi-label datasets asymptotically preserve marginal label frequencies in each fold, by the law of large numbers, ensuring representativeness without explicit stratification. The second analyzes WDES, an evolutionary algorithm that minimizes label distribution differences across folds. proves that the expected quality the best-found stratification improves at rate of O(I/GG), converging toward the global optimum as the number of generations and population size grow.
A.1 Asymptotic Representativenes, of Random Splits in Multi-Label Datasets
Theorem A.1. Let D = Vn)) n=1 be dataset where each Yn E {0, 1} L is binary label vector over fixed set of L labels. Suppose D is randomly partitioned into K disjoint folds s= ( ".,',..,,"* such that each fold satisfies | = r N, for a fixed target proportion vector E [0, with Σ r* 1. Then, for any label index l. E {1,..., L. the empirical label frequency in each fold converges almost surely to the true marginal label frequency as N - DC. That is, for all k- l,..., K,
PN (2) 0. p()),
where
(1)
Σ
and p(() E Elyne) P(yne =1).
|
Vne
nEsk
Proof. Fix a label index l E (1,..., L.. For each sample n, define the binary indicator:
0 Ynt {0, 1}.
By assumption, the data points Vn) are drawn i.i.d., so the sequence { Jn=1 is i.i.d Bernoulli with mean:
| = p(t).
Now consider the random assignment of samples to each fold S* where |S r* N. For sufficiently large N, each | - oo and the samples in each fold remain representative of the full dataset.
Let
(e) = 1
BESt
Then, by the Strong Law of Large Numbers (SLLN),
(2) a.a, E = p(t),
as | 00.
Therefore, for any k M (1...., K}.
(f) i a.s. 0,
and each fold becomes asymptotically representative of the true label distribution.
A.2 Optimality and Convergence of WDES
To quantify the effectiveness of WDES, we derive a convergence bound that characterizes how the quality of stratifications improves with the number of generations and population size under standard evolutionary algorithm assumptions.
Proposition A.1 (Convergence Rate of WDES to Empirical Optimum). Let FM. F denote the set of stratifications explored by the WDES algorithm after G generations with a population size M. Then the expected suboptimality of the final stratification SWDES FM.G satisfies:
E [[((nw))))]
LFFM 5
MG
where Liv denotes the optimal Wasserstein score found in the empirical subset.
21

---

Proof. WDES is a genetic algorithm that evolves population of M stratifications over G generations. The algorithm operates using elitist selection, where the best individual preserved across generations. This ensures that the sequence of best-found scores { | g=1 is non-increasing.
Each generation performs M fitness evaluations, totaling MG evaluations across the run. The search space F is finite because there are finitely many possible assignments of N labeled data points to K folds satisfying approximate size constraints. Hence. WDES induces finite-state stochastic process over F,G C F, the set of stratifications visited during execution.
Let D (x), n=11 be dataset with C classes, and F denote the space feasible stratifica- tions = (5" SK), ). where each fold 5 satisfies the approximate size constraint | = r N, for target proportion vector E [0, 1] with Σx 1.h 1.
Define the class distribution in fold k P 1 ΣnEsi Uy i]. and the global class distribution
P. N Σ Tlyn i). for 1, ,.... C. Let F Ei=1 Phe and F. Ei=1 P; be the cumulative class distributions fold and the full dataset, respectively.
The Label Wasserstein Distance of stratification S is defined as:
K
C
1 L(S) = K
Σ
Σ p - Fel.
Crucially, the mutation operator used in WDES is assumed to be ergodic, i.e., it has positive probability of reaching any feasible stratification from any other. This property, together with elitist selection, guarantees that the algorithm converges in probability the global optimum over F, as shown in [37]. However, in finite time, WDES only explores FM.G. and its best possible result L.M.a' The expected gap between the final stratification's score and this empirical optimum decreases with the number of independent evaluations MG. This follows from classical convergence results for evolutionary algorithms under elitist, ergodic dynamics, where the convergence rate to the best solution seen
E [[(SSDDS)) T.MM.
MG
This bound reflects the diminishing returns of increasing population size and generations, but confirms that more search budget leads to better empirical solutions.
Remark. The theorem guarantees convergence to the best stratification within the explored subset FM.G. not necessarily the global optimum L' minsef L(S). Nonetheless, due the ergodic nature of the mutation operator, the probability of reaching any region of the space is non-zero. Thus, 1 G 0S, and the algorithm converges the global optimum in probability.
1 F.M.
22

---

B Iterative Pixel Stratification Algorithm
Following the notation introduced in Section]
the algorithm for IPS is described in Algorithm
Algorithm Iterative Pixel Stratification
Input: A dataset D of C' classes, samples that have total of P pixels. The dataset is to be divided into K folds in the proportion r = P.
Output: Disjoint subsets SI..... Sk
1: Calculate desired number of samples per fold
2: [N*] f- -r
3: for c to do
4: Calculate number of pixels present per class
5: P :c:
6: #Calculate desired number of pixels per class per fold
7: [P=)] <- r
8: end for
9: while [D] > 0 do
10: Find rarest class
11: + argmin P.
for (1) ED; do
13:
# Find folds with largest desire for class
M 4 argmax Pl
kek
if M then
16:
m < M
else
18:
M' +- argmin N
KEM
19:
if MM then m M
21:
else
22:
m +
Random Element of (M')
23:
end if
24:
end
25:
# Assign sample to subset and remove from dataset
26:
5Th +- 5 U (x).
27:
28:
Nm <- N
29:
c C do
30:
p. Pm Px(
31:
end for
32:
end for
33:
end while
23

---

C
Wasserstein-Driven Evolutionary Stratification Algorithm
Following the notation introduced in Section the algorithm for WDES is described in Algorithm
Algorithm 2 Wasserstein- Driven Evolutionary Stratification (WDES)
Input: A dataset D of C classes and N samples, to be divided into folds according to proportions = (K); population size M; number of generations G; crossover rate Pe; mutation rate Pmu-
Output: Disjoint subsets SI...., Sx corresponding to the fittest individual.
1: # Initialize population of candidate stratifications
2: for i to M do
Generate random fold assignments according to 4: Store as individual I
3:
5: end for
6: for 4- 1 to do
Evaluate fitness of each individual for i to M do Evaluate fitness using Label Wasserstein Distance (Eq. 3 for fold assignments in 4 10: end for
7:
8:
9:
Select parents via tournament selection
12: Select pairs of individuals based on lowest LWD values
Apply crossover and correction for t 1 to number of parent pairs do randomm) < Pc. then
14:
16:
Exchange random segments of fold assignments between parents Apply correction to preserve fold proportions
18:
end if end for
19:
20:
Apply mutation
21:
for i M
22:
if (randoom) Pm then
23:
Randomly swap fold assignments of two samples in I;
24: end if
25: end for
26: end for
27: Select best pest-performing individual as final stratification
I argmin LWD(1;)
1:
29: Return S Sx the folds encoded in I*
24

---

D Genetic Algorithm parameters
We empirically selected the hyper-parameter for the genetic algorithm by evaluating performance across multiple datasets. For each parameter, we increased its value until further improvements were observed in the stratification quality. The final values are shown in Table S
Table 5: WDES Parameters
Parameter
Value
Number of Generations
50
Number of Individuals
100
Gene Mating Probability
0.5
Individual Mutation Probability
0.2
Selection Tournament size
3
E
Time and Space Complexity of WDES
The computational complexity of the Wasserstein Distance-basee Evolutionary Stratifier (WDES) can analyzed terms of the population size M, number of generations G, number of samples N, number of classes C, and number of folds K. Each individual in the population represents an assignment of all N samples into folds, and the fitness evaluation involves computing the Wasserstein distance across C-dimensional class histograms.
The overall time complexity is therefore (M G K .C), with an additional linear component N arising from crossover and mutation operations. The space complexity is O(M .N), the algorithm maintains population of M individuals, each encoding fold assignments for all samples.
In practice, the runtime of WDES is further influenced by the genetic algorithm parameters, including mutation and crossover probabilities, which determine how much of the population is updated between generations. These hyperparameters affect convergence speed and computational cost but do alter the asymptotic complexity. An empirical assessment of runtime performance is provided in the following section.
F Runtime Analysis
We evaluate the runtime performance of our stratification methods on an Apple MacBook Pro with an M3 Pro processor and 36 GB of memory. Figure&pprseent the runtime in seconds (with logarithmic scale on the y-axis) for Random splitting, IPS, and WDES across all datasets.
10'
LowaD
Camid
EndoVi
PesatVO
Cibesapps
Dalssets
Figure 4: Runtime comparison of Random splitting, IPS, and WDES across all datasets.
25

---

Table 6: Comparison of stratification methods on similarity- evaluation metrics. Results are averaged across datasets for Random splitting, IPS, CS-IPS, and WDES. While CS-IPS performs slightly better than IPS in some cases, WDES consistently achieves the best scores across all metrics, indicating stronger distributional alignment.
SD
PLD ox10
LWD (x10)
Dataset
Method
Mean
Std
Mean
Std
Mean
Std
Random
0.42
0.00
955
137
75.0
2.62
PascalVOC
CS-IPS
0.96
0.00
525
6.33
42.5
0.32
IPS
10.28
2.46
733
244
53.3
1.20
WDES
0.42
0.00
456
77.9
51.4
1.63
Random
0.00
0.00
63.4
3.73
183
5.08
Camvid
CS-IPS
0.60
0.00
58.0
0.39
175
1.48
IPS
6.68
1.18
76.6
5.30
209
8.44
WDES
0.00
0.00
36.7
4.47
138
7.84
Random
0.50
0.00
609
81.4
661
35.5
Endo
CS-IPS
0.80
0.00
485
8.00
549
3.34
IPS
11.42
1.05
769
12.2
764
75.2
WDES
0.50
0.00
208
23.8
399
10.4
Random
0.32
0.00
1110
109
1080
88.4
LoveDA
CS-IPS
0.64
0.00
935
11.9
892
7.58
IPS
8.06
1.15
1090
207
1100
87.4
WDES
0.32
0.00
377
31.4
635
16.0
Random
0.50
0.00
126
15.4
304
16.5
Cityscapes
CS-IPS
1.90
0.00
130
4.48
300
4.73
IPS
11.54
2.69
148
17.4
324
22.2
WDES
0.50
0.00
67
3.44
217
5.17
Random
1
3.0
3.2
Ranking
CS-IPS
2
2.2
2.0
IPS
3
3.8
3.8
WDES
1
1
G Class-Sensitive Iterative Pixel Stratification (CS-IPS)
The Class-Sensine Soass-Seniitive Iterative Pixel Stratification (CS-IPS) algorithm introduces an alternative iterative approach for constructing balanced folds based on pixel-level class distributions. The method begins by ranking all classes the dataset according to their rarity, determined by the total number of pixels belonging each class. Starting from the rarest class, all samples containing pixels of that class identified and sorted in descending order based on the number of pixels associated with the current class. The sorted samples are then distributed across folds in zig-zag manner to ensure balanced representation of the class across partitions. This procedure is repeated iteratively for each subsequent class until all samples have been assigned. The resulting folds aim maintain pixel-level balance while accounting for class frequency in straightforward, iterative manner.
We report the results of CS-IPS alongside those obtained using random splitting, IPS, and WDES in Table 6 While CS-IPS achieves slightly improved performance compared to IPS in several cases, WDES consistently attains the highest overall performance across datasets. These results reinforce our motivation for adopting a more flexible, globally optimized approach such as the genetic algorithm employed in WDES.
The corresponding segmentation results, summarized in Table exhibit a similar trend. CS-IPS provides measurable improvements over IPS, indicating the benefits of incorporating class sensitivity the iterative process. However, WDES continues outperform all other methods, underscoring the advantages of aptimization-drivee stratification over purely heuristic approaches.
26

---

Table 7: Average segmentation performance of UNet under 10-fold cross-validation using different stratification methods. Reported values correspond Accuracy, F1 Score, and loU. CS-IPS shows improved stability compared to IPS but remains below WDES, which continues to yield the most consistent and reliable fold performance.
Accuracy
F1
IoU
Dataset
Method
Mean
Std (x10-3)
Mean
Std (x10
Mean
Std (x10
Random
0.76
20.6
0.58
32.5
0.44
34.3
PascalVOC
CS-IPS
0.71
12.2
0.45
26.2
0.32
25.3
IPS
0.76
14.5
0.58
30.3
0.44
31.6
WDES
0.75
11.0
0.57
24.0
0.43
24.2
Random
0.94
0.68
0.91
1.16
0.89
1.17
Camvid
CS-IPS
0.94
0.70
0.91
1.32
0.89
1.24
IPS
0.94
0.74
0.91
1.35
0.89
1.27
WDES
0.94
0.67
0.91
1.09
0.89
1.06
Random
0.94
19.4
0.86
27.0
0.80
28.1
Endo
CS-IPS
0.92
13.6
0.86
21.2
0.78
21.4
IPS
0.94
14.1
0.87
29.3
0.79
30.7
WDES
0.94
13.0
0.85
17.9
0.79
18.9
Random
0.88
6.63
0.80
6.39
0.69
8.38
LoveDA
CS-IPS
0.88
7.30
0.80
8.19
0.69
10.3
IPS 0.88
7.39
0.80
9.29
0.69
10.7
WDES
0.88
7.49
0.80
9.86
0.69
11.0
Random
0.79
11.2
0.61
9.12
0.50
8.64
Cityscapes
CS-IPS
0.79
12.2
0.61
21.5
0.51
18.2
IPS
0.79
10.3
0.61
14.9
0.50
15.2
WDES
0.79
11.2
0.61
14.4
0.50
12.8
Random
2,6
2.2
2.2
Ranking
CS-IPS
2.4
2.6
2.6
IPS
2.6
3.2
3.6
WDES
1.8
1.8
1.8
H
Additional Evaluation Metrics
To ensure that the evaluation of stratification quality is not biased toward specific similarity metric, we further assess all stratification methods using additional distributional measures: the Linear-kernel Maximum Mean Discrepancy (L-MMD) and the Linear Energy Distance (L-ED). These metrics provide complementary perspectives on distributional similarity between folds and the overall dataset, capturing higher-order differences beyond those measured by the Label Wasserstein Distance (LWD).
Table g reports results for all methods-random splitting. IPS, CS-IPS, and WDES across these alternative metrics. Consistent with the findings based on LWD, WDES achieves the lowest average discrepancy across datasets, demonstrating that its advantages generalize across diverse statistical criteria. IPS performs competitively the VOC dataset, which attribute its higher proportion of single-class samples. Overall, these results confirm that the performance gains of WDES are robust to the choice of similarity metric.
Pixel Label Pair Distribution (PLPD) In addition the aforementioned distributional measures. we incorporate a class co-occurrence metric inspired by the Label Pair Distribution (LPD) proposed in literature 25). This measure captures second-order relationships between classes that tend to appear together within the same sample. We adapt this concept to the pixel level, computing the Pixel Label Pair Distribution (PLPD) to quantify the co-occurrence consistency between folds and the global dataset. The results, presented in Table 8 extend our analysis by evaluating how well each method preserves inter-class relationships. The previously introduced CS-IPS method is also included for completeness.
27

---

Table 8: Evaluation of stratification quality using alternative similarity measures: Pixel Label Pair Distribution (PLPD), Linear-kernel Maximum Mean Discrepancy (L-MMD), and Energy Distance (L-ED). Results are averaged across datasets for Random splitting, IPS, CS-IPS, and WDES. WDES consistently outperforms the other methods across all three metrics, demonstrating that its advantages generalize beyond Wasserstein- evaluation.
Dataset
Method
PLPD
L-MMD Ix10
L-ED
Random
0,67
5.67
3.64
CS-IPS
0.69
2.68
2.91
PascalVOC
IPS
0.66
1.84
2.56
WDES
0.76
2.94
2.49
Random
4.70
12.2
8.93
CS-IPS
5.97
11.1
8.28
Camvid
IPS
4.62
14.7
9.91
WDES
4.58
7.16
6.98
Random
0.36
46.8
14.7
CS-IPS
0.29
35.7
11.7
EndoVis
IPS
0.27
55.2
16.4
WDES
0.25
17.9
9.53
Random CS-IPS
1.35x10
78.6
20.9
1.22x10-b
62.7
17.2
LoveDA
IPS
x
69.9
21.3
WDES
1.20x10-5
27.7
12.8
Random
3.73
20.7
7.70
CS-IPS
3.75
21.2
7.44
Cityscapes
IPS
3.85
23.3
8.24
WDES
3.68
11.8
5.79
Random
3
3.2
3.2
CS-IPS
3,2
2.2
2.2
Ranking
IPS
2.2
3.2
3.6
WDES
1.6
1.4
1.0
These additional evaluations further corroborate the overall trends observed in the main results: while iterative approaches such IPS and CS-IPS produce reasonable fold-level balance, WDES consistently achieves superior alignment with global label distributions across all tested metrics.
I
Dataset details
I.1 Cityscapes
Cityscapes is large benchmark dataset for training and testing pixel-level and instance-level semantic labeling. It contains diverse stereo video sequences from street scenes 50 cities. Of these frames, 5,000 images have high-quality pixel annotations across 30 visual classes, grouped into eight categories: flat, construction, nature, vehicle, sky, object, human, and void. We use only the left-camera images (the annotated view) and run our experiments the training set, since it's the largest split.
1.2 CamVid
The Cambridge-driving Labeled Video Database (Cam captures footage from the perspective of driving automobile, unlike most videos that are filmed with fixed-position CCTV-style cameras. From this footage, 701 frames were sampled Hz and manually labeled with 32 semantic classes. Similar to CityScapes, we only use the largest subset, the training set.
28

---

Figure 5: Example Cityscapes data: the original image (left) and its enhanced annotation mask (right),
for better visualization.
Figure 6: Example Cam Vid data: the original image (left) and its enhanced annotation mask (right).
1.3 Pascal VOC 2012
The Pascal VOC 2012 serves as comerstone resource for training and comparing semantic segmen- tation models. It comprises 20 classes, including entities like people, animals, vehicles, and indoor objects. The dataset comprises 1,464 training images, 1,449 validation images, and a private testing Each image in this dataset is annotated with pixel-level segmentation, bounding box, and object class information. For our experiments, we exclusively utilized the training set.
Figure 7: Example Pascal VOC 2012 data: the original image (left) and its enhanced annotation mask (right).
L.4 Endo Vis2018
The Vis201 dataset, used in the Robotic Scene Segmentation Challenge of MICCAI 2018, contains 16 robotic nephrectomy procedures recorded using da Vinci Xi systems in porcine labs. Each procedure comprises 149 training frames and 250 testing frames, each with a resolution of x 1024. The dataset includes images from both the left and right eye cameras, as well as the stereo camera calibration parameters. However, labels are only available for the left eye camera, with 12 categories. In this case, we utilized only the training set.
1.5 LoveDA
The LoveDA dataset, a comprehensive collection of remote sensing images, designed semantic segmentation. It comprises 5,987 high-resolution images captured across three Chinese cities,
29

---

Figure 8: Example EndoVis2011 data: the original image (left) and its annotation mask (right).
capturing a diverse range of urban and rural scenes. These images are meticulously annotated with 166,768 object annotations, categorized into distinct land-cover classes: background, buildings. roads, water bodies, barren land, forests, and agricultural areas. Again, only the training set is utilized.
Figure 9: Example LoveDA data: the original image (left) and its enhanced annotation mask (right).
Acknowledgments
The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR) of the FAU Gelangen-Nürnbern The hardware is funded by the Deutsche Forschungsgemeinschaft (DFG).
30


Reviews:
[{"review_id": "bYS9nMmX9C", "reviewer": "Reviewer_Rioe", "summary": "In this paper, the authors argue that random splitting of image segmentation datasets into training, testing and validation data leads to unrepresentative test sets, resulting in biased evaluations and poor model generalization. Building on existing concepts from the stratification literature, which have addressed label distribution imbalance in classification tasks (in applications outside computer vision), the authors investigate stratification for image segmentation. This has specific challenges   due to the multi-label structure per sample (image) and the severe class imbalance that often occur in segmentation data. Essentially, the authors minimize criteria, which encourage the pixel-level class distributions in the data subsets (e.g., training, validation and testing subsets or folds) to match as closely as possible the class distribution of the overall dataset (Eqs. 2 and 3). The first criterion (Eq. 2) is the L1 norm between distributions and is optimized with the iterative stratification method in [23], which uses a greedy approach. The latter assigns samples to data subsets, to match a given number of pixels per class within each subset. The second criterion is the Wasserstein distance between distributions (Eq.3), optimized with an evolutionary algorithm like Evosplit [28], which evaluates a population of potential subset assignments (individuals), with the fittest individuals undergoing selective crossover and mutations for refinement. The authors evaluate the proposed stratification methods and random splitting across five image segmentation benchmarks, and used the standard deviation of accuracy, F1-score, and Intersection over Union across 10-fold cross-validation as measure of data splitting quality (with lower deviation indicating better performance).", "questions": "-\tFollowing on my main concern above, could the authors provide applications’ examples where it is relevant in image segmentation to ensure label distribution consistency across data splits (I do not see any). There are efforts in the segmentation literature that build benchmarks with distributions shifts (to simulate realistic settings), which is actually the opposite of what this paper is advocating. \n\n-\tFollowing on my comment above, did the author explore deterministic optimizers that enable subset assignments under cardinality constraints (I am not sure that genetic algorithms are the best option for optimizing the considered objective functions).", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is clear, technically sound and well written. \n\n- Semantic image segmentation and its evaluation remain important problems. \n\nWeaknesses\n\n-\tMy main concern with this paper is the following: The proposed methods are essentially removing (artificially) the class-distributions shifts between the testing and training data in segmentation, making the problem easier. I believe this does not reflect real and practical settings in image segmentation, where class proportions could change significantly (e.g. due to camera motion) and it is often the case that all possible class proportion are not covered in the training data. Therefore, artificially removing label-distribution shifts could lead to over-optimistic performance evaluations of segmentation algorithms (making the real-world problem much easier). In fact, there is a whole literature on unsupervised domain adaptation (UDA) for segmentation, which addresses specifically the challenges in distribution shifts (both in label distributions and image data distributions), which occur often in practice. In fact, there are even UDA benchmarks for segmentation that simulate such shifts and class imbalance to simulate real situations (and this is essentially the opposite of what this paper is doing). So, I am not sure that the main target of the paper (ensuring consistent class distributions across data subsets) is relevant in practice. This might explain why stratification has not been investigated before for image segmentation. I understand that stratification strategies are well used in other applications domains (other than computer vision).\n\n-\tThe main conclusion following the experiments is not surprising at all: Essentially, the proposed data splits methods yield lower performance variance across splits. Of course, such target data splits are artificially removing shifts between training and testing data, effectively making the problem easier. \n\n-\tI might be missing something but, in this specific case, sample-to-subset assignment by minimizing distances between distributions amounts to a discrete subset assignment problem under cardinality constraints. This is due to the fact that all the subsets have known cardinality (so, we just need to enforce constraints on class cardinality, not class distributions). Discrete subset assignment with cardinality constraints could be solved with a simple ranking, unless I am missing something. Essentially, for each class, you rank images in decreasing order using the number of class pixels within the image. Then you assign one or more images to each subset following this order and matching the required number of pixels for each subset.     \n\n-\tMinor: Theorem 3.1 (Convergence Rate of WDES to Empirical Optimum). I am not an expert in evolutionary algorithms, but the technical ingredients of the proof (Appendix 2) seem to follow standard textbook knowledge in the area, in which convergence results for evolutionary algorithms are established under some common assumptions (e.g., elitist model, ergodic dynamics) – see, for instance, [G. Rudolph, Convergence of evolutionary algorithms in general search spaces, Proceedings  of the IEEE International Conference on Evolutionary Computation, 1996]. So stating the result as a theorem in the main paper and claiming the derivation of convergence bound is, in my humble opinion, a bit of an over-claim. It gives readers the impression that Theorem 3.1 and its proof are new technical results. I would suggest presenting Theorem 3.1 as a Proposition and just state that the proof follows standard arguments in the literature (and cite proper references on the covergence of evolutionary algorithms).", "was_revised": true, "final_justification": "The authors did a great job addressing my comments and those of the other reviewers. I am increasing my score the \"borderline accept\".", "rebuttal": "Thank you for your feedback. We'd like to address your main questions as follows:\n\n**Unsupervised Domain Adaptation and Distribution Shifts**\n\nWe agree with the sentiment that distribution shifts can never be fully ruled out in real-world conditions and remain underexplored. However, we see our work as tangential to benchmarks or algorithms aiming to improve performance in the presence of these shifts. Single train-test splits remain common in practice, and any given test split is more representative when applying our stratification method. It thus allows practitioners to evaluate the generalizability of their models as drops in performance occur due to an inability to adapt to similar images rather than to different images. In practice, both the in-distribution performance (capability to interpolate) and the out-of-distribution performance (capability to extrapolate) you mention are of relevance. By stratifying, these two effects can be disentangled as we get a better estimate for in-distribution performance. As a final note, we'd like to point out that even in the domain of computer vision, standard stratification remains an important tool to evaluate and eliminate bias in image classification, in particular in the medical domain [1]. We thus continue to believe that this topic is relevant and will further emphasize these points in the revised paper.\n\n[1] C. Mosquera et al., \"Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance,\" *Imaging Informatics and Artificial Intelligence*, vol. 34, pp. 7895–7903, 2024.\n\n\n**Class-sensitive ranking-based stratifier**\n\nThank you for your comment. A simple ranking-based algorithm does not account for the fact that, once an image with a specific class is chosen, the other classes also present in that image may negatively influence the overall distribution. This interplay is exactly why we propose the genetic algorithm, and presumably why our proposed IPS, which follows a similar greedy line of thinking, generally does not perform as well.\n\nTo explore this further, we implemented an additional iterative stratifier, `CS-IPS`. It assigns samples to folds by prioritizing class rarity. Specifically, for all classes, samples are ranked based on the number of pixels of the class. We then start with the rarest class to assign all corresponding samples to folds. This process is repeated for the next rarest class until all samples are placed. \n\nWe show its results alongside those of random splitting, IPS, and WDES in the following table. While `CS-IPS` performs slightly better than IPS in some cases, WDES consistently outperforms all three methods on average across datasets. This supports our motivation for using a more flexible and globally optimized approach like the genetic algorithm.\n\n| Dataset | Method | SD ↓ | PLD (×10⁻⁵) ↓ | LWD (×10) ↓ |\n|---------|--------|----|-----------|---------| \n| **PascalVOC** | Random | **0.42** | 955 | 75.0 |\n| | CS-IPS | 0.96 | 525 | 52.5 |\n| | IPS | 10.28 | 733 | 53.3 |\n| | WDES | **0.42** | **456** | **51.4** |\n| **Camvid** | Random | **0.00** | 63.4 | 183 |\n| | CS-IPS | 0.60 | 58.0 | 175 |\n| | IPS | 6.68 | 76.6 | 209 |\n| | WDES | **0.00** | **36.7** | **138** |\n| **EndoVis** | Random | **0.50** | 609 | 661 |\n| | CS-IPS | 0.80 | 485 | 549 |\n| | IPS | 11.42 | 769 | 764 |\n| | WDES | **0.50** | **208** | **399** |\n| **LoveDA** | Random | **0.32** | 1110 | 1080 |\n| | CS-IPS | 0.64 | 935 | 892 |\n| | IPS | 8.06 | 1090 | 1100 |\n| | WDES | **0.32** | **377** | **635** |\n| **Cityscapes** | Random | **0.50** | 126 | 304 |\n| | CS-IPS | 1.90 | 130 | 300 |\n| | IPS | 11.54 | 148 | 324 |\n| | WDES | **0.50** | **67** | **217** |\n| **Ranking** | Random | **1.0** | 3.0 | 3.2 |\n| | CS-IPS | 2.0 | 2.2 | 2.0 |\n| | IPS | 3.0 | 3.8 | 3.8 |\n| | WDES | **1.0** | **1.0** | **1.0** |\n\n**Theorm -> Proposition**\n\nWe thank the reviewer for this helpful observation. We agree that the convergence argument relies on standard results from the evolutionary algorithms literature, and we will revise the presentation accordingly. Specifically, we will rename Theorem 3.1 to a Proposition and clarify in the text that the result follows from well-established arguments under common assumptions. We will also update the appendix to include appropriate citations to properly acknowledge prior work."}, {"review_id": "GXNPlK6Ls5", "reviewer": "Reviewer_PqJg", "summary": "The authors introduce two stratification methods for image segmentation: iterative pixel stratification, a pixel-level adaptation of iterative stratification, and Wasserstein-driven evolutionary stratification, a genetic algorithm that minimizes Wasserstein distance between class distributions in dataset splits. These methods address the limitations of random splitting, which can cause unrepresentative folds and biased evaluation, especially in small or imbalanced datasets. Results show that the latter proposed method made more balanced splits and lower variance in accuracy for more reliable model evaluation.", "questions": "1. Can the authors provide evidence for how WDES leads to better model selection or generalization after doing cross-validation with the folds that WDES found most optimal?\n2. Can the authors potentially improve the IPS method to be more competitive or explain why it was chosen over other potentially stronger iterative baselines?\n3. Is there a reason why Wasserstein difference was chosen as the metric for this problem as opposed to any other similarity metrics?\n4. How do the time and space complexity of WDES grow asymptotically?\n\nAddressing these questions effectively may help raise my score.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors include a proof for how WDES approaches the optimal stratification as the number of generations and the population size increase.\n- The paper is clear, well-written, and provides good motivation for the proposed method.\n- The focus on segmentation is significant since it is a method that is generally used but has limited labelled data due to the cost of pixel-level labels.\n- The authors provide results on five different datasets of vastly varying domains to show the effectiveness of their proposed method.\n\nWeaknesses:\n- IPS performs quite poorly compared to both random splitting and WDES. It is unclear why this was used and why it was not tuned further to use a non-greedy strategy.\n- Random splitting seems to perform quite a bit better than WDES in high-entropy datasets, which restricts the effectiveness of the proposed method to low-entropy datasets.\n- Lower performance variance is the main metric used to see if the proposed method is effective. There seems to be no effect or a slightly negative effect on accuracy when using the method.\n- With larger datasets, the runtime and computational load may be significantly higher for WDES.", "was_revised": true, "final_justification": "Thanks to the authors for thoroughly addressing my points. I will raise my score from a 3 to a 4 accordingly.", "rebuttal": "Thank you for taking the time to provide valuable feedback. We’d like to address your questions and related weaknesses as follows.\n\n**How WDES leads to better model selection or generalization**\n\nWe appreciate the reviewer's question. Our primary concern in this work is to improve the evaluation of models given a single train-test split, which is well-established in the image segmentation benchmarking literature. We used cross-validation only as a means to quantify the variance in evaluation metrics (e.g., accuracy, F1, IoU) across different proposed split that could be generated by each algorithm. Stratification makes performance estimation more reliable, which in turn strengthens downstream tasks such as model comparison and hyperparameter tuning. Without this, fluctuations caused by skewed class distributions can obscure true model performance. Finally, WDES is equally applicable to single train-test splits, where class imbalance in the test set can be even more problematic. In both settings, it provides a principled way to construct evaluation splits that better reflect the underlying dataset.\n\n\n**Potentially stronger iterative baseline**\n\nWe thank the reviewer for the suggestion. We took the spirit of the suggestion and implemented an alternative iterative strategy that assigns samples to folds by prioritizing class rarity. Specifically, for all classes, samples are ranked based on the number of pixels of the class. We then start with the rarest class to assign all corresponding samples to folds. This process is repeated for the next rarest class until all samples are placed.\n\nPreliminary results show that this variant performs slightly better than IPS on our stratification metrics, though it still falls short of WDES as it doesn't take class distributions inside folds into consideration. We call it `CS-IPS` in the following tables where we report the performance on the evaluation metrics. We are currently also evaluating its performance on the cross validation experiments on five datasets and hope to report these results before the end of the rebuttal period.\n\n| Dataset | Method | SD ↓ | PLD (×10⁻⁵) ↓ | LWD (×10) ↓ |\n|---------|--------|----|-----------|---------| \n| **PascalVOC** | Random | **0.42** | 955 | 75.0 |\n| | CS-IPS | 0.96 | 525 | 52.5 |\n| | IPS | 10.28 | 733 | 53.3 |\n| | WDES | **0.42** | **456** | **51.4** |\n| **Camvid** | Random | **0.00** | 63.4 | 183 |\n| | CS-IPS | 0.60 | 58.0 | 175 |\n| | IPS | 6.68 | 76.6 | 209 |\n| | WDES | **0.00** | **36.7** | **138** |\n| **EndoVis** | Random | **0.50** | 609 | 661 |\n| | CS-IPS | 0.80 | 485 | 549 |\n| | IPS | 11.42 | 769 | 764 |\n| | WDES | **0.50** | **208** | **399** |\n| **LoveDA** | Random | **0.32** | 1110 | 1080 |\n| | CS-IPS | 0.64 | 935 | 892 |\n| | IPS | 8.06 | 1090 | 1100 |\n| | WDES | **0.32** | **377** | **635** |\n| **Cityscapes** | Random | **0.50** | 126 | 304 |\n| | CS-IPS | 1.90 | 130 | 300 |\n| | IPS | 11.54 | 148 | 324 |\n| | WDES | **0.50** | **67** | **217** |\n| **Ranking** | Random | **1.0** | 3.0 | 3.2 |\n| | CS-IPS | 2.0 | 2.2 | 2.0 |\n| | IPS | 3.0 | 3.8 | 3.8 |\n| | WDES | **1.0** | **1.0** | **1.0** |\n\n**Why was Wasserstein Distance chosen?**\n\nWe thank the reviewer for this question. We selected Wasserstein distance because it:\n- Remains well-defined even when some classes are absent from a fold.\n- Captures both how much and how far mass must be moved to align distributions.\n- Naturally fits histogram-based distributions from pixel-level label counts.\n\nTo ensure our results are not tied to a specific similarity metric, we also evaluate stratification quality using Linear-kernel Maximum Mean Discrepancy (L-MMD) and Energy Distance (L-ED). As shown in the following table, WDES consistently outperforms IPS, random splitting, and the newly introduced `CS-IPS` on average across these alternative metrics. We note that IPS performs particularly well on the PascalVOC dataset, likely due to its high proportion of single-class samples. These results indicate that the benefits of WDES generalize beyond Wasserstein distance and remain consistent under other well-established distributional similarity measures.\n\n| Dataset | Method | L-MMD (×10²) ↓ | L-ED ↓|\n|---------|--------|--------------|------|\n| **PascalVOC** | Random | 5.67 | 3.64 |\n| | CS-IPS | 2.68 | 2.91 |\n| | IPS | **1.84** | 2.56 |\n| | WDES | 2.94 | **2.49** |\n| **Camvid** | Random | 12.2 | 8.93 |\n| | CS-IPS | 11.1 | 8.28 |\n| | IPS | 14.7 | 9.91 |\n| | WDES | **7.16** | **6.98** |\n| **EndoVis** | Random | 46.8 | 14.7 |\n| | CS-IPS | 35.7 | 11.7 |\n| | IPS | 55.2 | 16.39 |\n| | WDES | **17.9** | **9.53** |\n| **LoveDA** | Random | 78.6 | 20.9 |\n| | CS-IPS | 62.7 | 17.2 |\n| | IPS | 69.9 | 21.3 |\n| | WDES | **27.7** | **12.8** |\n| **Cityscapes** | Random | 20.7 | 7.70 |\n| | CS-IPS | 21.2 | 7.44 |\n| | IPS | 23.3 | 8.24 |\n| | WDES | **11.8** | **5.79** |\n| **Ranking** | Random | 3.2 | 3.2 |\n| | CS-IPS | 2.2 | 2.2 |\n| | IPS | 3.2 | 3.6 |\n| | WDES | **1.4** | **1.0** |\n\n**Time and Space Complexity of WDES**\n\nWe thank the reviewer for this question. The time and space complexity of WDES, as a genetic algorithm, depend on the population size *M*, number of generations *G*, number of samples *N*, number of classes *C*, and number of folds *K*.\n\nEach individual encodes an assignment of all *N* samples into *K* folds, and the fitness evaluation involves computing the Wasserstein distance over *C*-dimensional class histograms. This results in a time complexity of **O(M · G · K · C)**, with additional linear overhead in *N* from crossover and mutation. The space complexity is **O(M · N)**, as we maintain a population of *M* individuals.\n\nWe also note that the actual runtime is sensitive to the choice of mutation and crossover probabilities, which influence how much of the population changes from one generation to the next."}, {"review_id": "4w53SqzrQC", "reviewer": "Reviewer_Nnkd", "summary": "The paper introduces 2 new techniques, namely Iterative Pixel Stratification (IPS) and Wasserstein-Driven Evolutionary Stratification (WDES) to address  the shortcomings of existing sampling strategies, such as random splitting, for generation of representative folds from the original dataset for image segmentation. \n\nWhile IPS is a greedy approach where predefined class distribution is achieved based on the number of positive pixels in each fold and inspired by Iterative Stratification (IS) [23], WDES, a genetic algorithm inspired by Evosplit [28], optimizes the class distribution by minimizing the Label Wassertein Distance (LWD) and increasing the similarity between folds and the original dataset. \n\nThe paper evaluates the performances of IPS and WDES with 5 public segmentation datasets by using 3 metrics, and compares them against that of the random splitting. The results show that WDES generally produces better splits than the random splitting and IPS, especially when the evaluation data is a low-entropy segmentation dataset.", "questions": "An important question:\n-- **Will the authors release the code as open-source as the supplementary material says it is only for the academic review?**\n\nBased on the weaknesses above, \n-- Have the authors already evaluated the proposed method using another architecture? If not, could the authors provide some perspective on the effect of the architecture for the evaluation of the stratification method?\n-- Could the authors elaborate on the scalability and the feasibility of applying WDES to large datasets?\n-- What do the authors think about the performance improvements of the models for the underrepresented classes for low-entropy classes with WDES or IPS? Have they done any evaluation in this regard?\n-- Could the authors please comment on the unexpected high mean PLD for WDES for Cityscapes dataset?\n-- Can the authors check the Figure 4 in D Runtime Analysis and make sure the legend is correct (what does PCIS correspond to)?", "limitations": "As mentioned in the weaknesses, I would encourage the authors to investigave the co-occurance imbalance as this is highly relevant for the segmentation tasks. Incorporating different architectures into the evaluation, along with additional metrics would offer a more compherensive understanding of the proposed method. Finally, I would suggest that the authors should add some comments about the application of WDES to large datasets in terms of the scalability and the feasibility, considering its high computation cost.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n-- Generating representative test sets by splitting the original dataset is crucial for unbiased evaluation. The paper aims to address this common challenge in the field of machine learning, which has not been fully explored yet, especially for image segmentation tasks.\n-- To the best of my knowledge, the proposed methodology is a novel idea. IPS nicely extends the existing IS and WDES looks sensible approach for finding the similar class distributions across folds.\n-- This paper is well-written with a strong structure and thorough content that provides a good level of details. \n-- The evalution is supported by several public datasets from various domains including medical, satellite and street images as well as general-purpose dataset. The paper provides results from 10-fold cross validation evaluated using multiple metrics, namely accuracy, F1 score and Intersection over Union (IoU), along with their means and standard deviations.\n-- In my opinion, the paper presents important insights that may stimulate further discussion in the field.\n\nWeaknesses:\n-- I think there are a couple of points that need to be addressed for better understanding of the proposed method.\n-- Firstly, each evaluation dataset has a large number of classes. However, the provided metrics are the means of the fold means, so the label imbalance is not evaluated in terms of co-occurence imbalance as some label combinations may appear only in one fold, which limits the ability to accurately gauge the true performance of the proposed method. Although the authors mentioned this issue as \"class co-occurence\" in the conclusion section, this important limitation should be thoroughly examined for the full picture.\n-- Secondly, the paper uses a UNet with a resnet34 encoder as its underlying architecture. This is a specific architectural choice by the authors so I wonder whether the results would hold if a different architecture or even a different encoder were used.\n-- Thirdly, the addition of precision and recall into the evaluation would be helpful.\n-- Although WDES might provide better stratification under certain conditions, its computational cost is high. It would be beneficial for the readers if the authors could elaborate on the scalability and the feasibility of applying this method to large datasets.\n-- Another improvement of better stratification might be for the performances of the machine learning models for underrepresentative classes, especially in low-entropy classes. Although this is not necessary, it would be great to examine if WDES can specifically improve the model performance for rare classes in PascalVOC or CamVid.\n-- Finally, I am surprised about the mean PLD of WDES for the Cityscapes dataset. The value is unexpectedly high and warrants further investigation.", "was_revised": true, "final_justification": "I think that the reviewers raised great questions, highlighting key limitations, while the authors' rebuttals offered valuable insights to clarify these limitations.\n\nMy rating, which stays the same, is based on the quality of the study conducted by the authors. As acknowledged by the reviewers, the paper is well written and technically sound. The paper provides good evaluation supported by a number of experiments using diverse public datasets. I think that the paper may stimulate further discussion in the field.\n\nWhile I have some concerns regarding the limitation of the architectural choice, which may be relevant for certain domains, I think that the authors addressed the majority of my questions in their rebuttal.", "rebuttal": "Thank you for your thoughtful comments.\n\nRegarding the questions raised:\n\n**Class Co-occurence**\n\nThe paper from Szymański et al [2] introduced a metric to track class co-occurrence called Label Pair Distribution (LPD) aimed at tracking 2nd order relationship between classes appearing in the same sample. We re-implemented this metric by considering pixel counts and show the results the following table under the column PLPD (Pixel LPD). A previously introduced method, CS-IPS, as a reponse to reviewer PqJg has also been included in the table.\n\n[2] P. Szymański et al., \"A network perspective on stratification of multi-label data,\" *First International Workshop on Learning with Imbalanced Domains: Theory and Applications.*, pp. 22–35, 2017.\n\n| Dataset | Method | PLPD ↓ |\n|---------|--------|------|\n| **PascalVOC** | Random | 0.67 |\n| | CS-IPS | 0.69 |\n| | IPS | **0.66** |\n| | WDES | 0.76 |\n| **Camvid** | Random | 4.70 |\n| | CS-IPS | 5.97 |\n| | IPS | 4.62 |\n| | WDES | **4.58** |\n| **EndoVis** | Random | 0.36 |\n| | CS-IPS | 0.29 |\n| | IPS | 0.27 |\n| | WDES | **0.25** |\n| **LoveDA** | Random | 1.35×10⁻⁵ |\n| | CS-IPS | 1.22×10⁻⁵ |\n| | IPS | 1.21×10⁻⁵ |\n| | WDES | **1.20×10⁻⁵** |\n| **Cityscapes** | Random | 3.73 |\n| | CS-IPS | 3.75 |\n| | IPS | 3.85 |\n| | WDES | **3.68** |\n| **Ranking** | Random | 3 |\n| | CS-IPS | 3.2 |\n| | IPS | 2.2 |\n| | WDES | **1.6** |\n\n**Open Source Code**\n\nYes, we intend to release the full codebase as open source upon acceptance, under a permissive open-source license to support reproducibility and further research. The current restriction noted in the supplementary material applies only during the review period.\n\n**Using UNet + Resnet34 encoder**\n\nWe thank the reviewer for this important observation. Our choice of a U-Net with a ResNet-34 encoder was guided by the goal of using a widely adopted, representative architecture to demonstrate the effect of stratification on evaluation stability during training convergence. While we did not conduct extensive experiments across architectures in this submission, we do not expect our main conclusions to be architecture-specific. Our primary focus was to show that stratification leads to more consistent and reliable evaluation.\n\n**Scalability and Feasibility of WDES to Large Datasets**\n\nWe appreciate the reviewer’s comment. While WDES introduces more computational overhead than random splitting or simple heuristics, it is intended for small to medium-sized datasets where sampling variance and class imbalance can undermine evaluation reliability.\n\nFor larger datasets, random splits often suffice, as supported by Theorem A.1, which shows that class proportions per fold converge to the global distribution as dataset size increases.\n\nWe also point the reviewer to our response to Reviewer PqJg for a detailed discussion of WDES's time and space complexity. Finally, since stratification is typically done once during dataset preparation, we believe the added cost is often acceptable in practice.\n\n**Under-representative classes**\n\nWe thank the reviewer for this thoughtful suggestion. To assess the impact of stratification on rare classes, we analyzed the segmentation performance of three underrepresented classes in PascalVOC: bicycle, boat, and potted plant.\n\nAcross all three classes, WDES consistently yields lower standard deviation in accuracy, F1 score, and IoU compared to random and IPS-based splits. This mirrors the overall trend observed in our main results, suggesting that WDES not only improves stability at the global level but also benefits evaluation consistency for rare classes in low-entropy datasets. We will emphasize these findings in the final version of the paper.\n\n| Class | Method | Accuracy |  | F1-Score |  | IoU |  |\n|---------|--------|----|----|-------------|----|-----------|----|\n|         |        | Mean | Std ↓ | Mean | Std ↓ | Mean | Std ↓ |\n| **Bicycle** | Random | 0.62 | 0.04 | 0.31 | 0.08 | 0.19 | 0.06 |\n|             | IPS | 0.62 | 0.04 | 0.31 | 0.09 | 0.19 | 0.06 |\n|             | WDES | 0.62 | **0.03** | 0.31 | **0.07** | 0.19 | **0.05** |\n| **Boat** | Random | 0.72 | 0.05 | 0.53 | 0.09 | 0.37 | 0.08 |\n|          | IPS | 0.70 | 0.06 | 0.49 | 0.14 | 0.33 | 0.12 |\n|          | WDES | 0.72 | **0.04** | 0.55 | **0.08** | 0.38 | **0.08** |\n| **Potted Plant** | Random | 0.71 | 0.08 | 0.47 | 0.16 | 0.32 | 0.14 |\n|                  | IPS | 0.71 | 0.09 | 0.51 | 0.16 | 0.36 | 0.14 |\n|                  | WDES | 0.70 | **0.07** | 0.47 | **0.12** | 0.31 | **0.10** |\n\n**Correction 1**\n\nWe thank the reviewer for pointing this out. Upon rechecking, we realized that it is actually a typographical error. The correct mean PLD for WDES on the Cityscapes dataset is 67, not 671 as originally shown. We have corrected this mistake in the manuscript.\n\nWe appreciate the reviewer’s attention to detail and apologize for the oversight.\n\n**Correction 2**\n\nWe thank the reviewer for catching this. The label “PCIS” in Figure 4 is a typo and should correctly read “IPS”, referring to the Iterative Pixel Stratification baseline. We have corrected this mistake and updated the legend in the manuscript."}, {"review_id": "90QbgXUyqM", "reviewer": "Reviewer_o5rr", "summary": "Datasets used in machine are often randomly partitioned into disjoint sets that are used as folds for training and testing. This paper introduces methods that try to enforce that the proportion of pixels in each class for each fold approximates the number of pixels in associated classes for the whole dataset. The objective with this is to get more representable folds that yield more representative training results. To test the hypothesis, variance of the performance over the folds is recorded for the different fold-generating processes, and it is shown that the folds generated with the introduced method WDES produces models with lower variance in comparison to the introduced alternative method IPS as well as the classical random split method.", "questions": "**Question 1:** The studied problem looks like it has a bin-packing kind of structure – do you think it is NP-hard? \n\n**Question 2:** Why was a genetic algorithm approach used instead of a integer/mixed integer problem approach?\n\n**Question 3:** Would it be easy to adapt the technique to represent different volumes with the right frequency instead of only looking at pixel-frequency? What effect would you suspect this to have?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality:**\nThe work is technically sound and backed by reasonable experiments. The questions that come to mind are related to methodology.\n1) It would be interesting to have some discussion on whether the problem Is NP-hard and if there could be a polynomial and non-random method for solving it. \n2) A genetic algorithm is introduced, when, I would assume, a integer or mixed integer problem approach equally well could have been formulated and passed to a standard solver depending on branch and bounds techniques. It would be interesting to see this as a baseline reference.\n3) Whereas the pixel frequency is of importance, I think it would be even more interesting if it handled the distribution of the volumes (number of pixels per sample). That is, say that the dataset contains three distinct volumes for a class: x, 2x and 3x and that they occur equally often (with volume frequency 1/3), which means that the pixel frequency for that class would be 2x. Then it would be nice if the proposed method would differentiate between a set having the volume frequencies (1/2,0,1/2) and a set having the volume frequencies (0,1,0) with respect to the volumes (x,2x,3x). In other words, so that one fold does not only get samples with average volume 2x whereas another fold gets 50% samples with volume x and 50% with volume 3x.\n\n**Clarity:**\nThe writing is excellent, and the ideas are very clearly formulated. Even though the genetic algorithm was explained in text, it would have been nice to include a more formal description of the algorithm because of how central it is to the paper. \n\n**Significance:**\nIf the variance of performance of the number of folds necessary for testing a method in image segmentation can be reduced, then the reliability of the performance is increased. Potentially also the number of necessary folds can be decreased. These types of results are of very high interest to the community.\n\n**Originality:**\nI haven’t seen this kind of approach taken in image segmentation before. I am however not familiar with the literature in other domains of application so I cannot tell how close it is to previous work in other domains of application.", "was_revised": true, "final_justification": "I think this paper addresses the problem of generating representative folds in an interesting, simple and clear way that opens further research questions on how various \"fold difference loss functions\" could be defined and optimized. \n\nMy main concern was if the genetic algorithm really was the most suitable algorithm for the optimization.\nPrecisely my thinking was (1) if the problem was not NP-hard it would have been ideal to have a polynomial method and if one wasn't known then at least it would have been good to have a discussion about this  (2) if the problem was NP-hard then I would have argued that a mixed integer based approach would have been a more natural starting point than a genetic algorithm.\n\nThe authors clarified that the problem was NP-hard and showed me the problems they have found when considering a mixed integer approach.\n\nEven though I think the point that the reviewer Nnkd brings up about overgeneralization is important, I find it unlikely that there are properties related to the chosen architectures that affects the results in \"non-random\". Consequently, I am not as concerned about this.", "rebuttal": "We thank you for your valuable feedback. We answer the questions posed as follows.\n\n**Is the problem NP-Hard?**\n\nCreating the best split for segmentation tasks is formally NP-hard because it can be reduced from the multi-way partition [3] problem, a known NP-hard problem. In segmentation, each image contains pixel-level label distributions across multiple classes, and the goal is to partition the dataset into K folds such that the per-fold class distributions closely match the overall dataset, typically by minimizing a distributional distance like the Wasserstein distance. This is analogous to dividing a multiset of weighted elements into K subsets with balanced sums; except here, the \"weights\" are pixel counts per class and the balance is across multiple classes simultaneously. Computing the Wasserstein distance for a candidate split can be done in polynomial time, placing the problem in NP, but the underlying optimization is as hard as multi-dimensional multi-way partitioning, making it NP-hard. We will clarify this in the paper.\n\n3. https://en.wikipedia.org/wiki/Multiway_number_partitioning\n\n**Exploration of ILP or Mixed-ILP**\n\nWe thank the reviewer for this suggestion. In response, we experimented with formulating the stratification problem using mixed integer quadratic programming (MIQP) by minimizing the Linear-kernel Maximum Mean Discrepancy (L-MMD) between individual fold distributions and the global distribution. Unlike the Wasserstein distance, which requires sorting, L-MMD yields a solvable constrained objective function.\n\nDespite significant effort, the Gurobi solver failed to converge to an optimal solution within the imposed time limit of 12 hours. We report the best solution found (for a single seed) below for comparison. This result was notably worse than those obtained using WDES or even IPS, both in terms of stratification quality and runtime feasibility.\n\n| Dataset | Method | SD |  | PLD (×10⁻⁵) ↓ |  | LWD (×10) ↓ |  |\n|---------|--------|----|----|-------------|----|-----------|----|\n|         |        | Mean | Std | Mean | Std | Mean | Std |\n| **PascalVOC** | Random | **0.42** | **0.00** | 955 | 137 | 75.0 | 2.62 |\n|         | Gurobi | 0.42 | 0.00 | 940 | 0.00 | 57.8 | 0.00 |\n|         | IPS | 10.28 | 2.46 | 733 | 244 | 53.3 | 1.20 |\n|         | WDES | **0.42** | **0.00** | **456** | **77.9** | **51.4** | **1.63** |\n\n**Volume Frequency vs Pixel Frequency**\n\nWe thank the reviewer for this insightful question. After carefully considering the example provided, we understand the concern to be about not just total pixel frequency per class, but also the distribution of object volumes across samples, specifically, ensuring that folds reflect not only the average volume but also the diversity of object sizes (e.g., x, 2x, 3x) for a given class.\n\nOur current formulation focuses on pixel-level frequency distributions and does not explicitly control for the sample-wise volume profile of each class. While this could be extended, for example by stratifying on quantized volume bins per class, it would introduce an additional layer of complexity to the objective function, likely requiring multi-dimensional matching or joint constraints across volume profiles and class frequencies.\n\nWe agree this is an interesting direction and potentially important in settings where object scale is a key factor. However, we have not explored it in this work, and the effect on stratification quality and downstream model performance remains an open question.\n\n**Formal Description of WDES**\n\nWe have included a formal description of the WDES genetic algorithm to the appendix in the manuscript and will be visible in the camera-ready version."}]

Author rebuttal:

Meta-review: This paper presents an alternative to random sampling of data to generate training/test/validation splits, which the authors argue leads to biased/unrepresentative test sets. This could impact model generalization and affect evaluation in the community. In order to improve up on this for semantic segmentation, which has a number of challenges (e.g. severe label imbalance, etc.) the authors propose a criterion to drive pixel-level distributions in the subsets to match the overall dataset distribution, and these aspects can be optimized via iterative stratification methods. The authors compare to random splitting and show improved (reduced) standard deviations over metrics such as accuracy across a 10-fold cross validation scheme. Reviewers all appreciated that the paper is both well-written and technically sound, with a strong motivation and theoretical analysis of the proposed method. The main concerns included artificial treatment of class-distribution shifts between training/testing, and simpler baselines (ranking-based methods, CS-IPS), time/space complexity, effect of architecture, and alternative similarity metrics. Through a lengthy rebuttal and discussions, the authors provided a number of new results and explanations that overall satisfied the reviewers, who all recommend acceptance. I agree with the reviewers, and believe this paper provides an interesting perspective for data split generation that is understudied, and the paper proposes a well-formed and justified method. I recommend that the authors include the rebuttal elements in the updated version.
Decision: Accept (poster)