DIVERSE: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration
Authors: Gilles Eerlings
Brent Zoomers
Jori Liesenborgs
Gustavo Rovelo Ruiz
Kris Luyten
Venue: ICLR 2026 Poster (2026)
Keywords: Rashomon Set
Rashomon Effect
Feature-wise Linear Modulation (FiLM)
CMA-ES
Model Multiplicity
Predictive Multiplicity
Neural Network Diversity
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: We propose DIVERSE, a framework for systematically exploring the Rashomon set of deep neural networks, the collection of models that match a reference model’s accuracy while differing in their predictive behavior. DIVERSE augments a pretrained model with Feature-wise Linear Modulation (FiLM) layers and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search a latent modulation space, generating diverse model variants without retraining or gradient access. Across MNIST, PneumoniaMNIST, and CIFAR-10, DIVERSE uncovers multiple high-performing yet functionally distinct models. Our experiments show that DIVERSE offers a competitive and efficient exploration of the Rashomon set, making it feasible to construct diverse sets that maintain robustness and performance while supporting well-balanced model multiplicity. While retraining remains the baseline to generate Rashomon sets, DIVERSE achieves comparable diversity at reduced computational cost.

Published as a conference paper at ICLR 2026
DIVERSE: DISAGREEMENT-INDUCNN VECTOR EVOLUTION FOR RASHOMON SET EXPLORATION
Gilles Eerlings, Brent Zoomers, Jori Liesenborgs, Gustavo Rovelo Ruiz & Kris Luyten Digital Future Lab - Flanders Make
UHasselt Hasselt University
Diepenbeek, Belgium
(firstname, ame) 3 Suasseelt be
ABSTRACT
We propose DIVERSE, a framework for systematically exploring the Rashomon set of deep neural networks, the collection of models that match reference model's accuracy while differing in their predictive behavior. DIVERSE aug- ments pretrained model with Feature-wise Linear Modulation (FiLM) layers and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search latent modulation space, generating diverse model variants without retraining gradient access. Across MNIST, PneumoniaMNIST, and CIFAR-1O, DIVERSE uncovers multiple high-performing yet functionally distinct models. Our exper- iments show that DIVERSE offers a competitive and efficient exploration of the Rashomon set, making it feasible construct diverse sets that maintain robust- ness and performance while supporting well-balanced model multiplicity. While retraining remains the baseline for generating Rashomon sets, DIVERSE achieves comparable diversity at reduced computational cost.
INTRODUCTION
In many machine learning tasks, there exists large set of diverse models that achieve similar performance for the same tasks, yet differ substantially in the pathway leading their predictions. These differences can manifest in decision boundaries, internal representations, or feature impor- tance (Fisher al. 2019 Black et al. 2022). This phenomenon, where models attain similar per- formance yet differ in their decision pathways, is known as the Rashomon Effect (Breiman) (001), often also referred to as model multiplicity (Black et al., 2022 Dai et al. 2025). The collection of these functionally diverse yet accurate models is known as Rashomon set (Semenova al. 2022
Within a Rashomon set, models may produce divergent predictions for identical inputs, a property termed predictive multiplicity (Marx et al., 2020). These disagreements are not errors, but valid vari- ations within the model family, revealing that equally performant models can still differ on specific samples. This variation highlights when predictions depend on the model rather than the data. This has been leveraged for tasks such as assessing model uncertainty (Ferlings et al., (2023), promoting fairness (Dai et al. 2025). improving interpretability (Semenova et 2022), and increasing model flexibility (Jain et al.] 2025). At the same time, multiplicity poses practical challenges in high-stakes applications, since equally accurate models may assign different outcomes which might be consid- ered threat to faimess and trust in machine learning (Meyer et al. 2025). Rashomon sets capture the hidden flexibility of models while highlighting the need for systematic, computationally feasible methods to explore them.
While the Rashomon Effect is better understood in simpler model families such as decision trees (Xin et al.) (022) and generalized additive models (Zhong et al., 2023), its characterization in deep neural networks remains limited. The challenge is to explore their vast hypothesis space, while staying within a narrow margin of optimal performance, making Rashomon set exploration both computationally demanding and conceptually complex. Recent work has approximated Rashomon sets in neural networks through model retraining (Ganesh 2024 Eerlings al.) 2025). Adversarial Weight Perturbation (AWP) (Hsu & Calmon (022). and dropout- dropout-based sampling (Hsu et al.) 2024). These approaches offer valuable insights and strong foundation for our work, in which

---

Published as a conference paper at ICLR 2026
plore important trade-offs: retraining is computationally costly, AWP can scale poorly, and dropout sampling offers limited control over diversity. Building on previous work, our work proposes an alternative perspective that emphasizes efficiency and explicit diversity control.
We propose DIVERSE, a gradient-free framework for systematically exploring a local Rashomon set of deep neural networks without costly retraining. By augmenting the pretrained model with Feature-wist wise Linear Modulation (FiLM) layers (Perez et al. 2018). we perform a controlled search of the local environment around the model by modulating the pre-activations using latent vector z. This design enables fine-grained adjustments to internal representations while leaving the orig- inal weights untouched. By varying 2, the model traverses modulation space, a low-dimensional region of the hypothesis space where functionally distinct variants can be realized without retrain- ing. To search this space, we employ Covariance Matrix Adaptation Evolution Strategy (CMA- ES) (Hansem, 2006), which optimizes over candidate vectors z to induce predictive disagreement with the pre-trained model while maintaining accuracy within a predefined margin, thereby delin- eating the Rashomon set.
Contributions. (1) We introduce a robust Rashomon exploration method that combines FiLM- based modulation with CMA-ES, requiring no retraining gradient access; (2) on MNIST, Pneu- moniaMNIST, and CIFAR- 10, DIVERSE discovers accurate and diverse model sets, outperforming retraining in efficiency and dropout most of the time in diversity.
2 BACKGROUND & RELATED WORK
Defining and Approximating Rashomon Sets. We consider the standard supervised learning set-
ting with dataset DD = ((Ts }i) inputs x; E X C. RR9 and corresponding labels y; E 3 R* and hypothesis space H. where each model S H is parameterized by weights E RR.. Perfor- mance of each model is evaluated by a loss function L,, with empirical risk, following the principle of Empirical Risk Minimization /aannik 1995):
N R(fm) Σ ((fm( 2=1
(1)
N
Standard supervised learning seeks model that minimizes empirical risk, obtained through standard training, which we denote as the reference model fref. However, multiple parameter configurations can minimize the empirical risk equally well, yielding hypotheses fw E H. that match the perfor- mance of fres- The existence of many such solutions gives rise to the Rashomon set R. defined as the subset of models whose empirical risk lies within e-tolerance, known the Rashomon parameter (Semenova et al. 2022 2023), of the reference model, for e 2 0 (Fisher et al. 2019).
Enumerating the complete Rashomon set is computationally infeasible in the high-dimensional hy- pothesis space of deep neural networks. Following Hsu et al. (2024), we therefore work with an empirical Rashomon set, defined over the finite collection of models produced during search:
R.T (sw E Hm (R(f.,,) S R(Fres) +
where H.m H the finite candidate pool. As m grows, RT increasingly approximates R. (Hsu et al., (2024). For brevity, we refer to RT simply as the Rashomon set in the remainder of this paper. The Rashomon Ratio (Semenova et 2022) summarizes this set by measuring the fraction of candidates in Hm that meet the constraint, providing scalar view of Rashomon set size.
Several strategies have been proposed to approximate Rashomon sets in deep neural networks. Retraining from scratch with different seeds. hyperparameters, or augmentations (Ganesh, [024]] Eerlings et al. 2025) offers global exploration but is computationally costly and does not always guarantee models with equivalent performance. Adversarial Weight Perturbation (Hsu & Calmon 2022) perturbs a trained model's weights to induce disagreement while preserving accuracy, but this requires costly optimization for each (sample, class) pair, making it way slower than retraining and impractical for large neural networks (Hsu et al.) 2024). Dropout- sampling (Hsu et al, 2024) provides scalable, training-frt free alternative by stochastically generating subnetworks at test time, while showing substantial speedups over retraining, the resulting diversity limited and not explicitly controlled.
2

---

Published as a conference paper at ICLR 2026
FiLM Layers for Latent Modulation. Feature-wis Linear Modulation (FiLM) PPerez et al. 2018 is a conditioning mechanism that applies affine transformations to neural activations, enabling controlled modulation of intermediate representations. A FiLM layer has the form
( = 7 B,
(3)
where h is hidden pre-activation, and 7, are the modulation parameters that may be dynamically computed by another function.
FiLM has been widely adopted for conditional computation across range of settings. In do- main generalization, MixStyle (Zhou et al. 2021) introduces stochastic affine perturbations, which are FiLM-like, to simulate domain shilts. In few-shot learning, Tseng al. (2020) use feature- wise transformations to adapt representations to new tasks. A related use is found in FiLM- Ensemble (Turkoglu et al.) (2022). where ensemble members are defined by distinct, learned FiLM parameters optimized for epistemic uncertainty.
In our work, we repurpose FiL.M layers to define family of modulated models centered around fixed reference network (fret). Rather than learning new weights for each instance, we freeze the network and FiLM projection layers, and search over a latent vector from which y and are derived. This defines continuous subspace of the hypothesis space, the modulation space, enabling controlled exploration without retraining. We denote the model induced by latent vector as fz.
CMA-ES for Diversity-Orienter Search. Covariance Matrix Adaptation Evolution Strat- egy (CMA-ES) is a widely used optimization algorithm for continuous, non-convex search spaces (Hansen 2006). Unlike gradient-basee methods, it does not require access to objective gradients. Instead, it adapts a multivariate Gaussian distribution over the search space by repeat- edly sampling candidate solutions, evaluating their fitness, and updating the distribution parameters based on the relative ranking of candidates. Specifically, candidates at generation are drawn as:
(g) z
(m(()) o(9)² ((m)))),o0)) (()))(()))
where m()) is the current best guess (or mean of the search space), σ(و) is the global step size, and C(a) is the covariance matrix that shapes the search distribution. These parameters updated across generations based on the relative fitness of sampled candidates. The mean is shifted toward better-performing solutions, the covariance matrix adapted to reflect the directions of successful search steps, and the step size is adjusted using the length of a cumulative path.
Because CMA-ES adapts full covariance matrix, can capture correlations between variables, a crucial property in our setting, where the FiLMMmodulated latent vector induces a non- separable landscape: each coordinate of z affects many FiLM layers simultaneously, leading to strong interactions across dimensions. Full-covariance CMA-ES is designed for such coupled struc- tures through its rotational invariance and ability to learn arbitrary covariance patterns during the search (Hansen) (2006). Empirical comparisons confirm this: CMA-ES outperforms alternative evo- lutionary strategies, such as Differential Evolution and Particle Swarm Optimization, on low- to mnderaae-dimeesioonal non-separable or ill-conditioned problems (Auger et al. 2009: Omidvar & Li 201T). making a suitable optimizer for exploring our modulation space
However, full-covariance CMA-ES becomes expensive in high dimensions (c.g., over several hun- dred dimensions) (Omidvar & Li 201T). In our work, we leverage these strengths, namely its ability to capture correlations, handle non-separable landscapes, and learn rich covariance structures, in the context of latent vector z search, while considering the scalability limits of CMA-ES, which will influence our choice of latent dimensionality.
Quantifying Diversity. To explore distinct solutions in the z-space, we require measures that quantify functional differences between models. We distinguish two complementary notions:
Hard disagreement. A standard way to quantify diversity between two classifiers is to measure the proportion of inputs on which their predicted labels differ. Formally, for two models fw. and fws evaluated on dataset D, hard disagreement is given by:
N
Dis(Fw. fww)) =
Σ 1=1
1 10
+
0 = -x0, = -fa(
(5)
N
3

---

Published as a conference paper at ICLR 2026
where 1[-] denotes the indicator function, equal to 1 if the condition inside holds and 0 otherwise. This notion of disagreement originates in ensemble learning. where it was introduced as a measure of classifier diversity (Skalak 1996. Ho 1998, Kuncheva & Whitaker 2003).
Soft disagreement. To quantify differences between two models' predictive behaviors, we compare their output probability distributions the same inputs. While Kullback-Leibler (KL) (Kullback & Leibler 1951) and Jensen-Shannon (JS) (Lin 1991) divergences are common choices for measuring distributional differences, both rely logarithmic ratios that assume overlapping support between distributions. This assumption becomes problematic when models produce near-deterministic out- puts, leading to zero probabílities and numerical instabilities (Thomas M. Cover, 2005). We instead adopt the Total Variation Distance (TVD) (Devroye et al. 1996) as a more robust metric:
N TVD(P.Q) Σ
P Qil.
(6)
where P and Q are probability distributions from the two models. The TVD is bounded in [0, 1]. making it numerically stable even for deterministic predictions, and maintains connections to information-theorctic measures through Pinsker's ' inequality (Pinsker 1964).
Measuring Predictive Multiplicity. Having obtained collection of models that satisfy the Rashomon criteria as defined in Eq. (2). we require metrics to characterize the extent and struc- ture of predictive multiplicity. Unlike the metrics discussed in Section 2 these operate at the set level and summarize the properties of an entire collection of near-optimal solutions. We provide an overview of all the metrics used, their formal definitions, and brief explanation in Appendix A.I
Decision- based Metrics. One approach is to evaluate predictive multiplicity directly through model outputs. Ambiguity measures the proportion of instances whose predicted labels vary across models, while discrepancy captures the maximum proportion of disagreements between a baseline model and any alternative. Together, these metrics characterize whether multiplicity manifests as scattered disagreements or as systematic divergence across the Rashomon set (Marx et al. 2020).
Probability-bused Metrics. Beyond decision-level measures, probability- -based metrics capture how predictive multiplicity manifests in model output scores. The Viable Prediction Range (VPR) WWatson-Danich et al.] (223) summarizes, for each sample, the range of predicted probabilities assigned across the Rashomon set, indicating how wide predictions can vary while maintaining ac- curacy. The Rashomon Capacity (RC) (Hsu Calmon| 2022) instead measures score variations in the probability simplex using information- thhoreetic quantities, quantifying the effective number of distinct predictive distributions while avoiding the overestimation that can occur with thresholded decisions. Together, VPR highlights the spread of probabilities for class, whereas RC captures the richness of distributional variation, providing complementary perspectives on multiplicity.
MODULATING ACTIVATIONS FOR RASHOMON SET GENERATION
Our method, DIVERSE, explores the Rashomon of neural networks by embedding a pretrained model into a FiLM-moddule latent space and optimizing this space with CMA-ES. The approach unfolds in three steps: (i) training reference model, (ii) defining a FiLM-modulated space of models, and (iii) exploring this space with CMA-ES under Rashomon constraints.
Reference model. We train a reference network fref under standard supervised learning, with training stopped once validation performance stabilizes. The reference performance sets the Rashomon threshold, defined as the minimum performance required for inclusion the set. All subsequent variants are initialized from fret's learned weights.
FiL.M-modultedd model space. To induce controlled variation without retraining, we wrap the pretrained reference model with frozen FiLM layers. The reference latent vector is initialized to Zref = 0, which recovers the unmodified network. Any nonzero choice of z defines a candidate model f= that inherits all weights from fret but differs in its internal activations.
We consider three different insertion strategies depending on the underlying architecture, as shown in Fig. Regardless of the used strategy, FiLM acts on the pre-activations h, and all inserted FiLM layers share the same latent vector z. Coneretely, this means that this single vector modulates the model in coordinated, network-wide wide fashion.
4

---

Published as a conference paper at ICLR 2026
Formally, FiLM applies feature- wise affine transformation with modulation parameters (y and (), FiLM (l;; x) - (2) о + (()). 7(2) 1+ tanh(=W<,,, B(z) tanh(=W'),, (7)
where WB E Rd are frozen projection matrices with random initialization drawn from (0 0.52).The use of tanh bounds modulation to 7 E [0, 2] and / E [-1,1], preventing desta- large DC shifts during search. By construction,
0 recovers the ref- erence model (7 1. 3 = 0), while nonzero vectors induce structured, bounded variations. For dense outputs h I BxC FiLM acts feature-wise. For convolutional outputs |xxH* 7,8 E BBxC are reshaped to (B, 1, C) and broadcast across spatial dimensions, ensuring consis- tent per-channel modulation.
bilizing amplification
By freezing the projection matrices, the latent vector defines reproducible, low-dimensional mod- ulation space of functional variations. This design eliminates additional hyperparameters, making the method simple and stable while remaining expressive enough to generate diverse behaviors.
Inpat
Activation
Output
(a)
Input
Convolutional
Normalization
Batch
FILM
Activation
Output
(b)
Imput
Convolutional
Normalization
Batch
FiLM
Activation
Output
Optional
FILM
Skip connection
Zevector Madilatiion
Figure 1: FiLM placement (bluc): (a) after dense layers, (b) after convolutional blocks, after batch normalization when present, otherwise directly after the convolution, and (c) on residual skip con- nections. A shared latent vector provides the modulation parameters for all FiLM layers per model.
Latent Vector Design. The latent vector Rd parameterizes the FiLM modulations defined in Eq. 77). This provides a modulation space: instead of operating in the full weight space of the network, CMA-ES explores a lower-dimensional latent space that indirectly controls network behavior. Its dimensionality d determines the effective size of the search space explored by CMA- ES. Small d values yield efficient exploration but could restrict the diversity of functional variants, since the same latent dimensions are projected across many features. Larger d values could permit richer modulations and a broader range of behaviors, but increase computational cost and strain CMA-ES, which scales poorly in very high dimensions (Omidvar & 2011).
Initialization also plays an important role. As previously discussed, z = O recovers the pretrained reference model, making it natural anchor point for search. However, because CMA-ES adapts its distribution locally, different starting means can bias exploration toward different regions of the modulation space, potentially uncovering distinct Rashomon members. To mitigate this sensitivity and study how initialization affects the Rashomon set that is uncovered, we evaluate multiple starting points, probing both near-identity and stronger modulation regimes (see Section 4).
Searching the Latent Modulation Space with CMA-ES. Given a choice of dimensionality and initialization (Section 33). exploration of the modulation space is performed with CMA-ES. At each generation, CMA-ES samples candidate latent vectors from a multivariate Gaussian distribution (Eq. (4) evaluates their fitness, and then adapts the sampling distribution toward higher-scoring regions. This makes it well-suited for setting: the search space is continuous, non-convex, and gradient information is unavailable since underlying network weights are frozen.
The key design choice is the fitness function, which must balance two objectives: (i) keeping can- didates close to the reference model in performance, so they remain the Rashomon set, and (ii) maximizing predictive disagreement with the reference, that discovered models diverse. To do this, we compute class probabilities on the training set pz (x),, and form loss- and diversity- aware objective. Let (rr)(2)) be the reference model's probabilities, and denote the cross-entropy
5

---

Published as a conference paper at ICLR 2026
(CE) training loss of fz and fret by (x)ai( and L respectively. increase
We define the relative loss
Lerain (z) L ref
(z))
train
(8)
"train + 10
where the denominator is stabilized with small constant to prevent division by zero. Then we apply Gaussian penalty centered
(x(2) = exp( A(2)
2x
which softly enforces the Rashomon parameter candidates with losses close to the reference (i.e., A(z) = 0) receive little penalty: larger deviations are exponentially down-weighted rather than hard-rejected. This choice avoids discarding potentially diverse candidates that lie near the Rashomon boundary, allowing CMA-ES to explore the trade-off between accuracy and disagreement more flexibly.
To capture functional diversity, we combine soft (Eq. (6)) and hard disagreement (Eq. (5) with mixing weight À E [0, 1J. This weighting allows tune their relative influence: when X the score reflects only hard disagreement, while À = 1 reduces it to purely soft disagreement. In our experiments, we fix À = 0.5 balance the contributions. that results are mostly insensitive this choice, due to the strong correlation between soft and hard disagreement our FiLM-modulate models. The resulting diversity score defined as:
brief A-ablation Appendix AA. shows
Div = XTVD(pter(t) (x) + (1 - A)Dis(Jret J=)-
(10)
Finally, combine the diversity with the Gaussian penalty into single fitness function:
F(z) Div
(z). (z))
(11)
Taken together, the fitness design steers CMA-ES towards models that remain within the Rashomon set while exhibiting meaningful disagreement at both the decision and probability levels. The Gaus- sian penalty softly preserves accuracy, preventing premature rejection of boundary cases, while the mixture of hard and soft disagreement captures complementary notions of diversity. To ensure these properties extend beyond the training distribution, we enforce the Rashomon constraint on the val- idation set, retaining only z-candidates within of the reference model (Eq. (2)). The surviving models are then evaluated on a held-out test set, ensuring that reported Rashomon members reflect genuine diversity rather than overfitting the search process.
EXPERIMENTAL SETUP
Datasets and Reference Models. We evaluate DIVERSE on three complementary benchmarks, each with an appropriate reference model. On MNIST (LeCun et al., 1998) (70k samples, 10 classes), we use a 3 3-layer MLP (98% val/test), testing applicability beyond convolutional architec- tures. On PneumoniaMNISTT YYagg et al. (023) (5.8k samples, classes), we adopt an ImageNet- pretrained 5esNet-50 (98% val, 91% test), reflecting standard transfer learning in medical imaging. On CIFAR-10 Krizhevsky 2009) (60k samples, 10 classes), we use an
VGG- 16 (77% val, 75% test), a common baseline for moderate-scale vision tasks. Reference models are trained with Adam and carly stopping on validation performance; their accuracies define Rashomon thresholds and remain frozen during CMA-ES search. When no validation set is available, 10% of the training data is reserved. All subsequent variants are FiLM Moodullddd according to the insertion strategies Section and illustrated in Fig. [
Z-vectors. The latent vector Id parameterizes FiLM modulations described in Section
We study E (2, 4, 8, 16, 32, 64), balancing search reach against CMA-ES scaling. As discussed in Section 3 initialization biases exploration toward different regions of the latent space. To probe this effect, for each d we run initializations: the natural anchor (z = 0), a stronger perturbation (z 1), and eight additional draws from Gaussians centered at or 1 (o 0.1). The same initialization seeds are reused across all experiments for comparability.
CMA-ES Search Protocol. We configure CMA-ES using full covariance settings, following CMA-ES defaults (Hansen 2016)), using a population size of popsize = 4 + 3 3 d for latent di- mension d. To control the search horizon, we use a parameter k, which specifies the number of

---

Published as a conference paper at ICLR 2026
evaluations we allocate per latent dimension. Setting k = 80 yields target budget of kd total eval- uations, ensuring that the computational effort scales linearly with the size of the search space and that higher-dimensionn
receive proportionally more exploration. While CMA-ES has no strict rule for this choice, community practice often uses budgets of tens to hundreds of evaluations per dimension; our setting aligns with this and keeps computation tractable.
Because CMA-ES evaluates popsize candidates per generation, the evaluation budget kd trans- lates into Nlgenerations = ttd/popsizel which serves as the stopping criterion: the search runs until Vggenerations full generations have been completed. Each generation produces popsize candidates, re- sulting in m = popsize- Nggenerations total models per run. Since both popsize and Agenerations depend on the latent dimension d. the total number of models m generated by DIVERSE is fixed once d chosen. The ceiling operator may cause m to exceed the nominal budget of k:d by at most one additional generation, but this remains negligible. Finally, to assess the sensitivity of DIVERSE to the choice of initial exploration scale, we ablate over initial step sizes dp E {0.1, 0.2, 0.3, 0.4, 0.5).
Metrics. We evaluate the Rashomon sets along two dimensions: their size and their internal structure. The Rashomon Ratio (Section 2 quantifies the proportion of candidates that meet the Rashomon constraint, capturing test size. To assess structural diversity, we measure ambiguity and discrepancy to capture decision-level variation, and employ the Variable Prediction Range (reported as the width of the range rather than its bounds) and Rashomon Capacity (Section 22 to characterize distributional variation, with the latter reflecting the number of distinet predictive distributions in the set. Together, these metrics provide a comprehensive characterization of multiplicity within the discovered Rashomon sets.
Baselines. We compare DIVERSE against two families of baselines. First, retraining-based, ob- tained by reinitializing and retraining the reference model with different seeds, represents the stan- dard but computationally costly approach to Rashomon exploration. Second, we include dropout- based Rashomon exploration (Hsu et al. 2024), utilizing the Gaussian sampling method introduced by them, which implicitly samples functional variations by applying dropout masks at evaluation time. Unlike our method, however, dropout defines Rashomon membership directly on the test set, which risks optimistic estimates since the same data is used for both selection and evaluation. Our protocol instead enforces Rashomon constraints on validation set, and reports diversity metrics only on held-out test data, yielding stricter but unbiased comparisons. Because of the high computa- tional cost and impracticality of AWP for large neural networks, as discussed in Section we do not include Adversarial Weight Perturbation (Hsu& & Calmon| (002) as a baseline. For baseline compar- isons, report DIVERSE under the hest-performing hyperparameter configuration per e, selected by averaging performance across all multiplicity metrics; the corresponding configurations are listed All methods were executed the same hardware; full details are in Appendix A.3
in Table 3
5 RESULTS
We evaluate DIVERSE against retraining and dropout baselines on MNIST, PneumoniaMNNIST, and CIFAR-10. Our analysis addresses three questions: (i) how large and diverse are the Rashomon sets discovered, (ii) how robust is the method to hyperparameters, and (iii) how does DIVERSE compare the baselines.
Rashomon Sets and Latent Dimension. Fig. 2 reports five metrics across Rashomon thresholds
c: Rashomon Ratio, discrepancy, ambiguity, Viable Prediction Range (VPR), and Rashomon Capac- ity. On MNIST, nearly all modulated models satisfy the tolerance (ratios = 1.0), while discrepancy and ambiguity grow with e, showing that DIVERSE uncovers functionally varied solutions. Pneu- moniaMNIST and CIFAR-10 yield lower ratios, reflecting the challenge of preserving performance in deeper models, yet diversity metrics still increase with e, confirming meaningful multiplicity.
Latent dimension d further shapes these outcomes. On MNIST, all combinations of and d yield Rashomon sets, with larger d providing only small gains in diversity. At d = 32 and d = 64, the Rashomon Ratio shows jitter at the strictest thresholds (c {0.01, 0.02)). On CIFAR-10 and PneumoniaMNIST, the effect is more restrictive: d E {2,4] yields sets across all c, d = 8 only at 2 0.03, and sets recovered for d > 16.
Robustness to Hy perpparraesers. We evaluated sensitivity to the CMA-ES step size oo (Fig. 6 and latent initialization (Fig. 7). Initializing the latent vector at consistently produced
7

---

Published as a conference paper at ICLR 2026
d-2
d-16
d-32
d-64
MNIST
CIFAR-10
PreamoniaMMIST
Dicerppncy
0.02
0.10
0.01
0.05
0.05
Ambigiity
0.3
0.15
0.2
0.10
0.06
0.1
0.05
VPR
0.1
0.2
0.0
0.0
Itah. Cap.
1.1
1.0
1.0
1.0
Rash. Rtaio
0.5
0.5
0.01
0.02
0.03
0.01
0.05
0.01
0.02
0.03
0.04
0.06
0.01
0.02
0.03
0.04
0.05
Rashomon parameter (e)
Figure 2: Diversity metrics (discrepancy, ambiguity, Rashomon caascityccaaacity... VPR) and Rashomon Ratio across Rashomon thresholds (c) for varying latent dimensions (d). Each group of points corresponds to one horizontal spacing within groups is only for readability and carries no semantic meaning. Markers show medians, bars the interquartile range (IQR) across latent initializations, CMA- ES step sizes (oo), and d.
Rashomon members across datasets, whereas initialization around often failed. An ex- ception MNIST, where such initializations still yielded Rashomon members, though with reduced Rashomon ratios. The effect of also varies across datasets: larger initial step sizes increase diver- sity on MNIST, while smaller step sizes are more effective on PneumoniaMNISTT and CIFAR-I0.
Baseline Comparison and Runtime Efficiency. Fig, comppree DIVERSE, dropout-based sam- pling, and retraining across diversity metrics and the Rashomon Ratio. Retraining achieves the highest diversity metrics in most cases, but does not always yield the largest Rashomon Ratios. On MNIST, DIVERSE exceeds retraining on discrepancy at higher values and outperforms dropout all metrics except VPR. On CIFAR-10, DIVERSE surpasses dropout across all metrics. On PncumoniaMNISTT DIVERSE outperforms dropout and retraining discrepancy at the highest e-level, but falls short stricter thresholds. Across datasets, DIVERSE attains consistently lower VPR values. Table compares the runtimes to generate m candidates. Retraining is prohibitively costly, requiring hours on PneumoniaMNISTT and CIFAR-10, dropout samples models within sec- onds, and DIVERSE completes search in minutes, but remains slower than dropout- dropout-based sampling.
Qualitative Illustrations of Multiplicity To complement the quantitative metrics, we provide qualitative view of predictive multiplicity in Fig. For each dataset (MNIST, CIFAR-10, and Pneu- moniaMNIST),, we select the sample with the highest model disagreement and display the input alongside the flss-frrquency histogram over the Rashomon set found by DIVERSE. These exam- ples reveal how models that are all equally optimal under the Rashomon constraint can nevertheless make different decisions the same input, making predictive multiplicity visually explicit.
Layerwise Sensitivity. We additionally performed a layerwise ATVD sensitivity analysis to iden- tify which FiLM layers contribute most disagreement (details in Appendix A.5). Across archi- tectures. diversity is localized: early FiLMs dominate MNIST, mid-level convolutional FiLMs in VGG16, and early mid-stage FiLMs in ResNet50. These patterns suggest that only subset
8

---

Published as a conference paper at ICLR 2026
MNIST
CIFAR
PneumonniaM
1.0
1.0
% of Rashomon Set
0.8
0.6
0.4
0.1
0.2
0.2
0.2
0.0
0.0
0.0
7 8
9
5
6
Class
Figure 3: Highest-disagremment samples for MNIST, CIFAR-10, and PneumoniaMNIST.. Each panel displays the input image and the class-frequency distribution over the Rashomon set members; the true class is shown in green, and all other classes are represented in gray.
of FiLM sites meaningfully drives diversity, pointing toward future approaches that focus CMA-ES search on sensitivity umsitivity-identiffee identified layers rather than modulating the entire network.
Taken together, these results demonstrate that DIVERSE consistently generates non-trivial Rashomon sets under principled validation constraints, with efficiency that enables exploration at scale. We next discuss implications for multiplicity research and limitations of our approach.
Table I: Runtime (hh:mm:ss) to obtain candidate models under the same Rashomon constraint. For DIVERSE, m is determined by the latent dimension d (see Section 4) in the settings shown, = yields m 162 models and d 8 yields 640. All baselines are configured generate the same m for fair comparison.
MNIST
PneumoniaMNIST
CIFAR-10
Method
m=162
m=640
m=162
m=640
m=162
m=640
Retrain
00:29:39
01:57:36
02:09:00
08:16:00
03:17:26
12:37:27
Dropout
00:00:30
00:01:57
00:01:32
00:05:47
00:01:58
00:06:30
DIVERSE
00:00:50
00:03:16
00:01:49
00:07:15
00:02:11
00:08:42
DISCUSSION
Key Findings. Our study shows that DIVERSE reliably uncovers non-trivial Rashomon sets across varying datasets. It recovers functionally distinct models under strict e, robust ini- tialization and step size settings, and produces candidate sets within minutes. Compared to existing approaches, DIVERSE is far more efficient than retraining while delivering equal or better diversity than dropout under principled validation and test constraints. These findings establish DIVERSE as practical tool for large-scale multiplicity research deep neural networks.
Interpretation of Results. The experiments reveal how dataset complexity and architecture in- fluence which configurations yield valid Rashomon sets. On MNIST, Rashomon members appear for all combinations of and latent dimension though diversity gains decline at larger d and the Rashomon Ratio exhibits jitter at very small values. This suggests boundary effects under tight tolerances, where small fluctuations in validation performance determine membership. By

---

Published as a conference paper at ICLR 2026
DIVERSE
Dropent CI
Retraining
MNIST
PneamoniaaMNS7
Ambiguity Dicrrppncy
0.02
0.15
0.10
0.10
0.01
0.05
0.50
0.10
0.05
0.25
0.1
0.50
0.50
VPR
0.50
0.25
0.225
Raas. Cap.
1.2
1.50
1.1
1:25
Rash. Rttio
0.75
0.5
0.01
0.02
0.03
0.01
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.06
Rashomon parameter (e)
Figure 4: Best hyperparameter configuration for DIVERSE compared against the pre-defined pa- rameters of the dropout authors vs retraining.
trast, Pneumonia.MNIST and CIFAR-10 admit Rashomon sets only at small latent dimensions, with 16 failing entirely. This points to the more complex search landscapes created by deeper net- works with many FiLM injections, where increased dimensionality makes it harder for CMA-ES locate candidates within the Rashomon tolerance. Hyperparameter sensitivity follows the same pattern: larger initial step sizes encourage exploration in the simpler MNIST MLP, whereas smaller values are more effective for deeper models, where rugged landscapes benefit from finer search granularity. These results also contextualize comparisons with baselines: retraining provides broad exploration at high cost, while dropout is nearly instantaneous but bypasses search on the training set and defines Rashomon membership directly the test set, leading to diversity estimates that may be optimistically biased by model selection on test data. Taken together, these findings suggest that multiplicity in deep networks is shaped not only by optimization but by the interplay of search dimensionality, architectural complexity, tolerance thresholds, and the evaluation protocol itself.
Extension Transformers. While our main experiments focus MLPs and CNNs, the FiLM- based modulation in DIVERSE also applies to dense layers in Vision Transformers (ViT) (Dosovit- skiy et al. 202T). We include small preliminary ViT experiment on CIFAR-10 in Appendix A.6 which provides an initial indication that the modulation- search mechanism extends to Transformer architectures and motivates a more comprehensive future study.
Limitations. While DIVERSE shows that FiLM- modulation with CMA-ES can efficiently uncover non-trivial Rashomon sets across multiple architectures and domains, several opportunities remain for improvement. First, full-covariance CMA-ES becomes costly as latent dimensionality grows, restricting us to moderate-sized modulation spaces; scalable variants such as DD-CMA- ES (Akimoto & Hansen 2019), offer promising path toward higher-dimennionall search. Second, FiLM modulation provides controlled functional variation but does not guarantee that the discovered causally meaningful ways. Third, although our evaluation spans three datasets, extending to additional modalities, including a more extensive Vision Transformer study (Appendix (.6)), would further test the generality of our conclusions.
models differ in interpretable
10

---

Rashomon membership Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our work. All datasets used (MNIST, CIFAR-10, and PneumoniaMNIST) are publicly available, with preprocessing and splits described in Section 4 Detailed descriptions of the methodology. including FiLM modulation, CMA-ES search, and the fitness function, are provided in Section [}] with full metric definitions in Appendix A.I] Hyperparameter settings and search protocols
outlined in Section 4 and additional sensitivity analyses are reported in Appendix AA.5 A complete specification of the hardware setup is provided in Appendix A..3 To facilitate the exact reproduction of our ex- periments, we release the full source code and experiment scripts at httpt::p///// hittps://githbb com/
ETHICS STATEMENT
This work focuses on methods for exploring the Rashomon set of neural networks. All datasets used (MNIST, CIFAR-10, and PneumoniaMNISST) are publicly available and widely adopted for academic research under appropriate licenses. No personally identifiable information or sensitive private data were collected or generated in this study. We have taken care to ensure faimess by reporting results across multiple datasets and by using a validation- criterion to avoid biased or overly optimistic evaluations. The potential risks of this research are limited to the misuse of diverse model sets for generating deceptive predictions; however, we em- phasize that intended contribution is to enhance the robustness, transparency, and understanding of predictive multiplicity.
ACKNOWLEDGMENTS
This work was funded by the Flemish Government under the "Onderzoeksprogrammi Artificiële In- telligentie (AI) Vlaanderen" programme, R- R-13509. This research was partly funded by the Special Research Fund (BOF) of Hasselt University (R- 14436) and the FWO fellowship grant (ISHDZ24N). The resources and services used in this work were partly provided by the VSC (Flemish Supercom- puter Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government.
REFERENCES
Youhei Akimoto and Nikolaus Hansen. Diagonal acceleration for covariance matrix adaptation evolution strategies. CoRR, fass/195.05580 2019. URL http://aaxxlv..rrg/bbb///00
05:85
Anne Auger, Nikolaus Hansen, Jorge Pérez Zerpa, Raymond Ros, and Mare Schoenauer. Experi- mental comparisons of derivative free optimization algorithms. pp. 3-15, 06 2009. ISBN 978-3- 642-02010-0. doi: 10. 00.100779978-3-64--00001 1-7_3.
Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability; and Transparency. FAceT '22, pp. 850-863, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781 478353333. doi: 10.114575331466353311
URL https://doi.
org/10 10. 11 45 3555/531155 655553331116633311111 3533149
Leo Breiman. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statistical Science, 16(3):199 231, 2001. doi: 10. 00012144s/10009213722 httpss
Gordon Dai, Pavan Ravishankar, Rachel Yuan, Daniel B. Neill, and Emily Black. Be intentional about fairesss Fairness, size, and multiplicity in the rashomon set, 2025. URL https://
Lue Devroye, László Györfi, and Gábor Lugosi. A Probabilistic Theory of Pattern Recognition. Springer New York, 1996. ISBN 9781461207115 doi: 10.1007/978-1-4612-000 URL
doi www....rg..... wtt../............00007111111111

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. /ttttps://aaii.....ttt abs 22000 92.
Gilles Eerlings, Sebe Vanbrabant, Jori Liesenborgs, Gustavo Rovelo Ruiz, Davy Vanacken, and Kris Luyten. Ai-spectra: A visual dashboard for model multiplicity to enhance informed and trans- parent decision-making. In Engineering Interactive Computer Systems. EICS 2024 International Workshops, pp. 55-73, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-91760-1
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying entire class of prediction models si- multancously. Journal of Machine Learning Research, 20(177):1-81, 2019. URL http:
7/jmlr. wwtpp//ppppppp///pppr./ppp
html
Prakhar Ganesh. An empirical investigation into benchmarking model multiplicity for trustworthy machine learning: A case study on image classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4488 4497, January 2024.
Nikolaus Hansen. The CMA Evolution Strategy: A Comparing Review, pp. 75-102. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006. ISBN 978-3-400-324494444- doi: URL /ttptppp////ooooorr001000//////2
1.4.
Nikolaus Hansen. The CMA evolution strategy: tutorial. CoRR, abs/1604.00772, 2016. URL
Tin Kam Ho. The random subspace method for constructing decision forests. JEEE Transactions on Pottern Analysis and Machine Intelligence, 20(8):832-844, 1998. doi: 10.1 1109/44779600)
Hsiang Hsu and Flavio Calmon. Rashomon capacity: A metric for predictive multiplicity in clas- sification. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httphhtss///peeeeiiiiii
Forum? dd= www.dddCCyWdyaW.c....
Hsiang Hsu, Guihong Li, Shaohan Hu, and Chun-Fu Chen. Dropout-based rashomon set exploration for efficient predictive multiplicity estimation. In The Twelfth International Conference Learn- ing Representations, 2024. /ttlhttts////oeeeeeee
Shomik Jain, Margaret Wang, Kathleen Creel, and Ashia Wilson. Allocation multiplicity: Evaluat- ing the promises of the rashomon set. FAceT '25, pp. 2040-2055, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714825, doi: 10.1 10.1145/77152253332213...
_ http:::/p//...........:::
100 1145/115111 5275. 11113112775772222111111 3732138
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79- - 86, 1951. doi: 10. 10.1214/amms/11777 17770m/9/11777994,.. HtR.https://ddo org/
10.1214/aai aoms/ 1177729694
Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181-207, May 2003. ISSN 1573-0565. doi: 10. 102289990000 URL htt.htt::// /ttt:::/dxx.eee doi. war.io.rrr/1..
A:1022859003006
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232) November 1998.
J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 3111)145--151 1991. doi: 10.1/.9/1...11..
Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6774. PMLR. 13-18 Jul 2020. URL https: httpss:/proceedingg mmrr. www.ppresssiiii mttesss/211/m/m222a. html
12

---

Published as a conference paper at ICLR 2026
Anna P. Meyer, Yea-Seul Kim, Loris D'Antoni, and Aws Albarghouthi. Perceptions of the fairness impacts of multiplicity in machine learning. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, New York, NY. USA, 2025. Association for Computing Machinery. ISBN 9798400713941, doi: 4411445770699988733324 URL https:
7/doi. aeddi.org/10 10. 1ew.111153377653 3713524
Mohammad Nabi Omidvar and Xiaodong Li. A comparative study of cma-es on large scale global optimisation. In Jiuyong Li (ed.), Al 2010: Advances in Artificial Intelligence, pp. 303-312, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-17432-22
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. volume 32. Association for the Advancement of Artificial Intelligence (AAAI,, April 2018. doi: 10.1609/aaai..321111. v.2.......2.11111111111 URL http://///xx
doi. doi.org/10. 1609 aaai ww2.aaaai................1......
ww2.1.2211...... 11671
M.S. Pinsker. Information and Information Stability of Random Variables and Processes. Holden- Day series in time series analysis. Holden-Day, 1964. ISBN 978081660 66268047.
Lesia Semenova, Cynthia Rudin, and Ronald Parr. On the existence of simpler machine learning models. In 2022 ACM Conference Fairness Accountability and Transparency, FAceT '22, pp. 1827-1858. ACM, June 2022. doi: 10.1 1463353323.. URL http:http/// dof org/ 10. 10.1145/33 3531 46. 3533232
Lesia Semenova, Harry Chen, Ronald Parr, and Cynthia Rudin. A path to simpler models starts with noise. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL net forum id=Uzi22W=>>>>>
David B. Skalak. The sources of increased accuracy for two proposed boosting algorithms. In AAAI Conference on Artificial Intelligence, 1996.
Joy A. Thomas Thomas M. Cover. Elements of Information Theory, chapter 2, 11, pp. 13 55, 347-408. John Wiley and Sons, Ltd, 2005. ISBN 9780471748823. 10. N0.100/0047174822
doi: https://
[RRlhtttss///nniiee wwwlll.rrrr.... wi wiw www...llll... wwt.coo/////..... don//da//// /
Hung-Y Yu Tseng, Hsin- Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, 2020. URL http::::
net/ Seett//orrmid--S1555pp
Mehmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüi, Mina Rezaei, Bernd Bischl, Ro- drigo Caye Daudt, Stefano D' Aronco, Jan Dirk Wegner, and Konrad Schindler. Film-ensemble: probabilistic deep learning via feature-wise linear modulation. In Proceedings of the 36th Inter- national Conference Neural Information P Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088
Vladimir N. Vapnik. The nature of statistical learning theory. Springer- Verlag, Berlin, Heidelherg, 1995. ISBN 0387945598.
Jamelle Watson-Daniels, David C. Parkes, and Berk Ustun. Predictive multiplicity in probabilistic classification. In Proceedings the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, MAAT doi: 10.16099aaai..33799.226 26227. URL https:
AAAI Press, 2023. ISBN 978-1-57735-880--0
//doi. otop/oorgg100. 1wt9/6660.10////// www.a..v....... 26227
Rui Xin, Chudi Zhong, Zhi Chen, Takuya Takagi, Margo Seltzer, and Cynthia Rudin. Exploring the whole rashomon set of sparse decision trees. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2 large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1), January 2023. ISSN 2052-4463 doi: 10.1038/ 012-1-21-----11-
541557-022-01721--8 URL.tttp://ds dof org/100.
13

---

Published as a conference paper at ICLR 2026
Chudi Zhong. Zhi Chen, Jiachang Liu, Margo Seltzer, and Cynthia Rudin. Exploring and interacting with the set of good sparse generalized additive models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023.
Curran Associates Inc.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
net/forum id 6xH33 /MVxxp
A APPENDIX
A.1 PREDICTIVE MULTIPLICITY METRICS.
We begin by introducing the notation used throughout this appendix. Let D ((ri, ) denote the dataset with N samples. Let fret be the reference model obtained under standard training. The empirical Rashomon set, constructed according Eq. (2), is given by f(M)).
R {f())
where each f(i) is candidate model within the tolerance For model f() R.", we denote its predicted label on input x; by
0 f
and the prediction of the reference model by re (vet
Rashomon Ratio. As introduced by (Semenova et al. 2022), the Rashomon Ratio measures the relative size of Rashomon set. Adapting this definition to the empirical Rashomon set R it is given by:
R
RR,
(A.1)
Hmm
Ambiguity. Following Marx et al. (2020), the fraction of samples for which at least one Rashomon model disagrees with the reference prediction:
1 N
Ambiguity(RR) (R fret)
Σ
| ......,,,, 0f क:
(A.2)
Diserepancy. Following Marx al. (2020), the worst-case disagreement rate between the reference model and any Rashomon model:
N Σ i=1
Discropancy(R." fret)
max jE(L....M))
j + 0
(A.3)
N
Viable Prediction Range (VPR). As introduced by Watson-Daniele et al. (203)), for sample
model f(6 R." outputs class pross-probabbltty vector p! E A For class E {1...., k}, define the per-class VPR as
vPR;, max () min ( 1SSSM
P
Piee
(A.4)
ISJSM
In practice we report the true-class width VPR.,,, and summarize [VPRG, Ji over the test set.
Rashomon Capacity (RC). Proposed by Hsu & Calmon (2022) and empirically computed as in Hsu ct al. 2024), the RC measures the effective number distinct predictive distributions the set: we fix and view the empirical Rashomon set discrete channel with input W {1...., M)
(model index) and output Y {l....,k, }} (predicted class), where P(Y = c W = j) = Pi. The per-sample Rashomon Capacity is the channel capacity:
())
M
M
RC; = max 1(W;)) = max Σ DKL (
P..
9
Σ TeP:
(bits).
(A.5)
rEAM
FEAM
==1
t=1
We compute RC, via the (mean/median) reported over (RC
algorithm (using log,).
Set-level summaries
14

---

Published as a conference paper at ICLR 2026
A.2 IMPACT OF MIXING WEIGHT
To assess the influence of the mixing weight À in Eq. 10. which balances soft (TVD-based) and hard (label-based) disagreement, we conducted a controlled ablation on all three datasets. We fixed all other hyperparameters to (d = 20 z0zeros, {0.0, 0.25, 0.50, 0.75, 1.0]. This setup isolates the effect of A by ensuring that it is the only variable that changes.
0.05) and varied À
MNIST
CIFAR-10
PneumonisMMiST
Discrpancy
0.0065
0.0950
0.185
0.0060
0.0055
0.085
Ambiquity
0.11
0.010
0.10
0.008
0.19
0.011
0.11
VPR
0.135
0.10
0.009
Rush. Cap.
1.060
1.003
1.060
1.002
1.06
Roch. Raiio
LAKKIINIKUG
0.90
1000000000
0.75
0..00
1.00
Balancing weight (A)
Figure 5: Ablation over the mixing weight A. Across all datasets, diversity and Rashomon-set metrics change only marginally over X E 0.0, 0.25, 0.5, 0.75, 1.0. Median curves and IQR bands are computed over different 2g initializations of the FiLM latent vector.
As shown in Fig. 5 the X-ablation reveals that diversity metrics remain nearly invariant for most settings, yet a closer cross-datased comparison uncovers the beginnings of an architectural trend. Our three benchmarks form natural complexity hierarchy: MNIST (a shallow 3-layer MLP), CIFAR-10 (VGG-16), and PneumoniaMNNIST (ResNet-50). Recall that in the diversity score of Eq. (T)) setting = 0 makes the score depend only on hard disagreement, while À makes it depend entirely on TVD-basee soft disagreement; this score is used directly by CMA-ES to rank candidate latent vectors at each generation. We observe that model complexity increases, soft (TVD-based) and hard (label-based) disagreement begin to decouple. This effect is most visible in PneumoniaMNNIST. where higher À which place full weight TVD in the diversity score score-produce small but noticeable increase in diversity. In contrast, MNIST shows virtually no X-dependence, consistent with its highly overdetermined and near-deterministic predictions. These results suggest that deeper, nigher-capacio architectures support richer probability- variation that does not immediately translate into label flips, making soft disagreement increasingly informative. Consequently, À may become a more influential hyperparameter for Rashomon exploration in more expressive models, motivating future studies on wider CNNs and Transformer-baat architectures.
A.3 HARDWARE SETUP.
All experiments were run on workstation equipped with an NVIDIA GeForce RTX 4090 GPU (24 GB VRAM), Intel Core 19- 1490KK CPU, and 64 GB of system memory.
15

---

Published as a conference paper at ICLR 2026
A.4 IMPACT OF INITIAL STEP SIZE AND Z INITIALIZATION
We analyzed how CMA-ES hyperparameters influence Rashomon recovery. Figure 6 shows that the initial step size do affects the balance between exploration and precision: larger values promote greater diversity on simpler architectures such as MNIST, while smaller values are more effective on deeper networks, where fine-grained search helps maintain Rashomon membership. Figure illustrates the effect of latent vector initialization. Starting from the anchor & = 0 consistently yields Rashomon members across datasets, while initializations centered around stronger perturbations
(2 1) often fail to recover candidates, except on MNIST. These results confirm that both hyperpa- rameters shape the diversity and robustness of the discovered sets, underscoring the importance of initialization when exploring the modulation space.
0-1.1
a 0.3
0.4
0=0.5
MNIST
CIFAR-10
PheumoniaMNIST
Discrepancy
0.10
0.01
0.05
Aubiguit
0.3
0.15
0.10
0.2
0.06
0.05
VPR
0.1
0.1
0.0
0.0
0.0
Rash. Cap.
1.10
1.10
1.1
1.05
1.00
1.0
1.00
Itah.. Ratio
0.50
0.5
0.25
0.0
0.0
0.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Rashomon parameter (e)
Figure 6: Diversity metrics (discrepancy, ambiguity, Rashomon capacity, and Viable Prediction Range) and Rashomon Ratio across Rashomon thresholds (c) for varying step sizes (oa). The x- axis values are discrete: each group corresponds to the same level, and the horizontal spacing within groups is for readability only and carries no semantic meaning. Markers denote the median across error bars show the interquartile ranges (IQR), reflecting variability across latent vector initializations and CMA-ES initial step sizes (or) and latent dimensions (d).
A.5 LAYERWISE SENSITIVITY STUDY
To determine which FiLM layers contribute most to predictive disagreement, performed lay- erwise ATVD sensitivity analysis on the best-performing Rashomon set obtained for each dataset. Because each FiLM layer in reconstructed models is assigned explicit, architecture-spedt name (c.g.. film 1, f 1m conv 7, film res 12), we can unambiguously control, disable, and interpret individual modulation sites. These FiLM layers correspond distinet functional locations
16

---

Published as a conference paper at ICLR 2026
20seed 20seed2
#0serd3
a0seed17
Dneros
alseed0 MNIST
afsseed
#0eeed
CIFAR-10
PresnooiiaMNIST
Discrepaney
0.10
0.02
0.05
0.05
Ambiguity
0.3
0.10
0.2
0.10
0.06
0.1
0.05
0.2
VPR
0.1
0.1
0.0
0.0
Rash. Cap.
1.10
1.06
1.05
1.00
1.0
1.00
1.0
Rash. Ratio
0.50
0.5
0.5
0.25
0.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.06
0.01
0.02
0.03
0.04
0.05
Rashomon parameter (e)
Figure Diversity metrics (discrepancy, ambiguity, Rashomon capacity, and Viable Prediction Range) and Rashomon Ratio across Rashomon thresholds (e) for varying latent vector initializations. The seeds sampled around are: z0seed1, z0seed3, z0seed5, z0seed7, while: z0seed0, z0sced2, z0seed4 and v0seed6 are sampled around 1. The x-axis values are discrete: each group corresponds to the same e level, and the horizontal spacing within groups is for readability only and carries no semantic meaning. Markers denote the median across runs; error bars show the interquartile ranges (IQR), reflecting variability across latent vector initializations and CMA-ES initial step sizes (ap) and latent dimensions (d).
the base architecture: modulation of dense-layer activations in the MNIST MLP, modulation be- fore ReLU after each convolution in VGG16, and modulation between BatchNorm and ReLU or along residual branches ResNet50. This naming scheme ensures that the effect of turning off a particular FiLM site has clear architectural interpretation.
After CMA-ES identifies a Rashomon set using all FiLM layers, we take each member's latent and measure its baseline disagreement with the reference model (all FiLM layers active) using Total Variation Distance (TVD, Eq. (6)) We then disable one FiLM layer at a time, recompute predictions, and record the resulting change in disagreement. A positive ATVD indicates that the removed FiLM layer resulted in more disagreement (its modulation increased diversity), whereas negative value suggests the opposite
ATVD TVD FiLM-on
TVD I'Dayye-od
(12)
Since all FiLM layers share the same latent vector and initialization, differences ATVD arise solely from the architectural location of each FiLM site, making the sensitivity profile directly in-
17

---

Published as a conference paper at ICLR 2026
terpretable. Averaging ATVD across all Rashomon members aggregates the contributions of many different latent vectors, meaning that different models may rely on different FiLM layers to generate disagreement. Thus, the reported values capture the average marginal influence of each FiLM layer across the Rashomon set. This analysis does not account for potential interactions between FiLM layers (e.g., the joint effect of disabling layers 1 and 3), which leave interesting direction for future work.
Across architectures, disagreement is localized: early FiLM layers dominate in MNIST, mid-level convolutional FiLMs contribute most in CIFAR 10, and early to mid-stage FiLMs dominate in Pneu- moniaMNIST. Several deeper FiLM sites yield negative ATVD, indicating that late-stage modula- tions often stabilize rather than diversify predictions.
These findings provide structural interpretation of how FiLM layers influence multiplicity and suggest a promising direction for future work: using ATVD sensitivity scores to automatically select FiLM sites, allowing DIVERSE to focus CMA-ES search on layers that meaningfully affect functional diversity.
Diieesity-generating (+ATVD)
Diversity-suppresinng (-ATVD)
MNIST
0.006
0.002
nim.1
Rin.2
Rlm.a
CIFAR-10
ATVD influence
0.000
-0.002
Rimurome22
Fim cune
EimuzonesS
Alm.conv...
Elm canv 12
Rim
PneumoniaMNIST
0.005
0.010
Bln bn
Binalma
Rim AnL &
dlan hall
almn ilm--baa
baald
tim noe
Glan 19
almn lis 20
24
alm hoaao
rve
almubs 25
iitu roud4
ba ba dimamma
tilm bes 26
lite new 40
atan bout
las 45
Minu Minuresto recio
Hitu rou
attn bos
Min. ree
FiLM layer
Figure 8: Layerwise ATVD sensitivity across MNIST, CIFARIO, and PneumoniaMNIST. Positive
while negative bars (red) indicate layers whose removal increases agreement (diversity
bars (green) indicate FiLM layers whose removal reduces disagreement
18

---

Published as a conference paper at ICLR 2026
20 ones
20 20_seed_
20 20_seed_3 3
x0 seed.e 5 x0_seed.6
x0 _seed_
x0 zeros
20. -sedd.0
20.seed.2
20.seed_4
Vision Transformer CIFAR-10
Discrepancy
0.10
0.05
Ambiguity
0.2
VPR
0.10
0.05
Rash. ratio Rosh. cap.
1.050
1.025
0.50
0.25
0.01
0.02
0.03
0.04
0.05
Rashomon parameter (e)
Figure 9: Diversity metrics and Rashomon Ratio for the Vision Transformer across Rashomon thresholds (t), grouped by latent vector initialization. Each point shows the median; error bars indicate IQR across runs. Horizontal spacing within each group is for readability only.
A. APPLICABILITY ON VISION TRANSFORMERS
To further assess whether DIVERSE generalizes beyond convolutional and MLP- based architec- tures, we conducted an additional experiment on a Vision Transformer (ViT) (Dosovitskiy et al.) 2021). This test evaluates whether transformer, with deep stacking, multi-head self-attention, and nigh-diimessooal MLP blocks, can also be embedded into a FiLM-modulated latent space that CMA-ES can effectively explore.
For this experiment, we use compact ViT trained on CIFAR-I0. Images are resized to 72x72, split into small patches, and embedded into low-dimensional sequence representation. The encoder consists of 8 transformer blocks with 4-head self-attention, followed by a lightweight MLP classifi- cation head. The reference ViT is trained with standard supervised learning using AdamW. a batch size of 256, and up to 75 epochs, achieving 73% test accuracy before FiLM modulation.
To apply DIVERSE, we wrap the ViT with FiLM layers following the same dense-layer insertion rule shown in Fig. Concretely, every dense layer inside the eight transformer MLP blocks, as well as the dense layers in the classification head, FiLM-modulated. Attention layers, patch embeddings, and positional encodings remain unchanged in this study, though modulating these components presents an interesting direction for future work. As in all other architectures, every Rd defined in Eq. (7). CMA-ES is exactly as described in Section 4 For this initial test, we evaluate only d 2, as the goal is to verify architectural generality rather than perform exhaustive hyperparameter study.
FiLM layer shares the same latent vector
19

---

Published as a conference paper at ICLR 2026
this configuration, with initial step size do = 0.1, Rashomon tolerance = 0.05, and initialization at zerss... The resulting timings broadly comparable to those of VGG16 reported in Table T with the ViT being slightly slower, possibly due to its deeper sequence- processcg architecture.
All VIT runtime measurements (reported in Table
Across different Zg initializations, the resulting Rashomon exhibit diversity metries comparable to those obtained with VGGI6 with = (Fig. 2 However, both the diversity outcomes and the timing measurements should be regarded as preliminary: more extensive experimentation is needed to fully validate performance on transformer-basse models, including exploring larger latent dimensions and selectively modulating only the FiLM layers that meaningfully influence diversity (as suggested by the sensitivity analysis in Appendix A.5p. Nonetheless, this experiment provides encouraging evidence that DIVERSE can operate on transformer-based architectures as well.
Table 2: Runtime (hh: to obtain m candidate models for the Vision Transformer under the same Rashomon constraint.
Vision Transformer
Method
m=162
m==40
DIVERSE
00:02:16
00:08:52
A.7 BEST PERFORMING HYPERPARAMETER FOR BASELINE COMPARISON
The hyperparameter settings used for comparing DIVERSE against the retraining and dropout base- lines are reported Table For each dataset and Rashomon tolerance e, we select the configuration that achieves the best average performance across all multiplicity metries: discrepancy, ambiguity, viable prediction range (VPR), Rashomon Capacity (RC), and Rashomon Ratio. The hyperparam- eters considered the latent modulation dimension d, the CMA-ES initial step size du, and the latent initialization Zg.
Table 3: Best performing hyperparameter settings for each dataset across the five Rashomon toler- ances
MNIST
CIFAR-10 (VGG)
PneumoniaMNIS" (ResNet-50)
d
20
d
20
d
2o
0.05
64
0.2
z0. (e_zeros
8
0.1
z0_seed_5
4
0.1
z0 seed_5
0.04
32
0.2
z0 20_seed_5 5
8
01
z0 zeros
0.2
z0 seed_5
0.03
32
0.2
z0_seed_7
4
0.1
z0 zerzeros
0.1
z0_zeros
0.02
64
0.1
z0.zeros
2
0.1
z0 zeros
2
0.1
z0_seed_7
0.01
32
0.1
z0 zeros
2
0.1
z0 zeros
0.1
z0 zeros
A.8
LLM USAGE
Large language models (LLMs) were used in the preparation of this paper exclusively general- purpose assistive tools. They supported tasks such as grammar correction, improving readability, and rephrasing for clarity. L.MS were not used to generate original research ideas, design experiments, produce results, fabricate citations. All technical contributions, experimental designs, analyses, and conclusions in this paper were conceived, implemented, and validated by the authors. The authors remain fully responsible for the accuracy and integrity of the content.
20


Reviews:
[{"review_id": "QxXfvYBsCy", "reviewer": "Reviewer_mafH", "summary": "The authors introduce DIVERSE, a framework for exploring the Rashomon set of deep neural networks, innovative way to find high quality and diverse models that match a reference model's accuracy but differ in their predictions. DIVERSE adds Feature-wise Linear Modulation (FiLM) layers to a pretrained model and uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search a latent modulation space, producing diverse model variants without retraining or gradients. On MNIST, PneumoniaMNIST, and CIFAR-10, DIVERSE finds multiple high-performing models that behave differently", "questions": "Why CMA-ES and not any other evolutionary algorithm?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Innovative approximation of Quality-Diversity evolution\n- Well written paper", "weaknesses": "- Limited Ablation study on evolution side.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7T7oltaviB", "reviewer": "Reviewer_CZF8", "summary": "The paper adresses the problem of exploring the Rashomon set of a trained machine learning model in a cost-efficient and diverse manner. The $\\epsilon$-Rashomon set is the set of all machine learning models that reach the same empirical risk as the reference model with a tolerance $\\epsilon$. In this set lies a multiplicity of different models that may produce different predictions for identical individuals. Being able to explore $\\epsilon$-Rashomon sets model is interesting to construct diverse ensemble of machine learning models for uncertainty quantification or improved predictive performance.\n\nAs exploring the $\\epsilon$-Rashomon set of a deep neural network can be compute intensive with naive methods (retraining from scratch or exploring the whole parameter space), this paper proposes to explore a subspace of the $\\epsilon$-Rashomon set using Feature-wise Linear Modulation (FILM), a low-dimensional parameterized transformation of a neural network. In this low dimensional space, a black-box and derivative free optimization algorithm is used (CMA-ES) for exploration. The constraint on exploring models included in the $\\epsilon$-Rashomon set is relaxed by adding a penalization in the objective function minimized by CMA-ES.\n\nThe proposed approach (DIVERSE) is then evaluated on three small to medium scale image classification datasets. Empirical results show that DIVERSE can be a good compromise between exploration and compute. Extensive experiments and ablation studies are conducted to evaluate the impact of each introduced hyperparameters.", "questions": "Has Rashomon set exploration been done for different training tasks such as regression?\n\n### Remarks\nThe columns of Figure 2 and 3 are not in the same order.\nThe impact of $\\lambda$, the mixing coefficient between soft and hard agreement is not evaluated in the ablation study.\nMore diverse datasets could be used in the experimental protocol. MNIST (and I think PneumoniaMNIST also) is quite an easy dataset where a linear model trained on the raw pixel can achieve very high accuracy. Even though not ideal, Fashion-MNIST and K-MNIST might be better alternatives. MNIST-1D, even though not a image classification dataset but a time-series classification one, could be a potential candidate, as linear model have poor performances on it.\n- Fashion-MNIST : Xiao, Han, Kashif Rasul, and Roland Vollgraf. \"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.\" 2017.\n- K-MNIST : Clanuwat, Tarin, et al. \"Deep learning for classical japanese literature.\" 2018.\n- MNIST-1D : Greydanus, Sam, and Dmitry Kobak. \"Scaling down deep learning with mnist-1d.\" 2020.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with a clear objective, all notions are introduced clearly.\n2. The proposed approach has the potential to be a fundamental tool in many domains of machine learning.\n3. The proposed approach is sound and well grounded in the literature.\n4. The ablation study is quite furnished and the experimental protocol sound", "weaknesses": "1. The paper lacks qualitative or illustrative experiments to helps the reader gain intuitions on the significance of the results.\n2. The paper lacks experiments with downstream tasks (uncertainty quantification, ensembling, ...) to better asses if generated models with DIVERSE are actually effective for practical tasks.\n3. The paper lacks quantification about how smaller the explored set of models induced by FiLM is to the true $\\epsilon$-Rashomon set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wg3bW6RMXX", "reviewer": "Reviewer_HCic", "summary": "The authors introduce DIVERSE, a novel method to find other models in the ***Rashomon Set*** of a reference model. A *Rashomon Set* is defined as the set of models that have similar predictive accuracy on a given task.\n\nThe DIVERSE algorithm is based on two main components: FiLM layers which apply affine transformations to pre-activations, and CMA-ES for gradient-free optimisation of the parameters of the FiLM layers. Since CMA-ES is not scalable to high dimensions, DIVERSE uses it to optimise a low-dimensional vector \\$z\\$, which is then projected to higher dimensions using random fixed matrices.\n\nThe authors evaluate their method using both prediction-based and probability-based metrics, in all cases after the last layer. They compare against two baselines: re-training using different seeds, and dropout-based Rashomon exploration. In terms of diversity, the generated models are generally less diverse than with full re-training, but for a runtime orders of magnitude lower.", "questions": "Q1: The definition in the Introduction defines the Rashomon Set as the set of models that achieve a similar performance on a same task, whereas in Equation (1) this set is constrained to a hypothesis space of models parametrised by weights \\$w \\in \\mathbb{R}^p\\$. Could the authors please clarify whether the Rashomon Set is constrained to models that use the same architecture?\n\nQ2: In Table 1, it is unclear which size \\$d\\$ is used for the vector \\$z\\$. Since CMA-ES struggles with higher dimensionalities, how do higher values of \\$d\\$ impact the runtime of DIVERSE? Is there a trade-off between diversity, Rashomon ratio and runtime as \\$d\\$ increases?\n\nQ3: The paper only explores using DIVERSE on CNNs. Would DIVERSE be applicable to Transformers? If yes, demonstrating this applicability in the paper could further strengthen it. If not, what are the potential challenges?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This paper proposes a relatively simple, inexpensive and elegant solution to the Rashomon Set exploration problem.\n\nS1: The proposed method is relatively simple, yet effective. This makes it very relevant to solve the Rashomon Set exploration problem.\n\nS2: The evaluation of the algorithm is good, with metrics covering both class predictions and output probabilities. Furthermore, multiple hyperparameters are explored and DIVERSE is compared to existing baselines.\n\nS3: The paper is generally clear and well-written. The introduction effectively contextualises the Rashomon Set problem, experimental setup and results are generally clear.", "weaknesses": "I think that the paper is overall quite solid. However, I believe its main weak points are related to its impact and motivation.\n\nW1: Based on the Introduction and the Conclusion of the paper, I do not understand why Rashomon Set exploration is an important problem to solve. I would like the authors to better motivate why their work is impactful for research in Machine Learning.\n\nW2: Furthermore, I think the results insufficiently show that DIVERSE is better than dropout-based Rashomon exploration. In particular, dropout-based Rashomon exploration is less computationally expensive than DIVERSE, and shows better performance in most cases for PneumoniaMNIST. I believe the authors should better highlight where their method outperforms the existing baselines, and should clarify under what conditions DIVERSE is preferable, such as specific datasets, architectures, or computational constraints.\n\nW3: For the experiments comparing DIVERSE with its baselines, it is unclear to me what hyperparameters are being used. I think this should be described in the experiment setup.\n\nW4: This is a minor point, but I believe it would be preferable that the metric mathematical definitions, currently in Appendix A.1, be integrated to the main text. This would improve the clarity and readability of the results Section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XP4ANgJ4zM", "reviewer": "Reviewer_gMcV", "summary": "This paper introduce a gradient-free method called  DIVERSE to explore the Rashomon set of deep learning models. This set contains models with similar accuracy but different predictive behaviors. DIVERSE takes a pre-trained model and augments it with \"Feature-wise Linear Modulation\" (FiLM) layers. DIVERSE then uses a search algorithm \"Covariance Matrix Adaptation Evolution Strategy\" (CMA-ES) to find different model variations without needing to retrain them from scratch.  The experiments show that DIVERSE can uncover multiple high-performing and functionally distinct models efficiently. It offers a competitive way to explore the Rashomon set, achieving comparable diversity to retraining but at a much lower computational cost.", "questions": "Which layers contribute most to disagreement. Please provide a layerwise or stagewise sensitivity analysis of FiLM norms versus diversity.\n\nHow portable is the method across backbones with similar top line accuracy. A compute matched comparison on two architectures would clarify whether results are model local or architecture agnostic.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\nThe paper reframes latent exploration as Rashomon set exploration in deep networks. It searches a bounded FiLM modulation space around a fixed model and balances an accuracy tolerance epsilon with explicit control over behavioral diversity. This perspective differs from weight generation or full retraining by focusing on efficient, local, and controllable exploration around a single reference model.\n\nQuality:\nThe methodology is clearly specified and reproducible, with objectives, constraints, and data splits stated in enough detail. The experimental design is sound for the stated goals and uses appropriate datasets and baselines. The evaluation employs complementary metrics including Rashomon Ratio, discrepancy, ambiguity, VPR, and RC, which together provide a comprehensive view of diversity under an accuracy constraint. The analysis includes sensitivity to key hyperparameters and highlights dataset dependent effects.\n\nClarity:\nThe paper is clearly written and easy to follow. The flow from problem setup to method and experiments is logical. Notation is consistent, and the figures make the FiLM based search space and the role of the latent variable z intuitive.\n\nStrengths:\nThe approach is practical and training free for a given reference model, which makes audits of the local Rashomon set feasible under realistic compute budgets. The joint use of decision level and probability level metrics supports a nuanced interpretation of disagreement. By mapping accuracy constrained behavioral variants, the method helps characterize the local performance diversity landscape of a trained model and can inform stress testing, ensembling, and selective prediction. The compute footprint is small compared to retraining, enabling broader exploration on larger models and datasets.", "weaknesses": "Scaling of the search. The method relies on full-covariance CMA-ES over FiLM latents, which does not scale well as dimensionality increases. This limits exploration on deeper or more complex models and constrains the approach’s practical reach.\n\nArchitecture locality. Because FiLM layers are inserted into a specific pretrained network, the results are tied to that architecture. It is unclear whether conclusions transfer across backbones with comparable accuracy, and cross-architecture comparability is not established.\n\nObjective agnostic to why models disagree. The fitness targets disagreement under an accuracy tolerance but remains insensitive to the underlying cause. As a result, discovered variants may be superficial perturbations rather than models with meaningfully different reliance on features, robustness characteristics, or fairness profiles.\n\nExperimental scope and baseline breadth. Experiments are confined to image classification on moderate-scale datasets. The behavior of the Rashomon set may differ substantially on larger benchmarks (e.g., ImageNet) or in other modalities (e.g., NLP with attention). Moreover, “retraining” is treated as a single comparator despite encompassing diverse regimes, leaving the trade-off landscape underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)