Adaptive Conformal Prediction via Mixture-of-Experts Gating Similarity
Authors: Jingsen Kong
Wenlu Tang
Dezheng Kong
Linglong Kong
Guangren Yang
Bei Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: Conformal prediction; Mixture of Experts; Distribution-free inference.
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Mixture-of-Experts Conformal Prediction (MoE-CP)

Abstract: Prediction intervals are essential for applying machine learning models in real applications, yet most conformal prediction (CP) methods provide coverage guarantees that overlook the heterogeneity and domain knowledge that characterize modern multimodal datasets. We introduce Mixture-of-Experts Conformal Prediction (MoE-CP), a flexible and scalable framework that uses the gating probability vectors of Mixture-of-Experts (MoE) models as soft domain assignments to guide similarity-weighted conformal calibration. MoE-CP weights calibration residuals according to the similarity between gating vectors of calibration and test points, producing prediction intervals that adapt to latent subpopulations without requiring explicit domain labels. We provide theoretical justification showing that MoE-CP preserves nominal marginal validity under common similarity measures and improves conditional adaptivity when the gating captures domain structure. Empirical results on synthetic and real-world datasets demonstrate that MoE-CP yields more domain-aware, interpretable, and often tighter intervals than existing conformal baselines while maintaining target coverage. MoE-CP offers a practical route to reliable uncertainty quantification in latent heterogeneous, multi-domain environments.

Published as a conference paper at ICLR 2026
ADAPTIVE CONFORMAL PREDICTION VIA MIXTURE- OF- OF-EXPERTS GATING SIMILARITY
Jingsen Kong*l.2 Wenlu Tang"?, Dezheng Kong", Linglong Kong", Guangren Yang" && Bei Jiang
l Jinan University, Guangzhou, China
University of Alberta, Edmonton, Canada bjinsen(111@sttu2 (wenlu4, dezheng2, peil, 1kong (kongg@ualll www.laalllabaaacooommomoo
inu, tygrê jnu eedu. edu. en
ABSTRACT
Prediction intervals are essential for applying machine learning models in real applications, yet most conformal prediction (CP) methods provide coverage guar- antees that overlook the heterogeneity and domain knowledge that characterize modern multimodal datasets. We introduce Mixture-of-Expert Conformal Pre- diction (MoE-CP), a flexible and scalable framework that uses the gating prob- ability vectors of cottuure-of-Experrt (MoE) models as soft domain assignments to guide monilarity-weighhee conformal calibration. MoE-CP weights calibration residuals according to the similarity between gating vectors of calibration and test points, producing prediction intervals that adapt to latent subpopulations without requiring explicit domain labels. We provide theoretical justification showing that MoE-CP preserves nominal marginal validity under common similarity measures and improves conditional adaptivity when the gating captures domain structure. Empirical results synthetic and real-world datasets demonstrate that MoE-CP yields more domain-awand awwar, interpretable, and often tighter intervals than existing conformal baselines while maintaining target coverage. MoE-CP offers practical route to reliable uncertainty quantification in latent heterogeneous, multi-domain environments.
INTRODUCTION
Modern machine learning increasingly handles heterogeneous and latent multi-domain data. By heterogeneous, we mean data whose characteristies such as noise level, feature distribution, or la- bel variability that differ across regions of the input space. By multi-domain, we mean that these differences often align with implicit or explicit groups (Hahnloser & Seung. 2000), such as pa- tients from different hospitals, users across geographic regions, or sensor modalities in autonomous systems. Specifically, such domains are not always labeled in advance, and they emerge as latent aub-populations embedded in the data. In these settings, a single "one-size-fitse aill-all" predictive un- certainty such as standard conformal prediction (Vovk et al. 2005 Lei et al. 2018)) is not adequate. Intervals that are valid average may be overly wide in cowvvariacce domains and narrow in high- variance ones. Reliable prediction intervals must therefore adapt to the underlying domain structure in both observed or latent patterns while preserving rigorous coverage guarantees.
To address this challenge, we introduce Mixture-of of--xxerts Conformal Prediction (MoE-CP), a new method that combines conformal prediction with the representational power of Mfxture-of-Expeee (MoE) models (Jacobs et al. 1991). MoE architectures decompose complex prediction tasks into a set of specialized expert functions, each responsible for modeling sub-regime of the data. gating network assigns probability vector over experts for each input, effectively acting as soft domain classifier. This structure has proven highly successful in scaling large language models (Shazeer) et al. (017). multimodal learning (Nguyen et al., 2023). and domain adaptation, precisely because it can capture heterogeneous or dommi- structure without explicit domain labels. In our setting, the gating vector plays central role that it provides natural similarity measure that identifies which
"Equal contributions in alphabetical order "orresponniig author

---

Published as a conference paper at ICLR 2026
calibration points belong to the same latent regime test point. By weighting calibration residuals according to this similarity, MoE-CP tailors prediction intervals to the local predictive regime of the test input. In effect, the MoE component solves the challenge of latent domain identification, while the conformal component ensures rigorous coverage guarantees. Together, this integration yields intervals that are both valid and adaptively sensitive to heterogeneous, multi-domain data. We show the workflow of our framework in Figure I
This modification fundamentally adapts conformal prediction to data complexity. In heterogeneous or multi-domain datasets, test points may lie in regions with very different conditional variances or expert specialization. MoE-CP adjusts intervals using the most relevant calibration points, becoming wider in noisy regions and narrower where predictions are reliable. In multi-domain data, the gating distribution naturally links different domains, so intervals adapt across domains without manual splitting. This work is related to randomly localized conformal prediction (RI.PP) (Hore & Barber, 2025), where calibration points are chosen according to their proximity the covariate space. By contrast, MoE-CP identifies relevant calibration points based proximity in a learned, label-aware latent regime space defined by the gating probabilities the mixture-of-experts. This makes the intervals not only valid, but also sharper, more flexible, and easier to interpret than other local weighted conformal methods, see (5 Remark
In this work, also provide theoretical analysis showing that MoE-CP (i) preserves marginal validity when using KI. or cross-entropy divergences, (ii) retains asymptotic validity under a wide class of similarity measures, and (iii) approaches conditional coverage under mixture representa- tion under some conditions. Empirically, we show that MoE-CP consistently improves efficiency and adaptivity across synthetic experiments and real multidomain data.
In summary, we highlight our contributions as follows.
We propose MoE-CP, a conformal prediction framework that leverages MoE gating vectors to adapt intervals to local domain structure.
We establish theoretical guarantees for coverage under KI., cross-entropy, and general di- vergences, as well as approximate conditional coverage under a mixture representation.
We demonstrate empirically that MoE-CP yields domain-adaptive intervals compared to existing conformal methods, particularly heterogeneous and multi-domain settings.
Training
Mixture of Expert Model
Test point
Calibration
Galnng
Weighted score distrbuttion
Riby Rio faz
(ala)
MoE Conformal Interval
Rescetu
p(x) D (x)
Random
(X,,.
R: R: OeIggN
Similarity-baas based Weight
D (F (X.+s) (X.)]
Similar expert Highh
Temperata 7
Control the spread
"Peacetul and actorable unicom!
Different expertl
Figure 1: A MoE model assigns each input to experts through a gating network, producing both predictions and soft domain probabilities. Calibration residuals are weighted by the similarity of gating vectors. The final conformal interval adapts automatically across domains.
2

---

Published as a conference paper at ICLR 2026
2
METHODOLOGY
2.1 PROBLEM SETUP AND NOTATIONS
We work in supervised prediction setting where the goal is to produce valid prediction intervals for unseen data, potentially coming from multiple latent domains. Let
D Drain ((,, Y) Den = {(X,, YO)
be the training and calibration set respectively. Both are drawn independently and identically dis- tributed (i.i.d.) from an arbitrary distribution P that contains K latent domains. prediction set č, (Xx+1) is called marginally valid interval if for any new test point (Xn+1, Yn+1) ~ P.
P(Y,+1 (Xn+ 2 a.,
where E (0,1) is the desired miscoverage rate, and the probability P is over the randomness in training, calibration, and test sampling. While marginal validity is standard target in conformal prediction, it does not guarantee coverage for each specific covariate z. The stronger property is conditional validity, that is, for any N,
P[Y++1 (x(+1))) Xn+1 x} 2 a, Vr.
In practice, exact conditional validity unattainable without strong assumptions (Lei & Wasserman, 2014 Barber et al.) (02T), but it motivates our method: assigning higher influence to calibration points that are most similar the test point.
2.2 MIXTURE-OF- (MoE) PREDICTOR
We consider the observed data are input-output pairs (X;, Y,) E X' R, which in practice may arise from latent and unknown multi-domain structure. However, these domain assignments are not observed. What we see (X,, Y,) is drawn from overall distribution P, which we treat exchange- able. Thus, although the settaageneratngg mechanism inherently heterogeneous and multi-domain, the observed dataset appears single mixed population.
To model this heterogeneous and multidomain data, we use a Mixture-of- -fispere-of-Experts (MoE) architecture. The MoE partitions the input space into K sub-regimes and assigns each to specialized expert function (X) X' + R. Given an input X, gating network produces probability vector
K Σ
TXX) = (TI(X), ..... (x())) E 1],
TX)
1,
where each (X) that approximates the latent domain probabilities +X( which reflects the de- gree to which expert is responsible for predicting X. Each TH (X) is parameterized via softmax:
exp(4x(X))
(X) = Σ eX)({,(X))
where (X) is a logit score for expert of given X, from a Multi-layer perceptron (MLP) or transformer block. The expert functions P4 and gating logits la are trained jointly on the training dataset Dunin to minimize a loss function such as mean squared error. The choice of K is discussed in Section A.3 The MoE prediction is then defined the weighted average of each expert:
M(X) Σ Th (X)))x((X) k=1
where 2 (X) denotes the output the k-th expert model, and (X) E [0, is the gating prob- ability assigned to expert l at input X defined by (I) After training, each prediction ja() comes with an interpretable domain probability vector (œ)). This allows us to exploit domain structure in data-driven way when constructing adaptive conformal prediction intervals.
3

---

Published as a conference paper at ICLR 2026
2.3 MOE-BASED SIMILARITY WEIGHTED CONFORMAL
We now describe how to construct valid and adaptive prediction intervals using the domain structure captured by the MoE model. Our approach extends standard conformal prediction by assigning higher importance to calibration points that share similar domain affinities with the test point, where similarity measured in the MoE's learned gating space.
Given the trained MoE model from Section 2.2 we first define a nonconformity score S for each calibration point. While conformal prediction can be applied with many different scores, in this work we use the absolute residual:
= - A(X)), (x), E Deal.
where ((X,) = E (x) is the MoE's point prediction for X, and TT represents the model's gating probability that input X; belongs to expert/domain k. The MoE also outputs a vector π (X.) = ( (X:),...,, TK (X:)).
While we focus on absolute residual-based scores (Lei et al. 2018) for elarity, our MoE-hhighted framework is general and easily extensible to other conformity scores, such quantile (Romano iet al. 2019) or distributional conformity (Chernozhukov et al., 2021a) scores. These extensions would allow MoE-CP to adapt not only to mean prediction errors but also to richer forms of predic- tive uncertainty, further enhancing its applicability in heteroscedastic or multi multi- environments.
Remark In heterogeneous or multidomain datasets, calibration points are not equally informative for every test point. Two points with the same covariates X distribution but different domain label may exhibit different residual distributions. The gating vector x (x) acts as a soft domain assignment, capturing the model's ss internal partitioning of the input space. Thus, calibration points with similar T (.) to the test point are statistically closer in the model's 's latent space, and their residuals provide a more accurate basis for interval calibration.
Randomized gating vector for validity. For a test input Xu+1, we first obtain its gating vector (X,+1). To ensure coverage validity, we introduce randomization following the principles of Zhhng & Candès 2024) and Hore & Barber (2025) Specifically, we generate:
L Multinomial (T, (X=+1)), tt (Xm+1)== I
where E N temperature parameter controlling the randomization magnitude such that small r introduces more randomization and involves greater variability in weights, while large 7 will let approach (X,+1) and introduce less variability. This randomized 7 (X,+1) plays the same role as randomized covariates in localized conformal prediction (Hore & Barber 2025) but is computed in gating space rather than the raw input space.
Similarity- based weighting, Our key idea of the weighting lies on the fact that calibration points with similar gating vectors π (zx) to the test point should have more influence. And thus we formalize similarity via divergence between probability vectors. Given म (Xn+1) and each gating vector v (X,), we compute similarity-based weights:
i = =xp
exp |-r
D (# (Xn-1), n (X,))),
(4)
where D(.-)) is a divergence between probability vectors. The weights are then normalized across calibration points Deat {(X,, Y,), E Ima (1,...,,) .n)) and the test point:
w.
W +++ j= mj
i Teal U U
(5)
This normalization ensures that weights form probability distribution over the calibration set plus the testing point (Tibshirani et al. 2019 Barber et al. 2023).
Remark 2 Common choices of D(..)) include Kullback- Leibler divergence, cross-entropy, cosine distance, Euclidean distance. It aims to measure the similarity between *(Xn+1) and <(.)). Larger similarity between म (X,41) and (X.) results in smaller divergence, and thus carry out higher weight wi.
4

---

Published as a conference paper at ICLR 2026
Remark 3 Although there is no well-established existing works choice of show some prac- tical implications to guide our choice. If T(()) is accurate and stable, we prefer larger T to ex- ploit fomain--peciife residuals for tighter intervals. However, if π(x)) is noisy or high-variance, smaller r will be chosen to avoid over-committing to potentially spurious domain assignments. As extreme cases, setting - reduces the method to standard split conformal prediction with uniform weights, while letting + oo places essentially all weight on the test point itself.
MeE-weighted conformal interval.
The final WeE-weighted conformal prediction interval is:
m
0, + un 10-10+00 )}
GMaE (Xn+1)= = y E R: ly- A (Xn+1)) S Q1-a Σ
where Q1-a()) (.) is the weighted empirical (1 - a >quantilee of the nonconformity scores. The term 54000 corresponds to the standard conformal adjustment ensuring coverage (Tibshirani et al.) 2019 Barber et al., 2023). This approach yields adaptive, valid, and interpretable prediction intervals that reflect the multimodal similarity structure inherent in the gating mechanism of MoE. is particularly effective when training examples are heterogencously distributed across latent regimes defined by multimodal signals. Algorithm| iis is the workflow of our MoE-weighted conformal framework, and the full pipeline of the framework is provided in Appendix A.1
Algorithm 1 MoE- Weighted Conformal Prediction
Require: Derain: Deal, Xn+1, S, K, D, T, a
Ensure: (X
1: Fit MoE:
learn experts and gate {uu: Th on Derain by []: let (ix) Σ To (x
2: Calibration: for i E Teal, compute 8, = S(x,, Y.) and n. π (X,) = (TI (X.)..... me (X:)).
3: Randomize: compute πn+1 = ((n+1)); draw and π by
4: MoE weights: For i E Tcal U U + 1}. let w, exp( - n. ) normalize w; by 5
5: Quantile: form F = Eiez.n W; 8r, + unt1 d+ and let Q1 be its (1 a) quuuntlle..
6: return CNoEE (MMoB((X++1 (X++1) {y: S(X 9) S Q1-a).
Remark 4 The MoE-CP framework is flexible: users may specify the divergence measure, the num- ber of experts K, and the temperature т. Our theoretical and empirical results show that MoE-CP is largely insensitive to these choices and admit simple practical defaults (see Appendix A.3 for guidelines and Appendix 30..3dix(A.4.3111 for empirical ablation results). In practice, we recommend using the KL. divergence by default (it additionally satisfies exact marginal validity in our analysis); choosing a small K (e.g., K = 2, 3) when the training/calibration sample size is modest (e.g., n. < 1500) and allowing larger K when large amount of data is available; and setting to moderate value (e.g., T E [100, 300) to balance adaptivity and effective sample size. When needed, standard model- selection procedures such as cross- validaatin or an over over-parani sere-paammterized gating network with sparsity regularization can be used to select or automatically prune K.
Remark 5 The core idea of our construction is to assign larger weights to calibration points whose gating distributions are close to ग (((++1). an estimate of the true gating probability π (Xn+11) A randomization step is included to guarantee marginal validity (see the next section for theoretical details). The novelty of our framework is to define conformal weights via MoE gating probabilities, which-by leveraging the strong predictive performance of MoE models-yields improved adaptivity and efficiency. Related localized conformal methods GGuan 2023 Hore & Barber 2025) weight calibration points by the Euclidean distance between covariates; such an approach may fail to identify latent domains or heterogeneity, and may perform poorly in digh-dimensional settings where many features are irrelevant. Zhang & Candès (2024) instead fit a mixture model on the conformity scores to obtain weights, but this requires refitting the mixture for every candidate Yn+1 computationally expensive. By contrast, our weights are computed from an MoE. gating network trained once on the training data, making the procedure both adaptive and computationally efficient.
which
Remark 6 The MoE-CP admits several natural extensions. It can be paired with alternative confor- mity scores; for example, one may use the conformalized quantile score (Romano et al.) 2019 Sesia
& Romano 202 : S(X, Y) max max ((a,/( (X) Y. Y 91 91-a/2( (X)}, where 4a.22 and q1 ()
5

---

Published as a conference paper at ICLR 2026
lower and upper quantile regressors jointly estimated within MoE model. The framework also
extends naturally to online and time-series settings (Gibbs & Candes 2021); ; for instance, one can combine MoE- CP with time-aware MoE architectures (e.g., Tome-MoE (Shi et al. (022 to produce adaptive, sequentially valid prediction intervals. Further details are given in Appendix A.6
THEORETICAL RESULTS
We now present theoretical guarantees for the proposed MoF-weighted conformal prediction. Our goal is to show that the method preserves valid coverage while adapting to domain similarity as reflected the gating space.
Theorem Suppose that (X,, Y.), E [m+1], are exchangeable. For the Kl.-based or cross-entropy based divergence, and for any T E N+, the MoE weighted conformal interval satisfies
P(Y++ C (Xn+1)) #(x++1)) 2 -
Marginalizing the randomization # (Xn+11), the MoE-weighted conformal interval is therefore marginally valid:
P(Y E EMoE (X.- 2 -
Theorem I shows that when the divergence used in weighting is KL divergence cross-entropy, the constructed interval has the standard conformal property that covers the label of a new test point with probability at least a. Importantly, this holds conditional the randomized gating vector # (1) and remains valid after marginalization. This result ensures that our method retains the baseline of conformal prediction even after introducing MoE-based similarity weighting. We next consider divergences other than KL. To formalize this, we impose the following mild assumption.
Assumption Denote Wi KL t, as the KL-based weight, as T oc, the similarity measure D(,,)) satisfies WiD = KL + o(1)
Assumption D holds for any divergence measure that shares the same unique minimizer as the KL divergence; this claim is formalized in Proposition [ in Appendix A..2 In particular, provided sufficiently large calibration set, the assumption is satisfied by many divergences, e.g., Jeffreys di- vergence, Hellinger distance, Euclidean distance, etc. See Proposition [ a detailed discussion.
Theorem Under the assumption in Theorem 1 for any similarity measure D(...) satisfying As- sumption 7 and assuming the nonconformity score S has continuous distribution, the MoE con- formal interval based on D(,,)) satisfies
P(Yn+1
C (MoE (X ((x+11) 2 - -
Theorem states that even if KL divergence replaced with other similarity measures, the resulting intervals still maintain the conformal coverage property up to vanishing enror term. In other words, validity is robust the choice divergence, while the choice of D may influence efficiency. Finally, analyze the case where the nonconformity scores have latent finite mixture representation.
Theorem Under the assumption Theorem 7 suppose that the nonconformity score satisfies S ~ Ek Th (X,)), where Tg(X (X)) ) the true domain probability and f% is the score distri- bution within domain k. Define the oracle MeEghteghted conformal interval using the true gating probabilities:
e Mo (Xn+1)=
VER: S(X = Q1
Σ
i ös. + N. 1
where the oracle weight wł is defined as
cxpl-TD(x** (X (Xn+1), (X,))] i E exp (X (Xn+1), T² (X,))] (X,))]
ie In 1).
for any similarity measure D(..) and T E N+. Then,
P(Y+11 C.
AMEE, (MMo.+(Xx+t)) Xa+1) P(Y++1 C GMo,E (Xn+1) (Xu+1)).
nt (NoE..(Xo+1)

---

Published as a conference paper at ICLR 2026
Theorem B shows that if the true domain probabilities (XX) were known, then coverage conditional on the covariates X equivalent to coverage conditional on the domain probability vector т ((X). This implies that our method achieves approximate conditional validity whenever the learned gating vector ((X) is close to the true π² (X). In practice, the better the MoE captures domain structure, the closer approach conditional coverage.
4 NUMERICAL EXPERIMENTS
In this section, we conduct synthetic data and real-data experiments to demonstrate the performance of the proposed method. Due to space constraints, details of the experiments and comprehensive results are provided in the Appendix A.4
4.1 SYNTHETIC DATA EXAMPLE
The synthetic data (X, Y) are drawn from a three-domain distribution:
X t
p P1;
Y
X2 + p P25
X3 + p= P3i
where X Uniform([-3,3),
N(0, 1), and p;. 1,2,3, are the proportions of each domain
that sum up to 1. We consider two settings: (1) Unbalanced proportions: pl = 0.2, = 0.3. Pa = 0.5; (2) Balanced proportions: P1 = P2 P3 1/3. The sample size for training, calibration, and testing are set equally to be n = 500. The experiments are repeated over 50 times. The miscoverage level is set to be
0.1. We denote the MoE weighted conformal prediction as MoECP. The divergence measure in MoECP is the KI. divergence. The number of experts K and the temperature for MoECP are set be K and 150, respectively. Several state state-of-the of-the-art methods are compared, including CQR (Romano al. 2019). CHR (Sesia & Romano 2021). SCP+CC (Gibbs) jet al. 2025), RLCP (Hore & Barber) 2025), and PCP (Zhang Candès 2024). Details of experimental setup provided in the Appendix A.4.I
1.0
27.5
Coverage rate
Interval Langth
22.5
17.5
0.7
12.5
CQR
CHR
SCP+CC
RLOP
POP
McEOP
CQR
CHR
SCP4C0
RLOP
PCP
McEOP
Method
Method
Figure 2: Marginal coverage and interval length for the setting 1 (unbalaaced proportions) in syn- thetic data over 50 experiments,
Here, we report results for the unbalanced setting: results for the balanced setting appear in the Appendix (.4.2 shows marginal coverage and interval length for competing methods. All methods satisfy the coverage guarantee, but PCP is conservative. Our method, MoECP, yields the shortest prediction intervals while maintaining coverage. Notably, although MoECP based on mean regression, it outperforms two quantile-rggression-beee though quantile regression typically more adaptive to heterogeneity. In addition, MoBCP is more efficient than the mean-regression baased methods SCP+CC, RLCP, and PCP.
approaches (CQR and CHR),
To highlight the differences among methods, Figure 3 presents the conditional coverage for com- peting approaches in representative experiment. The coverage each data point is computed by averaging over its 100 nearest neighbors. Among all methods, MoECP achieves conditional cov- erage closest to the nominal level. Figure further displays the corresponding prediction intervals.

---

Published as a conference paper at ICLR 2026
NcECP
1.0
Conditons Coverace
0.9
0.7
0.6
0.5
Figure 3: Local average coverage rates of conformal intervals for the setting (unbalanced propor- tions) in synthetic data. The local coverage of a data point is average by its 100 nearest data points.
Presitionn Interal for
Fesk Internal OHR
70-
20
-15
-20
-20
Interaa
20
20
-40
Figure 4: Prediction intervals for CQR, CHR, SCP+CC, RLCP, PCP, and MoECP for the setting (unbalanced proportions) in synthetic data.
Compared with MoECP, SCP+CC, RLCP, and PCP produce noticeably wider intervals. Although CQR and CHR attain interval lengths comparable to MoECP (see Figure 2 their prediction inter-
vals fail to adapt when substantial domain shifts occur, i.e., X E 3. 2] U [2, In contrast,
MoECP adapts effectively across all regions, achieving both adaptivity and efficiency.
4.2 REAL-DATA ANALYSIS
We evaluate methods public datasets. The Bike-sharing dataset contains 17,389 samples with 13 features (temporal indicators such as ycar/month/holide and environmental variables including weather. humidity, and wind speed); the response rental count. The Temperature dataset comprises 7,750 summer observations (2013-2017) used to evaluate bias correction for LDAPS temperature forecasts over Seoul.
For both datasets, the training, calibration, and test sets are each fixed at = 1500, and the nominal miscoverage level interval length, and worst-slice coverage (Romano et al. 2020 Cauchois et al. (021). Figure reports the results. RL.PP and PCP are slightly conservative, while remaining methods attain nominal marginal coverage. Worst-slice coverage is approximately 0.90 for all methods, indicating satisfactory conditional coverage. Importantly, MoECP produces the shortest prediction intervals, demonstrating superior efficiency relative to the competing approaches.
0.1. We evaluate methods using three metrics: marginal coverage,
is worth noting that the MoECP framework highly flexible, allowing us to vary the divergence measure, the number of experts, and the temperature parameter. To assess this flexibility, duct comprehensive ablation studies on world datasets (Appendix (A.4.)). The results show that MoECP is robust with respect both the divergence measure and the temperature parameter. Fur-
8

---

Published as a conference paper at ICLR 2026
20
1.8
88
0.0
1.0
REOP
Pat
Mosson
cor
PCP
con
REOO
par
Mostop
10
Iranva
2.0
0.0
Chel
RE.o
Paa
Mesco
comt
Ruco
Motcco
con
sorvoo
Por
Madico
Mothol
Metrol
Mobos
Figure 5: Marginal coverage, interval length and worst-slice coverage 50 experiments. The first row is the results for the Bike-sharing dataset; the second row is the results for the Temperature dataset. From left right, each column corresponds to marginal coverage, interval length, and worst-slice coverage, respectively.
thermore, when the sample size sufficiently large, MoECP also exhibits robustness to the choice of the number of experts.
RELATED WORK
Conformal prediction (CP) provides a distribution-free framework for uncertainty quantification, producing prediction intervals with guaranteed finite-sample coverage under minimal assumptions (Vovk et al. 2005). The split conformal method calibrates predictor by computing nonconformity
scores on held-out set and forming intervals from quantiles of these scores (Lei et al., 2018 Ro- mano et al. 2019). Its simplicity and rigorous guarantees have led to wide adoption across fields such as medicine, natural language processing, and drug discovery (Cortés-Cirianl)
Bender 2020).
A key limitation is that standard CP ensures only marginal coverage: intervals contain the true label with the desired probability on average, but coverage can vary across subgroups or regions of the feature space (Cauchois et al. 2020, Tibshirani et al., 2019 Gibbs & Candes 202T). This gap is problematic in heterogeneous data, where certain populations may be undercovered.
To address this, several extensions have been developed. Atribute-based methods partition the calibration data by known labels to achieve label-conditional validity (Boström et al. (202)). Lo- calized conformal prediction assigns more weight calibration points close to the test instance, improving adaptivity to local structure (Guan 2023 Zhang & Candès 2024 Hore & Barber 2025).
Another line of work considers distribution shifts, where covariate-weighted conformal prediction
adjusts calibration to maintain validity under shifted test distributions (Sugiyama et al.)
2007 Tib-
shirani al. 2019 Candès al. 202 Chernozhukov al. 2021b), Recently, several
works have
leveraged generative models to construct adaptive prediction intervals (Colombo) 2024
Fang et al.
2025).
Across these developments, a common scene emerges: heterogeneous data require conformal pre- diction methods that adapt conformity score to each example. Our work builds on this insight, introducing a MoE perspective that leverages domain similarity.
CONCLUSION
In this paper, we proposed Mixture-of-Expene Distributed Conformal Prediction (MoE-CP), framework that leverages MoE gating vectors to construct adaptive and interpretable prediction

---

Published as a conference paper at ICLR 2026
intervals for heterogeneous and multi nulti-domain data. MoE-CP demonstrates that combining expert specialization with conformal calibration yields a principled and practical path toward reliable un- certainty quantification in complex data environments. By producing intervals that adapt to hetero- geneous subpopulations, MoF-CP may help improve fairness and transparency in real-world appli- cation. We assume that all observations are i.i.d. samples from overall distribution P. which can be modeled as mixture of K latent domains; this guarantees the exchangeability condition needed for the saitle marginal validity. Extending MoE-CP settings with known domain labels or to more general non-exchangeable regimes is an interesting direction for future work. Looking forward, extending to various conformity scores and scaling large MoE architectures opens new paths for trustworthy uncertainty quantification in real-world applications.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and area chairs for their great feedback on the paper. Jingsen Kong was supported by Jinan University. Wenlu Tang was supported by NSERC Discovery Grants and Alberta Innovates. Guangren Yang's research was supported by the Na- tional Social Science Fund of China grant (24BTJ070). Bei Jiang and Linglong Kong were partially supported by grants from the Canada CIFAR AI Chairs program, the Alberta Machine Intelligence Institute (AMII), and Natural Sciences and Engineering Council of Canada (NSERC), and Linglong Kong was also partially supported by grants from the Canada Research Chair program from NSERC.
REFERENCES
Foyge Rina Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution-fii free conditional predictive inference. Information and Inference: A Journal of the IMA, (N2):455-482 2021.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2): (2(2))866662) 1100858666811111- 2023.
Henrik Boström, Ulf Johansson, and Tuwe Löfström. Mondrian conformal predictive distributions.
In Conformal and Probabilistic Prediction and Applications, pp. 24-38. PMLR, 2021.
Emmanuel J Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. arXiv preprint arXiv:2103.09763, 2021.
Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. arXiv preprint arXiv:2008.042677 2020.
Maxime Cauchois, Suyash Gupta, and John C Duchi. Knowing what you know: valid and validated confidence sets multiclass and multilabel prediction. Journal of Machine Learning Research, 22(81):1-42, 2021.
Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. Pro- ceedings of the National Academy of Sciences, 118(48), 2021a.
Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, pp. 1-16, 2021b.
Nicolò Colombo. Normalizing flows for conformal regression. In Conference Uncertainty in Artificial Intelligence, 2024.
Isidro Cortés-Ciriand and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery. 2020.
Zhenhan Fang, Aixin Tan, and Jian Huang. Contra: Conformal prediction region via normalizing flow transformation. In The Thirteenth International Conference Learning Representations, 2025.
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Ad- vances in Neural Information Processing Systems, 34:1660-1672. 2021.
10

---

Published as a conference paper at ICLR 2026
Isaac Gibbs, John Cherian, and Emmanuel Candès. Conformal prediction with conditional guar- antees. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. qkaf008, 2025.
Leying Guan. Localized conformal prediction: generalized inference framework for conformal prediction. Biometrika, 110(1):33-50 2023.
Richard Hahnloser and H Sebastian Seung. Permitted and forbidden sets symmetric threshold- linear networks. Advances in neural information processing systems, 13, 2000.
Matthew Harrison. Conservative hypothesis tests and confidence intervals using importance sam- pling. Biometrika, 99(1):57-69, 2012.
Rohan Hore and Rina Foygel Barber. Conformal prediction with local weights: randomization en- ables robust guarantees. Journal the Royal Statistical Society Series B: Statistical Methodology, 87(2):549-577 578, 2025.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-88) 87, 1991.
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Sociery Series B: Statistical Methodology, 76(1):71-96, 2014.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 (523):1094-1111, 1111, 2018
Huy Nguyen, Trung Nguyen, and Nhat Ho. Demystifying softmax gating function in gaussian mixture of experts. Advances in Neural Information Processing Systems, 36:4624-466 4652, 2023.
Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. Ad- vances in Neural Information Processing Systems, 32, 2019.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive cover- age. Advances in Neural Information P Processing Systems, 33:3581-3591, 2020.
Matteo Sesia and Yaniv Romano. Conformal prediction using conditional histograms. Advances in Neural Information Processing Systems, 63:5304-6315, 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: /701. 70656838,,, 2017.
Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time- moe: Billion-scale time series foundation models with mixture of experts. arXiv preprint arXhv:2409.16040 2024.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Roberr Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal pre- diction under covariate shift. Advances in Neural Information Processing Systems, 32, 2019.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning a random world. Springer Science & Business Media, 2005.
Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference Machine Learning, pp- 25834 -5866. PMIR, 2022.
Yao Zhang and Emmanuel Candès. arXiv:2409. 19712, 2024.
Posterior conformal prediction. arXiv preprint
11

---

Published as a conference paper at ICLR 2026
A APPENDIX
Appendix A.1 outlines the full pipeline the MoE-weighted conformal prediction procedure. Ap- pendix A.2 contains the theoretical proofs. Appendix A.3 provides detailed guidance on hyper- parameter selection. Appendix A..4 describes the experimental setup and results, including imple- mentation details (Appendix A.4.I additional results for the unbalanced setting in synthetic data (Appendix )A.4.2), and ablation studies on the real-data example (Appendix (A.4.3). Appendix [A.5 discusses the computational complexity analysis of MoECP. Appendix A.6 discusses the extension of MoECP to the online setting.
A.1 ALGORITHM
Algorithm MoE-Weighted Conformal Prediction
Require:
Training set Dinin {(x,, YJ), E Itra} and calibration set Deal - ((X;, Ical test point X nonconformity score function -), number of experts K, divergence D(.,.), temperature т, desired level
a.
Ensure: MoE weighted conformal prediction interval CMoE (X,-1)
1: Step 1: Train MoE model
2: Initialize expert functions (14(X)} and gating logits (ex(X))
3: Define gating probabilities TV. (X) م (X])
ex)(
4: Define MoE prediction ji() = 1=
> TA (x
5: Optimize parameters by minimizing loss function training set Drrinn
6: Step 2: Compute nonconformity scores
7: for Tent do
8:
Compute nonconformity score S == 5(X,, Y,) and gating vector x (X,)
9: end for
10: Step 3: Compute weights for test point Xn+1
11: Compute gating vector π (X)++1)
Sample Multinomial (T, (Xn+1))
Compute (Xx++1) = L/T
14: for E Teal U (n do Compute randomized weight:
wi = explp11DDD (Xn+1), , (X:)))
16: end for
Step 4: Normalize weights
18: for i E Teat
U [n 1) do
ii E 25czeau(M
20: end for
21: Step S: Construct weighted conformal interval
22: Form weighted empirical distribution: EiEI. w;ds, + 0x+18
Let Qi-a be the a) quantile of this weighted distribution
24: return (MoE (Xn+1) {y E IR S(Xx+1 v) S Q1-a}
A.2 THEORETICAL PROOF
Proof of Theorem 1 Denote Z: (X,, YJ), [n + 1). and 2; be the realization of Z;, for i [n + 1). By exchangeability,
1
PP 2
1+1 i [n 1].
12

---

Published as a conference paper at ICLR 2026
Conditional on ((x++1) L/T, and using the Bayes theorem, have
P(ā(Xn+11) Zn+1 #2}F(Zn+ = 2/}
P(Zn+1 = 24 [(Xx+1))
EØ=1 P( (xn+1)
Zn+1 - ( 2z}
Th= (T (1x)))
j== I (Tx (2,)]
Wi
i [1.
Here,
K
[m exp
Σ La log ta (1i))
K
# (xn+1) log Tx (xi)
= exp
T
R=1
exp (- (# 1), x(x;)),
where -(:-)) is Cross Entropy. Likewise, dividing both the numerator and denominator by the same number does not change the result. We know that
K
(x(2))
Lx
II
(x
where KL((-I)) is the KL divergence. Note that
Yu+1 $ MMol (Mot((N)t) x Sa+1 2 Q1-a
wjds, + 1n+1000
n+1 w;dss
Snt1 2 Q1-a
(Lemmaa A.l in Guan (2023)
n+1
w. Sx+1) S
Therefore, for the MoE weighted conformal prediction with Cross Entropy or KL divergence-basee weights, the probability conditional on 7 (Xn.:)) is
((Y+1 &
| #(Xn-1))
P(Yn+1 & CMoB (Xn+1) ((X++1), Zuti 23} P(Z,+1 2g #((,+1))
a+1
n+1 w.T(S 2 S,) 5a)saa )
B
E
1=1
h+
n+1
T, 2 22) 5a) )
E
ر T
< [[]
where I(.) is the indicator function, and the relationship "<" holds due to Lamma Al in Harrison 2012 Therefore,
P(Y++1
AMOE (Xn+1) # #(X 2 1
13

---

Published as a conference paper at ICLR 2026
Proof of Theorem 2
By Assumption
| for each there exists r; ((1) with
7+
W KL + The and
T3 0.
Define
n+1
n+1
F n+1
(s)
Σ
MiD I(S 8), and F(KL) (s)
Σ
Wi,LL
I(S S s).
For any fixed 8,
n+1
(() F(KT) (s) S Σ
Then, [ - f/(K))) o(1)
Let
(s) 2 21-aa; and ((x1)) inf(s n+1 (s) 2 ((KT)(s) (D) =if(s (s)
a).
As S is continuous, have
01 aiaa + o(1)
(D) (KL)
The MoE weighted conformal interval with similarity measure D is
(()) (X {v: -(Xm+1, S )
Likewise, the interval with KL divergence is
(KKL (Xn+1) ((x+1((X+1,3) S (KL), }.
We know that
(KL) P(Yn+1 E Ch (XkL(X+t)) ([Xx+1)) = P(S S 9i-a [#(x++1) 21
Thus,
P(Y E ((P) (Xx+1) #(X,+1)) PPS S 5-iaa म i(xx+1))
(D)
=P(S,+1 5-aaa o(1) #(x,+1))
(KL)
2 + o(1),
which completes the proof.
Proof of Theorem 3 The proof is somewhat similar to Proposition in Zhang & Candès (024))
while they consider different problem setting. Note that
(X xt (X,)) expl exp(-TD(T> (Xm+1), T (X.)) (X,))'
ie [n
+++ i=
Therefore, for any similarity measure D(.)) )) and T E N+, we have wi. th 1 Xa+1 Tt" (Xx+))) By the construction of (MoE, (Xn+1))
a+1
Yn+1 & G (MIBB,((X+t)) (Xn+1) + p = Σ
(MoE,
w T(S 2 Sat1) S a,
which means that the event (Yn+1 $ (MoE, (Xo (Xa+1)) depends on X only through S,+1 and
n
n (Xn+1) in the weights W i E [n + 1]. If there is finite mixture representation such that
S ΣK (X))f, In 1). we have
Sat Xa+1 n" (Xn+1)-
14

---

Published as a conference paper at ICLR 2026
Therefore, the probability of the event given n² (XXn+1) and X is independent of X ie.,
P(P S Xn+1, TT" (Xn+1)) P(P o (X++1)).
Conversely, x(x+11) can be taken as function of X,+1- Once Xn+1 is given, x ((x++1) can be taken as fixed. Then,
P(P S Xn+1, HT' (Xn+1)) P(P Xn+1).
Consequently,
P(P: Sa x (X = PP S a X
Taking the complement on both sides completes the proof.
Proposition related to Assumption 1 Here, state proposition that Assumption D under a specified condition. This condition satisfies many divergence measures if there are enough calibration points. The rationale behind the claim is that, for given म == # ()) if the calibration set is sufficiently large and contains point whose gating vector иj = T (rry) equals (or arbitrarily closely approximates) тr, then the KL divergence satisfies [L = O. Since KL q) = if and only if q म. such calibration point attains the minimizer of the KL divergence. For any divergence that likewise vanishes only at equality (for example, Jeffreys divergence, the Hellinger distance, or the Euclidean distance, etc), we have D Ty) 0. Hence, the minimizer condition is satisfied.
is satisfied
Proposition If a divergence D(--)) shares the same minimizer as with the KL divergence, then Assumption 7 is verified, ie., WipD kI, + o(1, as t o0.
Proof of Proposition 1 If the divergence D) shares the same unique minimizer with KI. divergence, we have
arg min W arg min KL EE(++1]
sE[++1]
Let
f: D(T,
9- = KL(F,T), E I1.
where से = =[==II))) V7 (I.)). Denote
f()) min f. f())
min fc. efffan
similar definition for 9()) 9(2)- Suppose that there exists the same minimizer, i.e.,
Sc f(y) fr. = 9(1-
We define
8 min (f2) fe. 9() - 9r > Then, the normalized weights for the divergence D and KL can be expressed
e
T9h
(7) Σ e-
(T)
a1 Σ
j j= T9
We can rewrite the denominator as
m++
S; =
Σ c f, = -T5: (1 As)) := Σ = (1 + Ag),
1=1
3=1
where Ay = Ejer e (()--1=),,
=E E (9)-- Clearly,
0 As, Ag -15
In case i = i", we know that
(f) -//
(a)
E T8,
W
w
S
1 + A,'
S
1 + A
15

---

Published as a conference paper at ICLR 2026
Therefore,
(0)
[Ag Apl
S SIAg A,1 max(Ag. S <ne-
(6)
(1+ Ay)(1 + Ag)
In case i + i", we know that
(5) e -- "
Tg e: --
(a)
Sj
1 1 As
Sg
1 1 Ag
Let As: := 1- Ay: = 9i gi We know that Aʃ; 2 Ag. 2 8, for + 1² Then, the
absolute difference between two weights can be decomposed as
e- -g
|0! ///0
1 1+ A 1 Ag
e TAK - Tdg
1
+ TAg
1 1 1+A, A
1+ Af
1 Ag
|e TA- --roi
+ -e-roo
Ag Asl
maxfe -Tasi -
+ -Dar max (A,, As}
I e --8 + ne 2r5 -255
(7)
Based on Equation (6) and we can conclude that
W K. o(1),
as T > 00, the divergence D shares the same the minimizer with the KI. divergence.
A.3 CHOICE OF HYPERPARAMETERS
The MoECP framework allows the user to specify the divergence measure, the number of experts K, and the temperature т. The ablation studies (Appendix A.443) provide a detailed comparison among these hyperparameters. Here, provide practical guidance with respect these choices.
Divergence measure D...)). We demonstrate both theoretically (Theorem 2 and empirically (Appendix A.4.3) that MoECP is largely insensitive to the particular divergence measure employed. In addition, we prove that the KL divergence enjoys exact marginal validity within our framework. For these reasons, combining theoretical guarantees and empirical robustness, we recommend the KI. divergence as the default choice for MoECP in practice.
Numbers experts K. Our ablation studies show that whether K materially affects MoECP's interval length depends primarily the training and calibration sample size. Intuitively, increasing K partitions the calibration data across more experts and thus reduces the effective sample size per expert; when the total sample size is modest (for example, I 1500), this reduction widen prediction intervals, so a smaller K (c.g., K 3) is preferable. For larger overall sample sizes, each expert still receives sufficient data and the difference in interval length across different K becomes negligible. In practice, choose K by balancing statistical efficiency and modeling flex- ibility: use cross-validation or information criteria (BIC/AIC) when possible, taking into account domain knowledge and computational constraints. As an alternative strategy, one may fit an over- parameterized model and apply sparsity regularization the gating network automatically prune unused experts, thereby producing an effective K without exhaustive manual tuning.
noticeably
Temperature parameter т. Our ablation studies (with scanned in the range 50-300) indicate that MoECP is robust to the choice of temperature: marginal coverage, conditional coverage, and interval length remain stable across a wide range of т. Intuitively, т controls the sharpness of the MoE weights-very small produces nearly uniform weights and reduces adaptivity, while very large 7 concentrates weight on a few points (reducing the effective calibration sample size and widening intervals). A moderate therefore provides good trade-off between adaptivity and statistical efficiency; we recommend 7 E [100, 300] as a practical default. When needed, 7 can be tuned by simple grid search validation.
16

---

Published as a conference paper at ICLR 2026
A.4 EXPERIMENT SETUP AND RESULTS
A.4.1 DETAILS OF THE EXPERIMENTAL SETUP
We use the Mixture of Experts (MoE) predictor as the base estimator for the proposed MoECP and implement an MoE model with experts and single gating network. Each expert is a fully con- nected feedforward network with three hidden layers and ReLU activations; the gating network uses the same hidden-layer structure and outputs softmax distribution over the K experts to weight their predictions. A dropout layer with rate 0.1 is applied in both the experts and the gating network to reduce overfitting. Models were trained with mini-batch gradient descent (batch size 64), a learning rate of 1 x 10- and early stopping ceiling epochs. For the synthetic-data experiments set = 3. temperature т = 150, use hidden layers of size (32, 32, 32) for both experts and gate, and train up to 600 epochs. For the real-data experiments set K 2, temperature 100, use hidden layers of size (64, 64, 64), and train up to 2000 epochs.
For Conformalized quantile regression (CQR) RRomano et al.) 2019) and Conformalized histogram regression (CHR) (Sesia & Romano 202T), we used the original authors' implementation using the base estimator as quantile random forest. For Split conformal prediction with conditional cal- ibration (SCP+CC) (Gibbs et al. 2025), Randomly-localize
conformal prediction (RLCP) (Hore) & Barber 2025), and Posterior conformal prediction (PCP) (Zhang & Candès 2024), we use the implementation in Zhang & Candès (0224) using the base estimator as a random forest regressior which use sklearn. ensemble e Randomoorest Rreerroreeeseeessoooooe
in scikit ceikit-learn, with de- fault hyperparameter. Specifically, the bandwidth of RLCP is chosen such that the effective sample size is 100 to balance the trade-off between the adaptivity and efficiency (Hore & Barber (025)).
To evaluate conditional coverage in the real-data experiments, we adopt the worst-slice coverage metrie, following Romano et al. (2020): Cauchois et al. (202T). This metric approximates the worst coverage across all small subpopulations of the form x arbitrarily.
[a, b]}, where b, and v may vary
The Bike-sharing dataset (UCI Machine Learning Repository P comprises 17,389 observations from an urban bike-sharing system. Each record includes 13 explanatory features that capture temporal context (e.g., year, month, weekday, holiday indicators, and working-day flags) and environmental conditions (e.g.. ambient temperature, perceived temperature, humidity, wind speed, and a coarse weather category). The response is the total rental count. Categorical and ordinal variables were encoded appropriately (binary flags or integer codes), and all numeric predictors and the response were standardized zero mean and unit variance prior to model training, calibration, and evaluation.
The Temperature dataset (UCI Machine Learning Repository @ comprises 7,750 summer observa- tions (001322077) assembled to evaluate bias-correction methods for LDAPS numerical temperature forecasts over the Scoul region. Each record pairs LDAPS forecast outputs with the corresponding ground-truth observation and includes temporal indicators (e.g., date/time, day-of-year) and local meteorological covariates used for bias modeling. In our experiments we use the next day's 's mini- mum temperature as the response; we drop the fields s staticn Date, and Next.Tmaxx and retain the remaining 21 variables as predictors. All variables both features and response- are standardized to zero mean and unit variance prior to training, calibration, and evaluation.
A.4.2 ADDITIONAL. RESULTS IN SYNTHETIC DATA EXAMPLES
In section 4.1 the synthetic data YX are drawn from a three-domain distribution:
X + p= P1i ==
X2 p p= P25 33 +e, p= P3i
where X ~ Uniform([-3, 3)) N(O 1), and p,, 2, 3. are the proportions of each domain that sum up to 1. We consider two settings: (I) Unbalanced proportions: p1 = 0.2, p2 = 0.3, pa =
hautss//ithhbbccmmyyaaaaag2 http:///ww..coooooooooooooooeoen
17

---

Published as a conference paper at ICLR 2026
30.0
1.0
27.5
25.0
Coverage rate
0.9
Interval Interval Length
22.5
20.0
0.8
17.5
0.7
15.0
12.5
0.6
10.0
CQR
CHR
SCP+CC
RLCP
PCP
MoECP
CQR
CHR
SCP+CC
RLCP
PCP
MoECP
Method
Method
Figure 6: Marginal coverage and interval length for the setting 2 (balanced proportions) in synthetic data over 50 experiments.
CQR
CHR
SCP+CC
RLCP
PCP
MoECP
1.0
Conditional Coverage
0.9
0.8
0.6
0.5
Figure 7: Local average coverage rates of conformal intervals in the setting 2 (balanced proportion) in synthetic data. The local coverage of data point is average by its 100 nearest data points.
0.5; (2) Balanced proportions: P1 = P2 Ps 1//3. The sample size for training, calibration, and testing are set equally to be n = 500. The experiments are repeated over 50 times. The miscoverage level is set to be 0.1.
Figures and present the results under Setting (unbalanced proportions), while Figures 6 7 and 8 correspond to Setting 2 (balanced proportions). Across both settings, the experimental re- sults demonstrate that the proposed MoECP framework achieves reliable marginal coverage, strong conditional coverage, and high efficiency with adaptively small interval lengths.
A.4.3 ABLATION STUDY
The MoECP framework flexible, allowing different choices of divergence D(.,)), )), number of ex- perts K, and temperature т. To examine the impact of these hyperparameters, we conduct ablation studies on the Bike-sharing and Temperature datasets. The reported results are averaged over 10 independent runs.
Ablation studies divergence D(..)) and temperature We consider several divergence measures, including KL divergence, Rényi divergence, Jeffreys divergence, Hellinger distance, Eu- clidean distance, and Cosine distance. The temperature parameter is varied from 50 to 300, while
18

---

Published as a conference paper at ICLR 2026
Weekiton mnte
Prodciion Interva CHA
-20
-00
Frdiction SCPeCC
Fredetiion
-20
-40
Prediction Inteval
Predciioo Marde faritecr
-20
Figure 8: Prediction intervals for CQR, CHR, SCP+CC, RLCP, PCP and MoECP for the setting (balanced proportions) in synthetic data.
1.8
1.15
0.8
07
9.5
306
250
1.0
Butbon
VOONE 4008 CONRPIIR 138
0.
132 200 200
110
100
270
Figure 9: Ablation studies divergence D(:-)) and temperature parameter The sample size for training, calibration and testing is set equally to be = 1500. The number of experts is K = 2. The first row is the results of the Bike-sharing dataset; the second row the results of the Temperature dataset. From left to right, each column corresponds to marginal coverage, prediction interval length, and worst-slice coverage, respectively.
the number of experts is fixed at K = 2. For all experiments, the training, calibration, and testing sets are of equal size, with n 1500 samples each.
Figure 9 reports the results of ablation studies over divergence measures and temperature, demon- strating the robustness the MoECP framework with respect to these hyperparameters. In particu- lar, the marginal coverage remains close to the nominal level across different divergences and values of r. The interval length also largely stable, with only minor variations across divergences and temperatures. As increases, observe slight increase in interval length. This is expected, since a larger reduces the effective sample size, thereby widening the prediction intervals (in the extreme case - +00, the normalized weights collapse i = 1 and w; 0 for all 1,... n, yield- ing an infinite prediction interval). Moreover, the worst-slice coverage is consistently robust with respect both divergence and temperature.
The expressions of KL divergence, Rényi divergence, Jeffreys divergence, Hellinger distance, Eu- clidean distance, and Cosine distance are listed below. For ((X++1), m (X,) E [0, 1|K
19

---

Published as a conference paper at ICLR 2026
KL divergence:
K
#x(X++1)
(xL( = Σ
nx (xxXa+11 log
TT (Xi)
k=
Jeffreys divergence:
Dietieys ( (Xx+)), x( + (xL
Rényi divergence with a = 0.5:
DRémyi
log Σ th (X (xxntt))
a
Hellinger distance:
Diellineer ( (f(Xx+t))) (Xx))) V2 (a((N+t)) (((XX)))-V((X)))) (T(()))
Euclidean distance:
DEuclidem ( (X(n)))), = || - xXX))1l
Cosine distance:
((x+11) (x,)
Deosine (aceine(((AAe) = - || ( )la (xx), 11
Note that for the cross-entropy,
K
D Deross entropy (T(Xx+1),T(X.)) = (xx, -
Σ nu (Xn+t))) fx (
1=1
direct calculation shows that the normalized weights in the MoE weighted conformal prediction satisfy
ÚKL = eeross-entropy.i
Therefore, it is unnecessary include the cross-entropy in the ablation studies, although it can be used equivalently.
Ablation studies the experts K. We further conduct ablation studies by varying the number of experts K E [2, 3, 4, 5). In addition, different sample sizes for training and calibration are con- sidered, ranging from 500 to 2500 (e.g., n 1000 means both training and calibration sets contain 1000 samples each). The test set size is fixed at n 1000 throughout. The temperature is fixed at 100, and the divergence measure chosen the KL divergence.
Figure I0 presents the results of ablation studies with varying numbers of experts K and different sample sizes. We observe that the marginal coverage consistently matches the nominal level across all settings, and the worst-slice coverage remains robust with respect to both K and the sample size. Regarding interval length, it naturally decreases as the training and calibration sample sizes increase. When fixing the sample size and varying K, MoECP with smaller K generally yields shorter intervals. This occurs because smaller corresponds to a larger effective sample size, leading to narrower prediction intervals. For example, with calibration size 500 and assuming balanced domains, samples per domain, thus widening the intervals. However, this effect diminishes as the overall sample size grows (e.g., n 3000), since each domain still has sufficient data to construct tight prediction intervals even with larger K.
= 2 results in 250 samples per domain, whereas K = leaves only 100
20

---

Published as a conference paper at ICLR 2026
Wore-d Ioe covenaee 1EE
9.8
Interal
23
1.5
1.3
0.8
12
Sansle
Figure 10: Ablation studies on the experts number K with different sample size. The temperature parameter is set to be T = 100. The divergence measure is the KL divergence. The sample size for testing is set to be n 1000, while the sample size for training and calibration varies from 500 to 2500. The first row is the results of the mike-sharing dataset; the second row is the results of the Temperature dataset. From left to right, each column corresponds to marginal coverage, prediction interval length, and worst-slice coverage, respectively.
0.05
Bike sharing Temperature
0.8
50 100 :50 300 250
300
Figure 11: Ablation studies on the impact of multinomial sampling on MoECP interval variability and stability. The divergence measure is the KL. divergence, and the number of experts h = 2. The sample size for training and calibration is set equally to be n = 1500. The sample size for the test set to calculate the variability measure is set to 100, and the multinomial sampling times are 500 for each test point.
Ablation studies on the impact of multinomial sampling on MoECP interval variability and stability For MoECP, the randomized gating probability #(X,+1) is generated using the multi- nomial sampling scheme in (3) with temperature parameter т. Our goal is to examine how this randomized construction influences the variability and stability of the resulting prediction intervals. To quantify this effect, we adopt variability metric inspired by Hore & Barber (002)):
MAD (X(CMoE (X
D(7)= E
Median (X(CMOt (XCMME((Xx+))) (Xm+1))
where X(-) denotes the Lebesgue measure prediction interval and MAD(X) = Median( Median(X)]p The only source of randomness comes from the multinomial sampling. Thus, D(r) captures the degree to which the MoECP interval width fluctuates. Experiments are performed on the Bike-Sharing and Temperature datasets, with training/calibration size 1500 and test size 100. The MoE model configuration follows Section 4.2, using KL divergence and two experts (K = 2). For each test point, the multinomial sampling is repeated 500 times to estimate the variability.
21

---

Published as a conference paper at ICLR 2026
Figure 11 presents the interval variability and stability results. The variability metrie D(r) is stable across values the temperature parameter T for both real datasets, indicating that the choice of is robust interval variability and stability.
A.5 COMPUTATIONAL COMPLEXITY ANALYSIS
For MoECP, it needs to train an MoE model in the training phase, and calculate the conformal score in calibration phase. Here, we aim to investigate the computational complexity of MoECP. We highlight that dominant computational cost lies in training the MoE model, while the calibration step of MoE-CP is lightweight.
Training cost (dominant). Training the MoE involves forward/backward passes through the gating network and all experts. If m is the training sample size, K the number of experts, and the number of epochs, the overall training cost is on the order of O(Em(ggat + KC Cexp),, where Cgate and Cexp denote one corward-backwand gate and a single expert, respectively. This one-time offline cost and represents the main computational burden of the method. In our experiments, training is performed once per dataset and is easily handled.
cost of the
Calibration cost (lightweight). Once the MoE trained, we precompute and cache the gating vectors π (X,),, é [mi] of the calibration set. For each test point, MoE-CP computes (1) one forward pass of the gating network, and (2) pairwise divergences between the test gating vector and the calibration gating vectors. This results in O(n) cost, which is extremely fast practice. Actually, the computational complexity of the calibration stage is analogous to RLCP (Hore& Barber 2025).
We evaluate the practical runtime on the real datasets, averaging each result over 10 independent runs. All experiments conducted
AMD Ryzen 5 4600H processor with Radeon graph- ics. Note that the reported runtime would be substantially lower with more advanced computing hardware. The details of the experimental setup provided in Appendix A.4.1
Table reports the runtime for each method on the Bike-sharing and Temperature datasets. The computational cost of MoE-CP is comparable that of CHR and PCP. Figure 12 illustrates the run- time as a function of the training/calibratin sample size, showing an approximately linear increase in computational time with sample size.
Table 1: Runtime (seconds) for each method and dataset
Methods
CQR
CHR
SCP+CC
RLCP
PCP
MoE-CP
Bike-sharing
3.3
151.7
27.0
5.1
134.7
153.2
Temperature
3.8
153.4
39.1
6.4
157.6
155.5
Shaidhaiig
222
200
175
20
220
1303
2003
Figure 12: Runtime (seconds) of MoECP under different sample sizes. For instance, a 1000 sample size corresponds to training and calibration size equally to be 1000.
22

---

Published as a conference paper at ICLR 2026
A.6 EXTENSION OF MOECP TO THE ONLINE SETTING
In online setting Gibbs & Candes 2021 Zaffran et al. 2022). we have To observations
Yi), (X Yro) in Ra I. We aim construct prediction intervals for Ti subsequent
observations X1o+1:" ... XTo+Ti sequentially: at any prediction step E [7o + 1, To + Ti). Y(-1 YI-To....
have been revealed. Therefore, the data (X,-To YI-To): (X,-1, Yz-1) 1 available to construct the prediction interval for Y,
are
The online MoECP proceeds by maintaining and adaptively updating target miscoverage level {au) based on recent performance and similarity similarity-weighter weighted errors. Let the nominal miscoveron be E (0. 1) and initialize 0' a. For each prediction time To + 1...., To + Tı, form the MoE-based prediction set
MMB (X,) tu e I:: S(X,, Q1-a.h
Computtee
where Qi-a, denotes the (1- as -quuntlte of the (weighted) conformity scores computed on the current calibration window. The miscoverage level then updated by
= ai -
a1+1 1
W. 1fY, & EMOE (X.))
a To+1
with step size y >> 0. The normalized cumulative weight w, for [t To + 1, is defined as
Ei=-T++1 Wi
0g Li=t-Ti To
To+1 Wi
where each w; is computed analogously Equation using the current covariate X₁- This update combines the recent empirical miscoverage frequency with similarity information between calibration and test points, making the online MoECP adaptive to recent errors while accounting for domain similarity. A thorough theoretical and empirical investigation of this online strategy is left for future work.
23


Reviews:
[{"review_id": "jPR4GAOMl5", "reviewer": "Reviewer_TKSb", "summary": "The authors introduce Mixture-of-Experts Conformal Prediction (MoE-CP), a flexible method that uses the gating probability vectors of Mixture-of-Experts (MoE) models to estimate the similarity between test sample and calibration samples, thereby achieve adaptive conformal prediction.", "questions": "1. What about extending your idea to classification? What challenge will you face?\n\n2. What is the difference between the similarity weights from gate model and a density estimator?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Sufficient discussion about background and clear introductive figure.\n\n2. Extensive ablation study on hyperparameter selection of the proposed method.", "weaknesses": "1. This work basically follows the logics of prior research: compute the similarity between calibration samples and a give test sample to estimate local conformal score density, which is used to output a local 1-alpha threshold for prediction sets. Thereby, the overall contribution is incremental.\n\n2. The gate model typically plays a role of PCA to facilitate the density estimation. In other words, the number of expert models K must be sufficiently smaller than feature dimensions to be functional, which limits the application of the proposed method.\n\n3. Density-estimation-based localized CP , such as [1,2], are sensitive to hyperparameter selection. The authors only mention which base predictive models are used, without discussing if their hyperparameters are tuned carefully. Hence, the experiement results may be unfair. 4. Recently, generative models for adaptiveness progressed, such as [3,4], which are missed in the work. \n\n[1] Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika, 110(1):33–50, 2023.\n[2] Rohan Hore and Rina Foygel Barber. Conformal prediction with local weights: randomization enables robust guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, 87(2):549–578, 2025.\n[3] Colombo, Nicolo. \"Normalizing flows for conformal regression.\" arXiv preprint arXiv:2406.03346 (2024).\n[4] Fang, Zhenhan, Aixin Tan, and Jian Huang. \"CONTRA: Conformal prediction region via normalizing flow transformation.\" The Thirteenth International Conference on Learning Representations. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rnAJTjQRe0", "reviewer": "Reviewer_ZRke", "summary": "The paper proposes Mixture-of-Experts Conformal Prediction, which integrates conformal prediction with mixture-of-experts models. It assigns higher weights to calibration samples whose gating vectors are more similar to the test point’s, thereby adapting prediction intervals to latent domain structures.", "questions": "1- The authors should clarify the precise meaning of $\\mu(x)$ and $\\pi(x)$. While both are formally defined, their conceptual interpretation and how they are learned from data remain vague.\n\n2- It is unclear how the number of experts and the temperature parameter are selected in practice. Please explain how you chose these parameters.\n\n3- Since the approach involves computing similarity-based weights for each test point, the computational complexity may be substantial for large calibration sets. It would be helpful if the authors could provide a computational complexity analysis and discuss practical runtime implications.\n\n4- There are existing conformal prediction approaches that incorporate expert advice [1, 2]; the authors are encouraged to provide a comparison with these methods, either conceptually or empirically\n\n5-The impact of the randomization step (via multinomial sampling in Eq. 3) on interval variability and stability is not analyzed. Please provide an analysis to show its impact\n\n[1] Hajihashemi, E. and Shen, Y., 2024. Multi-model ensemble conformal prediction in dynamic environments. Advances in Neural Information Processing Systems, 37, pp.118678-118700.\n[2] Gibbs, I. and Candès, E.J., 2024. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162), pp.1-36.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1- The paper introduces an interesting and well-motivated idea that combines Mixture-of-Experts (MoE) models with conformal prediction\n\n2- The paper provides a theoretical analysis, proving marginal validity and approximate conditional coverage", "weaknesses": "Please check the questions!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KOkSwCj7UJ", "reviewer": "Reviewer_qsRG", "summary": "This paper proposes MoE-CP, which uses MoE gating vectors as soft domain assignments to weight calibration residuals in conformal prediction. The method aims to produce adaptive prediction intervals for heterogeneous data without explicit domain labels, with theoretical guarantees for marginal validity and empirical validation showing tighter intervals than standard conformal methods.", "questions": "The authors should address the concerns raised in the weakness section above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Creative use of MoE gating vectors for conformal prediction weighting addresses heterogeneous data limitations.\n\nComprehensive analysis including marginal validity (Theorem 1), robustness to divergence choice (Theorem 2), and conditional coverage under mixture representation (Theorem 3).", "weaknesses": "Requires training an MoE model before conformal calibration, adding significant computational cost compared to standard conformal methods.\n\nMultiple hyperparameters require careful tuning, and there is no detailed computational cost analysis compared to baseline methods.\n\nCore assumption that MoE gating vectors meaningfully capture domain structure may not hold in practice, especially when true domains don't align with MoE's learned partitioning.\n\nLimited discussion of how to validate whether MoE gating provides good domain separation.\n\nMoE models are trained to minimize MSE loss, not to learn domain boundaries, creating fundamental misalignment between training objective and intended use.\n\nReal datasets are low-dimensional, limiting generalizability to modern large-scale applications. Synthetic experiments only consider simple polynomial relationships with clear domain boundaries.\n\nRandomization step in equation (3) appears somewhat ad-hoc without strong theoretical justification.\n\nExchangeability assumption may be violated in truly multi-domain data where domains have different distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeAGCmFyvz", "reviewer": "Reviewer_c8Cp", "summary": "The paper proposes a variant of the randomly localized conformal prediction (RLCP) introduced by Hore and Barber. The approach evaluates similarities between inputs in the latent space defined by the vector of probabilities output by the gating mechanism of a MoE model. The key idea is that inputs with similar statistics are also likely to be routed to the same experts if the gating mechanism is well trained.", "questions": "1) Why is the same temperature parameter tau used in both (3) and (4)?\n\n2) Under what conditions is Assumption 1 verified?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach is sound, retains statistical validity, and leverages advances in MoE architectures. \n\nThe experimental results, while limited, are sufficient to support the main claims of the paper.", "weaknesses": "The submission may not be sufficiently clear in the earlier sections about the relationship of this work with Hore-Barber. I think that this should be made clearer when presenting the contribution. As is, this is only discussed in Remark 5 on p. 5.\n\nIt is unclear a priori why the same temperature parameter tau is used in both (3) and (4). \n\nAssumption 1 is not clear. The authors claim that the assumption holds for a variety of divergences, but they do not provide any supporting evidence for this.\n\nThere are some typos, such as \"conformapl\" on p. 1 and the missing space in \"by(3)\" in Algorithm 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)