Uncertainty-driven Embedding Convolution
Authors: Sungjun Lim
Kangjun Noh
Youngjun Choi
Heeyoung Lee
Kyungwoo Song
Venue: ICLR 2026 Poster (2026)
Keywords: Probabilistic embeddings
Embedding convolution
Uncertainty-aware similarity
Primary area: foundation or frontier models, including LLMs

Abstract: Text embeddings are essential components in modern NLP pipelines. Although numerous embedding models have been proposed, no single model consistently dominates across domains and tasks. This variability motivates the use of ensemble techniques to combine complementary strengths. However, most existing ensemble methods operate on deterministic embeddings and fail to account for model-specific uncertainty, limiting their robustness and reliability in downstream applications. To address these limitations, we propose Uncertainty-driven Embedding Convolution (UEC). UEC first transforms deterministic embeddings into probabilistic ones in a post-hoc manner. It then computes adaptive ensemble coefficients based on embedding uncertainty, derived from a principled surrogate-loss formulation. Additionally, UEC employs an uncertainty-aware similarity function that directly incorporates uncertainty into the similarity scoring, providing a theoretically grounded and efficient surrogate to distributional distances. Extensive experiments on diverse benchmarks demonstrate that UEC consistently improves both performance and robustness by leveraging principled uncertainty modeling.

Published as a conference paper at ICLR 2026
UNCERTAINTY-DRIVA EMBEDDING CONVOLUTION
Sungjun Lim
Kangjun Noh
Youngjun Choi
Heeyoung Lee
Kyungwoo Song'
Yonsei University
ABSTRACT
Text embeddings are essential components in modern NLP pipelines. Although numerous embedding models have been proposed, no single model consistently dominates across domains and tasks. This variability motivates the use of en- semble techniques to combine complementary strengths. However, most existing ensemble methods operate on deterministic embeddings and fail to account for model-specific uncertainty, limiting their robustness and reliability in downstream applications. To address these limitations, we propose Uncertainty-drives Em- bedding Convolution (UEC). UEC first transforms deterministic embeddings into probabilistic ones in a post-hoc manner. It then computes adaptive ensemble coeffi- cients based on embedding uncertainty, derived from a principled surrogate- formulation. Additionally, UEC employs an aicertainty-awarr similarity function that directly incorporates uncertainty into the similarity scoring, providing a the- oretically grounded and efficient surrogate to distributional distances. Extensive experiments on diverse benchmarks demonstrate that UEC consistently improves both performance and robustness by leveraging principled uncertainty modeling."
INTRODUCTION
Embeddings are core building blocks in modern NLP, capturing semantic meaning for words, sen- tences, and documents. They support tasks like similarity (Gao al., 2021), retrieval (MacAvancy et al., 2019), QA (Devlin et al., 2019), and classification (Cer et al., 2018). Numerous embedding models (Devlin et al., 2019; Liu et al., 2019; Reimers Gurevych, 2019) have emerged with diverse architectures and training objectives, but their performance varies across tasks and domains. No single model excels universally; instead, models offer complementary strengths. This motivates combining multiple embeddings to leverage their diverse capabilities.
Embedding models can be ensembled at two levels: parameter and representation. the parameter level, techniques like model merging combine model weights (Wortsman et al., 2022; Ilharco et al., 2022: Yang al., 2024). However, this approach is often limited by strict architectural constraints. On the other hand, combining embeddings at the representation level, i.e., ensembling the output embeddings themselves, is more practical and widely applicable strategy. However, most existing ensemble approaches rely deterministic aggregation methods such uniform averaging, which fail to consider the reliability or uncertainty of individual embeddings (Khan et al., 2023; Fang et al., 2019). Without modeling uncertainty, these methods treat all embeddings as equally reliable, which can lead to suboptimal or unstable performance, especially when some models are poorly calibrated or mismatched to the target task. This is evident in Figure where averaging conflicting 'animal' and 'car' interpretations for 'jaguar' leads directly to a retrieval failure.
To address these limitations, we propose Uncertainty-driven Embedding Convolution (UEC), framework for combining embeddings in principled, uncertainty- manner. UEC consists of three key components. First, it converts pre-trained deterministic embeddings into probabilistic embeddings in a post-hoc fashion, allowing each embedding to represent both its mean and uncertainty. Second, it computes adaptive ensemble weights based on estimated uncertainty, down-weighting less reliable embeddings. This weighting strategy is grounded in a principled, amcertainty-awant surrogate-loss Yormulation. Third, UEC introduces an uncertainty-awar similarity function that incorporates both distance and variance into the similarity score, offering theoretically grounded and
"Corresponding Author: kyungwoo. song@gma i 1, com
Code is available https: / / thub. ccm MLA Yonsei /URC.

---

Published as a conference paper at ICLR 2026
Candidate It hunts in the jungle.
Query:
The jaguar fast and dangerous.
Candidate B: It has powerful engine.
Deterministic Embedding Ensemble
UEC (Ours)
Deterministic Embeddings
Ensembled Embeddings
Probabilistic Embeddings
Ensembled Embeddinge
* Query
Ensemble
Convolution
*
A
8
x Model
33 Model II C Modal III
A: 0.50 = : 0.50
X
A: 0 : 0.40
Figure I: Comparison of deterministic embedding ensemble and UEC. Deterministic ensemble (left) uniformly averages embeddings without considering their reliability, often leading to suboptimal decisions. In this example, both candidate embeddings contribute equally, resulting in incorrect retrieval. In contrast, the proposed UEC (right) adjusts weights based on the uncertainty, assigning higher importance to the more reliable candidate and successfully retrieving the correct answer.
efficient surrogate distributional distances. Consequently, empirical results on diverse benchmarks confirm that UEC reliably improves performance and robustness by leveraging principled uncertainty modeling.
In summary, our key contributions are as follows:
We propose post-hoe mechanism to transform pre-trained deterministic embeddings into probabilistic representations. thereby enabling inherent uncertainty quantification for diverse existing embedding models.
We introduce the UEC framework, which adaptively and data-dependently combines multiple embeddings by weighting their contributions based estimated, query query-specific uncertain- ties, following a principled surrogate-loss formulation.
We develop an uncertainty-aware similarity function that explicitly incorporates embedding variance into the similarity scoring process, serving as a theoretically grounded and efficient surrogate to distributional distances.
2 RELATED WORKS
2.1 EMBEDDING ENSEMBLE
Embedding ensemble methods aim to improve performance and robustness by combining multiple pre-trained models. Shuang et al. (2019) proposed CDWE, which handles polysemy by generating multiple word prototypes through deconvolution. Fang et al. (2019) enhanced knowledge-aware embeddings by re-weighting knowledge-gree variants using cross-attention guided by pseudo-labels. Sahlgren (2021) distilled multiple encoders into one by matching their averaged outputs. Khan et al. (2023) explored fusion strategies for domain- specific BERT models, yielding minor improvements. Despite these advances, most approaches rely on deterministic embeddings, limiting their ability model uncertainty. In contrast, method ensembles probabilistic embeddings, capturing both distributional information and model diversity through principled uncertainty estimation.
edges. Liu et al. (2025) fused BERT (Devlin, 2018)
2.2 PROBABILISTIC EMBEDDING
Probabilistic embeddings represent inputs distributions rather than points, thereby modeling uncer- tainty in representation learning. Early approaches such as Gaussian (Vilnis McCallum, 201 2014), Gaussian mixture (Chen al., 2015), and Bayesian embeddings (Barkan, 2017) captured word-level distributional semantics. Sen2Pro (Shen et al., 2023) extended this idea to sentences using pre-traine trained models, but it requires fine-tuning, lacks post-hoe conversion, and does support scalable similarity.
2

---

Published as a conference paper at ICLR 2026
Post-hoc Probabilistic Embedding
Uncertainty-driven Embedding Convolution
Uncertainty-aware Similarity
Convolution
In
The jaguar is fast...
Probabilistic Cosine Similarity sqg. Ni,, o,)
(x
huntts jungle
Comvolution
aodd - Minue's Mooiit
Calibrated Cosine Similarity J4x
Post-hoc Probabilistic Models
Figure 2: Overview the UEC framework: UEC first transforms deterministic embeddings from multiple encoder models into probabilistic representations using Laplace approximation. These probabilistic embeddings are then adaptively combined by computing uncertainty- driven ensemble coefficients based per-dimension variances. Finally, similarity is measured using uncertainty- aware metric that accounts for both the mean and uncertainty of the ensembled embedding.
Subsequent works explored multimodal and vision-language tasks: Chun et al. (2021); Chun (2023) improved cross-modal retrieval and image-text alignment, yet remained modality-specific; Prob- VLM (Upadhyay et al., 2023) and Bayes VLM (Baumann et al., 2024) incorporated uncertainty into frozen vision-lange language models through adapters or post-hoc strategies, but with limited generality. Other studies investigated hedging against ambiguous inputs (Oh et al., 2018), compositional multi- modal retrieval (Neculai et al., 2022), incertainty-awan multimodal pre-training (Ji et al., 2023), and scalable probabilistic embeddings with Gaussian process latent variable models (Venkataramanan et al., 2025). Our work provides a Gaussian convolution formulation that not only generates proba- bilistic embeddings in post-hoc manner but also supports theoretically grounded and lightweight similarity estimation for downstream tasks.
UNCERTAINTY-DRIVER EMBEDDING CONVOLUTION
We propose Uncertainty-drriv driven Embedding Convolution (UEC), a principled framework for combining multiple embedding models by modeling predictive uncertainty.
UEC performs three simple steps (Figure 2): (1) converts each deterministic embedding model into probabilistic one that outputs Gaussian embedding (Section 3.1). (2) combines these probabilistic embeddings so that more confident models contribute more (Section 3.2), and (3) computes an uncertainty-awaat similarity score (Section 3.3). This enables robust and adaptive ensemble without manual tuning.
3.1 POST-HOC PROBABILISTIC EMBEDDING MODEL
Laplace Approximation To convert deterministic embedding model into probabilistic one, we estimate a Gaussian posterior over its last-layer parameters using the Laplace Approximation (LA) (MacKay, 1992; Ritter al., 2018). Given training data D and last-layer weights ( LA constructs a second-order Taylor expansion of the negative log-posterior log (L) (D) around the maximum posteriori (MAP) estimate W(4);
- log (log((W(u)) D) ~ log W W(L)D) + (W(L) W())
was (W(L) W(
where H We the Hessian of the negative oog-osterrirr evaluated at the MAP. This MAP solution corresponds to the final-layer parameters of the pre-trained embedding model. Prior works show that final-layer LA is effective and efficient (Daxberger et al., 2021; Hobbhahn al., 2022), yielding a Gaussian weight posterior without retraining. The full derivation of LA is provided in Appendix A.1.
3

---

Published as a conference paper at ICLR 2026
Gaussian Embedding Generation Building the LA posterior derived above, convert the pre-trained embedding model into probabilistic one in a post-hoe manner. Let f(x) be an em- bedding model with last-layer parameters W(2) and penultimate representation (xz-1)(x) In the deterministic setting, the embedding is z(x) W(L) h (x)). By applying last-layer LA to pre-trained deterministic embedding model, we convert it to probabilistic embedding model where
the last-layer following Gaussian posterior p p(((L) D) N (W()) H W(Ls):
The embedding
produced by the model becomes a Gaussian random vector obtained by propagating the probabilistic last-layer through the fixed representation:
(x) N W h (x),,, h(2-1)
W(L) H Wes ((=)))))))))))))
We adopt diagonal approximation to H Way for computational efficiency, following prior works (Daxberger et al., 2021; Zhdanov et al., 2025). These Gaussian embeddings are then leveraged construct an uncertainty-aware convolution across multiple embedding models.
3.2 UNCERTAINTY-DRRIERR COEFFICIENTS FOR EMBEDDING CONVOLUTION
Gaussian Convolution Suppose have K independent embedding models, each transformed through the procedure in Section 3.1 into a probabilistic embedding model that produces 2u (x(x) ~ (x), [x)) To integrate these into single representation, we perform Gaussian convolution in the embedding space. Formally, we define the convolutional embedding zxx) as the result of a weighted combination of independent Gaussian variables:
z(x)
(x) (x).
where (x) denotes the convolution coefficient assigned to model k. and Ek=1 (x(x))= =1 Since each zą (x) is Gaussian and the coefficients (x) are deterministic, the resulting embedding z(x) follows a Gaussian distribution as well, according to the properties of linear combinations of independent Gaussians (Shao, 2008):
z(x)~ ~ N
T& (x)
Σ -((x)
Gaussian convolution aggregates the centers (means) and spreads (covariances) of the imput Gaussians into single distribution. This process inherently allows for the propagation of uncertainty in closed form, enabling the ensemble to capture and represent epistemic uncertainty arising from disagreements or variations across different models.
Uncertainty-awane Coefficients To determine the coefficients (x) for Gaussian convolution, introduce an uncertainty- surrogate loss that captures how reliably the embeddings of positive pairs expected align. This formulation enables principled aggregation of multiple embedding models while explicitly accounting for model-specific epistemic uncertainty.
Text embeddings are often trained with contrastive objectives such as InfoNCE (Oord et al., 2018). For lz-normalized features, the squared Euclidean distance is directly related to cosine similarity, I|u v 2(1 u T v) (Wang & Isola, 2020). These losses also encourage alignment of positive pairs and uniformity on the unit hypersphere. Leveraging this connection, we adopt the squared loss as principled surrogate for contrastive objectives.
Let (x, x') denote a positive document pair. For each model k. the corresponding embeddings follow 2y (x) ~ (x) Ex (x)), z (x') (x)xxx)), Σg (x))). We define the uncertainty- aware surrogate loss, multi-task objective that aggregates model-specific embedding errors weighted by uncertainty-awe task coefficients, as:
Laur (T; x,,x
Σ Th (x) Ex((K) [llzx(x) zi 2x
(x)
[ un mx(x')) tr (x)) tr (x'))
(2)
Fidelity
Uncertuinty
4

---

Published as a conference paper at ICLR 2026
This surrogate loss decomposes into fidelity and epistemic uncertainty. While p& (x(xx) and (x( are available at retrieval time, computing (xxxx') and Σx(x') requires evaluating the embedding model every document. However, in practical retrieval systems, document embeddings are precomputed once and stored in the index, whereas the query x arrives only at inference time (Zhou et al., 2022; Zhao al., 2024). Since the surrogate loss in Eq. 2 contains terms that depend on x', which cannot be recomputed per query, we discard all document-dependeen terms and retain only the query-dependent components. This yields vetrieval-feasible approximation of Eq. 2, leading to the following entropy-regularized convex optimization:
K
K
min
(x) (tr(Ex + pa (x)13 - T TH(((x))
S.t.
T& (x = 1, TT. (x) 2 0, (3)
where T > 0 controls sensitivity to uncertainty and Hx*(x)) denotes the Shannon entropy of the coefficient vector π (x) : (x),,..... ................................ T (x)), encouraging smooth allocations across models and preventing concentrated solutions. From Bayesian perspective, this entropy regularization is equivalent (up constant) KL divergence between (x) and uniform prior over models, encoding a preference against overconfident model selection when uncertainty evidence is limited. Such entropy- -arropy-based regularization has been widely adopted to stabilize soft reweighting and gating mechanisms under heterogeneous or noisy conditions (Sagawa et al., 2019; Nguyen et al., 2024).
The entropy-regularived objective admits a closed-form solution with temperature-connrolled coeffi- cients:
exp - tr tx())(x))/TT)
T& (x; T)
Σド= exp (x))((
This approximation follows from & normalization, under which I| (x))12 is approximately constant across models. With this formulation, UEC aggregates multiple embedding models in query- adaptive and data-dependent manner. Rather than using fixed global weights, each encoder's contribution is modulated by its estimated epistemic uncertainty for the query assigning lower influence to unreliable models and emphasizing stable representations. Consequently, UEC performs principled convolution over probabilistic embeddings that adapts to query-wise heterogeneity and distributional shift. Full derivation and discussion are provided in Appendix A.2.
3.3
UNCERTAINTY-AWARE SIMILARITY ESTIMATION
Measuring distances between embeddings is fundamental to many downstream applications. While distributional distances like KL divergence (Kullback & Leibler, 1951) or Wasserstein dis- tance (Chhachhi & Teng, 2023; Gelbrich, 1990) are theoretically well-founded, their computational cost is often prohibitive. To address this, we propose a lightweight and principled similarity estimator.
Let the ensembled embeddings for x and x' be q 2x) ~ .V(qq Eq) and N(ee Ee), respectively. Our similarity measure a follows the properties of the base embedding models, which are typically trained with contrastive objectives &g no-nomaalized outputs. This training results in mean vectors plq He that are near unit-norm, making the small-variance assumption plausible. We explicitly normalize the mean vectors, allowing by the dot product, x q c. This preserves the Gaussian form, and the small variance ensures the approximation remains accurate. We then model the resulting dot product s as a Gaussian via moment matching (Randone et al., 2024; Mallik & Sagias, 2011):
z(x')
approximate cosine similarity
Mg Mc: To incorporate uncertainty, we adopt a probit approximation (Eschenhagen et al., 2021; Gibbs, 1998):
NNHu,, ))), Me
of = M. EoHay + me EqHe + tr tr(E.
(5)
(6)
The incorporates predictive uncertainty in lightweight manner without additional sampling. Beyond its computational efficiency, the estimator is theoretically grounded, as it approximates the squared 2-Wasserstein distance with provably bounded error, ensuring consistent ranking be- havior. The following result holds under a mild small- variance scaling assumption, formalized in Appendix A.4.
5

---

Published as a conference paper at ICLR 2026
Theorem (Bounded approximation to the squared 2- W-aassesttein distance). Let q Ni Eq) and с ~ Nue. Σe) be two Gaussian embeddings with fz-normalized mean vectors and diagonal Ee ti... -e.a). Assume the small-variance Then, our similarity estimator approximates the squared
covariances Σn diag( 1................ ........ oq. regime, where = max; {o q,3-
2. 0.
2- Wasserstein distance W22 as:
- +
Hence, ranking by induces the same ranking as minimizing W22 up to O error.
Our similarity estimator thus serves as a theoretically grounded surrogate for established distributional distances, achieving nearly identical ranking performance while being significantly more efficient. We provide detailed derivation the estimator Appendix A.3, the proof for Theorem in Appendix A.4, and empirical comparisons with alternative distances in Appendix B.2.
EXPERIMENTS
nDCG@10
Recall@10 En
AUC@10 (Normalized)
En
En
A
Ru Ar
Ru An
Ru
100
00
200
Zh
Zh
Zh
Uniform
Weighted
UEC
Oracle
Figure 3: Performance MIRACL Subset across ensemble methods. The oracle represents the upper bound by selecting the best language-specific model per language. UEC achieves performance comparable to the oracle and even surpasses in cases, with particularly strong gains in AUC@ 10.
4.1 MIRACL SUBSET
1.0
En
0.472
0.312
0.216
Setting We first build a toy example with
language-specialized models: E5 (Wang et al., 2022) (Fnglish/Russian), BGE-zh (Xiao al., 2023) (Chinese), and MATRYOSHKA (Kusu-
Dataset
Ru
zh
0.172
0.689
0.139
0.4
pati et al., 2022; Henderson et al., 2017)
(Arabic). From the MIRACL Hard Negative dataset (Zhang et al., 2023), we sample 100
0.2
0.096
0.029
0.875
query-passage pairs each in English, Russian, Chinese, and Arabic. Baselines include uniform and weighted ensembles, along with oracle upper bound that selects the best model per lan- guage. We report nDCG @ 10 and Recall @ 10 for retrieval and AUC@ 10 for uncertainty (Enevold- sen et al., 2025). Full experimental details are in Appendix B.3.
MATRYOSHKA
0.00
Ensemble Coefficient
Figure 4: Heatmap of model-wise coefficients as- signed by UEC per language. Each corre- sponds to a language-specific input, and each col- umn an ensemble coefficient. UEC computes ensemble coefficients that are adaptively modu- lated by the uncertainty of each embedding.
Results As shown in Figure 3, UBC consistently outperforms uniform and weighted ensembles. For retrieval metrics (nDCG@ 10, Recall @ 10), it achieves performance comparable to the oracle. UEC explicitly models embedding uncertainty and adaptively leading to similarity scores that better aligned with retrieval correctness. This enables UEC to surpass the oracle on the uncertainty metric AUC@ 10, highlighting that uncertainty- aware ensembling not only preserves strong retrieval performance but also delivers superior confidence calibration.
weights unreliable representations,

---

Published as a conference paper at ICLR 2026
To qualitatively assess how UEC adapts ensemble weights via uncertainty, visualize the averaged model-wise coefficients per language in Figure 4. The 4 x 3 heatmap shows languages (rows: English, Russian, Chinese, Arabic) against ensembled models (columns). UEC assigns distinct patterns by language-for example, Arabic inputs emphasize Arabic embedding, while Chinese inputs favor Chinese embedding. It demonstrates that UEC effectively leverages embedding uncertainty assign coefficients.
4.2 MMTEB
We further evaluate on a subset of MMTEB (Enevoldsen et al., 2025), focusing on three representative tasks: retrieval, classification, and Semantic Textual Similarity (STS). For each task, we carefully select datasets to cover a diverse range of domains and topics. Additional task results, which show performance consistent with the main results, are provided in the Appendix B.4.5. We use three SBERT-style base models-BGE (Xiao et al., 2023), E5 (Wang et al., 2022), and GTE (Li et al., 2023) and competitive multilingual baseline (GTE-MB (Zhang al., 2024)). As a probabilistic embedding baseline, we adopt GroVE (Venkataramanan et al., 2025), applying it on top of the strongest individual model GTE-MB for fair comparison. To the best of our knowledge, this work presents the first systematic evaluation of medel-m guniform/weighte et al., 2022)) and ensemble (uniform/weighted) techniques for this benchmark. See Appendix B.4 for more experimental details.
task arithmetic (Ilharco
4.2.1 RETRIEVAL
Setting We evaluate retrieval performance five datasets: SCIDOCS (Cohan et al., 2020), Legal- BenchhorpprrteLoobbying (Guha et al., 2023), BelebeleRetrieval (Lovenia et al., 2024), WikipediaRe- trievalMultilinguall, and StackOverflow (Li et al., 2024). We use nDCG@ 10 and Recall 100 as retrieval metrics. In addition to standard retrieval metrics, we use AUC@ 10 (Enevoldsen et al., 2025) as key quantitative measure assess how well the model's confidence scores are calibrated with its actual performance, directly evaluating the quality of its uncertainty estimation. We additionally compare Borda count (Emerson, 2013) as a representative rank aggregation technique, natural strategy for retrieval tasks where each embedding model yields its own ranks.
Table Comparison of retrieval performance. The best and second-bes results per row are bolded and underlined. UEC achieves the highest average performance across all metrics, with particularly strong gains on the AUC@ 10 uncertainty metric, highlighting the efficiency for uncertainty calibra- tion. (Borda=Borda count, Uni Wt.=Weighted, TA=Task arithmetic).
Individual Models
Model Merging
Ensemble
Borda
Dataset Metrie
BGE
E5
GTE
GTE-MB
GroVE
Uni.
Wt.
TA
Uni.
Wt.
UEC
SCIDOCS
nDCG@ 10
21.47
18.30
23.14
19:46
21.88
22.18
21.73
22.43
22.66
22.67
22.11
24.01
Recall 100
51.04
41.71
53,27
45.03
50.16
51.97
49.00
51.31
52.87
52.98
50.53
54.36
10
27.25
34.42
31.63
32.52
34.03
5.84
29.42
30.99
30.72
28.52
32.08
35.21
Lobbying
aDCG@11 10
90.42
91.54
91.81
91.23
92.35
92.32
93.56
Recall 100
99.70
100.00
99.70
99.70
99.70
99.70
100.00
99.70
99.41
100.0
100.0
100.0
AUC@10
55.56
58.24
55.02
54.62
32.83
57.49
53.17
53.18
52.07
55.79
60.04
Belehele
nDCG@ 10
93.64
94.91
93.76
94.09
94. 14
96.26
94.60
94. 86
93.87 87
2,
93.64 64
95.82
Recall 100
99.44
99.55
99.77
99.66
99.44
99.44
99.66
99.66
99.44
99.66
99.55
99.77
AUC 10
83.83
84,78
86.18
82.53
86.57
69.17
86.39
85.11
86.03
82.99
83.34
88.72
Wilipedia
nDCG@ 10
92.17
93.32
92.71
91.55
92.71
93.12
91. 98
93.08
92.99
93.32
93.84 84
94.24
@ 100
99.80
99.93
99.93
99.80
99.86
99.55
99.86
99.86
99.80 80
99.86
99.86
99.93
AUCe 10
59.84
61.06
61.57
62.08
62.36
49.26
58.61
60.60
62.62
62.41
62.01
64.25
StackOverflow
nDCG@ to
79.93
87.14
80.54
91.17
87.76
85,36
78.93
81.31
80.55
85.76
88.71
90.26
Recall 100
96.59
97.54
97.66
98.79 79
97.54
98.09
96.28
97.44
96.54
98.09
98.09
99.40
AUC 10
69.59
84,00
68.28
17.14
69.69
73.89
71.27
82.43
89.83
Avg. aDCG@101 1
75.52
77.04
76.39
77.33
77.48
77.11
75.90
75. 69
76.19
77.76
49
79.58
Avg. Recall@ 100
89.31
87.74
90.06
88.60
89.34
89.75
88.96
89.59
89.61
90.12
89.61
90.69
Avg. AUC@ 10
54.65
64.70
60.74
64.07
66.16
34.85
60.52
60.75
60.76
60.30
63.13
67.61
Results As shown in Table 1, UEC attains the highest average performance across both retrieval and uncertainty estimation, surpassing even the strong GTE-MB baseline by ensembling three weaker
7

---

Published as a conference paper at ICLR 2026
models. These results demonstrate that uncertainty-awart ensemble framework effectively integrates heterogeneous embeddings, achieving robust and accurate retrieval across diverse datasets.
4.2.2 CLASSIFICATION
Setting We evaluate on five classification datasets covering finance, law, poetry, multilingual intent, and social media: FinancialPhrasebuni wlanssniiiaassitatttonn (Malo et al., 2014), SwissJudgemennCussiff cation (Niklaus et al., 2021), ), PremmSentimenCCassffiiaaa Classification (FitzGerald et al., 2022), and TeeeeeopppsSnnlleCassieeeeeee Accuracy, Fl score, and AUROC are reported as evaluation metrics, where AUROC (Kuhn et al., 2023) serves key quantitative measure assess how well the model estimates the degree of uncertainty in distinguishing between classes.
(Sheng & Uthus, 2020), Massivelntent- (Antypas et al., 2022).
Table 2: Comparison of classification performance. The best and second best results per row are bolded and underlined. UEC achieves the highest average scores, demonstrating robust performance across diverse classification tasks (Uni.UUnifrrm, Wt.=Weighted, TA=Task arithmetic).
Individual Models
Model Merging
Ensemble
Dataset Metric
BGE
E5
GTE
GTE-MB
GroVE
Unif.
Wt.
TA
Unif.
Wt.
UEC
Financial
Accuracy
80.47
82.92
81.89
75.07
76.73
77.64
79.39
79.53
82.15
82.64
83.02
F1
78.38
79.90
79.10
73.50
75.65
75.53
77.85
77.93
79.76
80.18
80,30
AUROC
92.46
93.04
92.02
90.62
91.13
89.63
91.92
92.38
92.86
93.16
94.07
Judgement
Accuracy
57.16
58.58
56.37
55.45
55.44
56.82
55.90
55.44
57.09
57.71
58.36
F1
46.63
47.25
46.34
46.41
45.82
46.55
45.87
45.73
46.61
47.02
48.52
AUROC
50.48
50,22
50.65
50.67
50.48
50.13
50.24
50.37
50.43
50.51
51.02
Poem
Accuracy
51.92
53.07
47.88
56,92
53.13
50.28
49.90
51.82
52.30
52.01
56.81
40.76
37.53
45.53
41.82
40.93
40.96
41.00
44.46
AUROC
56.70
57.38
53,41
62.21
57.31
55.13
54.36
55.76
56.71
56.27
61.39
Intent
Accuracy
73.46
68.05
65.09
75.19
74.29
72.95
74.07
75.33
68.78
68.31
77.08
FI
70.01
63.00
60.62
72.79
72.80
71.01
72.10
73.65
66.43
65.46
74.78
AUROC
66.74
64.54
63.09
67.23
66.98
64.07
67.15
67.02
65.26
65.
67.91
Topic
Accuracy
71.75
69.96
71.97
70.99
71.24
71.18
70.59
70.53
73.60
73.47
74.20
55.04
53.09
54.65
$4.85
54,93
55.48
54.78
54.82
56.32
56.13
57.14
AUROC
87.02
87.83
88.54
89.76
87.95
88.64
88.12
88.33
90.07
90.12
90.73
Avg. Accuracy
66.95
66.52
64.64
66.73
66.
65.77
65.97
66.53
66.79
66.83
68.89
Avg. F1
58.16
56.93
55.65
58.62
58.20
57.57
57.99
58.61
58.02
57.96
61.04
Avg. AUROC1
70.68
70.60
69.54
72.09
70.77
69.52
70.35
70.77
71.06
71.03
73.02
Results As shown in Table 2, UEC attains the highest average accuracy, F1 score, and AUROC
across five classification datasets. Notably, UBC also yields the best average AUROC, indicating that the model effectively captures the degree of uncertainty. These results demonstrate the effectiveness of UEC in classification scenarios where embedding uncertainty plays crucial role.
4.2.3 SEMANTIC TEXTUAL SIMILARITY
Setting We evaluate STS performance on 10 datasets, including STSBenchmark, FinParaSTS, SICK-R Marelli et al. (2014), STS22.v2 Chen et al. (2022), SemRe124STS (Ousidhoum et al., 2024). and STS12-STSI7 (Agirre al., 2012; 2013; Bandhakavi et al., 2014; Biçici, 2015; Cer al., 2017) Spearman correlation is used to measure alignment with human-annotated similarity scores.
Results As shown Table 3. UEC achieves the highest average performance across the STS tasks. It ranks first on out of the 10 datasets. These results demonstrate that uncertainty- aware aggregation not only enhances robustness across diverse semantic similarity tasks but also outperforms existing deterministic and ensemble baselines.
8

---

Published as a conference paper at ICLR 2026
Table Comparison of STS performance. The best and second-best results per column are bolded and underlined. UEC achieves the highest average performance, ranking first on 8 out of the 10 datasets evaluated (Uni. Unnffrrm,,,, Wt.=Weighted, TA=Task arithmetic).
Models
STSB
FinPara
SICK-R
SemRel22
STS12
STS13
STS14
STS15
STS17
STS22
Avg.
BGE
86.41
9.43
80.30
79.52
78.02
84,18
82.27
87.95
86.41
66.53
74.10
85.47
15,99
78.39
81.37
73.49
82,99
80.44
88.18
88.89
66.51
74.17
Individual
GTE
85.73
12.95
78.85
77.79
75.70
85,72
81.51
88.80
87.88
67.65
74.26
GTE-MB
85.41
15.18
75.71
75.02
74.82
86.63
78.75
85,39
88.31
72.28
73.75
GroVE
85.93
11.53
80.02
78.99
76.26
83.33
79.03
85.27
88.89
66.98
73.62
Model
Uni
86.78
11.75
79.85
79.49
75.43
86.79
82.07
89.12
88.71
68.42
74.84
Merging
WL
87.22
11.64
80.16
78.80
77.36
87.08
83.39
59.49
88.43
67.51
75.11
TA
86.83
10.22
80.25
78.61
78.40
85.85
83.18
88.67
87.41
66.31
74.57
Uni.
87.44
12.03
80.73
80.24
78.00
85,90
83.62
89.07
86.21
68. 15
75. 34
Ensemble
WL
87.44
14.13
80.41
81.08
77.07
85.76
83.45
89.17
89.18
68.68
75.64
UEC
87.55
15.53
81.37
81,89
78.85
87.62
83.75
89.50
89.24
69.55
76.49
4.3
UNCERTAINTY ESTIMATION DIAGNOSIS
To assess whether the post-hoe probabilistic embeddings and the resulting similarity variances provide meaningful uncertainty estimates, we evaluate two aspects: (1) calibration of the Laplace- based probabilistic embedding model used in UEC, and (2) whether the similarity variance o faithfully reflects predictive uncertainty. For all experiments, we fit LA following the same configuration with Section 4.2, and evaluate uncertainty using a binary classification proxy task on MS MARCO.
4.3.1 CALIBRATION OF PROBABILISTIC EMBEDDINGS
Table 4: Calibration of prob- abilistic embeddings derived via LA. Probabilistic embed- dings outperform the deter- ministic ones consistently.
We first measure how well the Laplace-based probabilistic embed- dings capture epistemic uncertainty at the model level. Across all models, LA consistently improves calibration, and UEC yields the best-calibrated predictions. These findings demonstrate that the post- hoc probabilistic embedding model provides stable and meaningful estimate of parameter uncertainty. Further analyses- covlyses-cvvarincce structure, number of layers included in LA, and data sparsity- are provided in Appendix C, where we observe consistent robustness across all factors.
ECE 4
Deter:
Prob.
E5
0.063
0.036
BGE
0.067
0.042
GTE
0.077
0.043
UEC
0.075
0.032
4.3.2 VARIANCE CALIBRATION OF THE SIMILARITY SCORE
We next assess whether UEC's similarity variance o3 meaningfully reflects predictive uncertainty.
Since standard ECE does not apply continuous sim- ilarity values, we introduce a variance-error metric (Var-ECE) that measures the alignment between predicted variance and empirical squared error. For each example, higher predicted variance should correspond to a larger squared residual. We group examples into variance-quantile quantile bins and compute the absolute discrepancy be- tween predicted variance and empirical squared error. The full derivation and extended diagnostics remain in Appendix C.2.
Var-ECE
Table 5: Variance calibra- tion on SEMREL24STS. UEC achieves the lowest Var-ECE, indicating that o? faithful esti- mate of similarity uncertainty.
Model Var-ECE 4
E5
0.035
Results On the multilingual SEMREL24STH benchmark, UEC achieves the lowest Var-ECE (Table 5), showing that of provides a numerically faithful estimate of similarity uncertainty.
BGE
0.051
GTE
0.038
UEC
0.028
4.4 EFFICIENCY AND COMPUTATIONAL CHARACTERISTICS
Table compares the computational characteristics of UEC against common ensemble basclines. Three key capabilities considered: automatic coefficient selection, taie coefficient estimation, and uncertainty- similarity estimation. Among all methods, only UEC supports all three features (V), while others lack adaptability require manual tuning (X, -).
Despite its added functionality, UEC introduces negligible overhead. It maintains the same asymptotic time and memory complexity (DD), where D is the embedding dimension. UEC increases only

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of computational characteristics. UEC supports automatic coefficient selection and uncertainty-awar aware similarity while preserving efficiency. "Rel." and "Comp." denote relative time and complexity, respectively. Symbols: / (Yes), (No), (Not applicable).
Auto Coeff.
Data-wise Coeff.
Uncertainty-awan Similarity
Memory Complex.
Time
Method
Selection
Rel.
Complex.
Uniform Ensemble Weighted Ensemble
X
O(KD)
1.000
O(KD)
x
X
O(KD)
1.000
O(KD)
Task Arithmetic
X
X
O(KD)
1.000
O(KD)
UEC (Ours)
O(KD)
1.006
O(KD)
0.6% in actual computation time for similarity estimation, demonstrating strong adaptivity without sacrificing efficiency and indicating its suitability for real-time deployment.
4.5 ABLATION STUDY
Table 7: Ablation for UEC's components: uncertainty-ave awware similarity (Une Sim) and adaptive convolution (Unc Conv). Removing either degrades performance, with the joint removal yielding the worst results.
We perform ablation study to evaluate each mod- ule's contribution. Following the protocol in Sec- tion 4.1, replace (i) the uncertainty-aware simi- larity with an uncalibrated one (- Unc Sim), (ii) the uncertainty-driier convolution coefficient with uni- form one (- Unc Conv), and (iii) both (- Une Sim, Conv). Table 7 shows that removing any component noticeably drops performance, highlighting their in- dividual and complementary contributions.
nDCG@
Recall@
UEC Une Sim
59.65%
80.07%
91,0%%
58.72%
78.13%
82.4%%
(L 0.93%)
(11.949)
(18.56%)
Une Conv
48.45%
66.69%
10.30%
(L 11.20%)
(1) 13.38%)
(1 30.74%)
Une Sim, Conv
46.78%
62.66%
4.01%
( 22.87%)
(1 17.41%)
(L $7.03%)
4.6 CHALLENGING CASE
Figure shows a qualitative example from the setting in Section 4.1. Although the query is in English and E5 is expected to perform well, all individual models fail to retrieve the cor- rect passage, which appears at ranks 12, 28, and 37. In contrast, UEC retrieves it at rank 6, thereby correctly retrieving the answer un- der both nDCG@ 10 and Recall @ 10 metries, which are evaluated over the top 10 candidates. This example highlights UEC's ability to handle model limitations by using uncertainty- ensembling and to generalize beyond language- specific encoders.
What language they speak
Buenos multicultural city, being multiple ethnic and religious groups. Several languages are spoken the city in addition to Spanish...
Positive
Passago
Models
(En/RRu)
E5
bge-dh (2h)
Matryoshka (Ar)
(Ours)
UEC
Rank
12
28
37
nDCG@10
Recal1@10
Figure 5: Challenging case where only UEC cor- rectly retrieves the positive passage, demonstrating its ability to leverage uncertainty.
CONCLUSION AND DISCUSSION
We propose the envertainty-driven Embedding Convolution (UEC) framework, which converts pre-trained deterministic embeddings into probabilistic representations and adaptively ensembles them via Bayesian coefficient estimation and an uncertainty-awarr similarity function. To the best of our knowledge, this introduces the first fully post-hoc, uncertainty-calibrate ensemble approach for embedding models, yielding consistent and reliable gains across diverse tasks. While UEC currently models only epistemic uncertainty and relies on diagonal LA, it does not yet capture aleatoric or full predictive uncertainty. Extending UEC to incorporate these additional uncertainty components and to explore richer posterior structures with efficient computation represents promising direction for enhanced robustness. Furthermore, UBC assumes that embeddings share the same dimensionality and may inherit biases from the underlying models. Relaxing the dimensionality constraint and developing uncertainty mechanisms that explicitly account for bias and fairness offer important avenues for responsible real-world deployment. Finally, applying UEC to multimodal settings, in which heterogencous sources of uncertainty interact across vision, speech, and text, presents an exciting opportunity broaden the framework's 's impact.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea govermmenndMSSSTT/88822044444440
REFERENCES
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. Sib-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. arXiv preprint arXiv:2309.0742 2023a.
David Ifcoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure FP Dossou, Akintunde Oladipo, Doreen Nixdorf, et al. Masakhanews: News topic classification for african languages. arXiv preprint arXiv:2304.09972, 2023b.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot semantic textual similarity. in sem 2012: The first joint conference on lexical and computational comantics-voluns and volume 2: Proceedings of the sixth international workshop on semantic evaluation (semeval 2012). Proceedings the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, QC, Canada, pp. 7-8, 2012.
Proceedings of the main conference and the shared task,
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics ( SEM), volume proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32-43, 2013.
Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados Leonardo Neves, Vitor Silva, and Francesco Barbieri. Twitter Topie Classification. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea, October 2022. Inter- national Committee on Computational Linguistics.
Anil Bandhakavi, Nirmalie Wiratunga, Stewart Massie, et al. Generating a word-emotion lexicon from# emotional tweets. In Proceedings of the third joint conference on lexical and computational semantics (* SEM 2014), pp. 12-21, 2014.
Oren Barkan. Bayesian neural word embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Amo Solin, and Martin Trapp. Post-hoc probabilistic vision-language models. arXiv preprint arXiv:2412.06014, 2024.
Ergun Biçici. Rim-deu: Predicting semantic similarity with referential translation machines. In Proceedings the 9th international workshop on semantic evaluation (SemEval 2015), pp. 56-63, 2015.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint M.XXiv:558..5222. 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual sumilarity-mutiliingg arXiv: (00i:0700800055. 2017.
and cross-lingual focused evaluation. arXiv preprint
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Ceeiiadddo-Cseeess.... Steve Yuan, Chris Tar, et Universal sentence encoder. arXiv preprint arXiv: 100.111117711 2018.
Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David Jurgens, and Mattia Samory. SemEval-2022 task 8: Multilingual news article similarity. In Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer,
11

---

Published as a conference paper at ICLR 2026
Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 1094-1106, Seattle, United States, July 2022. Association for Computational Linguisties. doi: https: / acl antho org/20222 s
1.155. URL
semeva 1 155.
Xinchi Chen, Xipeng Qiu, Jingxiang Jiang, and Xuanjing Huang. Gaussian mixture embeddings for multiple word prototypes. arXiv preprint arXiv: 1511.1224.666 2015.
Saurab Chhachhi and Fei Teng. On the 1-wasserstein distance between location-scale distributions and the effect of differential privacy. arXiv preprint arXiv:2304, 14869, 2023.
Sanghyuk Chun. Improved probabilistic image-text represetttttr arXiv preprint arXiv:2305. 2023.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross- retrieval. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Benjamin Clavié. Jacolbert and hard negatives, towards better japanese-fins first emheddings for retrieval: Early technical report. arXiv preprint 12Xiv:2312.1614: 2023.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. Specter: Document-
level representation learning using citation-infonnn informed transformers. ACL, 2020.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in neural information processing systems, 11-20089-201055 2021.
Leon Derczynski and Alex Speed Kjeldsen. Bornholmsk natural language processing: Resources and tools. In Proceedings the Nordic Conference of Computational Linguistics (2019), pp. 338-344. Linköping University Electronic Press, 2019.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint MiXiv1181.004555 2018
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pp. 4171-4186, 2019.
Peter Emerson. The original borda count and partial voting. Social Choice and Welfare, 40(2): 353-358, 2013.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński,, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Çagatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafal Pošwiata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Bjöm Plüster, Jan Philipp Harries, Loic Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Suppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang. Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lå, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff,, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhay Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv 2025. doi: 10. 10.48550/ar Xiv 13595. URL https: rxiv. org/ abs 2502 13595.
Hippolyte
12

---

Published as a conference paper at ICLR 2026
Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of laplace ap- proximations for improved post-hoc uncertainty in deep learning. arXiv preprint atXiv22111033577
2021.
Lanting Fang, Yong Luo, Kaiyu Feng, Kaiqi Zhao, and Aiqun Hu. Knowledge-eehhanced ensemble learning for word embeddings. In The World Wide Web Conference, WWW '19, pp. 427-437, New York, NY. USA, 2019. Association for Computing Machinery. ISBN 45056666677 doi:
111455/300855.331132222 URL https: /doi. /tt/10....0......../// 1145/ 3308558. .01.558.333440 3313425.
Christian Federmann, Tom Koemi, and Ying Xin. Ntrex 128-news test references for mt evaluation of 128 languages. In Proceedings of the first workshop on scaling up multilingual evaluation, pp. 21-24. 2022.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. Massive: A Im-example multilingual natural language understanding dataset with 51 typologically-divere languages, 2022.
Jay Gala, Pranjal A Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, et al. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. arXiv preprint arXiv:2305.16307, 2023.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0880 08821 2021.
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: transformer with extendable qa agent classifiers. arXiv preprint arXiv:2104.07088 2021.
Matthias Gelbrich. On formula for the 12 wasserstein metric between measures on euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185-203, 1990.
Mark N Gibbs. Bayesian Gaussian processes for regression and classification. PhD thesis, Citeseer, 1998.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Whoo,, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Ratt Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. Legalbench: A collaboratively built benchmark for measuring legal reasoning large language models, 2023.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. Efficient natural language response suggestion for smart reply, 2017.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers),
pp. 4129 4138, 2019.
Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for classifica- tion with bayesian deep networks. In Uncertainty in artificial intelligence, pp. 822-832. PMLR, 2022.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution, pp. 162-190. Springer, 1992.
13

---

Published as a conference paper at ICLR 2026
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujuu Yang. Map: Multimodal uncertainty-aware vision-language pre-training model. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 23262- 23271, 2023.
Mohammad Aflah Khan, Neemesh Yadav, Mohit Jain, and Sanyam Goyal. The art of embedding fusion: Optimizing hate speech detection. arXiv preprint arXiv:2306. 14939, 2023.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302009664 2023.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of
mathematical statistics, 22(1):79-86, 1951.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-SSyder Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
Antoine Lefebvre-Brossardd Stephane Gazaille, and Michel C Desmarais. Alloprof: a new french question-answer education dataset and its use in information retrieval case study. arXiv preprint arXiv:2302.07738, 2023.
Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. Coir: A comprehensive benchmark for code information retrieval models. arXiv preprint vtXiv:2407.02883 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint 23.30223, 2023.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
Ziwei Liu, Qi Zhang, and Lifu Gao. Optimizing sentence embedding with pseudo-labeling and model ensembles: A hierarchical framework for enhanced nlp tasks, 2025. URL https: / //arxiv.. org/abs/2500 25011 15876.
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo. William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria. Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov. Wei Qi Leong, Quyet Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Ngee Chia Tai, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng-Xin Xin Yong, and Samuel Cahyawijaya. Seacrowd: A multilingual multimodal data hub and benchmark suite for southeast asian languages. arXiv preprint arXiv: 2406. 10118, 2024.
Xing Han LU, Zdenčk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024.
Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. Cedr: Contextualized embed- dings for document ranking. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp. 1101-1 1104, 2019.
14

---

Published as a conference paper at ICLR 2026
David JC MacKay. Bayesian interpolation. Neural computation, 4(3):415 447, 1992.
Ranjan K Mallik and Nikos C Sagias. Distribution of inner product of complex gaussian random vectors and its applications. IEEE transactions on communications, 59(12):3353- 3362, 2011.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216- 223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http:/ / /www. rec- con o org/ http/p/oceediiggeeo http:/iinnnnn//e/e/////e rec2014 Feee2204//dff/355551
ptt01//////3333Papppede3e Papeeeeeeeepppe_33PP pdf..
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.073 2022.
Andrei Neculai, Yanbei Chen, and Zeynep Akata. Probabilistic compositional embeddings for multimodal image retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4547-4557, 2022.
Huy Nguyen, Pedram Akbarian, and Nhat Ho. Is temperature sample efficient for softmax gaussian mixture of experts? arXiv preprint arXiv:2401.13875 2024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. Ms marco: human generatdd machine reading comprehension dataset. 2016.
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. tiiis-judgment-preddie A multilingual legal judgment prediction benchmark, 2021.
Scong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. Modeling uncertainty with hedged instance embedding. arXiv preprint arvi: 11x::vv11000 1800.00319, 201 8.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Xiv: 807.007..770 .03488, 2018
Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, et al. Semrel2024: A collection of semantic textual relatedness datasets for 13 languages. arXiv preprint arXiv:2402.08633 2024.
Francesca Randone, Luca Bortolussi, Emilio Incerto, and Mirco Tribastone. Inference of probabilistic programs with mnoment-matchinn gaussian mixtures. Proceedings of the ACM on Programming Languages, ((POPL)::882-199))) 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint Xiv: 1908. 1008.10088 1008,, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th international conference on learning representations, ICLR 2018-conference track proceedings. volume 6. International Conference on Representation Learning, 2018
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pp. 410- 420, 2007.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 1et.111111077777 108731, 2019.
15

---

Published as a conference paper at ICLR 2026
Fredrik Carlsson Magnus Sahlgren. Sentence embeddings by ensemble distillation, 2021. URL https://axxiv/ org/ alss/ 06719.
Jun Shao. Mathematical statistics, Springer Science & Business Media, 2008.
Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint arXiv:1906.03744 2019.
Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In International Conference on Artificial Intelligence and Statistics, pp. 7694-7772 PMLR, 2023.
Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. Sen2pro: A probabilistic perspective to sentence embedding from pre-trained language model. arXiv preprint arXiv:2306.02244 2023.
Emily Sheng and David Uthus. Investigating societal biases in poetry composition system, 2020.
Kai Shuang, Zhixuan Zhang, Jonathan Loo, and Sen Su. Convolution-deconvooli word embed- ding: An end-to-end multi- prototype fusion embedding method for natural language processing. Information Fusion, 53, 06 2019. doi: fo.100//jinfffus.. fefffuus.2019000 2019.06.009.
Harman Singh, Nitish Gupta, Shikhar Bharadwaj. Dinesh Tewari, and Partha Talukdar. Indiegenbench: multilingual benchmark to evaluate generation capabilities of Ilms on indic languages. arXiv preprint ar ArXiv:2404. 16816, 2024.
Jörg Tiedemann and Santhosh Thottingal. Opus-nt-building open translation services for the world. In Amnual Conference of the European Association for Machine Translation, pp. 479-480. European Association for Machine Translation. 2020.
William Timkey and Marten Van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109, .04404, 2021.
Uddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Probvim: Prob- abilistic adapter for frozen vison-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1899-1910, 2023.
Aishwarya Venkataramanan, Paul Bodesheim, and Joachim Denzler. Probabilistic embeddings for frozen vision-language models: Uncertainty quantification with gaussian process latent variable models. arXiv preprint arXiv:2505.055 0666,, 2025.
Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. arXiv preprint arXiv: 14xiv:1412.666 141.6663, 2014.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly supervisd contrastive pre-train pre-training. arXiv preprint arXiv:2212.03533. 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-9939. PMLR, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tume tnn-ttund models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xi- angsheng Li, Haitao Li, Yiqun Liu, al. T2ranking: A large-scale chinese benchmark for passage ranking. In Proceedings the 46th International ACM SIGIR Conference on Research 2023.
and Development in Information Retrieval, pp.
16

---

Published as a conference paper at ICLR 2026
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.0766 07666,, 2024.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1393-1412, 2024.
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114-1131 2023. doi: 16.1/all a 00595. URL https: / /t////aaaaaantooo////e/
org/ 2023. tacl - 16..l1116.........
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey, ACM Transactions on Information Systems, 42(4):1-60) 2024.
Maksim Zhdanov, Stanislav Dereka, and Sergey Kolesnikov. Identity curvature laplace approximation improved out-of-distribution detection. In 2025 IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 7019-7028. IEEE, 2025.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, and Ji-Rong Wen. Dynamicretriever: A pre-training model- based system with neither sparse nor dense index. arXiv preprint arXiv:2203.00537 2022.
17

---

Published as a conference paper at ICLR 2026
A DERIVATION AND PROOF
A.1 DETAILED DERIVATION OF THE LAPLACE APPROXIMATION
Let D {(x,, ye) denote the training set and W(L) the parameters of the final linear layer of the embedding model. The posterior over W(L) is given by Bayes' rule:
p (W(L)) (L) D) p(D (()))((()))
P(D)
The exact posterior is typically intractable. The Laplace Approximation (LA) expands the negative log-ponn peg-pottrrio around its maximizer (the MAP estimate)
W arg max (WL)) D). W
Quadratic Expansion of the Log-Posterior
Define the log- log-posterior
((W(L) log p(D) w log p( (())..
A second-order Taylor expansion around W (L) gives
((w(1)) = (((()) (w(1) W
-W())
(8)
Since (LL) is a maximizer,
V((W())) = O.
Define the negative Hessian (precision matrix)
H Was --"(((WL)) v log W(() log log W(D))
w(o)=
Thus the posterior is approximated by
p(W(1) (D) N (w
H W
Posterior over Embeddings Following Section 3.1, let fxx be an embedding model with last- (x)).. (x). The embedding given by the
layer parameters ((L) and penultimate representation (4-1) final linear projection
z(x) (T-11 (x).
Under the Gaussian posterior
W N
WiZ)
the embedding z(x) becomes Gaussian random variable since a linear transformation of WL))
Using standard results on affine transformations of Gaussian variables, the predictive distribution of the embedding is given by
zxx) ~ - (W-)))))( (4-1)) (x) T Wa hx4-)) Wa,
(9)
Therefore, the deterministic embedding model converted into a probabilistic one that outputs a Gaussian embedding for each input x, where the mean corresponds to the original embedding and the covariance captures epistemic uncertainty induced by the last-layer parameter posterior.
Diagonal Precision Approximation In practice, computing the full Hessian is computationally expensive. We therefore adopt a diagonal approximation of the precision matrix:
H Waa & diag H w
This approximation preserves tractability while retaining model-specific epistemic uncertainty infor- mation, yielding a closed-form Gaussian embedding distribution used in the UEC framework.
18

---

Published as a conference paper at ICLR 2026
A.2 DERIVATION OF UNCERTAINTY-DRIVEN COEFFICIENT FOR EMBEDDING CONVOLCTION
In this section, we provide a detailed derivation of the uncertainty-driver coefficients used in the embedding convolution described in Section 3.2. In contrast hard selection induced by linear objective, we derive a lemperature-connrol regularization, which yields stable and query-adaptive coefficients.
closed-form solution by introducing entropy
Problem Setting
Let there be K independent embedding models. Each model & E K} produces probabilistic embedding
z (x) (x)xxx), Ex(x))
for imput Similarly, for positive example x', we define
Zk (x') (H)x (x
The ensembled embedding is defined as a convex combination:
K
K
z(x)
T& (x)
Tx(x) = 1, (x(x)) 2 0.
k=1
Surrogate Loss Following prior analyses of contrastive representation learning (Wang & Isola, 2020), adopt the squared Euclidean distance surrogate objective. For Ez-normalized features, minimizing squared distance is equivalent to maximizing cosine similarity, which captures the alignment property of contrastive learning while avoiding explicit negative sampling.
For positive pair (x, x'), we define the uncertainty-awar surrogate loss as
Lour (T)
Σ
(xx ||
Expected Distance Between Gaussian Embeddings For independent Gaussian variables 2x(x) A (()xx)) (x)x) and (x') ~ (x)xx(x), (x)))) the expected squared distance admits the closed-form expression:
E 2 (x) z (x())] || [xx(x) (x')) tr(Exxx)) + tr(Ex(x)).
This decomposition separates a fidelity term, capturing semantic alignment, and an epistemic uncer- tainty term, reflecting model confidence.
Setrieval-fessbbee Approximation In large-scale retrieval systems, document embeddings cor- responding to x' are precomputed and fixed, while the query x arrives only at inference time. To respect this constraint, discard all document-dependeer terms and retain only the query-dependent components, yielding the following query-time objective:
K
min (x) (tr (I(EE(())= (x)) s.t. Σ (x) 1, (x)x) 2 0, (10)
where H(T(()) = -x (x(x) (x)) denotes the Shannon entropy.
The entropy term prevents degenerate solutions that collapse onto a single model and can inter- preted as a KL divergence to uniform prior over models, thereby promoting robust aggregation under noisy, query-specific uncertainty.
Closed-form Solution The optimization problem in Eq. 10 is strictly convex. Introducing Lagrange multiplier X for the simplex constraint, the Lagrangian
L(T, A) Σ T&Ck + Σ Th log Th +
Σ Th -1)
k
19

---

Published as a conference paper at ICLR 2026
where c₁ (x() + (x)]1.
Setting a = yields
1 log n3
A
1,
T
which implies
Ck (-T))
TT X exp
Normalizing over k gives the closed-form solution
exp( -kk(-ck/7 Cx/
(x)
(11)
ビコ=1 exp(-cy/T)
Approximation under l2 Normalization In practice, embeddings are lg-normalized, implying |/ (x)]l² across models. Treating this term as approximately constant, Eq. 11 reduces to
exp(-ttrExxxx))12 (x) Σ, (x
which is equivalent to the power-law form in Eq. 4 of the main text.
The resulting coefficients assign higher weight to models with lower epistemic un- certainty for the given query, while entropy regularization ensures smooth, query-adaptive aggregation. This yields principled uncertainty-awe aaware embedding convolution that is robust to heterogeneous model reliability and distributional shift.
Interpretation
A.3 DERIVATION OF UNCERTAINTY-WWAR SIMILARITY ESTIMATION
In this section, we provide a detailed derivation of the uncertainty- aware similarity estimation method introduced in Section 3.3. When embeddings modeled multivariate Gaussian distributions, the similarity between them becomes a random variable. We aim to derive the distribution of this similarity score and propose calibrated approximation that accounts for predictive uncertainty.
Let q = z(x) ~ (pq. Eg) and с = z(x') ~ N(pe: Ee) denote two ensembled probabilistic embeddings generated by the proposed UEC method. Each embedding represented as a Gaussian distribution with a mean vector Rd and a covariance matrix Σ E Rdxd
Problem Setup
We define the cosine similarity between q and as:
q C
5 = cos(q, c)
||||| ||c||*
which itself is a random variable, since both q and are random. In our implementation, both q and are fo-normalized in expectation (i.e., ||Ma|| < ||Me|l 1), and their variances are relatively small. Therefore, we simplify cosine similarity to the inner product:
~ q
as is common in prior work (Mallik & Sagias, 20 Randone et al., 2024).
Distribution of the Inner Product via Moment Matching To characterize the distribution of q c, we approximate it as a Gaussian random variable using moment-matching (Randone et al., 2024). Since q and are independent Gaussian vectors, their inner product follows an approximately Gaussian distribution whose mean and variance can be computed in closed form.
Mean By linearity of expectation:
Ns = E(q Elq] B[c] = Mg Pe-
(12)
Variance Using the law of total variance:
Var(q c) = Eg [Var(q q)] + Vara (B(qTe ql)
Eg 'q Eegl Varg (q pe),
20

---

Published as a conference paper at ICLR 2026
where assume q and independent.
We evaluate each term as follows. For the first term, using the identity E/x Ax] = tr(AE) to² TAA for x ~ (H, Σ) (Shao, 2008):
Bq E.g] = tr( tr( 14 Salq.
For the second term, using the variance of a linear form in Gaussian:
Varq(q Me)) Me Sq
Combining both, we obtain:
of Var(q M4 Selaa 140 Dqlte tr(Eq
Therefore, under moment-matching, we approximate the distribution of similarity as:
$ ==q (
Probit-based Calibration of Similarity Scores quantify uncertainty, but still need scalar value to rank candidates. To account for this uncertainty in principled yet computationally efficient way, we apply a probit-based calibration to the mean similarity score:
The similarity distribution provides way to
Us
RC
(13)
This formula arises from approximating the expectation of a sigmoid over a Gaussian random variable (Gibbs, 1998; Eschenhagen et al., 2021), i.e., [[((z)) ss a (n/ /1 go2) when (N,, and (-) is the logistic sigmoid function.
Interpretation and Robustness The numerator μls captures alignment between the means, while the denominator serves as a soft penalty for high uncertainty. As o² increases, the similarity score å downscaled, reflecting reduced confidence in the similarity estimate. This makes the scoring robust in the presence of model disagreement, variance, noise, particularly useful in retrieval settings.
This uncertainty-aware formulation preserves the computational simplicity cosine similarity while incorporating predictive variance in a principled manner, enabling better performance under high uncertainty.
A.4 DERIVATION OF THEOREM 1
We prove that the proposed estimator provides bounded approximation the squared 2- Wasserstein distance W3 under diagonal-covariance assumptions, and we extend this bound to the Jeffreys divergence J under additional regularity assumptions.
Recall that
q z(x) Niqq Eq); with nomomalliee means |وg | 1, and diagonal covariances
c= z(x') NMee Ec),
Σg diag(
(2 q......... ... ... aga).
Ee diag e........ ... o2.).
We define the per-coordinate maximum variance
:: max(o 1.
A.4.1 PRELIMINARY
In Section 3.3 the dot product & = q7 c was approximated by a Gaussian with mean and variance
Us 14 Me: o² = p4 Eept + 1e EqUe + tr (E
The probit-calibrated similarity estimator is then approximated as
TT He
~3
V1 3o?
Using || ||Me|| 1, the mean term admits the exact identity
Mg Me
1
|Mq 1ell2
(14)
21

---

Published as a conference paper at ICLR 2026
A.4.2 APPROXIMATION OF UNCERTAINTY-AWARE SIMILARITY ESTIMATION
1 + (2²) for small æ, we linearize the Applying the Taylor expansion (1 + x) denominator:
000 + O
(15)
V 8
Substituting Eq. 14 and Eq. 15 yields
$
b Mg Mell
16 o + 0 o + (
Me pe 2
Rearranging and discarding the additive constant which does not affect ranking gives the expansion
1 al Mell 16 + O
(16)
A.4.3 COMPARISON WITH SQUARED WASSERSTEIN DISTANCE
We now compare the expansion Eq. 16 with the closed form of the squared 2- 2-Wasserstein distance for diagonal covariances:
W = 11qq Mell +
> (aq. oe,i)² 1=1
(17)
It is useful to align affine factors so that the leading mean term cancels: using Eq. 16 we have
2(1 - || Hell + o + 20(a?))
(18)
Define the residual (difference) between W? and the affine-aligned estimator:
A W3 2(1 - 8).
(19)
Substituting Eq. and the aligned form gives
(114g) Mell3 +0 + +20
A ||gg Mell 2 Σ (oqs 0ct)))
Tagi -o,))
0o,))
00² -0
(20)
Thus the mean separation term is exactly cancelled by aligning 2(1- 5) with W22: the residual A depends only on variance-related terms and the higher-order remainder (o The task is to bound A.
Elementwise Expansion
Write each term elementwise:
Σ aq.. T2,)2 Σ
( + 000 -20
o,2 =
( ( + oquter + 22))
Substituting into Eq. 20 yields
d
A
(oz + o², - 20q.)oe.)) - 7 (020/// + 02 22 + 02 222022)) - 20(o))) (21)
i=1
22

---

Published as a conference paper at ICLR 2026
General Bound Without Extra Scaling Assumptions and L). S e, we obtain the estimate
Using only the trivial bounds 0 1 mi), S
A S
(og + oe, + 30 og + t + +(0(0)))
i=1
S Co tr ( + tr(2e) + (Zq (
for some numerical constant C. (e.g. depending on πτ. Since tr [,,,,) ) O(²²) and tr (x) Oc)). the coarse bound yields = ((s) in general. This already justifies the statement that 2 - 8) approximates W up to variance-controlleed error; however the bound is not yet the tight ((e²) form.
Tighter Bound Under Smallevariance Scaling To obtain the strong ( residual we introduce additional mild scaling assumptions which are natural in the embedding-ensemblle small-uncertainty regime:
Assumption (Small-variance trace scaling).
tr(Eq) O(e),
(r(2)) 0
Assumption 2 (Per-dimension dispersion). The per-coordinate variances are dispersed across dimensions, iee.
max oa S d'
Both assumptions are mild in practice when the posterior variances of embedding coordinates small and not concentrated few coordinates (a common situation for high-dimensional embeddings).
Elementwise Cancellation Argument Under Assumptions 1-2 we refine the termwise bounds in Eq. 21. Since i I 1 and Assumption yields o3ys I e/d, each element in the bracket of Eq. 21
is
E²
(o2, + de. 20. [ + 22 + =2.0 -
d
Summing over i = 1,..., d yields
Oss2) d2
Moreover, the remainder term ( is O(s²) under assumptions. Hence A = O(E?), This proves the quantitative bound claimed in Theorem 1:
- W +
A.4.4 COMPARISON WITH JEFFREYS DIVERGENCE
We now show how a similar first-order argument extends to the Jeffreys divergence
J (N Ng),N(He Ea)) = DKI. ( E.)) + DRI. (N Ee)l Eq)),
which for diagonal covariances admits the closed form
[ Ma)) (EE=" (He Ma) +
- 2).
(22)
Unlike the Wasserstein case, the Jeffreys mean-term involves inverse covariances. To linearize these terms we need regularity assumption that the diagonal variances and c are not only small but also close common positive baseline. Specifically, make the following:
23

---

Published as a conference paper at ICLR 2026
Assumption (Common-scalee regularity). There exists a baseline > 0 and constants K, k² such that for all
10 --| S Ke,
l0 - Kε,
and T is bounded away from zero (i.e. 2 Tmin
0). Moreover, tr(Eq), tr() O in
Assumption
Assumption 3 asserts that each diagonal entry is a perturbation of a common (nonzero) scale T; the smallness parameter is the same used above. Under this assumption we can Taylor-expand reciprocal diagonal entries.
Expansion of Inverse Covariances
For each i, write
2 = + 8q,i-
Le. = T 8e,i
with [6(3)al I Ke.
Then, using the scalar expansion
(T + 8)-1 -- -- 28 0(8), we obtain
Σ = I 5 diag 1, 8g,d) + 0(E)) D ---1 diag (0, (fc.1...... Sc,) (E
(23)
Hence
Σ- Σg =2 -r- -- 2iag 5e,1,....,, , a
dc,a) O
Mean Term J Substitute Eq. into the mean quadratic term of Eq.
(Pee (4) T + (pe Ma) 1/2 (Me M4m] TT) (the 14)
- (Mo - ma) r - diag(6a. d0,) Ma) O(s2)
-- -.-1.1luu bllaa Mell + O(E/d) + (ss²).
(24)
Variance-only Term in J Consider the variance-only summation in Eq. 22:
o². -2) 9.
Vy =
Write each ratio using 0². = r + dq.io etc., and expand:
2²
+ Eq.i + Se,i
=1 óq de.i (dq.s de,i) 0(s²).
Similarly for the reversed ratio. Summing and simplifying yields
o2.
- (6q. - de,1)
72 + 0
a.
Hence
V, 2r
Σ (6q. +
Note that each 6(),5 = 0(e), so V, = 0s=2).
Putting Mean And Variance Parts Together
Combining Eq. 24 and the expression for V, we
J J= 1 11llq Mell + O(e) O(s2).
(25)
The dominant term is proportional to b11pq Mell with coefficient 1/r; the variance-only contribu- tion is Oss2).
24

---

Published as a conference paper at ICLR 2026
Relation Between J And 2(1 Recall from Eq. 18 that
2(1 8) || Hell + o + O(o.).
Comparing this with Eq. 25, we see that the mean-dependent parts of and 2(1 - 8) agree up to a constant scaling factor 1/T. Therefore, defining the scalar scaling factor
we have
a a= = Pa Mell + +0(E)
Consequently
a- 2(1 [5)] (2 ue 2 + 0(E)) (IHa- Mell 22 0(62))
2 M4 - Mel 2 (e)) + 0ss2)
(26)
Multiplying both sides by 11 and rearranging gives relation of the form
2(1 - a a 0()) O(*)).
Under Assumption 1-2, the (e)) terms refine to O(e²) as in the Wasserstein case (the same element- wise cancellation logic applies to the lincar-in-ò components), yielding:
Theorem (Affine approximation Jeffreys divergence). Under Assumptions 1-3, there exists scalar a >> and constant C > independent of d such that for sufficiently small E,
a c) 2(1 5) | 5..
Equivalently,
s= == -01.00-v(9.))+00
A.4.5 RANKING EQUIVALENCE
Combining Theorem (Wasserstein affine bound) and Theorem (Jeffreys affine bound), conclude that under the stated small-variance and common-scale assumptions the estimator å provides an affine, ) surrogate for both W? and (up constant scaling) the Jeffreys divergence; therefore, induces the same ranking as these distributional distances up to O(e²) errors.
However, our bound relies on first-order Taylor expansions and variance dispersion assumptions, yielding ((a²) residuals. If these assumptions are violated (e.g.. with highly concentrated variances or heterogeneous scales). the guarantee can degrade to O Developing refined analyses that relax these conditions and provide tighter bounds remains an important direction for future work.
B EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
B.1 PRELIMINARY ANALYSIS OF EMBEDDING SUBSPACE ALIGNMENT
Before ensemble the embeddings, potential concern is whether embeddings from different models indeed lie in sufficiently aligned subspace. We argue that in our setting this assumption is reasonable and empirically supported.
First, the embedding models considered in our experiments share (i) the same underlying architec- ture and (ii) largely overlapping training data. As established prior work (Hewitt & Manning, 2019), such conditions encourage independently trained models to discover consistent linguistic atructures-suc such as syntax, semantic relations, and contextual dependencies and to encode them in geometrically comparable ways in their embedding spaces.
Second, our ensembling procedure applics f2 normalization to embeddings before aggregation. This post-processing step has the effect of further mitigating potential subspace differences, effectively aligning embeddings to a common hypersphere and making their scales directly comparable (Timkey & Van Schijndel, 2021).
Finally, before ensembling, we empirically assessed the degree of subspace overlap by computing Canonical Correlation Analysis (CCA) (Hotelling, 1992) between models. We considered model
25

---

Published as a conference paper at ICLR 2026
groups: (i) E5, BGE-zh, and MATRYOSHKA (Section 4.1), and (ii) BGE, E5, and GTE (Section 4.2). Since CCA quantifies the similarity and alignment between two sets of representations, we computed pairwise scores within each group of three models. For the first group, CCA measured using the same dataset as Section 4.1. For the second group, it was measured on Wikipedia dataset, which used in the Retrieval task. As shown in Table 8, the results indicate that embeddings exhibit substantial overlap, supporting the view that they inhabit approximately aligned subspaces.
Table 8: Analysis of embedding subspace overlap using CCA (%). Results are reported for pairs in two different model groups.
Model Group
CCA
Model Group
CCA 1
(E5, BGE-zh)
99.9 %
(BGE, E5)
99.9
(E5, MATRYOSHKE
99.8 %
(BGE, GTE)
99.4 %
(BGE-zh, MATRYOSHKA)
98.3 %
(E5, GTE)
99.5
B.2 COMPARISON WITH ALTERNATIVE SIMILARITY METRICS
In this section, we compare proposed uncertainty-aware similarity (UEC) against widely used probabilistic similarity metrics. We restrict our comparison to similarity functions that admit closed-form solutions between Gaussian distributions, thereby avoiding the computational over- head of Monte Carlo sampling. Specifically, we consider the KLD (Kullback Leibler, 1951), I-Wasserstein (Chhachhi & Teng, 2023), 2-Wasserstein (Gelbrich, 1990), and Closed-form sampled distance (CSD) (Chun, 2023).
KLD Given multivariate Gaussian distributions M1 N(1,, En) and N2 NN Ez), the closed-form expression for KLD is:
det E₂
DRL. (Ni ([22
Er) (122 p1) (12 m1) d log
(27)
det E
where d is the dimensionality of the distribution. While informative, this metric involves matrix inversion and log-determinants, making it computationally expensive for large-scale retrieval.
1- Wasserstein Distance The 1-Wasserstein distance between two Gaussians N₁ and Ng is given by:
(1 ~ | - 122l2 tr(v/tr)))))) E1) tr( tetttt((/))
(28)
Here, the distance accounts for both mean shift and scale (via trace square-rood covariance). Although it is more interpretable, it requires square-root matrix operations which are non-trivial,
2- Wasserstein Distance The Wasserstein distance between Ni and N2 is defined as:
(2) = |41 pell2 + tr Σ + Eg 2
/2 Σ ///21//22)
(29)
which also integrates both mean and covariance alignment. However, its computation is notably expensive due to the presence of matrix square roots and requires full-rank covariance matrices.
CSD The Closed-form Sampled Distance (CSD), proposed in PCME+ (Chun, 2023). key advantage of CSD is its tractable closed-form given by:
A2) || M2 1 + Σ Σz/11
(30)
where || - || denotes the element- wise é -norm. CSD captures both distributional shift and total uncertainty spread in simple additive form, and is more computationally efficient than metrics involving matrix square roots or inverses.
We evaluate the similarity metrics under the MIRACL subset retrieval task described in Section 4.1. As shown in Table 9, UEC achieves the best overall retrieval performance while exhibiting the lower runtime.
26

---

Published as a conference paper at ICLR 2026
We note that the asymptotic complexities of KLD and Wasserstein distances be reduced to D) when restricted to diagonal covariance matrices. However, despite having the same big- O complexity, these metrics remain consistently slower than UEC in practice. This gap arises from fundamental differences in their computational structure. Specifically, KLD and Wasserstein still require per-dimension normalization, division, logarithmic, or square-root operations, well as multiple intermediate tensor constructions. In contrast, UEC consists solely of element-wist additions and multiplications followed by single aggregation, with no normalization constants, matrix operations, or transcendental functions.
As result, UEC has substantially smaller constant factor and lower memory access overhead, enabling faster execution and better hardware utilization, particularly in large-scale retrieval settings. Moreover, UEC operates entirely in closed form without sampling or iterative solvers, making well-suited for deployment in latency-sensitive systems.
Table 9: Comparison of similarity metries under the MIRACL evaluation (Section 4.1). "Rel." and "Comp." denote relative runtime (normalized to UBC=1.00) and asymptotic complexity per similarity computation. Although several metrics share the same asymptotic complexity under diagonal covariance assumptions, their empirical runtimes differ due to operation-leve costs and implementation overheads, which are not captured by big-O notation.
Memory Complexity
Time
Similarity
nDCG@
Recall@ 10
AUC@ 10
Rel.
Complex.
KLD
58.14 18.14
78.33
87.60
O(KD)
1.52
O(KD)
I-Wasserstcin
57.87
88.21
79.92
O(KD)
1.21
O(KD)
2- Wasserstein
57.71
88.48
81.05
O(KD)
2.37
O(KD)
CSD
58.23
79.10
86.92
O(KD)
0.88
O(KD)
UEC (Ours)
59.65
80.07
91.04
O(KD)
1.00
O(KD)
B.3
EXPERIMENTAL DETAILS: MIRACL SUBSET
B.3.1 FITTING LAPLACE APPROXIMATION
To quantify uncertainty for each embedding model, we apply a diagonal Laplace approximation (LA) to the final layer using small, labeled dataset. In this experiment, we fit the approximation using the MIRACL Hard Negative (Zhang et al., 2023) Subset. For E5, we use 50 labeled examples each from English and Russian: for BGE-zh, use 50 examples each from English and Chinese; and for MATRYOSHKA, we use 50 examples each from English and Arabic.
Each query-passage pair is assigned a binary label indicating whether the passage is relevant (1) or not (0) to the query. These binary-labele pairs are then used to train a logistic classifier with a cross-entropy objective.
To ensure computational scalability, we adopt diagonal Gaussian approximation. We fit the LA using the laplace-reel redux library². The posterior mean corresponds the MAP estimate of the final layer's weights, and the posterior variance is derived from the inverse of the diagonal Hessian. This provides lightweight probabilistic embedding representation without requiring backpropagation or full model fine-tuning.
B.3.2 DATASET
For the MIRACL subset evaluation, we constructed a test set by sampling 100 queries each from four languages (English, Russian, Chinese, and Arabie) from the multilingual MIRACL Hard Negative dataset (Zhang et al., 2023). This selection aimed to cover diverse language families and scripts, ensuring a balanced multilingual evaluation.
hhttps:/aaleximerr wtt.m/m.w.....mmmmmmam// wow/
27

---

Published as a conference paper at ICLR 2026
B.3.3 METRICS
We employed three commonly used retrieval metrics: nDCG@ 10, Recall 10, and AUC@ 10.
nDCG@ @ 10 (Normalized Discounted Cumulative Gain) measures the ranking quality based on the positions of relevant passages among the top 10 retrieved items. assigns higher weights to relevant items appearing earlier in the list, penalizing misranked relevant results logarithmically.
Recall @ 10 evaluates whether at least one relevant passage is retrieved within the top 10. is computed as the fraction of queries for which a relevant document appears among the top- 10 predictions.
AUC@ 10 (normalized Area Under the Curve) measures how well model's confidence scores align with its actual retrieval performance when abstention is allowed. Specifically, the model allowed abstain on queries with low confidence scores, and AUC 10 tracks how the average performance improves more uncertain queries are excluded. We compute AUC@ 10 by plotting nDCG@@ 10 function of the abstention rate-i.e., the proportion of oow-cofidence queries discarded and measuring the area under this curve. To enable fair comparison across models, we normalize this area between two bounds: (1) a baseline curve where abstention does not improve performance, and (2) oracle curve where the worst-per porst-performing queries are perfectly abstained first. The normalized AUC then computed as:
nAUC AUCActual AUCRassline
AUCoradte AUC
Intuitively, higher nAUC values indicate that the model's confidence scores are well- calibrated: as the model abstains on low-confidence queries, its average performance improves meaningfully. Conversely, negative nAUC implies that the model's s confidence scores are misleading i.e.. low-confidence queries were actually high-performing, or high-confidence queries were often wrong.
B.3.4 HYPERPARAMETERS
We search hyperparameters for weight ensemble and UEC (Ours). In UEC, we introduce a temperature parameter sharpen the ensemble coefficients, and additionally incorporate hyperparameter / to scale the influence of the variance of in the uncertainty-awe aware similarity estimation.
Weight Ensemble: We performed grid search over weight combinations such as 0,1,0.8),, 0.. [0.8, 0.1), ensuring the weights sum up across the three models.
UEC (Ours): We fixed the temperature T as 1.5 for all experiments. We search 3 in range of [0.0001, 0.001, 0.01, 0.1].
B.3.5 AUXILIARY RESULTS
Table 10 shows detailed results for individual models and combination methods. UEC achieves the highest average performance across all metrics, with particularly strong gains in AUC.
B.4
EXPERIMENTAL DETAILS: MMTEB
B.4.1 FITTING LAPLACE APPROXIMATION
To estimate uncertainty each embedding model, we fit a diagonal Laplace Approximation (LAA) to the last layer of each model using small subset of labeled data. We choose datasets that are commonly used in the pre-training or fine-tuning of the considered embedding models to ensure consistency between model behavior and fitting data.
Specifically, we use the MS MARCO (Nguyen et al., 2016) and SNLI (Bowman et al., 2015) datasets for fitting. From MS MARCO (consisting of roughly 80K samples), randomly select 3,983 pairs. From SNLI (550K examples), we use 3,775 sentence pairs. Each triplet consists of query (or premise), positive passage (or hypothesis), and randomly sampled negative passage.
28

---

Published as a conference paper at ICLR 2026
Table 10: Performance on MIRACL Subset across models and combination methods. The best and second-best results per row are bolded and underlined. UEC shows the highest performance on average across all metrics.
Individual Models
Ensemble
Language/ / Metric
UEC
E5
BGE-zh
MATRYOSHKA
Uniform
Weighted
English
nDCG@ 10
80.27
67.05
40.02
69.82
76.30
78.29
Recall @ 10
97.19
90.50
54.83
92.66
97.80
97.98
AUC@
37.37
-28.31
-86.74
-24.38
6.71
86.72
Russian
nDCG@ 10
39.12
12.25
5,96
11.65
21.78
26.61
Recall @
48.98
15.98
7.21
15.93
30.77
41.38
AUC 10
60.18
38.22
-46.00
17.14
39.65
91.49
Chinese
nDCG@1 10
17.74
75.16
0.73
39.17
35.26
60.68
Recall @ 10
28.65
97.45
2.00
57.02
54.60
91.04
AUC@
60.99
-53.35
27.89
13.65
38.01
92.14
Arabic
nDCG@
13.36
0.64
71.47
66.47
61.97
73.18
Recall @ 10
17.87
1.33
92.55
85.01
79.43
89.88
AUC@ 10
67.45
42.06
85.74
9.62
39.12
93.82
Avg. nDCG@ 10
37.62
38.77
29.55
46.78
48,83
59.65
Avg. Recall@ 10
48.17
51.32
39.15
62.66
65.65
80.07
Avg. AUC@10
56.50
-0.34
-4.78
4.01
30.87
91.04
We assign binary labels indicating whether a passage is relevant (1) or not (0) with respect to the query. These binary-labeled pairs used to fit logistic classifier using cross-entropy loss.
We adopt the diagonal Gaussian approximation to ensure computational efficiency, and perform LA fitting using the laplace-res redex library. The mean corresponds the MAP estimate of the final layer weights, and the (inverse) diagonal Hessian is used to compute the posterior variance. This yields probabilistic representation of each embedding without requiring re-training backpropagation through the base model.
B.4.2 DATASET
For broader multilingual evaluation, we conducted experiments on MMTEB (Muennighoff et al., 2022; Enevoldsen et al., 2025), leveraging the official benchmark codebase available at https : / github. com/ of tasks from the MMTEB benchmark, including five tasks from Classification, five from Retrieval, and ten from STS. The datasets for each task were chosen to span multiple domains within the same task type to assess domain-robust performance.
// eb. We mainly selected representative subset
B.4.3 METRICS
In MMTEB evaluations, we used the main score recommended by the benchmark as the primary evaluation metric for each task. Additionally, we reported commonly adopted metrics for each task type. For Retrieval, we reported nDCG @ 10, Recall @ 100, and AUC@ 10. For classification, we reported Accuracy, Fl score, and AUROC. The Fl score is the harmonic mean precision and Recall, providing balanced measure of classification performance, especially under class imbalance. AUROC, short for the area under the receiver operating characteristic curve, measures a model's ability to distinguish between classes across varying decision thresholds. It is particularly suited for evaluating models under uncertainty, as it considers both sensitivity and specificity to assess confidence calibration. For STS tasks, used Spearman correlation, the main metric in MMTEB,
29

---

Published as a conference paper at ICLR 2026
which measures the rank correlation between predicted similarity scores and auman-annotated ground truth.
B.4.4
HYPERPARAMETER
Weight Merging: We used grid search over weight combinations including [0.1, 0.1, 0.8]. 00.1, [0.8, 0.1, 0.1), ensuring the weights sum 1.
Task Arithmetic: We implemented task arithmetic following Ihharco et al. (2022), where the merged embedding is computed by linearly combining base model embeddings with the direction vector between two other models. Formally, given three models A, B, C, the merged embedding is:
meged Ambed_ + a (Embedg - Embedc),
where a is hyperparameter controlling the direction scaling. We performed grid search over {0.0001, 0.001, 0.01, 1.0, 1.03.
Weight Ensemble: We used grid search over weight combinations including (0.1, 0.8). 00,1, .... [0.8, 0.1, 0.1]. ensuring the weights sum 1.
UEC (Ours): The temperature T fixed at 1.8 throughout the experiments. We search B, scaling parameter for o², in range of [0.0001, 0.001, 0.01, 0.1].
B.4.5 ADDITIONAL TASKS
In addition to the three representative tasks (Retrieval, Classification, STS), we also conduct exper- iments on other MMTEB tasks, including Bitext Mining, Clustering, and Reranking. For each of these tasks, we randomly select subset of datasets and observe that the proposed UEC consistently achieves strong performance.
Bitext Mining Table 11 summarizes performance multilingual bitext mining across six represen- tative datasets from MMTEB: Bornholm (Derczynski Kjeldsen, 2019), Flores (Goyal et al., 2022), IN22Gen (Gala et al., 2023), IndicGenBench (Singh et al., 2024), NTREXB (Federmann et al., 2022), NorwegianCourts (Tiedemann & Thottingal, 2020). For evaluation, we report F1 scores following MMTEB's standard setup.
Table 11: Comparison of Bitext Mining performance. The best and second-best results per dataset are bolded and underlined. UEC achieves the highest average performance overall.
Individual Models
Model Merging
Ensemble
UEC
Dataset
BGE
E5
CTE
GTE-MB
Uniform
Weighted
TaskArith
Uniform
Weighted
BornholmmBiietMMiii
27.60
38:49
32.16
27.64
33.72
37.21
37.03
34,50
36.99
RloeesBitexcMiiii
13.03
19.42
13.66
17.48
15.93
18.21
19.38
15.89
18.38
19.55
4.54
8.85
3.52
5.88
5.94
8.13
8.26
6.11
8.52
8.60
NTREX BitectMining
18.78
27.68
19:77
24.54
23.08
26.34
25.10
22.51
26.01
27.72
90.67
92.91
92.03
91.45
91.28
92.04
91.87
91.52
92.62
92.96
Avg.
Our proposed UEC method achieves the best average performance, outperforming both individual models and all ensemble or merging strategies. Notably, UEC obtains the top result on four out of six datasets, demonstrating its ability robustly fuse knowledge from diverse pretrained models under high uncertainty.
Clustering We measure the performance on multilingual clustering across eight represen- tative datasets from MMTEB: ArXivHierarchical, BigPatent (Sharma al., 2019), Biorxiv, MasakhaNEWS (Adelani al., 2023b), SIB200 (Adelani et al., 2023a), StackExchange (Geigle et al., 2021), WikiClustering.
The V-Measure an external clustering evaluation criterion defined as the harmonic mean of homogeneity and completeness (Rosenberg & Hirschberg, 2007). Homogeneity measures whether each cluster contains only members of a single class, while completeness measures whether all members of given class assigned the same cluster. V-Measure ranges from 0 (no alignment)
30

---

Published as a conference paper at ICLR 2026
Table 12:
Comparison of Clustering performance. The best and second-best results per dataset are bolded and underlined. UEC achieves the hightest average performance overall.
Individual Models
Model Merging
Ensemble
UEC
Dataset
BGE
ES5
GTE
GTE-MB
Uniform
Weighted
TaskArith
Uniform
Weighted
59.75
58.05
59.85
59.50
59.38
60.14
59.56
50.29
58.42
61.43
58.05
54.91
57.67
58.16
58.95
58.63
57.08
38. 38.87
58.33
59.32
BigPaacatCuusiernnge
30.69
31.95
Biorsis Clustering22P.22
41.65
38.01
40.58
41.91
40.98
40.86
40.56
41.37
40.68
43.06
MasaktaNEWSC
1usteringS28
40.47
41.11
39.41
37.32
41.08
42,36
41.37
40.21
42.63
4.1.07
S1B200CTusternnS52
Stack Exchtnne
59.08
55.99
59.26
61.3 31
60.18
61.78
61.12
60.54
61.44 44
62.06
WikiClusteringP22P22
24.93
25.02
24.87
25.47
25.16
25.74
25.30
25.55
25.61
26.23
Avg. V-Meusrre
40.93
to 100 (perfect clustering), and is widely used in multilingual and scientific document clustering benchmarks such as MMTEB.
Table 12 summarizes performance on clustering tasks. UEC achieves the best average performance. outperforming all individual models and ensemble baselines. Notably, it ranks first on 6 out of 8 datasets. In short, UEC consistently improves by leveraging eeeerainty-calibbatne similarity and adaptive ensemble weighting, enabling more accurate cluster assignment under distributional shift and model disagreement.
Reranking Table 13 presents the results of reranking experiments five benchmark datasets: Alloprof (Lefebvre-Brossard et al., 2023), T2 (Xie et al., 2023), VoyageMMarco (Clavié, 2023), WebLIIXCCandidttes (Lu al., 2024), WikipediaReranking We evaluate performance using Mean Average Precision (MAP), standard retrieval metric that reflects both the precision and the ranking quality of relevant items. MAP is computed as the mean of the average precision scores over all queries, where average precision measures how well relevant documents are ranked near the top. Higher MAP values indicate that relevant passages are not only retrieved but also correctly prioritized.
Table 13: Comparison of Reranking performance. The best and second-bes results per dataset are bolded and underlined. UEC achieves the hightest average performance overall.
Individual Mudels
Model Merging
Ensemble
UEC
Dataset
BGE
E5
GTE
GTE-MB
Uniffrm
Weighted
TaskArith
Uniform
Weighted
Allopro/Rerunking
62.30
64.82
66.54
68.72
65.65
66.04
65.86
65.65
67.65
69.14
T2Reranking
60.93
60.55
61.20
65.24
61.83
62.98
62.77
61.39
63.34
65.08
NoyageM Mircoenninnnn
30.98
33.03
31.37
37.41
32.08
33.13
31,93
31.76
33.47
36.89
Webl. LNXCandddnness
13.72
10.84
13,51
18.02
13,88
14.73
14.32
13:73
15.49
17.65
WikipdiikkrakkngMMlliinnaaaa
75.21
76.74
Avg. MAP
48.11
49.21
49.28
$2.78
49.85
30.64
50.07
49.55
51.54
53.33
The proposed UEC method achieves the highest overall performance with an average MAP, outper- forming all other methods. It ranks first three datasets and second the remaining two. These results demonstrate UEC's effectiveness in capturing uncertainty and enhancing similarity estimation for high-precision reranking.
B.5 MULTILINGUAL SETS
Retrieval In multilingual retrieval, UBC consistently outperforms all baselines. Across seven languages, UEC attains the best average nDCG@ 10, Recall@ 100, and AUC@ 10, while never degrading performance any metric compared to the strongest individual model. Notably, UEC yields substantial gains on low-resource languages such as Bengali and Hindi, where single models exhibit complementary strengths: BGE and E5 provide strong geurrol-purpose multilingual retrieval, whereas GTE-MB is tailored to multilingual benchmarks. By reweighting models according to their epistemic uncertainty. UEC effectively selects the most reliable expert per query, improving both top-ranked retrieval quality and the calibration of uncertainty curves (AUC@ 10).
UEC yields the best average Accuracy, F1, and AUROC across all three languages. While GTE-MB is competitive strong multilingual
Classification On SwISsJ
31

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of retrieval performance the WIAPEDDIRRTREEVAAMCTENN benchmark. The best and second-best results per row are bolded and underlined. UEC achieves the highest average performance across all languages and metrics, with particularly strong gains on AUC @ 10, indicating more reliable uncertainty-aware ranking. (Uni.=Uniform, Wt.-Weighted, TA=Task arithmetic...
Individual Models
Model Merging
Ensemble
Language
Metric
BGE
E5
GTE
GTE-MB
Uni.
Wt.
TA
Uni.
Wt.
UEC
Bulgarian
nDCG@ @ 10
29.11
45.61
33.97
55.29
35.62
39.12
38.87
32.51
38.88
48.03
Recall @ 100
52.86
71.60
58.13
81.53
63.40
68.39
68.24
56.33
67.63
74.12
AUC@ 10
57.36
62.63
55.06
62.12
58.29
60.16
59.93
56.32
60.77
65.28
Bengali
nDCG@
8.11
5.54
3.16
6.52
7.74
6.36
7.91
9.86
Recall
23.93
17.40
10.46
19.75
21.97
21.03
19.66
23.74
25.73
AUC 10
39.71
19.28
29.58
47.36
26.83
24.97
36.22
31.65
30.78
45.97
Danish
nDCG@ 10
63.45
75.18
71.16
69.91
67.83
73.42
72.11
68.94
75.14
76.86
Recall @ 100
87.26
94.33
92.00
91.33
91.24
92.47
92.33
91.49
93.81
95.03
AU 10
64.22
73.07
65.35
68.92
66.67
69.39
69.72
68.04
73.01
76.28
Persian
nDCG@
9.19
14.44
8.97
35.69
11.36
13.28
12.97
12.06
14.23
16.92
@ecall 100
26.80
33.40
27.73
61.33
28.83
30.05
30.16
29.97
33.32
37.73
AUC@ 10
33.55
39.01
28.95
59.47
34.26
36.83
35.55
35.92
39.23
45.19
Hindi
nDCG@ 10
11.99
21.55
5.23
17.16
15.37
20.05
17.83
16.32
21.11
23.56
Recall @ 100
29.33
45.93
14.87
31.33
31.62
42.18
40.17
39.20
45.02
47.22
AUC 10
37.84
49.91
46.51
53.29
46.97
48.64
48.03
50.26
49.99
52.38
Portuguese
nDCG@ 10
72.29
81,12
75.40
75.94
75.32
79.17
78.93
76.66
80.64
83.13
Recall 100
92.73
96.13
95.13
95.66
94.83
95.27
95.21
95.17
96.08
96.94
AUC@ 10
54.28
71.07
68.13
70.52
64.82
70.34
68.73
71.26
72.35
77.17
Serbian
mDCG@
41.52
33.99
44.15
35.52
41.19
41.82
37.12
41.89
44,38
Recall 100
54.20
67.73
58.40
74. 06
64.93
67.71
69.74
69.18
72.09
74.81
AUC@ 10
21.25
63.42
61.25
54.97
62.65
66.16
69.21
Avg. nDCG@ 10 f
32.18
41.07
33.47
43.04
35.36
39.15
38.61
35.71
39.97
43.24
Avg. Recall 100
52.03
61. .68
51,95
63.52
56.37
59.72
59.55
57.28
61.67
64.51
Avg. AUC
44.03
54.05
51.61
60.41
50.40
53.34
54.61
53.73
56.05
61.64
encoder, it is dominated by UEC once epistemic uncertainty is taken into account the ensemble level. Crucially, UEC does not trade off performance between language groups: it matches or improves upon the strongest baseline in German and French, and achieves the largest gains on Italian, the smallest and most imbalanced language split. This suggests that UEC remains robust under multilingual label imbalance and does not sacrifice minority-language performance when weighteig uncertain embeddings.
STS On SEMREL24STS, UEC further improves over strong multilingual encoders on sentence- level similarity, While GTE-MB and E5 already achieve competitive performance on high-resource languages such as English and Afrikaans, UEC attains the best average score and ranks first on 9 out of 12 languages, including several low-resource African and Indic languages (e.g.. Hausa, Hindi, Marathi). This indicates that uncertainty- driven ensembling not only preserves the strengths of specialist models but also yields more stable semantic similarity estimates challenging multilingual and low-resource regimes.
32

---

Published as a conference paper at ICLR 2026
Table 15: Comparison of multilingual legal judgment classification performance on SWISSJUDGE- The best and second-best results per row are bolded and underlined. UEC achieves the highest average scores over German, French, and Italian, demonstrating robust perfor- mance across languages and label distributions. (Uni.=Uniform, Wt.=Weighted, TA=Task arithmetic.)
Individual Models
Model Merging
Ensemble
Language/ Metric
BGE
E5
GTE
GTE-MB
Uni.
Wt.
TA
Uni
Wt.
UEC
German
Accuracy
50.28
47.85
47.73
53.08
48.26
49.73
48,08
49.14
51.75
53.13
43.86
41.96
43.01
46.10
42.67
44.61
44.43
43.24
44.61
45.97
AUROC
50.32
50.17
50.86
51.12
50.17
50.21
50.83
50.72
50.26
51.83
French
Accuracy
57.16
58.58
56.37
55.45
56.82
55.90
55.44
57.09
57.71
58.36
F1
46.63
47.25
46.34
46,41
46.55
45.87
45.73
46.61
47.02
48.52
AUROC
50.48
50.22
50.65
50.67
50.13
50.24
50.37
50.53
50.51
51.02
Italian
Accuracy
53.47
57.61
53.00
58.36
54.76
55.89
54.83
54.92
57.65
58.72
Fl
45.64
46.09
45.72
47.79
45.88
46.03
45.31
45.62
46.72
48.03
AUROC
50.73
50.39
50.42
51.36
50.89
51.25
51.08
50.56
50.84
51.89
Avg. Accuracy
53.64
54.68
52.37
55.63
53.28
53.84
52.79
53.72
55.65
56.74
Avg. Fl
47.38
45.10
45.02
46.77
45.03
45.50
46.
45.16
46.
47.51
Avg. AUROC1 1
50.51
50.26
50.64
51.05
50.40
50.57
50.76
50.60
50.54
51.58
Table 16: Comparison of multilingual STS performance on SEMREL24STS. The best and second- best results per column are bolded and underlined. UEC attains the highest average Spearman correlation, ranking first on 9 out of 12 languages and consistently outperforming both individual encoders and heuristic ensembles. We evaluate on 12 languages: Afrikaans (Afr.), Amharic (Amh.), Modern Standard Arabic (Arb.), Algerian Arabic (Arq.), ). Moroccan Arabic (Ary.), English (Eng.), Hausa (Hau.), Hindi (Hin.), Indonesian (Ind.), Kinyarwanda (Kin.), Marathi (Mar.), and Telugu (Tel.). (Unf.=Uniform, Wt.=Weighted, TA=Task arithmetic.).
Model / Language
Afr:
Amh.
Arb.
Ary.
Ary.
Eng.
Hau.
Hin.
Ind.
Kin.
Mar.
Tel.
Avg.
BGE
22.93
79.52
33.69
42.73
E5
77.16
15.43
27.58
40.42
12.35
81.37
41.30
47.28
51.99
49.41
34.20
29. 25
43.97
Individual
GTF
77.29
15.21
22.01
34.19 19
12.88
77.79
35.25
40.69
45.20
47.33
45.66
32.36
40.48
GTE-MB
77.03
18.95
28.42
42.34
31.32
75.02
43.70
24.11
42.71
49.95
44.30
35.81
42.81
Model
Uni.
75.87
13.43
26.91
38.24
14.18
78.99
38.42
44.34
48
47.01
52.31
28.87
42.22
Merging
Wt.
76.34
14,17
27.94
38.43
13.98
80.05
38.79
46.26
49,73
48
24
28. 49
42.91
13.77
27.33
79.43
52.33
42.57
Uni.
75.79
13.822
27.18
38.19
14.25
79.16
38.
444855
48.34
46.91
52.42
23.99
42.34
Ensemble
WL
76.46
15.12
28.06
39.18
13.83
80.86
39.44
46.52
50.93
48.88
53.02
28.26
43.38
UEC
77.84
16.77
31.02
40.88
23.88
82.28
43.81
50.11
52.46
50.56
55.48
34.17
46.61
C UNCERTAINTY ESTIMATION DIAGNOSIS
C.I UNCERTAINTY ESTIMATION OF POST-HOC PRORABILISTIC EMBEDDING MODEL
We evaluate the behavior of the Laplace-based post-hoc probabilistic embedding model under several factors: covariance structure, the number of layers included in the curvature approximation, and the amount of data used for fitting. For all experiments, we fit LA following the same configuration in Appendix B.4, and evaluate uncertainty using a binary classification proxy task on MS MARCO.
Covariance Structure We first compare diagonal and KFAC covariance structures applied to the same penultimate-layer parameters with ES model. KFAC (Kronecker-Factore) Approximate Curvature) approximates the full Fisher or Hessian matrix using a Kronecker factorization of layerwise curvature, enabling structured (second-order) uncertainty modeling that is substantially richer than diagonal approximation. In principle, this allows correlations between parameters within the same linear layer to be captured, providing more expressive posterior.
33

---

Published as a conference paper at ICLR 2026
Table 17: Diagonal vs. KFAC covariance for LA. m, d, and N denote penultimate-layer width, embedding dimension, and number of LA fitting samples, respectively. Diagonal covariance achieved near identical calibration performance without extra computational overhead compared to KFAC.
Cov Type
ACC1
ECE 4
Time Comp.
Mem. Comp.
Diag KFAC
83.57
0.036
((Nmd) O(N(m²+d + m +d
O(md) O(m2+d2)
83.57
0.028
Table shows that both predictive accuracy and calibration (as measured by ECE) are nearly identical across the two settings. Despite capturing higher-fidelity curvature structure, the KFAC variant does not yield measurable improvements in uncertainty quality for this setting while incurring significantly larger computational overhead. Given this trade-off, we adopt the diagonal covariance in the main experiments.
To examine whether deeper curvature modeling affects post-hoc uncertainty, we vary the number of layers included in the LA from only the final linear projection layer (the standard configuration) up six layers.
Number of Layers
Table 18:
Effect of increasing the number of layers included in LA. Extending the curvature approxi- mation to multiple layers yields negligible gains while significantly increasing computational cost.
# Layers
1 (Last)
6
ACC1 f ECE L
83.57
83.57
83.56
83.57
0.036
0.035
0.038
0.035
As shown in Table 18, both accuracy and ECE remain stable across all configurations. This behavior aligns with recent findings in Daxberger et al. (2021) and Sharma et (2023), which show that last-layer LA already provide strong and computationally efficient uncertainty estimates for large pretrained encoders. Our observations support this conclusion: extending the curvature approximation to multiple layers yields negligible gains while significantly increasing computational cost.
Data Sparsity We further examine how the amount of data used for fitting the Laplace Approx- imation affects uncertainty estimation. We subsample the fitting set to 50%, 10%, and 1% of the original 7.5K samples, and additionally include 0% condition corresponding to purely deterministic embeddings without LA to isolate the contribution of the posterior variance.
Table 19: Effect of Laplace fitting data sparsity uncertainty estimation. Calibration error increases gradually under extreme sparsity, but remains well-behaved for moderate subsampling.
Sparsity ACC
100% (Original)
50 %
10 %
1 %
0% (Det.)
83.57
83.57
83.56
83.56
83.57
ECE
0.036
0.038
0.051
0.061
0.063
Table 19 shows that accuracy is unaffected by data sparsity, confirming that LA does not harm predictive quality. More importantly, all LA-based settings outperform the deterministic baseline (0%) in calibration. Even with only 1% of the fitting data (~75 samples), LA yields mild but consistent calibration gains. With 10% of the data (~750 samples), the calibration error nearly matches the full-data setting, indicating that LA requires only modest amount of data to produce stable variance estimates. Overall, these results demonstrate that LA is not particularly sensitive to noise sparsity, and that uncertainty estimation remains reliable even under substantial subsampling.
C.2 VARIANCE CALIBRATION OF o²
To directly assess the quality of o3 as a variance estimator, we perform a variance error calibration analysis, mentioned in Section 4.3.2.
34

---

Published as a conference paper at ICLR 2026
Setup Standard calibration metrics such ECE designed for probability outputs and not directly applicable to friance-based confidence measures such as o. To address this limitation, we introduce Var-ECE, variance-oriented calibration metrie that evaluates whether lower predicted vari- ance corresponds to higher empirical correctness. For each sentence pair (I1, x2) in SEMREL 24STS, human similarity score y [0, 1] is provided. Under UEC, the similarity is modeled as
x(x1, ~ (11( 22), (21,
where As is the predicted mean similarity and o is the epistemic variance. If a is an accurate estimate of predictive variance, then examples with large predicted variance should average incur larger squared error. We therefore compute the empirical squared emror
r(a1, x2) = (1,, x2) y( y(x1,,))) x2))²
and directly compare of and r across groups of examples.
Eertance-EFror Binning To reduce noise and obtain stable estimates, we partition the predicted variances into M quantile bins {Bm) } based on (T)) (we use M = 10 in all experiments). For each bin, we compute the average predicted variance and the average squared error:
0 m
Σ
(21,
Fm
Σ
22)1,22)
By
Bm
(#t BD))
(B
If of is well estimated, then a² should be close to Im for every bin: examples assigned higher predicted variance should indeed exhibit higher empirical error.
Variance Calibration Error (Var-ECE) empirical variances via
We measure the discrepancy between predicted and
Bm
Var-ECE
Σ
Tm
0² m
(31)
m==1
where N is the total number of evaluated examples. A small Var-ECE indicates that the predicted variance o numerically matches the observed squared error across uncertainty levels, meaning the UEC provides an accurate estimate of predictive variance.
D SENSITIVITY TO 3 AND TEMPERATURE T
We analyze the sensitivity of UEC to two key hyperparameters: (1) the coefficient 3 that deter- mines how strongly the estimated variance influences the uncertainty-awarc similarity, and (2) the temperature parameter T that sharpens the ensemble coefficients.
MIRACL Subset Figure 6 summarizes the effect of varying 3 and T on MIRACL Subset. While the curves exhibit noticeable movement, UEC maintains stable behavior across broad range of values. This reflects that both hyperparameters meaningfully control how uncertainty modulates similarity and coefficient sharpening. Performance consistently peaks near / 10 and T 1.5. which provides balanced off across all retrieval metrics.
MMTEB similar analysis on the StackOverflow task (MMTEB) presented in Figure 7. Re- markably, the optimal region remains highly consistent with the MIRACL case: the best performance again emerges around 10-2 and requires only mild temperature sharpening (here, =s 1.8).
E STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
In this study, we use LLMs in a limited, supporting capacity. The usage was restricted to polish writing, specifically for grammatical corrections, sentence refinement, and improving overall writing consistency, and experimental codes.
35

---

Published as a conference paper at ICLR 2026
nDC6@10
nDCG@10
90
Recall@10
90
Recall@
AUC8-00
AUC810
85
80
Score
75
70
70
65
65
60
10
10
10
10
0.5
1.5
2.0
B
Temperature
(a) Sensitivity to 3
(b) Sensitivity temperature T
Figure 6: Sensitivity analysis of UEC on MIRACL. Both and T influence performance in inter- pretable ways controlling variance contribution and sharpening. respectively. UEC exhibits stable optimum region around (B = 10-2, T = 1.5), making tuning straightforward in practice.
95
96
90
nDCG@10
Score
Recall@ 100
AUC@10
85
92
90
80
nDCG@10
Recallg100
AUC@10
10
10
10
10
0.5
1.0
1.5
2.0
B
Temperature
(a) Sensitivity to 3
(b) Sensitivity temperature T
Figure 7: Sensitivity analysis of UBC on StackOverflow (MMTEB). Despite dataset differences, the optimal hyperparameter region closely matches that of MIRACL, indicating that UEC's hyperparam- eter tuning is stable and transferable across tasks.
36


Reviews:
[{"review_id": "uelRkNFrMQ", "reviewer": "Reviewer_ed1F", "summary": "This paper introduces Uncertainty-driven Embedding Convolution (UEC) which is a method to construct ensemble of embeddings in an uncertainty aware method. UEC consists of three key components. First, it converts pre-trained deterministic embeddings into probabilistic embeddings in a post-hoc fashion. Second, it computes adaptive ensemble weights based on estimated uncertainty, down-weighting less reliable embeddings. This weighting strategy is grounded in a Bayes-optimal solution under a surrogate loss. Third, UEC introduces an uncertainty-aware similarity function that incorporates both distance and variance into the similarity score, offering a theoretically grounded and efficient surrogate to distributional distances. Extensive experiments on diverse benchmarks demonstrate that UEC consistently improves both performance and robustness by leveraging principled uncertainty modeling", "questions": "- Can the autors give some insight on how they are converting the deterministic embeddings into probabilisitic ones in section 3.1? It is very hard to follow currently.\n- The solution of eqn 3 should put probability 1 on the lowest value. Why do the authors claim the result in eqn 4?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work provides a way to ensemble embeddings in an uncertainty aware manner.", "weaknesses": "- It seems like this work contains a lot of jargon without providing any insights on what the proposed method is and how does it fit into the existing literature.\n- This work has major theoretical flaws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCNduFbzep", "reviewer": "Reviewer_TC4d", "summary": "### Summary\n\nThe paper proposes **Uncertainty-Driven Embedding Convolution (UEC)**, a post-hoc probabilistic ensemble method for text embedding models. It combines uncertainty estimation, probabilistic ensembling, and calibrated similarity scoring to improve robustness in retrieval and representation tasks.\n\nUEC works in three steps:\n1. **Laplace Approximation (LA)** on the last layer produces a Gaussian posterior over embeddings, modeling epistemic uncertainty.\n2. **Gaussian convolution** fuses multiple model embeddings into a combined mean and covariance with theoretically optimal coefficients.\n3. **Probit similarity** computes uncertainty-aware similarity, provably approximating the Wasserstein-2 and Jeffreys divergences.\n\nExperiments on MMTEB (retrieval, STS, reranking, clustering, bitext mining) show consistent improvements over deterministic ensembles with negligible computational overhead.", "questions": "### Questions for Authors\n\n1. Which **type(s) of uncertainty** does UEC explicitly model? From the derivations, it seems to capture **epistemic uncertainty** only (via parameter uncertainty in the last layer). Please confirm and discuss this limitation.\n2. How sensitive are your Bayes-optimal coefficients to correlation among models? Would cross-covariances alter the results?\n3. How robust is the Laplace fit when performed on out-of-domain data (e.g., fitting LA on MS MARCO, evaluating on STS)?\n4. Could you provide calibration diagnostics (ECE, reliability curves) to evaluate whether \\(\\sigma_s^2\\) reflects meaningful confidence?\n5. Would deeper (full-network) Laplace approximations substantially alter uncertainty estimates?\n6. Please include a sensitivity study on \\(\\beta\\) and temperature parameters.\n7. Can you empirically validate that the probit similarity preserves ranking order with respect to exact Wasserstein/Jeffreys distances?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Strengths\n\n1. **Principled probabilistic foundation** — Correct derivation of Gaussian embeddings from Laplace posteriors.\n2. **Theoretical rigor** — Ranking-preserving link between probit similarity, Wasserstein-2, and Jeffreys divergence.\n3. **Practicality** — Post-hoc, model-agnostic, and nearly cost-free in computation.\n4. **Empirical consistency** — Strong performance across MMTEB tasks and clear ablations.\n5. **Trustworthy AI alignment** — Explicit uncertainty propagation enhances interpretability and robustness.\n6. **Reproducibility** — Clear methodology, dataset references, and open hyperparameter reporting.", "weaknesses": "### Weaknesses\n\n1. **Limited type of uncertainty.**  \n   The method models only **epistemic uncertainty** (parameter uncertainty via Laplace Approximation on the last layer). It does not capture **aleatoric** or **predictive uncertainty**, nor propagate epistemic uncertainty through deeper layers.  \n\n2. **Strong independence and diagonal assumptions.**  \n   The Gaussian convolution assumes independent embeddings and diagonal covariances. These assumptions are rarely valid for transformer-based embeddings.  \n\n3. **Calibration not analyzed.**  \n   The paper assumes variances from LA are well calibrated, but provides no empirical evidence (e.g., ECE or reliability plots).  \n\n4. **Benchmark scope limited.**  \n   Only MMTEB is used. Out-of-distribution or multilingual setups would better demonstrate uncertainty benefits.\n\n5. **Incremental conceptual contribution.**  \n   The method is a sophisticated recombination of known tools (Laplace, Gaussian averaging, probit correction).  \n\n6. **Fairness risk.**  \n   Down-weighting uncertain embeddings could amplify demographic bias if uncertainty correlates with data imbalance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rhgrg3wtFT", "reviewer": "Reviewer_jJ7t", "summary": "This paper proposes Uncertainty-driven Embedding Convolution (UEC), which ensembles pre-trained text-embedding models by converting their deterministic embeddings into Gaussian representations via a post-hoc Laplace approximation, weighting and combining them by convolution, and using a probabilistic similarity metric that approximates distributional distance. The resulting framework improves robustness and retrieval accuracy across multilingual and multi-domain tasks, while remaining computationally lightweight. Most of the methods are very solid with many inspiration taken from applied statists literature.", "questions": "Please see my questions from weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is conceptually clean and well grounded from existing literature.\n\n2. The paper is clearly written and mostly mathematically correct, although with some times assumptions not specifically defined.\n\n3. The empirical results are convincing, the authors covering retrieval, classification, and semantic textual similarity.\n\n4. The approach is practical, post-hoc, scalable, and effective without model retraining.", "weaknesses": "1. The core components are standard statistical techniques: Laplace approximation inverse-variance weighting for Gaussian combination, and uncertainty-aware similarity based on probit approximations. Hence, I think the paper’s novelty lies mostly in assembling these ingredients into a single coherent pipeline for embedding ensembles, not in introducing fundamentally new theory.\n\n2. The “Bayes-optimal” claim is slightly overstated given that the surrogate loss drops query-dependent terms (Eq. 3).\n\n3. It would be valuable to analyse uncertainty calibration (ECE, reliability diagrams) to confirm that the modellled variances truly reflect epistemic uncertainty rather than functioning as learned scaling factors.\n\n4. Why not included some other baselines (e.g., Bayesian model averaging or deep ensemble uncertainty fusion)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lsNPFgaGMy", "reviewer": "Reviewer_ivFt", "summary": "This paper addresses the limitation of deterministic embedding ensembles, which fail to account for model-specific uncertainty, by proposing the Uncertainty-driven Embedding Convolution (UEC) method. UEC first converts pre-trained deterministic embeddings into probabilistic representations via a post-hoc Laplace approximation to quantify their uncertainty. It then derives Bayes-optimal ensemble weights that adaptively favor more confident models through a probabilistic contrastive loss. Based on this, the authors introduce an uncertainty-aware similarity function. Extensive experiments demonstrate that UEC consistently enhances performance and robustness across diverse NLP tasks by leveraging principled uncertainty modeling. The work provides a theoretically grounded and practical method for creating more reliable embeddings through ensembles.", "questions": "1. Would a general model, such as an LLM, be sufficient to handle the diverse tasks robustly? How would the proposed ensemble method perform compared to LLMs when applied to different tasks? \n2. How would the hyperparameters affect the performance of UEC?\n3. In the poem classification task, UEC underperforms an individual model GTE-MB, when it ensembles three weaker models. Can the ensemble apply to stronger models, such as GTE-MB, and still yield performance gains?\n4. How well could the proposed method estimate the mean and variance for the individual embeddings and the ensembled one, under different data sparsity settings? Are the estimations sensitive to noise?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of moving from deterministic to probabilistic embeddings for ensemble embedding learning is interesting and well-motivated. It allows quantifying model-specific uncertainty, thus improving robustness. The \"jaguar\" example effectively illustrates the need for this method.\n\n2. The paper provides strong theoretical foundations. Deriving the ensemble weights as a Bayes-optimal solution under a surrogate loss and connecting the similarity function to the 2-Wasserstein distance with a bounded error are significant contributions that add rigor and credibility to the proposed method.\n\n3. Extensive experiments have demonstrated the effectiveness of UEC, covering multiple tasks (Retrieval, Classification, STS, Bitext Mining, Clustering, Reranking) across diverse benchmarks (MIRACL, MMTEB). The consistent outperformance of UEC over strong baselines, including model merging and other ensemble techniques, provides robust evidence for its effectiveness.", "weaknesses": "1. The paper is very dense, especially Section 3. The derivations, while valuable, could be summarized more intuitively in the main text, with full details reserved for the appendix. Besides, the transition from the loss function in Eq. (2) to the tractable approximation in Eq. (3) is somewhat abrupt.\n2. There are certain hyperparameters, such as temperature and β. However, there is no analysis of UEC's sensitivity to the choices of these hyperparameters. \n3. While this ensemble method combines the strengths of specialists, i.e., different embedding models, LLMs perform like generalists. A comparison with LLM’s embeddings could justify the need for the ensemble method in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)