Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free
Authors: Ziyue Li
Tianyi Zhou
Venue: ICLR 2025 Oral (2025)
Keywords: Mixture of Experts
Primary area: foundation or frontier models, including LLMs

Abstract: While large language models (LLMs) excel on generation tasks, their decoder-only architecture often limits their potential as embedding models if no further representation finetuning is applied. Does this contradict their claim of generalists? To answer the question, we take a closer look at Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE LLMs can serve as an off-the-shelf embedding model with promising performance on a diverse class of embedding-focused tasks, without requiring any finetuning. Moreover, our extensive analysis shows that the MoE routing weights (RW) is complementary to the hidden state (HS) of LLMs, a widely-used embedding. Compared to HS, we find that RW is more robust to the choice of prompts and focuses on high-level semantics. Motivated by the analysis, we propose MoEE combining RW and HS, which achieves better performance than using either separately. Our exploration of their combination and prompting strategy shed several novel insights, e.g., a weighted sum of RW and HS similarities outperforms the similarity on their concatenation. Our experiments are conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding Benchmark (MTEB). The results demonstrate the significant improvement brought by MoEE to LLM-based embedding without further finetuning.

Published as a conference paper at ICLR 2025
YOUR MIXTURE-OF-EXPERT LLM Is SECRETLY AN EMBEDDING MODEL FOR FREE
Ziyue Li, Tianyi Zhou
Department of Computer Science
University of Maryland, College Park
(1itzy619, t tianýi @ @umd.
Project: https: ctwp://....h...o....oootooo.ao http:////.....aaoooo/o/oo//a////
ABSTRACT
While large language models (LLMs) excel generation tasks, their decoder-only architecture often limits their potential as embedding models if no further repre- sentation finetuning is applied. Does this contradict their claim of generalists? To answer the question, we take a closer look at ceature-off-Eppeeee Our study shows that the expert routers MoE LLMs can serve as an off-the-shelf embedding model with promising performance on diverse class of embedding- focused tasks, without requiring any finetuning. Moreover, our extensive analysis shows that the MoE routing weights (RW) is complementary to the hidden state (HS) LLMs, widely-used embedding. Compared to HS, we find that RW is more robust to the choice of prompts and focuses on high-level semantics. Moti- vated by the analysis, we propose MoBE combining RW and HS, which achieves better performance than using either separately. Our exploration of their combi- nation and prompting strategy shed several novel insights, e.g., a weighted sum of RW and HS similarities outperforms the similarity experiments are conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding Benchmark (MTEB). The results demonstrate the significant im- provement brought by MOBE to LLM-based embedding without further finetuning.
(MoE) LLMs.
their concatenation. Our
INTRODUCTION
HS
MoEE (ours)
72.4
70.7
58.2
58.3
59.7
56.0
Performance
48.8
38.1
34.5
29.3
24.6
24.4
Classification
Clustering
Pair Classification
Re-ranking STS
Summarization
Comparison of hidden state (HS) and MOBE (ours) on six types of tasks from the Massive Text Embedding Benchmark (MTEB), where MOEE consistently outperforms HS on all tasks. Both HS and MOEE are extracted from DeepSeekMMo-1668 (Dai al. 2024) without further finetuning.
Figure I:
Mixture-of oxiperss-o-Exxerrs (MoE) JJccbss et al. 1991 Jordan & Jacobs 1994). as a versatile architecture
originally developed in the 1990s, can improve model generalization and reduce inference cost by distributing tasks to specialized experts (Shazeer et al. (017). Over time, MoE gaining prominence
in fields such natural language processing (Shen al. 2023) and computer vision (Li et al. 2023

---

Published as a conference paper at ICLR 2025
Zong et al. 2024 Lin et al. 2024| Shi et al. 2024), especially attracting growing attention in the development of large language models (LLMs) (Muennighoff et al. 2024a] | Dai et al. 2024. Jiang et al. 2024). A key component of MoE is the dynamic routers, which intelligently assign each imput to the most relevant expert. This allows MoE to tailor its computations to the unique characteristics of each input, optimizing both efficiency and accuracy.
However, most recent LLMs and MoE LLMs are built upon the decoder-only architecture trained for autoregressive next-token prediction. While excelling on generative tasks, their final or intermediate hidden state (HS) is not designed to capture the key features of input tokens and cover all their information. Instead, HS can be biased towards the information of the next output token. Although it is a common empirical practice to extract the last token's 's hidden state (HS) as embedding (Wang) eet al., 2024), it may even perform much poorer than smaller encoder models specifically trained for embedding tasks (Lei et al., 2024) Muennighoff et al., (2024b). Take classification as an example, inputs with subtly different semanties may be associated with the same label, so the last HS aiming to predict the label may ignore the input difference. Although extra finetuning specifically for representation learning (Lee et al. 2024; Muennighoff et al.) (2024b) can greatly strengthen LLM's capability as embedding model, it raises the question of whether pre-trained LLMs can be claimed as generalists, given the broad application of embedding tasks.
HS v. RW x
HS x, RW <
Both x
Can we extract high-quality embedding directly from LLMS without additional training? In this paper, find Yes-answer to the question when studying MoE LLMs. Our main discovery is that the routers in MoE can serve as an off-the-shelf embedding model and the produced routing weights (RW) provide complementary infor- mation to the widely used HS as embedding. Compared to HS focusing on the final prediction results from the input, RW reflects the interme- diate reasoning choices of MoE on the input for each layer of LLMs. Hence, as byproduct of the routing mechanism, RW completes the input information missing in HS. As evidence, our comparative analysis of RW and HS shows that they reveal different clustering structures and topics of inputs, while RW captures the input's underlying themes and semantic structures. Moreover, we conducted an error analysis of the embedding task instances on which either HS or RW failed. As shown in Fig. 2 the proportion of cases where one embedding succeeds and the other fails exceeds 50% indicating large room for improvement combining RW and HS.
# Cases
(%)
Samples where least embedding fails
50
100
40.0%
23.8%
36.2%
STS12 Datasets
31.1%
25.5%
43.4%
30.4%
24.3%
45.3%
31.5%
20.7%
47. 47.8%
(upper bound) new success cases from combining HS and RW
failure cases even after combination
Figure 2: Complementarity of Doe-Seek.MEE- 16B's routing weights (RW) and hidden state (HS) as embedding in the task of similarity ranking STS12 datasets. In the error analysis of instances where at least one embedding failş"] we report the proportion of (1) HS succeeds V and RW fails x; (2) HS fails and RW succeeds, and (3) both RW and HS fail. In most cases, the proportion of (1)+(2) exceeds (3), indicating the complementarity of RW and HS.
Motivated by the analysis, we propose the first attempt to combine RW and the widely-used HS of MoE LLMs, resulting in a training-free. contextual-rich, and holistic embedding called "MoE Embedding (MoEE)" that excels in embedding tasks. Specifically, we experiment with various combination strategies and find that while simple concatenation of RW and HS (denoted by MOEE (concat)) improves either of them, weighted sum of the two similarities computed on RW and HS separately (denoted by MOEE (sum)) often achieves the best results. The weighted sum of similarities avoids the fusion and alignment between the two different types of embedding while allowing us to balance output-dependent information with input- senuiisensitive features, optimizing performance across diverse tasks.
We conduct extension evaluations of MOEE and compare it with baselines on the Massive Text Embed- ding Benchmark (MTEB) (Muennighoff' al. 2022). , which covers a wide range of tasks designed to test embedding quality. MOEE consistently outperforms embedding derived solely from HS or MoE's
Suucess/Failure is determined by how closely the ranking hased on the embedding matches the ground truth, with deviations heyond threshold marked as failures.
2

---

Published as a conference paper at ICLR 2025
RW, shown in Figure Notably, MoEE (sum) achieves significant gains in tasks requiring an in-depth understanding of the input, such as semantic textual similarity, classification, and clustering.
The rest of the paper is organized follows: 2 reviews related work on existing embedding methods and MoE. outlines our methodology for integrating RW of MoE with the widely-used HS embed- ding. reports experimental results MTEB, highlighting MoEE's advantages performance and interpretability. Finally, $ discusses the implications and future research directions. Results in the paper except $ are conducted on DeepSeckMoE- 16B
(Dai et al. 2024 unless specified.
2 RELATED WORK
Training-Basee -ased Embedding (pre-LIMM) Early work on sentence embedding, such as SkipThought (Kiros al. 2015), leveraged the distributional hypothesis by predicting surrounding sentences from a given input. These methods typically employed sequence-to-sqqeence architectures, following the success of Word2Vec (Mikolov) 2013). Recent advancements have shifted toward con- trastive learning, which has gained prominence for its effectiveness in self-suuer representation learning. Contrastive methods, such as SimCSE (Gao al.] [202T], exploit different views of the same sentence through data augmentation dropout, treating different outputs positive pairs and negative pairs as unrelated sentences. This approach helps models better capture semantic similarities by max- imizing the similarity between positive pairs while minimizing it between negative ones. Contrastive learning has been widely applicd in sentence embedding due to its simplicity and competitive per- formance (Wu al. 2020 Wang et al. 2021 Meng et al. 202T). Other methods like InfoNCE (Oord) et al.] 2018) and MoCo (He et al. (020) have also contributed to the development of contrastive frame- works, further enhancing embedding quality. While effective, these approaches rely on static architec- tures that may overlook input variability. In contrast, MoE models dynamically route inputs through specialized experts, producing more nuanced, context-aware embedding without additional training.
Training-Basee Based Embedding with LLMs Recent advances in language modeling have demonstrated the potential of LLMs to generate high-quality sentence embedding (Muennighoft et al. 2024b] Meng ct al. 2024). For instance, some methods, such as Sentence-TT T5 (Ni et al.] [202]]. employ contrastive learning and are capable of generating embedding that rivals fine-tuned models, even with billions of parameters. However, these methods often depend on complex pretraining and large-scale contrastive objectives, limiting their flexibility for new tasks without retraining.
Training-Fre Embedding with LLMs Training-free approaches seek to directly extract embedding from pre-trained L.LSs without the need for additional finetuning. While this process relatively straightforward for encoder-decoder models (Ni et al. 2021). it presents challenges for the more common decoder-only LLMs, where deriving meaningful embedding is less intuitive. Current approaches typically utilize the generated hidden state(s) of these models (Jiang et al. 2023). To improve the quality of these embedding, prompt-based techniques have gained traction (Jiang et al. 2022 Lei et al.; 2024). One such method, Prompt with Explicit One Word Limitation (PromptEOL) (Jiang et al. 2023), distills sentence meaning into a compact embedding by prompting the model with the instruction: 'This sentence: "[texi]" means in one word:
In pre-trained decoder-only L.Mss, embedding is typically derived from the hidden state of the final layer. Given an input sequence x = [x1, T2,,... .r,, let (()) E TRxx represent the hidden state at the 1-th layer, where T is the sequence length, d is the hidden state dimension, and = 1,2,..., is the layer index.
To extract a single embedding eus that represents the entire input sequence, one approach is to use the last token's 's hidden state in the final layer, expressed as:
H (1) E Rd
ens
Another approach is to apply pooling over all tokens in the last layer. For example, mean pooling averages the hidden states as: 1 ΣT H,(") These methods provide flexibility based task requirements, with the resulting embedding capturing the context of the input sequence as modeled by the LLM.
Mixture-offExperee (MoE) MoE models have been predominantly used in multitask learning and efficient large-scale training scenarios (Shazeer et al. 2017). However, their potential for generating
3

---

Published as a conference paper at ICLR 2025
instance-level embedding has been underexplored. Our method leverages the routing decisions made by MoE models to generate embedding that are sensitive to the input's structure and semantics. This results in more flexible and interpretable embedding compared to static models, without the overhead of task-specific retraining.
3 MIXTURE-OF-EXPERT EMBEDDING (MoEE)
Our approach leverages the dynamic routing mechanisms of pre-trained, decoder-only LLMs equipped with MoE modules to generate enriched, input-sensitive embedding. This section details the key steps of our methodology, including embedding extraction, expert routing across layers, and the final integration of embedding all achieved using pre-trate pre-trained models without any additional training.
3.1 MoE ROUTING WEIGHTS (RW) EMBEDDING
Our approach capitalizes on the dynamic routing capabilities of MoE models embedded in pre-trained, decoder-only LLMs. These MoE modules operate across multiple layers, enabling the model to specialize in processing different aspects of the input at varying depths.
Each MoE model layer 1 consists of N experts, denoted by E,", where i 1, NOO), Each expert is a specialized sub-network that focuses on specific input characteristics at that layer, allowing for more granular understanding of the input as it passes through the network. However, the true strength of this architecture lies in the dynamic routing mechanism, governed by a gating function ((("((H)) (H()) R which determines which experts will be activated at each layer based on the input.
This gating function outputs a probability distribution over the available experts in each layer, dynami-
cally selecting the most relevant ones for the current input. The routing weights gi (H()) indicate the
(1)
contribution of each expert to the final output of layer I, formulated as: E/=1 9: (H (H))) (H))),
(1)
Nr( o.") ()) 1, ensuring weighted combination of experts. The gating function is where Σ
typically implemented softmax over a of logits z (9) (H)),, making the routing decision both flexible and data-driven:
exp(x (HO))
(H)) = INO exp(2) (1) (H())
By leveraging the routing weights from all layers, our approach captures richer representation of the input that accounts for both shallow and deep contextual features. This enables the model to provide nuanced information at every level of abstraction, which is critical for tasks requiring sensitivity both lewelevel and high-level input details.
By concatenating the dynamic routing weights from all layers, we form a comprehensive routing- based embedding eRw:
eRw = (1)
(1)
(H( ))....
REE, NO
This embedding captures how the input is routed through different experts across all layers, offering a holistic view of the model's interaction with the input. Importantly, it reflects the full depth of the model's decision-malne makinng process, making a powerful representation for downstream tasks where diverse semantic and structural features of the input are essential.
3.2 COMPARATIVE & COMPLEMENTARY ANALYSIS OF ROUTING WEIGHTS & HIDDEN STATE
In this section, we investigate how routing weight (RW) embedding and hidden state (HS) embedding, generated from MoE models, capture different aspects of input data. Understanding the distinct roles these embedding play is crucial to determining how they complement each other. While HS embedding from pre-trained LLMs provides a broad, context-driven representation of sentences, they may overlook the nuanced, token-specific information that RW embedding can capture through MoE's dynamic routing.
This distinction suggests that RW and HS may excel in different contexts, potentially encoding complementary information. To explore this, we first analyze their clustering behavior using k-means
4

---

Published as a conference paper at ICLR 2025
Table 1:
Correlation of the clustering re- sults achieved on the routing weight (RW) and hidden state (HS) embedding extracted from MoE LLMs. Low scores indicate the complementarity RW and HS.
Cluster 1
Cluster 2
Cluster 3
las
twanted
dotsan
hl fly paralens
S rike
pla e
take
HS
mur der
beatplaan tlane
today pm
dennehy
ship heavly
air
neorrs
vote
Metric
Score (max value)
MoEE mo placetravel
cause
al id
sa
would
siicin
uearlll
ruchane
Adjusted Mutual
surv Survive
Information (AMT)
0.29 (1.00)
ced busit
eseoper.oo
Normalized Mutual
Information (NMI) 0.29 (1.00)
Figure 3: Word clouds of the top-20 topics from
Jaccard Similarity 0.06 (7.00)
clusters achieved RW and HS separately, highlighting their captured distinct semantic features.
Exact Matching (%) 45.54% (I00.00000
prompt
Table 2: Prompts used in Fig
prompt
prompt
Prompt
prompt
This sentence: "sent* means in one KA
HS
prompt
0,51
word:
one word, describe the style the following sentence *sent*: In one word, describe the sentiment of the following sentence (positive, neutral, or negative) *sent*: In one word, deseribe the tone of the following sentence *sent* (e.g., formal, informal, humorous, seri- ous):
prompt 7
prompt
prompt
prompt
prompt
prompt
RW
prompt
5
0.51
0.63
prompt
prompt
prompt
In one word, describe the intent be- hind the following sentence (e.g.. request, suggestion, command) *sent#:
prompt
nompar 2
proppt 3
promps 5
hommpt 5
poomi 7
mompa B-
poomps s
prompt 4
pompar 5
momm 6
pommp
prommn
momm
In one word, rate the complexity of the following sentence (simple, mod- crate, complex) *sent#:
HS
RW
Figure 4: Heatmap of Spearman's rank correlation between RW and HS embedding achieved using nine different prompts (defined in [.. The top-left (HS-HS) and bottom-right (RW-RW) blocks show the correlations between embedding when using different prompts, with mean scores of 0.52 and 0.63 (excluding the diagonal entries), respectively. This implies RW is more robust to varying prompts than HS. The top-right and bottom-left blocks reflect correlations between RW and HS when using the same or different prompts, both with mean score of 0.51. This lowest score indicates the complementarity between RW and HS.
In one word, describe whether the following sentence is subjective objective *sent
In one word, describe the language style of the following sentence (e.g.. academic, conversational, literary)-
*sent*:
In one word, describe the grammat- ical structure of the following tence (simple, compound, complex) *sent#:
clustering and perform correlation analysis to quantify the differences between their respective cluster structures. We then leverage the BERTopic framework (Grootendorst) (022) to examine the topies associated with each cluster, providing insights into the embedding' capacity to capture thematic content. Finally, we evaluate their performance in identifying semantically similar text pairs, further confirming their complementary nature.
RW and HS embedding exhibit distinet clustering behaviors and encode different topics. Our analysis shows that the clustering results from RW and HS embedding are markedly different. As reflected in Table I the clustering metrics show moderate overlap (AMI and NMI 0.29), but with a low Jaccard Similarity of 0.06 and only 45.54% exact matching between clusters, underscoring the distinct ways each method structures the data. This difference clustering behavior is further reflected the topics captured by the embedding. As shown in Figure the word clouds reveal that
Exxac matching refers to the proportion of data points that are grouped into identical clusters by two different methods (in this case, RW and HS embeddings).
5

---

Published as a conference paper at ICLR 2025
the clusters from RW and HS embedding emphasize different thematic topics, highlighting how the two methods capture divergent aspects of the input data.
Complementary nature of RW and HS embedding. Previous analyses suggest that RW and HS embedding capture different aspects of input data. To validate this hypothesis and quantify their complementarity, need to examine how these two embedding relate to another. We approach this by conducting a Spearman correlation analysis using the STS12 dataset, which contains 6,216 sentence pairs. For each pair, we generate embedding from both RW and HS and calculate the similar- ity between the sentences to assess how each embedding captures semantic relationships. To ensure that any observed differences are not caused by prompt variation, we employ nine distinct prompts (listed in Table As shown in Figure 4 notably, the correlation between RW and HS embedding is the lowest across all comparisons, with mean value of 0.51. This low correlation highlights that RW and HS capture largely unrelated aspects of the data, reinforcing their complementary nature. Further evidence supporting this complementarity is presented in the error analysis (Figure and the experimental results (Section 4h.
3.3 THE PROPOSED MOE EMBEDDING (MoEE)
Building on the analysis of routing weight (RW) and hidden state (HS) embedding, we propose our comprehensive embedding representa- tion. We introduce two approaches for this combination as follows.
method MoEE, which combines RW and HS to form
Concatenation--ase Combination. In this method, the embedding generated by the hidden state (eus) and the routing weights (eRw) are concatenated to form the final embedding. This approach is denoted as MOBE (concat), and the final embedding is computed as:
Gfiall [eus; eRw] E
R + dew
where dHs is the dimensionality of the hidden state embedding, and dRw is the dimensionality of the routing weight embedding. This method preserves the distinct information captured by each component while allowing downstream tasks leverage the combined representation.
Weighted Sum Integration. The second method performs weighted sum of the similarity scores calculated from RW and HS embedding, denoted as MoEE (sum). For tasks like STS, given sentence pair ($1, $2), we first compute the similarity score between the two sentences using both HS-based embedding and RW-based embedding independently. as (s)) ens($2), egw(s ($1). and CRw (82). Then, a weighted sum of the similarity scores performed before comparing the result to the ground truth:
simms cosine Sssine_simillrit((essSS (Hs()))),
simgw cosine_similaripn similarity(CRw (s1) CRw (82))
The final similarity score is then computed as:
sim simps + a: simgw:
where a is used as hyperparameter to control the contribution of RW. To maximize the comple- mentary strengths of HS and RW, we optimize a adaptively at test time. Specifically, a is tuned using a gradient-based approach to maximize the Spearman rank correlation between siminal and its components (simHs and simgw) over samples for the given task. This process does not require ground truth labels, focusing instead on enhancing complementarity between HS and RW. Once optimized, a remains consistent for the given task and is applied uniformly during testing.
Finally, we compute the rank correlation (e.g., Spearman's 's rank correlation) between the predicted similarity scores simanal and the ground truth similarity. This framework can be applied consistently across other tasks, adapting the weighted sum to tpsk-specific needs.
4 EXPERIMENTS
4.1 EVALUATION SETUP
We evaluate MoEE on 6 task categories from the MTEB, including Classification, Clustering, Pair Classification, Re-ranking, Semantic Textual Similarity (STS), and Summarization. We focus

---

Published as a conference paper at ICLR 2025
Caatencc-to-Sentenn (S2S) tasks as they provide direct and widely-used benchmark for embedding quality. Multilingual datasets are excluded since most MoE LLMs are trained primarily on English data. To control computational costs, include all STS and Summarization tasks and select tasks with manageable sample sizes from other categories: top-3 for Re-ranking and Classification, top-2 for Clustering, and top-2 for Pair Classification, based task statistics in Muennighoff et al. (2022). For consistent and fair comparisons, we adopt the MTEB evaluation framework and use task-specifie metrics: Accuracy (Classification), V-Measure (Clustering), Average Precision (Pair Classification), Mean Average Precision (Re-ranking), nDCG (Retrieval), and Spearman's correlation (STS and Summarization).
Our experiments use three MoE models:
DeepSeekMoB-166 (Dai al. 2024): 28 layers, with 64 experts per layer.
Qwenl.5-Moo 15-.l..---22..27 (Team 2024): 24 layers, each containing 60 experts.
OLMoE-IB-7B (Muennighoff et al. 2024a): 16 layers, with 64 experts per layer.
All models use per-token routing, but MOEE uses the last token's 's routing weights, which consistently
outperform averaging across all tokens. For the hidden state (HS) embeddings, use the last-layer hidden state of the last token. Ablation studies supporting this choice are provided in Section
Baselines Our goal is to extract advanced embedding from MoE LLMs by combining hidden state (HS) and routing weights (RW) without further training. To demonstrate the effectiveness of MOEE, compare against both RW and HS individually, well as several self-supervised and supervised methods that require training. We also assess performance across different prompt strategies, specifically comparing methods without prompts and with PromptEOL (Jiang et 2023). For an in-depth analysis of the impact of sequence length on PromptEOL embedding quality, please refer Appendix B
Table 3: Performance across MTEB Tasks withoul prompts, including Classification (CLF), Clustering (Clust.), Pair Classification (Pair CLF)), Re-ranking (Rerank), STS, and Summarization (Summ.).
MTEB Tasks
CLF
Clust.
PairCLE
Rerank
STS
Summ.
Avg.
DeepSeckMoE-116
Hidden State (HS)
44.79
25.87
44.34
38.13
34.54
24.51
35.36
Routing Weight (RW)
44.06
17.53
50.59
35.94
41.11
26.22
35,91
MOEE (concat)
44.93
24.15
51.88
41.20
46.82
31.17
40.03
MOFE (sum)
48.74
32.83
52.12
47.88
48.34
29.89
43.30
Swwnn.5--Mo-A22..
Hidden State (HS)
46.41
24.31
44,43
44,91
28.36
22.65
35.18
Routing Weight (RW)
38.99
10.55
42.26
33.53
23.97
27.44
29.46
MoEE (concat)
44.81
26.75
49.79
49.23
37.93
27.61
39.35
MoEE (sum)
50.70
31.35
51.87
49.82
45.75
24.00
42.25
OLMoE. IB-7B 7B
Hidden State (HS)
44.23
23.79
47.56
45.60
35,44
20.94
36.26
Routing Weight (RW)
43.54
17.66
53.12
40.91
44.68
28.68
38.10
MOEE (concat)
44.62
22.83
51.64
46.58
48.84
31.67
41.03
MoEE (sum)
48.54
30.67
50.93
47.77
49.45
28.77
42.69
4.2
MAIN RESULTS
Our method demonstrates consistent performance improvements across variety of MTEB tasks, as shown in Tables and 4 Results for datasets under each task type are detailed in Appendix MoFE that combines routing weights with hidden state consistently outperforms both standalone methods (RW and HS) in most cases, highlighting the complementary nature of these two components.
For tasks evaluated without prompts, the results show that MOEE (sum) achieves the highest average performance across models, with notable improvements in tasks such as Classification, Re-ranking, and STS. Specifically, DeepSeekMoE shows a substantial boost from 35,36 (HS) to 43.30 (MOEE (sum)), a 22.45% improvement. This pattern holds across Qwen1.5-Mol and OLMoE, where MoEE (sum) achieves consistent gains over both individual methods. When PromptEOL is introduced (Table 4 observe greater performance gains, with 25.9 96. improvement for DeepSeekMoF.
7

---

Published as a conference paper at ICLR 2025
Table Performance across MTEB Tasks when PrompiEOL (Jiang et al. 2023 is applied MoE. Baselines marked with * are sourced from the MTEB leaderboard (Muennighoff et al. 2022 and require training.
MTEB Tasks
CLF
Clust.
PairCLF
Rerank
STS
Summ.
Avg.
Self-Supervike Methods
Gloves (Pennington 2014
51.04
23.11
62.90
48.72
60.52
28.87
45.86
Komninose [Komnnos & Manandhar
2016
50.21
24.96
66.63
50.03
61.73
30.49
47.34
BERT+ Devim 2018
52.36
23.48
66.10
48.47
52.89
29.82
45.52
MinCSE-BEKI-une
Gao al 2021
54.80
22.59
70.79
52.42
75.00
31.15
51.13
Supervised Methods
SimCSE-BERI-sup+
58.98
29.49
75.82
53.61
79.97
23.31
53.53
coCondenser-msmarcce Gao & Callan 2021
53.89
32.85
74.56
60.08
76.41
29.50
54.55
SPECTER* (Cohan et al. 2020
42.59
27.94
56.24
55.87
60.68
27.66
45.16
LaBSE Feng et al. 2020
54.31
24.05
73.68
54.63
70.95
31.05
51.45
LASER2
42.54
14.01
70.52
46.99
64.52
26.80
44.23
1-5 Muennighof 2022
53.28
26.59
68.80
53.65
75.01
30.26
51.27
DeepSeekMoFE-In 16B
Hidden State (HS)
58.24
24.64
48.76
38.13
59.66
24.38
42.30
Routing Weight (RW)
49.52
19.97
68.30
37.48
59.52
29.26
44.01
MOEE (concat)
54.21
26.10
72.44
53.31
67.59
28.89
50.42
MOFE (sum)
58.31
34.52
70.95
55.99
70.66
29.22
53,28
Owenl.5-MoE-A2.775
Hidden State (HS)
59.34
29.50
74.29
56.51
67.39
23.01
51.67
Routing Weight (RW)
47.84
16.74
64.85
43.55
51.71
27.74
42,07
MOEE (concat)
54.23
27.18
73.93
56.12
68.52
28.57
51.43
MOFE (sum)
59.57
38.33
72.21
56.25
72.78
31.09
55.04
OLMoB-
Hidden State (HS)
58.18
32.83
72.10
58.31
72.91
27.96
53.72
Routing Weight (RW)
45,02
19.93
61.58
43.91
54.33
29.49
42.38
MOEE (concat)
52.59
33.92
71.85
56.69
71.13
30.21
52.73
MOEE (sum)
57.46
36.46
71.26
60.43
74.63
30.71
55.16
Across all models, MOEE (sum) again leads to the best results, with OLMoE achieving the highest M. following closely at 55.04, While MOEE shows marginal gains over HS in the Classification task, this is expected, as the final layer HS is more aligned with output-specific features, which benefits classification.
overall average of 55.16 and Qwen1.5-MofF
Although MOEE initially trails behind self-supervised and supervised methods without prompts, the introduction of PromptEOL leads to a significant shift. As shown in Table MoEE surpasses supervised approaches like SimCSE and coCondenser, achieving superior performance without requiring additional training. This underscores both its effectiveness and efficiency.
4.3
ABLATION STUDY
Table 5: Ablation study different ways of using routing weights (RW) and hidden state (HS).
STS Datasets
STS12
STS13
STS14
STSI5
STS16
Avg.
DeepSeekMoE: 16B
HS last token, last layer
51.99
69.56
54.68
58.04
68.47
60.40
HS last token, all layers
59.82
60.59
45.20
51.08
58.88
55.03
HS all tokens, last layer
30.95
34,42
26.77
34.90
37.11
32.78
HS all tokens, all layers
60.81
62.46
46.90
52.38
59.99
56.34
RW last token
61.97
65.86
51.38
65,86
62.49
61.18
RW all tokens
50.76
46.42
41.47
43.68
48.37
46.03
MoEE (best)
67.39
81.43
68.98
67.76
74.26
71.75
This ablation study investigates how different methods of extracting routing weights (RW) and hidden state (HS) affect embedding quality across the STS12-16 datasets, with results presented in Table
8

---

Published as a conference paper at ICLR 2025
As detailed in Section 3.1 multiple depths. In contrast, HS from only the last layer may miss important intermediate details. Therefore, evaluate the use of hidden states from all layers (HS last token, all layers) to see if it can match RW, which naturally leverages multi-layered information.
RW integrates routing decisions across all layers, capturing information at
We also assess the impact of using only the last token versus averaging across all tokens. While the last token often condenses crucial sequence information, mean pooling across all tokens may offer a broader view by incorporating contributions from every token. Thus, we compare HS last token with HS- - all tokens, and RW- last token with RW- - all tokens. For multi-layer or multi-token cases, mean pooling is applied.
Our results show that focusing on the last token, whether from HS or RW. consistently delivers the best performance. This indicates that the last token captures the most critical semantic information, while pooling across tokens or layers introduces noise. Notably, RW outperforms HS, underscoring its superior ability to capture nuanced, dynamic information that HS alone cannot replicate.
4.4 A STABILITY COMPARISON OF RW AND HS USING DIFFERENT PROMPTS
(ii)
(iii)
DeepSeekMoE-16B
Qwen1.5 Srren15-MME-Ae MMME-22.7
OLMoE-1B -B-7B
HS
RW
50
40
30
0
5TS12
STS13
5T514
STS15
57516
57512
STS13
57514
STS15
57516
51S12
STS13
5TS14
STS15
51S16
Figure 5: Box plots of the performance of the two embedding methods (RW or HS) using nine different prompts (listed in Table [ on five STS datasets, evaluated on three MoE models: (i) DeepSeckMoE, (ii) Qwenl M.M. and (iii) OLMoE. The higher variance and wider spread of HS in the box plots indicate its sensitivity to the prompt choice, while RW is more robust (lower variance) with better mean performance.
Prompts are commonly used to boost the performance of embedding models across diverse downstream tasks (Lei et al. 2024). as shown by the improved results of PromptEOL (Table 3 compared no prompts (Table 4 However, the effectiveness of these prompts can vary, and method's ' robustness depends on its ability to handle these variations. To assess the prompt sensitivity of RW and HS, we measure their Spearman correlation scores across STS12-16 datasets using different prompts listed in T The amalysis is performed on three MoE models: DeepSeckMoE, Qwen MoM,,, and OLMoF, which differ in model size and architecture, allowing us to investigate the generalizability of our findings. For each model, we compute the mean and variance of these scores for each dataset, capturing how performance fluctuates under different prompt conditions and whether the methods remain stable when exposed to prompt variations.
Figure highhlghts the performance variance for both methods. HS exhibits significantly higher variance, indicating that its performance is highly dependent on the specific prompt used. This suggests that HS is more sensitive to prompt formulation, leading to inconsistent results that could hinder its reliability in broader applications. Figure 4 (see Section 3.2) further supports this from another perspectivd showing a smaller mean correlation of 0.52 for HS using different prompts, reflecting higher variance than RW.
In contrast, RW demonstrates greater stability, with consistently lower variance and narrower box plots across all datasets, indicating its robustness to prompt choice. In Figure 4 RW also achieves
'The Spearman correlation in Figure 5 a performance metric, is between HS/RW and the ground truth, while the Spearman correlation Figure is compare different emhedding.

---

Published as a conference paper at ICLR 2025
higher mean correlation of 0.63 between different prompts, underscoring its ability to maintain stable performance across different prompts. This makes MoEE a more reliable option for tasks where prompt variability is expected.
Notably, Qwen L.5-MoE and OLMoE exhibit greater sensitivity to prompt variations compared to Mee,SeeKMMF. Despite this, HS embeddings consistently demonstrate significantly higher variance than RW embeddings across all tasks and models. This pattern of RW robustness holds consistently across the three MoE models, reinforcing its stability in diverse settings.
4.5 CASE STUDY: WHEN HS OUTPERFORMS RW & VICE VERSA
Table 6: Semantically similar sentence pairs cor- rectly predicted by HS embedding but not by RW embedding. Differences between the sentences are highlighted to show subtle variations that in- fluence prediction outcomes.
Table 7: Semantically similar sentence pairs cor- rectly predicted by RW embedding but not by HS embedding.
Sentence
Sentence
He did, the imilia- tive did not get very far.
What happened that the initiative does go very far.
Sentence
Sentence 2
1 the vote will take place today at 5.30 p.m
the vote will take
2 then perhaps could have avoided catastrophe
might have been able to prevent disaster
place at 17h30
2 the standards are scarcely comparable, alone transferable
norms are hardly comparable and still less transferable
3 it increases the power the hig countries at expense the small countrics festive social celebration
it has the eflect of augment- ing potency the big countries the detriment of babies
3 that provision could open the door wide to arbitrariness
this point of proce- dure opens the door to the arbitrary
occasion which peo- ple assemble for cial interaction and enter- tainment.
4 woman puts flour on piece of meat
woman putting flour onto some meat. fishermen inactive, tired and disappoint- ment
5 the fishermen inactive, tired and disappointed
5 group of people de- fined by a specific profession
organization of performers and associated personnel (espccially theatrical).
In this section, we analyze instances where HS embedding performs better than RW embedding (Table well cases where RW outperforms HS (Table 7 This helps identify the strengths and weaknesses each method and offers insights into when one may be preferred over the other.
From the results, HS embeddings excel in capturing formal linguistic consistency, particularly when sentence structure undergoes only superficial changes. They effectively represent the overall structure and meaning of sentences, making them useful in cases with minimal semantic variation. In contrast, RW embedding performs better when handling paraphrasing, synonym use, and nuanced stylistic shifts. The RW mechanism's sensitivity input variations allows capture deeper contextual changes, even when the overall meaning of the sentence is preserved.
CONCLUSION
In this paper, we explore the untapped potential of MoE as effective embedding generators without extra training. Our analysis reveals that RW derived from MoE models complements the widely-used HS embedding, offering a deeper understanding of input semanties. By leveraging both RW and HS, we propose MOFE, which significantly improves embedding performance across diverse tasks in the MTEB benchmark. Our results demonstrate that combining RW and HS boosts generalization. making MoE models versatile tools for embedding tasks. Future work would further explore how to leverage MOEE adaptively for task-specific scenarios.
ACKNOWLEDGMENTS
We would like thank Rocco Zhang for his preliminary attempts and experiments in the earlier-stage exploration of this project's 's initial idea. We appreciate the reviewers and area chairs for their insightful comments and suggestions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter:
Document-level representation learning using citation-informen transformers. arXiv preprint arXiv:2004.07180, 2020.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401. 00666666, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 111v.1810.00800, 2018.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic bert sentence embedding. arXiv preprint urXiv:2007.018522 2020.
Luyu Gao and Jamie Callan, Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108 055400 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882) 2021.
Maarten Grootendorst. Bertopic: Neural topic modeling with class-based tf-idf procedure. arXiv
preprint arXiv:2203.05794, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9129997299999 9738, 2020.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel. Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL /tttpps:/arxxv..oo org/ abs 2201 04088
Ting Jiang, Jian Jiao, Shaohan Huang. Zihan Zhang, Deqing Wang, Fuzhen Zhuang. Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint arXiv:2201 223.03333.. 2022.
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.1666: 2023.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures experts and the em algorithm. Neural computation, 6(2):1812144 1994.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
Alexandros Komninos and Suresh Manandhar. Dependency based embeddings for sentence clas- sification tasks. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1490-1500, 2016.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint 2rXiv:2405.17422 2024.
Yibin Lei, Di WU, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. Meta-task prompting elicits embedding from large language models. arXiv preprint arXiv:2402.
1884402114888 18458, 2024.
11

---

Published as a conference paper at ICLR 2025
Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, and Dongsheng Li. Simple: Specialized model-sample matching for domain generalization. In The Eleventh International Conference on Learning Representations, 2023.
Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer. Mike Lewis, Gargi Ghosh, Luke Zealeemoyr, and Armen Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality-awaee experts, 2024. URL BtRthhtps://axxx wwwpaaawww...o.g.eeeeaeee
21770
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding-mistrae enhance text retrieval with transfer learning. Salesforce AI Research Blog, 3, 2024.
Yu Meng,
Chenyan Xiong, Payal Bajaj. Paul Bennett, Jiawei Han, Xia Song, et al. Coco-Im: Correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, 34:23102-2311 2314, 2021.
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv.1301.3781. 2013.
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv:2202.08904, 2022.
arXiv preprint
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022. doi: 1044850/ARXI2211007336 URL
petts://aariie org www..ag./....o....../....t...e.t 221.007311001
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.0220 02200, 2024a.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.0990 0.9999 2024b.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.0887. 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arxiv: 1wwp:11000000 007488, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint Xrv: 7017701100500 .65386668822 2017.
Sheng Shen, Le Hou, Yangi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph. William Fedus, Xinyun Chen, al. Mixture-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305.14700 2023.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. arXiv:2408.1 15998, 2024.
Qwen Team. Qwen 1-5mee: Matching 7b model performance with 1/3 activated parameters", February 2024. URL https: / qwwnlm.mmmmmmmmmmmmmmm...mm....m github ic wwtp//oo//wweeee mooeennme//
Kexin Wang, Nils Reimers, and Iryna Gurevych. Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. arXiv preprint LtXiv:2104.06977 2021.
12

---

Published as a conference paper at ICLR 2025
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1 11897-11916, 2024.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive
learning for sentence representation. arXiv preprint arXiv:2012.15466 2020.
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts multimodal context, 2024. URL https://arxx
2404. 13046
13

---

Published as a conference paper at ICLR 2025
A
MTEB RESULTS
We present detailed evaluation results task types, including STS (Table 8) classification (Table pair classification (Table [O), clustering (Table IT), and re-ranking (Table 12 tasks. We show the performance of our method across different models and prompts, and compares it to baseline methods like Hidden State (HS) and Routing Weight (RW).
Table Detailed Results of STS Tasks. The DeepSeekMoE. Qwenl.5-MoE, and OLMoE models evaluated on tasks from STS12 to STSBenchmark. The MOEE method (without and with PromptFOL.) significantly improves performance across most benchmarks.
Prompt
STS12
STS13
STS14
STSIS
STS16
BIOSSES
SICK-R
STSBenchmark
DeepSeckMoE-100
Hidden State (HS)
none
20.90
43.39
24.02
31.75
47.15
29.87
42.66
30.61
Routing Weight (RW)
45.22
41.38
28.75
38.63
50.36
34.14
51.98
38.44
MoEE (concat)
none
46.26
55 88
37.90
42.37
54.19
41.20
53.66
43.06
MoEE (sum)
none
46.41
60.58
41.50
42.85
54.98
42.33
53,70
44.36
Hidden State (HS)
PrompiEOOL
51.99
69:36
54.68
58.04
68.47
45.29
63.78
65.48
Routing Weight (RW)
PromptEOL
61.97
65. 86
51.38
65 86
62.49
53.97
$7.93
56.68
MoFE (concat)
F0
63.56
71.22
61.96
MOEE (sum)
PromptEOL
67.39
81.43
68,98
67.76
74.26
62.09
69.98
73.41
Qwen 155MM
Hidden State (HS)
8.39
15.23
15.76
22.08
38.11
28.69
51.73
36.88
Routing Weight (RW)
none
27.96
18.89
13.88
17.11
36.29
25.40
29.42
22.80
MoEE (concut)
33.36
36.30
24.68
25.86
47.16
39.06
53.92
43.09
MoEE (sum)
none
35.72
47.29
31.51
31.00
50.61
53.40
62.35
54.11
Hidden State (HS)
PromptEOL
55.05
71.48
63.63
73.60
73.49
61.42
67.01
67.42
Routing Weight (RW)
PrompiEO1.
54.39
59.05
45.49
48.1
56,96
43.65
$5.46
50.54
MoFF (concat)
PromptFO1.
64.44
77.38
64.05
67.18
71.48
64.87
69.01
69.71
MoEE (sum)
PromptEOL
65.54
82.44
75.43
67.84
71.15
75.57
OLMoE-
Hidden State (HS)
none
21.33
22.71
39.885 88
51.49
22.36
Routing Weight (RW)
none
47.16
43.92
32.62
43.87
51.91
44,30
52.89
40.77
MoEE (concat)
none
48.82
52.69
37.48
46.80
56.06
54.58
52.24
42.02
MOEE (sum)
49.59
54.19
38.87
47.27
56.11
54.58
$2.82
42.16
Hidden State (HS)
PromptF01.
65.51
81.86
69.37
77.64
77.19
73.54
66.62
71.51
Routing Weight (RW)
PromptEOL
60.01
57.88
56.28
56.02
50.72
MOEE (concat)
PromptEOL
67.35
80.13
68.42
68.76
73.35
73.02
67.51
70.47
MoEE (sum)
PromptEOL.
68.84
84,34
74.02
73.81
76.88
73.02
70.56
75.58
14

---

Published as a conference paper at ICLR 2025
Table 9: Detailed Results of Classification Tasks, including sentiment extraction, emotion classifi- cation, and toxic conversations classification. The performance of different methods (Hidden State, Routing Weight, and MoEE) with and without PromptEOL is shown.
TweetSentiment-
Toxic-
Emotion-
Prompt
Extraction- Classification
Conversations- Classification
Classification
DeepSeekMoE-1d
Hidden State (HS)
none
49.14
27.55
57.69
Routing Weight (RW)
52.37
26.49
53.32
MoEE (concat)
52.64
28.02
$4.12
MOEE (sum)
none
50.32
27.52
68.39
Hidden State (HS)
PromptEOL
60.13
49.11
65.47
Routing Weight (RW)
PromptEOL
57.68
35.57
55.32
MoBE (concat)
PromplEOL
61.12
45.59
55.93
MOEE (sum)
PromptEOL
59.32
46.86
68.76
Qwen Owenl.55MoFF- EA-277
Hidden State (HS)
48.83
31.02
59.38
Routing Weight (RW)
42,80
20.63
$3.53
MOBE (concat)
none
49.60
30.93
53.90
MOFE (sum)
none
48.84
32.76
70.50
Hidden State (HS)
PromptEOL
61.14
48.09
68.80
Routing Weight (RW)
PromptEOL
55,33
33.82
54.37
MOËE (concat)
PromptEOL
60.78
46.10
55.82
MOÉE (sum)
FOO
60.72
47.97
70.03
OLMoE-1B-777
Tidden State (IIS)
50,29
30.29
52.10
Routing Weight (RW)
50.15
25.53
54,93
MoEE (concat)
none
51.59
28.76
$3.51
MOEE (sum)
none
51.00
29.75
64.86
Hidden State (HIS)
PromptEOL
59,58
47.50
67.46
Routing Weight (RW)
PromptE:L
52.79
28.51
53.75
MoBE (concal)
POomptEOL.
59.72
42.78
55.27
MOBE (sum)
PrompiEOL
59.92
45.63
66.84
Table 10: Pair classification task results on TwitterURR..Crppu and 1wit
Prompt
TwitterURL.Corpus
TvitterSemEval2015
Dol-SeckMoo-16h
Hidden State (HS)
none
49.04
39.63
Routing Weight (RW)
none
53.39
47.79
MORE (concat)
57.27
46.48
MoEE (sum)
none
58.99
45.25
Hidden State (HS)
PromptEOL.
36.72
60.79
Routing Weight (RW)
PromptEOL
76.58
60.01
MOEE (concat)
PromptEOL.
80.08
64.79
MOEE (sum)
PromptEOL
79.20
62.70
QwenI.5-MoE-A2.778
Hidden State (HS)
45.71
43.14
Routing Weight (RW)
none
48.78
35.74
MoEE (concat)
none
53.74
45.83
MoEE (sum)
57.78
45.95
Hidden State (HS)
Prompt5O1
82,50
66.07
Routing Weight (RW)
PromptEOL.
73.72
55.98
MoEE (concat)
PromptEOL.
82.34
MOEE (sum)
PromptEOL
80.21
64.20
OLMoE-
Hidden State (HS)
none
55.07
40.04
Routing Weight (RW)
none
MoEE (concat)
none
56.97
46.31
MoFE (sum)
none
57.03
44.82
Hidden State (HS)
PromptF:Q
82.32
61.87
Routing Weight (RW)
PromptEOL.
70.37
52.79
MOEE (concat)
PrompdEOL
82,32
61.38
MoEE (sum)
PromptEOL.
80.98
61.53
15

---

Published as a conference paper at ICLR 2025
Table 11: Clustering task results, showing performance on Tweatyylosgrruusclsseriii and Medrx- ivClusteringS2S. MOEE (sum) consistently performs best without prompt, while the MoEE method with PromptEOL delivers substantial gains.
Prompt
MedtrinCCusterrnng222
DeepSeckMoE-110
Hidden State (HS)
none
25.62
26.11
Routing Weight (RW)
15.33
19.72
MOEE (concut)
22.94
25.35
MOEE (sum)
none
31.44
34.22
Hidden State (HS)
PromptOL
27.02
22.26
Routing Weight (RW)
PrompEOL
21.89
18.04
MOEE (concut)
PrompiE0L
29.13
23.06
MOFE (sum)
PromptFOL.
35.77
33.27
Owenl.5-MoE-A2.77
Hidden State (HS)
26.14
22.48
Routing Weight (RW)
none
9.71
11.38
MOEE (concat)
none
28.99
24.51
MOEE (sum)
32.07
30.62
Hidden State (HS)
PrompiOL
34.04
24.95
Routing Weight (RW)
PromptE:O.
16.94
16.54
MoEE (concat)
PromptEOL.
30.45
23.91
MOFE (sum)
Prompt1:11.
42.05
34,60
OLMoE-1BB-71
Hidden State (HS)
none
21.05
26.52
Routing Weight (RW)
17.14
18.17
MoEE (concut)
none
20.72
24.94
MOEE (sum)
none
27.58
33.75
Hidden State (HS)
PromptF:OL.
26.69
Routing Weight (RW)
PromptEOL
22.13
17.72
MoEE (concat)
PromptEOL.
41.23
26.60
MOEE (sum)
PromptEL
38.58
34,33
Table 12: Re-ranking task results, showing performance on AskUbuntu, SciDocsRR, and SvackOver- flow duplicate questions re-ranking tasks.
Prompt
AskUDuntuDuppQustion
SeiDoesRR
StackOOverfooDDuugqesttttss
DeepSeek MoE-16b
Hidden State (HS)
none
43.75
43.23
25.79
Routing Weight (RW)
none
41,97
42.65
23.21
MoEE (concat)
44.10
53.43
26.06
MoEE (sum)
nome
45.26
70.79
27.58
Hidden (HS)
PromptFOL
Routing Weight (RW)
PromptEOL
46.37
42.65
23.21
MoFE (concat)
PromptEOL.
50.66
72.63
36.65
MOEE (sum)
PromptEOL
52.93
76.17
38.88
QwenI.S-NoE-A2..7
Hidden (HS)
none
43.71
Routing Weight (RW)
41.00
36.85
22.75
MoFF (concat)
none
44.95
68.42
34,31
MoEE (sum)
44,30
70.85
34.31
Hidden State (HS)
PromptEOL
54.69
75.06
39.79
Routing Weight (RW)
FO0
44.65
$5.03
30.96
MOEE (concat)
PromptEOL
52.15
75.69
40.51
MOFF (sum)
PromprOL
51.30
74.53
42.91
OLMoE-1B-7BB
Hidden State (HS)
none
43.67
69.08
24.05
Routing Weight (RW)
none
42.83
$4.17
25.72
MoEE (concat)
43.91
25.49
MoFF (sum)
none
44.57
72.54
26,20)
Hidden State (HS)
PromptEOL
55.32
78.24
41.36
Routing Weight (RW)
PromptEOL
45.
55.43
31.20
MoEE (concat)
PromptELL
52.81
77.14
40.13
MOEE (sum)
PromptEOL
56.68
43.41
16

---

Published as a conference paper at ICLR 2025
B ANALYSIS: IMPACT OF TEXT LENGTH ON PROMPTEOL EMBEDDING QUALITY
2000
Frequency
1500
(i)
1000
500
0
Average Accuracy
of HS
0.8
(ii)
0.6
1.0
(ii)) Average Accuracy of RRW
0.8
0.6
20
40
60
80
100
Sentence Length
Figure 6: (i) Distribution of sentence lengths in the STS12-16 dataset, concentrated short medium-length sequences. (ii) and (iii) show that sentence length has no significant negative impact on the average accuracy of Hidden State (HS) and Routing Weights (RW) embeddings, respectively.
PromptEOL is designed to condense the meaning of a sentence into single word, making well- suited for shorter sequences. However, its robustness for longer more complex inputs raises impor- tant questions. Specifically, does sequence length affect the quality of the embeddings? Addressing this is crucial for evaluating PromptEOL's effectiveness and identifying areas for improvement.
This analysis leverages the STS12-16 datasets, which provide diverse sentence pairs commonly used in Sentence-to-Sentle
(S2S) tasks. These datasets offer representative sample of real-world sentence lengths, making them ideal for studying the relationship between length and embedding quality. We first examine the length distribution understand the typical input range and then analyze how sequence length correlates with embedding quality, measured by accuracy. Accuracy is determined by how closely the embedding- bbsedd ranking aligns with the ground truth, with deviations beyond threshold considered correct.
Figure 6 illustrates the length distribution in the STS12-16 datasets, with a concentration of short to medium-length sequences and a median of approximately 25 tokens. Figure (ii) and (iii) shows the relationship between sentence length and average accuracy for HS and RW embeddings generated using PromptEOL, respectively. The results indicate no significant negative correlation, confirming that sentence length does not adversely affect embedding quality within the S2S context for both Hidden State (HS) and Routing Weights (RW). These findings demonstrate that PromptEOL effectively captures the semantic meaning shorter sequences without being sensitive to variations in length.
17

---

Published as a conference paper at ICLR 2025
However, for tasks involving longer sequences, such Sangraaph-ooPaargraae (P2P) embeddings, challenges may emerge. Compressing extensive information into a single word may limit embedding quality. Addressing this limitation for instance, by segmenting longer texts or introducing multi- token sinks-remaie an important direction for future work.
18


Reviews:
[{"review_id": "BjIztYwnCv", "reviewer": "Reviewer_CUz1", "summary": "The paper investigates the potential of Mixture-of-Experts (MoE) large language models (LLMs) as effective, training-free embedding models by leveraging their routing weights (RW) in addition to the commonly used hidden state (HS) embeddings. The main contributions include:\n- **Complementary Embeddings**: The authors discover that RW embeddings capture high-level semantic features and nuances that HS embeddings may miss, making RW a valuable, complementary embedding source. RW is also found to be more robust to prompt variations than HS.\n- **MoEE**: Based on these findings, the authors introduce MoEE, a method combining RW and HS embeddings. They evaluate two strategies: concatenating RW and HS and a weighted sum of their similarities. The weighted sum approach performs better, balancing output-specific information from HS with the broader, input-sensitive features from RW.\n- **Performance on Benchmark Tasks**: Experiments on the Massive Text Embedding Benchmark (MTEB) across tasks like classification, clustering, and semantic textual similarity demonstrate MoEE’s effectiveness, consistently outperforming RW and HS alone. MoEE shows particular advantages in tasks requiring deep semantic understanding, even surpassing some fine-tuned embedding models when prompt tuning (PromptEOL) is applied.\n\nThe findings suggest that combining RW and HS in MoE models offers a rich, stable embedding approach, expanding the utility of LLMs in embedding tasks without the need for further fine-tuning.", "questions": "Following the weakness, I have the questions below\n- **Aggregation Methods**: Given that the weighted sum variant outperforms concatenation for similarity calculations, have you considered more complex aggregation techniques? Could alternative methods further enhance the complementary benefits of HS and RW embeddings?\n- **Robustness of RW Embeddings**: In claiming that RW embeddings are more robust than HS, have you explored robustness across other dimensions such as model size or architectural variations? What other factors do you think could influence this robustness?\n- **MTEB Task Selection**: Could you provide insights into why only a subset of MTEB tasks was chosen for evaluation? Were there specific criteria for this selection, and do you anticipate different results if other tasks or datasets were included?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces an innovative use of routing weights (RW) as a standalone language model embedding, presenting a fresh perspective on embedding capabilities in Mixture-of-Experts LLMs.\n- The authors conduct extensive experiments to underscore the complementary nature of hidden states (HS) and routing weights (RW), providing thorough insights into their distinct contributions.\n- The empirical evaluation is well-rounded, demonstrating consistent performance improvements when HS and RW are combined, showcasing the effectiveness of the proposed approach across diverse embedding tasks.", "weaknesses": "- While MoEE introduces two methods for combining HS and RW embeddings (concatenation and weighted sum), the concatenation variant appears simplistic and less effective than the weighted sum in terms of similarity calculation. Future work could explore more sophisticated aggregation methods to fully leverage the complementary strengths of HS and RW.\n- The claim that RW embeddings are more robust than HS, based solely on prompt variation tests, lacks comprehensive support. Other factors, such as model size (or maybe architectural variations), should be examined to substantiate this claim.\n- The choice to evaluate on only a subset of the Massive Text Embedding Benchmark (MTEB) raises questions about generalizability; it would be helpful to understand the criteria behind this selection and whether other tasks or datasets might yield different insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRWVozT6cd", "reviewer": "Reviewer_A92b", "summary": "This paper first shows that RW and HS of MoE LLM are complementary for embedding-focused tasks. And they proposed two method to combine these two representations, concat and sum respectively. They benchmark the performance of the method on 6 embedding tasks of MTEB and show that it can outperform both RW and HS, as well as other previous baselines. In addition, they demonstrate the robustness of the method to different prompts.", "questions": "1. Could author provide more recent baselines, cause the best baseline in this paper from 2021?\n2. Could author explain their method is not sensitive to the scale of models?\n3. Could authors also report the stability of baselines from the main experiments?\n4. Could authors provide more details about how to tune alpha and is it consistent across experimentation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper explores the utility of the RW and HS components of MoE LLMs for embedding-focused tasks, an area that has not yet been explored by others. \n2. They have comprehensive experiment on 6 embedding tasks of MTEB with three different MoE LLMs and the method is stable across different prompts.\n3. The logic is easy to follow and understand.", "weaknesses": "1. Although the authors have compared their method to previous approaches, the baselines used are somewhat outdated. For instance, the latest baseline of this paper is published in 2021. This a old topic or authors just missed some latest baselines ( I think this could limit the contribution of this paper to the field.)?\n2. The method seems not sensitive to the scale of MoE LLM, could author provide some explanation?\n3. Though authors demonstrate the stability of the method compared to HS, it is not comprehensive without stability of baselines in the main experiment.\n4. It seems the performant variant of their method \"sum\". And \"sum\" depends on the hyper parameter alpha. Could authors provide more details of how to tune alpha? Is alpha consistent across all experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r057F28vVn", "reviewer": "Reviewer_rfVb", "summary": "The paper explores the use of the routing weights of a mixture-of-expert LLM for embedding tasks. Concatenating those routing weights over layers gives them surprisingly good scores. The paper claims that the routing weights are more robust to varying prompts than the contextualized representations and that the routing weights complement the standard contextualized representations well for embedding-centric downstream tasks. The experimental part covers tasks from the MTEB benchmark, e.g., on textual similarity, classification, and clustering. The results shows that the combination of routing weights with hidden states are preferable over the current best-practice of just using the hidden states of language models for embedding-centric tasks.", "questions": "Do you have any insights how the length of the sequence has an influence on embedding quality?\n\nGiven last-layer last-token HS performs best in the ablations, I assume that is what you use for the main table. Is this correct or did I miss anything?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is original, and the experimental results confirm its effectiveness. The MoEE technique (essentially combining hidden states with routing weights) scores higher in 5 out of 6 tasks and gives equal performance on the 6th task. This hints at the approach of making use of MoE routing weights could be generally considered for embedding-centric tasks. The paper thoroughly tests the contribution of hidden states and routing weights to the overall performance (Sec 3.2). The paper further explores different variants of dealing with multiple sequences, i.e. concatenation or calculating and interpolating cosine similarity separately for hidden states and routing weights. Here the results show that the sum variant is in most cases preferable. Generally, a strength of the approach is that no further training is required, but that those embeddings just need to be extracted from a language model. The paper further presents an ablation study comparing different token/layer selection strategies for the hidden states and routing weights -- interestingly here, routing weights already perform better than the hidden states.", "weaknesses": "To some extent the paper relies PromptEOL, i.e. prompting the model to distill the meaning of a sentence into one word. While this is clearly stated in the paper already, it could potentially hint at further limitations; for example, when a large context would need to be compressed into a single word.\n\nAlthough the baseline selection is solid, a few more commonly used models for embedding-centric tasks could be employed. SentenceBERT, ColBERT, & BertScore come to my mind, as those are commonly employed for text similarity and such tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)