Adapting General-Purpose Embedding Models to Private Datasets Using Keyword-based Retrieval
Authors: Yubai Wei, Jiale Han, Yi Yang
Accepted at: acl25

Adapting General-Purpose Embedding Models to Private Datasets Using Keyword-based Retrieval

Yubai Wei, Jiale Han and Yi Yang
Hong Kong University of Science and Technology
yubaiwei@ust.hk, jiahehan@ust.hk, imyiyang@ust.hk

Abstract
Text embedding models play a cornerstone role in AI applications, such as retrieval-augmented generation (RAG). While general-purpose text embedding models demonstrate strong performance on generic retrieval benchmarks, their effectiveness diminishes when applied to private datasets (e.g., company-specific proprietary data), which often contain specialized terminology and lingo. In this work, we introduce BMEmbed, a novel method for adapting general-purpose text embedding models to private datasets. By leveraging the well-established keyword-based retrieval technique (BM25), we construct supervisory signals from the ranking of keyword-based retrieval results to facilitate model adaptation. We evaluate BMEmbed across a range of domains, datasets, and models, showing consistent improvements in retrieval performance. Moreover, we provide empirical insights into how BM25-based signals contribute to improving embeddings by fostering alignment and uniformity, highlighting the value of this approach in adapting models to domain-specific data. We release the source code1 for the research community.

1 Introduction
Text embeddings serve as a cornerstone for various AI applications, particularly in information retrieval and retrieval-augmented generation (RAG) systems (Izacard et al., 2022; Gao et al., 2023). With the widespread adoption of AI, companies like OpenAI and Cohere now provide general-purpose text embedding APIs, enabling organizations to quickly integrate AI into their RAG systems. However, while these general-purpose embedding models show impressive performance on generic benchmarks, they often face significant challenges when applied to private datasets, such as domain-specific or company-specific proprietary data, which often contain specialized terminology and jargon (Anderson et al., 2024; Tang and Yang, 2024a).

For instance, consider a pharmaceutical company that seeks to build a RAG system over its vast internal dataset. The company’s employees may query the system for information about an internal product code (e.g., Product Code: PHX-121). However, general-purpose models, not trained on this proprietary dataset, may fail to properly interpret or retrieve relevant documents containing such specific terms, leading to suboptimal answers.

Current practices in RAG systems often attempt to address this issue by combining traditional keyword-based retrieval with embedding-based retrieval. One popular hybrid approach is reciprocal rank fusion (RRF), which reranks results based on a mathematical formula without fine-tuning the underlying embedding model (Cormack et al., 2009). While simple and effective, RRF remains heuristic, with its effectiveness potentially limited by the lack of fine-tuning to the private dataset. This leads us to the following question: Can we fine-tune general-purpose embedding models to better align with private datasets?

One of the key challenges in adapting embedding models to domain-specific datasets is the lack of available tuning signals. While general-purpose embedding models are often trained on large, curated QA datasets using contrastive learning (Tan

![An illustration of tailoring an embedding model to a private domain.](page_682_370_682_246.png)

Figure 1: An illustration of tailoring an embedding model to a private domain.

1The code is available at: https://github.com/BaileyWei/BMEmbed.
et al., 2022; Zhou et al., 2022; Moreira et al., 2024), private datasets, which often consist of free-text data without annotations, pose a particular challenge. This leads to an important sub-question: How can we generate supervisory signals for adapting general-purpose embedding models to private, unlabeled datasets?

In this work, we introduce BMEmbed, an automated framework designed to adapt general-purpose text embedding models to private datasets. Our method leverages BM25 (Robertson and Zaragoza, 2009), a well-established keyword-based retrieval function based on TF-IDF, to generate supervisory signals from the ranking of keyword-based retrieval results. The BMEmbed framework consists of three main components: (1) domain query generation, where a large language model generates synthetic queries based on domain-specific events extracted from the private corpus; (2) relevant sampling, which uses BM25 to retrieve lexically related paragraphs and samples from different intervals of the ranking list to ensure informative signals; and (3) listwise fine-tuning, where the embedding model is optimized using a listwise loss function on the curated ranking lists, fully leveraging the ranking supervision. Unlike traditional in-batch negative contrastive learning (van den Oord et al., 2018; Chen et al., 2020), our approach uses ranked BM25 results to guide the fine-tuning process.

We evaluate BMEmbed across multiple domains and datasets, using two general-purpose embedding models with varying scales. Compared to base embedding models, BMEmbed consistently achieves substantial improvements in retrieval accuracy. Our experiments further show that BMEmbed outperforms or achieves competitive performance compared to two commonly used techniques in current RAG systems: (1) fine-tuning with in-batch negative contrastive learning, and (2) the RRF hybrid approach. To better understand the inner workings of BMEmbed, we investigate the alignment and uniformity properties of the adapted embeddings (Wang and Isola, 2020). We find that BMEmbed successfully improves embedding uniformity while maintaining good alignment, leading to improved retrieval performance.

In summary, this paper introduces a simple yet effective method for adapting general-purpose text embedding models to private datasets. Given the increasing adoption of RAG systems across industries, we believe our method provides a practical solution to enhance domain specificity, leading to more accurate and contextually relevant retrieval results in real-world applications.

2 Background

2.1 Text Embedding Models

Text embedding refers to the numerical representation of a piece of text that captures its semantic meaning, transforming texts of varying lengths into fixed-size vectors. Previously, fine-tuning models like BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) to adapt to embedding downstream tasks was the dominant approach (Reimers and Gurevych, 2019; Ni et al., 2022). However, with the development of LLMs, the landscape is shifting. The focus has now moved toward building LLM-based, general-purpose embedding models, including Qwen (Li et al., 2023), LLM2Vec (BehnamGhader et al., 2024), NV-Embed (Lee et al., 2024), etc. These LLM-based embedding models have demonstrated their superiority on massive text datasets, e.g., MTEB (Muennighoff et al., 2023).

Current embedding models (Izacard et al., 2022; Wang et al., 2022; Li et al., 2023; Chen et al., 2024; Tang and Yang, 2024c) are primarily trained using contrastive learning, with the widely adopted InfoNCE loss (van den Oord et al., 2018) as the objective, which aims to distinguish semantically relevant text pairs from irrelevant ones. While effective, the performance of contrastive learning heavily depends on the selection of high-quality positive and negative samples (Tan et al., 2022; Zhou et al., 2022; Moreira et al., 2024). When adapting the embedding model to a specific domain, constructing relevant and irrelevant samples from a private corpus can be a challenging task. In this work, we propose leveraging BM25 to construct lexically relevant samples, addressing the challenge of sample selection in an unsupervised manner.

2.2 Keyword-based Retrieval: BM25

BM25 (Robertson and Zaragoza, 2009) is a well-established retrieval method based on TF-IDF, which ranks documents by considering the uniqueness and significance of terms relevant to a given query. The BM25 score for document \( d \) with respect to query \( q \) is defined as:

\[
\text{BM25}(d, q) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 - b + b \cdot |d|)}
\]
where \( f(t, d) \) is the term frequency of term \( t \) in document \( d \), \( |\hat{d}| \) is the normalization of document length, \( \sum_{t \in q} \mathrm{IDF}(t) \) is the inverse document frequency of term \( t \) in the corpus, \( k_1 \) and \( b \) are hyper parameters that control the impact of term frequency and document length, respectively. Previous works have demonstrated the effectiveness of using BM25 as a weak supervision signal for training small models (Dehghani et al., 2017; Haddad and Ghosh, 2019; Karpukhin et al., 2020).

Despite significant progress in dense retrieval (Karpukhin et al., 2020; Xin et al., 2022), BM25 remains a robust retrieval algorithm. Its rule-based, keyword matching approach enables strong generalization, maintaining competitive performance in scenarios where keyword matching is more crucial than semantic matching. As a result, hybrid approaches, such as Reciprocal Rank Fusion (RRF) (Cormack et al., 2009), have been used to combine and rerank results from both dense retrieval models (embedding-based) and sparse retrieval models (BM25-based). However, RRF relies on heuristics to rank these hybrid results. In contrast, this paper aims to fine-tune general-purpose embedding models to a specific dataset, enabling true adaptation rather than simply combining results from different retrieval methods.

3 BMEEmbed: Domain Adaptation for General-Purpose Embeddings

In this section, we present BMEEmbed, an automated framework designed to tailor general-purpose embedding models to private datasets consisting of unannotated text. The method contains three steps, and the overall process is illustrated in Figure 2.

![An overview of the BMEEmbed framework.](page_186_120_1077_410.png)

Figure 2: An overview of the BMEEmbed framework.

3.1 Domain Query Generation

The first step is to prompt an LLM (e.g., GPT-4) to generate synthetic queries focused on domain-specific events in the private corpus, rather than on general concepts.

Event Extraction We require the LLM to extract all the events and their associated arguments from the private corpus. In addition, the original context from which the events are extracted is also generated, serving as the evidence for the queries used in the baseline method in subsequent experiments.

Query Synthesis Then, we feed both the corpus and the extracted events into the LLM, prompting it to automatically generate queries \( Q \) for each event. The detailed prompts are provided in Appendix A.

3.2 Relevant Sampling via BM25

The second step is to construct ranked retrieval results using keyword retrieval method BM25.

BM25 Searching We divide the private corpus into multiple chunks and calculate the BM25 score between query \( q \in Q \) and each chunk. The top-\( k \) scoring chunks, denoted as \( C = \{c_1, c_2, \ldots, c_k\} \), are selected, where each chunk \( c_i \) is associated with its respective BM25 score \( r_i \).

Ranking List Partition We further partition \( C \) into \( m \) intervals, denoted as \( \{\mathcal{P}_1, \mathcal{P}_2, \ldots, \mathcal{P}_m\} \). This approach allows positives and negatives to be sampled from different intervals, which amplifies
the scope of sampling space across diverse relevance tiers, effectively mitigating noise in BM25 pseudo labels. The partitioning can follow either a uniform or a fine-to-coarse strategy. Uniform intervals divide the range of BM25 scores into equally sized segments, ensuring a consistent distribution of samples across all intervals. In contrast, fine-to-coarse partitioning strategy intervals prioritize finer segmentation of higher-relevance scores, leading to more granular sampling for positively ranked examples. For instance, given \( m = 4 \), the top-20 ranking list can be divided into intervals [0, 2), [2, 6), [6, 12), [12, 20) using the fine-to-coarse strategy, whereas the uniform strategy divides it into [0, 5), [5, 10), [10, 15), [15, 20).

Ranking-Based Sampling For each interval \( \mathcal{P}_j \), we randomly select one sample \( p_j \) along with its retrieval score \( r_j \), forming a ranking list \([q, p_1, p_2, \ldots, p_m, r_1, r_2, \ldots, r_m]\).

3.3 Listwise Fine-Tuning

Since BM25 retrieval results produce a ranked list, we hypothesize that this ranking contains valuable information that can be better utilized through a listwise training objective, rather than the commonly used in-batch negative contrastive learning objective, where ranking information is typically ignored. To this end, we employ a listwise training objective to fully leverage the ranking information obtained from BM25 retrieval.

Given \([q, p_1, p_2, \ldots, p_m, r_1, r_2, \ldots, r_m]\) and a base embedding model \( e(\cdot) \), we first obtain the embeddings of \( q \) and \( p_j \) for \( j \in [1, \ldots, m] \), denoted as \( e(q) \) and \( e(p_j) \), respectively. Then, we calculate the cosine similarity \( s_j = \operatorname{sim}(e(q), e(p_j)) \). Following the work of ListNet (Cao et al., 2007), the listwise loss is calculated as follows:

\[
\mathcal{L}(s, r) = - \sum_{q \in Q} \sum_{j=1}^m p_j^r \log(p_j^s)
\]

where \( r = \{r_1, r_2, \ldots, r_m\}, s = \{s_1, s_2, \ldots, s_m\}, p^r \) and \( p^s \) are the distributions normalized by softmax over the \( r \) and \( s \), respectively. We introduce a temperature scaling factor \( \alpha \) on the target score list \( r \), with:

\[
p_j^r = \frac{\exp \left( \frac{r_j}{\alpha} \right)}{\sum_{i=1}^m \exp \left( \frac{r_i}{\alpha} \right)}
\]

Here, \( \alpha \) controls the sharpness of the target distribution, with smaller values leading to a more concentrated distribution, and larger values resulting in a smoother distribution.

<table>
  <tr>
    <th>Dataset</th>
    <th>Multihop</th>
    <th>Finance</th>
    <th>LegalBench</th>
  </tr>
  <tr>
    <td>evaluation queries</td>
    <td>2,255</td>
    <td>498</td>
    <td>1,676</td>
  </tr>
  <tr>
    <td>corpus tokens</td>
    <td>1,453k</td>
    <td>840k</td>
    <td>7,109k</td>
  </tr>
  <tr>
    <td>synthesized queries</td>
    <td>5,972</td>
    <td>1,009</td>
    <td>685</td>
  </tr>
  <tr>
    <td>chunk size</td>
    <td>256</td>
    <td>1,024</td>
    <td>1,024</td>
  </tr>
  <tr>
    <td>k</td>
    <td>1,000</td>
    <td>1,000</td>
    <td>4,000</td>
  </tr>
  <tr>
    <td>m</td>
    <td>9</td>
    <td>6</td>
    <td>6</td>
  </tr>
</table>

Table 1: Statistics and implementation details of the datasets.

4 How does BMEmbed Perform?

4.1 Experimental Setup

Base Embedding Models We use the following two general-purpose embedding models: gte-Qwen2-1.5B-instruct\(^2\), a small yet strong model, and e5-mistral-7B-instruct\(^3\), a larger model based on Mistral-7B. Both two models perform competitively on the MTEB leaderboard (Muenninghoff et al., 2023).

Baselines We compare models fine-tuned by BMEmbed with the following methods: 1) **BM25**, with parameters \( k_1 = 1.2 \) and \( b = 0.75 \); 2) **Base**, the base embedding model. 3) **CL**, the embedding model fine-tuned using contrastive objective InfoNCE loss (van den Oord et al., 2018), where LLM-generated evidence is used as positives (as detailed in Section 3.1), along with in-batch negatives. 4) **RRF**, Reciprocal Rank Fusion (Cormack et al., 2009), which is a hybrid search method combining rankings from multiple sources into a unified ranking:

\[
RRF(d) = \sum_{a \in A} \frac{1}{u + a(d)}
\]

where \( d \) is a document, \( A \) is the set of rankers (retrievers), \( a(d) \) is the rank of document \( d \) in ranker \( a \), and \( u \) is a constant set to 40. Here we combine BM25 rankings with the base embedding model. 5) **RRF+BMEmbed**, the combination of the BM25 and the BMEmbed-finetuned model.

“Private” Datasets In our experiments, we choose three publicly available retrieval datasets as evaluation benchmarks. However, these datasets are released after the base embedding models, meaning the models are unlikely to have been trained on them. Therefore, while the datasets are

2 https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct
3 https://huggingface.co/intfloat/e5-mistral-7b-instruct
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">Multihop-RAG</th>
    <th colspan="5">Finance-RAG</th>
    <th colspan="5">LegalBench-RAG</th>
  </tr>
  <tr>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>41.06</td>
    <td>65.01</td>
    <td>79.02</td>
    <td>25.93</td>
    <td>28.51</td>
    <td>46.18</td>
    <td>57.43</td>
    <td>37.46</td>
    <td>0.12</td>
    <td>7.58</td>
    <td>14.62</td>
    <td>1.62</td>
  </tr>
  <tr>
    <td>Base</td>
    <td>33.97</td>
    <td>59.69</td>
    <td>76.50</td>
    <td>22.22</td>
    <td>23.69</td>
    <td>41.37</td>
    <td>53.82</td>
    <td>32.84</td>
    <td>8.00</td>
    <td>16.65</td>
    <td>23.09</td>
    <td>6.34</td>
  </tr>
  <tr>
    <td>CL</td>
    <td>31.53</td>
    <td>55.96</td>
    <td>74.72</td>
    <td>21.48</td>
    <td>25.50</td>
    <td>43.57</td>
    <td>58.43</td>
    <td>35.20</td>
    <td>6.44</td>
    <td>17.90</td>
    <td>25.48</td>
    <td>5.45</td>
  </tr>
  <tr>
    <td><b>BMEmb</b></td>
    <td>40.58</td>
    <td>68.34</td>
    <td>83.06</td>
    <td>26.54</td>
    <td>26.31</td>
    <td>45.38</td>
    <td>57.03</td>
    <td>36.21</td>
    <td>8.95</td>
    <td><b>20.64</b></td>
    <td><b>28.52</b></td>
    <td><b>7.47</b></td>
  </tr>
  <tr>
    <td>RRF</td>
    <td>38.76</td>
    <td>66.30</td>
    <td>82.04</td>
    <td>25.80</td>
    <td><b>31.73</b></td>
    <td>49.80</td>
    <td>63.45</td>
    <td>40.97</td>
    <td>8.47</td>
    <td>18.32</td>
    <td>24.76</td>
    <td>6.45</td>
  </tr>
  <tr>
    <td>RRF+BMEmb</td>
    <td><b>43.28</b></td>
    <td><b>71.09</b></td>
    <td><b>84.35</b></td>
    <td><b>28.30</b></td>
    <td><b>31.73</b></td>
    <td><b>51.61</b></td>
    <td><b>64.46</b></td>
    <td><b>41.62</b></td>
    <td><b>9.43</b></td>
    <td>19.69</td>
    <td>28.46</td>
    <td>7.19</td>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Base</td>
    <td>29.49</td>
    <td>54.99</td>
    <td>75.39</td>
    <td>20.33</td>
    <td>19.28</td>
    <td>36.55</td>
    <td>48.80</td>
    <td>28.10</td>
    <td>7.76</td>
    <td>17.42</td>
    <td>23.75</td>
    <td>6.48</td>
  </tr>
  <tr>
    <td>CL</td>
    <td>21.11</td>
    <td>48.34</td>
    <td>69.40</td>
    <td>16.67</td>
    <td>24.30</td>
    <td>46.79</td>
    <td>57.43</td>
    <td>35.08</td>
    <td>7.88</td>
    <td>16.65</td>
    <td>21.06</td>
    <td>5.37</td>
  </tr>
  <tr>
    <td><b>BMEmb</b></td>
    <td>45.28</td>
    <td><b>71.49</b></td>
    <td>85.63</td>
    <td>27.60</td>
    <td>28.11</td>
    <td>48.39</td>
    <td>62.25</td>
    <td>38.40</td>
    <td><b>9.96</b></td>
    <td>19.03</td>
    <td><b>27.27</b></td>
    <td>7.08</td>
  </tr>
  <tr>
    <td>RRF</td>
    <td>42.26</td>
    <td>67.58</td>
    <td>82.13</td>
    <td>27.04</td>
    <td>30.72</td>
    <td>47.39</td>
    <td>61.85</td>
    <td>39.55</td>
    <td>9.79</td>
    <td><b>19.09</b></td>
    <td>24.34</td>
    <td><b>7.23</b></td>
  </tr>
  <tr>
    <td>RRF+BMEmb</td>
    <td><b>45.72</b></td>
    <td><b>71.44</b></td>
    <td><b>85.72</b></td>
    <td><b>28.36</b></td>
    <td><b>32.33</b></td>
    <td><b>52.21</b></td>
    <td><b>64.06</b></td>
    <td><b>41.92</b></td>
    <td><b>9.96</b></td>
    <td>19.03</td>
    <td><b>27.27</b></td>
    <td>7.08</td>
  </tr>
  <tr>
    <td>e5-mistral-7B</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Retrieval performance of different methods across three datasets. Best results are highlighted for each embedding model on each dataset.

publicly available, they effectively simulate “private” datasets in our experiments, also ensuring fair comparison and reproducibility.

Specifically, the three datasets are: Multihop-RAG (Tang and Yang, 2024b), a multi-hop question answering (QA) dataset from the financial news domain; Finance-RAG4, a long-context QA dataset based on financial reports, released as part of the ACM-ICAIIF’24 FinanceRAG competition; and LegalBench-RAG (Pipitone and Alami, 2024), a challenging long-context legal domain QA dataset. Each dataset contains questions, their corresponding relevant evidence, and the original corpus. We use the evidence as the label to evaluate the retrieval performance. Detailed statistics are provided in Table 1.

Implementation and Training Details For domain query generation, we use GPT-4o for accurate event extraction and GPT-4o-mini for query synthesis to minimize costs. We generate 5,972, 1,009, and 685 queries for Multihop-RAG, Finance-Bench, and Legal-Bench, respectively, based on corpus size. A real case, including the input corpus, intermediate events, and the final generated query, is showcased in Appendix B. During relevant sampling, we set the chunk size of 256 for Multihop-RAG and 1,024 for the other two datasets with long context. The fine-to-coarse partitioning strategy is used by default. We adopt \( m=9 \) for Multihop-RAG and \( m=6 \) for the others, with \( k=1,000 \) for MultiHop-RAG and Finance-RAG, and \( k=4,000 \) for LegalBench-RAG. The impact of different \( m \) and partitioning strategies is further discussed in Section 5.2. The results under different \( k \) are shown in Appendix D. For finetuning, we use a fixed batch size of 16 for CL, while the batch size is equivalent to \( m \) for BMEmb. The temperature \( \alpha \) is set to a moderate value between 1.0 to 5.0, with further adjustments on different datasets and models, which we provide a detailed discussion in Section 5.3. We finetune the model using LoRA (Hu et al., 2022) with a rank of 16 for 1,000 steps. Training Qwen on 4×3090 GPUs takes about 1.5 hours, while training e5-mistral on 8×H800 GPUs takes approximately one hour.

4.2 Results and Discussion

Table 2 presents the experimental results of BMEmb and all baselines across two embedding models and three datasets. It can be observed:

1) The vanilla embedding models perform suboptimally in specific domains. In most cases, base models underperform BM25 on Multihop-RAG and Finance-RAG, even with large model sizes. This finding highlights the necessity of further adaptation when applying general-purpose embedding models to specific domains. Furthermore, the BMEmb consistently outperform BM25 across models and datasets, despite being trained with supervisory signals derived from BM25. This demonstrates that BMEmb is not merely mimicking BM25. Instead, we treat BM25 as a weak lexical teacher and design both our sampling strategy and training objective to guide the model toward learning relevance information beyond BM25’s direct outputs.

2) Contrastive learning does not consistently lead to performance improvements for embedding model adaptation. Surprisingly, we find that applying CL to base models do not always improve
Figure 3: Retrieval performance of MAP@10 for different \( m \) and sampling strategies.

Figure 4: Alignment and uniformity for different \( m \) and sampling strategies.

Figure 5: Retrieval performance of MAP@10 for different \( \alpha \).

Figure 6: Alignment and uniformity for different \( \alpha \).

performance. We hypothesize that noise in the positive evidence generated by the LLM might interfere with model optimization. This indicates that contrastive learning is sensitive to the quality of positive and negative samples, and such an approach does not always result in promising improvements for embedding adaptation.

3) Our BMEmbed consistently delivers improvements, benefiting from the supervision signals provided by BM25. Our framework boosts the base models across all embedding models and datasets, especially on the metrics Hit@4. Compared to RRF which combines BM25 ranking information with dense retrieval from embedding models, BMEmbed achieves a remarkable improvement, which illustrates that our framework deeply deciphers the ranking confidence signals from BM25, achieving a better embedding model adaptation.

4) Furthermore, BMEmbed can be combined with other hybrid retrieval methods to achieve further enhancement. This is demonstrated in experiments comparing RRF+BMEmbed with RRF alone. In most cases, RRF+BMEmbed shows clear performance gains, except in the case of LegalBench-

RAG, where the BM25 baseline performs poorly and BMEmbed+RRF does not achieve further performance gains.

4.3 Generality under Alternative Settings

To further explore the generality of the BMEmbed framework, we conduct additional experiments under three different settings: (1) applying BMEmbed to a smaller embedding model, (2) replacing the loss function in listwise fine-tuning, and (3) evaluating the adapted embedding model on other embedding task. Full experimental setups and results are provided in Appendix C.

In Setting 1, we choose all-MiniLM-L6-v2\(^5\) from the Sentence Transformers family as a smaller embedding model. We observe that even small model can achieve performance comparable to larger general-purpose model after BMEmbed adaptation, while requiring significantly fewer computational resources and training time. This highlights the practicality and efficiency of our framework in resource-constrained scenarios. In Setting 2, we replace the cross-entropy loss with a

\footnotetext{5 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2}
maximum likelihood loss, ListMLE (Xia et al., 2008) in listwise fine-tuning. The adapted model still shows consistent improvements, demonstrating that BMEmbed is robust across different listwise training objectives. In Setting 3, we evaluate the adapted embedding Qwen2-1.5B on FinSTS (Liu et al., 2024), a semantic textual similarity task. Despite being trained solely on listwise signals derived from BM25 rankings and without any direct supervision on the STS task, the adapted models achieve noticeable improvements. This suggests that our approach effectively captures domain-specific semantic nuances, further highlighting its broader utility.

5 Why BMEmbed Enhances Embedding Adaptation? An Investigation from Uniformity and Alignment

In this section, we further investigate why BMEmbed leads to improvements. We conduct ablation experiments to study how our samplers and temperature interact with retrieval performance. Moreover, we introduce the Alignment and Uniformity properties, which reflect the quality of the embedding, to gain a deeper theoretical understanding. The reported experiments are based on the Multihop-RAG dataset and the Qwen2-1.5B model by default. The complete ablation study setup and results are presented in Appendix D. As observed in the ablation study, our experiments empirically reveal a strong agreement between embedding properties and retrieval performance, suggesting that the enhancement from BMEmbed results from the optimized embedding properties. Here, we discuss our key observations and conclusions.

5.1 Alignment and Uniformity

A good embedding should bring similar data points closer together while preserving as much useful information as possible (Bachman et al., 2019; Hjelm et al., 2019) to distinguish different data points, leading to lower alignment and higher uniformity. Here, we adopt alignment and uniformity for evaluating an embedding following the work of Wang and Isola (2020), with further details and discussion in Appendix E.

5.2 Ablation Study of Different Partitions

To explore the effect of different partitions during relevant sampling via BM25 in BMEmbed, we investigate the impact of various partition factors, including the number of partitions and the partitioning strategies. Specifically, we conduct experiments with \( m \) ranging from 6 to 10, using both uniform and fine-to-coarse sampling strategies, with the temperature \( \alpha \) set to 1 and \( k \) set to 1,000.

Figure 3 shows the relationship between retrieval metrics MAP@10 and fine-tuning with different \( m \) and sampling strategies, while Figure 4 presents a comparison of uniformity and alignment of the fine-tuning models shown in previous figure. We observe that the fine-to-coarse strategy achieves better retrieval performance and superior alignment compared to the uniform strategy. In contrast, the uniform strategy is suboptimal in retrieval performance due to its overly uniform embedding distribution, which leads to a loss of alignment. In addition, as \( m \) increases from 6 to 7 under the fine-to-coarse sampling strategy, we observe a measurable improvement in MAP@10 performance, suggesting that moderately expanding the sampling scope captures more relevant items. However, further increasing \( m \) causes performance fluctuations and a gradual decline in overall effectiveness. These findings highlight the importance of carefully calibrating \( m \) to optimize retrieval performance.

5.3 Ablation Study of Listwise Fine-Tuning with Varying Temperatures

We examine the effect of varying temperatures \( \alpha \). For convenience, we work with its reciprocal, \( 1/\alpha \), with values of 0.1, 0.2, 0.5, 0.7, and 1.0. We set \( k=500, m=10 \), and adopt the fine-to-coarse sampling strategy.

Figure 5 shows the trend between MAP@10 and fine-tuning with different \( 1/\alpha \), with the corresponding alignment and uniformity results shown in Figure 6. Our analysis shows that smaller temperature achieve better retrieval performance by fostering good uniformity in the embedding distribution. In contrast, as temperature increases, uniformity decreases, even lowering it compared to the base model. This is because the higher temperature smooths the label distribution, which diminishes the distinction between learning samples and causes the embeddings to become overly clustered. Such clustering may hurt the performance of downstream tasks which require clear distinction between embeddings, as observed in our experiments, where it led to a degradation in retrieval performance.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Multihop-RAG</th>
    <th colspan="2">Finance-RAG</th>
    <th colspan="2">LegalBench-RAG</th>
  </tr>
  <tr>
    <th>Alignment↓</th>
    <th>Uniformity↑</th>
    <th>Alignment↓</th>
    <th>Uniformity↑</th>
    <th>Alignment↓</th>
    <th>Uniformity↑</th>
  </tr>
  <tr>
    <td></td>
    <td colspan="6">Qwen2-1.5B</td>
  </tr>
  <tr>
    <td>Base</td>
    <td>1.2422</td>
    <td>2.7665</td>
    <td>1.1562</td>
    <td>1.6567</td>
    <td><b>1.3203</b></td>
    <td>1.1599</td>
  </tr>
  <tr>
    <td>CL</td>
    <td>1.3516</td>
    <td>2.8022</td>
    <td>1.2188</td>
    <td><b>2.9437</b></td>
    <td>2.0000</td>
    <td><b>2.2382</b></td>
  </tr>
  <tr>
    <td>BMEEmbed</td>
    <td><b>1.2031</b></td>
    <td><b>3.3266</b></td>
    <td><b>1.1484</b></td>
    <td>2.6631</td>
    <td>1.6691</td>
    <td>2.1426</td>
  </tr>
  <tr>
    <td></td>
    <td colspan="6">e5-mistral-7B</td>
  </tr>
  <tr>
    <td>Base</td>
    <td>1.1875</td>
    <td>1.7430</td>
    <td>1.1797</td>
    <td>1.0353</td>
    <td><b>1.2891</b></td>
    <td>0.7317</td>
  </tr>
  <tr>
    <td>CL</td>
    <td>1.5156</td>
    <td>2.7649</td>
    <td>1.3281</td>
    <td>3.0445</td>
    <td>2.7969</td>
    <td><b>1.7913</b></td>
  </tr>
  <tr>
    <td>BMEEmbed</td>
    <td><b>1.1797</b></td>
    <td><b>3.7768</b></td>
    <td><b>1.0859</b></td>
    <td><b>3.2144</b></td>
    <td>1.6797</td>
    <td>1.6182</td>
  </tr>
</table>

Table 3: Alignment and Uniformity of Embedding Models. Lower alignment (↓) and higher uniformity (↑) are preferred. Best results are highlighted for each embedding model on each dataset.

<table>
  <tr>
    <th>Original Query</th>
    <th>Masked Query</th>
    <th>Substituted Query</th>
  </tr>
  <tr>
    <td>What variables are considered on top of the value at <b>1 January</b> when calculating the value at 31 December for <b>government grants</b> that are included within trade and other <b>payables</b>?</td>
    <td>What variables are considered on top of the value at [MASK] when calculating the value at 31 December for [MASK] [MASK] that are included within [MASK] and other [MASK]?</td>
    <td>What variables are considered on top of the value at New Year's Day when calculating the value at 31 December for public subsidies that are included within commerce and other liabilities?</td>
  </tr>
</table>

Table 4: A comparative example of three query perturbation types.

5.4 BMEEmbed Balances Alignment and Uniformity Optimization

Our ablation experiment and analysis have demonstrated that using the fine-to-coarse strategy with a smaller temperature is an effective way to leverage BM25, supported by both theoretical reasoning and practical results. Since main experiment we conducted in Section 4.2 is based on this strategy, here we report the uniformity and alignment of corresponding fine-tuned embedding models in Table 3 for further analysis.

<b>Embedding models fine-tuned with BMEEmbed achieve better retrieval results due to increased uniformity compared to the base model, while maintaining relatively low alignment.</b> Comparing with CL with in-batch negatives, we observe that although uniformity has increased significantly, it does not effectively maintain or improve the alignment of the base model. This imbalance leads to instability in retrieval performance, and in some cases, even performance degradation. Specifically, we identify the ideal optimization direction, as indicated by the red arrow in the in Figure 4. BMEEmbed achieves this theoretical direction on both Multihop-RAG and Finance-RAG, demonstrating its potential to balance the optimization of both uniformity and alignment.

6 How Does BM25 Signal Boost Embedding? Integrating Lexical Sensitivity with Semantic Generalization

While BMEEmbed is fine-tuned using weak supervision signals derived from BM25, it remains unclear what specific capabilities this adaptation imparts to the embedding model. To investigate this, we design a set of controlled experiments and introduce a two-part decomposition of model behavior: semantic generalization, defined as the ability to capture general semantic patterns, and lexical sensitivity, defined as the sensitivity to domain-specific lexical cues. To assess how BMEEmbed balances these two aspects, we conduct three groups of query perturbation experiments: (1) retrieval using original queries, (2) retrieval with domain-specific keywords masked, and (3) retrieval with query keywords substituted by synonyms.

The experimental pipeline involves three stages. First, we use an LLM to extract domain-specific keywords and generate semantically appropriate synonyms for each query (see prompt details in Appendix F). Next, we create two perturbed versions
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Perturbation Method</th>
    <th colspan="1">Hit@1</th>
    <th colspan="1">Hit@4</th>
    <th colspan="1">Hit@10</th>
    <th colspan="1">MAP@10</th>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="3">BM25</td>
    <td>original</td>
    <td>28.51</td>
    <td>46.18</td>
    <td>57.43</td>
    <td>37.46</td>
  </tr>
  <tr>
    <td>masked</td>
    <td>0.40<br>(.28.11)</td>
    <td>6.65<br>(.39.53)</td>
    <td>11.29<br>(.46.14)</td>
    <td>3.17<br>(.34.29)</td>
  </tr>
  <tr>
    <td>substituted</td>
    <td>5.85<br>(.22.66)</td>
    <td>12.10<br>(.34.08)</td>
    <td>16.13<br>(.41.30)</td>
    <td>8.94<br>(.28.52)</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2-1.5B</td>
    <td>original</td>
    <td>23.69</td>
    <td>41.37</td>
    <td>53.82</td>
    <td>32.84</td>
  </tr>
  <tr>
    <td>masked</td>
    <td>2.41<br>(.21.28)</td>
    <td>4.62<br>(.36.75)</td>
    <td>5.82<br>(.48.00)</td>
    <td>3.31<br>(.29.53)</td>
  </tr>
  <tr>
    <td>substituted</td>
    <td>8.87<br>(.14.82)</td>
    <td>17.14<br>(.24.23)</td>
    <td>24.40<br>(.29.42)</td>
    <td>13.08<br>(.19.76)</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2-1.5B + BMEmbed</td>
    <td>original</td>
    <td>26.31</td>
    <td>45.38</td>
    <td>57.03</td>
    <td>36.21</td>
  </tr>
  <tr>
    <td>masked</td>
    <td>2.21<br>(.24.10)</td>
    <td>4.42<br>(.40.96)</td>
    <td>8.63<br>(.48.40)</td>
    <td>3.76<br>(.32.45)</td>
  </tr>
  <tr>
    <td>substituted</td>
    <td>9.27<br>(.17.04)</td>
    <td>18.95<br>(.26.43)</td>
    <td>26.81<br>(.30.22)</td>
    <td>14.30<br>(.21.91)</td>
  </tr>
</table>

Table 5: Controlled Retrieval Experiments with Query Perturbations on Finance-RAG.

of each query by either masking the identified keywords or substituting them with their corresponding synonyms (examples shown in Table 4). Finally, we evaluate model performance across these variants to assess the impact of each perturbation type. The evaluation is conducted on the Finance-RAG dataset using three methods: BM25, the base Qwen2-1.5B embedding model, and Qwen2-1.5B fine-tuned with BMEmbed. As shown in Table 5, the results provide several insights of BMEmbed:

1) Semantic Generalization: Compared to BM25, BMEmbed exhibits significantly less performance drop under synonym substitution (Hit@10 drop: 30.22 vs. 41.30), indicating stronger semantic generalization. Notably, even when compared to the base Qwen2-1.5B model, BMEmbed achieves slightly better absolute performance (Hit@10: 26.81 vs. 24.40) despite experiencing a similar level of performance drop (Hit@10 drop: 30.22 vs. 29.42). This suggests that our fine-tuning process not only preserves but also enhances the model’s semantic generalization ability.

2) Lexical Sensitivity: Under keyword masking, BMEmbed shows a larger performance drop than the base model (Hit@4 drop: 40.96 vs. 36.75), implying that BMEmbed has become more sensitive to domain-specific lexical cues, especially to the high ranking items. This indicates that while BMEmbed preserves semantic understanding, it also better incorporates keyword-level information.

These results suggest that BMEmbed effectively combines the strengths of both lexical and semantic information. This dual capability makes it particularly well-suited for domains that require adaptation to specialized terminology, or for proprietary enterprise datasets.

7 Conclusion

With the growing adoption of AI in real-world applications, particularly RAG systems, adapting general-purpose models to domain-specific data remains a critical challenge. In this paper, we present BMEmbed, a novel method for adapting text embedding models to private datasets (e.g., company-specific proprietary data). Since private datasets often contain specialized terminology and domain-specific language, we leverage keyword-based retrieval as a supervisory signal to fine-tune general-purpose embedding models. Experimental results demonstrate that BMEmbed effectively enhances retrieval performance, producing more accurate query results on private datasets. As AI continues to transform industries, we hope that our proposed method can further advance the adoption and adaptation of AI in domain-specific applications, ensuring more effective and contextually relevant retrieval.
8 Limitations

This study has several limitations that present opportunities for future research. First, our current method primarily focuses on the retrieval task in embedding models. However, text embeddings are also widely used in domain-specific NLP tasks such as clustering and semantic textual similarity (STS). An interesting direction for future research is exploring task-specific supervisory signals to better adapt general-purpose embedding models to private datasets for applications beyond retrieval, including clustering and STS. Second, while our method aims to develop embedding models tailored to private datasets (such as company-specific proprietary data), we evaluate it on public datasets. These datasets are chosen because they are released after the base embedding models we assess, ensuring fair comparison and public reproducibility. However, applying this method to proprietary datasets in real-world RAG scenarios remains an important next step. We hope future research will explore these practical applications to further validate and refine our approach.

Acknowledgment

This work is partially supported by a research grant provided by HSBC. We also thank the anonymous reviewers for their thoughtful and constructive comments.

References

Peter Anderson, Mano Vikash Janardhanan, Jason He, Wei Cheng, and Charlie Flanagan. 2024. Greenback bears and fiscal hawks: Finance is a jungle and text embeddings must adapt. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: EMNLP 2024 - Industry Track, Miami, Florida, USA, November 12-16, 2024, pages 362–370. Association for Computational Linguistics.

Philip Bachman, R. Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 15509–15519.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961.

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, volume 227 of ACM International Conference Proceeding Series, pages 129–136. ACM.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 2318–2335. Association for Computational Linguistics.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR.

Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA, July 19-23, 2009, pages 758–759. ACM.

Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 65–74. ACM.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6894–6910. Association for Computational Linguistics.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
Dany Haddad and Joydeep Ghosh. 2019. Learning more from less: Towards strengthening weak supervision for ad-hoc retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 857–860. ACM.

R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.

Vladimir Karpukhin, Barlas Ouzu, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769–6781. Association for Computational Linguistics.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Jiaxin Liu, Yi Yang, and Kar Yan Tam. 2024. Beyond surface similarity: Detecting subtle semantic shifts in financial narratives. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2641–2652.

Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831.

Niklas Muenninghoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 2006–2029. Association for Computational Linguistics.

Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1864–1874. Association for Computational Linguistics.

Nicholas Pipitone and Ghita Houri Alami. 2024. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. arXiv preprint arXiv:2408.10343.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980–3990. Association for Computational Linguistics.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Haochen Tan, Wei Shao, Han Wu, Ke Yang, and Linqi Song. 2022. A sentence is worth 128 pseudo tokens: A semantic-aware contrastive learning framework for sentence embeddings. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 246–256. Association for Computational Linguistics.

Yixuan Tang and Yi Yang. 2024a. Do we need domain-specific embedding models? an empirical investigation. arXiv preprint arXiv:2409.18511.

Yixuan Tang and Yi Yang. 2024b. Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries. arXiv preprint arXiv:2401.15391.

Yixuan Tang and Yi Yang. 2024c. Pooling and attention: What are effective designs for llm-based embedding models? Preprint, arXiv:2409.02727.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929–9939. PMLR.

Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192–1199.

Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero-shot dense retrieval with momentum adversarial domain invariant representations. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4008–4020. Association for Computational Linguistics.

Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Debiasied contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 6120–6130. Association for Computational Linguistics.

A Prompts Used for Domain Query Generation

The LLM prompts used in the domain query generation stage are detailed as follows:

Event Extraction Prompt:
Given a document, please extract all the events and their associated topics and organization in the context.
Note: 1. The event should not contain ambiguous references, such as 'he', 'she', and 'it', and should use complete names.
2. You should give at least one passage in the original text associated to the event you extract, DO NOT make up any event.
3. If there are multiple paragraphs associated to the extracted event, please list and number all of them.
4. If the event does not contain some of the arguments mentioned above, please leave it empty.
5. The type of Event involves fine-grained events and general events, where fine-grained events focus on specific facts and details while general events are summarizations of happened fine-grained events.
6. Please return the fine-grained events first, then return general events.
The document is:
{doc}
Please return the extracted event in the following format with following arguments:
[Event];
[Topic];
[Original context]: 1. ......
2. ......
...
[Type]:
Events you extract are:

Query Synthesis Prompt:
Given several events and their original source document, please ask several questions according to the information and give the original reference paragraph following this format:
[Event];
[Question];
Note: 1. Don’t need to mention all the arguments in your question.
2. You can involve the original document information, but make sure that your question is about the topic of the given event.
3. You should ask questions separately to different events.
Document:
{doc}
Event:
{event}
Your question towards given event:

B Case Study of Query Generation

In this section, we present a real query generation process, showcasing the input document, intermediate extracted events, and the final generated query.

Document Chunk from Multihop-RAG:
Table of Contents Table of Contents Echo, Fire TV, and Kindle deals Apple deals TV deals Laptop deals Headphone and earbud deals Tablet deals Gaming deals Speaker deals Vacuum deals Kitchen deals Smart home deals Fitness deals Beauty tech deals Drone deals Camera deals Lego deals Gift card deals UPDATE: Nov. 27, 2023, 5:00 a.m. EST This post has been updated with all of the latest Cyber Monday deals available at Amazon. Amazon is dragging out the year’s biggest shopping holiday(s) into 11 days of deals. The retail giant began its Black Friday sale in the early morning of Friday, Nov. 17 (a week ahead of schedule) and was on top of making the switch to Cyber Monday language in the wee hours of Saturday, Nov. 25. Official Cyber Monday mode, which is currently on through Monday, Nov. 27, includes both a ton of deals carried over from Black Friday plus some new ones.

GPT-4o Extracted Events:
### Fine-Grained Events
1. [Event]: Amazon began its Black Friday sale.
[Topic]: Black Friday sale
[Original context]: "The retail giant began its Black Friday sale in the early morning of Friday, Nov. 17 (a week ahead of schedule)..."
[Type]: Fine-grained
2. [Event]: Amazon switched to Cyber Monday language.
[Topic]: Cyber Monday sale
[Original context]: "...and was on top of making the switch to Cyber Monday language in the wee hours of Saturday, Nov. 25."
[Type]: Fine-grained
3. [Event]: Amazon’s Cyber Monday mode includes deals carried over from Black Friday plus some new ones.
[Topic]: Cyber Monday sale
[Original context]: "Official Cyber Monday mode, which is currently on through Monday, Nov. 27, includes both a ton of deals carried over from Black Friday plus some new ones."
[Type]: Fine-grained
<table>
  <tr>
    <th>Model</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>33.97</td>
    <td>59.69</td>
    <td>76.50</td>
    <td>22.22</td>
  </tr>
  <tr>
    <td>e5-mistral-7B</td>
    <td>29.49</td>
    <td>54.99</td>
    <td>75.39</td>
    <td>20.33</td>
  </tr>
  <tr>
    <td>MiniLM</td>
    <td>17.52</td>
    <td>39.96</td>
    <td>55.79</td>
    <td>12.55</td>
  </tr>
  <tr>
    <td>MiniLM+BMEEmbed</td>
    <td>32.77</td>
    <td>60.18</td>
    <td>78.27</td>
    <td>22.40</td>
  </tr>
  <tr>
    <td></td>
    <td>(+15.25)</td>
    <td>(+20.22)</td>
    <td>(+22.48)</td>
    <td>(+9.85)</td>
  </tr>
</table>

Table 6: Retrieval Performance of Different Models on MultihopRAG.

<table>
  <tr>
    <th>Method</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
  </tr>
  <tr>
    <td>ListNet</td>
    <td>40.58</td>
    <td>68.34</td>
    <td>40.58</td>
    <td>26.54</td>
  </tr>
  <tr>
    <td>ListMLE</td>
    <td>39.87</td>
    <td>67.98</td>
    <td>83.10</td>
    <td>26.29</td>
  </tr>
</table>

Table 7: ListMLE vs. ListNet under identical training settings.

<table>
  <tr>
    <th>Model</th>
    <th>without BMEEmbed</th>
    <th>with BMEEmbed</th>
    <th>Improvement</th>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>0.2566</td>
    <td>0.2727</td>
    <td>+0.0161</td>
  </tr>
  <tr>
    <td>e5-mistral-7B</td>
    <td>0.2678</td>
    <td>0.3024</td>
    <td>+0.0346</td>
  </tr>
</table>

Table 8: Evaluation on FinSTS with Cosine Spearman correlation.

C Experimental Details for Generalization Settings

Setting 1: We use the all-MiniLM-L6-v2 model, which has only 22.7M parameters. We fine-tune it on the Multihop-RAG dataset using a learning rate of 2e-5 for 500 steps, with the same sampling strategy used in our main experiments for this dataset. The full fine-tuning process required only 24 minutes on a single NVIDIA 3090 GPU, with a peak memory usage of 4GB. The results are presented in the Table 6.

Setting 2: We conducted an additional experiment using ListMLE (Xia et al., 2008) on the Qwen2-1.5B model and the Multihop-RAG dataset, under the same settings as our main experiment. This setup is compared against ListNet-based training used in the our main experiment. The results are presented in Table 7.

Setting 3: We conducted an additional experiment to evaluate the adapted embeddings on a semantic similarity task. Specifically, we used the model fine-tuned on the Finance-RAG dataset and evaluated it on the FinSTS dataset (Liu et al., 2024), a well-annotated benchmark designed to detect subtle semantic shifts in financial narratives. Since both datasets are based on financial reports, FinSTS serves as a "private evaluation set" in this context. In this evaluation, we adopted a last-token pooling configuration and used the Cosine Spearman Correlation as the evaluation metric. The results are presented in Table 8.

D Ablation Study

D.1 Ablation Study of Query Generation Module

We conduct experiments to investigate the impact of the number of synthetic queries used for fine-tuning. Specifically, we compare three settings: (1) using the full set of synthetic queries, (2) using a randomly sampled 50% subset, and (3) using a randomly sampled 25% subset. To control for the total number of training samples, we change the number of listwise samples generated per query. Specifically, we increase the number of sampled ranking lists per query accordingly when using fewer queries, ensuring the overall amount of training data remains constant. All experiments are conducted on the Multihop-RAG dataset using the Qwen2-1.5B model. All other settings are kept fixed, including the sampling strategy, number of training steps (1,000), and the temperature (1.0) used in listwise fine-tuning.

As shown in Tabel 9, no significant performance difference is observed across the three settings, suggesting that the number of synthetic queries has limited impact on the model’s performance. This indicates that BMEEmbed can compensate for fewer queries by generating multiple listwise samples per query, thereby maintaining training signal quality.

D.2 Ablation Study of Relevant Sampling Module

We conduct three sets of experiments on Multihop-RAG and Qwen model while controlling different variables, investigating four key factors according to our pipeline:

• selection of \( k \), we explore values of \( k \) at 200, 500, and 1000;

• selection of \( m \), we examine \( m \) values ranging
<table>
  <tr>
    <th>Setting</th>
    <th>Samples per Query</th>
    <th>Total Samples</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
  </tr>
  <tr>
    <td>full set</td>
    <td>1</td>
    <td>5,972</td>
    <td>41.02</td>
    <td>69.36</td>
    <td>84.79</td>
    <td>26.96</td>
  </tr>
  <tr>
    <td>subset(50%)</td>
    <td>2</td>
    <td>5,972</td>
    <td>39.91</td>
    <td>68.43</td>
    <td>84.21</td>
    <td>26.30</td>
  </tr>
  <tr>
    <td>subset(25%)</td>
    <td>4</td>
    <td>5,972</td>
    <td>40.31</td>
    <td>68.03</td>
    <td>84.08</td>
    <td>26.48</td>
  </tr>
</table>

Table 9: Ablation study of Query Generation Module.

from 6 to 10;

• sampling strategy, compared fine-to-coarse and uniform approaches, fixing the first partition from 0 to 3 for an informative positive sample, while dividing the remaining partitions based on the chosen strategy. Specifically, when using the fine-to-coarse strategy, for a given \( k \) and \( m \), the length of the next interval is twice the length of the previous interval. This can be represented by the formula:
\[
L(\mathcal{P}_{i+1}) = 2L(\mathcal{P}_i);
\]

• hyperparameter \( \alpha \), for convenience, we work with its reciprocal, \( 1/\alpha \), with values of 0.1, 0.2, 0.5, 0.7, and 1.0.

Our experiments are structured as follows:

1. We fix temperature = 1 and \( k=1000 \), and conduct experiments with different values of \( m \) and sampling strategies.

2. We fix temperature = 1, \( m=10 \), and the fine-to-coarse strategy, then investigate different values of \( k \).

3. We fix \( k=500, m=10 \), and the fine-to-coarse strategy, then examine the effect of varying temperature.

Our ablation experiment results in Table 10 demonstrate that, **fine-tuned embedding model with lower alignment and higher uniformity tend to achieve better result on retrieval task**. We observe a strong correlation between retrieval performance and these two properties. Specifically, embedding models with better alignment tend to achieve superior retrieval results. Moreover, when alignment is similar, models with larger uniformity exhibit better retrieval performance. This suggests that we can leverage our strategy to adjust alignment and uniformity, ultimately optimizing retrieval performance.

E Alignment and Uniformity: Details and Discussion

In the work of Wang and Isola (2020), Alignment, which measures how well similar data points are positioned in the embedding space, is quantified by the mean Euclidean distance between the embeddings of all positive pairs. **Uniformity**, which reflects how well the data points are distributed across the embedding space, is quantified using the Gaussian potential kernel, capturing the pairwise similarity across all data points in the distribution, they are denoted as follows:

\[
\text{Alignment} = \mathbb{E}_{x,y \in pos}[\|e(x) - e(y)\|_2^2]
\]
\[
\text{Uniformity} = \log \mathbb{E}_{x,y \in p_{data}}[exp(-2\|h(x) - h(y)\|_2^2)]
\]

where \( x, y \in pos \) represents the positive pairs in the dataset, and \( p_{data} \) is the data distribution of all data points, \( e(\cdot) \) is the embedding model that maps input data points to their corresponding embeddings in a high-dimensional space. In our experiments, \( x, y \in pos \) refer to the question and its corresponding evidence chunk, while we randomly sample chunks from each document, forming a set of \( p_{data} \) to compute uniformity.

Since fine-tuning can further amend the model’s alignment (Gao et al., 2021), making it difficult to compare across different models, we introduce a scaling factor to address this. A model with high alignment does not necessarily perform worse in retrieval than one with low alignment. If a high-alignment model also ensures that negative samples are more dispersed relative to positive ones, it can still achieve strong retrieval performance. Considering this, we define the distance between the query and its nearest embedding in the database as a scaling factor for alignment. In the following experiments, we use the normalized version of alignment, which denotes as follows:

\[
\text{Alignment}_{norm} = \mathbb{E}_{x,y \in pos} \left[ \frac{\|e(x) - e(y)\|_2^2}{\|e(x) - e(y_{\text{nearest}})\|_2^2} \right]
\]

, where \( e(y_{\text{nearest}}) \) refers to the closest embedding in the database to the question embedding \( e(x) \).
<table>
  <tr>
    <th>Method</th>
    <th>Alignment</th>
    <th>Uniformity</th>
    <th>Hit@1</th>
    <th>Hit@4</th>
    <th>Hit@10</th>
    <th>MAP@10</th>
  </tr>
  <tr>
    <td>Base</td>
    <td>1.2422</td>
    <td>2.7624</td>
    <td>33.97</td>
    <td>59.69</td>
    <td>76.50</td>
    <td>22.22</td>
  </tr>
  <tr>
    <td>m=6 k=1000 fine-to-coarse</td>
    <td>1.2031</td>
    <td>3.1258</td>
    <td>38.94</td>
    <td>67.45</td>
    <td>82.44</td>
    <td>25.94</td>
  </tr>
  <tr>
    <td>m=7 k=1000 fine-to-coarse</td>
    <td>1.1953</td>
    <td>3.1907</td>
    <td>41.02</td>
    <td>69.00</td>
    <td>83.99</td>
    <td>26.76</td>
  </tr>
  <tr>
    <td>m=8 k=1000 fine-to-coarse</td>
    <td>1.1953</td>
    <td>3.3276</td>
    <td>39.38</td>
    <td>68.91</td>
    <td>83.33</td>
    <td>26.29</td>
  </tr>
  <tr>
    <td>m=9 k=1000 fine-to-coarse</td>
    <td>1.2031</td>
    <td>3.3266</td>
    <td>40.58</td>
    <td>68.34</td>
    <td>83.06</td>
    <td>26.54</td>
  </tr>
  <tr>
    <td>m=10 k=1000 fine-to-coarse</td>
    <td>1.2031</td>
    <td>3.3267</td>
    <td>40.04</td>
    <td>68.43</td>
    <td>83.55</td>
    <td>26.43</td>
  </tr>
  <tr>
    <td>m=6 k=1000 uniform</td>
    <td>1.2734</td>
    <td>3.6012</td>
    <td>36.98</td>
    <td>64.79</td>
    <td>80.27</td>
    <td>24.41</td>
  </tr>
  <tr>
    <td>m=7 k=1000 uniform</td>
    <td>1.2656</td>
    <td>3.5860</td>
    <td>36.76</td>
    <td>65.19</td>
    <td>81.37</td>
    <td>24.79</td>
  </tr>
  <tr>
    <td>m=8 k=1000 uniform</td>
    <td>1.2578</td>
    <td>3.6276</td>
    <td>38.67</td>
    <td>67.49</td>
    <td>82.35</td>
    <td>25.61</td>
  </tr>
  <tr>
    <td>m=9 k=1000 uniform</td>
    <td>1.2578</td>
    <td>3.6222</td>
    <td>38.18</td>
    <td>65.90</td>
    <td>81.46</td>
    <td>25.24</td>
  </tr>
  <tr>
    <td>m=10 k=1000 uniform</td>
    <td>1.2734</td>
    <td>3.6265</td>
    <td>36.50</td>
    <td>64.26</td>
    <td>80.71</td>
    <td>24.39</td>
  </tr>
  <tr>
    <td>k=1000 unifom m=10</td>
    <td>1.2734</td>
    <td>3.6265</td>
    <td>36.50</td>
    <td>64.26</td>
    <td>80.71</td>
    <td>24.39</td>
  </tr>
  <tr>
    <td>k=500 unifom m=10</td>
    <td>1.2578</td>
    <td>3.6303</td>
    <td>36.76</td>
    <td>65.45</td>
    <td>81.46</td>
    <td>24.72</td>
  </tr>
  <tr>
    <td>k=200 unifom m=10</td>
    <td>1.2422</td>
    <td>3.6452</td>
    <td>37.69</td>
    <td>66.39</td>
    <td>82.97</td>
    <td>25.23</td>
  </tr>
  <tr>
    <td>k=1000 fine-to-coarse m=10</td>
    <td>1.2031</td>
    <td>3.3267</td>
    <td>40.04</td>
    <td>68.43</td>
    <td>83.55</td>
    <td>26.43</td>
  </tr>
  <tr>
    <td>k=500 fine-to-coarse m=10</td>
    <td>1.1953</td>
    <td>3.3675</td>
    <td>40.71</td>
    <td>68.74</td>
    <td>83.50</td>
    <td>26.67</td>
  </tr>
  <tr>
    <td>k=200 fine-to-coarse m=10</td>
    <td>1.1953</td>
    <td>3.3896</td>
    <td>38.85</td>
    <td>68.65</td>
    <td>83.10</td>
    <td>26.11</td>
  </tr>
  <tr>
    <td>1/α=0.1 k=1000 fine-to-coarse m=10</td>
    <td>1.1953</td>
    <td>2.1774</td>
    <td>35.48</td>
    <td>63.02</td>
    <td>78.14</td>
    <td>23.96</td>
  </tr>
  <tr>
    <td>1/α=0.2 k=1000 fine-to-coarse m=10</td>
    <td>1.1875</td>
    <td>2.6560</td>
    <td>37.83</td>
    <td>66.43</td>
    <td>81.46</td>
    <td>25.47</td>
  </tr>
  <tr>
    <td>1/α=0.5 k=1000 fine-to-coarse m=10</td>
    <td>1.1875</td>
    <td>3.2849</td>
    <td>40.09</td>
    <td>67.63</td>
    <td>82.88</td>
    <td>26.34</td>
  </tr>
  <tr>
    <td>1/α=0.7 k=1000 fine-to-coarse m=10</td>
    <td>1.1953</td>
    <td>3.3411</td>
    <td>39.96</td>
    <td>68.29</td>
    <td>83.10</td>
    <td>26.45</td>
  </tr>
  <tr>
    <td>1/α=1.0 k=1000 fine-to-coarse m=10</td>
    <td>1.1953</td>
    <td>3.3675</td>
    <td>40.71</td>
    <td>68.74</td>
    <td>83.50</td>
    <td>26.67</td>
  </tr>
</table>

Table 10: Ablation study of Relevant Sampling Module.

Finally, the original uniformity is a negative value, in our experiments, we report the absolute value of uniformity. This makes comparison and analysis easier, and a larger absolute value indicates that the embedding model distribution is more uniform.

F Prompts Used for Query Perturbation

The LLM prompts used in the keywords masking experiments are detailed as follows:

Prompt for Extracting Keywords:
Given a query and a paragraph including the answer of the query, please extract all the common keywords that query and paragraph both have:
Note:
1. The definition of keywords is: words in the query and paragraph that are particularly distinctive and related to the main topic. Less important pronouns or frequently occurring words do not fall into this category.
2. The words you extract must appear in both the query and the paragraph.
3. Do not output other format, just list all the words as follows:
investigation, Eastwood, Filing
Query:
{query}
Paragraph:
{paragraph}
keywords:

Prompt for Generating Synonyms:
Given a query and a set of its keywords, generate substituted words or phrases for these keywords that preserve the original semantic meaning of the query.
Note:
1. Ensure the number of keywords remains unchanged, with one substitution for each keyword. Maintain the query’s intent, context, and grammatical correctness.
2. Avoid altering the overall structure and purpose of the query.
3. Return the substituted keywords in the same format with Keywords like: investigation, Eastwood, Filing
Query:
{query}
Keywords:
{keywords}
Your substituted keywords:

Reviewer paper summaries: - The authors propose an unsupervised embedding model tuning methods using LLMs as well as the private corpus.
- More specifically, the authors utilize both LLMs as well as BM25 to generate synthetic queries and their rankings. Then, using listwise ranking loss to train the embedding models.
- In experiments, the authors provide promising results where the fine-tuning methods show consistent improvements over zero-shot baselines as well as naive in-batch training methods.

This paper introduces BMEmbed, a novel method for adapting general-purpose text embedding models to private datasets, which often contain specialized terminology and domain-specific language. The authors address the challenge of fine-tuning embedding models for domain-specific retrieval tasks by leveraging BM25, a well-established keyword-based retrieval technique, to generate supervisory signals. The framework consists of three main steps: (1) domain query generation using LLMs, (2) relevant sampling via BM25 to construct ranked retrieval results, and (3) listwise fine-tuning to optimize the embedding model. Experiments across multiple domains and datasets demonstrate that BMEmbed consistently improves retrieval performance compared to baseline methods, including contrastive learning and hybrid retrieval approaches. The paper also provides empirical insights into how BM25-based signals enhance embedding properties like alignment and uniformity, contributing to better retrieval outcomes.
Summary of strengths: - The proposed method does not utilize any labeled data to do embedding model customization unlike other related works which need some labeled data. Here, the source of gains comes from LLMs internal knowledge as well as the BM25.
- The experimental results show that the proposed method achieves better results compared with zero-shot approach as well as naive fine-tuning methods (with LLM generated synthetic samples).

1. Novel Methodology: The paper introduces a practical and innovative approach (BMEmbed) to adapt general-purpose embedding models to private datasets, addressing a critical gap in retrieval-augmented generation (RAG) systems.
2. Comprehensive Evaluation: The authors conduct extensive experiments across multiple domains and datasets, validating the effectiveness of BMEmbed against strong baselines, including hybrid retrieval methods like RRF.
3. Theoretical Insights: The paper provides a deep analysis of how BM25-based signals improve embedding properties (alignment and uniformity), offering valuable theoretical grounding for the observed performance gains.
Summary of weaknesses: 1. Fine-tuning on closed models
- In many scenarios, closed embedding models (such as OpenAI or Gemini embedding models) perform well (currently MTEB 1st model is Gemini embedding model).
- However, this fine-tuning method cannot be applicable with those closed models.
- In that case, maybe using RRF with a better closed embedding model + BM25 would be better. It would be good to compare this setting as well.

2. Hyper-parameter tuning
- As we know, in this setting, we do not have ground truth query-corpus pairs and rely on the synthetic query corpus pairs.
- In that point of view, how can we do robust hyper-parameter tuning?
- It seems like the performance is somewhat sensitive on the hyper-parameters. Could we have the ablation study for all three datasets? Then, we can evaluate whether the hyper-parameter tuning is significantly important or not with this method.

3. Experimental details
- Why do we use different chunk sizes, k, m in Table 1? Also, how does the number of synthesized queries determine? (The number of extracted events?)
- Why do we use Hit@4? (why not MAP@4?) Can we also use nDCG@k types of metrics?
- Why do you think BMEmbed shows worse results with Legal Bench-RAG data (especially in Table 3)?

1. Dependence on BM25: The method relies heavily on BM25 for supervisory signals, which may inherit limitations of keyword-based retrieval (e.g., handling semantic but lexically divergent queries). 
2. The impact of BM25's weaknesses on BMEmbed could be discussed further.
3. Scalability: The computational cost of fine-tuning large models is non-trivial. A discussion on scalability or efficiency optimizations (e.g., for smaller models) would be beneficial.
Comments, suggestions and typos: See above

**Comments after rebuttal**

Thanks a lot for providing some clarifications to my comments.
For the private data point, in the current manuscript, the authors only highlight the challenges of private data for special jargons (e.g., domain-specific, company-specific). Therefore, the authors' claims in the rebuttal may not be convincing for me to not including the experiments with the best embedding model without tuning.
Also, for the hyper-parameters, due to the unsupervised setting, if the performance is highly sensitive to the hyper-parameter selection, the practicality of the unsupervised tuning is significantly reduced.
In those points, I will stand my original score.

1. Consider discussing how BMEmbed handles semantically similar but lexically divergent queries (e.g., synonyms), and whether augmenting BM25 with lightweight semantic signals could improve robustness.
2. Consider providing clearer analysis of trade-offs between performance gains and fine-tuning costs.
3. Briefly evaluate adapted embeddings on clustering or semantic similarity tasks to demonstrate broader utility.
Overall assessment scores: 2.5

3
Metareview score: 3.5
Metareview: This paper introduced BMEmbed, a method that adapts text embedding models to domain-specific datasets by using BM25 to generate synthetic signals. The reviewers acknowledge the innovation and usefulness of the approach for RAG on specialized domains (e.g. legal / finance).
Summary of reasons to publish: - The integration of BM25 signals as listwise fine-tuning is an interesting technique
- Compared against various baselines including contrastive learning and hybrid methods
- The analysis of alignment and uniformity provides insights to the model's effectivenss