Do We Need Domain-Specific Embedding Models? An Empirical Investigation

## Abstract

Abstract Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts?
In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB’s higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns. Moreover, we find that the performance of general-purpose embedding models on MTEB is not correlated with their performance on FinMTEB, indicating the need for domain-specific embedding benchmarks for domain-specific embedding models. This study sheds light on developing domain-specific embedding models in the LLM era. FinMTEB comes with open-source code at https://github.com/yixuantt/FinMTEB

## 1 Introduction

Embedding models, which transform text sequences into dense vector representations, play a crucial role in various natural language processing (NLP) tasks (Mikolov et al., ; Pennington et al., ; Peters et al., ). The quality of text embeddings directly impacts the effectiveness of information retrieval, semantic understanding, and other downstream applications. Recently, many state-of-the-art embedding models have been developed using large language models (LLMs) as the foundational model (Wang et al., ; Li et al., ; Meng et al., ).
Since LLMs are trained on massive text corpora spanning nearly every domain, these LLM-based embedding models have demonstrated superior and robust performance in general-purpose embedding benchmarks such as Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., ).

Given that general-purpose embedding models are becoming the backbone of NLP applications, and companies like OpenAI and Cohere are offering general-purpose embeddings that potentially serve a wide range of industry applications, a critical question arises: Do we still need domain-specific embedding models? The answer is not immediately clear. On the one hand, as mentioned earlier, state-of-the-art embedding models are primarily built from general-purpose LLMs that have been trained on vast text corpora covering nearly every domain. There is no strong evidence suggesting that these models cannot grasp domain-specific languages or linguistic patterns. On the other hand, while there has been limited development of domain-specific embedding models, researchers have advocated for training domain-specific LLMs (Gururangan et al., ) to better capture domain-specific terminology and semantics. For example, domain-specific LLMs such as BioMedLM (Bolton et al., ) for the biomedical domain, SaulLM-7B (Colombo et al., ) for the legal domain, and BloombergGPT (Wu et al., ) for the finance domain are pre-trained on large, domain specialized corpora.

To address this question, we empirically evaluate the necessity of domain-specific embedding models, focusing on the finance domain as our research context. We select the finance domain because financial NLP is a critical area within the research community, with a wealth of established financial NLP datasets (FiQA, ; Islam et al., ; Liu et al., ; Ju et al., ; Sinha & Khandait, ; Mukherjee et al., ; Malo et al., ). The importance of representing financial texts in downstream applications has also driven the development of finance-specific LLMs, such as BloombergGPT (Wu et al., ) and InvestLM (Yang et al., ). Moreover, the complexity and specificity of the financial domain provide a unique opportunity to assess how effectively general-purpose embedding models can represent specialized texts.

We first develop the Finance Massive Text Embedding Benchmark (FinMTEB), a finance-specific counterpart to MTEB. FinMTEB consists of 64 datasets and, like MTEB, covers seven distinct tasks, including classification, clustering, retrieval, pair-classification, reranking, and semantic textual similarity. Unlike MTEB, all datasets in FinMTEB are based on financial text data, which feature substantially longer text sequences and token lengths. Using seven state-of-the-art embedding models, we observe a significant performance drop on the FinMTEB compared to the general-purpose MTEB. ANOVA analysis further indicates that this average performance drop is primarily driven by the differences between the benchmarks, rather than model-specific factors.

While the performance drop on FinMTEB may seem expected given the domain shift, one concern is whether the datasets in FinMTEB are inherently more complex than those in MTEB. Is the reduced performance a result of the benchmark’s complexity, or do these models lack the necessary understanding of domain-specific context? If the FinMTEB datasets were of equal complexity to MTEB, we might not observe the same performance gap, suggesting that dataset complexity could be contributing to the performance decline.

To eliminate the confounding factor of dataset complexity, we propose four different measures to quantify complexity: ChatGPT’s response error rate, dataset readability, information entropy, and text dependency distance. Our analysis shows that even when controlling for complexity, general-purpose embedding models still perform worse on domain-specific texts. Moreover, the more complex the domain-specific data, the greater the performance drop—although this trend is less prominent in general-purpose tasks on the MTEB. Collectively, this evidence suggests that state-of-the-art, general-purpose embedding models may not fully capture the linguistic nuances and semantic patterns unique to a particular domain.

Moreover, we observe that the performance of general-purpose embedding models on MTEB does not correlate with their performance on FinMTEB. Models that perform exceptionally well on general embedding tasks do not necessarily maintain their superiority in the financial domain. This underscores the importance of evaluating embedding models within the specific context in which they will be applied and emphasizes the necessity of domain-specific embedding benchmarks.

Our contributions in this paper are threefold:

- •
Our main research contribution is the empirical investigation into the necessity of domain-specific embedding models. To the best of our knowledge, this is one of the first studies to address the critical question of whether domain-specific embeddings are required, especially given the widespread adoption of general-purpose embedding models across various industry applications.
- •
Our analysis on the necessity of domain-specific embedding models is based on a rigorous evaluation framework. Rather than simply developing a domain benchmark and demonstrating a performance drop, our analysis accounts for dataset complexity, eliminating potential confounding factors. This allows us to conclude that the performance gap is due to the models’ inability to encode domain-specific text, rather than inherent dataset complexity.
- •
The development of the FinMTEB dataset, as a byproduct of our study, may serve as a valuable resource for researchers and practitioners interested in building financial domain-specific embedding models.

## 2 Related Work

### 2.1 General-purpose Embedding Models

Embedding models like Word2Vec (Mikolov et al., ) and GloVe (Pennington et al., ) lay the groundwork by capturing word-level semantics through contextual co-occurrence. The introduction of transformer-based models such as BERT (Devlin et al., ) and RoBERTa (Liu, ) marks a significant shift by utilizing deep bidirectional encoders, enabling contextualized word embeddings. Building on these, Sentence-BERT (Reimers & Gurevych, ) is designed to generate semantically meaningful sentence embeddings using Siamese and triplet networks, improving performance on semantic similarity tasks. Recent advancements in LLMs have driven the development of LLM-based embedding models, such as e5-mistral-7b-instruct (Wang et al., ) and gte-Qwen2-1.5B-instruct (Yang et al., ), which have achieved state-of-the-art performance across a wide range of NLP tasks.

### 2.2 Domain-Specific Models

Different domains exhibit distinct linguistic patterns and terminologies, often requiring domain-specific models or adaptations for specialized tasks. Researchers have advocated for training domain-specific models or fine-tuning general models for particular domains (Gururangan et al., ). For instance, domain-specific LLMs like BioMedLM (Bolton et al., ) for biomedical text, SaulLM-7B (Colombo et al., ) for legal documents, and BloombergGPT (Wu et al., ) for financial applications are pre-trained on large domain-specific corpora. In addition, instruction-tuned domain-specific models such as InvestLM (Yang et al., ) and FinGPT (Yang et al., ) are fine-tuned for specific downstream tasks in the finance domain.

While domain-specific LLMs have been widely studied and developed, domain-specific embedding models have received relatively less attention. In the biomedical domain, models like BioWordVec (Zhang et al., ) and BioSentVec (Chen et al., ) generate word and sentence embeddings tailored to biomedical texts. In finance, FinBERT (Yang et al., ) is pre-trained on a large corpus of financial texts to enhance text encoding capabilities. However, most domain-specific embedding models are based on smaller language models instead of state-of-the-art LLMs. Since LLMs are trained on extensive data across multiple domains with numerous parameters, it remains unclear whether general-purpose LLM-based embeddings can adequately handle specialized texts. This paper aims to address this research gap.

### 2.3 Embedding Benchmarks

To comprehensively evaluate embedding models, benchmarks like the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., ) have been established. MTEB assesses embedding models across a wide array of tasks using numerous datasets and languages. This extensive evaluation provides insights into a model’s generalizability and effectiveness across different linguistic contexts and task types. Similarly, the BEIR benchmark (Thakur et al., ) focuses on the information retrieval task, encompassing 18 diverse datasets. While BEIR includes some domain-specific datasets such as FiQA (FiQA, ), it is not tailored for comprehensive domain analysis. The inclusion of a few specialized datasets does not fully address the unique challenges posed by domain-specific language and terminology. There are also scenario-specific RAG evaluation benchmarks like RAGeval (Zhu et al., ). These benchmarks acknowledge the necessity for domain-specific evaluations, particularly highlighting the impact of accurate retrieval in specialized contexts. However, they primarily focus on retrieval tasks and often overlook other crucial embedding tasks such as semantic similarity and clustering.

### 2.4 Domain-specific Model Benchmarks

Numerous benchmarks tailored to specific domains have been developed with the emergence of domain-specific large language models (LLMs). For example, in the finance domain, benchmarks such as CFLUE (Zhu et al., ), FinEval (Zhang et al., ), DocMath-Eval (Zhao et al., ), and FinanceBench (Islam et al., ) have been introduced to assess the comprehension capabilities of LLMs within financial contexts. Similarly, in the legal domain, LawBench (Fei et al., ) has been established to evaluate LLMs across a variety of legal tasks. Besides, MedBench (Liu et al., ), MedEval (He et al., ), and DrBenchmark (Labrak et al., ) have been developed to test the proficiency in understanding and generating medical information. Most of these benchmarking papers conclude that general-purpose LLMs may fall short on domain tasks (Zhu et al., ; Fei et al., ). The importance of domain adaptation has gradually gained attention (Ling et al., ). However, to our knowledge, there is little work benchmarking the embedding model’s performance on domain texts.

## 3 The FinMTEB Benchmark

In this section, we briefly introduce the proposed Finance MTEB (FinMTEB) benchmark, which serves as the foundation for our analysis. The construction of FinMTEB closely resembles the widely used general embedding benchmark, MTEB
(Muennighoff et al., ).

### 3.1 FinMTEB Tasks

Figure: Figure 1: An overview of tasks and datasets used in FinMTEB. All the dataset descriptions and examples are provided in the Appendix C.
Refer to caption: x1.png

Figure 1 provides an overview of the tasks and datasets included in FinMTEB. Similar to MTEB (Muennighoff et al., ), FinMTEB includes seven embedding tasks, but with datasets specifically tailored to the finance domain, as follows.

Semantic Textual Similarity (STS) involves assessing the semantic similarity between two sentences from the financial text. For this task, we employ datasets such as FinSTS (Liu et al., ) and FINAL (Ju et al., ) from company annual reports, along with other data types such as BQ-Corpus (Chen et al., ) sourced from the banking corpus.

Retrieval focuses on identifying the most relevant evidence in response to a query from a financial corpus. This task utilizes some popular finance QA datasets such as FinanceBench (Islam et al., ), FiQA2018 (FiQA, ) and HPC3 (Guo et al., ). These datasets pair each query with relevant contextual information. Additionally, we also develop specific queries for finance terms from various sources, such as the TradeTheEvent (Zhou et al., ), to further enhance the finance domain evaluation.

Classification involves predicting the label of a financial text based on its text embedding. The classification task includes multiple datasets, such as financial sentiment analysis (Malo et al., ; FiQA, ; Cortis et al., ; Lu et al., ), Fed’s monetary policy classification (Shah et al., ), and organization’s strategy, as well as forward-looking statement classification (Yang et al., ).

Clustering is the process of grouping sentences based on their embedding similarities. We compile a diverse and comprehensive corpus that includes consumer complaints from CFPB (^1^11https://huggingface.co/datasets/CFPB/consumer-finance-complaints), financial papers from arXiv, company industry descriptions (Qader et al., ), financial events and intent detection(Gerz et al., ).

Reranking includes a set of financial datasets that have the ranking of retrieved documents to user queries such as FinQA2018-Rerank (Chen et al., ).

Pair-Classification focuses on comparing the class label of two financial text. We use the data from AFQMC (^2^22https://tianchi.aliyun.com/dataset/106411) and finance news headline (Sinha & Khandait, ).

Summarization focuses on summarizing the main content of the financial text. The corpus used for this task includes earnings call transcripts (Mukherjee et al., ), financial news (Lu et al., ), and Form 10-K filings (El-Haj et al., ).

In summary, FinMTEB consists of a total of 64 datasets, spanning seven different tasks. The key difference between MTEB and FinMTEB is that all datasets in FinMTEB are finance-domain specific, either previously used in financial NLP research or newly developed by the authors. Semantic similarity between datasets in FinMTEB are shown in Appendix A. The detailed dataset information and descriptions are presented in the Appendix C. The main scoring metric for each task is the same as used with that of the MTEB benchmark, and the details are presented in the Appendix D.

### 3.2 Characteristics of FinMTEB

Having built the FinMTEB benchmark, we now provide an analysis to understand its characteristics.

Linguistic Pattern. Table 1 presents a comparative analysis of linguistic features such as average sentence length, token length, syllables per token, and dependency distance (Oya, ) across two benchmarks.
The results indicate that texts in FinMTEB consistently have longer and more complex sentences than those in MTEB, with an average sentence length of 26.37 tokens compared to 18.2 tokens in MTEB. This highlights significant linguistic differences between financial and general texts. However, does this difference warrant the need for a domain-specific embedding model? We will explore this question later.

**Table 1: Comparison of Text Characteristics Between FinMTEB and MTEB. The numbers represent the average scores across all samples from all datasets.**
| Benchmark | Sentence Length | Token Length | Syllables Per Token | Dependency Distance |
| --- | --- | --- | --- | --- |
| MTEB | 18.20 | 4.89 | 1.49 | 2.49 |
| FinMTEB | 26.37 | 5.12 | 1.52 | 2.85 |

High Semantic Diversity. We first examine the inter-dataset semantic similarity. Using the all-MiniLM-L6-v2 model(^3^33https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), we embed 100 samples from each dataset, compute their averages to represent the dataset embedding, and measure inter-dataset similarity using cosine similarity. As shown in Appendix A, most datasets in FinMTEB have an inter-dataset similarity score below 0.6, indicating high semantic diversity. Despite being finance-domain specific, this highlights the diverse narratives and contexts present in the financial datasets.

### 3.3 The performance of state-of-the-art embedding models on FinMTEB

General-purpose Embedding Models. We consider seven state-of-the-art, general-purpose embedding models in our experiments. Specifically, we consider the following models: bge-en-icl (Xiao et al., ) and e5-mistral-7b-instruct (Wang et al., ), which are developed from Mistral-7B-v0.1 (Jiang et al., ); gte-Qwen2-1.5B-instruct (Li et al., ), developed from Qwen2 (Yang et al., ); bge-large-en-v1.5 (Xiao et al., ) and all-MiniLM-L12-v2 (Reimers & Gurevych, ), both developed from BERT (Devlin et al., ); instructor-base (Su et al., ) from T5Encoder (Raffel et al., ); and OpenAI’s text-embedding-3-small (OpenAI, ).

We evaluate the performance of these embedding models on FinMTEB tasks, with the results presented in Table 5, alongside their performance on MTEB for comparison. The results clearly demonstrate a significant performance drop on the FinMTEB benchmarks. For instance, the best-performing model on MTEB, bge-en-icl, achieves an average score of 71.67, while its performance on FinMTEB is notably lower, at 63.09.

**Table 2: State-of-the-art Embedding Model Performance on MTEB and FinMTEB. The “MTEB Score” column represents performance on the MTEB benchmark (Muennighoff et al., ) as reported on the Hugging Face MTEB Leaderboard (^5^55https://huggingface.co/spaces/mteb/leaderboard). The “FinMTEB Score” column shows the average performance score evaluated on the proposed FinMTEB benchmark. To ensure a fair comparison, FinMTEB uses the same evaluation metrics as MTEB.**
| Embedding Model | Base Model | Embedding Dimensions | MTEB Score | FinMTEB Score |
| --- | --- | --- | --- | --- |
| bge-en-icl | Mistral | 4096 | 71.67 | 63.09 $(\downarrow-8.58)$ |
| gte-Qwen2-1.5B-instruct | Qwen2 | 1536 | 67.16 | 59.98 $(\downarrow-7.18)$ |
| e5-mistral-7b-instruct | Mistral | 4096 | 66.63 | 64.75 $(\downarrow-1.88)$ |
| bge-large-en-v1.5 | Bert | 1024 | 64.23 | 58.95 $(\downarrow-5.28)$ |
| text-embedding-3-small | - | 1536 | 62.26 | 61.36 $(\downarrow-0.90)$ |
| instructor-base | T5Encoder | 768 | 59.54 | 54.79 $(\downarrow-4.75)$ |
| all-MiniLM-L12-v2 | Bert | 384 | 56.53 | 54.31 $(\downarrow-2.22)$ |

#### 3.3.1 What Drives the Performance Gap?

Having observed a performance discrepancy between general-purpose embedding models across the two benchmarks, we aim to investigate what drives this difference. Specifically, we consider two possible factors: (1) the model used (i.e., which embedding model is applied), and (2) the domain (i.e., whether it’s the general benchmark, MTEB, or the domain-specific benchmark, FinMTEB). Since we evaluate seven embedding models across two domains, this results in 14 unique model-domain combinations.

To facilitate statistical analysis, we employ bootstrapping methods to generate a large sample dataset. For each task in both MTEB and FinMTEB, we aggregate the task’s datasets into a task pool. From each task pool, we randomly sample 50 examples to form a bootstrap sample and evaluate the embedding model’s performance on this bootstrap. We repeat this process 500 times, yielding 500 bootstraps for each combination. Thus, we have 14 unique combinations (model and domain), each with 500 bootstraps and corresponding performance scores.

**Table 3: ANOVA analysis results. The reported numbers represent the partial eta squared (effect size) for each factor (Model or Domain). Asterisks indicate statistical significance levels, with $**$ denoting $p$-value $<$ 0.05.**
|  | STS | Classification | Retrieval | Reranking | Clustering | Pair-classification | Summarization | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | 0.79** | 0.02** | 0.89** | 0.30** | 0.04** | 0.00 | 0.11** | 0.00 |
| Domain | 0.11** | 0.23** | 0.31** | 0.05** | 0.82** | 0.63** | 0.12** | 1.00** |

We present the ANOVA analysis results in Appendix B. First, the results indicate that the choice of embedding model (Model factor) significantly impacts performance in most tasks, such as STS (0.79), Retrieval (0.89), and Reranking (0.30), with the exception of Pair-classification (0.00), where model choice has no significant impact. Second, the Domain factor also shows significant effects across all embedding tasks. Interestingly, the average scores reveal that, from an overall perspective, the Model factor has little impact on performance, with an effect size of 0.00 and an insignificant p-value. This suggests that while individual models may excel at specific tasks, their performance discrepancies balance out when averaged. However, the Domain factor (1.00) demonstrates a much more prominent influence, underscoring the necessity for domain-specific models or fine-tuning when addressing specialized tasks like those in finance.

Research Question. While the performance drop on FinMTEB and the subsequent ANOVA analysis suggests that domain-specific embedding tasks may pose greater challenges for general-purpose embedding models, does this necessarily indicate a need for domain-specific models? Not necessarily. The difference in datasets between FinMTEB and MTEB could contribute to the observed performance drop. For instance, FinMTEB datasets might be inherently more difficult or linguistically complex compared to those in MTEB as illustrated in Table 1. If both benchmarks contained datasets of equivalent complexity, general-purpose models might even perform better on FinMTEB tasks. Therefore, the performance drop does not necessarily imply that the models fail to understand domain-specific language or concepts. To draw meaningful conclusions about the necessity of domain-specific models, we must first control for differences in dataset difficulty. In the next section, we will analyze model performance while considering these inherent differences between the FinMTEB and MTEB datasets.

## 4 Performance Analysis After Controlling for Dataset Complexity

To answer the above research question, we conduct a detailed analysis of the embedding models’ performance, while accounting for dataset complexity.

### 4.1 Quantifying Dataset Complexity

We propose four different measures to quantify a dataset’s complexity.

ChatGPT Error Rate. The first measure quantifies how challenging it is for ChatGPT to answer a dataset’s questions. Specifically, for each example in the dataset across different tasks, we reformat the example into a question-and-answer pair, as shown in Appendix F, and prompt GPT-4o mini (OpenAI, ). The rationale is that if ChatGPT fails to answer a question correctly, it indicates the difficulty level of the question. Additionally, since state-of-the-art LLM-based embedding models present each query with an instruction in a question-answer format, we use the ChatGPT error rate as an indicator of dataset complexity.

Information Theory. We borrow the concept of information entropy from information theory to measure the complexity of a text sequence. Information entropy is calculated as the average amount of information produced by a stochastic source of data. Higher Information Entropy indicates text that contains more information or is less predictable, potentially implying greater complexity.

Readability. We also use readability to measure dataset complexity, specifically applying the Gunning Fog Index (Gunning, ), which factors in sentence length and the number of complex words. The index estimates the years of formal education required to understand a text on the first reading. A higher Gunning Fog Index score indicates more complex sentences.

Mean Dependency Distance. Finally, we measure linguistic complexity using the dependency distance between two syntactically related words in a sentence (Oya, ). A longer dependency distance indicates that more context is needed for comprehension, reflecting greater sentence complexity.

For all of these four complexity measures, a higher score indicates higher dataset complexity. Details on the measures and their calculations are provided in Appendix E.

### 4.2 Subgroup analysis: Embedding Performance vs. Dataset Complexity

Figure: (a) ChatGPT Error Rate
Refer to caption: extracted/5897514/illustration/box2/openai.png

We conduct a subgroup analysis to examine the impact of the domain on embedding model performance. First, we calculate dataset complexity using one of the four complexity measures and categorize the datasets into three subgroups: low, medium, and high complexity. This ensures that METB and FinMTEB datasets within each subgroup have the same level of complexity. We then calculate the average performance score of seven LLM-based embedding models across datasets within each subgroup. The results of this analysis appear in Figure 2. The subgroup analysis reveals two key findings.

First, embedding models perform substantially worse on FinMTEB datasets compared to MTEB datasets, even after accounting for dataset complexity. The performance of embedding models on FinMTEB datasets is consistently lower than on MTEB datasets within the same group. This performance gap persists regardless of the complexity measure used. Given that datasets in the same subgroup have similar levels of complexity (e.g., comparable ChatGPT error rates), the lower performance on FinMTEB tasks suggests that state-of-the-art LLM-based embedding models struggle with encoding domain-specific terminologies and semantic patterns.

Second, embedding models perform worst on FinMTEB datasets with the highest complexity levels. For example, using readability as a complexity measure, the average performance of embedding models on high-complexity FinMTEB datasets is around 0.3, significantly lower than their performance on low-complexity datasets (around 0.5) and medium-complexity datasets (around 0.6). This further highlights the challenge embedding models face in capturing complex, domain-specific language and semantics.

### 4.3 Regression Analysis: Embedding Performance vs. Dataset Complexity

**Table 4: Regression analysis results. The numbers represent the estimated coefficient values, with standard errors in parentheses. $**$ denotes significance at the $p<0.05$ level.**
|  | ChatGPT Error Rate | Readability | Information Theory | Mean Dependency Distance |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Domain | MTEB | FinMTEB | MTEB | FinMTEB | MTEB | FinMTEB | MTEB | FinMTEB |
| Intercept | 0.6619** | 0.6097** | 0.8739** | 0.6976** | 0.5905** | 0.4766** | 1.2341** | 0.9357** |
|  | (0.003) | (0.003) | (0.002) | (0.005) | (0.002) | (0.002) | (0.007) | (0.008) |
| Complexity | -0.2193** | -0.4637** | -0.0278** | -0.0202** | 0.0168** | -0.0092** | -0.2703** | -0.1810** |
|  | (0.009) | (0.010) | (0.000) | (0.000) | (0.001) | (0.001) | (0.003) | (0.003) |

Furthermore, we conduct a regression analysis to examine the relationship between dataset complexity and embedding model performance. Specifically, for each dataset complexity measure, we run two ordinary least squares (OLS) regression models—one for the MTEB datasets and one for the FinMTEB datasets. We normalize the variables Performance and Complexity to a range between 0 and 1 using min-max normalization. The regression specification is as follows:

$$ $\text{Performance}=\text{Intercept}+\beta\times\text{Complexity}+\epsilon$ (1) $$

where $\beta$ is the coefficients of the covariate (i.e., dataset complexity), and $\epsilon$ is the error term.

The regression results are presented in Table 4 and are largely consistent with the findings from the subgroup analysis. First, there is a negative relationship between dataset complexity and embedding model performance, indicating that models significantly struggle with domain-specific texts exhibiting higher linguistic complexity. Second, the intercept for the MTEB datasets is consistently higher than that for FinMTEB. Given that the Complexity variable is normalized between 0 and 1, these results suggest a significant performance gap between embedding models on MTEB and FinMTEB, even when controlling for the same level of dataset complexity.

Overall, both the subgroup and regression analyses demonstrate that the performance drop reported in Table 2 is not driven by differences in dataset complexity between MTEB and FinMTEB benchmarks. Rather, it suggests that state-of-the-art, general-purpose embedding models may not fully capture the linguistic nuances and semantic patterns specific to a given domain.

## 5 Domain-specific Embedding Benchmark is needed

Another key consideration when discussing domain-specific embeddings is whether we need a domain-specific embedding benchmark. While it may seem intuitive to say yes, there is little empirical evidence supporting this assumption. To explore this question, we evaluate the performance ranking of embedding models on both the general MTEB and FinMTEB datasets, calculating Spearman’s rank correlation between the two. The results, shown in Table 5, indicate that the ranking correlation is not statistically significant (p-values all greater than 0.05). In other words, a general-purpose embedding model performing well on MTEB does not necessarily perform well on domain-specific tasks. This suggests the necessity of developing domain-specific embedding benchmarks for evaluating domain-specific embeddings. Therefore, the development of FinMTEB constitutes a significant contribution to benchmarking embedding models specifically for the financial domain.

**Table 5: Spearman’s correlation of embedding models’ performance on MTEB and FinMTEB across different tasks. The p-value indicates that all correlations are statistically insignificant, suggesting no relationship between embedding model performance on the two benchmarks.**
|  | STS | Classification | Retrieval | Reranking | Clustering | Pair-classification | Summarization |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Correlation | 0.30 | -0.80 | 0.30 | -0.10 | -0.70 | -0.30 | 0.60 |
| p-value | 0.62 | 0.10 | 0.62 | 0.87 | 0.18 | 0.62 | 0.28 |

## 6 Conclusion

In this study, we empirically investigate a seemingly intuitive yet practically important question: do we need domain-specific embedding models? To rigorously address this, we use the finance domain as an example and develop the FinMTEB benchmark, which comprises a large variety of domain-specific (i.e., finance) embedding tasks. Additionally, we propose four methods to quantify dataset complexity. Our comparative analysis reveals that state-of-the-art LLM-based embedding models exhibit a substantial performance gap between general (MTEB) and domain-specific (FinMTEB) benchmarks. More importantly, this gap persists even when accounting for dataset complexity.
The empirical results provide strong evidence that, despite being trained on vast amounts of data that likely include various domains, LLM-based embedding models still fall short in capturing domain-specific linguistic and semantic patterns. Given the widespread use of embedding models in information retrieval and semantic search applications, this highlights the need for further adaptation of these models to specific domains in order to improve their utility. Moreover, the development of the FinMTEB benchmark can serve as a valuable resource for researchers and practitioners interested in financial-specific embedding models.

While this study presents compelling evidence for the necessity of domain-specific embedding models, the challenge of how to train these models remains. Should we adapt domain-specific embeddings from a domain-specific LLM, or should we develop domain-specific datasets and fine-tune a general-purpose LLM? We hope to see more research in this direction to further advance AI’s capabilities in handling domain-specific tasks effectively.

## Appendix A Dataset Semantic Similarity

Figure 3 presents the semantic similarity across all datasets in the Finance MTEB benchmark. The semantic similarity is calculated by cosine similarity.

Figure: Figure 3: Semantic similarity across all the datasets in FinMTEB benchmark.
Refer to caption: x2.png

## Appendix B ANOVA DATA

Table 6 illustrates the full results of ANOVA analysis.

**Table 6: Analysis of Variance (ANOVA) Results Across Tasks and Factors. Factor represents the independent variables analyzed: Model Factor pertains to variations attributed to different models, and Domain Factor pertains to variations due to different domains (MTEB or FinMTEB). Residual refers to the unexplained variance. The Sum of Squares, Degrees of Freedom, F-Statistic, and p-value are presented for each factor within each task. Asterisks denote significance levels, with lower p-values indicating higher statistical significance. The Domain Factor consistently shows high significance across all tasks.**
| Task | Factor | Sum of Squares | Degrees of Freedom | F-Statistic | p-value |
| --- | --- | --- | --- | --- | --- |
| Classification | Model Factor | 4.17 | 6.00 | 25.55 | $3.41\times 10^{-30}$ |
| Domain Factor | 56.82 | 1.00 | 2086.30 | $\approx 0$ |  |
| Residual | 190.42 | 6992.00 | NA | NA |  |
| Retrieval | Model Factor | 104.25 | 6.00 | 9052.57 | $\approx 0$ |
| Domain Factor | 6.16 | 1.00 | 3207.72 | $\approx 0$ |  |
| Residual | 13.42 | 6992.00 | NA | NA |  |
| STS | Model Factor | 10.55 | 6.00 | 149.00 | $1.64\times 10^{-178}$ |
| Domain Factor | 304.09 | 1.00 | 25761.71 | $\approx 0$ |  |
| Residual | 82.53 | 6992.00 | NA | NA |  |
| Clustering | Model Factor | 0.29 | 6.00 | 47.60 | $1.59\times 10^{-57}$ |
| Domain Factor | 32.25 | 1.00 | 32161.37 | $\approx 0$ |  |
| Residual | 7.01 | 6992.00 | NA | NA |  |
| Summarization | Model Factor | 12.98 | 6.00 | 145.31 | $2.90\times 10^{-174}$ |
| Domain Factor | 14.49 | 1.00 | 973.32 | $3.60\times 10^{-200}$ |  |
| Residual | 104.07 | 6992.00 | NA | NA |  |
| Reranking | Model Factor | 5.38 | 6.00 | 489.05 | $\approx 0$ |
| Domain Factor | 0.64 | 1.00 | 346.78 | $1.39\times 10^{-75}$ |  |
| Residual | 12.84 | 7002.00 | NA | NA |  |
| Pair Classification | Model Factor | 0.25 | 6.00 | 1.97 | 0.07 |
| Domain Factor | 249.19 | 1.00 | 11989.92 | $\approx 0$ |  |
| Residual | 145.31 | 6992.00 | NA | NA |  |
| Average | Model Factor | 0.00 | 6.00 | 1.34 | 0.37 |
|  | Domain Factor | 0.08 | 1.00 | 253.87 | $\approx 0$ |
|  | Residual | 0.00 | 6.00 | NA | NA |

## Appendix C Datasets

The detailed description of each dataset used in this work is listed in the Table tables 9, 12, 10, 11, 7, 13 and 8.

**Table 7: Summary of STS Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| FINAL (Ju et al., ) | English | A dataset designed for discovering financial signals in narrative financial reports. |
| FinSTS (Liu et al., ) | English | A dataset focused on detecting subtle semantic shifts in financial narratives. |
| AFQMC (^6^66[https://tianchi.aliyun.com/dataset/106411](https://tianchi.aliyun.com/dataset/106411)) | Chinese | A Chinese dataset for customer service question matching in the financial domain. |
| BQ-Corpus (Chen et al., ) | Chinese | A large-scale Chinese corpus for sentence semantic equivalence identification (SSEI) in the banking domain. |

**Table 8: Summary of Retrieval Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| FiQA2018 (FiQA, ) | English | Financial opinion mining and question answering dataset. |
| FinanceBench (Islam et al., ) | English | Open book financial question answering dataset. |
| HC3(Finance) (Guo et al., ) | English | A human-ChatGPT comparison corpus in the finance domain. |
| Apple-10K-2022 (^7^77[https://lighthouz.ai/blog/rag-benchmark-finance-apple-10K-2022/](https://lighthouz.ai/blog/rag-benchmark-finance-apple-10K-2022/)) | English | A retrieval-augmented generation (RAG) benchmark for finance applications. |
| FinQA (Chen et al., ) | English | Financial numerical reasoning dataset with structured and unstructured evidence. |
| TAT-QA (Zhu et al., ) | English | Question answering benchmark combining tabular and textual content in finance. |
| US Financial News (^8^88[https://www.kaggle.com/datasets/jeet2016/us-financial-news-articles](https://www.kaggle.com/datasets/jeet2016/us-financial-news-articles)) | English | Finance news articles paired with headlines and stock ticker symbols. |
| TradeTheEvent (Trading Benchmark) (Zhou et al., ) | English | Finance news articles paired with headlines and stock ticker symbols. |
| TradeTheEvent (Domain Adaption) (Zhou et al., ) | English | Financial terms and explanations dataset. |
| TheGoldman-en | English | English version of the Goldman Sachs Financial Dictionary. |
| FinTruthQA (Xu et al., ) | Chinese | Dataset for evaluating the quality of financial information disclosure. |
| Fin-Eva (Retrieval task) (^9^99[https://github.com/alipay/financial_evaluation_dataset/tree/main](https://github.com/alipay/financial_evaluation_dataset/tree/main)) | Chinese | Financial scenario QA dataset focusing on retrieval tasks. |
| AlphaFin (Li et al., ) | Chinese | Comprehensive financial dataset including NLI, QA, and stock trend predictions. |
| DISC-FinLLM (Retrieval Part Data) (Chen et al., ) | Chinese | Financial scenario QA dataset. |
| FinQA (from DuEE-fin) (Lu et al., ) | Chinese | Financial news bulletin event quiz dataset. |
| DISC-FinLLM (Computing) (Chen et al., ) | Chinese | Financial scenario QA dataset focusing on numerical tasks. |
| SmoothNLP (^10^1010[https://github.com/smoothnlp/SmoothNLP](https://github.com/smoothnlp/SmoothNLP)) | Chinese | Chinese finance news dataset. |
| THUCNews (Sun et al., ) | Chinese | Chinese finance news dataset. |
| Fin-Eva (Terminology) (^11^1111[https://github.com/alipay/financial_evaluation_dataset/tree/main](https://github.com/alipay/financial_evaluation_dataset/tree/main)) | Chinese | Financial terminology dataset used in the industry. |
| TheGoldman-cn | Chinese | Chinese version of the Goldman Sachs Financial Dictionary. |

**Table 9: Summary of Classification Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| FinancialPhrasebank (Malo et al., ) | English | Polar sentiment dataset of sentences from financial news, categorized by sentiment into positive, negative, or neutral. |
| FinSent (Yang et al., ) | English | Polar sentiment dataset of sentences from the financial domain, categorized by sentiment into positive, negative, or neutral. |
| FiQA_ABSA (FiQA, ) | English | Polar sentiment dataset of sentences from the financial domain, categorized by sentiment into positive, negative, or neutral. |
| SemEva2017_Headline (Cortis et al., ) | English | Polar sentiment dataset of sentences from the financial domain, categorized by sentiment into positive, negative, or neutral. |
| FLS (Yang et al., ) | English | A finance dataset detects whether the sentence is a forward-looking statement. |
| ESG (Yang et al., ) | English | A finance dataset performs sentence classification under the environmental, social, and corporate governance (ESG) framework. |
| FOMC (Shah et al., ) | English | A task of hawkish-dovish classification in finance domain. |
| Financial-Fraud (^12^1212[https://github.com/amitkedia007/Financial-Fraud-Detection-Using-LLMs/tree/main](https://github.com/amitkedia007/Financial-Fraud-Detection-Using-LLMs/tree/main)) | English | This dataset was used for research in detecting financial fraud. |
| FinNSP (Lu et al., ) | Chinese | Financial negative news and its subject determination dataset. |
| FinChina (Lan et al., ) | Chinese | Polar sentiment dataset of sentences from the financial domain, categorized by sentiment into positive, negative, or neutral. |
| FinFE (Lu et al., ) | Chinese | Financial social media text sentiment categorization dataset. |
| OpenFinData (^13^1313[https://github.com/open-compass/OpenFinData?tab=readme-ov-file](https://github.com/open-compass/OpenFinData?tab=readme-ov-file)) | Chinese | Financial scenario QA dataset including sentiment task. |
| MDFEND-Weibo2 (finance) (Nan et al., ) | Chinese | Fake news detection in the finance domain. |

**Table 10: Summary of Clustering Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| MInDS-14-en (Gerz et al., ) | English | MINDS-14 is a dataset for intent detection in e-banking, covering 14 intents across 14 languages. |
| Consumer Complaints (CFPB, ) | English | The Consumer Complaint Database is a collection of complaints about consumer financial products and services that sent to companies for response. |
| Synthetic PII finance (Watson et al., ) | English | Synthetic financial documents containing Personally Identifiable Information (PII). |
| FinanceArxiv-s2s (^14^1414Collect from the Arixv) | English | Clustering of titles from arxiv (q-fin). |
| FinanceArxiv-p2p | English | Clustering of abstract from arxiv (q-fin). |
| WikiCompany2Industry-en (^15^1515Collect from the Wikipedia) | English | Clustering the related industry domain according to the company description. |
| MInDS-14-zh (Gerz et al., ) | Chinese | MINDS-14 is a dataset for intent detection in e-banking, covering 14 intents across 14 languages. |
| FinNL (Lu et al., ) | Chinese | Financial news categorization dataset. |
| CCKS2022 (CCKS, ) | Chinese | Clustering of financial events. |
| CCKS2020 | Chinese | Clustering of financial events. |
| CCKS2019 | Chinese | Clustering of financial events. |

**Table 11: Summary of Summarization Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| Ectsum (Mukherjee et al., ) | English | A Dataset For Bullet Point Summarization of Long Earnings Call Transcripts. |
| FINDSum (Liu et al., ) | English | A Large-Scale Dataset for Long Text and Multi-Table Summarization. |
| FNS-2022 (El-Haj et al., ) | English | Financial Narrative Summarisation for 10K. |
| FiNNA (Lu et al., ) | Chinese | A financial news summarization dataset. |
| Fin-Eva (Headline) | Chinese | A financial summarization dataset. |
| Fin-Eva (Abstract) | Chinese | A financial summarization dataset. |

**Table 12: Summary of Reranking Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| Fin-Fact | English | A Benchmark Dataset for Financial Fact Checking and Explanation Generation. |
| FiQA2018 | English | Financial opinion mining and question answering. |
| HC3(Finance) | English | A human-ChatGPT comparison finance corpus. |
| Fin-Eva (Retrieval task) | Chinese | Financial scenario QA dataset including retrieval task. |
| DISC-FinLLM (Retrieval Part Data) | Chinese | Financial scenario QA dataset. |

**Table 13: Summary of PairClassification Datasets**
| Dataset Name | Language | Description |
| --- | --- | --- |
| HeadlineAC-PairClassification (Sinha & Khandait, ) | English | Financial text sentiment categorization dataset. |
| HeadlinePDD-PairClassification (Sinha & Khandait, ) | English | Financial text sentiment categorization dataset. |
| HeadlinePDU-PairClassification (Sinha & Khandait, ) | English | Financial text sentiment categorization dataset. |
| AFQMC | Chinese | Ant Financial Question Matching Corpus. |

## Appendix D Main Metric

Semantic Textual Similarity (STS): The main metric used to measure performance in this task is Spearman’s rank correlation of predicted cosine similarity scores with the true similarity score.

Classification: The main metric for evaluating is accuracy, ensuring that the model’s assessment is based on different types of financial texts and frameworks.

Clustering: The main evaluation metric for this task is the V-measure (Rosenberg & Hirschberg, ), which assesses the quality of the clusters by examining both the completeness and the homogeneity of the data within each group.

Rerank: The main evaluation metric for reranking in Finance MTEB is Mean Average Precision (MAP).

Pair-Classification: The main evaluation metric for Pair-Classification is Average Precision (AP), which measures the model’s accuracy across various decision thresholds.

Summarization: Summarization is evaluated based on the correlation between dense embeddings derived from the summarized texts and those of the original texts, utilizing Spearman’s correlation coefficient as the main metric.

Retrieval: The main evaluation metric employed in this task is NDCG@10, which assesses the quality of the results based on their relevance and position in the list returned.

## Appendix E Complexity Score Calculation

### E.1 Shannon Entropy in Information Theory

The Shannon entropy is a measure from information theory that quantifies the average level of uncertainty or information content inherent in a set of possible outcomes. A higher Shannon entropy means higher uncertainty. To calculate the Shannon entropy $H$ of a text, we follow these steps:

- 1.
Count Tokens: Identify all unique tokens $w_{i}$ in the text and count their occurrences $n_{i}$.
- 2.
Calculate Probabilities: Compute the probability of each token $P(w_{i})$ by dividing its count by the total number of tokens $N$:
$P(w_{i})=\frac{n_{i}}{N},\quad\text{where}\quad N=\sum_{i=1}^{M}n_{i}$
Here, $M$ is the total number of unique tokens.
- 3.
Compute Shannon Entropy: Use the probabilities to calculate the entropy, and sum up all unique tokens in the text.
$H=-\sum_{i=1}^{M}P(w_{i})\log_{2}P(w_{i})$

### E.2 Readability: Gunning Fog Index

The Gunning Fog Index (Gunning, ) is a readability metric that estimates the years of formal education needed to understand a text upon first reading, it evaluates the complexity of English prose based on sentence length and the frequency of complex words. To calculate the Gunning Fog Index (GFI), follow these steps:

- 1.
Select a Representative Passage
Choose at least 1000 words from the text that represents the overall writing style.
- 2.
Calculate the Average Sentence Length (ASL)
$\text{ASL}=\frac{\text{Total Number of Words}}{\text{Total Number of Sentences}}$
(2)
- 3.
Identify Complex Words
•
Complex words are words with three or more syllables.
•
Exclude proper nouns, familiar jargon, compound words, and verbs with common suffixes (e.g., “-es”, “-ed”, “-ing”).
- 4.
Calculate the Percentage of Complex Words (PCW)
$\text{PCW}=\left(\frac{\text{Number of Complex Words}}{\text{Total Number of
Words}}\right)\times 100$
(3)
- 5.
Compute the Gunning Fog Index
$\text{GFI}=0.4\times\left(\text{ASL}+\text{PCW}\right)$
(4)

### E.3 Mean Dependency Distance

The mean dependency distance (MDD) (Oya, ) is introduced as a metric to quantify the syntactic complexity of sentences based on dependency parsing. A higher mean dependency distance indicates longer dependencies and potentially more complex syntactic structures. For each dependency relation between a word (a head) and its dependent in a sentence $d$ is calculated as the absolute difference of their positions in the sentence:

$$ $d=\left|\text{Position}_{\text{head}}-\text{Position}_{\text{dependent}}\right|$ $$

Here:

- •
$\text{Position}_{\text{head}}$ is the position index of the head word.
- •
$\text{Position}_{\text{dependent}}$ is the position index of the dependent word.

The sentence-level MDD is computed by averaging the dependency distances of all its $N$ dependency relations:

$$ $\text{MDD}_{\text{sentence}}=\frac{1}{N}\sum_{i=1}^{N}d_{i}=\frac{1}{N}\sum_{i =1}^{N}\left|\text{Position}_{\text{head}_{i}}-\text{Position}_{\text{ dependent}_{i}}\right|$ $$

Same with sentence-level, the document-level MDD averages the sentence-level mean dependency distances across all $M$ sentences in the document:

$$ $\text{MDD}_{\text{document}}=\frac{1}{M}\sum_{j=1}^{M}\text{MDD}_{\text{ sentence}_{j}}$ $$

In our experiments, we calculate the document-level MDD for a test sample by averaging the MDD of all its text. For example, to compute the MDD for a pair-classification data point, we average the MDD of sentence 1 and sentence 2.

## Appendix F Prompt For ChatGPT Error Rate

The detailed example prompt of each task for the ChatGPT Error Rate is listed in Table tables 14, 17, 16, 20, 18, 19 and 15.

**Table 14: Prompt for the ChatGPT Error Rate on Semantic Textual Similarity (STS).**
| Determine whether the following two sentences are similar and answer yes or no. |
| --- |
| Sentence1: Excluding the impact of merger-related costs, NSTAR Electric s earnings increased $67.4 million in 2013, as compared to 2012, due primarily to lower overall operations and maintenance costs and higher retail electric sales due primarily to colder weather in the first and fourth quarters in 2013. |
| Sentence2: NSTAR Electric’s earnings increased in 2014, as compared to 2013, due primarily to lower operations and maintenance costs primarily attributable to lower employee-related costs and higher transmission earnings, partially offset by higher interest expense, higher depreciation expense, higher property tax expense and the after-tax reserve recorded for the 2014 FERC ROE orders as compared to the reserve recorded in 2013 for the FERC ALJ initial decision in the FERC base ROE complaints. |

**Table 15: Prompt for the ChatGPT Error Rate on Classification.**
| Classify the sentiment of a given finance text as either positive, negative, or neutral. |
| --- |
| Text: Glencore shares hit 3-month high after refinancing key credit line |

**Table 16: Prompt for the ChatGPT Error Rate on Retrieval.**
| Given a financial question, retrieve user replies that best answer the question. Return the index.<br>Query: What is ’Obligor ’? |
| --- |
| Corpus: {A List of 20 different context} |

**Table 17: Prompt for the ChatGPT Error Rate on Clustering.**
| Identify industries from company descriptions. |
| --- |
| Choices: Banking;Retail;Automotive;Aerospace;Financial services |
| Text: Cobridge Communications was a cable television, high-speed internet, and digital telephone service provider. |

**Table 18: Prompt for the ChatGPT Error Rate on Reranking.**
| Given a financial question, retrieve user replies that best answer the question. Return the index.<br>Query: How to tell if you can trust a loan company? |
| --- |
| Corpus: {A List of 20 different context} |

**Table 19: Prompt for the ChatGPT Error Rate on Pair-Classification.**
| Classify the sentiment of a given finance text and determine whether label of two sentences are similar. Answer yes or no. |
| --- |
| Sentence1: gold falls as dollar strengthens, etf holdings decline |
| Sentence2: Gold futures succumb to profit-booking, global cues |

**Table 20: Prompt for the ChatGPT Error Rate on Summarization.**
| Determine whether the following are douments and summary. Answer yes or no. |
| --- |
| Text: deposits grew $ 167.8 million , or 7 % , to $ 2.504 billion at december 31 , 2020 , compared to $ 2.336 billion at december 31 , 2019. non-interest bearing deposits grew by $ 225.8 million in 2020 , or 20 % , and made up 54 % of total deposits at year-end . cost of deposits remained low at 0.11 % in 2020 , compared to 0.20 % in 2019. net interest income totaled $ 96.7 million and $ 95.7 million in 2020 and 2019 , resp…. |
| Summary: cash and cash equivalents : our cash and cash equivalents , which include federal funds sold and short-term investments , were $ 181.5 million at december 31 …. |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see my comments in the Weakness section.
soundness: 3
strengths: 1. **Valuable problem**: This paper focuses on a practical and important problem: creating high-quality text embeddings for the financial domain.

2. **Well-written paper**: The paper is clearly written and easy to follow.

3. **Comprehensive model evaluation**: The authors evaluate a diverse range of embedding models, providing a thorough overview of the current embedding model landscape. Additionally, the experimental results demonstrate the challenges posed by FinMTEB, highlighting the need for developing better financial embedding models.
summary: This paper introduces FinMTEB, a massive text embedding benchmark specifically designed for the financial domain. Experimental results show that state-of-the-art models that perform well on MTEB exhibit a significant performance drop when evaluated on FinMTEB. The authors further quantify the complexity of both FinMTEB and MTEB and conduct control experiments. They find that, at the same complexity level, model performance on FinMTEB is much lower than on MTEB, which they argue provides evidence that the challenges arise from domain-specific linguistic and semantic patterns rather than dataset complexity. Additionally, the authors observe that performance on MTEB does not correlate well with performance on FinMTEB, highlighting the need for domain-specific embedding benchmarks.
weaknesses: 1. **Should current general-purpose text embedding models perform well on financial embedding tasks?**: I have concerns about whether current SOTA text embedding models should be expected to perform effectively on financial embedding tasks. For example, the NV-Embed [1] model is trained on only a single finance-related dataset, FiQA. Given the limited financial data used in training, these embeddings may not perform well for finance-specific tasks.

2. **Unfair dataset language composition**: Most SOTA general-purpose text embeddings on the MTEB benchmark, including the models evaluated in this paper, are predominantly trained on English datasets. However, a substantial portion of FinMTEB is in Chinese, creating an imbalanced setting that may not be fair for evaluating these models.

3. **Dataset complexity measurement**: I have concerns about the reliability of the authors’ approach to quantifying dataset complexity. A) Complexity is measured independently for queries and documents, but retrieval is fundamentally a pairwise task, where the relationship between query and document can significantly affect complexity. For example, the BRIGHT [2] dataset requires reasoning to map queries to documents, which current models struggle with. B) I question the validity of using ChatGPT’s error rate as a complexity measure. If ChatGPT cannot answer a question correctly, is it due to the complexity of the question or the model’s lack of question-specific knowledge?

4. **Concerns about the argument of linguistic pattern**: I have concerns about the authors' claim that the higher token count in FinMTEB compared to MTEB indicates significant linguistic differences between the two benchmarks.

5. **High semantic diversity**: The inter-dataset semantic similarity is calculated using only 100 samples from each dataset. I question whether this sample size is sufficient to accurately represent the overall dataset’s characteristics.

6. **Unclear presentation of results**: This is a minor point, but only the average score across all FinMTEB tasks is shown. Without detailed results for each task type (e.g., clustering, retrieval), the performance of models on FinMTEB is not fully clear.

[1] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv preprint arXiv:2405.17428, 2024. 

[2] Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. arXiv preprint arXiv:2407.12883, 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: The paper claims that inter-dataset similarity scores below 0.6 indicate high semantic diversity. Could the authors clarify if this threshold reliably indicates diversity?

How were datasets divided into low, medium, and high complexity? Were the same thresholds used across datasets, or were they divided within each dataset independently? Further clarification here would enhance the validity of this categorization.

ChatGPT Error Rate as a Complexity Metric: Could the authors provide more details on how ChatGPT's error rate was used to categorize dataset complexity? Specifically, how was this metric interpreted to distinguish three complexity levels? An ChatGPT's judge (correct or wrong) might be able to be aligned with binary complexity in my understanding.

FinMTEB’s Dataset Source: In Tables 7 to 13, is the lack of citations for certain datasets indicative of new dataset construction? Clarifying this would help in assessing the novelty of FinMTEB.

In Section 5, the interpretation of p-values could be misleading. A large p-value indicates a lack of evidence for a relationship, not proof of no relationship. Rephrasing to clarify this distinction would avoid statistical misinterpretation.

The abstract mentions that the analysis provides "compelling evidence" of difficulties in capturing domain-specific patterns. Could the authors clarify where this evidence is presented in the analysis and summarize them, as it was not immediately apparent?
soundness: 2
strengths: This paper explores a critical question―whether domain-specific embedding models are necessary in the era of general-purpose, large-scale language models.
This paper presents the Finance Massive Text Embedding Benchmark (FinMTEB), a collection of existing or newly constructed evaluation datasets.
The dataset may be useful for emphasizing the finance-specific abilities in benchmarking because the performances on the (general-domain) MTEB and FinMTEB seem less correlated.
This paper evaluates dataset complexity in addition to model performances.
summary: This paper investigates the performance of text embedding models on two distinct benchmarks: a general-domain benchmark and a finance-specific benchmark, which is a newly collected dataset. The findings highlight a performance gap and lack of correlation between the two, suggesting the need for domain-specific benchmarks and embedding models better to capture specialized linguistic patterns in domains like finance.
weaknesses: The paper’s title question, "Do We Need Domain-Specific Embedding Models?" is not convincingly addressed. To conclude that domain-specific models are necessary, it would be important to demonstrate that (1) a domain-specific embedding model can outperform a general-purpose model within that domain and (2) the observed difference is specifically due to domain specialization. However, the paper does not provide such evidence. Therefore, the research question (and title) would be better phrased as "Do We Need Domain-Specific Embedding Benchmarks?" or "Are Current SoTA Embedding Models Sufficient for Domain-Specific Tasks?"

The analysis lacks sufficient logical rigor and is sometimes misleading. For example, describing the performance difference between FinMTEB and MTEB as a "significant performance drop" is misleading, as MTEB and FinMTEB are not directly comparable. These benchmarks differ in various aspects, making it speculative to attribute the performance difference primarily to financial domain knowledge. A clearer attribution of the observed effects is necessary. Even with additional analyses, it remains unclear whether any convincing conclusions can be drawn about the models solely based on this performance comparison.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: **Question:**

- From Table 7 to 13, I found that there are part of Chinese datasets in the benchmark. But the part of selected embedding models are English model, e.g. bge-large-en-v1.5 and bge-en-icl. Is this a potential problem?
- How long will it take to benchmark different scale models? It would be better to report the benchmarking time.

**Suggestions:**

- The related work on general-purpose embeddings is insufficient. More pilot and recent LLM sentence embeddings are encouraged to be considered in the related work, such as [1][2][3][4][5].
- According to the ICLR style, a table caption is better placed before the table, and a figure caption is better placed after the figure.

**Reference:**

[1] Li X, Li J. Angle-optimized text embeddings[J]. arXiv preprint arXiv:2309.12871, 2023.

[2] Lee J, Dai Z, Ren X, et al. Gecko: Versatile text embeddings distilled from large language models[J]. arXiv preprint arXiv:2403.20327, 2024.

[3] Li X, Li J. BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 792-804.

[4] Springer J M, Kotha S, Fried D, et al. Repetition improves language model embeddings[J]. arXiv preprint arXiv:2402.15449, 2024.

[5] Lee C, Roy R, Xu M, et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models[J]. arXiv preprint arXiv:2405.17428, 2024.
soundness: 3
strengths: - This paper presents a massive financial text embedding benchmark — FinMTEB, which consists of 64 financial datasets. I think it is a helpful resource for the financial NLP research community.
- This paper designs a novel and comprehensive evaluation framework to prove that the performance gap between MTEB and FinMTEB is due to the models’ capabilities rather than inherent dataset complexity.
- The paper is well organized and well written.
summary: This paper presents the Finance Massive Text Embedding Benchmark (FinMTEB). As a domain-specific counterpart to the general-purpose MTEB, FinMTEB includes 64 financial datasets curated for the evaluation of embedding models in the financial domain. A comparative analysis of model performance on both MTEB and FinMTEB suggests the need for domain-specific embeddings and benchmarks.
weaknesses: - From my point of view, the title is not good because the main contribution of this paper is the benchmark. Why not refer to the benchmark directly in the title?
- The evidence may be insufficient to support that the general embedding is not good at domain application since only one domain, i.e., finance, is proven in this paper.
- Lack of novelty: According to the previous work [1][2][3], domain fine-tuning can improve domain-specific performance, which is well-accepted evidence. Based on this, the FinMTEB benchmark should be the main contribution of this paper, not the "byproduct" as claimed in the paper.


**Reference:**

[1] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964.

[2] Huang Y, Wang K, Dutta S, et al. AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification[J]. arXiv preprint arXiv:2311.00408, 2023.

[3] Schopf T, Schneider D N, Matthes F. Efficient domain adaptation of sentence embeddings using adapters[J]. arXiv preprint arXiv:2307.03104, 2023.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: None
soundness: 3
strengths: - The development of the FinMTEB benchmark provides a valuable tool for evaluating embedding models within the financial domain.
- The experimental design is highly rigorous, systematically eliminating the potential confounding factors related to the models used and the inherent complexity of the datasets. This enhances the credibility and reference value of the experimental conclusions.
summary: This paper investigates the necessity of domain-specific embedding models given the prevalence of general-purpose embedding models. To address this, the authors first developed a financial domain benchmark, FinMTEB. Utilizing this benchmark, they observed a significant decline in performance and, through a series of experiments, ruled out the influence of the selected models and task complexity. This led to the conclusion that general-purpose models are inadequate for domain-specific tasks.
weaknesses: - The conclusion that general-purpose embedding models are not well-suited for specific domain tasks and that specialized models are necessary for particular domains is not particularly surprising.

Meta-review: The paper asks an intuitive question "Do We Need Domain-Specific Embedding Models?" and performed a case study in the financial domain, where the authors created a dataset. 

Reviewers generally gave borderline or rejection scores. The biggest concern is an overclaim from one specific domain to a general statement of their finding. In addition, the question researched does not differ drastically from previous studies of (L)LM domain adaptation. Of course, the authors can argue that the domain adaptation studies focus on different models/aspects, the paper has not offered insights that are interesting enough.

Decision: Reject