MUSTS: MUltilingual Semantic Textual Similarity Benchmark
Authors: Tharindu Ranasinghe, Hansi Hettiarachchi, Constantin Orasan, Ruslan Mitkov
Accepted at: acl25

MUSTS: MUltilingual Semantic Textual Similarity Benchmark

Tharindu Ranasinghe◊, Hansi Hettiarachchi◊, Constantin Orăsan♡ and Ruslan Mitkov◊
◊School of Computing and Communications, Lancaster University, UK
♡Centre for Translation Studies, University of Surrey, UK
{t.ranasinghe, h.hettiarachchi, r.mitkov}@lancaster.ac.uk
c.orasan@surrey.ac.uk

Abstract

Predicting semantic textual similarity (STS) is a complex and ongoing challenge in natural language processing (NLP). Over the years, researchers have developed a variety of supervised and unsupervised approaches to calculate STS automatically. Additionally, various benchmarks, which include STS datasets, have been established to consistently evaluate and compare these STS methods. However, they largely focus on high-resource languages, mixed with datasets annotated focusing on relatedness instead of similarity and contain automatically translated instances. Therefore, no dedicated benchmark for multilingual STS exists. To solve this gap, we introduce the Multilingual Semantic Textual Similarity Benchmark (MUSTS), which spans 13 languages, including low-resource languages. By evaluating more than 25 models on MUSTS, we establish the most comprehensive benchmark of multilingual STS methods. Our findings confirm that STS remains a challenging task, particularly for low-resource languages.

1 Introduction

Semantic textual similarity (STS) measures the extent to which two sentences convey the same meaning (Cer et al., 2017). Automatically measuring STS is a foundational natural language understanding (NLU) problem relevant to numerous applications (Mu and Lim, 2024), including machine translation (Wieting et al., 2019; Ranasinghe et al., 2020), text summarisation (Majumder et al., 2024), question answering (Mitkov et al., 2023), and information retrieval (Iida and Okazaki, 2021).

Over the years, researchers have developed various supervised and unsupervised approaches to calculate STS, ranging from training recurrent neural networks (RNNs) (Mueller and Thyagarajan, 2016; Ranasinghe et al., 2019b), fine-tuning pre-trained language models (Chandrasekaran and Mago, 2021) and prompting large language models (LLMs) (Chen et al., 2023). The organisation of SemEval shared tasks from 2012 to 2017 has fuelled the development of these neural network architectures (Cer et al., 2017; Agirre et al., 2016, 2015, 2014, 2013, 2012).

Natural language understanding (NLU) benchmarks such as GLUE (Wang et al., 2018) have been introduced to enable the systematic evaluation and comparison of STS methods. The multilingual adaptation of GLUE in languages such as Korean (Park et al., 2021), Japanese (Kurihara et al., 2022), and Sinhala (Ranasinghe et al., 2025) also included STS tasks. From the multilingual benchmarks, the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2023) includes 12 STS datasets spanning multiple languages, making it a valuable resource for multilingual text similarity evaluation (Su et al., 2023). However, despite its widespread adoption, several notable weaknesses are associated with the STS datasets included in MTEB.

(i) Language Coverage - None of the languages in the STS datasets of MTEB benchmark are low-resource (Muennighoff et al., 2023). According to Joshi et al. (2020), they all fall into the resource-rich 'Winners' or 'Underdogs' categories. Consequently, it remains uncertain how the top models in MTEB would generalise into low-resource languages as unsupervised STS methods.

(ii) Text Similarity vs. Relatedness - Several STS datasets in MTEB, including Polish (Dadas et al., 2020) and Finnish (Kanerva et al., 2021) have been annotated with a focus on relatedness rather than similarity. While similarity involves paraphrasal or entailment relations, relatedness accounts for all of the commonalities that can exist between two sentences (Abdalla et al., 2023; Morris and Hirst, 1991). As a result, datasets focused on relatedness cannot be classified as STS datasets.
<table>
  <tr>
    <th>Language</th>
    <th>Family</th>
    <th>|Train|</th>
    <th>|Test|</th>
    <th>Reference</th>
    <th>Source</th>
    <th>MTEB</th>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>Afro-Asiatic (Semitic)</td>
    <td>1080</td>
    <td>250</td>
    <td>Cer et al. (2017)</td>
    <td>SNLI</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Brazilian Portuguese</td>
    <td>Indo-European (Romance)</td>
    <td>2500</td>
    <td>2000</td>
    <td>Fonseca et al. (2016)</td>
    <td>SICK</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Czech</td>
    <td>Indo-European (Slavic)</td>
    <td>925</td>
    <td>500</td>
    <td>Sido et al. (2024)</td>
    <td>News</td>
    <td>×</td>
  </tr>
  <tr>
    <td>English</td>
    <td>Indo-European (Germanic)</td>
    <td>5750</td>
    <td>1380</td>
    <td>Cer et al. (2017)</td>
    <td>SNLI</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>French</td>
    <td>Indo-European (Romance)</td>
    <td>600</td>
    <td>410</td>
    <td>Cardon and Grabar (2020)</td>
    <td>Wikipedia & Vikidia</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>Japonic</td>
    <td>12500</td>
    <td>1460</td>
    <td>Kurihara et al. (2022)</td>
    <td>YJ Caption</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>Koreanic</td>
    <td>5750</td>
    <td>1380</td>
    <td>Ham et al. (2020)</td>
    <td>SNLI</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Portuguese</td>
    <td>Indo-European (Romance)</td>
    <td>2500</td>
    <td>2000</td>
    <td>Fonseca et al. (2016)</td>
    <td>SICK</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Romanian</td>
    <td>Indo-European (Romance)</td>
    <td>5750</td>
    <td>1380</td>
    <td>Dumitrescu et al. (2021)</td>
    <td>SNLI</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Serbian</td>
    <td>Indo-European (Slavic)</td>
    <td>953</td>
    <td>239</td>
    <td>Batanović et al. (2018)</td>
    <td>paraphrase.sr</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Sinhala</td>
    <td>Indo-European (Indo-Aryan)</td>
    <td>5000</td>
    <td>100</td>
    <td>Kadupitiya et al. (2016)</td>
    <td>SICK</td>
    <td>×</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>Indo-European (Romance)</td>
    <td>1620</td>
    <td>250</td>
    <td>Cer et al. (2017)</td>
    <td>SNLI</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Tamil</td>
    <td>Dravidian</td>
    <td>2500</td>
    <td>100</td>
    <td>Nilaxan and Ranathunga (2021)</td>
    <td>SICK</td>
    <td>×</td>
  </tr>
</table>

Table 1: Overview of the languages included in MUSTS, listing the language, language family, dataset size (train and test), reference, source, and whether the dataset is part of MTEB. We group the language into high-resource \( \mathbf{I} \), mid-resource \( \mathbf{I} \), and low-resource \( \mathbf{I} \) based on Joshi et al. (2020).

(iii) Machine Translations - Several STS datasets in MTEB, such as German and Russian, were generated by machine translations of English STS datasets without post-editing. These datasets risk propagating translation errors and introducing stylistic biases that affect model training and evaluation (Mahfuž et al., 2025; Hettiarachchi et al., 2025). As a result, the validity of using these datasets for model evaluation remains questionable (Mager et al., 2018).

Although some of these limitations are acceptable for an embedding evaluation benchmark like MTEB (Muennighoff et al., 2023), they prevent the datasets within MTEB from being suitable as an STS benchmark. Therefore, in this research, we compile MUSTS: MUltilingual Semantic Textual Similarity Benchmark addressing the limitations mentioned above. Furthermore, we evaluate several popular unsupervised and supervised STS methods on MUSTS and report the results.

The main findings of the paper are;
• LLM prompting and LLM-based encoders provide the best results as unsupervised STS methods for high and mid-resource languages. However, for low-resource languages, simple STS methods like smooth inverse frequency (Arora et al., 2017) and earlier transformer-based sentence encoders like LaBSE (Feng et al., 2022) outperform LLMs.
• MTEB ranking does not reflect in MUSTS, encouraging more low-resource language inclusion in multilingual benchmarks.
• Training transformers provide the best results for STS even for low-resource languages, aligning with the other regression-based NLP tasks (Qian et al., 2024; Ma et al., 2024).

2 MUSTS: Multilingual Semantic Textual Similarity Benchmark

Table 1 shows datasets in 13 languages that we identified fulfilling our requirements to be included in MUSTS. More statistics are available in Appendix D. We assign each language into one of the three categories following Joshi et al. (2020), where ‘Winners’ are high-resource, ‘Underdogs’ are mid-resource and the rest are low-resource. Compared to MTEB, MUSTS provides a better coverage in mid and low-resource languages as it includes languages like Sinhala and Tamil (De Silva, 2019).

Unlike MTEB, MUSTS only contains datasets that followed STS annotation guidelines from Agirre et al. (2013) (Appendix A) without mixing with other concepts such as relatedness and paraphrasing. While some languages, such as Portuguese (Fonseca et al., 2016) and Sinhala (Kadupitiya et al., 2016), include sentences from the SICK dataset (Marelli et al., 2014), which focus on semantic relatedness, they were reannotated for the STS labels. From the STS datasets that were machine-translated from an English source, such as Sinhala (Kadupitiya et al., 2016) and Tamil (Nilaxan and Ranathunga, 2021), we only include those that have undergone a post-editing process. Finally, we restrict our selection to STS datasets published in peer-reviewed papers.

Eliminated Datasets While Faroese (Snæbjarnarson et al., 2023) and Bengali (Shajalal and Aono, 2018) STS datasets satisfy MUSTS criteria, they contain less than 1000 instances. As MUSTS aims to evaluate supervised models as well, we decided not to include datasets with less than 1000 instances.
3 Methods

We evaluated the following unsupervised and supervised STS methods on MUSTS.

3.1 Unsupervised STS Methods

3.1.1 Vector Averaging

The first unsupervised STS method averages the word embeddings of both sentences and computes their cosine similarity. We used three multilingual transformer models; XLM-R Large (Conneau et al., 2020), RemBERT (Chung et al., 2021), and infoXLM Large (Chi et al., 2021) to obtain embeddings. For the words which are split into sub-tokens by a particular tokeniser, we used the first sub-token embedding as the word’s representation.

3.1.2 Smooth Inverse Frequency

Taking the average of word embeddings in a sentence gives equal weights for words such as but, just, etc. Smooth Inverse Frequency (SIF) addresses this in two steps (Arora et al., 2017);

(i) Weighting: SIF computes a weighted average of word embeddings, where each word is weighted by \( \frac{a}{a + p(w)} \), with \( a \) which we set to 0.001 and \( p(w) \) as the word’s estimated frequency in a reference corpus. For these experiments, we used HPLT (de Gibert et al., 2024).

(ii) Common Component Removal: It then computes the principal component of these embeddings across sentences and removes their projections onto the first principal component, reducing noise from frequent words like stop words.

After these two steps, we compute the cosine similarity between the two embeddings (Ranasinghe et al., 2019a). We employed the same multilingual transformer models used in §3.1.1 in SIF too.

3.1.3 Sentence Embeddings

We also utilised multilingual sentence encoders to generate sentence embeddings and calculate the cosine similarity to represent the STS. Specifically, we used LaBSE (Feng et al., 2022), all-MiniLM-L12-v2, and LASER (Heffernan et al., 2022) as the sentence encoders.

3.1.4 LLM Prompting

Following recent advances in NLP, we evaluated LLMs for measuring STS using a prompt-based approach. We conducted experiments with two recently released multilingual LLMs: Llama-3.1-8B-Instruct (Touvron et al., 2023) and Mistral-8B-Instruct (Jiang et al., 2023), using four prompting strategies.

(i) Zero-shot (ZS): We asked the models to predict similarity solely by following instructions without providing any examples.

(ii) Few-shot-English (FS-En): Under this variant, we provided five English examples randomly chosen from the training dataset with similarity scores spread across the entire range (0-5).

(iii) Few-shot-Monolingual (FS-Mono): Unlike FS-En, we provided the model with five examples in the same language as the given sentence pair for measuring STS. We used the same approach as (ii) to subsample training data.

(iv) Few-shot with Chain of thoughts (FS-CoT): We provided the model with a small set of examples, each demonstrating a series of intermediate reasoning steps that break down the task for better understanding. We utilised six English examples using the explanations in Table 3 in the Appendix A.

We adapted the original prompt templates from Wang et al. (2024a) and refined them based on our initial experiments. More details on prompt design and our prompts are provided in Appendix B.

3.1.5 LLM-Encoders

We also employed the top six LLM-based embedding models in MTEB leaderboard as of December 2024. Namely, they are NV-Embed-v2 (Lee et al., 2024), bge-en-ic1 (Li et al., 2024), stella_en_1.5B_v5, SFR-Embedding-2_R, gte-Qwen2-7B-instruct (Li et al., 2023) and gte-Qwen2-1.5B-instruct (Li et al., 2023). We compute the cosine similarity between the embeddings corresponding to the two sentences to represent the similarity.

3.2 Supervised STS Methods

3.2.1 Transformers

As the first supervised STS approach, we concatenated training sets from all the languages from MUSTS and trained three multilingual transformer models; XLM-R Large (Conneau et al., 2020), RemBERT (Chung et al., 2021), and infoXLM Large (Chi et al., 2021). The architecture and the configurations are available in Appendix C.1.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Low-resource</th>
    <th colspan="6">Mid-resource</th>
    <th colspan="4">High-resource</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>Sc</th><th>Sl</th><th>Ta</th>
    <th>Ar</th><th>Br-Pt</th><th>Cz</th><th>Ja</th><th>Ko</th><th>Pt</th><th>Ro</th><th>En</th><th>Es</th><th>Fr</th>
  </tr>
  <tr>
    <th colspan="15"><b>§3.1.1 Word Vector Average</b></th>
    <th></th>
  </tr>
  <tr>
    <td>XLM-R Large</td><td>0.499</td><td>0.246</td><td>0.259</td><td>0.333</td><td>0.460</td><td>0.416</td><td>0.265</td><td>0.388</td><td>0.394</td><td>0.420</td><td>0.318</td><td>0.383</td><td>0.610</td><td>0.384</td>
  </tr>
  <tr>
    <td>RemBERT</td><td>0.370</td><td>0.120</td><td>0.242</td><td>0.336</td><td>0.439</td><td>0.509</td><td>0.254</td><td>0.387</td><td>0.362</td><td>0.450</td><td>0.406</td><td>0.518</td><td>0.673</td><td>0.390</td>
  </tr>
  <tr>
    <td>infoXML Large</td><td>0.475</td><td>0.204</td><td>0.322</td><td>0.424</td><td>0.494</td><td>0.511</td><td>0.370</td><td>0.450</td><td>0.389</td><td>0.490</td><td>0.427</td><td>0.551</td><td>0.622</td><td>0.440</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.1.2 Smooth Inverse Frequency</b></th>
    <th></th>
  </tr>
  <tr>
    <td>XLM-R Large</td><td>0.389</td><td>0.193</td><td>0.266</td><td>0.504</td><td>0.461</td><td>0.544</td><td>0.294</td><td>0.459</td><td>0.483</td><td>0.505</td><td>0.400</td><td>0.577</td><td>0.762</td><td>0.464</td>
  </tr>
  <tr>
    <td>RemBERT</td><td>0.357</td><td>0.261</td><td>0.197</td><td>0.532</td><td>0.438</td><td>0.607</td><td>0.245</td><td>0.459</td><td>0.415</td><td>0.518</td><td>0.530</td><td>0.641</td><td>0.751</td><td>0.473</td>
  </tr>
  <tr>
    <td>infoXML Large</td><td>0.640</td><td>0.336</td><td>0.426</td><td>0.602</td><td>0.491</td><td>0.618</td><td>0.426</td><td>0.539</td><td>0.483</td><td>0.538</td><td>0.534</td><td>0.662</td><td>0.792</td><td>0.545</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.1.3 Sentence Encoders</b></th>
    <th></th>
  </tr>
  <tr>
    <td>LaBSE</td><td>0.726</td><td>0.499</td><td>0.551</td><td>0.690</td><td>0.708</td><td>0.782</td><td>0.761</td><td>0.705</td><td>0.730</td><td>0.715</td><td>0.722</td><td>0.808</td><td>0.891</td><td>0.716</td>
  </tr>
  <tr>
    <td>MiniLM-L12-v2</td><td>0.708</td><td>0.087</td><td>0.258</td><td>0.791</td><td>0.677</td><td>0.865</td><td>0.778</td><td>0.744</td><td>0.720</td><td>0.796</td><td>0.844</td><td>0.855</td><td>0.802</td><td>0.686</td>
  </tr>
  <tr>
    <td>LAESR</td><td>0.597</td><td>0.292</td><td>0.201</td><td>0.674</td><td>0.653</td><td>0.743</td><td>0.736</td><td>0.689</td><td>0.656</td><td>0.686</td><td>0.697</td><td>0.796</td><td>0.820</td><td>0.581</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.1.4 LLM Prompting</b></th>
    <th></th>
  </tr>
  <tr>
    <td>Llama-3.1-BB-Instruct ZS</td><td>0.746</td><td>0.396</td><td>0.549</td><td>0.713</td><td>0.735</td><td>0.759</td><td>0.822</td><td>0.729</td><td>0.747</td><td>0.706</td><td>0.801</td><td>0.802</td><td>0.806</td><td>0.717</td>
  </tr>
  <tr>
    <td>Llama-3.1-BB-Instruct FS-En</td><td>0.720</td><td>0.414</td><td>0.360</td><td>0.646</td><td>0.682</td><td>0.753</td><td>0.699</td><td>0.706</td><td>0.700</td><td>0.693</td><td>0.761</td><td>0.742</td><td>0.798</td><td>0.667</td>
  </tr>
  <tr>
    <td>Llama-3.1-BB-Instruct FS-Mono</td><td>0.745</td><td>0.483</td><td>0.313</td><td>0.681</td><td>0.723</td><td>0.689</td><td>0.789</td><td>0.750</td><td>0.735</td><td>0.740</td><td>0.761</td><td>0.678</td><td>0.862</td><td>0.689</td>
  </tr>
  <tr>
    <td>Llama-3.1-BB-Instruct FS-CoT</td><td>0.276</td><td>0.364</td><td>0.525</td><td>0.705</td><td>0.596</td><td>0.759</td><td>0.811</td><td>0.705</td><td>0.590</td><td>0.621</td><td>0.725</td><td>0.698</td><td>0.599</td><td>0.613</td>
  </tr>
  <tr>
    <td>Mistral-8B-Instruct ZS</td><td>0.360</td><td>0.085</td><td>0.354</td><td>0.741</td><td>0.598</td><td>0.740</td><td>0.807</td><td>0.717</td><td>0.565</td><td>0.655</td><td>0.743</td><td>0.775</td><td>0.705</td><td>0.603</td>
  </tr>
  <tr>
    <td>Mistral-8B-Instruct FS-En</td><td>0.512</td><td>0.313</td><td>0.256</td><td>0.730</td><td>0.663</td><td>0.803</td><td>0.819</td><td>0.754</td><td>0.647</td><td>0.682</td><td>0.759</td><td>0.802</td><td>0.705</td><td>0.650</td>
  </tr>
  <tr>
    <td>Mistral-8B-Instruct FS-Mono</td><td>0.569</td><td>0.327</td><td>0.351</td><td>0.733</td><td>0.697</td><td>0.558</td><td>0.687</td><td>0.750</td><td>0.716</td><td>0.737</td><td>0.759</td><td>0.799</td><td>0.671</td><td>0.643</td>
  </tr>
  <tr>
    <td>Mistral-8B-Instruct FS-CoT</td><td>0.628</td><td>0.342</td><td>0.311</td><td>0.610</td><td>0.554</td><td>0.717</td><td>0.807</td><td>0.695</td><td>0.622</td><td>0.632</td><td>0.764</td><td>0.716</td><td>0.750</td><td>0.627</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.1.5 Top LLM-based Encoders on MTEB (Muenighoff et al., 2023)</b></th>
    <th></th>
  </tr>
  <tr>
    <td>NY-Embed-v2</td><td>0.745</td><td>0.343</td><td>0.397</td><td>0.785</td><td>0.775</td><td>0.857</td><td>0.792</td><td>0.792</td><td>0.806</td><td>0.799</td><td>0.843</td><td>0.881</td><td>0.896</td><td>0.743</td>
  </tr>
  <tr>
    <td>bge-en-ic1</td><td>0.725</td><td>0.292</td><td>0.322</td><td>0.788</td><td>0.691</td><td>0.824</td><td>0.762</td><td>0.743</td><td>0.739</td><td>0.734</td><td>0.805</td><td>0.822</td><td>0.839</td><td>0.668</td>
  </tr>
  <tr>
    <td>stella_en_1.5B_v5</td><td>0.651</td><td>0.124</td><td>0.341</td><td>0.643</td><td>0.742</td><td>0.825</td><td>0.784</td><td>0.536</td><td>0.774</td><td>0.704</td><td>0.860</td><td>0.871</td><td>0.889</td><td>0.663</td>
  </tr>
  <tr>
    <td>SFR-Embedding-2_R</td><td>0.836</td><td>0.399</td><td>0.334</td><td>0.788</td><td>0.772</td><td>0.861</td><td>0.778</td><td>0.785</td><td>0.811</td><td>0.791</td><td>0.863</td><td>0.869</td><td>0.896</td><td>0.750</td>
  </tr>
  <tr>
    <td>gte-Qwen2-7B-instruct</td><td>0.695</td><td>0.280</td><td>0.397</td><td>0.818</td><td>0.723</td><td>0.816</td><td>0.772</td><td>0.759</td><td>0.740</td><td>0.759</td><td>0.816</td><td>0.857</td><td>0.878</td><td>0.711</td>
  </tr>
  <tr>
    <td>gte-Qwen-1.5B-instruct</td><td>0.591</td><td>0.312</td><td>0.353</td><td>0.746</td><td>0.724</td><td>0.776</td><td>0.751</td><td>0.737</td><td>0.756</td><td>0.682</td><td>0.853</td><td>0.864</td><td>0.888</td><td>0.690</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.2.1 Training Transformers</b></th>
    <th></th>
  </tr>
  <tr>
    <td>XLM-R Large</td><td>0.866</td><td>0.810</td><td>0.832</td><td>0.842</td><td>0.811</td><td>0.870</td><td>0.833</td><td>0.801</td><td>0.835</td><td>0.825</td><td>0.887</td><td>0.891</td><td>0.898</td><td>0.846</td>
  </tr>
  <tr>
    <td>RemBERT</td><td>0.868</td><td>0.821</td><td>0.841</td><td>0.848</td><td>0.820</td><td>0.876</td><td>0.841</td><td>0.808</td><td>0.841</td><td>0.829</td><td>0.889</td><td>0.899</td><td>0.900</td><td>0.852</td>
  </tr>
  <tr>
    <td>infoXML Large</td><td>0.869</td><td>0.825</td><td>0.848</td><td>0.851</td><td>0.825</td><td>0.878</td><td>0.844</td><td>0.811</td><td>0.848</td><td>0.839</td><td>0.893</td><td>0.904</td><td>0.905</td><td>0.880</td>
  </tr>
  <tr>
    <th colspan="15"><b>§3.2.2 Training LLM-based Encoders</b></th>
    <th></th>
  </tr>
  <tr>
    <td>stella_en_1.5B_v5</td><td>0.692</td><td>0.392</td><td>0.651</td><td>0.665</td><td>0.651</td><td>0.763</td><td>0.751</td><td>0.592</td><td>0.753</td><td>0.682</td><td>0.765</td><td>0.752</td><td>0.844</td><td>0.689</td>
  </tr>
  <tr>
    <td>gte-Qwen2-1.5B-instruct</td><td>0.683</td><td>0.420</td><td>0.770</td><td>0.648</td><td>0.683</td><td>0.758</td><td>0.732</td><td>0.638</td><td>0.672</td><td>0.650</td><td>0.741</td><td>0.770</td><td>0.864</td><td>0.691</td>
  </tr>
</table>

Table 2: Spearman (r) correlation between model predictions and human annotations. The best result for each language (any method) is in bold, and the best unsupervised result is underlined.

3.2.2 Training LLM-Encoders
We also selected two small LLM-Encoders that could be trained on an NVidia L40 48G GPU. We concatenated training sets from all the languages from MUSTS and trained these encoders using the sentence transformers architecture (details in Appendix C.2) (Reimers and Gurevych, 2019). We specifically used gte-Qwen2-1.5B-instruct and stella_en_1.5B_v5.

4 Results and Analysis

Table 2 shows the Spearman correlation between predictions of each model and human annotations. We describe our main findings in the following list.

(i) LLMs as Unsupervised STS Methods - The results show that both LLM-based unsupervised STS approaches, encoders and prompting, provide excellent results in mid and high-resource languages. Interestingly, some of these LLMs do not even directly support languages such as Czech and Romanian, yet they provide impressive results for the STS task. LLM-based encoders jointly outperformed prompting in nine out of ten mid/high resource languages. Finally, there is no clear winner among the four prompting strategies. However, we notice FS-Mono provides better results than FS-En in the majority of languages.

Both LLM-based encoders and prompting do not perform well in low-resource languages, particularly Sinhala and Tamil. Interestingly, SIF, a simple STS method, produced competitive results compared to LLM-based methods in low-resource languages. From the experimented unsupervised methods, LABSE (Feng et al., 2022) provided the best results for low-resource languages.

(ii) MUSTS ranking vs. MTEB Ranking - The results in row §3.1.5, Table 2 show that model ranking in MTEB is not reflected in MUSTS. For instance, SFR-Embedding-2_R, the fourth-ranked model in MTEB, ranked first among the text em-
bedding models in MUSTS. We attribute this discrepancy primarily to the language coverage of MUSTS and suggest that NLP benchmarks should incorporate low-resource languages for a fairer evaluation.

(iii) Supervised STS Models - Training transformer models provided the best results for STS. Among the multilingual transformer models tested, info-XLM Large achieved the highest performance across all 13 languages.

Interestingly, fine-tuning LLM-based encoders on MUSTS using the Sentence Transformers architecture did not lead to overall improvements. However, it did enhance performance in low-resource languages. We believe that techniques such as contrastive instruction tuning (Wang et al., 2024b) should be further explored to achieve better results.

5 Conclusions

In this paper, we compiled MUSTS, the most comprehensive multilingual STS benchmark up to date. MUSTS spans over 13 languages and includes carefully selected STS datasets in terms of task definition and human annotations. We make MUSTS publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard1.

We evaluated more than 25 STS approaches on MUSTS. Our results showed that STS remains a challenging task, especially for low-resource languages. Furthermore, we show that LLM-based methods thrive in the STS task for mid and high-resource languages but struggle in low-resource languages. Our findings highlight the need for fair multilingual evaluations in STS.

Limitations

We only used the models that could be experimented on an NVidia L40 48G GPU. Due to these computational limitations, we did not experiment with the larger models. Larger models may perform differently in this STS task.

The examples used in the few-shot scenarios ($3.1.4) were randomly sampled since we do not have the knowledge to prepare good-quality examples for all languages. Results might be different if these examples were carefully chosen by native speakers.

Ethical Considerations

All the datasets released with MUSTS are publicly available with a CC BY 4.0 licence. Furthermore, all the models that we experimented with in this paper are publicly available in HuggingFace (Lhoest et al., 2021).

Acknowledgements

We would like to thank the anonymous reviewers for their positive and valuable feedback. We further thank the creators of the datasets used in this paper for making the datasets publicly available for our research.

The experiments in this paper were conducted in UCREL-HEX (Vidler and Rayson, 2024). We would like to thank John Vidler for the continuous support and maintenance of the UCREL-HEX infrastructure, which enabled the efficient execution of our experiments.

References

Mohamed Abdalla, Krishnapriya Vishnubhotla, and Saif Mohammad. 2023. What makes sentences semantically related? a textual relatedness dataset and empirical study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 782–796, Dubrovnik, Croatia. Association for Computational Linguistics.

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Marítxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado. Association for Computational Linguistics.

Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland. Association for Computational Linguistics.

Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California. Association for Computational Linguistics.

1 Available at https://github.com/TharinduDR/MUSTS
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385–393, Montréal, Canada. Association for Computational Linguistics.

Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32–43, Atlanta, Georgia, USA. Association for Computational Linguistics.

Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations.

Vuk Batanović, Miloš Cvetanović, and Boško Nikolić. 2018. Fine-grained semantic textual similarity for Serbian. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Rémi Cardon and Natalia Grabar. 2020. A French corpus for semantic similarity. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6889–6894, Marseille, France. European Language Resources Association.

Daniel Cer, Mona Diab, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.

Dhivya Chandrasekaran and Vijay Mago. 2021. Evolution of semantic similarity—a survey. ACM Comput. Surv., 54(2).

Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. How robust is gpt-3.5 to predecessors? a comprehensive study on language understanding tasks. arXiv preprint arXiv:2303.00293.

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXML: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online. Association for Computational Linguistics.

Hyung Won Chung, Thibault Fevrier, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Slawomir Dadas, Michał Perefkiewicz, and Rafał Poświta. 2020. Evaluation of sentence representations in Polish. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1674–1680, Marseille, France. European Language Resources Association.

Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, and Jörg Tiedemann. 2024. A new massive multilingual dataset for high-performance language technologies. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1116–1128, Torino, Italia. ELRA and ICCL.

Nisansa De Silva. 2019. Survey on publicly available sinhala natural language processing tools and research. arXiv preprint arXiv:1906.02358.

Stefan Daniel Dumitrescu, Petru Rebeja, Beata Lorincz, Mihaela Gaman, Andrei Avram, Mihai Ilie, Andrei Pruteanu, Adriana Stan, Lorena Rosia, Cristina Iacobescu, Luciana Morogan, George Dima, Gabriel Marchidan, Traian Rebedea, Madalina Chitez, Dani Yogatama, Sebastian Ruder, Radu Tudor Ionescu, Razvan Pascanu, and Viorica Patraucean. 2021. Liro: Benchmark and leaderboard for romanian language tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Erick Rocha Fonseca, Leandro Borges dos Santos, Marcelo Criscuolo, and Sandra Maria Aluisio. 2016. Visão geral da avaliação de similaridade semântica e inferência textual. Linguamática, 8(2):3–13.

Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. KorNLI and KorSTS:
New benchmark datasets for Korean natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422–430, Online. Association for Computational Linguistics.

Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan, and Lasitha Randunu Chandrakantha Uyangode. 2025. Overview of the first workshop on language models for low-resource languages (LoResLM 2025). In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 1–8, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Hiroki Iida and Naoki Okazaki. 2021. Incorporating semantic textual similarity and lexical matching for information retrieval. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 582–591, Shanghai, China. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Jcs Kadupitiya, Surangika Ranathunga, and Gihan Dias. 2016. Sinhala short sentence similarity calculation using corpus-based and knowledge-based similarity measures. In Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016), pages 44–53, Osaka, Japan. The COLING 2016 Organizing Committee.

Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Iiro Rastas, Valtteri Skantis, Jemina Kilpeläinen, Hanna-Mari Kupari, Jenna Saarni, Maija Sevón, and Otto Tarkka. 2021. Finnish paraphrase corpus. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 288–298, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.

Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966, Marseille, France. European Language Resources Association.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvyaa Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. Making text embedders few-shot learners. arXiv preprint arXiv:2409.15700.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Wanqin Ma, Chenyang Yang, and Christian Kästner. 2024. (why) is my prompt getting worse? rethinking regression testing for evolving llm apis. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN ’24, page 166–171, New York, NY, USA. Association for Computing Machinery.

Manuel Mager, Elisabeth Mager, Alfonso Medina-Urrea, Ivan Vladimir Meza Ruiz, and Katharina Kann. 2018. Lost in translation: Analysis of information loss during machine translation between polysynthetic and fusional languages. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 73–83, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Tamzeed Mahfuz, Satak Kumar Dey, Ruwad Naswan, Hasnaen Adil, Khondker Salman Sayeed, and Haz Sameen Shahgir. 2025. Too late to train, too early to use? a study on necessity and viability of
low-resource Bengali LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1183–1200, Abu Dhabi, UAE. Association for Computational Linguistics.

Goutam Majumder, Vikrant Rajput, Partha Pakray, Sivaji Bandyopadhyay, and Benoît Favre. 2024. Text summary evaluation based on interpretable semantic textual similarity. Multimedia Tools and Applications, 83(1):3233–3258.

Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavík, Iceland. European Language Resources Association (ELRA).

Ruslan Mitkov, Le An Ha, Halyna Maslak, Tharindu Ranasinghe, and Vilelmini Sosoni. 2023. Automatic generation of multiple-choice test items from paragraphs using deep neural networks. In Advancing Natural Language Processing in Educational Assessment, pages 77–89. Routledge.

Jane Morris and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics, 17(1):21–48.

Wenchuan Mu and Kwan Hui Lim. 2024. Modelling text similarity: A survey. In Proceedings of the 2023 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’23, page 698–705, New York, NY, USA. Association for Computing Machinery.

Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1).

Niklas Muenninghoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

Satkunanantham Nilaxan and Surangika Ranathunga. 2021. Monolingual sentence similarity measurement using siamese neural networks for sinhala and tamil languages. In 2021 Moratuwa Engineering Research Conference (MERCon), pages 567–572.

Sungjoon Park, JiHyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chising Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021. Klue: Korean language understanding evaluation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.

Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanodia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. 2024. What do large language models need for machine translation evaluation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3660–3674, Miami, Florida, USA. Association for Computational Linguistics.

Tharindu Ranasinghe, Hansi Hettiarachchi, Nadeesha Pathirana, Damith Premasiri, Lasitha Uyangodage, Isuri Anuradha, Alistair Plum, Paul Rayson, and Ruslan Mitkov. 2025. Sinhala encoder-only language models and evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria. Association for Computational Linguistics.

Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2019a. Enhancing unsupervised sentence similarity methods with deep contextualised word representations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 994–1003, Varna, Bulgaria. INCOMA Ltd.

Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2019b. Semantic textual similarity with Siamese neural networks. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1004–1011, Varna, Bulgaria. INCOMA Ltd.

Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation quality estimation with cross-lingual transformers. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5070–5081, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Md Shajalal and Masaki Aono. 2018. Semantic textual similarity in bengali text. In 2018 International Conference on Bangla Speech and Language Processing (ICBSLP), pages 1–5.

Jakub Sido, Michal Seják, Ondřej Pražák, Miloslav Konopík, and Václav Moravec. 2024. Czech news dataset for semantic textual similarity. Language Resources and Evaluation, pages 1–18.
Vésteinn Snæbjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023. Transfer to a low-resource language via close relatives: The case study on Faroese. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 728–737, Tórshavn, Faroe Islands. University of Tartu Library.

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

John Vidler and Paul Rayson. 2024. UCREL - Hex; a shared, hybrid multiprocessor system. https://github.com/UCREL/hex.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.

Yuxia Wang, Minghan Wang, and Preslav Nakov. 2024a. Rethinking STS and NLI in large language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 965–982, St. Julian’s, Malta. Association for Computational Linguistics.

Zhimeng Wang, Pinzheng Wang, Juntao Li, Yibin Chen, and Min Zhang. 2024b. Achieving stronger generation via simple contrastive tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8986–8999, Miami, Florida, USA. Association for Computational Linguistics.

John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU: Training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344–4355, Florence, Italy. Association for Computational Linguistics.

A Annotation Guidelines

All the datasets in MUSTS follow the annotation guidelines in Table 3.

<table>
  <tr>
    <th></th>
    <th>The two sentences are completely equivalent, as they mean the same thing.</th>
  </tr>
  <tr>
    <td>5</td>
    <td>The bird is bathing in the sink.<br>Birdie is washing itself in the water basin.</td>
  </tr>
  <tr>
    <th></th>
    <th>The two sentences are mostly equivalent, but some unimportant details differ.</th>
  </tr>
  <tr>
    <td>4</td>
    <td>Two boys on a couch are playing video games.<br>Two boys are playing a video game.</td>
  </tr>
  <tr>
    <th></th>
    <th>The two sentences are roughly equivalent, but some important information differs/missing.</th>
  </tr>
  <tr>
    <td>3</td>
    <td>John said he is considered a witness but not a suspect. “He is not a suspect anymore.” John said.</td>
  </tr>
  <tr>
    <th></th>
    <th>The two sentences are not equivalent, but share some details.</th>
  </tr>
  <tr>
    <td>2</td>
    <td>They flew out of the nest in groups.<br>They flew into the nest together.</td>
  </tr>
  <tr>
    <th></th>
    <th>The two sentences are not equivalent, but are on the same topic.</th>
  </tr>
  <tr>
    <td>1</td>
    <td>The woman is playing the violin.<br>The young lady enjoys listening to the guitar.</td>
  </tr>
  <tr>
    <th></th>
    <th>The two sentences are completely dissimilar.</th>
  </tr>
  <tr>
    <td>0</td>
    <td>The black dog is running through the snow.<br>A race car driver is driving his car through the mud.</td>
  </tr>
</table>

Table 3: Similarity scores with explanations and English examples from Agirre et al. (2013).

B Prompts

As a starting point, we used the prompt templates provided by Wang et al. (2024a), developed through a comprehensive experimental study, as the basis for our study. We made a few key updates. First, we revised the task description to explicitly mention ‘semantic textual similarity’ to avoid confusion with syntactic similarity, which our initial experiments suggested could occur when using just the term ‘similarity’. We also added a new sentence to each prompt to clarify the expected output, aiming to reduce post-processing errors when extracting the scores. Table 4 summarises the final prompt templates.

C Supervised STS Models

C.1 Training Transformers

We trained the transformer models using the architecture in Figure 1.
• All our models use the AdamW (Loshchilov and Hutter, 2019) optimiser with a weight decay of 1e-8, learning rate of 2e-5, a warmup ratio of
<table>
  <tr>
    <th>task_description:</th>
    <td>Determine the semantic textual similarity between the following two sentences (S1, S2). The score should be ranging from 0.0 to 5.0, and can be a decimal.</td>
  </tr>
  <tr>
    <th>Strategy</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>ZS</td>
    <td>{task_description} Return the score only following the prefix 'Score:' without any other text or explanations. S1: {sentence1} S2: {sentence2}</td>
  </tr>
  <tr>
    <td>FS</td>
    <td>Five demonstration examples:<br><br>Example 1:<br>S1: Ebola UK: NHS staff 'panicked' after suspected Ebola cases.<br>S2: UK says investigating 2 suspected MERS cases.<br>Score: 0.0<br><br>Example 2:<br>...<br><br>{task_description} Return the score only following the prefix 'Score:' without any other text or explanations. S1: {sentence1} S2: {sentence2}</td>
  </tr>
  <tr>
    <td>FS-CoT</td>
    <td>{task_description} Return the explanation and score only following the prefixes 'Explain:' and 'Score:' without any other text.<br><br>Six demonstration examples with explanation for each:<br><br>Example 1:<br>S1: The black dog is running through the snow.<br>S2: A race car driver is driving his car through the mud.<br>Explain: S1 and S2 are completely dissimilar.<br>Score: 0.0<br><br>Example 2:<br>...<br><br>S1: {sentence1} S2: {sentence2}</td>
  </tr>
</table>

Table 4: Prompt templates used for experiments

0.06 from the training data and are trained for five epochs with a batch size of 32 (gradient accumulation is applied when needed), and a maximum length of 512 tokens. The values of the hyper-parameters (including the number of training epochs) were set to fixed values in order to ensure consistency between all the models.
• All the models were evaluated while training using a development set that had one-fifth of the rows separated from the training set before the start of the training process.
• The best checkpoints were selected on the development set. We use the Spearman correlation as the checkpoint selection criterion.
• We trained our models on an NVidia L40 48G GPU. All models were trained with half-precision (fp16) using the default PyTorch implementation.

C.2 Training LLM-based Encoders
We trained the LLM encoders using the architecture in Figure 2.

![Architecture for using Transformers in STS.](page_792_120_484_246.png)

Figure 1: Architecture for using Transformers in STS.

![Architecture for using Transformers in STS.](page_792_624_484_246.png)

Figure 2: Architecture for using Transformers in STS.

• All models were trained using the same configurations mentioned in Appendix C.1, apart from the learning rate and warmup ratio, which were set to 1e-6 and 0.1, respectively.

D MUSTS Statistics
In the following figures, we show label distribution and token overlap for each language.
Arabic

(1) Similarity distribution of Ar train set

(2) Similarity distribution of Ar test set

(3) Word share against similarity bins in Ar train set

(4) Word share against similarity bins in Ar test set
Brazilian Portuguese

(1) Similarity distribution of Br-Pt train set

(2) Similarity distribution of Br-Pt test set

(3) Word share against similarity bins in Br-Pt train set

(4) Word share against similarity bins in Br-Pt test set
Czech

(1) Similarity distribution of Cz train set
(2) Similarity distribution of Cz test set
(3) Word share against similarity bins in Cz train set
(4) Word share against similarity bins in Cz test set
English

(1) Similarity distribution of En train set

(2) Similarity distribution of En test set

(3) Word share against similarity bins in En train set

(4) Word share against similarity bins in En test set
French

(1) Similarity distribution of Fr train set
(2) Similarity distribution of Fr test set
(3) Word share against similarity bins in Fr train set
(4) Word share against similarity bins in Fr test set
Japanese

(5) Similarity distribution of Jp train set

(6) Similarity distribution of Jp test set

(7) Word share against similarity bins in Jp train set

(8) Word share against similarity bins in Jp test set
Korean

(1) Similarity distribution of Ko train set

(2) Similarity distribution of Ko test set

(3) Word share against similarity bins in Ko train set

(4) Word share against similarity bins in Ko test set
Portuguese

(1) Similarity distribution of Pt train set
(2) Similarity distribution of Pt test set
(3) Word share against similarity bins in Pt train set
(4) Word share against similarity bins in Pt test set
Romanian

(1) Similarity distribution of Ro train set

(2) Similarity distribution of Ro test set

(3) Word share against similarity bins in Ro train set

(4) Word share against similarity bins in Ro test set
Serbian

(1) Similarity distribution of Se train set

(2) Similarity distribution of Se test set

(3) Word share against similarity bins in Se train set

(4) Word share against similarity bins in Se test set
Sinhala

(1) Similarity distribution of Si train set

(2) Similarity distribution of Si test set

(3) Word share against similarity bins in Si train set

(4) Word share against similarity bins in Si test set
Spaish

(1) Similarity distribution of Es train set

(2) Similarity distribution of Es test set

(3) Word share against similarity bins in Es train set

(4) Word share against similarity bins in Es test set
Tamil

(1) Similarity distribution of Ta train set
(2) Similarity distribution of Ta test set
(3) Word share against similarity bins in Ta train set
(4) Word share against similarity bins in Ta test set

Reviewer paper summaries: The paper presents a new benchmark for semantic textual similarity (STS) in several languages, compiled from existing public datasets that match specific criteria.
A broad range of multilingual models is assessed in the benchmark, to conclude that LLM prompting and LLM-based encoders are the best unsupervised methods, except for low-resource languages. Yet, the best results are achieved by fine-tuning transformers in the training data.

This work introduces Multilingual Semantic Textual Similarity Benchmark (MUSTS), a comprehensive semantic textual similarity (STS) benchmark that covers 13 languages across high-, mid-, and low-resource levels. This paper evaluates both unsupervised and supervised STS methods on MUSTS, and shows the challenge of STS for low-resource languages.

The authors introduce the Multilingual Semantic Textual Similarity Benchmark (MUSTS) as a set of datasets in 13 languages (high, medium, and low resource) and seven categories of unsupervised and supervised STS methods through distinct models. Unlike a SOTA benchmark, MUSTS also includes low resource language coverage and is devoted purely to the semantic textual similarity (STS) task. Being a challenging task, especially for low-resource languages, MUSTS attempts to fill the gap for this being a plural benchmark for STS.
Summary of strengths: A new benchmark for STS, covering a broad range of languages, from high- to low-resource.

A broad range of approaches and open multilingual models compared in the new benchmark.

**Motivation**  This paper has a clear motivation. The proposed benchmark addresses the limitations of the commonly used MTEB benchmark: language coverage, relatedness, and the quality of machine translations.
  
**Evaluation** The evaluation is very comprehensive, covering a range of unsupervised and supervised STS methods including vector averaging, LLM prompting, and fine-tuning transformers.
  
**Clarity** This paper is clearly written with a well-defined motivation, a thorough description of the benchmark, a detailed explanation of the evaluation methods, and a clear presentation of the results and analysis. Tables 1 and 2 are particularly informative, with one providing details about the benchmark including language family, resource level, and the other one showing a comprehensive set of evaluation results with very nice visual aids.

- Set of specific datasets for the STS task (independent of the relatedness task)
- Coverage of low resource languages not frequently offered by the other SOTA benchmark
- The inclusion of, beyond the compilation of datasets, different methods (unsupervised and supervised) to STS along the 13 languages covered with the available code
- Being still a challenging task, there is a high likelihood of other researchers using this benchmark to achieve higher results
Summary of weaknesses: Some incorrections and missing information raise doubts on the dataset selection / integration process and in the obtained results. (clarified in the author's response)

The discussion of the results, which is probably the most interesting thing of the paper, should be longer.

**Contribution** My biggest concern is the sufficiency of this paper’s contribution. It seems that the authors did not truly address the limitations of METB mentioned in Section 1, especially the issue of relatedness (lines 62-71) and the quality of machine translations (lines 72-80). The authors handled these issues by simply omitting the problematic data from the compiled benchmark, rather than introducing new data points. The paper’s contribution is limited to benchmark compilation rather than actively resolving the limitations of MTEB.

- For a short paper, there is no space for further explanation of the distinct methods used
- The SOTA benchmark cited is referred to as having 12 STS datasets. Still, after reading carefully the given reference, it seems there are only 11 datasets. Please confirm this situation in "Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics."
Comments, suggestions and typos: Given that MUSTS is the main contribution of the work, the paper could provide more details on the search and inclusion criteria, report which datasets from MTEB were left out, and, ideally, include some statistics in the main body of the paper.

Authors claim that MUSTS covers 13 languages, but two of them are Portuguese and Brazilian Portuguese, i.e., two varieties of the same language. Moreover, the reference in Table 1 is for the ASSIN 2 task, which has a dataset of 6,500, 500, and 3,000 pairs, respectively for training, validation and test datasets, all of them in Brazilian Portuguese. So nor the reported numbers, nor the division into both varieties makes sense.
The original ASSIN task indeed contained half of the pairs in Brazilian Portuguese and another half in European Portuguese, but it did not use SICK as the source of pairs:

Fonseca, E. R., Borges dos Santos, L., Criscuolo, M., & Aluísio, S. M. (2016). Visão Geral da Avaliação de Similaridade Semântica e Inferência Textual. Linguamática, 8(2), 3-13.

Authors should probably consider adding this dataset, but first make sure that their numbers are correct.

Sec. 3.1.3 mentions all-MiniLM-L12-v2 as a multilingual sentence encoder, but it was trained for English only. By the way, some reference should be included for this model, either a paper or a URL.

Questions:

Q1: Why is only the embedding of the first token in a sub-word used in the vector averaging approaches?
Q2: How is p(w) estimated for SIF?
Q3: Sentence Transformers are often trained in sentence-level tasks, including STS. Did the authors check whether either these models or the LLM Encoders had been trained in some of the used datasets? If so, including them in the unsupervised category should be rethought.
Q4: When authors say that "fine-tuning LLM-based encoders on MUSTS using the Sentence Transformers architecture did not lead to overall improvements", what they comparing it to? Also, there should be a reference for the Sentence Transformers.

Form:
The categories introduced in Sec. 2, ‘Winners’, ‘Underdogs’, others end up not being used is the rest of the paper, including Table 1.
Appendix E is not referred in the main body of the paper.

1. It would strengthen the paper if the authors further emphasize the necessity of compiling this benchmark. For example, the authors could discuss how the original uncompiled data exists in varying formats, making it difficult to use.

2. Lines 252-254 “The results in row 5, Table 2 show…” are somewhat misleading. Since the “blocks” in Table 2 already have titles like §3.1.5 Top LLM-based Encoders on MTEB, it would be clearer to refer readers directly to this title instead of “row 5”.

3. I noticed that Serbian, though being a low-resource language, generally has higher Spearman correlation compared to other low-resource languages (even some mid-resource languages). Do you have any insights into this?

- Please clarify the observation in line 246 when the authors say, "Interestingly, SIF, a simple STS method, produced competitive results compared to LLM-based methods in low-resource languages". In table 2, I cannot detect such a situation.
Overall assessment scores: 3

3

4
Metareview score: 3.5
Metareview: This work introduces the Multilingual Semantic Textual Similarity Benchmark (MUSTS) which spans (up to) 13 languages (as there is some dispute if Portuguese and Brazilian Portuguese count as two separate languages). The evaluate many models (more than 25 on MUSTS) to establish a comprehensive baseline. 

The work is well motivated but the writing lacks the detail required of a benchmark if it is to be a widely used resource.
Summary of reasons to publish: -	Well-motivated work; a benchmark for STS focused on low resource language
-	Still a challenging task so other researchers are likely to use the benchmark