Semantic similarity prediction is better than other semantic similarity measures
Authors: Steffen Herbold
Venue: Accepted by TMLR ()

Abstract: Semantic similarity between natural language texts is typically measured either by looking at the overlap between subsequences (e.g., BLEU) or by using embeddings (e.g., BERTScore, S-BERT). Within this paper, we argue that when we are only interested in measuring the semantic similarity, it is better to directly predict the similarity using a fine-tuned model for such a task. Using a fine-tuned model for the Semantic Textual Similarity Benchmark tasks (STS-B) from the GLUE benchmark, we define the STSScore approach and show that the resulting similarity is better aligned with our expectations on a robust semantic similarity measure than other approaches.

Published in Transactions on Machine Learning Research (01/2023)
Semantic similarity prediction is better than other semantic similarity measures
Steffen Herbold
Faculty of Computer Science and Mathematics
University of Passau
Passau, Germany
Reviewed on OpenReview: https: // openreview. net/foruee id= dfbbfsNggN5
Abstract
Semantic similarity between natural language texts is typically measured either by looking at the overlap between subsequences (e.g., BLEU) or by using embeddings (e.g., BERTScore, S-BERT). Within this paper, we argue that when we are only interested in measuring the semantic similarity,
better directly predict the similarity using fine-tuned model for such task. Using a fine-tuned model for the Semantie Textual Similarity Benchmark tasks (STS-B) from the GLUE benchmark, define the STSScore approach and show that the resulting similarity is better aligned with our expectations on robust semantic similarity mcasure than other approaches.
1 Introduction
When we are considering research questions related to the quality of the output of generative models for natural language (e.g., Large Language Models/LLMs like GPT(-2) by Radford et al. (2019) and all the other currently very successful models that followed), we need to determine the generated text is a valid output for the desired task. For example, when we want to use such model for translation from English German, need to determine if the resulting German sentence has the same meaning the English sentence. Since there are typically many possible valid outputs for such tasks, cannot casily specify ground truth for benchnarks. While an expert would be able to determine if the oulpul is correct, automating this task with a computer is not easy, as this would effectively require solving the translation problem first. However, since scaling such evaluations to thousands or even millions of examples is not possible without automation, researchers created different methods to assess the quality of output using the idea of semanlic similarily.
The general idea is quite simple: while we cannot define the ground truth, we can measure the semantic meaning to one more correct outputs. Simple metrics such the Word Error Rate (WER) estimate this through the consideration of the differences between words, similar to the Levenshtein distance. Others, like BiLinnuall Evaluation Understudy (BLEU, Papineni et al. (2002)) or Evaluation (ROGUE, Lin (2004)) are rather looking at the overlap between subsequences of texts, assuming that a larger overlap means a better semantic similarity. Furthermore, some methods (c.g., BLEU) go beyond the comparison of single solutions a sample output and instead allow the comparison between the corpora of generated and sample solutions. For individual words without considering them their context, word embeddings like word2vec (Mikolov et al., 2013) in combination with cosine similarity are generally accepted way to estimate the semantic similarity between words, even though this approach has problems with ambiguities, e.g., caused by polysemes (Del Tredici & Bel, 2015). Similar methods were extended to whole sentences (Sharma et al., 2017), but they did not achieve the same level of performance.
for Gisting
The transformer architecture enabled the context-sensitive calculation of embeddings (Vaswani et al., 2017). Naturally, this was adopted for the calculation of the similarity. Two methods based on this are currently primarily used. The first BERTScore by Zhang et al. (2020), which is based on an optimal matching of the pair-wise similarities of words within contextual BERT embedding and has been used by thousands of

---

Published in Transactions on Machine Learning Research (01/2023)
publications since its publication in 2020. The second is Sentence-BERT (S-BERT) by Reimers & Gurevych (2019), who pool the embeddings of the tokens into an embedding for sentences. The cosine similarity can then be computed between these sentence embeddings,, the same as between words with word2vec or with carlier, less powerful, sentence embeddings (c.g. Sharma et al., 2017).
This success notwithstanding, we want to argue that this approach should not be further pursued in favor of simpler solution to estimate the semantic similarity, i.e, directly predicting the semantic similarity with regression model. We note that this idea is not new and was, c.g., also used to define approaches like BEER (Stanojevié & Sima'an 2014) or RUSE (Shimanaka et al., 2018), However, these models are from the pre-transformer era of natural language processing. As can be seen in large benchmarks like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), models based on the transformer architecture (Vaswani et al., 2017) provide a much better performance. Therefore, we formulate the following hypothesis for our research:
Hypothosis: Modern language models with encoder-only transformer architecture similar to BERT (Devlin el al., 2019) that are fine-tuned as regression models for the similarity between sentence pairs are also capable of robustly measuring the semantic similarity beyond their training data and are better measures for the semantic similarity than embedding- based and n-gram approaches.
We derive this hypothesis from the assumption that such a regression model fulfills its task, employing it as semantic similarity measure would be the natural use case beyond just benchmarking model capabilities. Within this paper, we present the results of confirmatory study that demonstrates that downloading fine- tuned RoBERTa model (Liu et al., 2019) for the STS-B task (Cer et al., 2017) from the GLUE benchmark from Huggingface and using this model to predict the similarity of sentences fulfills our expectations on a robust similarity measure better than the other models we consider. We refer to this approach as STSScorer. To demonstrate this empirically, we compute the similarity score for similarity related GLUE tasks and show that while the predictions with the STSScorer are not perfect, the distribution of the predicted scores is closer what we would expect given the task description, than for the other measures.
2 Method
Within this section, we describe the rescarch method we used evaluate our hypothesis. We first describe the STSScorer model for the prediction of the similarity in Section 2.1. Then, we proceed to describe our analysis approach in Section 2.2, including the expectations have regarding our hypothesis and the tools we used in Section 2.6.
2.1 STSScorer
Listing describes our approach: download fine-tuned model from Huggingface for the STS-B task (Held, 2022), which was based RoBERTa (Liu et al., 2019). The STS-B task contains sentence pairs from news, image captions and Web forums. Each sentence pair received a score between zero and five. These scores were computed as the average of the semantic similarity rating conducted by three humans such that five mcans that the raters believe the sentences mcan exactly the same and zero means that the sentences are completely unrelated to each other. We simply use a regression model trained for this task and divide the results by five to scale them to the interval [0, 1]. The regression model is trained with Mean Squared Error (MSE) loss, which is defined as MSE(y, yy") (y--""" where y is the predicted label and y" the expected label. Hereafter, we refer to this model STSScorer.
2.2 Analysis approach
Our analysis approach similar to the method used by Zhang et al. (2020) for the evaluation of the robustness of similarity measures also by Reimers & Gurevych (2019) for the evaluation of S-BERT: we utilize data labeled data for which we have an expectation of what to observe when measuring the semantic similarity.
2

---

Published in Transactions on Machine Learning Research (01/2023)
Listing 1: A simple class to define a fully functional semantic similarity scorer based pre-trained model for the STS-B tasks.
import transformers
class STSScorer:
def init init_(seilll (self):
model name 'WillHeld/ poceedddrrbbraaabbe btase-stsb"
self. sts tokenizer transformers. AutoTokenizer. from pretrained (
+ model model__nme)
self sts model
_sts____________________________ _sts_modell = transformers. t AutoloodFoosqquaccCsssstttttttt
4 from pretrained (model name)
self. sts evael.eval
def score (self, sentencel, sentence2): sts tokenizer output self. sts lokenizer Sakkenizer(seteencee (sentencel, sentence2, padding=
4 True, truncation=True, return return_tensorse (onsors='pt") sts model output = self. sts m*sel tokenizer_outee
# logits contain regression values # need to divide by five due to scoring approach of STS-B between 0 and 5 return sts model _ottuttuupuutt llogis '].item ()/5
There are four such data sets within the GLUE benchmark:
the Semantic Textual Similarity Benchmark (STS-B) data we already discussed above;
the Microsoft Research Paraphrase Corpus (MRPC, Dolan & Brockett (2005)) data, where the task to determine two sentences are paraphrases;
the Quora Question Pairs (QQP, Iyer et al. (2017)) data, where the task is to determine if two questions are duplicates; and
the Chinese to English translations from the WMT22 metrics challenge (WMT22-7H-EN, Freitag et al. (2022)), where the translation quality is labeled using the MQM schema Burchardt (2013).
For STS-B, MRPC, and WMT22, we use the test data. Since the labels for QQP's test data are not shared, we use the training data instead. To the best of our knowledge, this data was not seen during the training of the STSScorer and S-BERT models of the models we use, as Quora was not part of the pre- training corpus of RoBERTa, which mitigates the associated risks regarding data contamination. However, the model underlying S-BERT was fine-tuned using contrastive learning on a corpus of one billion sentences (Reimers & Gurevych, 2019), which contained about 100,000 instances from the QQP data, i.e., about quarter. Thus, S-BERT might have advantage this data.
On each of these data sets, we compute the similarity between all pairs of sentences with BLEU, BERTScore, S-BERT," 1 and STSScore. All methods we consider compute scores between zero (not similar at all) and one (same somantic meaning), which simplifics the direct comparison. This narrower view few models allows us to consider the results more in-depth. Specifically, we can go beyond the plain reporting of numbers and instead look directly the distributions of the similarity measures for different data sets. Due to the confirmatory design of our study, we formulate concrete expectations the results, given the properties of each data set. How well the similarity measures fulfill these expectations will be later used to evaluate our hesis. Based on the strong performance of BERT-based models the STS-B task in the GLUE benchmark, we predict, based on our hypothesis that STSScorer should have the best alignment with our expectations for all data sets.
From now on, simply refer calculating the similarity between embeddings with S-BERT S-BERT brevity.
3

---

Published in Transactions on Machine Learning Research (01/2023)
We note that while we could have added more approaches to this comparison, e.g., ROUGE (Lin, 2004). METEOR (Lavie & Agarwal, 2007), RUSE (Shimanaka et. al., 2018), and BEER (Stanojevié & Sima an, 2014), these models were all already compared to BERTScore, which was determined to provide a better measure for the similarity (Zhang et al., 2020). Further, we refer to a general overview of such metrics to the work by Zhang et al. (2020). Thus, instead of providing a broad comparison with many models, we rather compare our approach to the embedding- approaches S-BERT and BERTScore which are currently used de-facto state-of-the-art by most publications and the still very popular BLEU method that uses an n-gram matching approach.
Additional analyses regarding the impact of the sequence lengths, the use of different methods as an ensemble, as well as using a different model within the STSScorer can be found in the appendix.
2.3 STS-B data
While the comparison on the STS-B data may seem unfair, because the STSScorer was specifically trained for that model, the analysis of the behavior of the different scorers on this model still gives us interesting insights: for any semantic similarity measure, the distribution of the scores should be directly related to the label of STS-B,² which is a human judgment of the semantic similarity. Consequently, when we plot the label on the x-axis versus a semantic similarity score on the y-axis, we would ideally observe a strong linear correlation. Visually, we would observe this by the data being close to the diagonal. less ideal, but still good, result would be that the trend is monotonously increasing, indicating a rank correlation, which would mean that while the magnitudes of the similarity measure not aligned with the human judgments from STS-B, at least the order of values is. Any other trend would mean that the similarity mcasure is not aligned with the human judgments of STS-B. In addition to this visualization, we also measure the linear correlation between the scores and the labels with Pearson's r and the rank correlation with Spearman's p. as is common for the STS-B task within the GLUF benchmark (Wang et al., 2018) and was also used by Reimers & Gurevych (2019) for the evaluation of S-BERT.
Because the STSScorer was fine-tuned the STS-B data, we only utilize the test data. Nevertheless, because this is exactly the same context during the training of STSScorers models (same data curators, same humans creating the judgments) this model has a huge advantage over the other approaches. Due to this, we expect that. STSScorer is well aligned with human judgments and we observe the linear trend described above. If this fails, this would directly contradict our hypothesis, as this would not even work within-context. The BLEU, BERTScore,, and S-BERT models were created independently of the STS-B data, but given their purpose to estimate the semantic similarity, they should still be able to fulfill the desired properties. If this is not the case, this would rather be an indication that these models are not measuring the semantic similarity at least not according to the human judgments from the STS-B data.
2.4 MRPC and QQP data
The MRPC and QQP data are similar: both provide binary classification problems. With MRPC, the problem is paraphrasing. With QQP, the problem is duplicate questions, which can also be viewed as a type of paraphrasing, i.e., the paraphrasing of questions. Paraphrasing is directly related to semantic similarity, as paraphrased sentences should be semantically equal. Thus, similarity measures should yield high values for paraphrases and duplicate questions, ideally close to one.
For the negative examples of MRPC, a look at the data helps to guide our expectations. When considering the MRPC data, we observe that the negative samples are all sentences on the same topic, with different meaning. As an example, consider the first negative example from the training data:
Sentence 1: Yucaipa owned Dominick 's before selling the choin to Safeway in 1998 for $ 2.5 billion
Sentence 2: Yucaipa bought Dominick 's in 1995 for $ 693 million and sold it to Safeway for $ 1.8 billion in 1998
WWill STS-B is regression task and and it would be hetter to speak of a dependent variable here, we rather speak of labels all the time be consistent with the subsequent classification tasks.
4

---

Published in Transactions on Machine Learning Research (01/2023)
Both sentences are related to the ownership of Dominick's by Yuceipa and the sale to Safeway, but consider different aspects regarding the time and different amounts of money for the payment. Thus, we observe semantic relationship, but not a paraphrasing. While we have not read through all the negative examples from the MRPC data, we also did not find any examples where both sentences were completely unrelated. Consequently, we expect values for the semantic similarity of the negative examples to be significantly larger than zero, but also smaller than the positive examples of actual paraphrases with notable gap.
For the negative examples of QQP, the expectation is less clear. For most pairs, we observe that both questions are somewhat related, iee, different questions regarding the same topic. As an example, consider this negative example from the training data:
Sentence 1:
What causes stool color to change to yellow?
Sentence 2: What can cause stool to come out as little balls?
Both questions are about stool, but different aspects of stool. This kind of difference is similar to what have within the MRPC data. However, also observed examples, where the questions are completely unrelated. Consider the following instance from the training data of QQP:
Sentence 1: How not to feel guilty since I am Muslim and I'm conscious we won't have ser together? Sentence 2: I don't beleive I am bulimic,, but I force throw up alleast once a day after I eat something and feel guilty. Should tell somebody, and if so who?
While both questions are broadly related the concept of guilt, the rest completely unrelated and we would expect a very low semantic similarity. Consequently, while our expectation for the semantic similarity measures for the majority of the negative samples is similar to that of MRPC (i.e,, significantly larger than zero, but smaller than for the positive examples), we also expect to observe a strong tail in the distribution with lower similarities.
For both data sets, we visualize the distribution of the similarities per class. Additionally, we report the central tendency (arithmetic mean) and variability (standard deviation) per class in the data. Finally, we consider how the similarity metrics would fare classifiers based on the Area Under the Curve (AUC).
2.5 WMT22-ZH-EN
With the WMT22-ZH-EN data, we consider how the semantic similarity between a reference solution and alternative translations relates to manually labeled translation quality. Semantic similarity between the source and translation is an important aspect when we want to consider the quality of the generated translation. However, this is only one aspect of the Multidimensional Quality Metrics (MQM) framework for the analytic evaluation of translation quality (Burchardt, 2013). This framework uses multiple criteria such as the accuracy, fluency, terminology. style, and locale conventions. While the semantic similarity is directly related to the accuracy criterion, the others are rather regarding how a translations worded beyond its meaning- something we typically want ignore when measuring the semantic similarity. Nevertheless, aubedding-based approaches for computing the semantic similarity were very successful in past challenges ranking machine translation systems (Freitag et al,, 2022). Based this, expect that we can observe correlations between the semantic similarity and the translation quality, but we also expect that the observed correlations are weaker than those for the other data sets, because aspects like fluency, terminology, style, and locale conventions, are not considered by the semantic similarity.
We report the correlations the same manner as for STS-B, i.c., we visualize the relationship between the human judgments and the semantic similarity and report Pearson's and Spearman's p.
2.6
Tools used
We used the Huggingface transformer library to implement STSScore and the Huggingface evaluation library for BLEU. The RoBERTa model used for STSScore (Held, 2022) was trained with learning rate of
5

---

Published in Transactions on Machine Learning Research (01/2023)
STS-B
MRPC
QQP
WMT22 WMI22-ZH-EN
P
neg
pos
neg
pos
P
BLEU
0.34
0.32
0.26 (0.19)
0.39 (0.20)
0.11 (0.23)
0.18 (0.25)
0.14
0.13
BERISscore
0.53
0.53
0.53 (0.15)
0.68 (0.13)
0.44 (0.26)
0.67 (0.17)
0.32
0.37
S-BERT
0.83
0.82
0.71 (0.15)
0.83 (0.12)
0.56 (0.23)
0.86 (0.10)
0.19
0.24
STSScore 0.90
0.89
0.61 (0.18)
0.84 (0.13)
0.44 (0.24)
0.76 (0.18)
0.27
0.34
Table 1: Summary statistics of the results. Pearson's 's and Spearman's 's between the labels and similarities for STS-B and WM122-ZH-ENN Mean values with standard deviation in brackets for both classes of the MRPC and QQP data. We use neg/ pos to indicate the classes such that pos is the semantically equal class. All values are rounded to the second digit.
Results for STS-B with BLEU
Results for STS-B with BERTScore
1.0
0.75
ooca
Similarity
Similarity
0.50
0.5
0.25
0.0
0.00
00
010:0
GI010
GIGIDIO
0
2
5
Label
Label
Results for STS-B with S-BERT
Results for STS-B with STSScore
1.0
1.0
Similarity
Similarity
0.5
0.5
0.0
2
4
5
Label
Label
Figure 1: Exaluation of similarity measures the test data of STS-B. Ideally, the similarity correlates linearly with the labels, ie., the scores are close to the black line.
2 10- linear scheduler with warmup ratio of 0.06 for 10 epochs using Adam optimizer with 31 0.9, 0.999, and 10-8 using MSE loss. For S-BERT, used the all-MinilM-16-12 which was tuned for high-quality sentence embeddings using a constrative learning model (Rcimers, 2021; Reimers & Gurevych, 2019) and the python package provided by Reimers & Gurevych (2019). For BERTScore we used the default RoBERTa model for the English language and the python package provided by Zhang et al. (2020). We used Scaborn for all visualizations and Pandas for the computation of the correlation coefficients, mean values, and standard deviations. All implementations we created for this work are publicly available online: https://gi com/ saiengglab/stsseee
3 Results
Figure shows the results on the STS-B test data. The STSScore has the expected strong linear correlation with the labels. However, this is not surprising since the underlying model was fine-tuned for this task and the strong performance was already reported through the GLUE benchmark. Still, this confirms that the semantic similarity prediction fulfills the expectations that have semantic similarity measure.

---

Published in Transactions on Machine Learning Research (01/2023)
Results for MRPC with BLEU
Results for MRPC with BERTScore
label
3
label
1.5
Density
Density
2
1.0
1
0.5
0.0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Results for MRPC with S-BERT
Results for MRPC with STSScore
label
label
Density
Density
2
2
1
0
0.2
0.4
0.8
1.0
0.0
0.2
0.4 Similarity
1.0
Similarity
Figure 2: Evaluation of similarity measures on the test data of MRPC. Ideally, the positive class (1) has scores close to one and the negative class (0) has smaller values, but not close to zero.
S-BERT also seems to have the desired linear correlation, but with a general tendency to overcstimate the similarity as most values are above the diagonal. The same cannot be said about BLEU or BERTScore. Both are not aligned at with the expectations from the STS-B task. The values of BERTScore rather seem fairly randomly distributed, the values of BLEU are often exactly zero and otherwise often a lot lower than expected. An optimistic reading of the BERTScore results detects weak upward slope in the similarity scores that would be expected. The correlation coefficients depicted in Table match the results of analysis: STSScore is strongly correlated (r 0.90, p 0.89), followed by S-BERT that is also strongly correlated, but generally a bit weaker than STSScore (r = 0.83, P 0.82). BERTScore has only a moderate correlation (r = 0.53, 0.53) and the correlation of BLEU is weak (r 0.34, p = 0.32).
visual
Figure 2 shows the results for the MRPC data, Table shows the statistical markers. STSScore yiclds the expected results: the paraphrasings have a higher similarity than the negative examples, with lypically high scores (mean=0.84). However, the density plot shows that the scores are not always close one, though only few scores below 0.6. We also observe that the non-paraphrasings are almost always detected semantically somewhat similar (mean 0.61) with a high variability that covers nearly the complete range. However, we note that the density drops to almost zero at very high values (>0.9) and very low values (<0.1). This is in-line with our expectations: the similarity measure typically does not indicate unwarranted equality and it picks up the relationships within the data not dropping to zero. S-BERT is generally similar with high scores for the paraphrasings (mean=0.83).. The distribution looks a bit different from STSScore: while STSScore has a peak at around 0.82 and sccond peak exactly onc, S-BERT has only a single peak at about 0.92, but drops sharply after this peak. The non-paraphrasings have higher semantic similarity for STSScore (mean=0.71),, which aligns with the tendency of S-BERT to overestimate the similarity that we also observed with the STS-B data.
The results of BERTScore exhibit a lot of the expected properties, i.e., a larger mean value for the paraphrases and the similarity for the negative examples covers the whole range, except for the very high and very low values. However, we note that there are only few cases with a similarity close to one for the paraphrasos, even though this is actually the expected value. Moreover, the peak of the distribution is also bit lower than for STSScore and the tail towards lower values for paraphrases is also stronger. As result, the mean
7

---

Published in Transactions on Machine Learning Research (01/2023)
Results for QQP with BLEU
Results for QQP with BERTScore
label
label
1.5
Density
10
Density
1.0
5
0
0.0
0.2
0.4
0.8
1.0
0.0
0.4
0.6
Similarity
Similarity
Results for QQP with S-BERT
Results for QQP with STSScore
2.0
label
label
1.5
Density
3
Density
2
1
0.0
0.0
0.2
0.8
1.0
0.0
0.2
0.6
1.0
Similarity
Similarity
Figure 3: Evaluation of similarity measures on the training data of QQP. Ideally, the positive class (1) has scores close to one and the negative class (0) has smaller values, with only small fraction being close to zero.
ROC for MRPC
ROC for QQP
1.0
0.8
True Positive Rate
0.8
Positive
0.6
0.6
BLEU (AUC = 0.68)
BLEU (AUC = 0.59)
BERTScore (AUC 0.78)
BERTScore (AUC 0.76)
0.2
S-BERT (AUC 0.74)
0.2
S-BERT (AUC = 0.87)
STS (AUC 0.85)
STS (AUC 0.85)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive
False Positive Rate
Figure& 4: Receiver Operator Characteristics (ROC) and AUC measurements for using the semantic similarity metrics classifiers for the MRPC and QQP data. A larger area better.

---

Published in Transactions on Machine Learning Research (01/2023)
value for the paraphrases of BERTScore is only 0.68 for the positive examples. Moreover, these downward shifts in the distributions also lead to a larger overlap between the distributions for the paraphrases and the negative examples, ie., there is a stronger random element in observing large scores with BERTSscore than with STSScore.
For BLEU, the results are not fully aligned with our expectations for the MRPC data. BLEU displays the same tendency for often reporting similarities of exactly zero for both classes that we also observed for STS-B. Similarly, the values for both classes are fairly low, with only a mean similarity for the paraphrases of 0.39. However, the visual analysis shows that this mean value is somewhat skewed by the many values that are exactly zero, as the peak for the non-zero similarities is rather around 0.6. Morcover, both the distribution as well the lower mean value (0.26) indicate that the negative examples receive lower similarity scores, as expected. Still, based on the visual analysis, the distributions of the positive and negative samples strongly overlap, meaning that while the score trends in the expected direction at scale, it is not suited for individual results or calibrated as would be expected for this data.
For QQP, the results for all three models are comparable with respect to their alignment with our expectations to the MRPC: STSScore matches our expectations very well. We observe both a lot of very high values, and overall rather high semantic similarity for the duplicate questions. The mean is a bit lower than for MRPC. However, this seems to be a property of analyzing questions in comparison to regular sentences, as we observe such a downward shift across all classes and similarity measures. We also observe the expected trend towards very low values. S-BERT is a bit of a mixed bag for QQP. On the one hand, the results for the duplicate questions indicate a better measurement of the similarity than for STSScorer. On the other hand, the negative examples also receive higher similarity scores of the same amount and also have their peak at very high similarity of 0.8, though the distribution is spread out such that it is almost uniform between about 0.5 and about 0.8. When we consider these results in the context of the other data sets, this can be explained by the general property of S-BERT to produce higher values for the measurement of the similarity. We note that S-BERT has seen some of the data from QQP during the pre-training, which may be the reason for the very high similarity of the duplicates. This advantage notwithslanding, it seems that STSScore and S-BERT are comparable on the QQP data, with stronger separation observed with STSScore, but higher similarities for duplicates with S-BERT.
BERTScore is again exhibiting a lot of the expected properties but again fails to achieve very large values and has an overall lower similarity for the duplicate questions than STS-B. Same as above, this leads to a larger overlap between the distributions of the classes. The tendency to produce values of exactly zero is strongest for the QQP data. In general, one can say that BLEU failed for this data set: while there is still some difference the mean value, most similarity scores of BLEU are exactly zero for both classes.
Figure 4 provides another perspective on the results for the MRPC and QQP data, i.e., the performance measured through the AUC when using a threshold to directly classify the data given the similarity measures. This perspective augments the distributions we have analyzed above, as the overlap between the distributions is the cause for misclassifiattion for different thresholds. Consequently, a smaller overlap that allows for a better separation between the paraphrases, respectively duplicate questions and leads to a larger AUC. This ROC analysis supports the above conclusions regarding the differences: STSScorer is clearly the best for MRPC, followed by S-BERT, BERTScore, and BLEU, all with notable gaps. For the QQP data, STSScorer and S-BERT perform similar. As we observed before, the other models both perform notably worse.
Figure 5 shows the correlation between semantic similarity measures and the manually determined MQM judgments on the WMT22-ZH-EN data. The embeddings based approaches all correlate somewhat with the MQM judgments, but there is a huge spread within the similarity values for the same MQM judgments. BERTScore achieves the best correlation and has the lowest spread within the visualization. STSScore has a slightly lower correlation, which also shows through a larger spread of similarity values compared to BERTScore. S-BERT performs worse than the other approaches with a notable gap. A potential reason for the stronger performance of BERTScore is that the matching that is performed based on pair-wise similaritics of words may align well with the MQM criteria beyond accuracy, differences on the word level, while STSScore and S-BERT consider the full text at once. We do not have a clear explanation why S-BERT performs so much worse than the other embedding based approaches. All
this allows to mcasure
9

---

Published in Transactions on Machine Learning Research (01/2023)
Results for STS-B with BLEU
Results for STS-B with BERTScore
1.00
1.0
Similarity
Similarity
0.5
0.50
0.25
0.0
0.00
-0.5
-80
-60
40
-2
-80
-40
-20
MQM
MQM
Results for STS-B with S-BERT
Results for STS-B with STSScore
1.00
1.0
0.75
Similarity
Similarity
0.50
0.5
0.25
0.00
0.0
-80
-60
-40
-20
-80
-60
-40
-20
MQM
MQM
Figure 5: Evaluation of similarity measures on the MQM labelled lest data for translations from Chinese to English (ZH-EN) WMT-22 metries challenge. Ideally, the similarity correlates linearly with the MQM labels, i.e, the scores are close to the black line.
embedding approaches clearly performmmm BLEU, which is again only weakly correlated with the expectation and has many similarity values of exactly zero.
4 Discussion
Our most important result is that our experiments support our hypothesis: for three data sets, the transformer-based prediction approach STSScorer aligned best with the expectation from the data, and we suggest using such approaches for future calculations the semantic similarity. The exception is the translation task WMT-22, where STSScorer also aligned with our expectations, but where BoRTScore over- all was better aligned with the human MQM judgments. The weaker correlations on the WMT22-ZH data show that considering only single generic semantic similarity is not sufficient for more complex narios that consider different views semantic similarity (e.g., locale style). For the STS-B, MRPC, and QQP data especially, the S-BERT approach also yields good results, though it seems to have tendency to overestimate the similarity, even on the QQP data which was partially seen during training, BERTScore also yields promising results, but the scores are only moderately correlated with human labeled data (as measured with STS-B) and fail fully capture semanntc equality (as shown with MRPC and QQP). As could be expected, older n-gram basod approaches like BLEU are not comparable and yicld significantly worse results: the distribution of the similarity scores is overall fairly random, although the general trend of the labels still be (weakly) observed. This means that rankings based on BLEU are likely not misleading, if large amount of data is used and the effects between different models are large.
Another important aspect that we want lo stress is that the consideration of the actual distributions of the similarities computed by the different models gave us insights well beyond the statistical markers. Just looking the statistics, we would have missed important aspects like the tendency BLEU towards values of exactly zero, the tendency of BERTScore not yield very large values close one for semantically equal statements, the differences between STSScorer and S-BERT paraphrases within the MRPC data.
10

---

Published in Transactions on Machine Learning Research (01/2023)
We further note that our goal was to evaluate our hypothesis and provide easy method for better semantic scoring. As a consequence, we choose to simply re-use an existing model from Hugginglace instead of training our own model or using a checkpoint from elsewhere. While the model we use is not a lot worse than the top performers 0.93), it cannot be considered as stale-of-the-arl for STS-B. Nevertheless, our results hold, ie., STSScorer is a very good approach, though plugging in other models fine-tuned for semantic similarity prediction might yield (slightly) better results. We also note that Reimers & Gurevych (2019) also considered a fine-tuned version of S-BERT on the STS-B data by using the STS-B scores divided by five (same as us) to define cosine similarity loss, which seemed to improve the alignment. with STS-B when computing the cosine similarity. Nevertheless, based on the results reported by Reimers & Gurevych (2019), using the predictions directly should still be a better similarity measure than computing the cosine similarity of the embeddings. However, embeddings have the advantage, that they also enable other kinds of analysis, e.g., clustering or visualizations.
the GLUE benchmark (best performing models in STS-B achieve correlations of up to
Another aspect to consider when computing the semantic similarity is the computational effort. The three embedding- based approaches are almost the same in terms of computational effort, as they are all based on the BERT architecture. As the authors of BERTScore note, the speed of is only slightly slower than that of BLEU (Zhang et al., 2020). Thus, the efficiency is not a major factor when choosing between these models.
5 Limitations
Even though we believe that STSScorer is currently the best method for computing similarities, there is also an important drawback: when we use transformer-based approaches to evaluate other likely transformer- based approaches, the evaluation will possibly have all the problems regarding biases that transformers have, Thus, we may further encode these biases, because we selected models based on a biased evaluator.
Additionally, all the current work ignores that semantic similarity is not absolute and also depends the perspective of humans and can, e.g., be influenced by culture, social background, and other aspects, raising the question of whether we should rather specify our notion of similarity more carefully in the future and use multiple similarity measures that can capture such diffcrences. This is also shown on the WMT22-ZZ-EN lask, where not only the accuracy influences the translation quality, but also other aspects like the use of locale conventions.
Another notable limitation is that semantic similarity data needs to be available to target languages, i.c., we camnot easily use model that is fine-tuned on the English STS data to compute the semantic similarily in other languages, or even between languages. Approaches based on the cosine similarity of embeddings are more flexible in this regard, as they only require a suitable foundation model for the language, possibly augmented with further self-supervised pre-training as done by S-BERT (Reimers & Gurevych, 2019).
6 Conclusion
The jumps in performance for natural language processing enable us directly predict the semantic similarity instcad of using embcdding- basod approaches or hcuristics. Due the readily available models on platforms like Huggingface, switching to such models for the future evaluation of the semantic similarity of results should be easily possible.
Future work should address the current limitations, most notably, the general problem capturing semantics with a single number regardless of the approach! is likely problematic. To resolve this, similar models to STSScore can be created that consider different aspects of semantics, e.g., for different aspects of translation quality based on MQM (Burchardt, 2013) or based on Leech's seven types of meaning (Loech, 1974). Any such approach should carefully consider potential biases and their effect on the created semantic similarity measures.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Broader Impact Statement
For many low-level tasks, we now have very good classification and regression models. We should consider using these models more often as perlormance measures for more complex lasks. We demonstrate this this paper by showing that fine-tuned semantic similarity models are better at capturing expected characteristics of semantic similarity than relying only on the cosine-similarity of embeddings. However, when doing this, risk that biases and limitations from the fine-tuned model affect the evaluation of complex task in an adversarial manner, which should also be further studied.
References
Aljoscha Burchardt. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translaling and the Compuler 35, London, UK, November 28-29 2013. Aslib. URL https://acllanthology
tc-1
Daniel Cer, Mona Diab, Eneko Agirre, Tnigo Lopez-Gazpio, and Lucia Specia. SemFval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semanlic Evaluation (SemEval-2017) pp. 1 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2000 URL https: / //aclanthology. org/S17-2201..
Marco Del Tredici and Núria Bel. A word-embeeddingg-bset sense index for regular polysemy representation.
In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 70-78, Denver, Colorado, June 2015. Association for Computational Linguistics. doi: 10.3115/v1/W15- 1510. URL https://aclantholloo wwt.:/ooooooooo/1---1111 org/#15-1510.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653.. 15653/v1/N199111/ URL https://aclanntoooogy http:////aatttoloooooo//o///a/20 org/N19-1423,
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URI. https:// aclanthology. srg/105-5002.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. Results of WMT22 metrics shared task: Stop using better and more robust. In Philipp Koehn, Loie Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yopes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 46 68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL http:///wggggaaacccoo//llll/////
BLEU neural metrics
org/2022. wmt-1.2.
William Held. https://huugiingffceccom
2022.
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs, 2017. URL attps:///qorraatt..quoot www.a. htmm/First-uuora.. Dataaset-Reeesse-uueeeee
Alon Lavie and Abhaya Agarwal. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pp. 228-222-22222 Prague, Czech Republic, June 2007. Association for Computational Linguistics.
URL http://cclaaooolooooooor///////0
12

---

Published in Transactions on Machine Learning Research (01/2023)
Geoffrey Leech. Semantics: The Study of Meaning. Penguin Books, 1974.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Texl Summarizalion Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aclanthology. hwt.aaatooooooo.orr/AA-------0--
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrxiv: 1692, 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistios. doi: 10.3115 1/11130033 1073135. URL attps://aclantae http:://ccccaaaoooooooor/////-/s wwwp:/yorr///P/1///000/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nyya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nils Reimers.
2021.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence emboddings using Siamese BERT-networkss
In Proceedings of the 2019 Conference on Empirical Methods in Natural Longuage Processing and the 9th International Joint Conference on Natural Language Processing (FMNL.PIICNNPP), pp. 3982-3992 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD 1410.. URL https:
1ttph/oooooooor//1111110000 org/D19-1410.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised metrics in task-oriented dialogue for ovaluating natural language generation. arXiv preprint arXiv: 1rXii:1706.000 0979.9 2017.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. RUSE: Regressor using sentence embeddings for automatic machine trauslation evaluation. In Proceedings of the Third Conference on Machine Trans- lation: Shared Task Papers, pp. 751-755, Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10.18653/11W1186445 URL https: http:p:://aaaathloor/// org Org/W18-6456.
Miloš Stanojevié and Khalil Sima'a Sima'an. BEER: BEtter ovaluation ranking. In Proceedings of the Ninth Work- shop on Statistical Machine Translation, pp. 414-419 Baltimore, Maryland, USA, June 2014. Association URL https::/acllatthoolgy/ org/
for Computational Linguistics. doi: 10.3115/v1 /W14-3354.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you noed. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackbouNLP: Analyzing and Interpreling Newral Nelworks for NLP, pp. 353 355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-5446. URL https://aclanthon wrt.:/anthoollgyooo//o1-//----
org/W18-5446.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:/ pttps::/proceediing pagroceedingg..uurpp...... wtt.c//peee.eeeeepppp//////////e /ttpr/////////////////////////
13

---

Published in Transactions on Machine Learning Research (01/2023)
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Represenlations, 2020. URL https:
/
A Results of additional experiments
A.1 Results when using BERT-base for STSScorer
In order to compare how much impact our choice of using a fine-tuned RoBERTa model has on the results, we provide a comparison with a BERT-base model instead. Table shows the performance comparison 9 show the plots for the BERT-base model. The results indicate a clear gap, i.e.,
to RoBERTa. Figures the better score on the STS-B data of the larger and better trained RoBERTa model robustly translates into better semantic similarity estimation.
A.2 Results when using an ensemble for scoring
We also provide results for using an ensemble of STSScore, BERTSccree, and S-BERT. For this ensemble, we simply average the similarity of the three embedding-based approaches. Table shows the performance in comparison to the individual embedding based approaches. Figures 10 to 13 show the plots in comparison to the individual models. The ensemble is always between the two best models, ie., it has some upside regarding robust but in general, it seems better to use STSScore directly.
A.3 Impact of sequence length
Table 4 shows the impact of the sequence length measured in the number of characters on the STSScore. For this, we split the data into two (about) equally large subsets, such that the sequences shorter than the median are in one subset and the sequences equal to or longer than the median in the other. The median sequence lengths are 44 for STS-B, 116 for MRPC, 52 for QQP, and 123 for WMT22-ZH ZH-EN. We do not obsorve a stable trend when considering the impact on the length. For WMT22-7H-EN, the values on both subsets are larger, which indicates that this may be case of the Simpson paradox.
STS-B
MRPC
QQP
WMT22-2H-EN
0
neg
pos
neg 0.44 (0.24)
pos
r
P
RoBERTa
0.90
0.89
0.61 (0.18)
0.84 (0.13)
0.76 (0.18)
0.27
0.34
BERT-base
0.83
0.82
0.62 (0.18)
0.79 (0.11)
0.46 (0.23)
0.69 (0.15)
0.08
0.18
Table 2: Summary statistics of the results for the STSScore with different foundation models that were fine-tuned for the STS-B data. Pearson's and Spearman's between the labels and similarities for STS-B and WMT22-ZH-EN, Mean values with standard deviation in brackets for both classes of the MRPC and QQP data. We use neg/pos to indicate the classes such that pos is the semantically equal class. All values are rounded to the second digit.
STS-B
MRPC
QQP
WMT22-ZHH
r
0
neg
pos
neg
pos
T
P
BERTSccore
0.53
0.53
0.53 (0.15)
0.68 (0.13)
0.44 (0.26)
0.67 (0.17)
0.32
0.37
S-BERT
0.83
0.82
0.71 (0.15)
0.83 (0.12)
0.56 (0.23)
0.86 (0.10)
0.19
0.24
STSScore
0.90
0.89
0.61 (0.18)
0.84 (0.13)
0.44 (0.24)
0.76 (0.18)
0.27
0.34
Ensemble
0.88
0.87
0.62 (0.13)
0.78 (0.10)
0.48 (0.24)
0.77 (0.13)
0.30
0.35
Table 3: Summary statistics of the results. Pearson's and Spearman's p between the labels and similarities
for STS-B and WMT22- SMTT2--ZH-EN ZH-EN. Mean values with standard deviation in brackets for both classes of the MRPC and QQP data. We use neg/pos to indicate the classes such that pos is the semantically equal class. All values are rounded to the second digit.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Results for STS-B with STSScore with BERT base
ane saoe e $ one
en dn mi te
uef 600 10000
Similarity
0.4
18E 000
0.2
0.0
Label
Figure 6: Evaluation of similarity measures on the test data of STS-B with BERT-base. Ideally, the similarity correlates linearly with the labels, i.e., the scores are close to the black line.
Results MRPC with STSScore with BERT base
label
Density
2
1
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Figure 7: Evaluation of similarity measures on the test data of MRPC with BERT-base. Ideally, the positive
class (1) has scores close to and the negative class (0) has smaller values, but not close
STS-B
MRPC
QQP
WMMT2-ZZHER
P
ney
po8
neg
pos
A1l
0.90
0.89
0.61
(0.18)
0.84 (0.13)
0.44 (0.24)
0.76 (0.18)
0.27
0.34
Shorter
0.90
0.90
0.57 (0.18)
0.80 (0.14)
0.48 (0.24)
0.78 (0.18)
0.30
0.31
Longer
0.89
0.88
0.68 (0.15)
0.86 (0.12)
0.41 (0.23)
0.74 (0.18)
0.40
0.42
Table 4: Summary statistics the results. Pearson's and Spearman's 's between the labels and similarities for STS-B and WMT22-7H-FNN Mean values with standard deviation brackets for both classes of the MRPC and QQP data. We neg/po pos to indicate the classes such that pos is the semantically equal class. All values are rounded to the second digit.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Results for QQP with STSScore with BERT base
label
2.0
1.5
Density
1.0
0.5
0.0
0.0
0.2
0.4
0.6
0.8
Similarity
Figure 8: Evaluation of similarity measures on the training data of QQP with BERT-base. Ideally, the positive class (1) has scores close to and the negative class (0) has smaller values, with only small fraction being close to zero.
ROC for MRPC
ROC for QQP
1.0
1.0
True Positive Rate
Positive
0.6
0.6
0.4
0.4
0.2
0.2
STS BERT Base (AUC 0.81)
STS BERT Base(AUC 0.80)
0.0
0.0
0.0
0.4 False Positive Rate
0.2
0.8
False Positive Rate
Figure 9: Receiver Operator Characteristics (ROC) and AUC measurements for using the semantic similarity metrics as classifiers for the MRPC and QQP data with BERT-base. larger area is better.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Results for STS-B with an ensemble of BERTScore, S-BERT and STSScore
Results for STS-B with BERTScore
Similarity
0.75
Similarity
0.50
0.5
0.25
0.0
0.00
0
1
2
5
Label
Label
Results for STS-B with S-BERT
Results for STS-B with STSScore
1.0
Similarity
Similarity
0.5
0.0
0.0
0
1
2
5
0
4
5
Label
Label
Figure 10: Exaluation of similarity measures the test data of STS-B with an ensemble of the embedding based approaches. Ideally, the similarity correlates linearly with the labels, i.e., the scores are close to the black line.
Results for MRPC with an ensemble of BERTScore, S-BERT and STSScore
Results for MRPC with BERTScore
label
label
Density
Density
2
1
1-
0.8
1.0
0.2
0.4
Similarity
Similarity
Results for MRPC with S-BERT
Results for MRPC with STSScore
label
label
Density
Density
0.2
0.4 Similarity
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Figure 11: Evaluation of similarity measures on the test data of MRPC with an ensemble of the embedding based approaches. Idcally, the positive class (1) has scores close to and the negative class (0) has smaller values, but not close to zero.
17

---

Published in Transactions on Machine Learning Research (01/2023)
Results for QQP with an ensemble of BERTScore, S-BERT and STSScore
Results for QQP with BERTScore
label
2.0
label
0
1.5
Density
Density
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Results for QQP with S-BERT
Results for QQP with STSScore
2.0
label
label
1.5
0
Density
Density
1.0
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Similarity
Similarity
Figure 12: Evaluation similarity measures on the training data of QQP with an ensemble the embedding based approaches. Ideally, the positive class (1) has scores close one and the negative class (0) has smaller values, with only small fraction being close to zero.
ROC for MRPC
ROC for QQP
1.0
1.0
True Positive Rate
0.8
True Positive Rate
0.8
0.6
0.4
Ensemble (AUC 0.83)
0.4
Ensemble (AUC 0.85)
BERTScore (AUC = 0.78)
BERTScore (AUC 0.76)
0.2
S-BERT (AUC 0.74)
0.2
S-BERT (AUC 0.87)
STS (AUC 0.85)
STS (AUC 0.85)
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
Figure 13: Receiver Operator Characteristics (ROC) and AUC measurements for using the semantic similar- ity metrics as classifiers for the MRPC and QQP data with an ensemble of the embedding based approaches.
A larger area better.
18


Reviews:
[{"review_id": "9BaRLcELL9", "reviewer": "Reviewer_aEt1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is well-written, effectively beginning by covering the motivation behind using a fine-tuned model for predicting semantic similarity. \n- A comprehensive discussion is presented of the current methods for determining semantic similarity, which makes it clear why a new approach would be beneficial.\n- The hypothesis---that using transformers for measuring semantic similarity might outperform other methods---is clearly stated and well-motivated. \n- The authors provide a thorough approach to test their hypothesis, including calculations, visualizations, and detailed explanations of their analysis.\n- The method, STSScore, itself appears promising, as it's based on a robust fine-tuned RoBERTa model.\n\n## Weaknesses:\n- The use of only one model (RoBERTa) without comparing the performance of other transformer architectures may lead to biased conclusions. Other models could have potentially affected results.\n- The paper focuses only on the English language. It would be interesting to see a cross-linguistic analysis of the proposed method.\n- There is very little discussion about the limitations of the proposed method", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel method for predicting semantic similarity between natural language texts called STSScore, leveraging a fine-tuned model for the task from the GLUE benchmark. The authors discuss existing methods for measuring semantic similarity, such as BLEU and methods based on various embeddings like BERTScore and S-BERT, and argue for a regression model instead. The proposed model, STSScore, is built upon a fine-tuned RoBERTa model for the STS-B task. The authors conducted a confirmatory study to test their hypothesis that their approach, based on predicting the semantic similarity directly, is better aligned with expectations of a robust semantic similarity measure. In their analyses, they compared results from BLEU, BERTScore, S-BERT, and STSScore to evaluate the relationship between different measures across various tasks such as STS-B, Microsoft Research Paraphrase Corpus (MRPC), and Quora Question Pairs (QQP) data. The findings mostly aligned with their hypothesis, showcasing the potential of STSScore for semantic similarity tasks, although a detailed comparison against other potential models was not extensively discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparisons: Can you include comparisons using models other than RoBERTa if that's feasible?\n\n2. Multilingual Analysis: I'd recommend the authors to extend their approach to languages other than English to test the robustness and versatility of STSScore.\n\n3. Discuss Limitations: The paper has a distinct lack of discussion about limitations. Every approach has its limitations, and discussing them keeps the research balanced and provides a fuller picture of the method to the readers.\n\n4. Computation Efficiency: Understanding the computation cost of the proposed approach in comparison to others is important. Please include a discussion or a benchmark of the computational efficiency of STSScore.\n\n5. Future Work: Indicate directions for future research, such as how to improve the existing model or if there are any alternative approaches you would like to explore in the future.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GXSXsXT3cm", "reviewer": "Reviewer_KAVy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The problem of measuring semantic similarity is an important one in the field: Better semantic similarity metrics would enable us to more accurately evaluate our current models, truly understand the extent of our progress, and also enable us to design better retrieval systems and their various applications (e.g. which of the documents that we have are the most / least similar to this particular document that we care about?).\n\n2. The approach is very simple and easy to implement, and leverages an off-the-shelf pre-trained RoBERTa model that is fine-tuned on STS-B, which is readily available for the general public on libraries like HuggingFace.\n\n3. The research hypothesis is clearly stated on page 2, which helps crystallise what the claim is and what is being done.\n\n**Weaknesses**\n\n1. The main weakness of this paper is that the empirical scope of the paper is very narrow, and more concerningly, does not validate the claim that is being made. Concretely, the claim is that the semantic similarity scores from STS-B can be **broadly and generally** applied to evaluate the semantic similarity of two natural language texts, even in other domains that are quite different from the STS-B dataset. However, the paper only evaluates on the STS-B test set (which is in-domain), in addition to MRPC and QQP. The problem is that the MRPC and QQP datasets are: (i) both semantic similarity prediction tasks (which may not be representative of other types of NLP tasks), and (ii) only cover newswire text (MRPC) and web text (QQP). To what extent would the proposed semantic similarity metric work well for evaluating, e.g. machine translation outputs, or text summarisation outputs against the reference, human-generated output? And how would this metric compare with BLEU (default evaluation metric for machine translation) and ROUGE (default metric for text summarisation) for those tasks? And can the metric work well when being used to evaluate other domains / more noisy natural language text, such as Reddit / Twitter? Given the small size of the STS-B training set, I am skeptical that RoBERTa fine-tuned on STS-B would necessarily provide better evaluation metrics for other important NLP tasks, use cases, and domains (unless otherwise empirically shown, which the paper in its current form has not done yet by virtue of evaluating only on 3 mostly-similar semantic similarity prediction tasks that are not representative of many different NLU & NLG system use cases).\n\n2. For the case where the metric has to be used to provide negative or positive labels (e.g. given text \"A\" and \"B\", output a label of 1 if they are semantically paraphrases, or \"0\" otherwise), it is still very much unclear how we can **set the appropriate threshold**, which might differ from dataset to dataset. For instance, for MRPC, an STSScore of 0.7 and above should mean a label of 1 (paraphrase), whereas an STSScore of <0.7 should mean a label of 0 (not paraphrase). Can we come up with such a decision rule using this metric? If so, how should we determine this threshold, which would likely be dataset-specific? And how well do we perform on the binary classification task after setting that threshold (i.e. what MRPC accuracy do we get when using the threshold to determine whether or not two texts are paraphrases of one another?). \n\n3. Given the simplicity of the approach (which is a good thing as simple techniques can be adopted more easily), I would expect there to be more substantive content & analyses given the fairly generous 10-page content, such as: (i) experiments on more tasks (including on evaluating generation-based tasks like machine translation, summarisation, etc. which would be very useful to validate the claimed generality of the approach), (ii) ablation studies (how does the similarity metric performance differ for long vs short text, etc.), (iii) experiments with combining multiple metrics (e.g. can we combine STSScore and S-BERT to yield a better semantic similarity metric than either one of them alone), etc. In this paper, 2 full pages (pages 6 & 9) are dedicated to plots that could be much smaller and stacked together; this would then free up space for more analyses, etc. Also, a lot of the space is dedicated to explaining existing tasks (MRPC & QQP) in great detail, which is not central to the paper's contributions and could be described more succinctly in the main paper, with the details deferred to the Appendix.\n\n4.  For Figures 2 & 3, it would be nice to see a quantification of the \"overlap area\" between the negative and positive labels, with the assumption that the less overlap -> the better (i.e. the distributions of similarity for positive and negative classes are quite distinct from one another). This would allow us to better quantify the superiority of the STSScore compared to the baseline metrics.\n\n5. The paper has a non-trivial amount of typos and grammatical mistakes, and could benefit from careful copy-editing.\n\n**Questions and Suggestions**\n\n1. On page 1, it is stated that \"The transformer architecture enabled the context-sensitive calculation of embeddings\". This is incorrect: One can compute context-sensitive calculation of embeddings from other models like LSTMs or bidirectional LSTMs, for instance, and still have pretty good results. See the contextual word embedding of the ELMo model of Peters et al. (2018), which was originally based on bidirectional LSTMs. So the ability to do context-sensitive calculation of embeddings is not at all specific to Transformers. However, the Transformer architecture enables us to **do this at scale** by using a larger model size and pre-training on a larger dataset (e.g. compared to LSTMs), because we can parallelise across the time dimension and make better use of specialised hardwares like GPUs.\n\n2.  Some of the figures take up way too much space (e.g. Figures 2 and 3). It would be better to make them smaller and combine them to make room for other analyses, etc.\n\n**Typos / Presentation Suggestions**\n\n1. In page 1, \"ROGUE\" should be \"ROUGE\".\n2. In page 4, \"were created independent ...\" should be \"were created **independently** \".\n3. In page 4, \"As example, consider ...\" should be \"For example\" or \"As an example\" . \n4. In page 8, \"The questions indicates\" -> \"indicate\".\n5. In page 8, \"is again exhibit\" is a grammatical error.\n6. In page 10, \"a lot worse then\" -> \"than\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of **measuring semantic similarity**: Given two natural language texts A and B, we would like to measure how similar A and B are in terms of their meaning (irrespective of any superficial similarities or differences in the lexical surface form). Given the centrality of this problem in NLP --- particularly for evaluating how \"good\" / similar a model's output is with respect to a human-generated reference output --- various metrics have been proposed to address this problem throughout the years. Whereas earlier solutions have mostly focused on **lexical** similarity and overlap (e.g. BLEU, ROUGE, etc.), recent work has demonstrated the superiority of **contextual embedding-based similarity metrics**, such as sentence-BERT. Unlike metrics that are based on surface lexical overlap and similarity, these similarity-based metrics are instead computed by contextually encoding each text with a Transformer model, and measuring the similarity between the two contextual encoding of each text in the dense vector space.\n\nTo that end, this paper proposes a new, simple way of measuring semantic similarity. Rather than directly measuring the contextual embedding similarity in the dense vector space with an off-the-shelf pre-trained model like BERT or RoBERTa directly, the paper proposes to instead use the resulting similarity metric from a **RoBERTa model that is fine-tuned on the STS-B task**. As the STS-B task: (i) is designed to train the model to do well on semantic similarity prediction, and (ii) requires the model to output a semantic similarity score that ranges from 1 to 5, the paper argues that the output ratings from a fine-tuned RoBERTa model on STS-B (denoted as **STSScorer**) provides a natural, simple, and off-the-shelf way for measuring textual similarity, even when it is applied to other, more general text domains beyond the STS-B dataset. By directly leveraging the STS-B semantic similarity rating scale, this method circumvents the need for choosing additional vector similarity metrics, such as cosine similarity.\n\nExperiments on STS-B, MRPC, and QQP suggest that the resulting similarity score distributions from the STSScorer is better or equally correlated with human judgment compared to other alternative metrics and baselines, such as BERTScore and Sentence-BERT.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Critical**: Evaluate the metric on a much broader range of tasks, including for evaluating machine translation outputs vs the gold reference, and also for text summarisation, etc. This will empirically validate whether or not the proposed metric is truly a general-purpose metric for conducting semantic similarity on text, rather than something that only works well for text domains and NLP tasks that are similar to STS-B. It would also be nice to compare against existing contextual-embedding-vector-based metric for these tasks,for which prior work has dedicated a lot of work in the past (e.g. the WMT Shared Task on Quality Estimation).\n\n2. **Critical**: Report performance on tasks with binary labels (e.g. MRPC, QQP), where we need to discretise the predictions (e.g. an STSScore of >0.7 means the paraphrase label is positive, <0.7 means negative, etc.). Also come up with a scheme that would enable us to set the appropriate threshold for such prediction on a dataset-by-dataset basis, because arguably the appropriate threshold will be different for each dataset.\n\n3. **Critical**: Add additional analyses and ablation studies, such as how good the similarity metric is for long vs short sentences, whether or not we can ensemble & combine STSScore with other metrics to get a better distribution that more closely reflects the human similarity rating distribution, etc.\n\n4. **Recommended**: Incorporating the suggestions and resolving the questions that I raised above, and also careful copy-editing of the paper.", "broader_impact_concerns": "No broader impact concerns come to mind.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ec2EIOenDA", "reviewer": "Reviewer_2kRv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The experimental setup is quite thoroughly described, including definitions of the hypotheses and potential confoundres.\n- Examining the predicted distributions of similarity is interesting. \n\nWeaknesses:\n- The contribution is narrow. The model evaluated is already trained for the task it is evaluated on, and just downloaded from HF. \n- Semantic similarity is notoriously subjective (Goldstone et al. 1997, Similarity in context). This is a bigger problem with the framing of semantic similarity evaluation that goes beyond this particular paper, as discussed at the end of section 5. But more discussion throughout would be nice. E.g., it's unclear what labels the pairs of sentences in 3.0.2 _should_ have, because it's so relative.\n- The evaluation is quite narrow scope, just on three datasets which are essentially for paraphrase detection. Meanwhile, BERTScore also evaluates on tasks like machine translation and image captioning. Why not evaluate the proposed metric on more downstream tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates the use of BERT-based models fine-tuned for regression on predicting semantic similarity. The paper investigates how well this approach can serve as a metric for similarity for tasks like paraphrasing and compares against other common approaches like BLEU and BERTScore. The paper finds that the regression-based approaches outperform existing metrics on a number of these tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major changes:\n* Especially because the approach itself is not proposed by the paper, there should be much more thorough analysis of the metric beyond what is provided in the paper. E.g., evaluating on the myriad other tasks that rely on measures of similarity, like machine translation or image captioning.\n\nMedium changes:\n* STSScorer should at the very least be described in more detail in terms of how it was trained, even if the authors didn't train it. E.g., it is finetuned with a regression objective, right?\n\nMinor changes:\n* \"STS-B\" should be introduced before using the acronym / referring to it, especially in the abstract.\n* Typo: \"BiLinigual Evaluation Understudy\" --> \"Bilingual\"\n* Typo: \"since its publications in 2020\" --> \"since its publication in 2020\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies the effectiveness of semantic similarity prediction as a more general metric. It is overall a rather narrow contribution, but it documents valuable evidence that may be useful for some of the community. The revision has also substantially improved the presentation. This paper thus clears the TMLR criteria and I recommend acceptance after a few minor revisions:

1) please fix the typo in the "Hypothesis" block which reads "capable oh robustly measuring"
2) please include a brief appendix in the PDF documenting the new results mentioned in the responses ("our online materials"). This is important for completeness and archival purposes.
Decision: Accept (poster)