Improving word mover's distance by leveraging self-attention matrix
Authors: Hiroaki Yamagiwa
Sho Yokoi
Hidetoshi Shimodaira
Venue: EMNLP 2023 Findings (2023)
Keywords: word embeddings
word mover's distance
optimal transport
Gromov-Wasserstein distance
Fused Gromov-Wasserstein distance
Self-Attention
paraphrase identification
semantic textual similarity
TL;DR: By incorporating BERT's self-attention matrix (SAM) and the Fused Gromov-Wasserstein distance, the proposed method improves WMD by considering both word embeddings and sentence structure.

Abstract: Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similarity via the optimal alignment between the sets of word embeddings. However, WMD does not utilize word order, making it challenging to distinguish sentences with significant overlaps of similar words, even if they are semantically very different. Here, we attempt to improve WMD by incorporating the sentence structure represented by BERT's self-attention matrix (SAM). The proposed method is based on the Fused Gromov-Wasserstein distance, which simultaneously considers the similarity of the word embedding and the SAM for calculating the optimal transport between two sentences. Experiments demonstrate the proposed method enhances WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity.

press Improving word mover's distance by leveraging self-attention matrix
Hiroaki Yamagiwa'
Sho Yokoi 2,3
Hidetoshi Shimodaira 1,3
Kyoto University hiroaki. yywaaiiaasssaaaaaa..........a kww.:/aassy....oou.uuu..... jp, yokoiêtchoku. ac. jp, shimoli kyoto-u.aa jp
Tohoku University
RIKEN AIP
kyoto-u.
Abstract
sentence obama speaks to the
obama
sentence2
the
Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similar- ity via the optimal alignment between the sets word embeddings. However, WMD does not utilize word order, making it challenging to distinguish sentences with sigmificant overlaps of similar words, even if they are semantically very different. Here, attempt to improve WMD by incorporating the sentence structure represented by BERT's self-attention matrix (SAM). The proposed method is based on the Fused
press
media greets
president
the
illinois speaks media in illinois
president in chicago
chicago
greets
of WMD
(proposal) of SMD WSMD
Figure 1: An illustration of OT for word embeddings from sentence I sentence Words are aligned by word similarity in WMD; e.g., obama matches president. Words are aligned by sentence structure in SMD or by word similarity and sentence structure simultaneously in WSMD; e.g., obama matches press. See Section 2.
distance, which si- multaneously considers the similarity of the word embedding and the SAM for calculating the optimal transport between two sentences. Experiments demonstrate the proposed method enhances WMD and its variants in paraphrase identification with near-equivalent performance in semantic textual similarity. Our code is avail- able at https: /ttp:::///gtttttt github. com/ ymgw55/ WSMD.
the sets (Kantorovich, 1960). Word Mover's Dis- tance (WMD) computes Wasserstein distance by considering sentence a set of word embed- dings (Kusner et al., 2015).
However, WMD does not consider the word or- der of sentences, making it challenging to identify paraphrases. Let us see the following illustrative example of the paraphrases adapted from (Kusner et al.. 2015) and (Zhang et al., 2019):
1 Introduction
The task of measuring the semantic textual similar- ity (STS) of two sentences is essential for natural language processing with various applications (Cer et al., 2017). There are several methods for mea- suring STS, among which many methods using Optimal Transport (OT) distance have been pro- posed and have shown good performance (Kusner et al., 2015; Huang et al., 2016; Chen et al., 2019; Yokoi et al., 2020).
(a) Obama speaks to the media in Illinois.
(b) The President greets the press in Chicago.
(c) The press greets the President in Chicago.
Here (b) a paraphrase of (a), while (c) has a very different meaning from (a). However, all these sentences have a high overlap of words. Thus, WMD cannot distinguish these pairs.
OT theory gives a method to measure the dif- ference between two distributions by setting the transport cost of a unit mass and considering the allocation of the transported mass to minimize the total cost. This allocation is called optimal trans- port, and the total cost called the OT distance. basic form of OT distance the Wasserstein distance, which measures the similarity between sets using the distance between the elements in
To account for sentence structure, focused on BERT (Devlin al., 2019), attention-based model that has recently achieved remarkable perfor- mance in natural language processing tasks. BERT is a masked language model that performs task- specific fine-tuning after pre-training on a large

---

press
employs the Fused Gromov-Wassersed (FGW) distance (Vayer al., 2019, 2020), known the op- timal transport distance that combines both Wasser- stein and GW distances.
obama
speaks
greets
media
president
speaks
Illinois
chicago
media
press
Illnols
greets
president
Illustrative example
sentence1
sentence2
chicago
OT of WMD
Here we explain how our proposed method works for the example of the paraphrases in Section 1. The original WMD and its improvement with our proposal are computed for this example and shown in Table 1. We computed these values using the word embeddings taken from the input layer of BERT and the SAMs taken from the second head of the eighth layer of BERT. The details of the setting will be described in Section 7.2. The similarity should be high for the sentence pair (a) vs. (b), i.c., case (1), while the similarity should be low for the sentence pair (a) vs. (c), i.e., case (2).
(proposal) OT of SMD or WSMD
An illustration of OT for SAMs from sen- tence to sentence To avoid crowding the diagram, arrows are shown in only two SAM elements. The darker the color, the higher the value of the SAM el- ement (reflecting a real SAM used in Table 1). The difference of SAM values is small for the word align- ment of SMD or WSMD; e.g. (obama, speaks) matches (press, greets). The difference large for that of WMD; e.g., (obama, speaks) matches (president, greets). See Section 2.
Figure 2:
Looking at the values of the original WMD for the two cases, we find that they are about the same, Thus, WMD failed to give a reasonable sentence similarity. This failure is explained in Fig. 1 by illustrating word embeddings for case (2). WMD tries to find the closest word for each word, match- ing obama to president, media to press, and on; the word alignment is indicated as the short dotted arrows. Then, WMD is computed by averaging the length of these short arrows, indicating a high sen- tence similarity contrary to the fact that the actual similarity is low for case (2).
dataset. By inputting sentence into the pre- trained BERT, we can extract the Self-Attention Matrix (SAM), which represents the dependencies between words in the sentence. SAM encodes sen- tence structure information, such as syntactic in- formation (Clark et al., 2019; Htut et al., 2019; Luo, 2021). Therefore, sentence structure can be incorporated into STS by measuring the distance between SAMs.
We propose novel method called sentence Structure Mover's Distance (SMD) in Section 5 that measures the optimal transport distance be- tween sentence structures represented in SAMs. SMD aims to find the optimal word alignment by considering the transport cost between elements of SAMs, while WMD considers the transport cost between word embeddings. To compute SMD, we employ the Gromov-Wasserster (GW) dis- tance (Mémoli, 2011; Peyré and Cuturi, 2020), the optimal transport distance for measuring structural similarity between sets.
However, the word alignment for WMD is in- appropriate, given the word order and sentence structure. Our proposed method correctly matches obama to press and media to president. This word alignment is shown as the long arrows with solid lines in Fig. 1. WMD, in Table is computed by averaging the length of these arrows, i.e., the same formula as WMD but using the OT taking account of the sentence structure. We find that WMDX increases for case (2), correctly indicating low sentence similarity.
SMD is ineffective by itself because measures only the difference between sentence structures. Thus SMD is used in combination with WMD. We improve WMD- methods (i.e., WMD and its variants based on Wasserstein distance between sets of word embeddings) by combining SMD with them in Section 6. This proposed method is called Word and sentence Structure Mover's 's Dis- tance (WSMD). WSMD simultaneously measures the difference between word embeddings and the difference between sentence structures. WSMD
Fig. 2 explains how our proposed method takes account of the sentence structure. Given a word alignment between two sentences, we can think of matching of elements between the two SAMs by upplying the word aligmment to both the rows and the columns of the matrices. For example, the word alignment of WMD in Fig. induces the matching from (obama, speaks) to (president, greets), (me- dia, speaks) to (press, greets), and so on. SMD minimizes the average difference of SAM values

---

case
sentences
WMD
WMDx
WSMD
similarity
(a) obama speaks to the media in illinois.
12.54
12.54
7.26
high
(b) the president greets the press in chicago.
(a) obama speaks to the media in illinois.
(2)
12.55
13.30
8.03
low
(c) the press greets the president in chicago.
Table 1: Similarity measures for sentence pairs (a) vs. (b) and (a) vs. (c) of Section
between the matched elements. This recovers the correct word alignment in this example. WSMD
The weight and the cost in SynWMD are called Syntax-aware Word Flow (SWF) and Syntax-aware Word Distance (SWD), respectively. In Recursive Optimal Transport Similarity (ROTS) (Wang et al., 2020, 2022), a recursive optimal transport method is introduced to capture word dependencies in sen- tences. To mcasure sentence similarity, the pair- wise cosine similarity of words is weighted using this optimal transport and aggregated. In Mover- Score (Zhao et al., 2019), the inverse document frequency (IDF) used for the word weight, and the word embeddings from BERT are used for com- puting the cost. BERTscore (Zhang et al., 2020a) also uses IDF and BERT embedding, but it employs greedy matching instead of OT for word alignment.
minimizes the weighted sum of the objectives for WMD and SMD; we used the mixing ratio À = 0.5 here. As shown in Table 1, WSMD correctly indi- cates that the sentences in case (2) are less similar than those in case (1).
3 Related Work
This paper focuses on sentence similarity measures using the OT of word embeddings. For example, WMD (Kusner et al., 2015) uses uniform word weights and the L2 distance of word embeddings as the transport cost. By modifying the word weights and the transport cost, various WMD-like methods have been developed to compute sentence similar- ity based on the Wasserstein distance between sets of word embeddings. For example, Word Rotator's Distance (WRD) (Yokoi et al., 2020) uses the vec- tor norm of the word embedding for the weight and the cosine similarity for the cost. There are many attempts improve WMD by incorporating word order and sentence structure information into the weight, cost, and penalty terms, as explained be- low. However, none of them consider the optimal transport of sentence structures nor utilize SAMs from BERT.
4 Optimal transport of words
We review the computation of WMD. In the follow- ing, we denote two sentences as
((w:)==1+ s' (w/) C CR
where wi, w/ E Rd are word embeddings, and n, are sentence lengths. SAM is denoted A (Aw) E Rnxn and A' (A'r)) E Rmxm respec- tively, for and s'. The element Arr is the attention weight from w, to Wy, and the element Abp3 is the attention weight from w' to w'y. Each row of SAM is normalized as Ex= Aw = - A'p- 1. The weights on and w/J in the probability distributions for s and s' are denoted by u (u))_ E RR with E u = and u' (uf) E Bo- with Σド= un/ 1, respectively. In this paper, unless otherwise stated, the weights on words in a sentence are equal, i.e., the uniform distribution specified as
In OPWD (Su and Hua, 2019), the temporal dif- ference between word embeddings w, and w' is measured by the distance between their relative temporal positions (i/n j/ 3/m)², where n and m represent the lengths of the corresponding sen- tences. This difference is incorporated into the transport cost and an additional regularization term. WMDo (Chow et al., 2019) identifies consecutive words common to two sentences chunk and in- troduces penalty term according to the number of chunks. In Syntax-aware WMD (SynWMD) (Wei et al., 2022), the word weight is computed from the word co-occurrence extracted from the syntactic parse tree of sentences, and a word embedding in- corporates those from its subtree of the parse tree.
= (1/n) u' (1/m)
(1)
4.1 Wasserstein distance
Sentences and s' cannot be naively compared be- cause they are generally different in length, and the corresponding words are unknown. We then consider the word transport from s to s' to obtain

---

the word alignment. Interpret the sentence as the amount of mass u, position w,. First, we denote Púj E [0, 1], the amount of mass transported from position w; to position w'.- and consider the trans- port matrix P = (Pg) E 20 with nonnegative elements. Next, we define the distance function c(w,, w/) E R220 as the cost of transporting a unit mass from W; to w'J and specify the distance ma- trix C (Cty) (c(w,, w()) Ro" Given P and C, the transport from W, to w costs CiyPa. and the total cost of transport is
sentence
sentence2
J
j
P Pap
Pa
P
U
Pe5t
P
Figure 3: Transport of sentence structure induced by word transport. The array of transport from i-th word is P: and that from the ě'-th word is p The amount of transport from the (ú, 1 i')-th element of SAM defined by the outer product p, and Pa-
m
CipPi 1=1
(2)
Finding the optimal transport matrix P = (P.j) that minimizes the total cost, compute the Wasserstein distance between s and s' as the mini- mum value of the total cost Σ ΣØ Pij.ej.
2019). WMD does not distinguish the case (1) and the case (2) of Table in Section 2 using the Oth layer of BERT as a statistic word embedding. Morcover, the difference is slight even if the 12th layer of BERT is used as dynamic word embed- ding: WMD = 9.17 for case (1) and WMD = 9.26 for case (2).
4.2 Word mover's distance
Word Mover's Distance (WMD) (Kusner et al., 2015) is Wasserstein distance applied to word em- beddings. In the original WMD, the weight on words is the uniform distribution (1), and the dis- tance function is Euclid distance between word embeddings w; and w'.. The distance matrix is defined as
Optimal transport of sentence structures
In this section, we propose to apply the GW dis- tance to the SAM of BERT to measure the opti- mal transport distance between sentence structures. Later in Section 6, will attempt to improve WMD using this method.
f c(w; wy) = |w; w'll2-
(3)
Therefore, WMD between and s' is
WMD(s, min Pell(u,"')
ΣΣ
(4)
Here, Iu, u') the of all possible values of the transport matrix defined as
5.1 Gromoy Wassery-asserstein distance
As seen Section 2, the transport of sentence struc- ture from 8 to s'. i.e., transport from A to A', is induced by the transport of words. How can we de- fine the structure transport consistent with a given word transport?
n Σ 1=1
m
R P
I4 uu
= Pij=u
20
j=1
and u, u' are omitted on the left side of (4).
Let consider p: (P......, Pim), the array of transport amount P.j from the i-th word of to the j-th word of s'. As illustrated in Fig. 3, we apply p, to the i-th row A and py to the i'-th column of A. This defines the transport from Aw of s to A'py of s' as the outer product of and pr; the transport amount from position A, to position A is defined as P.jP.y:
4.3 Limitations of WMD
WMD cannot account for the word order of the sentences because it treats a sentence as a set of word embeddings to find the optimal transport dis- tance. Therefore, it is hard to distinguish sentence pairs with significant word overlaps in the case of static word embeddings. For some models, such BERT and ELMo, the problem remains even for dynamic word embeddings that depend on the context because the similarity is still high between word embeddings of the same word (Ethayarajh,
The above definition of transport between SAMs is consistent with the transport of words. From the definition of transport, the mass at Aw is Ens==1 P.p = u,ut, and the mass at Agy is

---

Popey ulddg. This implies that the marginal mass for rows and columns coincides with the mass of words, and thus the transport preserves the mass.
Therefore, by combining WMD-like methods (i.e., WMD and its variants obtained by modifying C, u and u') with SMD, we propose an optimal trans- port distance, Word and sentence Structure Mover's Distance (WSMD), that utilizes word features and considers word dependeney within a sentence. By specifying the mixing ratio parameter À E [0, 1]. obtain
In this paper, we specify the cost of transporting a unit mass as |AA -Ayy1 Then, the cost of transport AAg 1/ 1y,2,,, Pey,, and the total cost
of transport from A A'
(P) = Σ
11
Σ
m
Aw - A/g Pes
WSMD((s, A), A'))
=
n
m
W==1J/=1
min Pell(.)') 81 jJ'=1
Σ
Σ
{1- 1)Ci
(5)
Finding the optimal transport matrix P that min- imizes the total cost (P), we compute the Gromov-Wasserstei (GW) distance (Mémoli, 2011 I; Peyré and Cuturi, 2020) between A and A' the minimum value of (5).
+ X/: AA 1 12] PippPiys (7)
where A is a normalization parameter. For further details about k. refer to Appendix A. By noting D/= -y Pay 1, WSMD WMD for = For X == WSMD kSMD)
5.2 Sentence structure mover's distance
For an intermediate value À (0, WSMD considers both WMD and SMD. Let P be the op- timal transport matrix, i.e., P that attains the min- imum of (7). Substitute this into (2) and (5), denote them as WMD, and SMD,, respectively. Then we can write
Applying the GW distance SAMs, we propose the sentence Structure Mover's Distance (SMD), which is the optimal transport distance considering the dependency of words in a sentence.
min SMD(A, PEHl(,,))
(P.
(6)
WSMD = (1 A)WMD, + A&SMD
(8)
Note that SMD is not a metric of a metric space, while it can still measure the structural difference.
This decomposition explained in Appendix B.
The optimal transport distance that simultane- ously considers the Wasserstein and GW distances, as in WSMD, is known as the Fused Gromov- Wasserstein (FGW) distance (Vayer et al., 2019, 2020). However, like SMD, WSMD uses an asym- metric SAM, so WSMD is not metric in general.
The general definition of GW distance considers the p-th root of the total cost for the form |Aw A I" in (5), and it is metric if Aw is symmetric distance matrix. Thus using an asymmetric SAM is not GW distance in strict sense. We consider the case = and the square root is omitted because it does not affect the magnitude order.
Experiments
6 Optimal transport of words and sentence structures
We compare the performance of our proposed method and existing baseline methods on the task of measuring semantic textual similarity (STS) be- tween two sentences.
Here, we propose an optimal transport distance that simultaneously considers the word embeddings and sentence structure.
7.1 Datasets
6.1 Word and sentence structure mover's distance
7.1.1 PAWS dataset
Paraphrase Adversaries from Word Scrambling (PAWS) (Zhang et al., 2019) has a binary label for an English sentence pair indicating paraphrase (i.c., the sentence pair has the same meaning) non-paraphrase. The binary classification of these labels can be considered an STS task. If the trans- port distance for a pair small, we classify the pair as a paraphrase; otherwise, we classify it as a non- paraphrase. The effectiveness of this classification
As seen in Section 4.3, WMD computes Wasser- stein distance using Euclid distance of word embed- dings but cannot handle sentences with different meanings depending on word order. On the other hand, as seen Section 5.2, SMD computes the GW distance using the SAM of BERT, which en- codes the sentence structure, but it cannot handle individual word information like word embeddings.

---

Dev
Test
The word embeddings used in the experiments are the static embeddings taken from the input layer, i.e., the Oth layer, and the dynamic embed- dings taken from the final layer, i.e., the 12th layer. We call such word embeddings BERTO, BERT12, and so on. When computing embeddings for STS methods such WMD, recomputing the embed- dings for each method may result in different em- beddings due to the randomness of dropout. There- fore, to ensure an accurate comparison of STS methods, we had computed the embeddings once and used them for all the STS methods. We per- formed whitening for all the word embeddings be- cause the representation of BERT is known to be anisotropic (Ethayarajh, 2019).
PAWSQQP
1500 (24.7%)
677 (28.2%)
PAWSwii
1500 (42.3%)
8000 (44.2%)
STSB
1500
1379
Table 2: The number of sentence pairs used in our ex- periments. Train sets are not used. The percentage of paraphrases for PAWS is shown in parentheses.
is then evaluated using the AUC as the metric. There are two types of PAWS: PAWSqqP and PAWSwiKi. constructed from sentences in Quora and Wikipedia, respectively. Table 2 shows the number of sentence pairs and the percent- age of paraphrases used in our experiments. In PAWSqQr. since the test set is not provided, the first 1500 pairs were selected from the 11988 pairs in the train set to make a new dev set, and the original dev set was considered as a test set. In PAWSwiki. the first 1500 pairs were selected from the dev set.
7.3 Baseline methods
The following simple baseline methods for repre- senting sentence were selected. The similarity for a sentence pair is computed by the cosine similarity of the two vectors.
Bag-of-Words (BoW) is a high-dimensional vector whose elements the frequency of oc- currence in a sentence for all words.
7.1.2 STS Benchmark dataset
The STSB dataset (Cer et al., 2017) contains human-annotated scores for English sentence pairs, reflecting the average similarity on a six-point scale, Table 2 shows the number of sentence pairs. Spear- man's rank correlation (Spearman's 's p) used to compare these scores with optimal transport dis- tances.
Average Pooling (Avg. Pool.) is simply the av- erage vector of word embeddings in a sentence.
[CLS] token is the first token in the input quence, and in SimCSE it is used specifically for sentence embedding.
SIF (Arora et al., 2017) is an unsuper- vised method that sums word embeddings with frequency-based weights and subtracts the first sin- gular vector from the SVD.
7.2 Models
We used BERT, SimCSE (Gao et al., 2021) and RoBERTa (Liu et al.. 2020) from the Hugging Face transformers library (Wolf et al., 2020) and see Table 8 in Appendix C for details.
uSIF (Ethayarajh, 2018) an unsupervised method that replaces the dot product with arceo- sine in the probability model of SIF and subtracts multiple singular vectors instead of just the first. Conceptor Negation (Con. Neg.) (Liu et al., 2019) is unsupervised method that optimizes the performance of sentence embeddings for a new corpus while preserving the performance for older corpora when dealing with multiple datasets.
SimCSE is a sentence embedding model based on contrastive learning with BERT. It has two types: unsup-SimCSE and Sim-EE, depending on how positive samples are defined. In this study, unsup-SimCSE is used.
RoBERTa is a model that improves the perfor- mance of BERT by changing the pre-training set- tings of BERT, and by tuning the hyperparameters.
BERTScore (Zhang et al., 2020b) is an auto- matic evaluation metric for text generation and computes similarity score by comparing each token in a sentence with each token in another sen-
The BERT, SimCSE, and RoBERTa models we used have 12 layers and 12 heads, resulting in a total of 144 extractable SAMs. We input the sen- tences into BERT and removed the stopwords and special tokens such as [CLS] and [SEP] from the output tokens!
we did not remove the stopwords since removing them would have deteriorated the performance.
"Note that the embeddings taken from
layer are only
approximate statistic embeddings. because the segmentation embeddings and the position embeddings are added.
For one of the henchmarking OT methods (SynWMD),
IInstedd of [CLS] token, we used 
 token for RoBERTa.

---

PAWS dev
LnHi:: n-th layer's Lipe head
75
PAWS dev (AUC x 100)
n-th layer
0.5
SimCSE
Light
RoBERTa
Lage
0..4
BV
70
LAp Lnyso
0.3
LGg13
L8g11
LAWS DaR
0.2
65
L1g12
13p6
2
4
Layer
8
10
12
L30043
LSji
10
Figure 5: The layer-wise BV for BERT, SimCSE, and RoBERTa on the PAWSoqP dev set. BV is high from the fifth layer in each model.
12y15
60
L720
L1
LAHS
WM
L357
BERT
SimCSE
RoBERTa
70
75
80
85
PAWSooP dev (AUC x 100)
Oth Embs.
8
12th Embs.
8
6
Figure 4: The AUC values of WMD (4) and WSMD (+.
A) on the PAWSQQP (X-axis) and PAWSwak (y-axis) dev for BERTO. The symbol indicates the AUC value of the WSMD for a single head. The symbol indicates the AUC value of the average WSMD for multi-heads within layer. While there are significant variations in the symbols, the A symbols are more stable. In fact, the symbols have higher correlation coefficient than the symbols, as shown in Table 3.
Table 4: Topl layer for BERT, SimCSE, and RoBERTa. The layer was selected from the fifth onward based on the AUC of the average WSMD for single layer using the PAWSQQr dev set.
2019) with cost defined by L2 distance and cosine similarity. ROTS (Wang al., 2020, 2022) using the most effective method ROTS+SWC+mean. We used Python Optimal Transport (POT) (Flamary et al., 2021) to implement simple WMD-like meth- ods, SMD, and WSMD. We also used the OPWD part of the publicly available code for OWMD (Liu et al., 2018), but other parts could not be found.
WSMD calculation
#
p X 100
single head multi-heads within layer
144
69.30
12
74.50
Table 3:
The Spearman's p between AUC values of WSMD for PAWSqqr and those for PAWSwaa in Fig. 4. The average WSMD for multi-heads within layer has a higher correlation than the WSMD for single head.
7.5 Head selection to compute SAM
Which heads should we use for WSMD? Fig. 4 shows the AUC values of WMD and WSMD for each head on the PAWSQQP and PAWSWik dev sets. It also shows the AUC values computed for the WSMD averaged over multiple heads within a layer. Table 3 shows the Spearman's 's p for AUC val- ues in Fig. 4. These results suggest that when cal- culating WSMD, selecting an appropriate layer and using the average WSMD for multi -eati-heads within that layer likely to result in more stable perfor- mance than using a single head.
tence.
DynaMax (Zhelezniak al., 2019) is fully unsupervised and non-pare con-parametric similarity that dynamically extracts and max-pools good features depending on the sentence pair.
7.4 Benchmarking OT methods
The following OT methods computed from word embeddings were selected: WMD (Kusner et al., 2015) with the uniform and IDF weights. WRD (Yokoi et al., 2020) with the norm and IDF weights. SynWMD (Wei al., 2022) with SWF weights and cost defined by cosine similarity and SWD. WMDo (Chow et al., 2019) with cost de- fined by cosine similarity. OPWD (Su and Hua,
It is not desirable to change the heads or layers for each data set. Therefore, we first define bidirec- tional attention variability BV score for the layer containing SAMs that capture the context of sentences. We then consider averaging WSMD using the SAMs of the layers selected based on BV A. For details on BV A. see Appendix E. Fig. shows the layer-wise BV for BERT, Sim-

---

BERTI PAWSqar PARS
BERTIZ
WSMD is not attempted for ROTS, OPWD and WMDo because these methods cannot be expressed
SAM:
STSB
PAW PAWS
AUC
/ X 100
9,00
00.74
0.65
-1.98
in the form (4).
5-12
-0.53
0.00
-1.30
1-12
Table 5: Average score improvement of WSMD for WMD-like methods with BERT.
7.6.1 Comparison of WSMD Performance
Fig. 6 shows the experimental results compared to some baselines using BERT embeddings. For PAWSQQP the top of Fig. and PAWSwiki at the middle of Fig. 6, the best-performing layer se- lection is the top1 layer, and improvements were observed in WSMD with WMD, WRD, and Syn- WRD for both BERTO and BERTI2. The top1 layer is chosen based on the PAWSQqP dev set, there- fore the performance improvement is greater in PAWSQQP compared PAWSWiki- For STSB the bottom of Fig. there a small decrease in per- formance. However, especially with BERTO, per- formance can be improved on the PAWS datasets while still maintaining some level of performance on STSB
SimCCDD PAWSOGP PANSWILI AUC
SimCSR12
SAM;
STSB
PAWSOOP PAWSWikk AUC
S15B
100
100
100
5.44
1.67
00.02
0.37
0.59
0.53
S-12
4.39
0.50
-0.30
0.15
-0.6
-1.01
1-32
00.33
Table 6: Average score improvement of WSMD for WMD-like methods with SimCSE.
RuBERCOO PAWSOOP PAWSWIKi
ReBERTs12 PAWSOOP PAWSSWik
SAM;
STSU K 100
SISU
AUC 100
ADC 100
Px 100
top
11.04
2.49
0.74
0.01
0.37
-157
5-12
11:48
2-45
0.83
0.13
0.24
-1.38
1-12
10.42
2.28
-130
0.11
0.01
-1.1
Table 7: Average score improvement of WSMD for WMD-like methods with RoBERTa.
Similar to Fig. 6. the experimental results for SimCSE and RoBERTa are shown in Appendix Figs. 7 and 8, respectively. We observe that the sults are roughly similar to those for BERT, while the methods using ROBERTaO show better perfor- mance compared to those using RoBERTa12 on STSB. Detailed results for each dataset are shown in Appendix Tables 19, 20, and 21.
CSE, and RoBERTa the PAWSqqp dev set. BV is high from the fifth layer in each model. Thus, we consider the following three ways to compute the average WSMD*: (i) The average WSMD using the best single layer from the fifth onward (topl layer). (ii) The average WSMD over the fifth onwards (5-12 layers). (iii) The average WSMD over all the layers (1-12 layers). Table 4 presents the topl layer chosen for each model. It is worth noting that the topl layer selected using the PAWSQQU dev set applied to all other datasets. It also important to note that the selection of layers (i.e., 5-12 layers) in method (ii) was based only on BV A of sentences from PAWSqqP dev set without any reference to the labels for the binary classification. In particular, method (iii) is fully unsupervised in terms of layer selection.
7.6.2 Comparison of layer selection methods
Tables 5, 6. and 7 show the performance improve- ment of WSMD using different head selection. It is observed that the performance on PAWSQQP and PAWSwik improves for all models when the topl layer is selected. As shown in Fig. 6, partic- ularly significant performance improvements are observed when the Oth-layer embeddings are used.
In addition, even for unsupervised layer selec- tion, the performance improvements are seen on PAWSQQP and PAWS for all Oth-layer embed- dings. In particular, for RoBERTa on PAWS, while the performance improves when using layers from 5 to 12 based on BVA, the performance decreases when using layers from to 12. This may suggest that the early layers do not have enough contextual information, as also indicated by BV A in Fig. 5. For STSB, although no performance improve- ment is observed, the performance degradation is limited to a small 2%.
7.6 Results
Some WMD-like methods (WMD, WRD, Syn- WMD) are combined with SMD (as indicated as WSMD) and compared with the original method to see if an improvement by introducing WSMD. We use À 0.5 for WSMD. For more details about the parameters for the other methods, see Appendix D.
'We used PAWSqqe dev set for the layer selection be- cause it has better performance compared to PAW Swiki. and also because STSB is a datasct with different properties than PAWS.

---

7.7 Supplementary experiments
addition to the abovementioned experiments, we have conducted other experiments in different settings.
PAWSoop test (AUC x 100)
80
76
To evaluate the generality of our method, we con- ducted experiments with DiST (Sanh et al., 2019) model with different number of lay-
Score with WSMD
72
68
ers than BERT, and with BERT trained with dif- ferent seed (Sellam et al., 2022). We evaluated the performance of these models on PAWSQQPs PAWSwiki. and STSB. WSMD also showed strong performance on these models, especially on PAWS. More details can be found in Appendix G.
64
WHDDU
Tor
by AM.. J0b
60
56
52
48
44
We also extended our experiments to another dataset, SICK-R (Marelli et al., 2014), using BERT measure semantic textual similarity. Similar to STSB, the degradation in performance was within 2%. Detailed results of this additional study are also available in Appendix H.
44
52
56
60
72 76
Baseline Score
PAWS Wiki test (AUC x 100)
80
Score with WSMD
76
Given the small size of the PAWSqqP test set, which contains only 677 sentence pairs, we also present the scores on the PAWSQQP train set using our method for WMD in Appendix The result suggests that improvements in paraphrase identi- fication by WSMD are observed even when the dataset size increases.
72
wio Ao
68
64
60
Harg 200
56
WNS
SyweD 2Ay 350
These results confirm the effectiveness our method on different datasets, using different mod- els with SAMs.
52
48
52
56
64
68
76 80
Baseline Score
8 Conclusion
STSB test (Spearman's p x 100)
Since WMD treats sentence set of word em- beddings and computes sentence similarity, can-
84
78
Score with WSMD
not consider the word order in the sentence, There-
Sugaro Pee
fore, we focused on the fact that the SAM of the input sentence obtained from the pre-trained BERT
66
represents the relationship between words in the sentence and has information on the sentence struc- ture. We proposed optimal transport distance WSMD that improves existing WMD-like meth- ods by using FGW distance that simultancously measures the difference between word embeddings and the difference between sentence structures. We conducted experiments on paraphrase identifica- tion on PAW'S and sentence similarity on STSB, confirming the proposed method boosts PAWS per- formance with minimal impact STSB.
60
54
48
42
36
30
32
40
48
56
72
88
Baseline Score
Figure 6: Performance of WSMD with several WMD- like methods (SMD, WMD, WRD, SynWMD) for BERT. The scores (AUC Spearman's p) are com- pared with the original WMD-like methods. Methods that are not applicable WSMD are positioned on the diagonal line. Values above the diagonal line represent performance improvements achieved by WSMD.

---

Limitations
23H03355, JST ACT-X JPMJAX200S, and JST CREST JPMICR2IN3.
As seen in Section 7.6, there was not much improvement in scores on STSB compared to PAWS. Since methods like uSIF, which con- sider sentences sets of words, show good performance, it is assumed that in STSB, un- like PAWS, there are fewer sentences where word order changes significantly alter the meaning, and the SMD term WSMD con- sidered noise. In fact, in STSB, it was found that there is a strong correlation between the number of common words in a sentence pair and similarity. See the Appendix for details.
References
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-bert baseline for sentence embed- dings. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. Open- Review.net.
Daniel M. Cer, Mona T. Diab, Eneko Agirr,, lñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval- 2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Eval- uation, SemEval @ ACL 2017, Vancouver, Canada, August 3-4, 2017, pages 1-14. Association for Com- putational Linguistics.
As noted in the Section 7.6, WSMD shows a smaller performance improvement with 12th embeddings compared to Oth embeddings in PAWS. This is probably due to the fact that the 12th embeddings are contextualized, and the changes in word order within sentences in PAWS are reflected the embeddings.
Liqun Chen, Yizhe Zhang. Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Im- proving sewenceet--seueece learning via optimal transport. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA USA, May 6-9, 2019.
Sentences with the same meaning do not nec- essarily have the same structure. For example, don't think it makes sense and it doesn't make sense are a pair of sentences with the same meaning. Still, they have different structures, so using WSMD might decrease the sentence similarity score.
Julian Chow, Lucia Specia, and Pranava Madhyastha. 2019. WMDO: fluency-based word mover's ss distance for machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 Vol- 2: Shared Task Papers, Day 1, pages 494-500. Association for Computational Linguistics.
Compared to regular WMD, as the number of SAMs used in WSMD increases, the com- putation time also increases. However, since WSMD can be computed independently for each SAM, parallel processing possible resources are available.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of bert's attention. In Proceed- ings of the 2019 ACI. Workshop BlackboxNLLP An- alyzing and Interpreting Neural Networks for NLP, BlackboxNLPPG ACL 2019, Florence, Italy, August 1, 2019, pages 276-286, Association for Computational Linguistics.
WSMD cannot be applied directly other em- beddings, such as word2vee, because different tokenizers are used for each embedding. How- ever, may be possible to use WSMD through processing such as replacing the tokens of the model with words in the other embeddings vocabulary.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ethics Statement
This study complies with the ACL Ethics Policy.
Kawin Ethayarajh. 2018. Unsupervised random walk sentence embeddings: A strong but simple baseline. In Proceedings of The Third Workshop on Repre- sentation Learning for NLP, Rep4NLP@ACC AAAALP@@ACL... 2018, Melbourne, Australia, July 20, 2018, pages 91-100. Association for Computational Linguistics.
Acknowledgements
We would like to thank anonymous reviewers for their helpful advice. This study was partially sup- ported by ISPS KAKENHI 22H05106 22H03654,

---

Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? comparing the geome- try of bert, elmo, and GPT-2 embeddings. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Join! Conference on Natural Lunguage Pro- cessing, EMNLP-IJCNILP 2019, Hong Kong, China, November 3-7, 2019, pages 55-65. Association for Computational Linguistics.
the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACI-HLT 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2279. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2020. Roberta: A robustly optimized BERT pretraining ap- proach. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos. Kil- ian Fatras. Nemo Fournier, Léo Gautheron, Nathalie H. Gayraud, Hicham Janati, Alain Rakotoma- monjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tave- nard, Alexander Tong, and Titouan Vayer. 2021. POT: python optimal transport. J. Mach. Learn, Res.. 22.78:1-78:8.
Ziyang Luo. 2021. Have attention heads in BERT learned constituency grammar? In Proceedings of the 16th Conference of the European Chapter of the Association for Compulational Linguistics: Student Reseurch Workshop. EACL. 2021, Online, April 19- 23, 2021. pages 15, Association for Computational Linguistics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-17 November, 2021, pages 6894 6910. Association for Computational Linguisties.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raf- faella Bemardi,, Stefano Menini, and Roberto Zam- parelli. 2014. Semeval-2014 task Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and tex- tual entailment. In Proceedings of the 8th In- ternational Workshop on Semantic Evaluation, Se- mEval @ COLING 2014, Dublin, Ireland, August 23- 24, 2014, pages 1-8. The Association for Computer Linguistics.
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in BERT track syntactic dependencies? In Natural Lan- guage, Dialog and Speech (NDS) Symposium. The New York Academy of Sciences.
Facundo Mémoli. 2011. Gromov- wasserstein distances and the metric approach to object matching. Found. Comput. Math., 17444477-487
Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha. and Kilian Q Weinberger. 2016. Supervised word mover's distance. In Advances in Neural Infor- mation Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, De- cember 5-10, 2016, Barcelona, Spain, pages 4862- 4870.
Gabriel Peyré and Marco Cuturi. 2020. Computational optimal transport.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.
Leonid Kantorovich. 1960. Mathematical methods of organizing and planning production. Management Science, 6(4):366-42)
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistiIBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Sth Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019.
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kil- ian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedingg of the 32nd In- ternational Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of Proceedings of Machine Learning Research, pages 957-966. PMLR.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Ture, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The multib- erts: BERT reproductions for robustness analysis. In The Tenth International Conference on Learning Rep- resentations, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net.
Bang Liu, Ting Zhang. Fred X. Han, Di Niu, Kunfeng Lai, and Yu Xu. 2018 Matching natural language sentences with hierarchical sentence factorization. In Proceedingy of the 2018 World Wide Web Conference on World Wide Weh, WWW 2018, Lyon, France, April 23-27, 2018 pages 1237-1246. ACM.
Bing Su and Gang Hua. 2019. Order-preserving op- timal transport for distances between sequences. IEFE Trans. Pattern Anal. Mach. Intell., 41(12)22961 2974.
Tianlin Liu, Lyle Ungar, and João Sedoc. 2019. Contin- ual learning for sentence representations using con- ceptors. In Proceedings of the 2019 Conference of

---

Titouan Vayer, Lactitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. 2020. Fused gromov- wasserstein distance for structured objects. Algo- rithms, 13(9):212.
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLI 2019, Minneapolis, MN, USA, June 2-7. 2019. Volume I (Long and Short Papers), pages 1298- 1308. Association for Computational Linguisties.
Titouan Vayer, Nicolas Courty, Romain Tavenard, Lacti- tia Chapel, and Rémi Flamary. 2019. Optimal trans- port for structured data with application on graphs. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-13 June 2019, Long Beach, Califormia, USA, volume 97 of Pro- ceedings of Machine Learning Research, pages 6275- 6284. PMLR.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019, Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th Interna- tional Joint Conference Natural Language Pro- cessing, EMNLP-IJCNI 2019, Hong Kong, China, November 3-7, 2019, pages 563-578. Association for Computational Linguistics.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in transformer language model. CoRR, abs/1996.00228.
Zihao Wang. Jiaheng Dou, and Yong Zhang. 2022. Un- supervised sentence textual similarity with composi- tional phrase semantics. In Proceedings of the 29th International Conference on Computational Linguis- tics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 4976-4995, International Committee on Computational Linguistics.
Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, and Nils Ham- merla. 2019. Don't settle for average, go for the max: Fuzzy sets and max- pooled word vectors. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Zihao Wang, Yong Zhang, and Hao Wu. 2020. Structural-aware sentence similarity with recursive optimal transport. CoRR, abs/2002.00745..
Chengwei Wei, Bin Wang, and C.-C. Jay Kuo. 2022. Synwmd: Syntax-aware word mover's dis- tance for sentence similarity evaluation. CoRR, hhs/2206.100299
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu. Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38-45. Association for Computa- tional Linguistics.
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word rotator's distance. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 2944- 2960. Association for Computational Linguisties.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020a. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Tianyi Zhang, Varsha Kishore, Felix WU, Kilian Q Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT In 8th Inter- national Conference on Learning Representations,

---

A WSMD definition details
the optimal transport matrices in the computation of WMD, SMD, and WSMD are all different in general, and WSMD, the optimization considers the two components simultaneously.
For clarity, rewrite equation (7). By specifying the mixing ratio parameter À E [0, 1], obtain
WSMD((s, A) (s', A))
Interestingly, the decomposition (8) indicates that WMDD, can be interpreted improvement of WMD by utilizing (A/(1 A))&SMD, penalty term. Thus, in addition WSMD, WMDX can also be used as a sentence similarity measure, although this is not the main argument of this paper.
Σ Σ
{11 X)Ca
min Pell(u,
--'=1===1
A3yr (1]) P Pip
Ak Aw
where h: SMEAMSE is computed from
m
Cg
C Models
CM
mm
The Hugging Face models used in the experiments are presented in Table 8.
Aw 1
AMSE
n²m²
11=11-=11
D Parameters for each method
Parameters are shown for each method used in the experiments. We provide explanations for terms that may be unclear or less intuitive compared to the original notation in the paper. numSVToRe- move means the number of singular vectors to be removed.
Note that our definition of l is only to increase the interpretability of X, and l does not change the performance at all. By this definition, Cas repre- sents the average of Cij. and AMSE the average of |A 1/1.1. From this, we can roughly consider Cig ~ C., CM - 111 AMSE. we employ k. then K|A -A' 1 With this formulation, CM/AMSE AMSE Cij- can use easily in- terpretable value for A, such 0.5. By noting E f Ex Pzy = 1, WSMD = WMD for 0. A 1, WSMD = kSMD. For an inter- mediate value x c (0, 1). WSMD considers both WMD and SMD. We normalized À by introducing the factor & in (7). OM and A MSE: are interpreted as (2) and (5), respectively, by specifying Pij- = uuu/ with the uniform weight (1).
D.1 SIF
Parameters of SIF used the experiments are shown in Table 9.
D.2 uSIF
Parameters of uSIF used in the experiments are shown in Table 10.
D.3 Conceptor Negation
Parameters of Conceptor Negation used in the ex- periments are shown in Table
B Decomposition of WSMD into WMD and SMD components
D.4 ROTS
Let P be the optimal transport matrix of (7), i.e.. P that attains the minimum of (7). Substitute this P into (2) and (5), denote them as WMD, and SMD,, respectively. Then we rewrite equation (8) as follows:
WE apply the most effective technique. ROTS+SWC+mean ROTS. In this context, SWC+mean denotes a process that implements dimension-wise (S)caling (Ethayarajh, 2019), possesses SIF (W)eights, applies (C)onceptor Negation, and uses the mean in the aggregation of pair-wise cosine similarities of words. Parameters of ROTS used in the experiments are shown in Table 12. Preg means prior regularization, Creg means cosine regularization, and Ereg means entropy regularization. C also means interpolation coefficient. aggregation means how to handle different scores (mean, max, min, last, no).
WSMD (1 A)WMDA A&SMD
In case (2) of Table 1, we computed WSMD = 8.03, k 688, x 0.5. This is decomposed into the components WMD, 13.30 and ASMD, 2.76. On the other hand, we can also compute WMD 12.55 and &SMD = 2.76 in the same setting, which indicates that WSMD is not a simple interpolation of WMD and ASMD. This is because

---

model
Hugging Face model
BERT unsup-SimCSS RoBERTa
bert-base-uncased roberta-base
DistilBERT
di distilbeert-bas--nnaaed /ooglee/mult.bbrtsseeedd
BERT seed0
Table 8: Models of Hugging Face used in the experiments.
parameters
values
parameters values
weight nomSVTo
SIF
weight SIF
aCN
2
Table 9: Parameters of SIF used in the experiments.
Table 11: Parameters of Conceptor Negation used in the experiments.
parameters values
parameters
values
weight uSIF
numSVToRemove
5
weight
SIF
OCN
2
Table 10: Parameters of uSIF used in the experiments.
nunSVToRemove
parser
dependency tree mean
aggregation
D.5 OPWD
normed vectors
True
Parameters of OPWD used in the experiments shown in Table 13.
tree depth
5
Preg
[10, 10, 10, 10, 10]
Creg
0
D.6 WMDo
Ereg
0
Parameter of ROTS WMDo the experiments are shown Table 14.
C
Table 12: Parameters of ROTS used in the experiments.
D.7 SyD
Parameter of SynWMD used in the experiments are shown in Table 15.
parameters
values
A
10
A2
0.03
10
Iteration
20
Table 13: Parameters of OPWD used in the experiments.
parameter value
0.2
Table 14: Parameter of WMDo used in the experiments.
parameter value
Table 15: Parameter of SynWMD used in the experi-

---

E Details of attention variability
We employ attention variability as proposed in (Vig and Belinkov, 2019). This measures the degree of attention variability across different inputs. A high variability implies that the attention head is content- dependent, while low variability suggests that the head is content-independer attention variability V defined as:
PAWSoor (AUC x 100)
methods layer weight
BERT
SimCSE
RoBERTa
BOWS [CLS]
44.56
44.56
46.16
60.24
62.39
57.76
top
59.84
59.70
60.95
5-12
uniform
58.50
58.04
59.96
Mathematically,
1-12
56.23
58.35
57.27
SMD
top
60.25
59.77
59.48
5-12
IDF
57.99
59.30
57.19
VA ΣXEX Σ E (x) Ayl 2. ExEx EE E Azj(2)
1-12
57.82
57.71
55.62
(9)
Table 16: The scores on PAWSoqP test set for BOWS, [CLS], uniform- uniform-weighted and IDF-weighted SMD, methods that do not require word embeddings. Note that BERT and SimCSE have identical scores for BOWS because they use the same tokenizer.
where A represents the head contained in given layer, represents sentences of length |x|] in the dataset X, A(x) (a)) E R |||||| is the spe- cific SAM for each sentence æ, and Auj is the av-
erage of (z) over all X. V is defined for
GPT-2 (Radford et al., 2018), and the summation from 1 to for j reflects the unidirectional structure of GPT-2.
PAWSwiki (AUC
100)
methods layer weight
We adapt the attention variability V to BERT by defining a bidirectional attention variability BV A that measures the degree of attention variabil- ity across different inputs, where the summation for ranges from to [x]. It is defined as follows:
BERT
SimCSE
RoBERTa
BOWS [CLS]
48.86
48.86
50.59
57.94
58.02
56.29
top 1
61.68
58.51
58. 18
5-12
uniform
62.98
60,91
58.69
1-12
59.66
58.82
57.18
SMD
Exex Σ E (a)(2)) Asgl
top 1
62.73
59.22
53.86
BV
5-12
IDF
63.30
59,92
53.08
2. Σ EvEx Σ Σ Az((x)
1-12
60.83
58.96
51.89
(10)
Table 17: The scores the PAWS test for BOWS, [CLS), muiform-wigghed and IDF-weighted SMD, methods that not require word embeddings. Note that BERT and SimCSE have identical scores for BOWS because they use the same tokenizer.
F Detail results of experiments in Section 7
For PAWSoqP, PAWSwiki. and STSB, the scores from BOWS, [CLS], weighte-weighted and IDF- weighted SMD, methods that do not require word embeddings, shown in Tables 16, 17, and 18, respectively. The scores for methods that depend on word embeddings are shown Tables 19, 20, and 21. Similar to Fig. 6, the experimental results for SimCSE and RoBERTa are shown in Figs. and 8.
STSB (Spearman's 's x 100)
methods layer weight
BERT
SimCSE
RoBERTa
BOWS [CLS]
68.82
68.82
67.93
41.17
75.23
40.75
top
21.30
17.51
23.21
5-12
uniform
29.81
35.41
27.71
1-12
30.48
33.78
27.50
SMD
top
22.97
19.48
26.11
5-12
IDF
33.45
39.18
31.47
1-12
34.32
37.65
31.76
Table 18: The scores on the STSB test set for BOWS. [CLS], weiigtee and IDF-weighted SMD. methods that do require word embeddings. Note that BERT and SimCSE have identical scores for BOWS because they use same tokenizer.

---

PAWSQQP (ATC x 100)
methods
byer
weight
cost
BEXTO
BERT12
SinCSED
SinCSE12
RoBEXTal
RoBEXTa12
Avg. Pool.
52.96
64.33
30.56
63:00
05:05
63.04
BERTSSomm
unioocm
I13F
45.52
45.40
54.78
55.70
45.41
55.25
51.74
51.98
41.91
42.71
5L.61
$1.01
DynaMax SIF
51
51.23
Con, Neg ROTS
53.05
66.17
44.85
62.81
53,00
64.71
36.48
65.00
67.38
64.50
OPWD
miffeem
Lg
53.00
6648
53.01
65.58
68.35
66.89
cosise
$4,30
09.98
54.41
67.38
48.40
08.81
WMDo
uniSem
cosine
43.86
56.33
43.71
50.33
44.25
61.49
00.31
60.19
58.40
58.08
59.97
59.74
SMD
5-12
59.55
$6.30
57.78
56.16
50.27
$7.44
1-12
58.35
56.55
57.55
56.34
$7.59
$6,05
$3,07
66.48
53.01
65.58
48.35
66,69
62.58 (1
67.10
$7.87 0,80
65
5896
10.61)
0.05)
5.67)
56.35
65.44"00.3)
0.26)
1-12
(1
17 0.2
37.54
06.13(00005
51 9,99)
(.0T)
52.44
$2.47
top
(1 9.10)
66.75 (.67)
56.72 4255
38.88 030)
$7.511 9,24)
64.61 0.08)
5-12
57.90 5.40)
66.36 17 0.25)
55.88 (1 3,21)
10.66000000
572941 9,20)
64.05 (.12)
1-12
S8.47 (1 6.03)
66.11 (1 0.23)
56.33 4,36)
65,00 (S4)
$6 (8 8.50)
64.39 () -0.14)
54.53
69.07
33.71
66.99
6889
61,43
lop
66 80 12:15)
60.56 0.49)
60.18 6.67)
67.99 (-60))
60.7 11 180
67.50 0.16)
5-12
nerm
63.84 (1 9.19)
69:25 (7 0.18)
58.4 4.95)
67.32("0.3)
6234(1 15:455
67.4 (.19)
WRD
1-12
64.02 IT 997)
53.94
69.36 0.299
68.73
60.34 (1 7.13)
67.61""000)
61077
61.18(1 12.29
2386
67.54 (7.54 (.09)
62.09
68.6
(7 0.15)
61.8
6071 (1:33)
0.11)
70.00 (1 0.27)
63,66 9.65)
67.4("0057)
59.79
10.91)
66.15 0.03)
$9.17
lep
5-12
cosine
$6.29 (1 8.67)
60.37 (L 353
0.01)
56.78 88)
7.07)
5889 -026)
59030 4014)
60 17:56)
59 16.66)
10 -0.16)
(1 0.100
Sys WMD
1-12
SWF
36,57 LT 8.97)
68126 (L 0.46)
55.82 (1 7.90)
39.1(--)
58.00 15.73)
65.30 00.66
59:41
61.1
58.47
62:17
57.85
70.14
top 5-12
SWD
69.63 (1 1.22)
62. (1 3.33)
63.88 63.88 (38)
63:95 0.76)
60.15 1,68)
59.27 0.80
62.10(]-0.3 -.37)
62.35 -.44)
64.20 6.35)
6347 5.56)
09:97 -0.17)
70.11 (1-0.00)
1-12
0095)
62.09
Table 19: The scores on the PAWSQQP test set for the methods using the embeddings from BERT, SimCSE, and RoBERTa. For WMD, WRD, and SynWMD, the scores WSMD and the original method are compared for each layer selection. Score increases are highlighted in red and score decreases highlighted in blue. Maximum scores for each embedding are underlined.

---

methods
layer
weight
cost
BERTO
BERT12
PAWSSWi (AUC 100)
SimCSED
SinCSE12
RoBEXTal
RoBEKTAI2
Ang. Pool
49.43
60.06
49
56.71
51.11
99.13
BERTScose
miiforas
I1st
51.58
51.58
60
60.60
$2.66
52.71
57.02
56.92
52.09
$2.03
$5.70
57.57
DynaMax
38.38
66.00
02.37
5388
05
59.90
90.21
Nog. ROIS
49.40
60.31
51.18
19.24
49.35
00.88
53.76
57.45
52.15
00.08
DPWD
untioom
Eg
56.02
70.40
56.00
64.75
55.36
65.61
couite
$2.00
68.87
5208
61.87
52.51
64.35
WMDo
uniforrs
cosine
48.48
58.56
45.33
54.41
41.28
51.79
68.88
61.33
59.76
$7.94
56.42
58. 92
SMD
5-12
62.09
62.82
60,5
60.82
50.07
99.32
60.17
89.3
59.79
58.91
$7.47
58.24
56,02
70.40
56.00
64.78
35.36
03.61
uniform
63.43
(1)
70.51
11.36
2.23)
65.51 0.73)
$9.45
59.5841 0.22)
65
60.26
3.75)
70.62
kpp
5-12
11:0
62. 6.92)
(1 19.19)
71:22:1 1.10)
707247
36 0.5
$7 3.00)
64.3 ..1
65 0.78
53000) 2400
538011 2.84)
99,93 L1 0.30
0.00)
1-12
61.24 (1 5.800
70
20,90)
56 98
1.56)
54 -0000
$2944)
(.88)
90.51 () -0.73
32,01
69.06
5201
62.00
52.44
64.25
Top
55.04 (7 3.05)
69.75 (1 0.69)
534547 (.44)
62.62 0.622
540417 1,50)
63.05 ( 0.30)
5:12
$4.00 2.02)
6926 0.20
$250 0359
02.17(
St.12" 1.609
63.04 ( 0.29)
WRD
1:12
chimn
54.35 2.34)
69.22 0.06
53.5007 (.09)
02.37 0.37)
61.08
S4.44 (.70)
29.34
65.01 (7 0.26)
50.350
IDF
1.99
53.00 2,300
6836 0.14)
53,49 1,70)
61.39 0.311
50.32 0.98)
(1 0.01)
59.0
$1.63
$1.19
55.73
54.90 207)
5-12
cokine
50.24 (.99)
SK 15 0.26)
K5 0.95)
$5.02 (1 0.17)
$4.47 2.84)
65 0.76)
SynWMD
1-12
SWF
30.45 1.90)
49.94
59.35 00600
$9.08
50,2030 1.00)
$1.36
54.97 () .,.2
$6.54
54.48(1
$4.40
65.49 (1 0.51)
67.23
5-12
SWD
$2.11 2.17)
$1.22 1,28)
55751) 0.35
59.24 0.163
$2 L.0)
51.78 <h 0443)
16.11(] .0.4)
56.75 (1 0.213
56.951 2.53)
5695 2.53)
67.55 0.32)
03.65 (1 0.42)
51.31
5631
$1.95 <D 0.59)
$5.91
56.81
67.25
Table 20: The scores on the PAWS test set for the methods using the embeddings from BERT, SimCSE, and RoBERTa, For WMD, WRD, and SynWMD, the scores WSMD and original method are compared for each layer selection. Score increases are highlighted in red and score decreases highlighted in blue. Maximum scores for each embedding are underlined.

---

STSB (Speazman's X 190)
Methins
layer
wiight
COSE
BIFTO
BRRT12
SimCSFD
SimCSF12
RoBERBA
RoBERTH 12
Arg. Pool.
69.16
69.07
00.35
77.54
70.15
64..99
BERTSSomm
unifocm
63.27
65.13
62.90
70.59
63.38
58.71
LDW
70.96
64.09
DynaMax SIF
77.67
51
76.55
Con. Neg ROTS
77.19
68.69
69.27
61 81
77.36
69.58
64.
OPWD
unifocm
cosise
68.19
60.76
69.78
69
63:76
60.73
64.88
77.00
07.15
58.79
03.95
44.43
WMDo
unifoem
67.61
62.17
76.18
61.53
23.71
22.43
18.28
17.72
27.00
24.15
SMD
5-12
34.21
32.22
38.64
37.11
32.20
50.11
1-12
60.78
49.62
60.73
64.88
58.79
44,47
58,50
66 25 -3,37)
59.25 (L
63.17
52.33
42,48 -1
5-12
$8.96
42.25 (U-2.18)
1-12
-2,05)
1.729
42.00
62.78
top
62.22 (L 1.03)
898801 (.11)
63.26 (L 1000)
6828 1333
61,49
(2)2
45.04 (188)
5-12
63 10 -1225)
(07761-221)
63.71 (L .0:50
64.88
61.27 (C-1113 51
45,00 -1.97)
1-12
67. -1.7%)
5088 (1-218)
63.27 (L -1.03)
64.00 (1-177)
64:00
61 (_ A9)
45.05 () 919
70.68
70.30
65.96
76.87
60.52
64.09
lop
(1 0.08)
7002 (028)
58 (8388
7683 0004
69.35 (_ 0.00))
6161 (U -046)
5-12
70.65 ( 0.00)
CL 015)
05.05 0.31)
76831, 004
69.11 (. 0.11
63.19 () 0.11)
WRD
1-12
70.57 0.11)
70.55
701501-011
65 ()
67.01
1682(_-005)
77,07
69.71 (110.1))
63.56 00.3)
70.39
66.99
77.06
69.44
70.46
71000C 0001
.025)
77.11 (2
70.33 -0.22)
0.03) 71.08
66.58 (L -0445
77.15 0.08)
69.58 20)33
-042))
72.06
78.71
lep
5-12
cosine
71 -0.1))
-0,30)
70.89 (1-1.71)
-276)
68.0 (1 0.05)
-0.9%
17.30 (2
7.388.-1333
10.34 10.34 0.95
-081)
61
-1.76)
-1.99
Sys WMD
1-12
SWF
71.26 (1 (1.00))))
72.12
(1-85)
75.28
67.22 (058)
67.01
77.05 (_ 66)
19.15
70.30 (1-30 39)
72.88
67.00 (1 (15588)
70.54
SWD
72.13 (f 0.013
71055 () -2200
66.95 (1 .008)
78.19 0.9
72,14 0.54)
68.74 (1) -1,00)
5-12
0.15)
67.45 0.994
18.15
-1,00)
1-12
71.75
71.9701-1225)
7805
Table 21: The scores the STSB test set for the methods using the embeddings from BERT, SimCSE, and RoBERTa. For WMD, WRD, and SynWMD, the scores WSMD and the original method are compared for each layer selection. Score inereases are highlighted in red and score decreases highlighted in blue. Maximum scores for each embedding are underlined.

---

MAcrso-ymiNG
ha9CrBa2 Syrwwro
100077012-410 4170
OOW test (AUC x 100)
PAWSooP test (AUC x 100)
80
76
75
72
Score with WSMD
Score with WSMD
68
70
64
1D RC
65
WGO JOTLOCE
60
SyYAMo Ser 510
SyrNNo Set 5N0
60
VR0 ard
56
55
52
50
48
44
40
44
48
52
56
64
68
72
76
40
45
50
55
65
70
75
80
Baseline Score
Baseline Score
PAWS test (AUC x 100)
PAWS Wiki test (AUC x 100)
75
75
72
Score with WSMD
Score with WSMD
69
69
66
WED sore
66
63
63
60
SPAND
60
WRR
57
57
Turraco
54
Syiggs qur SAP
54
51
51
48
4848
66 Baseline Score
72
54
72 75 Baseline Score
57
63 66
STSB test (Spearman's x 100)
STSB test (Spearman's p x 100)
88
80
Score with WSMD
Score with WSMD
80
W85 1y RWS
72
72
SWIAND SAPT 940
Tyewoo om
64
64
56
56
48
48
40
40
32
56 64
80
88
32
40
64 72 80
Baseline Score
Baseline Score
Figure 7: Performance of WSMD with several WMD- like methods (SMD, WMD, WRD. SynWMD) for Sim- CSE. The scores (AUC or Spearman's p) are compared with the original WMD-like methods. Methods that are not applicable to WSMD are positioned on the diagonal line. Values above the diagonal line represent perfor- mance improvements achieved by WSMD.
Figure 8: Performance of WSMD with several WMD- like methods (SMD, WMD, WRD, SynWMD) for RoBERTa. The scores (AUC or Spearman's p) pared with the original WMD-like methods. Methods that are not applicable WSMD are positioned on the diagonal line. Values above the diagonal line represent performance improvements achieved by WSMD.

---

PoWS dev
from the third (top1 layer), third to sixth (3-6 lay- ers). and all layers (1-6 layers). For seRTs we compare performance using the same three settings as for BERT on the PAWSQQP dev set.
0.5
BV
0.4
Table 22 shows the top1 layer for each model. The scores for methods that depend on word embeddings are shown in Tables 23, 24, and 25,
0.3
BERT
2
6
10
12
Layer
H Experiments with SICK-R
Figure 9: The layer-wise BVA for DistilBERT and sERo on the PAWSqqP dev set. BV is high from the third layer in DistilBERT and from the fifth layer in BERT BERTsodd-- sond0-
In addition to STSB, we also extended our exper- iments to another dataset, SICK-R R (Marelli et al., 2014), using BERT to measure semantic textual similarity.
Similar to STSB, SICK-R contains human- annotated scores for English sentence pairs, reflect- ing the average similarity on a six-point scale.
DistilBERT
BERT sced0 seed0
Oth Embs.
9
Table 26 shows the number of sentence pairs in the SICK-R test set.
6th Embs.
12th Embs.
9
The scores for methods that depend on word embeddings are shown in Table 27.
Top1 layer for DistilBERT and BERTs The layer was selected from the third onward for Distil-
Table 22:
Experiments with PAWSQQP train set As seen in Section 7.7, the size of the PAWSQQP test set is small and the dataset contains only 677 sentence pairs. alleviate this problem, we present the scores on the PAWSQQP train set using our method.
BERT and the fifth onward for BERT scedo based the AUC of the average WSMD for single layer using the PAWSQQP dev set.
Experiments with other models
Table 28 shows the number of sentence pairs in the PAWSQQP train set.
To evaluate the generality of our method, con- ducted experiments not only with BERT, SimCSE, and RoBERTa, but also with DistilBERT (Sanh et al., 2019) and BERT trained with a different seed (Sellam et al., 2022). DistilBERT is a six- layer model distilled from BERT using BERT'S weights as initialization. Since BERT has 12 lay- ers, this initialization selects one layer from each corresponding pair of layers. For the model trained with different seed, chose BERT with seed referred as BERT seed0-
The scores for methods that depend on word embeddings shown in Table 29.
It should be noted that the top layer was deter- mined based on the first 1500 sentence pairs, so it implies a potential bias in the layer selection.
J Correlation between common words count and gold score in sentence pairs.
For PAWSoQP- PAWSwiki. and STSB, We present scatter plots the number of common words sentence pairs against their respective gold scores in Fig. 10. Compared to PAWS, STSB shows a strong correlation between the number of common words and the gold score. Therefore, in STSB, there is need to worry about gold score de- crease for sentence pairs with high word overlap. This suggests that even when sentences are con- sidered as sets of words, good performance can be achieved.
We used DistilBERT and BERT scodo from the Hugging Face transformers library (Wolf et al., 2020) and see Table 8 in Appendix C for details. Similar to Fig. 5, Fig. shows the plots of BV .A for each layer of DistilBERT and BERT seedo on the PAWSQQP dev set. For DiStilBERT, BV is high from the third layer, while for BERT seedo it is high from the fifth layer, although unlike BERT, the value decreases at the seventh layer.
Therefore, for DiST we compare perfor- mance using WMD on the PAWSqqP dev set with three layer selection methods: the best single layer

---

methods
weight
DisilBFRT
DisiBERTI2
PAWSctrap (AUC 100)
RFRTTaeddof
BERT, Feoci
Pool. BFRTSSoon
52.02
63.22
52,80
63.91
uniforu
54,74
45.50
52,05
IUE
35.30
45.41
DynaMax USIF Con Neg
52.02
70.00
54,59
66.71
12.26
63.03
$2.70
61.68
52.29
63.06
$2.85
63,00
52.15
63.12
$2.64
61.76
ROTS
51.71
64.95
55.25
64.37
UPWD
uniforn
56.75
53.03
67.33
20M4
55,09
53.51
08,03
66.90
58.53
53.05
67.33
53.53
66.99
top
uniforn
59:47
66.99
71 18.42)
58.87 5.84)
67.2 0.19)
66.44 13.13)
67.45
WMD
1/2
39.01 <T 5.08)
$2.46
67.38 0.05)
66
65.25
$3,06
11:72)
67.59 (1 0.600)
66,
top
5-12
1DF
57.97 41 5.53)
58.41 <1 5.97)
66.33 [] -0.66
66 66.96(7 0.19)
70 17:74)
65.63 12,57)
60. 21.23 (1 0.510)
67.35 (1 0.66)
1-12
58.07 (1 5.63)
66.87 0.08)
64 11.12)
67.29 (1 0.600
53.25
70.39
56,00
67.
63,02 (1 9U77)
70.35 (1 0000
71.65 <f 15.65)
67.93 ct 0.37)
5:12
62.88 <1 9.61)
70 0.12)
61 12.55)
67.79 (1 0:23)
WRD
1-12
62.95
54.46
0.10)
70.99
68.27
55,49
68 31
top
64-37 9.90
70.46 (0.13)
71 1634)
68.63 (1 0-31)
65.08 C (0.63)
70 0.15)
6885 (3.38)
68 (1 0.17)
1-12
04.97 (n 10.50)
7071 ct 0.12)
08 (f 13:22)
68.67 (T 0.35)
47.43
60.98
47,47
60,50
top
5-12
cosine
$4.38 (1 7:159
56.21 (1 8.78)
29,011 1.59)
60.47 (1 .0.51)
64.33 17.05)
60.43 < 12.90)
60.17 (1 .1.35)
60.29 LJ 0.233)
SynWMD
1:12
SWF
55.76 (1 8,33)
58.43
60.17 (1 0.51)
64.47
59.66 12.19)
59,51
60.27 (1 0..25)
65.20
59.73 1,30)
62.80 -167)
66.72 7.2
63,58 (] -191)
5-12
SWD
60.99
(]
63.04 -1.35)
<T 1.83)
60.50 0.97
63.08 3:57)
64.18 -1111)
Table 23: The scores the PAWSQQP test for the methods using the embeddings from DistiBBERT and BERT For WMD, WRD, and SynWMD, the scores of WSMD and the original method are compared for each layer selection. Score increases are highlighted in red and score decreases are highlighted in blue. Maximum scores for each embedding are underlined.
methods
layar
weight
cost
DiesiBFRRTO
DisiiBFRR11
PAWSSyiil (AUC x 106)
BERT D
BERT 12
Arg. Pool.
50.17
57.25
49.93
60.22
BERTS
unifrrm
IDF
52.75
52.55 58
56.12
$6.08
52.65
$2.80
57.32
$7.21
DynaMax SUF uSIP Neg ROTS
23.36
$0.34
52.50
56.44
50.10
60000
$0.05
$7.20
40.86
60.15
50.11
57.68
$4.76
6062
OPWD
unifomm
Lg
55.78
$1.98
66.96
69.55
52.31
68.76
7000
WMDo
unformm
cosine
18:45
57.16
45.62
58.79
55.78
69.95
55.69
70,90
top
5-12 1-12
uniform
38.60 <t 2.86)
58.15 (f 2.57)
64.55 (L 5.100)
69:20 (1 -0755
68:26 (1 12:57]
64.10 K41))
69.15 (1 -1.74)
68 L) -2.30
WMD
L
58 2.85)
$5.20
68 (1.-)11)))
20.19
k1 6.42)
(0 -2,60)
57.5 2.58)
(L.-5)))
67.10 11992)
-1.71
5-12
IF
57 36 2.16)
69 0.80)
630911 1.23
1-12
57 (1 2,80)
63.93 -1.26)
61.06 5.880
68.61 () -2.50)
$2.21
67:77
$1,45
68.79
top
54.75 (T 2.54)
6683 .0.44)
61 7.00)
69 071)
5:12
numm
53.98 (1: L.77)
67:2611 0,213
57.44 (159))
69161) (.37)
WRD
1-12
osiine
$4.34 2,13)
$1 1.776
67.32(1
66.69
$6.31 (1 417)
$1.06
68.99
68
0,20)
top
5-12
IDF
54.39 <t 2.63)
53.73 (.99)
66 (1-001))
16 0.20
ST 5005
59.70 7.62)
69.55 0.83)
15 0445)
1-12
54.11
(6.58 4.50
6893 0.21)
18.90
$7.08
48.69
59.52
top
asiine
51,00 2.100
55.29
56.64 7:55)
59.0
5-12
30.31 1,90
5683 0.25)
5193 5.24)
58.82 0,000
SynWMD
1-12
SWF
51,09 (.19)
$2.28
56.70 (1 0.38)
$7.10
$2.05
(1 4.16)
58.47 -1.05)
$0.11
top 5-12
SWD
53.14 0.86)
$2.45 c1 0.17)
38.09 (200
76 (L-7.
57.05 (1 5,02)
k5.11x1 3,06)
58.64 () -0.4)
(0.40 (1.7)
1-12
53:20
56.55 0.53
51.18 2.12)
58.05 (1 -1,05)
Table 24: The scores on the PAWS test set for the methods using the embeddings from DistiBERT and BERT For WMD, WRD, and SynWMD, the scores of WSMD and the original method are compared for each layer selection. Score increases are highlighted in red and score decreases are highlighted in blue. Maximum scores for each embedding are underlined.

---

(Spearann'
layer
weight
oost
DiSBBERTT
DicIBERTI2
BERTaerd0
seedo d
Arg.
BERTSSooe
witforal
60:97
DynaMax SIF s.IP Con, Neg.
70.61
72.66
60.35
51.06
61.35
71.76
59 20
47.3
65.25
71.61
59.34
47.40
63.8
71.86
61.76
53.44
ROTS
62.9
52.63
Connze Cosine
WMDo
unifors
61.05
top
5-12 1-12
uniforss
$8.50
-2.00
-7.81)
50.01 (1 -2.99
56331
4.30
10.36 (. -230)
WMD
1/2
58.07 -2.52)
64.36
40 51 -1.11)
55.95
56,09 1] -435%
61.61
41:25 (1-15 45)
48.31
top
IDF
$9,50
-6.00)
29,73 (1 6.25
56.23 (1442) .4,42)
(, (7.99
-1551)
61.26
61.60
60.73
71.30
67.06
47.85
tip
77.23
67.85
50.67
top
5-12
IDF
09.65 (1 .1.24)
88,0 L1 1.20
72.161) .00m)
71,45 (1 0.09
66.62 (1-1223)
65.16 2.09
49.08 (C-1 (51
47:26 L 3309)
1:12
09.56 () 40,39)
72.14 0.09
60,09
1110
49.18
(1499)
71.56
73,90
69.30
54.21
top
$7.29 (3.77
68.76 16.661 (511444
64.201] -5. 10
44.15
968
5-12
Con2ze
71,00 (1 -0.56)
72.56 -1.55)
67.03 -2.27)
49.75 (0.75(. 44446
SyWWMD
1-12
SWF
70,89 (1 -0.57)
70.86
72,05 -1.85)
74.67
67.101] -2.20
68.52
50,00 (_ (4.1919)
top
00.10)
1.15)
1-12
70.35 11 0.30
73.31 ( 1.300
6733 (1 -1.18
59.34 ( 2.75)
Table 25: The scores on the STSB test for the methods using the embeddings from DistilBERT and BERT seedD For WMD, WRD, and SynWMD, the scores of WSMD and the original method are compared for each layer selection. Score increases are highlighted red and score decreases are highlighted in blue. Maximum scores for each embedding underlined.
PAWSgge dev
PAWSwik dev
STSB dev
Gold Score
Gold Score
Gold Gold Score
2
10 20
30
10
30
20 Number of Common Words
Number of Common Words
Number of Common Words
(a) PAWSsge dev
(b) PAWSwa: dev
(c) STSB dev
Figure 10: Scatter plots of the number of common words in sentence pairs for each dataset against the corresponding gold scores. (a) Spearman's x 100 = -9.31 for the PAWSqoP dev set. (b) Spearman's x 100 -0.85 for the PAWSwik dev set. (c) Spearman's p 59.37 STSB dev set.

---

Test
SICK-R 4927
Table 26: The number of sentence pairs in the SICK-R K-R test set.
motheds
layer
weight
SICK-K (Spearman' a 100)
BERTO
BERT12
Asg. Pool.
41.10
62:
unifurn
50.00
58 55
BLRThoooee
IDD
50.54
58.26
DynaMax SuP
48.38
61.99
43.02
61.95
USIF
61.92
Neg.
OPWD
uniform
cosine
48.40
61.72
06
WMDo
uniform
onsine
16.93
60.70
49,97
54.06
top
uniform
50.13 (1 0.16)
$3.3911 0.87
5:12
50.14 17 0.17)
53.41 0.65)
WMD
1-12
50,00 11 0.100
5301 -1,02)
53.55
$2.9911-055 (1--665))
11
(1 0.09)
$2.771-00 -0.0)
-0000
61511 -0
5-12
42:75 0009)
61.801 0.10
WRD
1-12
oonine
47.69 .0.99)
48.13
61.34
61.29
..25)
5-12
INF
48.08 .0.05)
48.05 -0.00)
61200 .009
61 2.14
1-12
610201-22
50.50
63335
62.89
5-12
62.3511-1.00
SyWMDD
top
5-12
SWD
49.70 0.11)
49,92 11 0.33)
62 0.53)
62.7611 .0.76
1-12
49,70 0.11)
62.52 -1,02)
Table 27: The scores on the SICK-R test set for the meth- ods using the BERT embeddings. For WMD, WRD, and SynWMD, the scores of WSMD and the original method are compared for each layer selection. Score increases highlighted in red and score decreases highlighted in blue. Maximum scores for each embed- ding are underlined.
Train 11988 (31.50%)
PAWSQqP
Table 28: The number of sentence pairs in the PAWSqqP train set. The percentage of paraphrases is shown parentheses.

---

bayer
wright
PAWSQOP
ROBERTUI2
Ang.
ID0
58.51
50.73
52.52
59.3
DynaMax SIF USIF Con. Neg. ROTS
67.61
79.25
68.12
73.42
63.56
71.49
63.50
75.25
66.24
73.34
61.58
73.04
65.39
75.25
66.21
73.36
61.67
72.99
66.01
75.64
66.48
76004
61.60
73.69
76.59
74.01
F-2
76.95
cosne
uniform
cusime
68.85
top
5-12
uniform
70.42 11.77)
76,45
8.84
78.13 0.239
75.11 7.39)
7.04)
73.75 0.25)
0.55)
252100 9.15)
9.26)
23.02 (1
70.95 (-)
WMD
1-12
1/2
76.40 8,84)
67.41
78.10 0.30)
77.54
26.13 (1 8.36)
67.53
76.04 0.55)
75.08
75.06 8.71)
64.67
76.65 (.) -0.33)
76
5-12
3"
IDF
78,88 (1 11,47y
73
8:47)
78.66C7 0.62)
77.00 0.36)
74.66 7:139
75.22 (1 7.69)
75.53 0.25)
75.86 (" 0.58)
73.55 8.36)
73 8.38)
76.26 0.14) 76 () -0.04)
12:45(077.0))
Table 29: The scores on the PAWSoqP train set for the methods using the embeddings from BERT, SimCSE, and original method are compared for each layer selection. Score
RoBERTa. For WMD, the scores of WSMD increases are highlighted in red and score decreases are highlighted in blue. Maximum scores for each embedding are underlined. Note that the top layer was determined based on the first 1500 sentence pairs. For this reason, top) is marked with an asterisk.


Reviews:
[{"review_id": "ADMVuBL6Np", "reviewer": "Reviewer_YoLW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper. Below are our responses:\n\n---\n\n# Questions about the generalizability across other models and the ablation study\n> One limitation to note is that the authors exclusively tested their method using BERT and RoBERTa. An exploration into its generalizability across other leading language models and the ablation study of hyperparameters are not fully addressed.\n\nA1. Thank you for your valuable feedback. We acknowledge the concerns you've raised regarding the evaluation of our proposed method using only BERT and RoBERTa. We understand the need to evaluate its generalisability to other major language models and to perform a hyperparameter ablation study. \n\nAt this point, we cannot confirm whether the performance of all models with self-attention improves when WSMD is used. We then conducted experiments with DistilBERT, a model different from BERT and RoBERTa, and **observed performance improvements on PAWS for DistilBERT**, similar to BERT and RoBERTa. This suggests that WSMD may be applicable to a wider range of models than just BERT and RoBERTa. \n\n### Experimental details (If you are interested in the details of the experiments, please take a look.)\nDistilBERT is a light model with 6 layers and 12 heads, distilled from BERT. The BV_A values were as follows:\n|Layer|1|2|3|4|5|6|\n|---|---|---|---|---|---|---|\n|DistilBERT-$BV_A$|0.293|0.267|0.461|0.499|0.475|0.471|\n\nAlthough BERT and RoBERTa have 12 layers and DistilBERT has only 6 layers, the BV_A per layer, as seen in Fig. 5, starts low for the initial layers and increases thereafter. Therefore, we chose the layers top1, 3-6, 1-6 for WSMD in the DistilBERT experiments. The top1 layer was chosen based on the AUC from averaging the WSMD over a single layer starting from the 3rd layer using the PAWSQQP dev set, and the layers for the 0th and 6th embeddings were chosen as follows:\n||DistilBERT|\n|---|---|\n|0th Embs. |4|\n|6th Embs. |4|\n\nThen, the results on PAWSQQP, PAWSWiki, and STSB were as follows:\n### PAWSQQP\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 52.02 | 63.22 | \n| SIF |  | sif |  | 52.26 | 63.03 | \n| uSIF |  | usif |  | 52.29 | 63.06 | \n| Con. Neg. |  | sif |  | 52.15 | 63.12 | \n| BERTScore |  | uniform |  | 45.58 | 54.74 | \n| BERTScore |  | IDF |  | 45.46 | 55.30 | \n| ROTS |  | sif |  | 51.71 | 64.95 | \n| OPWD |  | uniform | $L_2$ | 53.03 | 67.33 | \n| OPWD |  | uniform | cosine | 54.73 | **70.93** | \n| WMDo |  | uniform | cosine | 44.18 | 55.42 | \n| WMD |  | uniform | $L_2$ | 53.03 | 67.33 | \n| WMD | top1 | uniform | $L_2$ | 61.53 (+8.50) | 67.77 (+0.44) | \n| WMD | 3-6 | uniform | $L_2$ | 58.69 (+5.66) | 67.64 (+0.31) | \n| WMD | 1-6 | uniform | $L_2$ | 59.01 (+5.98) | 67.38 (+0.05) | \n| WMD |  | IDF | $L_2$ | 52.44 | 66.79 | \n| WMD | top1 | IDF | $L_2$ | 60.38 (+7.94) | 67.32 (+0.53) | \n| WMD | 3-6 | IDF | $L_2$ | 57.90 (+5.46) | 67.13 (+0.34) | \n| WMD | 1-6 | IDF | $L_2$ | 58.07 (+5.63) | 66.87 (+0.08) | \n| WRD |  | norm | cosine | 53.25 | 70.39 | \n| WRD | top1 | norm | cosine | 65.44 (+12.19) | 70.66 (+0.27) | \n| WRD | 3-6 | norm | cosine | 62.57 (+9.32) | 70.58 (+0.19) | \n| WRD | 1-6 | norm | cosine | 62.95 (+9.70) | 70.50 (+0.11) | \n| WRD |  | IDF | cosine | 54.46 | 70.59 | \n| WRD | top1 | IDF | cosine | **66.81** (+12.35) | **70.93** (+0.34) | \n| WRD | 3-6 | IDF | cosine | 65.07 (+10.61) | 70.82 (+0.23) | \n| WRD | 1-6 | IDF | cosine | 64.97 (+10.51) | 70.71 (+0.12) | \n| SynWMD |  | SWF | cosine | 47.43 | 60.98 | \n| SynWMD | top1 | SWF | cosine | 58.71 (+11.28) | 61.07 (+0.09) | \n| SynWMD | 3-6 | SWF | cosine | 56.40 (+8.97) | 60.72 (-0.26) | \n| SynWMD | 1-6 | SWF | cosine | 55.76 (+8.33) | 60.17 (-0.81) | \n| SynWMD |  | SWF | SWD | 58.43 | 64.47 | \n| SynWMD | top1 | SWF | SWD | 61.51 (+3.08) | 63.99 (-0.48) | \n| SynWMD | 3-6 | SWF | SWD | 60.00 (+1.57) | 63.93 (-0.54) | \n| SynWMD | 1-6 | SWF | SWD | 60.06 (+1.63) | 63.50 (-0.97) | \n\n### PAWSWiki\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 53.36 | 60.17 | \n| SIF |  | sif |  | 53.25 | 60.20 | \n| uSIF |  | usif |  | 53.18 | 60.33 | \n| Con. Neg. |  | sif |  | 53.33 | 60.40 | \n| BERTScore |  | uniform |  | 52.06 | 59.78 | \n| BERTScore |  | IDF |  | 52.25 | 59.78 | \n| ROTS |  | sif |  | 56.73 | 62.68 | \n| OPWD |  | uniform | $L_2$ | 57.43 | 71.99 | \n| OPWD |  | uniform | cosine | 54.86 | 69.95 | \n| WMDo |  | uniform | cosine | 49.69 | 58.62 | \n| WMD |  | uniform | $L_2$ | 57.43 | 71.99 | \n| WMD | top1 | uniform | $L_2$ | **61.45** (+4.02) | 72.56 (+0.57) | \n| WMD | 3-6 | uniform | $L_2$ | 59.81 (+2.38) | 72.07 (+0.08) | \n| WMD | 1-6 | uniform | $L_2$ | 60.72 (+3.29) | 71.69 (-0.30) | \n| WMD |  | IDF | $L_2$ | 57.02 | 72.23 | \n| WMD | top1 | IDF | $L_2$ | 60.79 (+3.77) | **72.81** (+0.58) | \n| WMD | 3-6 | IDF | $L_2$ | 59.29 (+2.27) | 72.30 (+0.07) | \n| WMD | 1-6 | IDF | $L_2$ | 60.11 (+3.09) | 71.78 (-0.45) | \n| WRD |  | norm | cosine | 55.11 | 70.48 | \n| WRD | top1 | norm | cosine | 58.68 (+3.57) | 70.89 (+0.41) | \n| WRD | 3-6 | norm | cosine | 57.51 (+2.40) | 70.63 (+0.15) | \n| WRD | 1-6 | norm | cosine | 58.43 (+3.32) | 70.65 (+0.17) | \n| WRD |  | IDF | cosine | 54.38 | 69.77 | \n| WRD | top1 | IDF | cosine | 58.39 (+4.01) | 70.21 (+0.44) | \n| WRD | 3-6 | IDF | cosine | 57.17 (+2.79) | 69.91 (+0.14) | \n| WRD | 1-6 | IDF | cosine | 58.17 (+3.79) | 69.90 (+0.13) | \n| SynWMD |  | SWF | cosine | 50.43 | 60.70 | \n| SynWMD | top1 | SWF | cosine | 53.94 (+3.51) | 60.78 (+0.08) | \n| SynWMD | 3-6 | SWF | cosine | 52.47 (+2.04) | 60.42 (-0.28) | \n| SynWMD | 1-6 | SWF | cosine | 53.08 (+2.65) | 59.96 (-0.74) | \n| SynWMD |  | SWF | SWD | 52.94 | 60.52 | \n| SynWMD | top1 | SWF | SWD | 54.90 (+1.96) | 60.36 (-0.16) | \n| SynWMD | 3-6 | SWF | SWD | 53.95 (+1.01) | 60.11 (-0.41) | \n| SynWMD | 1-6 | SWF | SWD | 54.13 (+1.19) | 59.67 (-0.85) | \n\n### STSB\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 66.33 | 71.83 | \n| SIF |  | sif |  | 65.35 | 71.76 | \n| uSIF |  | usif |  | 65.25 | 71.64 | \n| Con. Neg. |  | sif |  | 65.81 | 71.86 | \n| BERTScore |  | uniform |  | 63.30 | 63.01 | \n| BERTScore |  | IDF |  | 64.16 | 63.92 | \n| ROTS |  | sif |  | 66.61 | 71.98 | \n| OPWD |  | uniform | $L_2$ | 60.59 | 52.63 | \n| OPWD |  | uniform | cosine | 67.29 | 70.93 | \n| WMDo |  | uniform | cosine | 66.59 | 70.13 | \n| WMD |  | uniform | $L_2$ | 60.59 | 52.63 | \n| WMD | top1 | uniform | $L_2$ | 58.88 (-1.71) | 50.20 (-2.43) | \n| WMD | 3-6 | uniform | $L_2$ | 59.24 (-1.35) | 50.66 (-1.97) | \n| WMD | 1-6 | uniform | $L_2$ | 58.07 (-2.52) | 49.52 (-3.11) | \n| WMD |  | IDF | $L_2$ | 64.36 | 55.98 | \n| WMD | top1 | IDF | $L_2$ | 63.11 (-1.25) | 53.90 (-2.08) | \n| WMD | 3-6 | IDF | $L_2$ | 63.39 (-0.97) | 54.25 (-1.73) | \n| WMD | 1-6 | IDF | $L_2$ | 62.53 (-1.83) | 53.38 (-2.60) | \n| WRD |  | norm | cosine | 69.73 | 71.30 | \n| WRD | top1 | norm | cosine | 69.57 (-0.16) | 71.15 (-0.15) | \n| WRD | 3-6 | norm | cosine | 69.63 (-0.10) | 71.19 (-0.11) | \n| WRD | 1-6 | norm | cosine | 69.47 (-0.26) | 71.09 (-0.21) | \n| WRD |  | IDF | cosine | 69.89 | 72.23 | \n| WRD | top1 | IDF | cosine | 69.68 (-0.21) | 72.16 (-0.07) | \n| WRD | 3-6 | IDF | cosine | 69.74 (-0.15) | 72.20 (-0.03) | \n| WRD | 1-6 | IDF | cosine | 69.56 (-0.33) | 72.14 (-0.09) | \n| SynWMD |  | SWF | cosine | **71.56** | 73.90 | \n| SynWMD | top1 | SWF | cosine | 71.17 (-0.39) | 72.20 (-1.70) | \n| SynWMD | 3-6 | SWF | cosine | 71.39 (-0.17) | 72.69 (-1.21) | \n| SynWMD | 1-6 | SWF | cosine | 70.69 (-0.87) | 72.05 (-1.85) | \n| SynWMD |  | SWF | SWD | 70.86 | **74.67** | \n| SynWMD | top1 | SWF | SWD | 70.89 (+0.03) | 73.26 (-1.41) | \n| SynWMD | 3-6 | SWF | SWD | 71.02 (+0.16) | 73.65 (-1.02) | \n| SynWMD | 1-6 | SWF | SWD | 70.55 (-0.31) | 73.31 (-1.36) | \n\nThese experimental results will be included in the camera-ready version. We appreciate your constructive comments.\n\n---\n\n# Question about the evaluation dataset size\n\n> The evaluation dataset size is quite small.\n\nA2. We acknowledge your point about the small size of the evaluation dataset. \n**We intend to repeat the experiments** using the full PAWS dataset and other STS datasets such as STS12-16 and SICKR,  and include the results in the camera-ready version as long as time permits. Thank you again for your constructive feedback.\n\nTo provide more context,  I explain the reason. The original dataset sizes for PAWSQQP, PAWSWiki, and STSB are as follows:\n\n|Dataset|Dev|Test|\n|---|---|---|\n|PAWSQQP|11988|677|\n|PAWSWiki|8000|8000|\n|STSB|1500|1379|\n\nSince the data sizes for the PAWSQQP and PAWSWiki test sets are significantly different in our study, we adjusted them for experimental efficiency. We adjusted the sizes of the PAWSQQP dev and PAWSWiki dev and test sets to the STSB dev size of 1500 because STSB is one of the most widely used datasets and we felt that the size of 1500 sentence pairs was not too small, although the PAWSQQP test set is relatively small.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors proposed a small modification for WMD, and effective for paraphrase identification.\n- The authors conducted a comprehensive empirical study on the proposed method.\n- The paper is well-written and easy to understand.", "reasons_to_reject": "- One limitation to note is that the authors exclusively tested their method using BERT and RoBERTa. An exploration into its generalizability across other leading language models and the ablation study of hyperparameters are not fully addressed.\n- The evaluation dataset size is quite small.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors proposed a small modification for World Mover's Distance (WMD) to improve paraphrases identification, They found that the World Mover's Distance (WMD) does not consider word order, which poses difficulties in distinguishing between sentences that have a substantial overlap of similar words but divergent semantic meanings. Therefore, they introduced an enhancement to the WMD by integrating sentence structures as represented by BERT's self-attention matrix (SAM). The suggested approach is grounded on the Fused Gromov-Wasserstein distance, which concurrently factors in the word embedding similarity and the SAM when determining the optimal transport between two sentences. Experimental results reveal that the proposed approach improves upon the WMD and its variants on BERT, particularly in paraphrase identification, while maintaining near-equivalent performance in semantic textual similarity tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LDVGDv09aK", "reviewer": "Reviewer_ycfD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper. Below are our responses:\n\n---\n\n# Question about the performance on STSB\n\n> The improvement in performance on the STS Benchmark dataset for semantic text similarity was very limited.\n\nA1. Thank you for highlighting this issue. Indeed, the performance improvement on the STS semantic text similarity benchmark dataset was quite limited. In the future, we are considering dynamically calculating the lambda for each sentence pair to ensure that the GW term adjusts as needed. \n\nTo provide more context, we explain the reason behind this issue. As can be seen in Fig. 9 in Appendix G, there is a trend in STSB where **the gold score increases as the number of common words increases**. As a result, methods that treat sentences as sets of words can achieve good performance, and in WSMD, the GW term can potentially act as noise.\n\n---\n\n# Question about the method limitation of the transferability and the general applicability\n\n> Different pre-trained language models use different tokenization methods, which limits the transferability of the proposed method and its general applicability.\n\nA2. Thank you for highlighting this issue. Certainly, different pre-trained language models use different tokenization methods, which challenges the transferability of our proposed method and its general applicability. We believe that **WSMD can address this issue by mapping tokens derived from each tokenizer**. \n\nFor example, both SynWMD and ROTS use different tokenizers than BERT to derive parsing trees from sentences. They address this problem in their implementations by using the mapping strategy.\n\n---\n\n# Question about the evaluation dataset size\n\n> The size and scope of the two datasets used in the paper were relatively small. If the proposed method could be validated on more diverse and large-scale datasets, the results would be more convincing.\n\nA3. Thank you for highlighting this issue. Indeed, the size and scope of the two datasets used in our paper were relatively small. \n**We intend to repeat the experiments** using the full PAWS dataset and other STS datasets such as STS12-16 and SICKR,  and include the results in the camera-ready version as long as time permits. Thank you again for your constructive feedback.\n\nTo provide more context,  I explain the reason. The original dataset sizes for PAWSQQP, PAWSWiki, and STSB are as follows:\n\n|Dataset|Dev|Test|\n|---|---|---|\n|PAWSQQP|11988|677|\n|PAWSWiki|8000|8000|\n|STSB|1500|1379|\n\nSince the data sizes for the PAWSQQP and PAWSWiki test sets are significantly different in our study, we adjusted them for experimental efficiency. We adjusted the sizes of the PAWSQQP dev and PAWSWiki dev and test sets to the STSB dev size of 1500 because STSB is one of the most widely used datasets and we felt that the size of 1500 sentence pairs was not too small, although the PAWSQQP test set is relatively small.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new idea was proposed to use the self-attention mechanism of BERT to represent the syntactic structure information of sentences, which is an aspect that has been less considered in previous work on short text representation.\n\nThe WSMD method was developed by adding sentence structure matching to the existing word vector similarity, optimizing both word vector distance and sentence structure distance, thus improving the quality of sentence representation.\n\nThe SMD based on sentence self-attention matrix was designed, providing an innovative way to measure structural similarity.\n\nThe effectiveness of the proposed method was thoroughly verified on the PAWS dataset for sentence similarity tasks, and the results show that WSMD outperforms the baseline methods, demonstrating its effectiveness.", "reasons_to_reject": "The improvement in performance on the STS Benchmark dataset for semantic text similarity was very limited.\n\nDifferent pre-trained language models use different tokenization methods, which limits the transferability of the proposed method and its general applicability.\n\nThe size and scope of the two datasets used in the paper were relatively small. If the proposed method could be validated on more diverse and large-scale datasets, the results would be more convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper mainly focuses on improving the Word Mover's Distance (WMD) method for short text representation. A new Word and Sentence Mover's Distance (WSMD) is proposed and validated to optimize both word vector distance and sentence structure distance, in order to more accurately characterize sentence semantics. \nThe main contributions are:\n\nThe Word Mover's Distance (WMD) method cannot consider word order information when calculating sentence similarity, which may result in high similarity scores for sentences with different meanings. The proposed Word and Sentence Mover's Distance (WSMD) method simultaneously utilizes both word vector distance and sentence structure distance, allowing sentence structure information to participate in the calculation of similarity.\n\nThe authors introduced the self-attention matrix (SAM) output by the BERT model to represent sentence structure and designed the SAM-based Sentence Structure Distance (SMD).\n\nThey also proposed the WSMD distance metric, which simultaneously optimizes both word vector distance and sentence structure distance. The WSMD effectively combines word vector information and sentence structure information to compute sentence similarity.\n\nThe effectiveness of WSMD was validated on the PAWS dataset for sentence similarity judgment tasks, and the results outperformed multiple baselines. This provides a new approach for sentence representation, characterizing sentence semantics through matching word vectors and syntactic structures.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "snhx8EueiH", "reviewer": "Reviewer_RMgW", "summary": "", "questions": "- Will other variations of BERT also comply the idea of using Self-Attention matrix for measuring semantic similarity?\n  - If yes, I think the metric contributes a lot to NLP field (but the authors should show the result)\n  - Otherwise, the contribution is limited to be applicable in BERT model.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper. Below are our responses:\n\n---\n\n# Question about the paper readability\n\n> In writing, the paper is hard to understand. In particular, Figure 4 and Figure 6 seem important to support the method, but it is not readable in this state.\n    \n\nA1. We appreciate your insightful suggestions. Indeed, we acknowledge that our paper contained sections that may have been difficult to understand, including Fig. 4 and Fig. 6. Therefore, we will review these sections and figures, and update the paper to improve clarity in the camera-ready version. \n\n### Brief explanation of our proposed method (please skip this part if you're already familiar with it)\nFirst, while the Wasserstein distance-based WMD is used to measure sentence similarity, it treats sentences as sets. This means that it cannot distinguish between two semantically different sentences such as\n\n- The President greets the press in Chicago.\n- The press greets the President in Chicago.\n\nTo address this problem, we focused on the Fused Gromov-Wasserstein distance, which combines the Wasserstein distance with the structural information provided by the Gromov-Wasserstein distance. We proposed the WSMD, which uses a WMD-like method for the Wasserstein distance term, and the Self-Attention Matrix (SAM), which encodes sentence structure information, for the Gromov-Wasserstein term. We believe that **our approach is novel for two reasons**:\n\n1. The use of the Fused Gromov-Wasserstein distance for sentence similarity is novel. While WMD based on Wasserstein distance is well known, to our knowledge, no method has been proposed to measure sentence similarity using the Fused Gromov-Wasserstein distance.\n2. Demonstrate high performance using SAM in the Gromov-Wasserstein term. Optimal transport distances allow for custom distribution and cost function definitions, providing significant research value. For example,  WRD, which improves on WMD, defines its distribution and cost function using word embedding length and cosine similarity, and is known to work effectively.\n\n### Explanation of Fig. 4 and Fig. 6\nRegarding Fig. 4, BERT has 12 layers and 12 heads, resulting in 144 SAM options for WSMD. **Fig. 4 is a scatter plot showing the AUC for both PAWSQQP and PAWSWiki calculated using WSMD for these 144 SAMs.** Since each layer contains 12 heads, we also show the scores averaged over the WSMDs using these heads. Performance varies significantly between different heads because they capture different information, but when averaged across a layer, the variability is reduced. Therefore, considering its use in other datasets such as STSB, we decided to select a layer, compute the WSMD using SAMs within that layer, and evaluate performance based on the average of those values. I will discuss the layer in more detail in response to A2.\n\nFinally, **Fig. 6 provides a visual representation of the results in Tables 18, 19, and 20 of Appendix F.** For WMD, WRD, and SynWMD, we set their respective scores as **Baseline Score** and the scores with WSMD as **Score with WSMD**. If a method is above the diagonal line, it indicates improved performance with WSMD. Note that certain baselines, such as SIF, cannot use WSMD, so they are plotted on the diagonal line.\n\n---\n\n# Recommendation to move ablation parts to the appendix\n\n> I recommend the authors to move ablation parts (e.g., selecting BERT's head) to Appendix and focus on highlightening their method in the main part.\n\nA2. We appreciate your insightful suggestions. We will rewrite our paper to make it more reader-friendly and may incorporate these changes in the camera-ready version. Thank you for your constructive feedback.\n\nTo provide more context, we explain the reason why Section 7.5 on ablation parts is in the main text. While WMD is a fully unsupervised method, WSMD requires the selection of SAM and lambda. So, we set lambda to 0.5, and if a dev set is available, we use the SAMs from the best performing layer on the dev set for WSMD. \n\nIn cases without a dev set, we propose a fully unsupervised method to select multiple layers and use their SAMs for WSMD. The most naive layer selection in the absence of a dev set would be to use all layers. However, in order to select layers with more structural information, we defined BV_A and also presented results when selecting layers 5-12, which have high BV_A values. \n\nGiven this context, we felt that the method of selecting the BERT head was important enough to be included in the main text, although it may be worth moving Section 7.5 to the Appendix to give more focus to our method.\n\n---\n\n# Question about no comparison with other general methods\n\n> No comparison with other general methods to measure semantic similarity. Although I can see the method shows better score than the traditional methods, at least, BERTScore should be presented.\n\nA3.  Thank you for highlighting this issue. Your point about preparing an appropriate baseline to argue for the empirical performance of our proposed method convincingly is well taken. We also appreciate the suggestion of BERTScore, a relevant prior work that measures sentence similarity based on word-to-word correspondences. \n\nWe then conducted comparative experiments using BERTScore as a baseline.  We find that **the performance of BERTScore was not high compared to WSMD, especially on PAWS**. This is due to the fact that BERTScore, like WMD and SIF, treats sentences as sets. This method struggles to identify paraphrases with high overlap. While BERTScore is widely used as an automatic evaluation metric for generated sentences, our results suggest that there is room for improvement in BERTScore.\n\n### Experimental details (If you are interested in the details of the experiments, please take a look.)\nThe DistilBERT results on PAWSQQP, PAWSWiki, and STSB were as follows:\n\n### PAWSQQP \n| method | weight | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|\n| BERTScore | uniform | 45.52 | 54.78 | 45.41 | 51.74 | 41.91 | 51.61 | \n| BERTScore | idf | 45.40 | 55.29 | 45.25 | 51.98 | 42.71 | 53.01 | \n\n### PAWSWiki \n| method | weight | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|\n| BERTScore | uniform | 52.16 | 62.45 | 52.15 | 56.09 | 53.26 | 60.34 | \n| BERTScore | idf | 52.29 | 62.53 | 52.27 | 56.10 | 51.96 | 59.29 | \n\n### STSB \n| method | weight | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|\n| BERTScore | uniform | 63.27 | 65.13 | 62.90 | 70.59 | 63.38 | 58.71 | \n| BERTScore | idf | 63.99 | 65.96 | 63.72 | 70.96 | 64.09 | 59.82 | \n\nUsing 1-12 layers, we then show the WSMD results of uniform weight WMD from Tables 18 and 19 on PAWS, and those of SWD cost SynWMD from Table 20 on STSB for unsupervised comparison with BERTScore:\n\n### PAWSQQP\n| method | layer | weight | cost | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|---|---|\n|WMD|1-12|uniform|$L_2$|59.41|66.71|57.54|66.13|58.33 |66.76|\n\n### PAWSWiki\n| method | layer | weight | cost | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|---|---|\n|WMD|1-12|uniform|$L_2$|61.95 | 71.72 | 59.52| 66.50 | 62.71| 69.99 |\n\n### STSB\n| method | layer | weight | cost | BERT0 | BERT12 | SimCSE0 | SimCSE12 | RoBERTa0 | RoBERTa12 |\n|---|---|---|---|---|---|---|---|---|---|\n| SynWMD | 1-12 | SWF | SWD | 71.73 | 71.97| 66.96 | 78.05 | 71.79 | 68.98|\n\nThese results show that the performance of BERTScore was not high compared to WSMD, especially on PAWS. These experimental results will be included in the camera-ready version. We appreciate your constructive comments.\n\n---\n\n# Quesiton about the applicablity of the method for other BERT-based models\n\n> Will other variations of BERT also comply the idea of using Self-Attention matrix for measuring semantic similarity?\n\nA4. Thank you for highlighting this issue. We believe the answer is yes. We have done similar experiments with DistilBERT, which has a different number of layers compared to BERT and RoBERTa, and **observed performance improvements with WSMD on PAWS**. \nThis suggests that WSMD may be applicable to a wider range of models than just BERT and RoBERTa. \n\n### Experimental details (If you are interested in the details of the experiments, please take a look.)\nDistilBERT is a light model with 6 layers and 12 heads, distilled from BERT. The BV_A values were as follows:\n|Layer|1|2|3|4|5|6|\n|---|---|---|---|---|---|---|\n|DistilBERT-$BV_A$|0.293|0.267|0.461|0.499|0.475|0.471|\n\nAlthough BERT and RoBERTa have 12 layers and DistilBERT has only 6 layers, the BV_A per layer, as seen in Fig. 5, starts low for the initial layers and increases thereafter. Therefore, we chose the layers top1, 3-6, 1-6 for WSMD in the DistilBERT experiments. The top1 layer was chosen based on the AUC from averaging the WSMD over a single layer starting from the 3rd layer using the PAWSQQP dev set, and the layers for the 0th and 6th embeddings were chosen as follows:\n||DistilBERT|\n|---|---|\n|0th Embs. |4|\n|6th Embs. |4|\n\nThen, the results on PAWSQQP, PAWSWiki, and STSB were as follows:\n### PAWSQQP\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 52.02 | 63.22 | \n| SIF |  | sif |  | 52.26 | 63.03 | \n| uSIF |  | usif |  | 52.29 | 63.06 | \n| Con. Neg. |  | sif |  | 52.15 | 63.12 | \n| BERTScore |  | uniform |  | 45.58 | 54.74 | \n| BERTScore |  | IDF |  | 45.46 | 55.30 | \n| ROTS |  | sif |  | 51.71 | 64.95 | \n| OPWD |  | uniform | $L_2$ | 53.03 | 67.33 | \n| OPWD |  | uniform | cosine | 54.73 | **70.93** | \n| WMDo |  | uniform | cosine | 44.18 | 55.42 | \n| WMD |  | uniform | $L_2$ | 53.03 | 67.33 | \n| WMD | top1 | uniform | $L_2$ | 61.53 (+8.50) | 67.77 (+0.44) | \n| WMD | 3-6 | uniform | $L_2$ | 58.69 (+5.66) | 67.64 (+0.31) | \n| WMD | 1-6 | uniform | $L_2$ | 59.01 (+5.98) | 67.38 (+0.05) | \n| WMD |  | IDF | $L_2$ | 52.44 | 66.79 | \n| WMD | top1 | IDF | $L_2$ | 60.38 (+7.94) | 67.32 (+0.53) | \n| WMD | 3-6 | IDF | $L_2$ | 57.90 (+5.46) | 67.13 (+0.34) | \n| WMD | 1-6 | IDF | $L_2$ | 58.07 (+5.63) | 66.87 (+0.08) | \n| WRD |  | norm | cosine | 53.25 | 70.39 | \n| WRD | top1 | norm | cosine | 65.44 (+12.19) | 70.66 (+0.27) | \n| WRD | 3-6 | norm | cosine | 62.57 (+9.32) | 70.58 (+0.19) | \n| WRD | 1-6 | norm | cosine | 62.95 (+9.70) | 70.50 (+0.11) | \n| WRD |  | IDF | cosine | 54.46 | 70.59 | \n| WRD | top1 | IDF | cosine | **66.81** (+12.35) | **70.93** (+0.34) | \n| WRD | 3-6 | IDF | cosine | 65.07 (+10.61) | 70.82 (+0.23) | \n| WRD | 1-6 | IDF | cosine | 64.97 (+10.51) | 70.71 (+0.12) | \n| SynWMD |  | SWF | cosine | 47.43 | 60.98 | \n| SynWMD | top1 | SWF | cosine | 58.71 (+11.28) | 61.07 (+0.09) | \n| SynWMD | 3-6 | SWF | cosine | 56.40 (+8.97) | 60.72 (-0.26) | \n| SynWMD | 1-6 | SWF | cosine | 55.76 (+8.33) | 60.17 (-0.81) | \n| SynWMD |  | SWF | SWD | 58.43 | 64.47 | \n| SynWMD | top1 | SWF | SWD | 61.51 (+3.08) | 63.99 (-0.48) | \n| SynWMD | 3-6 | SWF | SWD | 60.00 (+1.57) | 63.93 (-0.54) | \n| SynWMD | 1-6 | SWF | SWD | 60.06 (+1.63) | 63.50 (-0.97) | \n\n### PAWSWiki\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 53.36 | 60.17 | \n| SIF |  | sif |  | 53.25 | 60.20 | \n| uSIF |  | usif |  | 53.18 | 60.33 | \n| Con. Neg. |  | sif |  | 53.33 | 60.40 | \n| BERTScore |  | uniform |  | 52.06 | 59.78 | \n| BERTScore |  | IDF |  | 52.25 | 59.78 | \n| ROTS |  | sif |  | 56.73 | 62.68 | \n| OPWD |  | uniform | $L_2$ | 57.43 | 71.99 | \n| OPWD |  | uniform | cosine | 54.86 | 69.95 | \n| WMDo |  | uniform | cosine | 49.69 | 58.62 | \n| WMD |  | uniform | $L_2$ | 57.43 | 71.99 | \n| WMD | top1 | uniform | $L_2$ | **61.45** (+4.02) | 72.56 (+0.57) | \n| WMD | 3-6 | uniform | $L_2$ | 59.81 (+2.38) | 72.07 (+0.08) | \n| WMD | 1-6 | uniform | $L_2$ | 60.72 (+3.29) | 71.69 (-0.30) | \n| WMD |  | IDF | $L_2$ | 57.02 | 72.23 | \n| WMD | top1 | IDF | $L_2$ | 60.79 (+3.77) | **72.81** (+0.58) | \n| WMD | 3-6 | IDF | $L_2$ | 59.29 (+2.27) | 72.30 (+0.07) | \n| WMD | 1-6 | IDF | $L_2$ | 60.11 (+3.09) | 71.78 (-0.45) | \n| WRD |  | norm | cosine | 55.11 | 70.48 | \n| WRD | top1 | norm | cosine | 58.68 (+3.57) | 70.89 (+0.41) | \n| WRD | 3-6 | norm | cosine | 57.51 (+2.40) | 70.63 (+0.15) | \n| WRD | 1-6 | norm | cosine | 58.43 (+3.32) | 70.65 (+0.17) | \n| WRD |  | IDF | cosine | 54.38 | 69.77 | \n| WRD | top1 | IDF | cosine | 58.39 (+4.01) | 70.21 (+0.44) | \n| WRD | 3-6 | IDF | cosine | 57.17 (+2.79) | 69.91 (+0.14) | \n| WRD | 1-6 | IDF | cosine | 58.17 (+3.79) | 69.90 (+0.13) | \n| SynWMD |  | SWF | cosine | 50.43 | 60.70 | \n| SynWMD | top1 | SWF | cosine | 53.94 (+3.51) | 60.78 (+0.08) | \n| SynWMD | 3-6 | SWF | cosine | 52.47 (+2.04) | 60.42 (-0.28) | \n| SynWMD | 1-6 | SWF | cosine | 53.08 (+2.65) | 59.96 (-0.74) | \n| SynWMD |  | SWF | SWD | 52.94 | 60.52 | \n| SynWMD | top1 | SWF | SWD | 54.90 (+1.96) | 60.36 (-0.16) | \n| SynWMD | 3-6 | SWF | SWD | 53.95 (+1.01) | 60.11 (-0.41) | \n| SynWMD | 1-6 | SWF | SWD | 54.13 (+1.19) | 59.67 (-0.85) | \n\n### STSB\n| method | layer | weight | cost | DistilBERT0 | DistilBERT6 |\n|---|---|---|---|---|---|\n| Avg. Pool. |  |  |  | 66.33 | 71.83 | \n| SIF |  | sif |  | 65.35 | 71.76 | \n| uSIF |  | usif |  | 65.25 | 71.64 | \n| Con. Neg. |  | sif |  | 65.81 | 71.86 | \n| BERTScore |  | uniform |  | 63.30 | 63.01 | \n| BERTScore |  | IDF |  | 64.16 | 63.92 | \n| ROTS |  | sif |  | 66.61 | 71.98 | \n| OPWD |  | uniform | $L_2$ | 60.59 | 52.63 | \n| OPWD |  | uniform | cosine | 67.29 | 70.93 | \n| WMDo |  | uniform | cosine | 66.59 | 70.13 | \n| WMD |  | uniform | $L_2$ | 60.59 | 52.63 | \n| WMD | top1 | uniform | $L_2$ | 58.88 (-1.71) | 50.20 (-2.43) | \n| WMD | 3-6 | uniform | $L_2$ | 59.24 (-1.35) | 50.66 (-1.97) | \n| WMD | 1-6 | uniform | $L_2$ | 58.07 (-2.52) | 49.52 (-3.11) | \n| WMD |  | IDF | $L_2$ | 64.36 | 55.98 | \n| WMD | top1 | IDF | $L_2$ | 63.11 (-1.25) | 53.90 (-2.08) | \n| WMD | 3-6 | IDF | $L_2$ | 63.39 (-0.97) | 54.25 (-1.73) | \n| WMD | 1-6 | IDF | $L_2$ | 62.53 (-1.83) | 53.38 (-2.60) | \n| WRD |  | norm | cosine | 69.73 | 71.30 | \n| WRD | top1 | norm | cosine | 69.57 (-0.16) | 71.15 (-0.15) | \n| WRD | 3-6 | norm | cosine | 69.63 (-0.10) | 71.19 (-0.11) | \n| WRD | 1-6 | norm | cosine | 69.47 (-0.26) | 71.09 (-0.21) | \n| WRD |  | IDF | cosine | 69.89 | 72.23 | \n| WRD | top1 | IDF | cosine | 69.68 (-0.21) | 72.16 (-0.07) | \n| WRD | 3-6 | IDF | cosine | 69.74 (-0.15) | 72.20 (-0.03) | \n| WRD | 1-6 | IDF | cosine | 69.56 (-0.33) | 72.14 (-0.09) | \n| SynWMD |  | SWF | cosine | **71.56** | 73.90 | \n| SynWMD | top1 | SWF | cosine | 71.17 (-0.39) | 72.20 (-1.70) | \n| SynWMD | 3-6 | SWF | cosine | 71.39 (-0.17) | 72.69 (-1.21) | \n| SynWMD | 1-6 | SWF | cosine | 70.69 (-0.87) | 72.05 (-1.85) | \n| SynWMD |  | SWF | SWD | 70.86 | **74.67** | \n| SynWMD | top1 | SWF | SWD | 70.89 (+0.03) | 73.26 (-1.41) | \n| SynWMD | 3-6 | SWF | SWD | 71.02 (+0.16) | 73.65 (-1.02) | \n| SynWMD | 1-6 | SWF | SWD | 70.55 (-0.31) | 73.31 (-1.36) | \n\nThese experimental results will be included in the camera-ready version. We appreciate your constructive comments.\n\n---\n\n# Question about typos\n\n> Line286: m->j\n\nA5. We apologize for any confusion caused by unclear explanations. We would like to clarify that the point raised was **not a typographical error**. Such feedback is invaluable in helping us to produce a more accessible manuscript for readers. \n\nWe will provide a detailed explanation of the point raised. Regarding the comment about whether $m$ is a typo for $j$ on line 286, the notation $P_{ij}$ represents the transport amount from the $i$-th word of sentence $s$ to the $j$-th word of sentence $s'$, and $p_i=(P_{i1},\\cdots,P_{im})$ represents the array of transport amounts from the $i$-th word of $s$ to the $j$-th word of $s’$, where $j=1,\\cdots,m$. As shown in Fig. 3, the equation $\\sum_{j=1}^m P_{ij} = u_i$ holds. We acknowledge the complexity of the notation and will try to provide a clearer explanation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is simple and straightforward", "reasons_to_reject": "- In writing, the paper is hard to understand\n  - In particular, Figure 4 and Figure 6 seem important to support the method, but it is not readable in this state.\n  - I recommend the authors to move ablation parts (e.g., selecting BERT's head) to Appendix and focus on highlightening their method in the main part.\n\n- No comparison with other general methods to measure semantic similarity\n  - Although I can see the method shows better score than the traditional methods,\n  - at least, BERTscore should be presented.", "questions_to_authors": "- Will other variations of BERT also comply the idea of using Self-Attention matrix for measuring semantic similarity?\n  - If yes, I think the metric contributes a lot to NLP field (but the authors should show the result)\n  - Otherwise, the contribution is limited to be applicable in BERT model.", "paper_topic_and_main_contributions": "This paper tackled the problem of WMDs, which does not consider the order of words in a sentence.\nThus, the authors propose a method to incorporate GW and Self-attention matrix of pretrained BERT, which is called as SMD.\nFurthermore, they combine WMD and SMD with a hyperparameter, denoted by WSMD.\n\nHowever, I can hardly understand the results and the explanations.\n\n--- After rebuttal ---\n\nI updated my scores toward positive. Thanks for the clear explanation and I hope those explanations will be included in the final version.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line286: m->j"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the Word Mover's Distance (WMD), which measures sentence similarity, by incorporating structural information based on the self-attention matrix of encoder language models (e.g., BERT). Thus, the main contribution is a solid extension of an existing technique. Generally, this paper is well-written and easy to follow, although a Reviewer highlighted that some figures are hard to interpret. Despite the experiments being fairly extensive and based on significant sample sizes (as shown in the rebuttal), the results are mixed. In fact, the Authors report gains in paragraph identification but no change in performance in semantic textual similarity. Thus, the new method seems beneficial only in a subset of sentence similarity tasks. Based on the rebuttal, the Authors convincingly demonstrated that these findings hold true independent of the choice of backbone language model (although it remains uncertain how the method behaves at larger model scales). Potential limitations that remain to be fully addressed are 1) the fact that different encoders may rely on different vocabularies (as a result of different tokenizers), as the proposed solution (i.e., mapping) remains error-prone; 2) a thorough comparison with baselines for sentence similarity based on methods different from WMD, such as BERTScore, which already take into account structural information. The rebuttal has added some of these results to the initial submission; however, they remain incomplete and therefore no definitive conclusions can be drawn from them. Ultimately, I strongly recommend accepting this paper for Findings.
Decision: Accept-Findings