Unsupervised Paraphrasing of Multiword Expressions
Year: 2023

Abstract:
We propose an unsupervised approach to paraphrasing multiword expressions (MWEs) in context. Our model employs only monolingual corpus data and pre-trained language models (without fine-tuning), and does not make use of any external resources such as dictionaries. We evaluate our method on the SemEval 2022 idiomatic semantic text similarity task, and show that it outperforms all unsupervised systems and rivals supervised systems.1

heading: 1 Introduction
text: Multiword expressions (MWEs) are notoriously difficult to model because the meaning of the whole can diverge substantially from that of the component words, e.g. the meaning of swan song (“final performance”) is far removed from its component words2 (Sag et al., 2002; Baldwin and Kim, 2010). This hampers the capacity of pre-trained language models such as BERT (Devlin et al., 2019) to capture MWE semantics (Tayyar Madabushi et al., 2021; Zeng and Bhat, 2022). Similarly, nonnative speakers tend to have difficulty understanding MWEs, especially those that have no equivalent in their native language (Irujo, 1986; Arnaud and Savignon, 1996).
In this work, we propose a method for paraphrasing non-literal MWEs (e.g. swan song) into more literal expressions (e.g. final performance) to aid both humans and machines to understand their meanings. Importantly, we perform this in a fully unsupervised way: our method uses only monolingual corpus data and off-the-shelf pre-trained masked language models (MLMs), and does not make use of any labelled data or lexical resources
∗This work was partially done when the first author was at Riken.
1Code is available at: https://github.com/ twadada/mwe-paraphrase.
2The MWE is said to originate from an ancient legend that a swan sings beautifully before it dies.
such as WordNet (Fellbaum, 1998), in contrast with previous work (Liu and Hwa, 2016; Zhou et al., 2021, 2022). We base our experiments on SemEval 2022 Task 2 (Tayyar Madabushi et al., 2022), a task designed to evaluate how well models understand the semantics of MWEs in two highresource languages (English and Portuguese) and one low-resource language (Galician). We show that our model generates high-quality paraphrases of MWEs and aids pre-trained models to produce better representations for sentences that contain idiomatic expressions. Compared to system submissions to the shared task, our method performs better than all unsupervised systems and comparably with supervised systems.

heading: 2 Methodology
text: Given a target sentence t that contains an MWE x, our goal is to paraphrase x with more literal expressions in the context of t. To this end, we propose a fully unsupervised method which employs monolingual corpus data and an off-the-shelf masked language model as is, i.e. without any fine-tuning. Our method consists of the following five steps: (1) collect sentences containing x from a monolingual corpus; (2) cluster the sentences; (3) generate paraphrases of x for each cluster; (4) rerank the paraphrase candidates; and (5) select the most relevant cluster to the sense of x in the target sentence t, and paraphrase x. Figure 1 illustrates the overview of our model, and we describe the details of each step below.

heading: 2.1 Sentence Retrieval and Clustering
text: We first collect sentences that contain the target MWE x from a monolingual corpus.3 Next, we sparsify sentences that have very similar local con-
3Here, we do not perform lemmatisation of x and regard, say, ghost town and ghost towns as different instances because we aim to generate paraphrases that fit well both syntactically and semantically in the target sentence t.
4732
texts around x to ensure diversity in the data,4 and keep up to 300 sentences {si} for each MWE type. Then, we generate contextualised embeddings of x for each sentence and cluster them to try to separate out different senses of x. For instance, for x = closed book, there should be at least two clusters to represent its literal sense (“unopened book”) and idiomatic sense (“mystery”; e.g. This subject is a closed book to me). To generate the embeddings, we replace x with a single [MASK] token and obtain the representation of [MASK] just before the linear layer prior to the word prediction output.
As our clustering algorithm, we choose DBSCAN (Ester et al., 1996) since it adaptively determines the number of clusters (expected to roughly correspond to the number of senses of x), and dynamically removes outliers during clustering. We measure the distance using cosine similarity, and tune a couple of hyperparameters of DBSCAN on the dev set of the SemEval STS task.5 In Section 4.1, we show that DBSCAN is more effective than two other popular clustering methods: K-means (Lloyd, 1982; Arthur and Vassilvitskii, 2007) and X-means (Pelleg and Moore, 2000). As monolingual corpora, we use OSCAR (Ortiz Suárez et al., 2020) for English and Portuguese, and CC-100 (Wenzek et al., 2020; Conneau et al., 2020) for Galician.

heading: 2.2 Paraphrase Generation
text: Given the Nj sentences in the j-th cluster Sj : s1,j , s2,j .., sNj ,j , we generate paraphrase candidates of x. The key idea is to find words or phrases that are a good fit for the contexts of x in all sentences in Sj . To this end, we use BERT to generate 1- and 2-token paraphrases, independently. To pre-
4We look at 3 words surrounding x for each sentence, and discard sentences that share many words with other sentences.
5The tuned values are shown in Appendix A.
dict 1-token candidates y, we simply replace x in si,j with a single [MASK] token (denoted as M), and obtain the probability as:
P1(y|Sj) = g( 1
Nj
Nj∑
i=1
f(sx→Mi,j )M), (1)
where sx→Mi,j denotes the sentence si,j with x replaced by [MASK]; f(sx→Mi,j )M ∈ Rd denotes the [MASK] representation before the last linear layer; and g denotes the last linear layer followed by the softmax function. We calculate the probability distribution after averaging the [MASK] embeddings over the Nj sentences, and retrieve the top-10 words as the 1-token paraphrase candidates.
Similarly, when we generate 2-token paraphrases y: y1y2, we replace x with two [MASK] tokens (denoted as M1M2) and obtain the probability distribution for yk as:
P̃(yk|Sj) = g( 1
Nj
Nj∑
i=1
f(sx→M1M2i,j )Mk). (2)
Since extracting the most probable words independently from P̃(y1|Sj) and P̃(y2|Sj) does not always result in a valid phrase, we first extract the top-5 words in P̃(y1|Sj) and use each of them as the basis of M1, and then compute the probability of filling M2 given y1, as:
P̃(y2|y1, Sj) = g( 1
Nj
Nj∑
i=1
f(sx→y1M2i,j )M2). (3)
We extract the top-5 words for each of the top-5 y1 candidates, resulting in 25 unique phrases, some of which are single words consisting of two subword tokens. The (normalised) joint probability of y: y1y2 is estimated as:
P2(y|Sj) = √ P̃(y2|y1, Sj)P̃(y1|Sj). (4)
Lastly, we swap the mask-filling order of y1 and y2 and generate another 25 phrases, some of which overlap with the previous 25 phrases. We retain the top-10 paraphrases with the largest joint probabilities and combine them with the top-10 single-token paraphrases.6 While this algorithm could be extended to generating longer phrases, we focus on 1- and 2-token paraphrases based on the observation that many MWEs can be regarded as single semantic units (Chafe, 1968) and are thus paraphrasable with single words (e.g. kick the bucket with die) (Baldwin and Kim, 2010). In Section 3.2, we also experiment with using T5 (Raffel et al., 2020) instead of BERT to generate paraprhases without the token-length constraint.

heading: 2.3 Outer Probability Reranking
text: After generating paraphrase candidates with differing token lengths, the question is how to jointly rank them. One simple solution is to compare the mask-filling probabilities P1(y|Sj) and P2(y|Sj) directly. However, these values are not directly comparable because the former tends to get higher values due to the narrower search space. Also, these generation probabilities are affected by the word frequency of y, with rare words receiving smaller values.7 As such, we propose a new reranking method based on the “outer probability”: the probability of generating the context of x given the candidate. More specifically, for each sentence si,j in the cluster j, we first replace some context words in si,j with [MASK] tokens to create the masked sentence ŝi,j . Then, we replace x with one of the paraphrase candidates y and predict the masked tokens using an MLM. Our hypothesis is that if y represents the semantics of x very well, the model will predict the surrounding words with higher probabilities.8 Lastly, we calculate the reranking score S(y|Sj) by taking the average of the log probabilities over all the masked tokens in all the sentences
6We discard candidates that are highly similar to the target MWE (e.g. swan songs for swan song) based on a threshold on the normalised character-level edit distance between the candidate and x of 0.2 or less.
7For instance, Lau et al. (2020) measure the acceptability of a sentence based on the LM perplexity and show that it is crucial to normalise the perplexity by unigram probability of the words in the sentence because rare words tend to receive low probabilities even if they are used in a natural context.
8This resembles the training objective of skip-gram (Mikolov et al., 2013).
si,j in cluster j:
S(y|Sj) = Nj∑
i=1
log P(sx→yi,j |ŝ x→y i,j ),
where P(sx→yi,j |ŝ x→y i,j ) is the product of the probabilities of reconstructing the masked tokens. One advantage of this method is that the tokens we predict are always the same regardless of y, reducing the influence of the number of tokens in y.
Regarding which tokens to mask in si,j , one naive approach is to randomly mask words as performed during the MLM pre-training. Ideally, however, we want to select words that are semantically relevant to x; for instance, given the context: This show will serve as his swan song, as he plans to retire, the words show and retire are arguably more relevant to the meaning of swan song than other words such as will and plans. With this in mind, we mask words based on the self-attention weights: first, we replace x with two [MASK] tokens and calculate their self-attention weights from the other words in the last layer.9 We then mask the top-5 words with the highest weights, excluding punctuation and subword tokens.

heading: 2.4 Paraphrase Retrieval
text: Given the ranked paraphrase candidates for each cluster, we finally retrieve the paraphrases of x by retrieving the cluster that best represents x in the target sentence t. To this end, we first replace x in t with one [MASK] token and retrieve the closest cluster based on the cosine similarity between the [MASK] embedding and the centroids of the clusters. Note that all the previous steps can be done without t, meaning if we have a list of potential MWE types and pre-compute their paraphrase candidates, we can paraphrase them given an arbitrary context in an online manner.

heading: 3 Experiments

heading: 3.1 Idiomatic Semantic Text Similarity Task

heading: 3.1.1 Data
text: We first evaluate our model on SemEval 2022 Task 2 Subtask B (Tayyar Madabushi et al., 2022). This is a variant of the semantic textual similarity (STS) task, where given two input sentences, a model produces a score between 0.0 and 1.0 based on
9We average the weights across all attention heads and [MASK] tokens. We also tried using only one [MASK] token instead of two and obtained similar results.
the similarity of the sentences, and the evalation is based on Spearman’s rank correlation to the humanannotated scores. One of the input sentences E contains an MWE (e.g. E: Witten’s swan song was far from a hit) and the other sentence is a replica of E except that the MWE is replaced with either a correct paraphrase (E→c, e.g. Witten’s final performance was far from a hit) or an incorrect one (E→i, e.g. Witten’s bird song was far from a hit). The target MWEs are two-word nominal compounds that can contain adjectives (e.g. old flame). For E–E→c pairs, the STS score is 1.0 since they have almost identical meaning, while for E–E→i pairs, sentences are scored in the [0, 1) range. Data is provided in English, Portuguese, and Galician, and for English and Portuguese, general STS benchmark data sets — STS Benchmark (Cer et al., 2017) for English and ASSIN2 STS (Real et al., 2020) for Portuguese — are also included in the evaluation set to assess the model’s generalisability on both MWE and non-MWE data.

heading: 3.1.2 Models
text: There are two settings for the shared task: FineTune and Pre-Train. In the Fine-Tune setting, systems can be supervised on the train split of the MWE STS data,10 but in the Pre-Train setting, systems are not allowed to use this data but can be pretrained on other resources. Since our model works without any labelled data (or even fine-tuning), we compare our model against Pre-Train systems.
In the Pre-Train setting, the best-performing system (Phelps, 2022) expands the BERT vocabulary and obtains additional embeddings for each MWE type using monolingual data containing the MWEs. As such, it is built on similar data to our model. To obtain the MWE embeddings, they employ BERTRAM (Schick and Schütze, 2020), which was originally proposed for learning additional BERT input embeddings of rare words. The model with the MWE embeddings is then fine-tuned on the train split of general STS data. The major limitation of this approach is that the MWE embeddings need to be pre-trained for each model (e.g. BERT-base, BERT-large, T5) before they are finetuned. In contrast, our model directly paraphrases the input text E containing an MWE, and we feed it to an arbitrary pre-trained STS model to generate the sentence embedding of E, which is then used to measure the similarity of the sentence pairs.11
10For Galician, there is no train or development data. 11The other input text (E→c or E→i) is fed into the STS
That is, our paraphrasing model is completely separated from the task-specific models, providing more flexibility in terms of model selection and training. In the STS experiments, however, we use the same BERT models used in Phelps (2022) for both paraphrasing and STS models (and also for clustering) for fair comparison. Specifically, we use BERT-base-uncased (Devlin et al., 2019) for English, BERTimbau-Base (Souza et al., 2020) for Portuguese, and Bertinho-Base (Vilares et al., 2021) for Galician. To obtain STS models, we fine-tune the respective BERT models on the train split of STS Benchmark for English and ASSIN2 STS for Portuguese and Galician, following Phelps (2022).12

heading: 3.1.3 Results
text: Table 1 shows the results on the STS task, where the columns “MWE” and “General” denote the Spearman’s rank correlation on the MWE and general STS data, and “All” denotes the overall performance. Sem-Base denotes the mBERT-based baseline provided by the shared task organisers. OURS and OURS-ave3 denote our models, where the latter indicates the performance when we produce the sentence embedding of E by replacing the target MWE with the top-3 paraphrases and averaging the embedddings of the three sentences. In the shared task, the systems are ranked based on the “All” performance, and the table shows that our model outperforms Phelps (2022) and achieves the best score in the Pre-Train setting. It also shows
model without any paraphrasing. 12The Portuguese data is used for Galician since there is no STS Galician data set and these languages are very similar.
that using the top-3 paraphrases further improves performance.
The “Supervised” sub-table shows the performance of the submitted systems in the Fine-Tune setting, where the models are trained on both MWE and general STS data. Without any labelled data, our model outperforms the supervised task baseline (Sem-Base) and also the supervised model of Phelps (2022), which fine-tunes their unsupervised model on the MWE STS data and ranks 2nd in the Fine-Tune setting. OURS-ave3 performs slightly worse than the best supervised system (Liu et al., 2022), which however performs very poorly on the general STS data, suggesting it is overfitting the MWE data. Table 2 compares the performance of Phelps (2022) (w/ or w/o fine-tuning) and our model for each language. Overall, our model achieves better performance than the unsupervised baseline in all languages, and the supervised model in Portuguese and Galician. We expect further improvements by fine-tuning BERT on labelled or unlabelled data, which we leave for future work.

heading: 3.2 MWE Paraphrase
text: Next, we evaluate our model based on the matching accuracy of the MWE paraphrases. To this end, we first extract the pairs of MWEs and their correct paraphrases in context from the English “train” split of the SemEval STS data set provided for the Fine-Tune setting. Since we do not use any portion of this data to train or tune our model, we can regard it as a pseudo test set for this task;13 the size of
13We could not use the test set because the gold paraphrases are not publicly available.
each data split is shown in Table 10 in Appendix. In this task, in addition to BERT-base, we also combine our method with BERT-large (whole word masking), SpanBERT-large (Joshi et al., 2020), ALBERT-large (Lan et al., 2020) and T5 (Raffel et al., 2020).14 Unlike BERT and its variants, T5 generates an arbitrary number of tokens conditioned on the encoder hidden states of all input tokens, making it difficult to aggregate the hidden states or probability distributions across multiple sentences as done in Eqn. (1). Therefore, we propose another simple method tailored for T5 (which is somewhat different from the method described in Section 2.2): we use it to generate 20 paraphrase candidates for each sentence si,j in the cluster j15 independently using beam search, which produces 20Nj paraphrases in total.16 Then, we retain the paraphrases that contain 1 or 2 words (which can consist of more subword tokens)17 and rank them based on their generated counts; for the 2-word candidates, we double the count because one-word candidates often get higher values, leading to worse results. We also try reranking the candidates using the outer probability (Section 2.3), but randomly masking 5 consecutive words for T5 (and SpanBERT) rather than the high-attention words because these models are trained to fill random spans of text rather than separate tokens. As a strong baseline for this task, we use GPT-3 (davinci) (Brown et al., 2020). As a prompt, we feed several triples randomly retrieved from the dev set, which consist of a sentence that contains an MWE, a question asking what is the most appropriate substitute for it, and the correct paraphrase. We feed as many examples as possible until they reach the max token limit (2048), which correspond to about 35 triplets. Lee et al. (2021) show that this approach outperforms BERT in lexical substitution, and we use their prompt with minor modification.18
Table 3 (under “Matching Accuracy”) shows the results based on P@k: the proportion of instances where the gold paraphrase is included in the topk predictions. For GPT-3, we report only P@1 because it is prompted to produce the single best paraphrase only, as we have only one gold para-
14See the Appendix for model details (Table 9). 15We use the T5 encoder to create the clusters. 16To avoid generating lengthy phrases, we set the maximum number of tokens to generate to 10. 17We also tried including 3-word expressions but got worse results as most gold paraphrases are 1 or 2 words. 18See Appendix B for an example of the prompt.
phrase per input sentence. We can see that GPT-3 performs the best in P@1, followed by our models using T5 and BERT.19 However, note that they are not strictly comparable in terms of the number of the model parameters (shown as “# param”) as well as the amount of training data (with BERT trained on the least data). Also, GPT-3 is supervised on labelled data while our models are fully unsupervised.20 In P@5/10, T5-large performs the best, but using much more parameters than the other models. Our reranking method improves the performance of T5 overall, demonstrating its effectiveness. BERT performs reasonably well at P@10 with fewer parameters, possibly because BERT is conditioned on all sentences si,j in the cluster j simultaneously by averaging the mask embeddings.
We also evaluate the models on the STS task (using the English test split), and the result is shown under “STS” in Table 3. Notably, GPT-3 performs the worst in this metric, and this is likely because it sometimes copies the target MWEs or paraphrases them with another MWE (e.g. dead end with blind alley), which does not simplify the text very much. The comparison of the other models also reveals that larger models are not necessarily better at text simplification.
19One reason why P@1 is very low is that the SemEval data contains only one gold paraphrase for each MWE (e.g. final performance for swan song), missing many other possible candidates (e.g. last performance, final appearance, and farewell appearance).
20Also, GPT-3 implicitly memorises the definitions of some MWEs during pre-training, e.g. given a simple prompt like What is the meaning of “swan song”?, it returns its origin as well as its meaning.

heading: 4 Analysis

heading: 4.1 Effects of Clustering
text: We compare the performance of our model (BERTbase) using different clustering methods: K-means, X-means and DBSCAN. Table 4 shows the results on the MWE paraphrasing task. The first row “None” denotes the performance when we treat all sentences as a single cluster. Amongst all methods, DBSCAN performs the best at P@1/5, outperforming K-means and X-means. We conjecture that DBSCAN benefits from not only setting the cluster size per MWE type dynamically, but also from creating one outlier cluster and discarding sentences that do not provide sufficient context to infer the MWE semantics (which is not achieved by K/X-means). Without clustering, our method performs poorly at P@1/5, and yet is equivalent to DBSCAN at P@10. This is because our model with clustering completely fails when it retrieves a wrong cluster,21 whereas our model without clustering always generates a mixed bag of paraphrases with varying meanings, some of which are often relevant to the (common) MWE senses.
Table 5 shows some examples of sentences22
sampled from each DBSCAN cluster, as well as the best paraphrase generated for each cluster. The first sentences for each MWE are “outlier sentences” sampled from the outlier cluster, and are thus not used for paraphrase generation. We can see that the outlier sentences of inner circle and small fry have vague contexts that do not represent the MWE semantics very well. In contrast, the one of silver bullet is regarded as an outlier despite its specific context because the target MWE is conjoined with another MWE (magic pill) and the model wrongly
21This is particularly pronounced in the low performance of K-means (K = 4) in P@10.
22The input text fed to the clustering model usually contains the previous and/or following sentences as well.
predicts pill and drag as substitutes for silver bullet. Comparing the three clusters of closed book and their paraphrases, we can see that the paraphrases generated by the first and second clusters successfully distinguish its literal and idiomatic senses (“book” and “complete mystery”), but the paraphrase of the last cluster (fully written) does not retain the original meaning; in fact, it is very challenging to find a good (short) substitute for closed book in this context, as an inherent limitation of a paraphrasing approach. The table also shows that some clusters are formed based on syntactic constraints rather than the semantics of the MWEs; e.g. the clusters of silver bullet are formed based on whether the MWE is used as a noun or adjective. The last cluster of inner circle, which has similar semantics to the second cluster, is also created as a result of the local morphophonetic effect, where most of the sentences in this cluster include the MWE as an inner circle of and all the paraphrase candidates start with a vowel sound (e.g. entire army, elite group, alliance) due to the article– noun agreement effect. A similar result is observed by Wada et al. (2022) in lexical substitution, where they find that MLM predictions and representations are highly affected by such morphophonetic or morphosyntactic biases. Interestingly, the last cluster of small fry is composed of the sentences where fry is used as a component word of the other MWE fry pan, not of small fry. Other similar cases include high life (“expensive lifestyle”) used as high life expectancy and bad hat (“troublemaker”) used
as bad hat hair. We further discuss this problem in Section 7.

heading: 4.2 Effects of Reranking
text: In Table 3, we showed that both T5-base and T5large benefit from reranking. To further verify its effectiveness, we compare the performance of BERTbase/large w/ or w/o reranking in Table 6. The row “None” denotes the performance when we rank the candidates based on the mask-filling probabilities P1(y|Sj) and P2(y|Sj) in Eqn. (1) and Eqn. (4), and “Rand” and “Attn” show the performance with our reranking method, wherein we mask five words randomly or based on self-attention weights; for Rand, we average the performance over three runs. Our reranking method improves the P@5/10 performance of BERT-base in both English and Portuguese, but for BERT-large, there is no clear winner. When we compare Rand and Attn, there is no
noticeable difference in matching accuracy; therefore, we also examine the impact of our reranking methods on the STS task in Table 7 (which can measure the semantic fit of the paraphrases in a continuous manner). It shows that Attn performs the best overall on the dev and test sets. We also observe that the performance of Rand varies greatly across three runs (65.34, 65.91, and 64.96 in ALL on the test set), suggesting that the selection of the context words to reconstruct in our reranking method has a non-trivial impact on performance.
Table 8 shows examples of MWEs and their top-3 paraphrases generated by BERT-base w/ or w/o reranking (Attn). It shows that reranking produces semantically more relevant paraphrases. Furthermore, these results reveal a few plausible limitations of the scoring method based on the mask-filling probabilities; the limitations that are also relevant to T5 to some degree. Firstly, it tends to favour single-token paraphrases (e.g. bridge and road for zebra crossing) due to the narrower search space. Secondly, it often assigns high scores to phrases that have strong collocation (e.g. new world, regular customer) regardless of their semantic fit because of the higher probability P̃(y2|y1, Sj) in Eqn. (3) (or P̃(y1|y2, Sj)). Thirdly, it is inevitably influenced by word frequency, as evidenced by the fact that research organization gets much higher values than its British spelling research organisation (0.33 vs. 0.09) as the paraphrase of think tank; on the other hand, our reranking method assigns them very similar scores. On the other hand, one limita-
tion of our reranking method is that it occasionally gives high scores to syntactically ill-formed candidates, especially those that contain duplicated tokens (e.g. clock clock as the paraphrase for grandfather clock), and we alleviate this by removing one of the duplicated tokens.

heading: 5 Related Work
text: There is a line of work on MWE paraphrasing (or substitution), but unlike our method, previous methods resort to either human-annotated corpora or high-coverage dictionaries to generate paraphrases; consequently, they are only evaluated in English or other high-resource languages (e.g. Chinese). For instance, Liu and Hwa (2016) extract paraphrases of MWEs from their dictionary definitions,
which usually contain supplementary information as well as their core meanings and paraphrases. Similarly, Zhou et al. (2022) encode definitions of MWEs using a sentence embedding model and employ the embeddings to generate their paraphrases. They also propose another method that fine-tunes BART (Lewis et al., 2020) on a parallel corpus built by Zhou et al. (2021), in which the source sentences contain MWEs and the target sentences paraphrase them while leaving the other words unchanged. Similarly, Qiang et al. (2022) create analogous data in Chinese and fine-tune mT5 (Xue et al., 2021) on it to paraphrase Chinese MWEs. In contrast, Ponkiya et al. (2020) propose an unsupervised method using BERT/T5, but they focus on paraphrasing noun compounds (e.g. club house) using the same MWE tokens (e.g. house owned by a club) without context.
A more common and resource-efficient approach to handling MWEs is to regard them as individual lexical units (e.g. regard “kick_the_bucket” as one token) and train their embeddings using monolingual data (Salehi et al., 2015; Cordeiro et al., 2019; Phelps, 2022). Those embeddings can be used to retrieve similar words in the vocabulary based on embedding similarity (Otani et al., 2020). However, one drawback is that it increases the size of vocabulary (and parameters) significantly, due to the sheer volume of MWE instances (e.g. around 41% of entries in WordNet 1.7 (Fellbaum, 1998) are MWEs (Sag et al., 2002)). Recently, Zeng and Bhat (2022) addressed this limitation by training an additional adapter network (Pfeiffer et al., 2020) on top of an MLM to produce better embeddings for various MWEs, but they rely on dictionary definitions to train the network, arguing that such external knowledge is fundamental for learning MWE representations.

heading: 6 Conclusion
text: We propose a fully unsupervised method to paraphrase multiword expressions (MWEs) in context. Our method employs only a monolingual corpus and pre-trained language model, and does not rely on any labelled data. In our experiments, we show that our model generates good MWE paraphrases and aids pre-trained sentence embedding models to represent sentences containing MWEs.

heading: 7 Limitations
text: One limitation of our proposed method is that it requires the pre-identification of the target MWE in a sentence before paraphrasing it, a task that is not a walk in the park. In particular, it is very challenging to identify what is the “correct” span of a given MWE, which our model critically relies on. For instance, given the MWE lip service (“insincere agreement”), our model predicts more attention as the best paraphrase, likely because the MWE is usually used as pay lip service to (something), and attention is one of the few words that fits well in this context (in terms of collocation). Therefore, the whole phrase pay lip service to should be identified as an MWE instance23 when it is used in sentences like They pay lip service to the idea; however, lip service can also serve as one lexical unit in sentences like It wasn’t just lip service. A similar problem arises when we deal with nominal MWEs that follow indefinite articles (a or an) as discussed in Section 4.1, or verbal MWEs that are often followed by specific prepositions (e.g. turn a blind eye to ... means “deliberately ignore ...”) because the MLM prediction is affected by the syntactic constraint.24 MWE span identification is also important in our sentence collection process; e.g. as discussed in Section 4.1, the phrase small fry can be used as small fry pan rather than as the MWE meaning “insignificant”, and hence collecting sentences based on string match resulted in one additional cluster that is not relevant to either its literal or idiomatic senses.
Another limitation is that our model cannot handle discontinuous MWEs such as throw someone under the bus and not ... in the least because it is not clear which parts to mask and paraphrase in such cases. Similar problems arise when continuous MWEs undergo either internal modification (e.g. go completely cold turkey) or drastic syntactic transformation (e.g. the beans are split). However, note that all of these types of expressions, as well as the pre-tokenisation problem discussed above, become a pain in the neck for any approach that regards an MWE as a lexical unit and learns its holistic embedding.
Lastly, our method heavily relies on the quality
23In fact, it is registered as such in some English dictionaries.
24In languages where words have grammatical gender such as Portuguese and Italian, this problem can be more pronounced because context words including adjectives and determiners are affected by gender.
of the clusters and is thus prone to error propagation. For instance, our model using BERT always generates large fish as the best paraphrase for the MWE big fish and fails to capture its idiomatic sense (“an important person”), likely due to its rare occurrence in monolingual corpora (compared to its literal sense). One possible solution to this problem is to derive more senses by allowing the clustering method to create more clusters with fewer instances, but that institutes a trade-off between accommodating rare senses and creating too many clusters for common senses; hence, there is no silver bullet. In fact, this problem pertains to the longstanding question (with no single correct answer) among lexicographers: how to “split” and “lump” senses of words, and how fine-grained the sense distinctions should be (Hanks, 2000, 2012).

heading: A Hyper-Parameters
text: Table 9 shows the hyper-parameters of DBSCAN: minPts: the minimum number of points required to form a core point; and ϵ: the maximum distance between two points to be considered as neighbours. We tune ϵ for each model, and the table shows their shortcut names in the Transformers library (Wolf et al., 2020). For SpanBERT, we used the model in the original GitHub repository (https://github.com/ facebookresearch/SpanBERT).

heading: B GPT-3 Prompt
text: Here is an example of the GPT-3 prompt used in Section 3.2:
Witten’s **swan song** was far from a hit.
Q: What is the most appropriate substitute for **swan song** in the above text?
A: final performance
It is a triple of the target sentence with the target MWE marked with **, a question that asks the most approproate substitute for the MWE, and the gold paraphrase retrieved from the dev set. We borrow this template from Lee et al. (2021), but
change What are appropriate substitutes to What is the most appropriate substitute since we have only one gold paraphrase for each input text. We feed as many triples as possible until they reach the max token limit (2048), which correspond to about 35 triplets. Then, we append one “test triplet” that contains one sentence from the test set, the corresponding question, and the answer without the gold paraphrase (i.e. A:), and make the model predict the paraphrase. We make sure that the MWE in the test triplet is not included in the triplets retrieved from the dev set.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? 3
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use the data from the SemEval 2022 shared task, and we follow its intended use.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 3, 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 2, 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We expect further improvements by fine-tuning BERT on labelled or unlabelled data, which we leave for future work