Venue: EMNLP 2020

# Compositional Phrase Alignment and Beyond

Yuki Arase\({}^{1\star}\) &Junichi Tsujii\({}^{*2}\)

\({}^{1}\)Osaka University, Japan

\({}^{\star}\)Artificial Intelligence Research Center (AIRC), AIST, Japan

\({}^{2}\)NaCTeM, School of Computer Science, University of Manchester, UK

arase@ist.osaka-u.ac.jp, j-tsujii@aist.go.jp

###### Abstract

Phrase alignment is the basis for modelling sentence pair interactions, such as paraphrase and textual entailment recognition. Most phrase alignments are compositional processes such that an alignment of a phrase pair is constructed based on the alignments of their child phrases. Nonetheless, studies have revealed that non-compositional alignments involving long-distance phrase reordering are prevalent in practice. We address the phrase alignment problem by combining an unordered tree mapping algorithm and phrase representation modelling that explicitly embeds the similarity distribution in the sentences onto powerful contextualized representations. Experimental results demonstrate that our method effectively handles compositional and non-compositional global phrase alignments. Our method significantly outperforms that used in a previous study and achieves a performance competitive with that of experienced human annotators.

## 1 Introduction

Phrase alignment is a fundamental problem in modelling the interactions between a pair of sentences, such as paraphrase identification, textual entailment recognition, and question answering Das and Smith (2009); Heilman and Smith (2010); Wang and Manning (2010). Phrase alignment generally adheres to compositionality, in which a phrase pair is aligned based on the alignments of their child phrases. Nonetheless, non-compositional alignments involving long-distance phrase reordering are prevalent in practice Burkett et al. (2010); Heilman and Smith (2010); Arase and Tsujii (2017). Figure 1 shows an example of phrase alignment in which phrases of the same colours are alignable, _i.e_. they are phrasal paraphrases. The alignment of 'antivirus vaccines' and 'vaccines against the virus' is compositional, as supported by alignments of their child nodes although their orderings are reversed. Similarly, the alignment of their parents \(\tau_{2}^{s}\) and \(\tau_{2}^{t}\) is compositional. By contrast, the alignment of \(\tau_{1}^{s}\) and \(\tau_{1}^{t}\) is non-compositional in relation to the alignment of \(\tau_{2}^{s}\) and \(\tau_{2}^{t}\); although \(\tau_{1}^{t}\) and \(\tau_{2}^{t}\) are siblings, \(\tau_{1}^{s}\) is not a sibling of \(\tau_{2}^{s}\), _i.e_. not in the scope of the parent node of \(\tau_{2}^{s}\). To treat such a long-distance correspondence in non-compositional alignment, one has to consider candidate phrases outside the local scope and potentially the entire sentence.

In this study, we address the phrase alignment problem by combining a tree mapping algorithm with phrase representation modelling. We treat compositional alignment by an algorithm for an _unordered_ tree mapping Zhang (1996). For the algorithm to work, definition of the edit cost (_i.e_. dissimilarity between phrases) is crucial. We propose a novel phrase representation, by which the edit cost is defined, based on contextualized representations by the bidirectional encoder representations from transformers (BERT) Devlin et al. (2019). The proposed phrase representation models the similarity distribution in the entire sentence, thereby allowing the algorithm to be extended to treat non-compositional global alignments.

Phrase alignment can be difficult even for humans because there is unavoidable subjectivity in

Figure 1: Phrase alignments by the proposed method (phrases of the same colour are paraphrases)

acceptable semantic discrepancies between paraphrases. Our experimental results indicate that the proposed method achieves \(95.7\%\) of the alignment quality of trained human annotators for phrase alignment in paraphrase sentence pairs.

The contributions of this study are twofold. First, we formalise the compositional phrase alignment problem as an unordered tree mapping. Second, we propose a phrase representation model that allows non-compositional global alignments.

## 2 Related Work

### Tree Mapping and Phrase Alignment

_Ordered_ tree mapping has been employed to estimate the similarity of a pair of sentences for its ability to align syntactic trees Punyakanok et al. (2004); Alabbas and Ramsay (2013); Yao et al. (2013); McCaffery and Nederhof (2016). However, it is too restrictive in that the order of the aligned phrases in the sentences must be the same. Previous studies extended the algorithm to adapt the edit costs Bernard et al. (2008); Mehdad (2009); Alabbas and Ramsay (2013) and edit operations Heilman and Smith (2010); Wang and Manning (2010) to specific tasks. In contrast, the unordered tree mapping that we employ in this study is sufficiently flexible to assure identification of optimal compositional phrase alignments.

Parallel parsing also involves phrase alignment in its parsing process. As the tree isomorphism assumption is too restrictive, previous studies have employed various relaxation techniques that prefer but do not force synchronisation. Burkett et al. (2010) used weakly synchronised grammar, and Das and Smith (2009) used quasi-synchronous grammars Smith and Eisner (2006). Choe and McClosky (2015) used dual decomposition to encourage agreement between two parse trees. All of these methods allow excess flexibility beyond compositionality in alignment. Rule extraction for tree transducers also involves phrase alignments Martinez-Gomez and Miyao (2016) but disregards phrase boundaries to maximise the coverage of extracted rules. In contrast, the phrase alignment problem addressed in our study adheres to syntactic structures.

### Phrase Representation Generation

Researchers have proposed specialised phrase representations for specific tasks Arase and Tsujii (2019); Yin et al. (2020) on top of contextualised representations. In this study, we propose dedicated phrase representations for the alignment problem. Before contextualised representation, studies considered word alignment distributions for modelling semantic interactions between a pair of sentences He and Lin (2016); Parikh et al. (2016); Chen et al. (2017). We agree with their intuition that the pairwise similarities alone are not good enough to define the cost of alignment. In case there are other similar phrases, their pairwise similarities have to be properly adjusted. This adjustment is crucial for treating non-compositional global alignment.

## 3 Phrase Alignment Method

### Preliminaries and Notation

We refer to one of the paraphrasal sentences as the _source_, \(s\), and the other as the _target_, \(t\). Superscripts \(s\) and \(t\) represent source and target, respectively. The syntactic trees of the source and target, \(T^{s}=\{\tau_{i}^{s}\}_{i}\) and \(T^{t}=\{\tau_{j}^{t}\}_{j}\), determine the phrase structures; \(\tau_{i}^{s}\) and \(\tau_{j}^{t}\) are the source and target phrases. The alignments of their phrases are \(\mathds{H}=\{\mathds{h}_{i}=\langle\tau_{i}^{s},\tau_{i}^{t}\rangle\}_{i}\). We interchangeably use the subscript of a node as the index of the alignment or the index of the node in a tree whenever the meaning is apparent from the context. A phrase can align to an empty node \(\tau_{\emptyset}\) (\(\tau_{\emptyset}\notin T\)), which is called the _null_ alignment.

We define functions to traverse a tree: ds\((\tau)\) derives descendant nodes of \(\tau\), and lca\((\tau_{i},\tau_{j})\) derives the lowest common ancestor of \(\tau_{i}\) and \(\tau_{j}\). Additionally, function deg\((T)\) computes the maximum depth of \(T\), and \(|\cdot|\) counts the number of elements in a set; _e.g._\(|T|\) is the number of nodes in \(T\).

### Problem Definition

Based on Arase and Tsujii (2017), we reformalise conditions of legitimacy as a set of compositional phrase alignments \(\mathds{H}_{L}\).

**Definition 3.1**.: _Legitimacy conditions consist of the following:_

**Consistency**: _In_ \(\mathds{H}_{L}\)_, a phrase (_\(\neq\tau_{\emptyset}\)_) in the source tree is aligned with at most one phrase (_\(\neq\tau_{\emptyset}\)_) in the target tree, and vice versa._
**Monotonicity**: _For_ \(\langle\tau_{i}^{s},\tau_{i}^{t}\rangle,\langle\tau_{j}^{s},\tau_{j}^{t} \rangle\in\mathds{H}_{L}\)_,_ \(\tau_{i}^{s}\in\text{ds}(\tau_{j}^{s})\) _iff_ \(\tau_{i}^{t}\in\text{ds}(\tau_{j}^{t})\)_._
**Familiaress**: _For_ \(\langle\tau_{1}^{s},\tau_{1}^{t}\rangle\)_,_ \(\langle\tau_{2}^{s},\tau_{2}^{t}\rangle\)_,_ \(\langle\tau_{3}^{s},\tau_{3}^{t}\rangle\) _in_ 

lca_(\(\tau^{s}_{1},\tau^{s}_{2}\)) _is a proper ancestor_1 of \(\tau^{s}_{3}\) iff lca_(\(\tau^{t}_{1},\tau^{t}_{2}\)) _is a proper ancestor of \(\tau^{t}_{3}\)._

Footnote 1: A proper ancestor of a node \(i\) is any node \(j\) such that node \(j\) is an ancestor of node \(i\) and \(j\) is not the same node as \(i\).

The consistency condition ensures one-to-one alignment. The monotonicity condition regulates the retainment of the ancestor-descendant relation in the source and target sides. The familiness condition realises compositionality in the language, which constrains such that two separate subtrees of \(T^{s}\) should be aligned to two separate subtrees of \(T^{t}\).2 In other words, the familiaress condition prohibits a node in the source subtree to align to a node outside that target subtree. In Figure 1, \(\langle\tau^{s}_{1},\tau^{t}_{1}\rangle\) violates the famliness condition in relation to \(\langle\tau^{s}_{2},\tau^{t}_{2}\rangle\) and \(\langle\tau^{s}_{3},\tau^{t}_{3}\rangle\) because \(\tau^{s}_{3}\) is not a proper ancestor of \(\mathrm{lca}(\tau^{s}_{1},\tau^{s}_{2})\) , whereas \(\tau^{t}_{3}\) is a proper ancestor of \(\mathrm{lca}(\tau^{t}_{1},\tau^{t}_{2})\).

Footnote 2: Our definition is less constrained than that in Arase and Tsujiri (2017) as discussed in Appendix A.

We define non-compositional alignments \(\mathbb{H}_{\text{nc}}\) as alignments that satisfy the legitimacy conditions internally but do not satisfy them against \(\mathbb{H}_{L}\). For example, the alignment \(\langle\tau^{s}_{1},\tau^{t}_{1}\rangle\) in Figure 1 is compositionally composed and satisfies the legitimacy conditions for its internal alignments. However, it does not satisfy the legitimacy conditions against alignments of \(\langle\tau^{s}_{2},\tau^{t}_{2}\rangle\) and \(\langle\tau^{s}_{3},\tau^{t}_{3}\rangle\) for violation of the familiarnesss condition. We allow \(\mathbb{H}_{\text{nc}}\) to be added into \(\mathbb{H}_{L}\) if it is compatible;

**Definition 3.2**.: \(\mathbb{H}_{\text{nc}}\) _is compatible with \(\mathbb{H}_{L}\) iff for all \(\langle\tau^{s}_{i},\tau^{t}_{i}\rangle\in\mathbb{H}_{\text{nc}}\) (\(\tau^{s}_{i},\tau^{t}_{i}\neq\tau_{\emptyset}\)), both \(\langle\tau^{s}_{i},\tau_{\emptyset}\rangle\) and \(\langle\tau_{\emptyset},\tau^{t}_{i}\rangle\) are in \(\mathbb{H}_{L}\)._

When the compatibility condition is met, \(\mathbb{H}_{\text{nc}}\) can be safely added to \(\mathbb{H}_{L}\) by complementing null alignments without violating the consistency condition. We implement this process by a simple post-processing step (Section 3.4).

### Compositional Alignment

Finding the optimal set of legitimate compositional alignments (Definition 3.1) is equivalent to finding the minimum cost of constrained tree mapping (Zhang, 1996), which belongs to the problem of _unordered_ tree mapping (Bille, 2005). The edit operations of re-labelling, deletion, and insertion correspond to alignment of two nodes, null alignment of a source node, and null alignment of a target node, respectively. Although the unordered tree mapping problem is in general MAX SNP-hard (Zhang and Jiang, 1994), the constrained tree edit distance (CTED) algorithm (Zhang, 1996) achieves polynomial time complexity using the familiness condition. In essence, the CTED algorithm reduces the unordered tree mapping problem to a maximum matching problem by the familyiness condition. The reduction enables faster dynamic programming of \(\mathcal{O}(|T^{s}||T^{t}|(\text{deg}(T^{s})+\text{deg}(T^{t}))\log\left(\text{ deg}(T^{s})+\text{deg}(T^{t})\right))\). Details of the CTED algorithm are described in detail in Appendix B.

To apply CTED for phrase alignment, the edit cost function \(\gamma(\cdot)\rightarrow\mathbb{R}\) is the key, which should satisfy the properties of a proper distance metric. This function evaluates the _dissimilarity_ of a phrase pair, for which we propose a phrase representation model (Section 4). We use cosine distance as \(\gamma(\cdot)\in[0,2.0]\) because of its prevalence in measuring dissimilarity between representations. However, it is not a proper distance metric because it does not satisfy the triangle inequality property. In future work, we will investigate alternative distance metrics.

We also need to estimate the cost of a null alignment. It is not trivial to generate representation of such an empty phrase; hence, we decided to use a constant cost \(\lambda_{\emptyset}\), _i.e._,

\[\gamma(\langle\tau^{s},\tau_{\emptyset}\rangle)=\gamma(\langle\tau_{\emptyset },\tau^{t}\rangle)=\lambda_{\emptyset}\in[0,2.0].\]

The appropriate value of \(\lambda_{\emptyset}\) is determined using a development set.

### Non-compositional Alignment

We designed top-down post-processing for non-compositional alignment so that the legitimacy conditions (Definition 3.1) will be maximally satisfied in the final alignments. As Algorithm 3.1 shows, we add a set of alignments \(\mathbb{H}_{\text{nc}}\) that compose the non-compositional alignments into \(\mathbb{H}_{L}\) when they are compatible. Our post-processing aligns all the coloured phrase pairs in Figure 1 by allowing \(\langle\tau^{s}_{1},\tau^{t}_{1}\rangle\) and its descendant alignments.

Algorithm 3.1 takes matrices of edit distance and corresponding operations \(D\) and \(A\) as input, which are obtained by CTED. \(D[i+1][j+1]\) and \(A[i+1][j+1]\) store the total cost and operations, respectively, to compose alignment of \(\langle\tau^{s}_{i},\tau^{t}_{j}\rangle\). Note that index \(0\) is reserved for null alignments. The algorithm sorts null alignments in \(\mathbb{H}_{L}\) in descending order of the span covering the source and target phrases (line 2). For each null alignment, the algorithm finds candidates of non-compositional 

alignments achieving the minimum cost (line 5). Then, using the IsCompatible function, it checks whether a non-compositional alignment and its descendant alignments are compatible with the current set of alignments. If so, they are added to \(\mathds{H}_{L}\) by the UpdateAlignments function, replacing null alignments \(\langle\tau^{s}_{i},\tau_{0}\rangle\) and \(\langle\tau_{0},\tau^{t}_{j}\rangle\) in \(\mathds{H}_{L}\) with non-compositional alignment \(\langle\tau^{s}_{i},\tau^{t}_{j}\rangle\).

Our post-processing is a heuristic to maximally satisfy the legitimacy conditions, as finding the best combination of non-compositional alignments is computationally intractable.3 Our method ensures that non-compositional alignments improve the alignment cost by only allowing those with minimum cost.

Footnote 3: Arase and Tsuji (2017) do not assure maximal satisfaction of the legitimacy conditions.

## 4 Phrase Representation for Alignment

We propose a phrase representation model on top of the pre-trained BERT. One of the most common methods for obtaining a phrase representation from BERT is pooling outputs corresponding to tokens in the phrase. However, as we empirically show in Section 6, this method exhibits an unsatisfactory ability for modelling the similarity distribution in a sentence pair. Hence, we propose a novel method for generating phrase representations suitable for the phrase alignment problem.

Problem Statement and ApproachThe estimate of a phrase pair's similarity for alignment is unique, because their similarity should depend on similarities of other phrases in the sentence pair. That is, even if the pairwise similarity of \(\tau^{s}_{i}\) and \(\tau^{t}_{j}\) is high, the similarity score should be lowered if there is a phrase in the source sentence that is more similar to \(\tau^{t}_{j}\). Hence, we generate a phrase representation that reflects the similarity distribution within the sentence pair; this is particularly important for non-compositional alignments to find a globally plausible alignment pair.

We first generate a representation of the similarity distribution within the sentence pair. We then transform the phrase representation obtained from BERT, referring to the representation of the similarity distribution using an attention mechanism.

Similarity Distribution ModellingWe regard outputs of the last layer \(\mathbf{h}\in\mathbb{R}^{b}\) of BERT as token representations, where \(b\) is the hidden size determined by the BERT pre-training settings. Using the token representations, we generate a representation of similarity distribution \(\mathbf{e}^{c}\in\mathbb{R}^{b}\) (Figure 2).

We first compute cosine similarities between token representations of the sentence pair and obtain the similarity matrix. We then encode the similarity matrix using a convolutional neural network (CNN) and obtain \(\mathbf{e}^{c}\), called the _SimMatrix_ representation. Our CNN is shallow, under the assumption that a shallow model is sufficient to capture latent features in SimMatrix. A shallow model also allows training with a smaller corpus while fine-tuning BERT. The CNN consists of a one-channel convolution layer activated by the rectified linear unit function, a max-pooling layer, and a fully connected feed

Figure 2: Modelling similarity distribution (shades of the matrix represent word similarities)



forward neural network (FFNN).

Representation GenerationWe obtain a basic representation of for span to by simply pooling the token representations obtained from BERT: . Similarly, a basic representation of target phrase is obtained. We then transform to reflect the SimMatrix representation. For this, we use an attention mechanism as shown in Figure 3, which has the same architecture as the Transformer Vaswani et al. [2017]. The attention layer consists of multi-head attention and FFNNs. Our model takes , , and transforms into .

Loss FunctionTo train the phrase representation model, we use a triplet margin loss:

(1)

where and are transformed representations of positive (alignable) and negative (unalignable) pairs, respectively, and is a margin. Intuitively, the loss function makes representations of paraphrase pairs closer, whereas those of nonparaphrase pairs are more distant. For negative examples, we randomly sample phrases that are separated by more than one hop from the alignable pair in . At an inference, we transform the basic representation of a target phrase by our model and compute the cost .

We also tried models that discriminate alignable phrases or minimise the cosine similarity of an alignable pair. However, they were all inferior to the triplet margin loss.

## 5 Experiment Setting

### Creation of ESPADA

To train our phrase representation model, we need a corpus with phrase alignments annotated on sentence pairs. We extended the Syntactic Phrase Alignment Dataset for Evaluation (SPADE) Arase and Tsujii [2018], creating the Extended Syntactic Phrase Alignment DATaset (ESPADA). Following the same annotation scheme, we annotated sentence pairs sampled from NIST OpenMT4 corpora. ESPADA is now the largest annotation corpus for this problem and will be released by the Linguistic Data Consortium (LDC) soon.

Footnote 4: [https://www.nist.gov/litl/iad/mig/openmt](https://www.nist.gov/litl/iad/mig/openmt)

A linguist first annotated gold-standard syntactic trees on paraphrases based on the head-driven phrase structure grammar. Then, three native or near-native English speakers annotated the paraphrases in parallel to identify phrasal paraphrases; the total number of annotated sentences is . Before the formal annotation, there was a training phase to improve annotation agreement; all annotators annotated trial samples.5 One of the authors inspected the results and gave advice on any misunderstandings of the annotation guidelines. Appendix C provides further details of the annotation process.

Footnote 5: excluded from the formal annotation set

Table 1 shows the statistics for ESPADA and SPADE; = phrasal paraphrases were identified, among which unique pairs were agreed upon by at least two annotators and unique pairs were agreed upon by all annotators. The last two rows show, in ESPADA and SPADE, to of pairs did not satisfy the monotonicity condition, and to triplets did not satisfy

\begin{table}
\begin{tabular}{l r r r} \hline \hline  & ESPADA & \multicolumn{2}{c}{SPADE} \\  & & dev & test \\ \hline \# sentence pairs & 1,916 & 50 & 151 \\ \# phrases w/o tokens & 75,283 & 2,584 & 7,438 \\ Total \# pairs & 251,972 & 8,708 & 25,709 \\ \# unique pairs & 105,154 & 3,566 & 10,790 \\ \# pairs agreed by & 80,572 & 2,814 & 8,292 \\ \# pairs agreed by all & 66,246 & 2,328 & 6,627 \\ \hline Non-monotonicity & & 4.7\% & 3.2\% \\ Non-familiars & & 1.4\% & 1.2\% & 1.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics for ESPADA and SPADE (‘#’ stands for ‘number of’)

Figure 3: Phrase representation transformation



the familiariness condition in alignments agreed upon by at least two annotators. Note that the monotonicity and familiness conditions are defined on relations of alignment pairs and triples, respectively; hence, these percentages do not mean that these percentages of alignments are non-compositional.

### Evaluation Metrics and Upper Bounds

We used SPADE as an evaluation corpus; Table 1 shows statistics for its development (dev) and test sets. As evaluation metrics, we used alignment recall (ALIR), alignment precision (ALIP), and alignment F-measure (ALIF) [20, 3]. ALIR evaluates how gold-standard alignments can be replicated by automatic alignments, and ALIP measures how automatic alignments overlap with alignments identified by at least one annotator:

\[{\rm ALII} =\frac{|\{\mathrm{h}|\mathrm{h}\in\mathrm{H_{a}}\wedge\mathrm{ h}\in\mathds{G}\cap\mathds{G^{\prime}}\}|}{|\mathds{G}\cap\mathds{G^{\prime}}|},\] \[{\rm ALIP} =\frac{|\{\mathrm{h}|\mathrm{h}\in\mathrm{H_{a}}\wedge\mathrm{ h}\in\mathds{G}\cup\mathds{G^{\prime}}\}|}{|\mathds{H_{a}}|},\]

where \(\mathds{H_{a}}\) is a set of automatic alignments, and \(\mathds{G}\) and \(\mathds{G^{\prime}}\) are those obtained by two respective annotators. ALIF computes the harmonic mean of ALIR and ALIP. Because SPADE provides alignments by three annotators, there are three combinations for \(\mathds{G}\) and \(\mathds{G^{\prime}}\). The final ALIR, ALIP, and ALIF values are calculated by taking the averages.

Note that these evaluation metrics count null alignments also; hence, ALIP performs differently from a general precision metric in that stricter models will have lower ALIP scores. This is because a stricter model aligning only a small number of phrases (\(\neq\tau_{\emptyset}\)) increases the number of null alignments, making \(|\mathrm{H_{a}}|\) larger.

The agreement among the human annotators can also be measured using ALIR, ALIP, and ALIF by regarding one annotator as a test and the other two as gold-standard and then taking averages. The scores for the trained annotators were consistent between ESPADA and SPADE as shown in Table 2. This indicates that phrase alignment is difficult even for humans because acceptable levels of semantic divergence in paraphrases cannot be perfectly controlled. Hence, we regard these human scores as upper bounds for ALIR, ALIP, and ALIF.

### Comparison Method

As the comparison state-of-the-art syntactic phrase alignment method, we used Arase and Tsujii [54]. We re-implemented this method and compared the performance on aligning gold parse trees.

Additionally, we compared variations of our method via ablation studies. We investigated the effect of CTED by comparing it with alignments by a naive thresholding, which aligns phrases having cosine similarities above a threshold. The threshold was set to maximise the ALIF score on the SPADE development set.

To investigate the effect of our phrase representation model, we compared it with a simply fine-tuned BERT using Equation (1) but directly inputting basic phrase representations of \(\mathbf{e}_{p}^{t}\) and \(\mathbf{e}_{n}^{t}\). To investigate the effect of SimMatrix representation, we compared it with the representation of the [CLS] symbol (denoted as BERT+[CLS]). BERT defines its input to begin with the special symbol [CLS], whose representation has been commonly used as a representation of sentence pair [67]. The assumption here is that BERT may learn to embed information of similarity distribution into [CLS] representation.

As a pre-trained model for generating phrase representations, we compared the fine-tuning approach with the feature-based approach, FastText [8] and embeddings from language models (ELMo) [90]. For all pre-trained models, we used mean-pooling to generate a basic phrase representation, which consistently outperformed max-pooling in our preliminary experiments.

### Model Settings

We used the following public pre-trained models: 'crawl-300d-2M-subword'6 as FastText, 'Original (\(5.5\)B)'7 as ELMo, and 'BERT-Base, Uncased'8 as BERT. We implemented our method and its variations using PyTorch9 with libraries Transformers,10

Footnote 6: [https://dli.fchguploiffies.com/fasttext/vectors-empslish/crawl-300d-2M-subword.zip](https://dli.fchguploiffies.com/fasttext/vectors-empslish/crawl-300d-2M-subword.zip)

Footnote 7: [https://allennlp.org/elmo](https://allennlp.org/elmo)

Footnote 8: [https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)

Footnote 9: [https://pytorch.org/](https://pytorch.org/) (version 1.4.0)

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & ALIR (\%) & ALIP (\%) & ALIF (\%) \\ \hline ESPADA & \(93.3\) & \(90.2\) & \(91.7\) \\ SPADE (dev) & \(93.5\) & \(91.4\) & \(92.4\) \\ SPADE (test) & \(92.3\) & \(90.3\) & \(91.3\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Human performance 

AllenNLP,11 and NetworkX12 for solving the minimum cost maximum flow problem in CTED.

Footnote 11: [https://allennlp.org/](https://allennlp.org/) (version 0.9.0)

Footnote 12: [https://networkx.github.io/](https://networkx.github.io/) (version 2.4)

Our attention mechanism had eight heads; the other settings were the same as those for Transformer Vaswani et al. (2017). Dropouts of \(10\%\) and \(50\%\) were applied to the BERT and ELMo outputs, respectively, as recommended in their papers. The CNN had a kernel size of three in the convolution layer and two for the pooling layer. The SimMatrix was padded with zeros for sentences shorter than the maximum sequence length of \(128\).13

Footnote 13: None of the sentence pairs exceeded this limit.

All models used AdamW Loshchilov and Hutter (2019) as an optimiser, using default settings except on the learning rate. We tuned a few hyperparameters in our models to maximise the ALIF score on the development set of SPADE by a grid search. The value of null alignment cost \(\lambda_{\emptyset}\) was searched for in the range \([0.05,0.95]\) by intervals of \(0.05\), the margin \(\delta\) in the loss function was searched for in \([0.2,1.0]\) by intervals of \(0.2\), and the learning rate was chosen from among \(1.0\mathrm{e}-5\), \(3.0\mathrm{e}-5\), and \(5.0\mathrm{e}-5\).

### Training Settings

All experiments were conducted on an NVIDIA Tesla V\(100\) GPU. We trained our phrase representation model using ESPADA. We simply used all phrase alignments by the three annotators, regarding all of them as equally reliable, each sentence pair has three sets of phrase alignments. We split the entire dataset into training and validation sets (\(90\%\) and \(10\%\), respectively) after randomly shuffling the sentence pairs, which prevents the same sentence pair from appearing in both sets. The batch size was \(16\). Training was terminated by validation-based early-stopping with patience \(5\) and minimum delta \(0.005\).

To alleviate the randomness effects in initialising the neural networks, we trained and evaluated the models \(10\) times with random seeds and report means of the evaluation scores with \(95\%\) confidence intervals. Further, we tested the significance of differences in means of the evaluation scores by the randomised test Efron and Tibshirani (1994). Throughout the paper, we present the best scores with a significance level of \(<1\%\) using a bold font.

## 6 Experiment Results

### Overall Results

Table 3 compares the methods' performance. BERT\(+\)SimMatrix\(+\)CTED (last row) includes the full feature set; it transforms the phrase representation using SimMatrix representation and aligns phrases using CTED. This method performed the best overall, achieving an ALIF score of \(87.4\%\) with post-processing. This ALIF score is \(95.7\%\) of that achieved by humans (Table 2).

We investigated non-compositional alignments produced by BERT\(+\)SimMatrix\(+\)CTED with post-processing. We found that \(0.1\%\) of alignment pairs did not satisfy the monotonicity condition and \(1.2\%\) of alignment triplets did not satisfy the familiarness condition. These non-compositional alignments cover \(3.5\%\) and \(23.2\%\) of those of the gold standard that did not satisfy the monotonicity and familiness conditions, respectively (as shown in Table 1).

**Effect of CTED Algorithm and Post-Processing** The middle and last sets of rows compare CTED-based and thresholding-based alignments. Thresholding-based alignment greedily aligns phrases by disregarding compositionality. In contrast, the pure CTED-based alignment only allows compositional alignments and makes all non-compositional alignments null. Even though CTED is much stricter than thresholding, it achieved competitive ALIF scores. The scores of the CTED-based alignment further improves by allowing non-compositional alignments by post-processing; ALIR, ALIP, and ALIF improved by \(2.2\), \(3.4\), and \(2.8\) percentage points on average, respectively.

**Effect of Phrase Representation Model** The last set of rows shows the performance of alignments by CTED with different phrase representation approaches. BERT\(+\)SimMatrix\(+\)CTED significantly outperformed BERT\(+\)CTED and BERT\(+\)[CLS]\(+\)CTED. The superiority of SimMatrix representation over [CLS] was more pronounced on alignments with post-processing. Although ALIF of BERT\(+\)[CLS]\(+\)CTED with post-processing achieved \(94.4\%\) of the human score, SimMatrix representation further improved it by \(1.2\) percentage points.

These results indicate that a phrase representation that explicitly models the similarity distribution is crucial for handling non-compositional alignments. We conjecture that SimMatrix representation has two effects in phrase alignment.

 

First, it encourages a null alignment in CTED when there is a more similar phrase beyond the local scope. _I.e._, it implicitly relaxes the syntactic constraint when composing compositional alignments that could be too restrictive to handle non-compositional alignments. Second, the SimMatrix representation allows the post-processing to find a globally plausible alignment pair considering the entire similarity distribution.

Effect of Null-Alignment CostFigure 4 presents ALIR and ALIP by the cost of null alignment \(\lambda_{\emptyset}\) for BERT\(+\)SimMatrix\(+\)CTED with and without post-processing. A small \(\lambda_{\emptyset}\) causes the method to align only a small number of phrases and produce a large number of null alignments. In contrast, a large \(\lambda_{\emptyset}\) confuses the method by allowing a larger number of possible alignments. Both situations are harmful, but the former has a larger impact. This is because the constraint of CTED only allows a legitimate set of phrase alignments, which effectively prunes away incorrect alignments.

Figure 4 empirically confirms that the post-processing is effective in improving ALIR and ALIP scores; these scores with post-processing were always higher than those without. The same trend was also observed for BERT\(+\)CTED and for BERT\(+\)[CLS]\(+\)CTED. This occurs because our post-processing only allows non-compositional alignments of minimum cost. Hence, it also improves ALIR and ALIP scores when phrase representations are reliable.

### Effects on Feature-Based Approaches

Table 4 shows the effect on performance when CTED is combined with the feature-based approaches: FastText, ELMo, and BERT without fine-tuning.15 Specifically, we generated a phrase representation by simply mean-pooling token representations generated by these pre-trained models and aligned phrases by CTED or by thresholding. Note that these methods behave deterministically owing to the absence of neural network training.

Footnote 15: Although we also applied our phrase representation model to feature-based approaches, the results were inferior to those given here, as discussed in Appendix D.

BERT w/o fine-tuning\(+\)CTED achieved an ALIF score of \(84.7\%\) with post-processing, even though it only tunes the hyper-parameter \(\lambda_{\emptyset}\). Although it scored lower than the proposed method (BERT\(+\)SimMatrix\(+\)CTED), the result is still encouraging for conducting phrase alignment in domains for which no corpora are available for training our phrase representation model.

Improvements in ALIR, ALIP, and ALIF scores by CTED over thresholding were much greater with FastText than with ELMo or BERT; it showed average gains of \(6.0\) to \(8.6\) percentage points. Improvements ranged from \(-0.8\) to \(2.7\) for ELMo and from \(0.2\) to \(1.3\) percentage points for BERT. The CTED algorithm constrains alignments by the syntactic structures. FastText representations obviously do not retain such structural information. We conjecture that FastText-based alignment is com

Figure 4: ALIR and ALIP on SPADE development set by null alignment cost

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{4}{c}{w/o post-processing} & \multicolumn{4}{c}{w/ post-processing} \\ Method & \(\lambda_{\emptyset}\) & ALIR (\%) & ALIP (\%) & ALIF (\%) & ALIR (\%) & ALIP (\%) & ALIF (\%) \\ \hline Arase and Tsujii (2017) & – & \(81.8\) & \(75.6\) & \(78.6\) & – & – & – \\ \hline BERT\(+\)Thresh. & \(0.80\) & \(83.6\pm 0.2\) & \(83.4\pm 0.2\) & \(83.5\pm 0.2\) & – & – & – \\ BERT\(+\)[CLS]\(+\)Threshold. & \(0.65\) & \(84.7\pm 0.5\) & \(83.5\pm 0.6\) & \(84.1\pm 0.5\) & – & – & – \\ BERT\(+\)SimMatrix\(+\)Theresh. & \(0.60\) & \(84.1\pm 0.4\) & \(84.7\pm 0.3\) & \(84.4\pm 0.3\) & – & – & – \\ \hline BERT\(+\)CTED & \(0.90\) & \(85.3\pm 0.1\) & \(81.9\pm 0.1\) & \(83.5\pm 0.1\) & \(87.4\pm 0.2\) & \(85.2\pm 0.2\) & \(86.3\pm 0.2\) \\ BERT\(+\)[CLS]\(+\)CTED & \(0.80\) & \(85.6\pm 0.3\) & \(82.1\pm 0.5\) & \(83.8\pm 0.4\) & \(87.4\pm 0.4\) & \(85.0\pm 0.7\) & \(86.2\pm 0.5\) \\ BERT\(+\)SimMatrix\(+\)CTED & \(0.80\) & \(85.7\pm 0.2\) & \(82.7\pm 0.1\) & \(84.2\pm 0.2\) & **88.2**\(\pm 0.3\) & **86.6**\(\pm 0.2\) & **87.4**\(\pm 0.2\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: ALIR, ALIP, and ALIF scores with \(95\%\) confidence intervals 

pensated for by CTED. In contrast, the smaller improvements on ELMo and BERT imply that they obtain such structural information through their masked language model training. This result is consistent with previous studies (Hewitt and Manning, 2019; Jawahar et al., 2019; Reif et al., 2019) that confirmed that BERT learns syntactic structures.

## 7 Discussion and Future Work

In contrast to previous methods, ours can align phrases not only in paraphrasal sentence pairs but also in partially paraphrasal pairs. We plan to apply it to a comparable corpus of partial paraphrases and investigate the performance, with the aim of creating a large-scale syntactic and phrasal paraphrase dataset. We intend to expand our method to conduct forest alignments for making it robust against parsing errors, which are inevitable in handling large corpora. Further, as our method does not restrict input to syntactic trees but only assumes tree structures with arbitrary numbering (_e.g_. left-to-right post-order numbering) as input, we intend to try alignments of chunk-based trees, which is desirable for applications that process text fragments, _e.g_. those that perform information extraction.

#### Acknowledgments

Discussions on the tree mapping problem with Dr. Masakazu Ishihata and Mr. Taihei Oki have been illuminating. Dr. Joseph Tabolt, Mr. Michael Doerner, Ms. Tabatha Kobayashi, and Mr. Johannes Hoseth Klev made enormous contributions as annotators for creating ESPADA. We appreciate the anonymous reviewers for their insightful comments and suggestions to improve the paper. This work was supported by JST, ACT-I, Grant Number JP-MJPR16U2, Japan.

## References

* Alabbas and Ramsay (2013) Maytham Alabbas and Allan Ramsay. 2013. Optimising tree edit distance with subtrees for textual entailment. In _Proceedings of International Conference Recent Advances in Natural Language Processing_, pages 9-17.
* Arase and Tsujii (2017) Yuki Arase and Junichi Tsujii. 2017. Monolingual phrase alignment on parse forests. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1-11.
* Arase and Tsujii (2018) Yuki Arase and Junichi Tsujii. 2018. SPADE: Evaluation dataset for monolingual phrase alignment. In _Proceedings of the Language Resources and Evaluation Conference (LREC)_.
* Arase and Tsujiii (2019) Yuki Arase and Jun'ichi Tsujii. 2019. Transfer finetuning: A BERT case study. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5392-5403.
* Bernard et al. (2008) Marc Bernard, Laurent Boyer, Amaury Habrard, and Marc Sebban. 2008. Learning probabilistic models of tree edit distance. _Pattern Recognition_, 41(8):2611-2629.
* Bille (2005) Philip Bille. 2005. A survey on tree edit distance and related problems. _Theoretical Computer Science_, 337(1--3):217--239.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association of Computational Linguistics (TACL)_, 5:135-146.
* Burkett et al. (2010) David Burkett, John Blitzer, and Dan Klein. 2010. Joint parsing and alignment with weakly synchronized grammars. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 127-135.
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1657-1668.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{w/o post-processing} & \multicolumn{3}{c}{w/ post-processing} \\ Method & \(\lambda_{0}\) & ALIR (\%) & ALIP (\%) & ALIF (\%) & ALIR (\%) & ALIP (\%) & ALIF (\%) \\ \hline FastText+Thresh. & \(0.70\) & \(74.7\) & \(72.9\) & \(73.8\) & – & – & – \\ FastText+CTED & \(0.80\) & \(83.3\) & \(78.9\) & \(81.1\) & \(84.3\) & \(81.9\) & \(83.1\) \\ \hline ELMo+Thresh. & \(0.50\) & \(81.7\) & \(80.7\) & \(81.2\) & – & – & – \\ ELMo+CTED & \(0.75\) & \(84.3\) & \(79.8\) & \(82.0\) & \(85.7\) & \(81.8\) & \(83.7\) \\ \hline BERT w/o fine-tuning+Threshold. & \(0.80\) & \(82.3\) & \(79.6\) & \(80.9\) & – & – & – \\ BERT w/o fine-tuning+CTED & \(0.85\) & \(83.6\) & \(79.8\) & \(81.7\) & **85.9** & **83.5** & **84.7** \\ \hline \hline \end{tabular}
\end{table}
Table 4: ALIR, ALIP, and ALIF scores with feature-based approaches 

Do Kook Choe and David McClosky. 2015. Parsing paraphrases with joint inference. In _Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 1223-1233.
* Das and Smith [2009] Dipanjan Das and Noah A. Smith. 2009. Paraphrase identification as probabilistic quasi-synchronous recognition. In _Proceedings of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 468-476.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 4171-4186.
* Dolan et al. [2004] Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In _Proceedings of the International Conference on Computational Linguistics (COLING)_, pages 350-356.
* Efron and Tibshirani [1994] Bradley Efron and R.J. Tibshirani. 1994. _An Introduction to the Bootstrap_. Chapman and Hall/CRC.
* He and Lin [2016] Hua He and Jimmy Lin. 2016. Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 937-948.
* Hellman and Smith [2010] Michael Hellman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 1011-1019.
* Hewitt and Manning [2019] John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 4129-4138.
* Jawahar et al. [2019] Ganesh Jawahar, Benoit Sagot, and Djame Seddah. 2019. What does BERT learn about the structure of language? In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 3651-3657.
* Lan et al. [2017] Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1235-1245.
* Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Martinez-Gomez and Miyao [2016] Pascual Martinez-Gomez and Yusuke Miyao. 2016. Rule extraction for tree-to-tree transducers by cost minimization. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 12-22.
* McCafferty and Nederhof [2016] Martin McCaffery and Mark-Jan Nederhof. 2016. DTED: Evaluation of machine translation structure using dependency parsing and tree edit distance. In _Proceedings of the Workshop on Statistical Machine Translation (WMT)_, pages 491-498.
* Mehdad [2009] Yashar Mehdad. 2009. Automatic cost estimation for tree edit distance using particle swarm optimization. In _Proceeding of the Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 289-292.
* Parikh et al. [2016] Ankur Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2249-2255.
* Peters et al. [2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 2227-2237.
* Punyakanok et al. [2004] Vasin Punyakanok, Dan Roth, and Wen tau Yih. 2004. Mapping dependencies trees: An application to question answering. International Symposium on Artificial Intelligence and Mathematics (Special Session).
* Reif et al. [2019] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In _Proceedings of Conference on Neural Information Processing Systems (NeurIPS)_, pages 8594-8603.
* Smith and Eisner [2006] David Smith and Jason Eisner. 2006. Quasi-synchronous grammars: Alignment by soft projection of syntactic dependencies. In _Proceedings of the Workshop on Statistical Machine Translation (WMT)_, pages 23-30.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of Conference on Neural Information Processing Systems (NeurIPS)_, pages 5998-6008 

Mengqiu Wang and Christopher Manning. 2010. Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In _Proceedings of the International Conference on Computational Linguistics (COLING)_, pages 1164-1172.
* Weese et al. [2014] Jonathan Weese, Juri Ganitkevitch, and Chris Callison-Burch. 2014. PARADIGM: Paraphrase diagnostics through grammar matching. In _Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL)_, pages 192-201.
* Yao et al. [2013] Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In _Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 858-867.
* Yin et al. [2020] Da Yin, Tao Meng, and Kai-Wei Chang. 2020. SettiBERT: A transferable transformer-based architecture for compositional sentiment semantics. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 3695-3706.
* Zhang [1996] Kaizhong Zhang. 1996. A constrained edit distance between unordered labeled trees. _Algorithmica_, 15(3):205-222.
* Zhang and Jiang [1994] Kaizhong Zhang and Tao Jiang. 1994. Some MAX SNP-hard results concerning unordered labeled trees. _Information Processing Letters_, 49(5):249-254.
* Zhang et al. [1995] Kaizhong Zhang, Jason T. L. Wang, and Dennis Shasha. 1995. On the editing distance between undirected acyclic graphs and related problems. In _Annual Symposium on Combinatorial Pattern Matching_, pages 395-407.

## Appendix A Detailed Comparison with Previous Study

Arase and Tsujii (2017) include additional conditions that a legitimate set of compositional alignments should satisfy. One of these is called the root-pair containment condition, which requires the root nodes of trees to be aligned. This constraint firmly restricts their method such that it can only handle a paraphrasal sentence pair as input. Our method, by contrast, can align any pair of sentences, not only paraphrasal sentences but also sentences that are only partially paraphrasal.

Additionally, in their study, the familiariness condition is replaced by the maximum set condition. The maximum set condition, together with the monotonicity condition, constrains all the lowest common ancestors (LCAs) of any pair of non-null alignments in \({\mathbb{H}}_{L}\) to ensure that they are aligned. That is, for all \({\rm h}_{m},{\rm h}_{n}\in{\mathds{H}}_{L}\) of non-null alignments, \(\langle\tau^{s}_{i},\tau^{t}_{i}\rangle\in{\mathds{H}}_{L}\), where \(\tau^{s}_{i}={\text{lca}}(\tau^{s}_{m},\tau^{s}_{n})\) and \(\tau^{t}_{i}={\text{lca}}(\tau^{t}_{m},\tau^{t}_{n})\). Owing to this constraint, their method belongs to the class of LCA-preserving distance mappings (Zhang et al., 1995), whose constraint is tighter than the constraint edit distance mapping. In phrase alignment, this forces LCAs of two aligned nodes to be aligned as well, even though the majority of phrases under the LCAs are null alignments. By contrast, CTED allows such LCAs to have null alignments depending on the alignments of descendant nodes.

```
0: Source and target trees \(T^{s}\) and \(T^{t}\)
0: Tree edit distance matrix \(D[i|j]\), where \(1\leq i\leq|T^{s}|\) and \(1\leq j\leq|T^{t}|\)
1:\(D[0][0]=0\), \(F[0][0]=0\)
2forall\(i=1\) to \(T^{s}|\)do\(\triangleright\) target side is \(\tau_{\emptyset}\)
3:\(F[i][0]=\sum_{k=1}^{N}D[i_{k}][0]\)
4:\(D[i][0]=F[i][0]+\gamma(\langle\tau^{s}_{i},\tau_{\emptyset}\rangle)\)
5:forall\(j=1\) to \(|T^{t}|\)do\(\triangleright\) source side is \(\tau_{\emptyset}\)
6:\(F[0][j]=\sum_{k=1}^{n_{j}}D[0][j_{k}]\)
7:\(D[0][j]=F[0][j]+\gamma(\langle\tau_{\emptyset},\tau^{t}_{j}\rangle)\)
8:forall\(i=1\) to \(|T^{s}|\)do
9:forall\(j=1\) to \(|T^{t}|\)do
10: Compute \(F[i][j]\) (Equation (2))
11: Compute \(D[i][j]\) (Equation (3))
```

**Algorithm B.1**CTED algorithm (Zhang, 1996) 

forests rooted at nodes \(i\) and \(j\) (line 10):

\[F[i][j]=\\ \min\begin{cases}F[0][j]+\min_{1\leq k\leq n_{j}}\{F[i][j_{k}]-F[0][ j_{k}]\},\\ F[i][0]+\min_{1\leq l\leq n_{i}}\{F[i_{l}][j]-F[i_{l}][0]\},\\ \min_{\Pi_{i,j}}\gamma(\mathds{H}_{i,j}).\end{cases} \tag{2}\]

Then (line 11), it computes the minimum cost of \(\langle i,j\rangle\), \(\langle i,\tau_{\emptyset}\rangle\), and \(\langle\tau_{\emptyset},j\rangle\) as

\[D[i][j]=\\ \min\begin{cases}D[0][j]+\min_{1\leq k\leq n_{j}}\{D[i][j_{k}]- D[0][j_{k}]\},\\ D[i][0]+\min_{1\leq l\leq n_{i}}\{D[i_{l}][j]-D[i_{l}][0]\},\\ F[i][j]+\gamma(\langle\tau^{s}_{i},\tau^{t}_{j}\rangle).\end{cases} \tag{3}\]

In Equation (2), \(\gamma(\mathds{H}_{i,j})\) is the summation of the alignment costs between the forests:

\[\gamma(\mathds{H}_{i,j})=\sum_{\langle u,v\rangle\in\mathrm{H}_{i,j}}\gamma( \langle u,v\rangle),\]

where \(u\) or \(v\) is \(\tau_{\emptyset}\) for null alignments.

The algorithm searches for \(\mathds{H}_{i,j}\) that has the minimum cost by solving the minimum cost maximum flow problem on a graph \(G(V,E)\), as shown in Figure 5. The vertex set consists of \(V=\{s_{0},s_{t},\tau^{i}_{\emptyset},\tau^{j}_{\emptyset}\}\cup I\cup J\), where \(s_{0}\) and \(s_{t}\) are the start and sink nodes, respectively, and \(\tau^{i}_{\emptyset}\) and \(\tau^{j}_{\emptyset}\) are null nodes. Each edge in \(E\) has a cost and capacity: Edges \((s_{0},i_{k})\), \((s_{0},\tau^{i}_{\emptyset})\), \((j_{l},s_{t})\), and \((\tau^{j}_{\emptyset},s_{t})\) are cost zero; \((i_{k},j_{l})\) is cost \(D[i_{k}][j_{l}]\); \((\tau^{i}_{\emptyset},j_{t})\) is cost \(D[0][j_{l}]\); \((i_{k},\tau^{j}_{\emptyset})\) is cost \(D[i_{k}][0]\); and \((\tau^{j}_{\emptyset},\tau^{j}_{\emptyset})\) is cost zero. All the edges have capacity one except \((s_{0},\tau^{i}_{\emptyset})\), \((\tau^{i}_{\emptyset},\tau^{j}_{\emptyset})\), and \((\tau^{j}_{\emptyset},s_{t})\), whose capacities are \(n_{j}\), \(\min(n_{i},n_{j})\),18 and \(n_{i}\), respectively. Obviously, the maximum flow of \(G\) is \(n_{i}+n_{j}\) and \(G\) is a network with integer capacities and non-negative costs. The minimum cost on the maximum flow of \(G\) is proven to be in agreement with \(\min_{\mathds{H}_{i,j}}\gamma(\mathds{H}_{i,j})\) in Zhang (1996).

Footnote 18: The flows in a solution should pass through all the non-null nodes; hence, the capacity between empty trees should be \((n_{i}+n_{j})-\max(n_{i},n_{j})=\min(n_{i},n_{j})\), which subtracts the minimum flows from/to non-null nodes from the maximum flow of \(G\). Zhang (1996) set this capacity to \(\max(n_{i},n_{j})-\min(n_{i},n_{j})\), but that produces a degenerate solution.

Algorithm B.1 only shows the computation of the alignment cost for brevity. However, the corresponding edit operations, _i.e._ alignments, can be computed simultaneously in the same manner as the edit cost.

## Appendix C Details of ESPADA Creation

To obtain paraphrasal sentence pairs to annotate, we sampled paraphrases from reference translations in NIST OpenMT corpora19 excluding sentences in SPADE. There are a variety of resources for constructing paraphrases, including reference translations Weese et al. (2014), news texts Dolan et al. (2004), and tweets Lan et al. (2017). Arase and Tsujiri (2018) discussed how paraphrases constructed from reference translations are authentic in the sense that they only pose paraphrastic phenomena because they are constrained by corresponding source sentences. By contrast, paraphrases extracted from other resources tend to have more diverse linguistic phenomena, such as additions and omissions of information and inferences requiring knowledge of the world.

Footnote 19: The flows in a solution should pass through all the non-null nodes; hence, the capacity between empty trees should be \((n_{i}+n_{j})-\max(n_{i},n_{j})=\min(n_{i},n_{j})\), which subtracts the minimum flows from/to non-null nodes from the maximum flow of \(G\). Zhang (1996) set this capacity to \(\max(n_{i},n_{j})-\min(n_{i},n_{j})\), but that produces a degenerate solution.

First, we recruited a linguist who is also a native English speaker to annotate the gold-standard syntactic trees on paraphrases based on the grammar of the head-driven phrase structure. Through this process, the linguist identified and discarded ungrammatical and/or non-paraphrasal pairs. The annotated trees were checked automatically for formatting, and the linguist corrected the annotations of trees with errors, such as trees with inconsistent bracketing. We then had three native or near-native English speakers annotate the phrase alignments.

Figure 5: Minimum cost maximum flow problem (values in parentheses represent the (capacity, cost) of each edge)





## Appendix D Phrase Representation with Feature-Based Approaches

We also applied our phrase representation model to ELMo and BERT, using them as feature generators. We trained only the attention and CNN models using ESPADA. For ELMo, we also trained the scalar weighting parameters.

Table 5 shows the results. Unfortunately, all of these methods are inferior to their counterparts that lack our phrase representation model: ELMo\(+\)CTED and BERT w/o fine-tuning\(+\)CTED, respectively. We conjecture that ESPADA may be insufficiently large for training our phrase representation model to adapt to a pre-trained model that behaves in a completely independent manner. BERT's ability to adapt quickly to a specific task by fine-tuning is a notable advantage.