Monolingual Phrase Alignment as Parse Forest Mapping
Year: 2023

Abstract:
We tackle the problem of monolingual phrase alignment conforming to syntactic structures. The existing method formalises the problem as unordered tree mapping; hence, the alignment quality is easily affected by syntactic ambiguities. We address this problem by expanding the method to align parse forests rather than 1-best trees, where syntactic structures and phrase alignment are simultaneously identified. The proposed method achieves efficient alignment by mapping forests on a packed structure. The experimental results indicated that our method improves the phrase alignment quality of the state-of-the-art method by aligning forests rather than 1-best trees.

heading: 1 Introduction
text: Monolingual phrase alignment, which identifies semantically corresponding phrase pairs in sentences, is a fundamental technique useful for paraphrase recognition (Das and Smith, 2009), textual entailment recognition (MacCartney et al., 2008; Heilman and Smith, 2010), question answering (Wang and Manning, 2010), and interpreting semantic textual similarity (Agirre et al., 2015; Li and Srikumar, 2016). Its ability to declare overlapping information across sentences is also useful for summarisation (Brook Weiss et al., 2021) and for interactive document exploration (Shapira et al., 2017; Hirsch et al., 2021). There are two approaches to phrase alignment: one aligns chunks of arbitrary spans (e.g., n-grams) (Yao et al., 2013; Ouyang and McKeown, 2019; Lan et al., 2021) while the other targets on syntactic phrases (Arase and Tsujii, 2017, 2020). In this study, we take the latter approach to identify phrasal paraphrases conforming to syntactic structures that allow modelling sentences based on syntax (Socher et al., 2013; Tai et al., 2015).
∗This work was completed at Osaka University.
The current state-of-the-art syntactic phrase alignment (Arase and Tsujii, 2020) has formulated the phrase alignment as the unordered tree mapping problem between trees of source and target sentences.1 Their method realised an efficient alignment with a solid theoretical background by adopting the constrained tree edit distance algorithm (Zhang, 1996), which aligns syntactic trees. However, their experiments were limited to using manually assigned gold syntactic trees, disregarding the effects of syntactic ambiguities that cause parse errors in practical parsers.
We address this problem by expanding the method proposed by Arase and Tsujii (2020) to align parse forests. Specifically, our method considers the likelihood of both syntactic structures and phrase alignment, i.e., it simultaneously identifies the syntactic structures of input sentence pairs and phrase alignment within. Figure 1 illustrates phrase alignment by our method, where the trees show syntactic structures of the source (top) and target (bottom). The alignment is colour-coded; the same colour nodes are pairs. For example, the pair of orange nodes represent that verb phrases ‘made this statement’ and ‘gave this statement’ are paraphrases. Remarkably, in the source tree, the
1We refer to one sentence of a pair as a source and another as a target for the sake of explanation.
449
prepositional phrase ‘at the Karachi Shipyard’ (the pink node) is correctly attached to the preceding verb phrase (the orange node) because the attachment ambiguity is resolved by referring to the target. In contrast, the 1-best tree failed to derive this structure.
The experimental results on the standard corpus indicated that the proposed method improves the phrase alignment quality of the state-of-the-art aligning 1-best parse trees. We also conducted a manual analysis that revealed attachment errors can be addressed by forest alignment.

heading: 2 Preliminary: Tree Alignment
text: Arase and Tsujii (2020) has formulated phrase alignment as the unordered tree mapping. They adopted the constrained tree edit distance (CTED) algorithm (Zhang, 1996) to identify optimal mappings of phrases in polynomial time. The CTED algorithm is based on dynamic programming; hence their method recursively aligns phrases from leaves to root nodes of source and target syntactic trees.
In the alignment algorithm, alignment of node i and j, denoted as ⟨i, j⟩, incurs a cost defined by a function γ(⟨i, j⟩) → R. In their method, the cost function is the cosine distance between phrase vectors of spans covered by i and j, where the vectors are computed by pooling token representations obtained by a fine-tuned pre-trained language model. A phrase is allowed not to have correspondence, i.e., null alignment, which is modelled as alignment to an empty node τϕ. The cost of null alignment is predetermined and given as a hyperparameter λϕ.
We denote Ti as the subtree rooted at node i. If we delete the node i from Ti, there remain a set of subtrees whose root nodes have been the children of i: {iℓ|i1, · · · , ini}, where ni is the number of the children. When we do not assume the order among these subtrees, they constitute an unordered forest, denoted as Fi. The CTED algorithm recursively computes the minimum cost to align subtrees of T si and T tj as follows.
D(T si , T t j ) =
min    D(τϕ, T t j ) + min 1≤k≤nj {D(T si , T tjk)−D(τϕ, T tjk)}, D(T si , τϕ) + min 1≤ℓ≤ni {D(T siℓ , T tj )−D(T siℓ , τϕ)},
D(F si , F t j ) + γ(⟨i, j⟩).
(1)
Specifically, Equation (1) computes the minimum cost among the cases regarding the alignment of i and j, i.e., ⟨τϕ, j⟩, ⟨i, τϕ⟩, and ⟨i, j⟩, which correspond to the first, second, and the third expressions,
respectively. Notice that the last case (i.e., ⟨i, j⟩) requires the alignment cost of forests under these nodes, i.e., F si and F t j . The cost to align F s i and F tj is computed as follows.
D(F si , F t j ) =
min    D(τϕ, F t j ) + min 1≤k≤nj {D(F si , F tjk)−D(τϕ, F tjk)}, D(F si , τϕ) + min 1≤ℓ≤ni {D(F siℓ , F tj )−D(F siℓ , τϕ)},
min M(i,j)
γ(M(i, j)).
(2)
Here, the first two expressions correspond to null alignment. The third expression identifies the constrained edit distance mapping between F si and F t j , denoted asM(i, j), of the minimum cost. It can be efficiently solved as the minimum cost maximum flow problem on a graph constructed based on these forests. For more details of the alignment algorithm, please refer to (Arase and Tsujii, 2020).
Some phrases may have long-distance correspondences (Heilman and Smith, 2010; Arase and Tsujii, 2017) that cannot be monotonically composed of alignment of descendant nodes, which hence cannot be identified by the CTED algorithm. Arase and Tsujii (2020) align such phrases by heuristicbased post-processing.

heading: 3 Proposed Method: Forest Alignment
text: We expand the alignment method proposed by Arase and Tsujii (2020) to align parse forests instead of trees. The syntactic structures of the input sentence pair are determined simultaneously with phrase alignment. A naive approach to align forests is considering combinations of all candidate trees and then finding the best one. However, this procedure is prohibitively computationally expensive considering the number of valid tree structures. We achieve efficient forest alignment by expanding the CTED algorithm to perform tree mapping on a packed forest structure (Miyao and Tsujii, 2008).
Syntactic Plausibility Studies on parallel parsing (Burkett et al., 2010; Choe and McClosky, 2015) have shown that syntactic ambiguity can be resolved by referring to sentences parsed in parallel with each other. Inspired by these studies, we consider the likelihood of parsing in the alignment cost function. Specifically, Equation (2) is expanded to consider the parsing likelihoods:
D̂(F si , F t j ) = D(F s i , F t j )− λs
S(T si )+S(T t j )
2 , (3)
where S(·) indicates the likelihood of a subtree obtained from a syntactic parser, and λs is a hyperparameter that balances both terms.
Alignment on Packed Forests The packed forest structure corresponds to the packed charts in the CFG parsing and can represent an exponential number of trees with a polynomial number of nodes. Specifically, a packed forests under the node i, PFi = {F ik}k, stores different possible syntactic structures (forests) under i. Figure 2 shows examples. Each box corresponds to a node where different possible structures are stored. In the left box, the source can be composed by combining (a) a verb phrase ‘made’ and noun phrase ‘this statement at the Karachi Shipyard’ and (b) a verb phrase ‘made this statement’ and prepositional phrase ‘at the Karachi Shipyard’.
Algorithm 3.1 illustrates our alignment mechanism. It computes the cost to align all combinations of possible structures on the packed forests and memorizes only the one with the minimum cost. That is, only the pair with the minimum cost needs to be considered in the alignment of the upper nodes. In the examples in Figure 2, there are 2 possible structures in the source and target. The proposed method computes the costs of the 2× 2 combinations and stores only the minimum cost and the corresponding structures.

heading: 4 Experiment
text: We evaluate the performance of syntactic phrase alignment of the proposed method compared to the previous state-of-the-art.

heading: 4.1 Evaluation Corpus
text: As an evaluation corpus, we used Syntactic Phrase Alignment Dataset for Evaluation (SPADE) (Arase
Algorithm 3.1 Packed forest mapping Input: Packed (unordered) forests under the node
i and j: PFs,i and PFt,j
1: cm ←∞ ▷ Minimum cost 2: MF← ∅ ▷ Pair of forests 3: for all F s,ik ∈ PFs,i do 4: for all F t,jℓ ∈ PFt,j do 5: c← D̂(F s,ik , F t,j ℓ ) ▷ Eq. (3) 6: if c < cm then 7: cm ← c 8: MF← {F s,ik , F t,j ℓ } 9: Compute D(T si , T t j ) with cm,MF ▷ Eq. (1)
and Tsujii, 2018).2 SPADE consists of English paraphrase sentence pairs assigned by their gold constituency trees annotated by linguistic professionals and phrase alignment identified by three native and near-native English speakers. It provides 50 sentence pairs as a development (dev) set and 151 sentence pairs as a test set. While these numbers of sentences may look small, the numbers of phrase pairs are sufficiently large to have statistically meaningful observations, i.e., 8, 708 phrase pairs and 25, 709 phrase pairs in the dev and test sets, respectively. Remind that our method does not require training; only its hyper-parameters should be tuned using the dev set.

heading: 4.2 Evaluation Metrics
text: Metrics for Alignment Quality Alignment recall (ALIR), alignment precision (ALIP), and alignment F-measure (ALIF) are the standard evaluation metrics defined by SPADE. ALIR evaluates how gold-standard alignment can be replicated by automatic alignment, and ALIP measures how automatic alignment overlaps with alignment pairs
2 https://catalog.ldc.upenn.edu/LDC2018T09
identified by at least one annotator.
ALIR = |{h|h ∈ H ∧ h ∈ G ∩ G′}|
|G ∩ G′| , (4)
ALIP = |{h|h ∈ H ∧ h ∈ G ∪ G′}|
|H| , (5)
where H is a set of identified pairs, G and G′ are those obtained by two respective annotators, and the operator | · | counts the elements in a set. ALIF computes the harmonic mean of ALIR and ALIP. Because SPADE provides alignment pairs by three annotators, there are three combinations for G and G′. The final ALIR, ALIP, and ALIF values are calculated by taking the averages.
Note that these evaluation metrics count null alignment pairs also; hence, ALIP performs differently from the general precision in that stricter models will have lower ALIP scores. This is because a stricter model aligning only a small number of phrases ( ̸= τϕ) increases the number of null alignment pairs, making |H| larger. Metric for Phrase Structure We also evaluated the correctness of phrase structures as the phrase span matching ratio (PSMR) against the gold trees. Specifically, PSMR computes the ratio of gold spans that exactly match with the spans in aligned trees. We compute the macro-average of PSMR of all source and target sentences.

heading: 4.3 Baseline
text: We compared our method to the state-of-the-art (Arase and Tsujii, 2020) on the SPADE corpus (denoted as TreeAligner hereafter). Their original experiments aligned gold syntactic trees annotated in SPADE. To replicate a realistic scenario where gold syntactic structures are unavailable, we used an off-the-shelf syntactic parser, namely, Enju (Miyao and Tsujii, 2008).3 We evaluated TreeAligner by inputting the 1-best trees obtained by Enju as the baseline. In contrast, the proposed method (denoted as ForestAligner hereafter) takes parse forests in the packed representation obtained by Enju as input.

heading: 4.4 Model Settings
text: For replicating TreeAligner, we used the released codes of the authors.4 We implemented our ForestAligner based on them using Pytorch5. As the
3 https://mynlp.is.s.u-tokyo.ac.jp/enju/ 4 https://github.com/yukiar/phrase_alignment_cted 5 https://pytorch.org/ (version 1.7.1)
phrase representation model in both TreeAligner and ForestAligner, we commonly used the bidirectional encoder representations from transformers (BERT) (Devlin et al., 2019) fine-tuned by Arase and Tsujii (2020)6. After inputting a sentence pair, a phrase representation was obtained by meanpooling the token representations consisting of the corresponding phrase.
The hyperparameters were tuned to maximise the evaluation metrics on the SPADE dev set. For TreeAligner, the hyperparameter of the null alignment cost, λϕ, was set to 0.75 to maximise ALIF on the dev set. For ForestAligner, λϕ and λs were set to 0.80 and 3.0 × 1011,7 respectively, to maximise the arithmetic mean of ALIF and PSMR on the dev set.

heading: 4.5 Results
text: Table 1 shows the experimental results. The ALIR, ALIP, and ALIF scores of TreeAligner significantly dropped when aligning 1-best trees compared to the case of aligning gold trees (the first row). Our ForestAligner improved ALIR by 1.3%, ALIP by 2.6%, and ALIF by 2.0% compared to TreeAligner with 1-best trees, which confirms the effectiveness of forest alignment.
For PSMR, ForestAligner moderately improved TreeAligner by 0.3%, which shows the parse errors in 1-best trees can be fixed through forest alignment. To investigate what kind of parse errors were addressed and newly introduced by forest alignment, we randomly sampled 40 sentence pairs where the PSMR score increased (20 sentences) and decreased (20 sentences) compared to TreeAligner. One of the authors observed resultant trees and manually categorised them into error types. Table 2 shows the results, indicating
6 https://zenodo.org/record/4686663#.YpcR2S_3LJQ
(Model: BERT1F_TripletMarginLoss_margin-1.0_lr-3e05_mean_100_ft-bert-base-uncased.pkl)
7The λs takes a large value due to different ranges of alignment cost and parsing likelihood. It was searched in the range of 1.0× 1011 to 9.0× 1011.
ForestAligner tends to fix noun phrase attachment errors while increases modifier attachment errors. The prepositional phrase attachment is a mixture of both improvements and deterioration.
Figure 1 illustrates alignment results by ForestAligner. For the source sentence, the correct structure of composing a phrase ‘made this statement at the Karachi Shipyard’ with a child verb phrase ‘made this statement’ (the orange node) and prepositional phrase ‘at the Karachi Shipyard’ (the pink node) were identified. In contrast, in the 1-best tree, the prepositional phrase was wrongly attached to a noun phrase of ‘this statement’ to compose a phrase ‘this statement at the Karachi Shipyard’, which prevented alignment of ‘made this statement’ and ‘gave this statement’ (the orange node pair).

heading: 5 Discussion: Alignment of Less-Similar Sentences
text: As discussed in Section 1, phrase alignment is coveted by various applications like paraphrase and textual entailment recognition and question answering. Such applications are different from SPADE, i.e., alignment of paraphrases, in that they require alignment of less-similar sentences, too. It is not a trivial difference as it sounds.
As a preliminary experiment, we aligned the test set8 of the semantic textual similarity (STS) benchmark (Cer et al., 2017) and converted alignment costs into similarity scores. Specifically, we normalised the root-level alignment costs by sentence lengths9 and scaled them to be compatible with the STS labels, i.e., from 0 (dissimilar) to 5 (equivalent). As a result, Pearson’s correlation coefficient of the predicted scores and human labels was limited to 0.51, which is comparable to estimating sentence-level similarity using static word embeddings.10
8We excluded 7 sentence pairs that Enju failed to output. 9Alignment costs obviously depend on sentence lengths.
10 https://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark
Table 3 shows the average similarity scores per human label. While ForestAligner outputs a noticeably high score on the most similar sentence pairs, other scores are almost uniform on less similar sentences. We conjecture that one of the factors causing this phenomenon is the lack of exposure to lesssimilar examples during development. The same can happen on existing phrase alignment methods trained on annotated corpora consisting of paraphrasal or highly similar sentence pairs (Thadani et al., 2012; Lan et al., 2021). The distributions of alignment pairs are largely different in semantically similar and less-similar sentences, where alignment is dense in the former but sparse in the latter. Hence, alignment methods trained only on similar sentences may tend to align phrases that should be unaligned.
While there are only a few corpora annotating alignment on less-similar sentences (Ernst et al., 2021), this direction is worth exploring to apply alignment techniques in practical applications. In future work, we will create corpora of this kind and explore robust phrase alignment on both similar and less-similar sentences.

heading: Acknowledgements
text: We sincerely appreciate the anonymous reviewers for their insightful comments and suggestions to improve the paper. This work was supported by JSPS KAKENHI Grant Number JP21H03564.

heading: Future Work
text: In future work, we will create corpora of this kind and explore robust phrase alignment on both similar and less-similar sentences