SlowGuess/NAACL_2022_OCR huggingface dataset:
Erguang Yang $^{1}$ , Chenglin Bai $^{1}$ , Deyi Xiong $^{2}$ , Yujie Zhang $^{1*}$ , Yao Meng $^{3}$ , Jinan Xu $^{1}$ , Yufeng Chen $^{1}$
$^{1}$ School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China
$^{2}$ College of Intelligence and Computing, Tianjin University, Tianjin, China
$^{3}$ Lenovo Research AI Lab, Beijing, China
{egyang,chenglin.bai,yjzhang,jaxu,chenyf}@bjtu.edu.cn,
dyxiong@tju.edu.cn, yaomeng1@lenovo.com
Syntax-controlled paraphrase generation aims to produce paraphrase conform to given syntactic patterns. To address this task, recent works have started to use parse trees (or syntactic templates) to guide generation. A constituency parse tree contains abundant structural information, such as parent-child relation, sibling relation, and the alignment relation between words and nodes. Previous works have only utilized parent-child and alignment relations, which may affect the generation quality. To address this limitation, we propose a Structural Information-augmented Syntax-Controlled Paraphrasing (SI-SCP) model. Particularly, we design a syntax encoder based on tree-transformer to capture parent-child and sibling relations. To model the alignment relation between words and nodes, we propose an attention regularization objective, which makes the decoder accurately select corresponding syntax nodes to guide the generation of words. Experiments show that SI-SCP achieves state-of-the-art performances in terms of semantic and syntactic quality on two popular benchmark datasets. Additionally, we propose a Syntactic Template Retriever (STR) to retrieve compatible syntactic structures. We validate that STR is capable of retrieving compatible syntactic structures. We further demonstrate the effectiveness of SI-SCP to generate diverse paraphrases with retrieved syntactic structures.
Paraphrases are texts or passages conveying the same meaning but with different surface realization. Paraphrase generation (PG) is a key technology of automatically generating a restatement for a given text, which has the potential use in many downstream tasks, such as question answering (Gan and Ng, 2019), machine translation (Zhou et al., 2019), and sentence simplification (Zhao
Figure 1: An example of a constituency tree structure. Parent-child relation:  $\mathrm{NP} \rightarrow \mathrm{NN}$ , sibling relation: NP NP VP Dot, alignment relation between nodes and words: (NN thing), (NP the same thing).
et al., 2018). However, a natural sentence can be paraphrased into various surface forms.
To obtain diverse paraphrases, controllable paraphrase generation (CPG) with specified styles has recently attracted growing interests e.g., satisfying particular sentiment (Hu et al., 2017; John et al., 2019; Dai et al., 2019; Lee et al., 2021) or syntactic structure (Iyyer et al., 2018; Chen et al., 2019; Liu et al., 2020; Kumar et al., 2020; Li et al., 2020; Yang et al., 2021). As CPG can produce diverse paraphrases by exposing syntactic control, it can be used in a wide range of application scenarios, such as dialogue generation (Niu and Bansal, 2018), data augmentation (Iyyer et al., 2018; Yang et al., 2021; Sun et al., 2021), and diverse question generation (Yu and Jiang, 2021), etc.
Generally, syntax-controlled paraphrase generation needs to tackle two major challenges. The first challenge is how syntax-controlled generating is achieved? For this challenge, Iyyer et al. (2018) use two encoders to encode input sentences and linearized parse trees to produce paraphrases. A constituency parse tree contains abundant structural information, such as parent-child relation, sibling relation, and the alignment relation between words and nodes, as shown in Figure 1. Linearizing parse trees, typically, result in loss of structural informa
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2079 - 2090 July 10-15, 2022 ©2022 Association for Computational Linguistics
tion. Kumar et al. (2020) encode parse trees in a top-down manner, and then a queue-based decoding mechanism is proposed to incorporate syntax information. Li et al. (2020) employ a path attention mechanism to capture the tree structure of the syntax. However, these methods still have some limitations. The top-down encoding manner and path attention only consider parent-child relation. Additionally, due to the complexity of the queue-based decoding mechanism (Kumar et al., 2020), the predicted pop sequences cannot be guaranteed to be perfect, which may cause error propagation.
The second challenge is how compatible syntactic structures can be retrieved to guide generation in practice? Previous works (Iyyer et al., 2018; Li et al., 2020) use common syntactic templates that appear most frequently from the corpus, which means that these works generate syntactically diverse paraphrases using a fixed set of syntactic structures for all input sentences. The diversity in syntax is hence limited. Moreover, not all sentences can be paraphrased into the same set of syntactic structures.
In order to address these problems, we propose a Structural Information-augmented Syntax-Controlled Paraphrasing (SI-SCP) model based on attention network. Particularly, we design a tree transformer to capture parent-child and sibling relation. To learn the alignment relation between words and nodes, we propose an attention regularization objective. The basic motivation is that a syntactic template contains several syntax nodes which guide the generation of different words, respectively. As shown in Figure 1, the NP node guides the generation of noun phrase. Learning alignment relation makes the decoder accurately select corresponding syntax nodes to guide the generation of words. Additionally, to enhance diversity in syntax-controlled generation, we propose a Syntactic Template Retriever (STR) to retrieve compatible syntactic structures for any input sentence.
We evaluate our model on two popular benchmark datasets. Experiment results show that S-SCP achieves the state-of-the-art performance in syntactic and semantic quality. The visualization results show the attention regularization makes the decoder accurately attend to corresponding syntactic nodes during decoding. Human evaluation also demonstrates that our method is able to generate semantically and syntactically better sentences
than previous methods. We further show that STR can retrieve more compatible structures compared with the common syntactic templates method. S-SCP can generate more syntactically diverse paraphrases with retrieved syntactic structures.
In summary, the major contributions of this paper are as follows:
- We build a novel syntax-controlled paraphrasing model that contains a tree-transformer and an attention regularization objective.
- To retrieve compatible syntactic structures in practice, we propose a syntactic template retriever.
- Experiments show that our SI-SCP achieves new state-of-the-art results in both semantic and syntactic evaluation on two popular benchmark datasets. We further demonstrate that STR is capable of retrieving compatible syntactic templates. The SI-SCP can produce more syntactically diverse paraphrases with retrieved syntactic structures.
We focus primarily on the task of syntactically controlled paraphrase generation, which has recently attracted increasing attention (Iyyer et al., 2018; Chen et al., 2019; Liu et al., 2020; Kumar et al., 2020; Li et al., 2020). According to the control element, previous works can be divided into two categories. The first strand of research uses sentential exemplar as syntactic control. Chen et al. (2019) and Liu et al. (2020) design a latent variable to learn syntax by encoding the exemplar itself, and then use a syntactic latent variable to guide the generation of paraphrases. However, a latent based approach might not offer enough explicit syntactic information as guaranteed by actual constituency parse trees (Kumar et al., 2020). The second strand of research takes a parse tree as a syntactic input, controlling the syntax of generated text with the structure specified by the parse tree. Iyyer et al. (2018) use two encoders to encode input sentences and linearized parse trees to produce paraphrases. Due to the linearization of syntactic tree, a lot of structural information is generally lost. To capture the tree structure of the parse tree, Kumar et al. (2020) encode parse trees in a top-down manner, and then a queue-based decoding mechanism is proposed to incorporate syntactic information. Li
Figure 2: The architecture of syntax-controlled paraphrasing model. The sequence encoder is built with the transformer network, which is used to encode leaf nodes in the syntactic template. Please refer Section 3.3 for details.
et al. (2020) design a syntax encoder based on a path attention mechanism. However, these methods only consider parent-child relation in parse trees.
Differing from previous approaches, we propose a novel tree-transformer to model parent-child and sibling relation for better capturing the tree structure of the parse tree. To learn the alignment relation between words and nodes, we also propose an attention regularization objective, which makes the decoder accurately select corresponding syntax nodes to guide the generation of words. Additionally, we propose a syntactic template retriever that can help to retrieve compatible syntactic structures for any input sentences.
3 Syntax-Controlled Paraphrasing
We formulate the problem of syntax-controlled paraphrase generation as follows. Given a source sentence  $\pmb{x}$  and a syntactic template  $\pmb{t}$ , the syntax-controlled text generation aims to generate target sentence  $\pmb{y}$  which conveys the meaning of  $\pmb{x}$  and conform to the syntactic structure of  $\pmb{t}$ .
The syntactic template  $t$  is a partial constituency parse tree that provides a general syntax skeleton. Due to different levels of syntax trees contain different information, to compare fairly, we use the top-4 layers (in this work) of the full parse tree of
$y$  as the syntactic template for all baselines. Of course, our model can also use syntactic templates from other layers.
As shown in Figure 2, our model contains a sentence encoder and a syntactic encoder, which encodes the source sentence  $\pmb{x}$  and the template  $\pmb{t}$  separately. We deploy a target sentence decoder to generate the final text. The details of the proposed approach will be presented below.
3.2 Sentence Encoder
The sentence encoder contains  $N$  stacked transformer blocks. The hidden states of the sentence encoder are calculated by:
$$
\boldsymbol {h} _ {\boldsymbol {x}} = \text {T r a n s f o r m e r} (\boldsymbol {x}) \tag {1}
$$
where  $h_x$  is final hidden state of source sentence  $x$ .
3.3 Syntactic Encoder
This encoder provides the necessary syntactic signal for the generation of paraphrases. A parse tree contains rich parent-child and sibling relations. To capture these information, we propose a tree-transformer with a syntax embedding layer, parent-child attention, and sibling attention modules.
Syntax Embedding Layer Given a syntactic template  $t$ , we use (node, level, index) format
sequences to represent it. Formally, let  $\pmb{t} = \{t_1, t_2, \dots, t_n\}$ , where  $\pmb{t}_i = (p_i, l_i, s_i)$ ,  $p_i$  is a parse tree node,  $l_i$  is its level,  $s_i$  is the index of the node at a specific level. For example, the syntactic template in Figure 2 is represented by {ROOT, 1, 1), (S, 2, 1), (NP, 3, 1), (DT, 4, 1), (VP, 3, 2), (VBZ 4, 2), (SBAR, 4, 3)}.
At the embedding layer, node tokens, level tokens, and index tokens are embedded respectively and then added together to produce the syntax embedding at position  $i$ :
$$
\operatorname {E m b} \left(\boldsymbol {t} _ {i}\right) = \operatorname {E m b} \left(p _ {i}\right) + \operatorname {E m b} \left(l _ {i}\right) + \operatorname {E m b} \left(s _ {i}\right) \quad (2)
$$
Parent-Child and Sibling Attention Then we calculate the hidden state of each node in a tree-structure manner. Specifically, we introduce an adjacency matrix to guide the calculation of the self-attention module:
$$
\boldsymbol {A} = \operatorname {S o f t m a x} \left(\left(\boldsymbol {Q} \boldsymbol {K} ^ {T} / \sqrt {d}\right) \odot \boldsymbol {M}\right) \tag {3}
$$
$$
\boldsymbol {h} _ {t} = \boldsymbol {A} \cdot \boldsymbol {V}
$$
where  $Q$  is a query matrix consisting of query vectors with dimension  $d$ ,  $K$  is a key matrix consisting of key vectors with dimension  $d$ ,  $V$  is a value matrix consisting of key vectors with dimension  $d$ .  $M$  is an adjacency matrix obtained from the syntactic template.  $\odot$  denotes the element-wise product. For example, Figure 2 shows examples of the parent-child adjacency matrix and sibling adjacency matrix. We encode parent-child and sibling information by parent-child and sibling attention operations. We stack multiple tree transformer blocks to make the information of the ROOT node flow to leaf nodes.
After obtaining hidden states of all nodes, we use another encoder to encode all the leaf nodes in a sequential way:
$$
\boldsymbol {h} _ {\text {l e a f}} ^ {\text {s e q}} = \operatorname {T r a n s f o r m e r} \left(\boldsymbol {h} _ {\text {l e a f}}\right) \tag {4}
$$
The reason we use this sequential encode is that parent-child and sibling attention modules would create a mismatch between the encoding and decoding process. Thus, it may be beneficial to introduce sequence information. We also demonstrate the effectiveness of the sequential encoder with experiments in Section 5.2.4.
Based on the input sentence hidden states  $h_x$  and syntactic template hidden states  $h_{leaf}^{seq}$ , the target
transformer decoder uses two cross-attention modules to jointly exploit the input sentence and syntactic template information to generate the target text  $\mathbf{y}$ . Cross attentions are calculated as follows:
$$
\boldsymbol {h} _ {y, t}, \boldsymbol {A} _ {t} = \operatorname {A t t n} \left(\boldsymbol {h} _ {y}, \boldsymbol {h} _ {\text {l e a f}} ^ {\text {s e q}}, \boldsymbol {h} _ {\text {l e a f}} ^ {\text {s e q}}\right) \tag {5}
$$
$$
\pmb {h} _ {y, t, x}, \pmb {A} _ {x} = \mathrm {A t t n} (\pmb {h} _ {y, t}, \pmb {h} _ {x}, \pmb {h} _ {x})
$$
where Attn is the multi-head attention module in transformer.  $h_y$  is hidden states of decoder,  $A_t$  and  $A_x$  are attention scores with the template and the input sentence, respectively. Through the feedforward network sub-layer,  $h_{y,t,x}$  will be used as the input of the next layer or used to predict the next word.
Syntax Attention Regularization The attention weight  $\mathbf{A}_t$  is essential for accurate syntactic control. We propose an regularization method to guide the learning of  $\mathbf{A}_t$  using the alignment of nodes and words in a parse tree. Specifically, we propose a attention regularization loss as follows:
$$
\mathcal {L} _ {a r} = \operatorname {M S E} \left(\boldsymbol {A} _ {t}, \hat {\boldsymbol {A}} _ {t}\right) \tag {6}
$$
where  $\hat{A}_t$  are oracle attention weights obtained from the alignment of nodes and words in a parse tree. MSE denotes the Mean Square Error loss function. By doing so, we can make learned  $A_t$  close to oracle attention distribution. We also introduce label smoothing (0.25) to reduce errors caused by parsing.
Because multi-layer transformer and multi-head attention mechanism produce multiple syntax attention matrices, we simply make each generated attention matrix close to the oracle attention. We leave alternatives to this simple method to our future work.
To train the above model, we optimize the following objective function:
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {c e} + \lambda_ {2} \mathcal {L} _ {a r} \tag {7}
$$
where  $\mathcal{L}_{ce} = -\sum_{t=1}^{N} \log p(y_t | \boldsymbol{x}, t, y_{1:t-1})$  is the cross-entropy loss for ground-true  $y$ ,  $\lambda_*$  are balancing hyper-parameters.
4 Syntactic Template Retriever
A sentence cannot be converted to any syntactic structures. Incompatible syntax will lead to imperfect paraphrase conversion and nonsensical
sentences. In this work, we propose a Syntactic Template Retriever (STR) to find compatible syntax for a sentence in practice. The basic assumption is that syntactic templates expressing the same meaning are relatively close in semantic space. Specifically, given an input sentence and its parse tree, the retriever retrieves compatible syntactic templates based on similarity from a pre-collected template library.
We use a template encoder to map any template  $t$  into a vector space. Meanwhile, for the input sentence  $x$  and its parse tree  $p_x$  (i.e., the query  $q$ ), we use a sentence encoder and parse tree encoder to map them to the same vector space, and then retrieves  $k$  syntactic templates whose vectors are closest to the query vector. We define the similarity between the query and the syntactic template using the dot product of their vectors:
$$
\operatorname {s i m} (\boldsymbol {q}, \boldsymbol {t}) = \boldsymbol {v} _ {q} \cdot \boldsymbol {v} _ {t} ^ {T}
$$
$$
\boldsymbol {v} _ {t} = \operatorname {E n c} _ {t} (\boldsymbol {t}) \tag {8}
$$
$$
\boldsymbol {v} _ {q} = \boldsymbol {W} \cdot \left[ \operatorname {E n c} _ {x} (\boldsymbol {x}); \operatorname {E n c} _ {p} (\boldsymbol {p} _ {x}) \right]
$$
where  $\mathrm{Enc}_*$  denote three different encoders,  $W$  is a linear layer, ' denotes concatenation operation.
The three encoders are built with the transformer encoder. We use linearized parse tree and syntactic template as input  $^{1}$ , prepend [CLS] token to each input, and then take the representation of the prepended token as the output of each encoder in a similar way to BERT (Devlin et al., 2019).
The goal is to create a vector space such that relevant pairs of queries and templates will have a smaller distance (i.e., higher similarity) than the irrelevant pairs. We use paraphrase pairs to build training data, where positive templates are from reference sentences.
Let  $(\pmb{q}_i, \pmb{t}_i^+, \pmb{t}_{i,1}^-, \dots, \pmb{t}_{i,n}^-)$  be a training instance that consists of one relevant (positive) template  $\pmb{t}_i^+$ , along with  $n$  irrelevant (negative) templates  $\pmb{t}_{i,j}^-$ . To train the retriever, we optimize the negative log
Model	BLEU↑	R-1 / R-2 / R-L↑	MTR↑	TED↓
ParaNMT-Small
Source-as-Output	18.5	50.6 / 23.2 / 23.1	47.6	11.9
SCPN (2018)	21.2	55.1 / 31.3 / 57.4	33.0	6.3
SGCP (2020)	7.9	34.7 / 13.7 / 36.9	17.9	12.5
GuiG (2020)	26.3	60.7 / 37.1 / 62.5	38.0	6.4
SynTrans	19.3	54.0 / 28.7 / 55.8	32.1	8.0
SI-SCP	27.8	62.8 / 39.5 / 64.4	39.9	5.7
- SeqEnc	27.4	61.4 / 37.4 / 63.2	39.1	5.8
- AttnRegu	26.8	61.6 / 38.9 / 63.7	38.5	6.0
- SibAttn	26.1	61.0 / 37.3 / 62.9	38.0	6.1
QQP-Pos
Source-as-Output	17.2	51.9 / 26.2 / 52.9	31.0	16.2
SCPN (2018)	28.1	59.1 / 36.5 / 62.1	33.1	8.3
SGCP (2020)	9.2	35.8 / 16.0 / 40.2	17.5	12.3
SynTrans	24.8	57.2 / 32.9 / 59.4	33.3	10.8
SI-SCP	53.5	77.3 / 61.0 / 78.8	54.5	5.2
- SeqEnc	53.1	76.9 / 60.5 / 78.6	54.4	5.3
- AttnRegu	31.4	61.6 / 38.5 / 63.5	38.0	9.6
- SibAttn	53.0	77.0 / 60.5 / 78.7	54.1	5.2
Table 1: Evaluation results on the ParaNMT-small and QQP-Pos datasets. All scores are reported as the mean over three runs. R-1: ROUGE-1. R-2: ROUGE-2. R-L: ROUGE-L. MTR: METEOR. '-SeqEnc': no sequential encoder used to encode leaf nodes, '-AttnReg': no attention regularization, '-SibAttn': no sibling attention module.
likelihood of the positive template:
$$
\begin{array}{l} \mathcal {L} \left(\boldsymbol {q} _ {i}, \boldsymbol {t} _ {i} ^ {+}, \boldsymbol {t} _ {i, 1} ^ {-}, \dots , \boldsymbol {t} _ {i, n} ^ {-}\right) \\ = - \log \frac {e ^ {s i m \left(\boldsymbol {q} _ {i} , \boldsymbol {t} _ {i} ^ {+}\right)}}{e ^ {s i m \left(\boldsymbol {q} _ {i} , \boldsymbol {t} _ {i} ^ {+}\right)} + \sum_ {j = 1} ^ {n} e ^ {s i m \left(\boldsymbol {q} _ {i} , \boldsymbol {t} _ {i , j} ^ {-}\right)}} \tag {9} \\ \end{array}
$$
At the training stage, we use the trick of in-batch negatives to learn the retriever.
In this section, we conducted experiments to answer the following questions:
- How does our model compare against previous models?
- Can our model produce diverse paraphrases with retrieved templates?
5.1 Controlled Paraphrase Generation
Implementation details are presented in Appendix A due to limited space.
Following previous work (Kumar et al., 2020), we used ParaNMT-Small and QQP-Pos datasets to
<sup>1</sup>We compared linearized and structured approaches and both have comparable performance, the main reason may be that controlled paraphrase generation requires learning relation between text and syntax (words and nodes), while the retriever learns relation between syntax. Based on the principle of simplicity, we used the linearized approach.
evaluate model performance for controlled paraphrase generation. ParaNMT-Small (Chen et al., 2019) contains 500k paraphrase pairs for training, 500 and 800 manually labeled paraphrase pairs for development set and test set. The ParaNMT-small is a subset of the original ParaNMT-50M dataset (Wieting and Gimpel, 2018) which is constructed automatically through back-translation of original English sentences. QQP-Pos (Kumar et al., 2020) is selected from Quora Question Pairs (QQP) dataset. It contains about 140K training pairs and 3K/3K data pairs for testing/validation.
- Source as output: Simply output the source sentence as output.
- SCPN (Iyyer et al., 2018) & GuiG (Li et al., 2020): They adopt a two-stage generation process. Iyyer et al. (2018) use two Bi-LSTM (Hochreiter and Schmidhuber, 1997) encoders to encode input sentences and linearized parse trees respectively. An LSTM decoder with attention mechanism pays attention to both semantic and syntactic hidden states to generate paraphrases. Li et al. (2020) use transformer network and propose a syntactic encoder with a path attention mechanism. GuiG only provided trained model on the ParaNMT-Small dataset.
- SGCP (Kumar et al., 2020): SGCP encodes syntactic templates with GRU network in a top-down manner. We directly used released SGCP model<sup>3</sup> and top-4 layers of the full parse tree of the reference paraphrases as syntactic templates.
- SynTrans: The framework is similar to SCPN, and we replace LSTM with Transformer (Vaswani et al., 2017). In this experiment, we directly used linearized syntactic templates to guide generation.
5.2.2 Automatic Evaluation
1. Semantic Metrics: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) scores between
Model	Semantic	Syntactic
SCPN	3.16	3.73
GuiG	3.67	3.77
SynTrans	3.05	3.51
SI-SCP	3.81	4.55
Table 2: Human evaluation results.
generated sentences and reference paraphrases in the test set were used as semantic metrics.
2. Syntactic Metrics: Following previous works (Chen et al., 2019; Kumar et al., 2020; Li et al., 2020), we used the tree edit distance (TED) against the parse tree of the reference.
5.2.3 Human Evaluation
We conducted the human evaluation on 100 randomly selected instances from the test set of ParaNMT-Small in a blind fashion. Three annotators evaluated generated paraphrases in terms of semantic and syntactic similarity (generations vs references); each aspect was scored from 1 to 5.
5.2.4 Results
As can be observed in Table 1, firstly, SGCP gets the lowest results, we observe that SGCP often produces sentences that end abruptly, thereby harming syntax and semantics<sup>4</sup>. GuiG achieves better performance than SCPN model among two-stage approach. This is because GuiG adopts the more advanced transformer network and a path-attention mechanism which can capture partial structural information. For SynTrans which uses linerized syntactic templates directly, it obtains relatively lower performance. Conversely, SI-SCP achieves significant improvements in both semantic and syntactic metrics, which means that modeling structural information can improve the quality of generations.
Among different variants of our model (ablation study), we see that removing any of the sequence encoder, attention regularization, and sibling attention module has a negative impact on performance, where attention regularization and sibling attention modules have a more significant impact.