Explicit Syntactic Guidance for Neural Text Generation
Year: 2023

Abstract:
Most existing text generation models follow the sequence-to-sequence paradigm. Generative Grammar suggests that humans generate natural language texts by learning language grammar. We propose a syntax-guided generation schema, which generates the sequence guided by a constituency parse tree in a topdown direction. The decoding process can be decomposed into two parts: (1) predicting the infilling texts for each constituent in the lexicalized syntax context given the source sentence; (2) mapping and expanding each constituent to construct the next-level syntax context. Accordingly, we propose a structural beam search method to find possible syntax structures hierarchically. Experiments on paraphrase generation and machine translation show that the proposed method outperforms autoregressive baselines, while also demonstrating effectiveness in terms of interpretability, controllability, and diversity.

heading: 1 Introduction
text: Natural language generation (NLG), such as paraphrase generation (Sun et al., 2021), text summarization (Lin et al., 2018), machine translation (Vaswani et al., 2017; Edunov et al., 2018), and language models (Brown et al., 2020; OpenAI, 2023), have shown remarkable progress in the past few years. Most of the highest-performing NLG models train the model based on source-target correspondence and conduct autoregressive inference, which achieves competitive empirical performances yet deviates from a range of desirable attributes of human language generation, e.g., lack of interpretability (Alvarez-Melis and Jaakkola, 2017; He et al., 2019; Li and Yao, 2021).
It has been shown that humans generate language by learning and manipulating language grammar (Zholkovskii and Mel’chuk, 1965; Montague, 1974), which generative grammar (Chomsky, 1965)
*Work was done during the internship at Tencent AI lab. †Corresponding authors.
considers as a finite rule set that combines words to form grammatical sentences, thereby avoiding enumeration of surface sequences, which can significantly increase data sparsity and reducing learning efficiency (Li et al., 2021; Dankers et al., 2022). In this process, syntax plays a crucial role, imposing constraints on how to construct sentences. Syntax knowledge has been found implicitly contained by deep neural models (Kovaleva et al., 2019; Clark et al., 2019) and also useful for NLG tasks (Yang et al., 2020a; Sun et al., 2021; Xie et al., 2021). However, relatively little recent work has considered explict syntax in NLG (Wang et al., 2018).
Inspired by the above psycholinguistic observation, we propose a syntax-guided generation scheme, which generates text by following a welldefined grammar. As shown in Figure 1, instead of sequential generation, the model generates the sentence in a hierarchically top-down manner guided by the constituency parse tree, starting with the root node <T>. Syntactic categories such as noun phrases <NP> and verb phrases <VP> are integrated with tokens in the generation process, and
14095
the model simultaneously considers multiple syntax structures at each tree depth, hierarchically exploring the syntax tree for reasonable hypotheses.
Intuitively, such a generation paradigm has the following advantages compared with autoregressive generation. First, akin to the language learning process of human beings, grammar learning breaks down non-enumerable surface sequences into finite pieces, acting as a training curriculum. Second, it provides an effective and interpretable pathway to probe into the generation process. Consequently, generation errors can be traced back to specific constituent expansion at the respective tree depth. Third, one can manipulate the generation process by exerting versatile control at arbitrary depths, e.g., modifying the translation of a verb phrase and constraining the paraphrase style with syntax templates. Forth, diverse sequences can be generated by exploring various syntax structures hierarchically throughout the syntax tree.
We implement the above process on Transformer (Vaswani et al., 2017). As shown in Figure 1, the generation process proceeds under the guidance of syntactic grammar. Starting from the root node “<T>”, the model recursively generates the infilling texts (e.g., “he” and ”seems <S>”) for each constituent in the current lexicalized syntax context (e.g, “<NP> <VP>.”.), and infills each one accordingly to construct the next-level lexicalized syntax context (e.g., “he seems <S>.”). The generation proceeds until there is no remaining constituent. The infilling texts are predicted by a Transformerbased model, which is trained by maximizing the likelihood of infilling texts for each constituent in the syntax context based on the source input. To explore more syntactically diverse and reasonable hypotheses during inference, we propose structural beam search, which searches promising syntax structures over the entire syntax tree in a top-down manner, as shown in Figure 1.
To isolate the effect of syntax and avoid the influence of other transformation factors, we conduct experiments on two sequence-to-sequence (seq2seq) tasks with semantic equivalence between the source and target sequences: paraphrase generation and machine translation. Empirical results demonstrate that our method can generate sequences with higher quality than the seq2seq baselines. Quantitative analysis demonstrates that the generation process can be interpreted effectively. In addition, our method demonstrates the ca-
pability of executing control from both syntax templates and fine-grained manual modifications. Finally, we show the diversity advantage through both automatic evaluation and human evaluation. We release the code on https://github.com/ yafuly/SyntacticGen.

heading: 2 Related Work
text: Syntax as Extra Input. A line of work incorporates syntax knowledge as extra input to boost task performance. In paraphrase generation, Iyyer et al. (2018), Chen et al. (2019), Kumar et al. (2020) and (Sun et al., 2021) additionally encode a constituency tree to produce controllable paraphrases. For machine translation, researchers utilize syntactic information to boost the neural machine translation system using syntactic encoders (Li et al., 2017; Ma et al., 2018; Eriguchi et al., 2019; Ma et al., 2020; Yang et al., 2020a), position encoding (Ma et al., 2019; Xie et al., 2021), attention mechanism (Chen et al., 2018; Peng et al., 2019), and auxiliary training objectives (Ma et al., 2019).
Syntax for Generation Guidance. Different from the above work, we focus on guiding generation explicitly following syntactic grammar. Typically, Aharoni and Goldberg (2017) and Le et al. (2017) learn the mapping from sequences to linearized constituency trees to improve machine translation. Eriguchi et al. (2017) proposes a hybrid decoder with RNNG (Dyer et al., 2016) to jointly learn parse actions and word predictions. Wu et al. (2017) and Wang et al. (2018) design a syntactic tree decoder based on LSTM (Hochreiter and Schmidhuber, 1997), with an extra rule decoder. Yang et al. (2020b) introduce a syntax-guided soft target template as extra prompts in Transformer. Different from their work, our method leverages Transformer strengths and breaks down the sequence-to-sequence generation process into a hierarchically top-down generation guided by the syntax tree.

heading: 3 Method

heading: 3.1 Baseline Transformer
text: Transformer models the correspondence between the source sequence x = {x1, . . . , x|x|} and the target sequence y = {y1, . . . , y|y|} in an end-to-end fashion. The Transformer encoder transforms the discrete source sequence x into a continuous representation, which the Transformer decoder utilizes
to generate the target sequence. The conditional probability p(y|x) can be factorized in an autoregressive way:
pθ(y|x) = |y|∏
t=1
pθ(yt|x, y1:t−1), (1)
where θ denotes the model parameters. Given a source-target training set D = {xi,yi}||D|i=1, the model is optimized by minimizing the cross-entropy (CE) loss:
LDce = − |D|∑
i=1
T∑
t=1
log pθ(y i t|xi, yi1:t−1). (2)

heading: 3.2 Syntax-guided Generation
text: In this section, we introduce syntax-guided generation, which generates texts by hierarchically expanding constituents in syntax contexts throughout the syntax tree, while also leveraging the strengths of Transformer. In general, the generation process can be decomposed into two stages: (1) neural generation: the neural decoder (Section 3.2.2) generates the infilling sequences based on the source sequence and the syntax context; (2) constituent expansion: predicted infilling sequences are mapped and filled into each constituent in the syntax context accordingly (Section 3.2.3), forming the next-level syntax context. To facilitate parallelism during training, we decompose the sequence-to-sequence dataset to a triplet set, where the neural decoder is optimized to maximize the probability of the infilled sequence (e.g., "<c> I <c> ate <NP> .") given the lexicalized syntax context (e.g., "<NP> <VP> ."), as shown in Figure 2.

heading: 3.2.1 Triplet Construction
text: Given a target sequence y, the corresponding constituency parse tree of depth |T| can be composed
by a set of labeled spans T:
T = {Td}||T|d=1 = {{(ak, bk, d, lk)}| |Td| k=1}| |T| d=1,
(3) where ak and bk represent the k-th constituent span’s fencepost positions at depth d, and lk represents the constituent label. Our model is optimized to predict the next-level span sets Td given the previous one and the source input, i.e., pθ(Td|Td−1,x).
Given the set of labeled spans at depth d, i.e., Td, we transform the target sequence into a lexicalized syntax sequence of length |sd|: sd = {sd;1, sd;2, . . . , sd;|sd|}, by keeping the lexical tokens and replacing the constituent spans with corresponding labels. For instance, the sequence “I ate an apple .” is transformed to s2 ={<NP>,<VP>,.} at depth 2, and is transformed to s3 ={I,ate,<NP>,.} at depth 3, as shown in Figure 2. The alignment between s2 and s3 can be modeled as a text-infilling task. For example, the {<NP>}, {<VP>} and at depth 2 are replaced by {I} and {ate <NP>} at depth 3, respectively. To generate the whole s3 based on s2 in one pass, we concatenate all the infilling texts with a special token “<c>”, yielding an infilling sequence f2 = {<c>,I,<c>,ate,<NP>}.
Similarly for each syntax context sd, we collect the respective infilling texts for each constituent in the lexicalized sequence at depth d+1, and concatenate them to construct the target infilling sequence of length |fd|: fd = {fd;1, fd;2, . . . , fd;|fd|}. In this way, a triplet is constructed for a source-target sequence pair at depth d: {(x, sd,fd)}. We traverse the target syntax tree in level-order to obtain the full set Φ of training triplets for a training instance:
Φ = {Φd}||T|−1d=1 = {(x, sd,fd)}| |T|−1 d=1 . (4)
Given a sequence-to-sequence training set D =
{xi,yi}||D|i=1, we go through the full training set to construct the complete triplet set Ψ:
Ψ = {Φi}||D|i=1 = {(xj , sj ,f j)}| ∑|D| i=1 |Φi| j=1 . (5)

heading: 3.2.2 Neural Decoder
text: Given a triplet instance Ψj , we construct the neural decoder based on Transformer to model the generative probability pθ(f j |xj , sj). The neural decoder takes the source sequence and the lexicalized syntax context as input and generates the corresponding infilling texts, as shown in Figure 2.
Besides the encoder that encodes source context, we introduce an extra Transformer encoder, i.e., syntax context encoder, to encode the lexicalized syntax context into a representation. On top of selfattention and source context attention, we insert an extra attention layer (syntax context attention) into each decoder layer to incorporate syntax contexts, as shown in the right part of Figure 2.
Similarly, the probability of the infilling sequence can be factorized as:
pθ(f |x, s) = |f |∏
t=1
pθ(ft|x, s, f1:t−1). (6)
We define the scoring function for an infilling sequence as the sum of the log probabilities:
score(x, s,f) =
|f |∑
t=1
log pθ(ft|x, s, f1:t−1). (7)
We adopt the standard cross-entropy loss (CE loss) to optimize our model, where the loss for the j-th triplet in the training set Ψ can be written as:
Ljce = − |fj |∑
t=1
log pθ(f j t |xj , sj , f j1:t−1), (8)
and the CE loss across the whole triple set Ψ becomes:
LΨce = |Ψ|∑
j=1
Ljce. (9)

heading: 3.2.3 Generation Process
text: Given a source sequence, our model generates the target sequence in a top-down manner which is grounded on syntactic grammar rules. As shown in Figure 2, the neural decoder first encodes the source sequence x into the source context representation hsrc, which remains fixed and can be reused
throughout the generation process. Initially, the neural decoder generates the infilling sequences t0 given x and s0 ={<T>}, based on Equation 6. Then the model proceeds with the generation process via iteratively generating infilling texts and expanding constituents.
At each iteration step (i.e., tree depth), the neural decoder generates the infilling sequence fd for the syntax context sd:
fd = argmax f ′
pθ(f ′|x, sd) (10)
Then the constituent expansion function yields the next-level syntax context given the syntax context and the infilling sequences predicted by the neural decoder:
sd+1 = expand(sd,fd). (11)
Specifically, we first separate the infilling sequences by the special separator “<c>” into a group of infilling texts, e.g., spliting f2 ={{<c>,I,<c>,ate,<NP>}} to {{I},{ate <NP>}}. Then we fill in each of the infilling texts into the corresponding constituent in the syntax context s2 to obtain the syntax context at the following level, e.g., s3={I,ate,<NP>,.}. The syntax context encoder encodes the updated syntax context sd+1 and starts the next iteration. The remaining decoding process loops between these two stages, until there is no constituent label in the syntax context, or a maximum tree depth is reached, as shown in Figure 2.
As the model behavior on expanding constituents over the entire syntax tree is completely accessible, the generation process can be effectively interpreted, as shown in Section 6.2. Moreover, manual modifications can be directly incorporated into the expansion process for each constituent throughout the syntax tree (Section 6.3). Finally, more than one syntax structure can be considered simultaneously at each tree depth, enabling searching for hypotheses of better syntactical diversity(Section 6.4).

heading: 3.2.4 Structural Beam Search
text: By default, our model selects the best infilling texts greedily in each iteration. We introduce structural beam search to explore the hypothesis space for a more accurate and diverse generation. Similar to standard beam search (Sutskever et al., 2014), structural beam search maintains a beam width of
candidates at each iteration. Thanks to explicitly traversing the constituency parse tree during inference, our method is able to search promising syntax structures throughout the syntax tree in a top-down manner. We show a real example of our model generating a paraphrase in Figure 3.
At each level, we apply standard beam search for neural generation and keep top k infilling texts along with their scores, computed by Equation 7. Taking previous predictions into consideration, we introduce a moving average mechanism to trade off confidence between the predictions from lower levels and the current-level prediction. Specifically, suppose si is the i-th syntax context in the k-width beam at the current depth, with an accumulated score of δsi ; and f j;si is the j-th infilling sequence candidate from the neural generation beam given the syntax context si, with a score of δfj;si . A beam of next-level syntax contexts is constructed, by filling in the current syntax context with the corresponding infilling sequences:
sik+j = expand(si,f j;si). (12)
The updated score for each of the next-level syntax contexts in the beam is given by:
δik+j = αδsi + (1− α)δfj;si , (13)
where α is a hyper-parameter (accumulation weight) that determines how much weight is put on predictions at lower levels. Then the beam is further pruned by their updated scores to maintain the beam width. For example, the first two candidate syntax contexts are selected at depth 2 in Figure 3. Algorithm implementation details can be referred to in Appendix A.

heading: 4 Experiment Setup
text: Datasets For paraphrase generation, we experiment on ParaNMT-small (Chen et al., 2019), which
contains 500K sentence-paraphrase pairs for training, 500 for validation, and 800 for testing. Both validation and test sets are provided with humanannotated sentence exemplars from which syntax information can be extracted for controlling paraphrase generation. For machine translation, we use NIST Chinese-English (Zh-En), WMT’16 Romanian-English (Ro-En), WMT’14 EnglishGerman (De-En), and WMT’14 English-German (En-De). For WMT datasets, we follow the official split for validation and testing. For NIST ZhEn, we use MT06 as the validation set and choose MT02, MT03, MT04, MT05, and MT08 as the test sets. For all datasets, we use Berkeley Parser (Kitaev and Klein, 2018; Kitaev et al., 2019) to obtain constituency parse trees and use the most frequent constituents (e.g., <NP>, <VP>, <PP> and <S>) for syntactic guidance.
Model Settings For Transformer baselines, we adopt the Transformer_Base configuration which consists of a 6-layer encoder and decoder. For our model, we keep the 6-layer source context encoder, and set the number of layers for both the syntax context encoder and the decoder as 3, resulting in a similar model size with Transformer_Base. The accumulation weight α is as 0.8 for structural beam search based on validation experiments. For machine translation, we adopt sequence-level distillation (Kim and Rush, 2016) for both our model and the corresponding baseline Transformer. More details are shown in Appendix B.
Evaluation We use the BLEU score (Papineni et al., 2002) to evaluate machine translation performance. For paraphrase generation, we also adopt ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005) as reference-based metrics. Besides,
we report iBLEU (Sun and Zhou, 2012):
iBLEU = r · BLEU(hypothesis, reference) −(1− r) · BLEU(hypothesis, source),
which evaluates the generation fidelity with novelty to the source sentence considered*. Following Bandel et al. (2022), we consider two referencefree metrics: (1) lexical diversity score, i.e., Dlex, which is the normalized character-level minima edit distance between the bag-of-words; and (2) syntax diversity score, i.e., Dsyn, which is the normalized tree edit distance. Both scores measure generated paraphrases with the source sequences unless specified.

heading: 5 Results
text: Paraphrase We compare our method with the baselines and previous work on syntax-control paraphrase generation. Another two baselines are also
*r is set as 0.7.
listed, i.e., copy the source input and use the reference as the output. The results are shown in Table 1. For paraphrase generation without syntax control (the center section in Table 1), our method achieves higher performance than the seq2seq Transformer, in both greedy and beam search settings. Typically, our method under greedy decoding obtains comparable results with the Transformer under beam search, and even outperforms under some metrics. The advantage of our method becomes larger for metrics such as iBLEU, Dlex, and Dsyn, which consider generation novelty compared with the source input. For example, compared with Transformer (beam 5), our method (beam 5) gives a much lower self-BLEU score (16.4 v.s. 33.8) and higher diversity scores (21.5 v.s. 16.2 for lexical diversity and 25.1 v.s. 18.1 for syntax diversity), indicating better generation diversity and contributing to a significant improvement on iBLEU (8.6 v.s. 2.2). With annotated exemplars (the lower section in Table 1), our model obtains further improvement over the non-exemplar setting and achieves better performance compared to previous work which utilizes full syntactic parse.
We extend our method to the pre-trained language model (PLM) setting and present the result in Table 3 (Details in Appendix A). It can be seen from the table that the utilization of BART (Lewis
et al., 2019) improves the generation diversity for the sequence-to-sequence model significantly. Despite the narrowed gap, our model outperforms the seq2seq counterpart in terms of iBLEU and lexical diversity by a considerable margin.
Machine Translation As shown in Table 2, our method achieves consistent performance (BLEU score) improvement over the Transformer baseline. The improvement is larger for the greedy setting (+1.5 BLEU scores on average), compared with the beam search setting (+1.2). This indicates that using syntax to guide and constrain generation yields more reasonable and high-quality hypotheses than the greedy autoregressive generation, and thus relies less on search algorithms (e.g., beam search). Note that compared with the Englishoriented datasets, our model obtains a smaller performance improvement on WMT’14 En-De. This can be because the German parser is less accurate than the English one (92.1 v.s. 96.3 for F1 score), resulting in a training set with lower quality.

heading: 6 Analysis
text: We first discuss the influence of grammar quality, then we understand the potential advantages of our method from three perspectives, i.e., interpretability, controllability, and diversity.

heading: 6.1 The Influence of Grammar Quality
text: Intuitively, learning syntactic grammar of higher quality results in better generation performance, e.g., the advantage of our method on Englishoriented datasets is larger than the German-oriented one. To further explore the influence of grammar quality, we randomly replace a certain ratio of the constituent labels with a random one to simulate a less accurate parser. We conduct experiments on the WMT’16 Ro-En dataset. By injecting noise of ratios of 0.2 and 0.4, the model performance deteriorates from 34.9 to 34.6 and 32.3 accordingly, indicating the quality of syntactic grammar exerts a large influence on model’s generation performance.

heading: 6.2 Interpretability
text: We evaluate the model’s interpretability based on its capability of providing explanations in understandable terms to a human (Doshi-Velez and Kim, 2017), i.e., whether it generates texts following language grammar. We trace each constituent expansion during generation and compare the modelinduced tree with the tree parsed by a benchmark
parser, e.g., Berkeley Parser. Specifically, we use the Berkeley parser to parse the same generated hypotheses by our model and treat the corresponding parsing results as golden parses. Quantitative results (Figure 4) show that our model achieves an average F1 score of 94.6 , which demonstrates the generation process highly corresponds to the syntactic grammar and thus can be effectively interpreted. Note that the score for WMT’14 En-De is lower (89.0), possibly due to the less accurate German parser for constructing the syntactic grammar, as discussed in Section 6.1.

heading: 6.3 Controllability
text: Control with Complete Syntax Template To leverage control signals from delexicalized syntax templates (e.g., “(S (NP) (VP (NP)))” for the sequence “I ate an apple.”), we introduce a reward γ into Equation 13:
δik+j = αδsi + (1− α)δfj;si + γ. (14)
If the updated syntax context sik+j matches the corresponding template pattern at depth d+ 1, the γ is a positive value otherwise 0. For example, the syntax context “<NP> <VP>” in Figure 3 matches the pattern “((NP)(VP))” at depth 2. Intuitively, the reward encourages the model to favor beam candidates that match the syntax template. We set the reward value as 0.32 based on validation results
(Appendix F). The testset of ParaNMT-small is provided with human-annotated exemplars and we use it to control generation, with results shown in Table 1. More generally, golden templates can be derived by parsing the reference sentences for each dataset with a parser (e.g., the Berkeley Parser). We present the results in Table 5. Guided by the reference syntax template, our model obtains consistent improvement in terms of hypothesis similarity with references, which is reflected by the decreased syntax edit distance to the references, i.e., Drefsyn. For the multi-reference dataset NIST Zh-En, our model can generate translations of different styles which are prompted by alternative syntax templates from multiple references.
Control with Partial Syntax Template We further explore whether the model can handle finegrained arbitrary controls. Specifically, we ask three annotators to modify the intermediate syntax contexts output by the model, based on the source input. 100 instances are randomly selected from the NIST Zh-En test set and each annotator gives different modifications for each instance. The modified contexts are fed to the model to predict the infilling texts. We then ask the annotators to evaluate whether their controls (i.e., modifications) are safely responded to by the model. We show some of the control examples in Appendix G. The average control success rate is 81%, which demonstrates the capability of our model to handle arbitrary fine-grained controls.

heading: 6.4 Diversity
text: Beam Diversity We expect the model to generate diverse hypotheses under beam search, while also maintaining generation quality. To this end, we measure the model’s beam diversity by computing two average scores: (1) the average of the mutual diversity scores of every two of the beam candidates, i.e., Dbeamlex and D beam syn ; (2) the average generation quality of the beam candidates, measured by BLEU scores. The results for paraphrase generation are shown in Table 6. In terms of generation quality, our model generates consistently better beam candidates on average than the baseline model. Besides, we can see that structural beam search can yield more diverse beam candidates, indicated by the higher mutual diversity (i.e., Dbeamlex and Dbeamsyn ) among beam candidates.
Effects of Accumulation Weight A larger accumulation weight (α in Eq. 13) indicates a larger
weight on previous decisions when re-ranking the newly updated beam candidates. As a result, early determined syntax structures are less likely to be surpassed throughout the whole structural beam search. On the contrary, a smaller α encourages the model to explore promising candidates at higher levels, and can therefore find more diverse hypotheses. We explore the effects of α with results shown in Figure 4. As the weight grows smaller, the model generates sequences of better syntactic diversity, i.e., Dsyn. However, an overly small weight deteriorates generation quality (iBLEU), which can be caused by the model’s overconfidence in local predictions without considering the predictions of syntax contexts at lower levels. Such deterioration is also seen for overly large weights (>0.95), due to limited exploration at higher levels.
Human Evaluation We further conduct a human evaluation to evaluate generation quality and diversity on paraphrase generation. We ask three annotators to vote for one of the two candidates: hypotheses from the seq2seq baseline and our method. The annotators are required to decide, which one is better by considering Fidelity, Novelty, and Diversity (See Appendix H for details). The results are shown in Table 7. As can be seen from the table, our method achieves much better generation novelty and beam diversity compared with the baseline, while maintaining semantic fidelity, which further
validates the results of the automatic evaluation.

heading: 7 Conclusion
text: We proposed a syntax-guided generation paradigm, which leverages the strengths of Transformer and generates sequences by hierarchically expanding constituents in the lexicalized syntax contexts throughout the syntax tree. The neural decoder was trained by maximizing the likelihood of the infilling texts for each constituent in the syntax contexts given the source sequence. Moreover, we proposed the structural beam search to better explore the hypothesis space. Empirical results demonstrated the advantage of generation quality over the seq2seq baseline, and also the effectiveness in terms of interpretability, controllability, and diversity.
Our method can be seen as a step towards explicit modelling of psycholinguistic structures during neural text generation , helping the model to have a degree of control over what it intends to generate, which can potentially address salient issues of current neural NLG, such as hallucination (Guerreiro et al., 2023; Dziri et al., 2022) and ethical issues (Sheng et al., 2019, 2021; Weidinger et al., 2021), if semantics, pragmatics, and other factors are also integrated.
Limitations
Despite the competitive performance, there are several limitations of this work: (1) As discussed in Section 6.1, the generation performance relies on the parser performance, which is strong enough for English but still less satisfactory for other languages. Dedicated methods need to be considered to compensate for the weak parser performance if we want to extend our method to more languages. (2) In this work, we consider two NLG tasks with semantic equivalence to testify if the proposed method can convey the source semantics accurately by following the target syntactic grammar. Other tasks such as summarization and dialogue generation can also be tested, where the semantics are not equivalent between the source and target. (3) To train the neural decoder parallelly, we break down the source-target dataset into a triple set. However,
the global dependency of the syntax parse tree is not considered, which can deteriorate generation performance. (4) Due to the recursive encoding of the syntax contexts, our model’s inference speed is approximately half that of the seq2seq counterpart (Appendix E). (5) Future work should include experiments on large language models (Brown et al., 2020; OpenAI, 2023; Zeng et al., 2022; Touvron et al., 2023; Taori et al., 2023). to further demonstrate the effectiveness of our method beyond pretrained language models.
Ethics Statement
We honor the ACL Code of Ethics. No private data or non-public information is used in this work. For human annotation (Section 6.3 and Section 6.4), we recruited our annotators from the linguistics departments of local universities through public advertisement with a specified pay rate. All of our annotators are senior undergraduate students or graduate students in linguistic majors who took this annotation as a part-time job. We pay them 60 CNY an hour. The local minimum salary in the year 2022 is 25.3 CNY per hour for part-time jobs. The annotation does not involve any personally sensitive information. The annotated is required to rank the system output and label factual information (i.e., syntactic annotation).

heading: Acknowledgement
text: We would like to thank all reviewers for their insightful comments and suggestions to help improve the paper. We thank Deng Cai and Xinting Huang for their insightful suggestions. This work is funded by the Ministry of Science and Technology of China (grant No. 2022YFE020038).

heading: A Algorithms
text: The scoring algorithm 7 can be rewritten with the source context x encoded into hsrc:
score(hsrc, s,f) =
|f |∑
t=0
logpθ(ft|hsrc, s, f1:t−1)
(15) The algorithm of structural beam search is demonstrated in Algorithm 1, which employs the standard beam search for autoregressive generation, depicted in Algorithm 2. The termination function in Algorithm 1 (i.e., terminated(·)) returns true if the there is no remaining constituent in the input sequence.

heading: B Experiment Details
text: For NIST Zh-En, we use parts of the bitext provided within NIST’12 OpenMT† and the final train set consists of about 1.8M sentence pairs. We apply BPE (Sennrich et al., 2016) on all datasets: the number of BPE operations is 6K for ParaNMTsmall, and 40K for the other datasets. We implement our model using Fairseq (Ott et al., 2019).
†LDC2005T06, LDC2004T07, LDC2003E07, LDC2000T46, LDC2000T47, LDC2000T50, LDC2003E14, LDC2005T10, LDC2002E18, LDC2007T09, LDC2004T08
Algorithm 2 Beam search Setup: k: beam size
tmax: maximum hypothesis length V: target tokens set score(·, ·, ·): scoring function (Eq. 15)
Input: s: syntax context hsrc: source context representations
1: t← 0 2: B0 ← {(0, ⟨bos⟩)} 3: while t < tmax do 4: B ← ∅ 5: for (δ,f) ∈ Bt−1 do 6: if f .last() = ⟨eos⟩ then 7: B.add((δ,f)) 8: continue 9: end if
10: for f ∈ V do 11: δ ← score(hsrc, s,f ◦ f) 12: B.add((δ,f)) 13: end for 14: end for 15: Bt ← B.top(k) 16: t← t+ 1 17: end while 18: return Btmax
We train the model using Adam (Kingma and Ba, 2015) optimizer. The learning rate increases to 7 · 10−4 in the first 10K steps and then anneals exponentially. We set the weight decay as 0.01 and label smoothing as 0.1. The dropout is 0.3 for ParaNMT-small, and 0.1 for the other datasets. The batch size is 64K tokens for ParaNMT-small, 256K for WMT’16 Ro-En and NIST Zh-En, and 512K for WMT’14 De↔En. All models are trained for a maximum update of 300K steps unless early stopped. We train the model using 4 V100s and increase gradient accumulation steps for large batch sizes. We choose the 5 best checkpoints based on validation sets and average them for inference. We set the beam width as 5 for beam search. For machine translation, the teacher models for knowledge distillation are Transformer_Base for NIST Zh-En and WMT’16 Ro-en, and Transformer_Big for WMT’14 De↔En.

heading: C Model Architecture
text: We conduct experiments to compare different model architectures to incorporate syntax context on the WMT’16 Ro-En validation set. We consider the following settings:
• Concat: concatenate the syntax context with the source sequence, with the vanilla Transformer unmodified.
• Extra-attention: reuse the source encoder for encoding syntax context and insert an extra at-
tention layer, i.e., the syntax context attention, into each decoder layer.
• Extra-encoder: introduce an additional encoder for encoding syntax context and also uses the syntax context attention.
Empirical results are shown in Table 9. Based on validation results, we adopt the Extra-encoder model in all experiments except for training on BART (Table 3), where we adopt the Concat model.

heading: D Experiments on PLM
text: In this section, we introduce our experiment settings of PLM. Following previous work (Sun et al., 2021), we use BART-base (Lewis et al., 2019) as our base model. All models are finetuned for 10 epochs with a batch size of 64k tokens. The learning rate is 3e-5 and the linear decay schedule, as recommended in BART’s official repository‡.
We use the Concat (Appendix C) model architecture for extending our method to BART. The source text and the syntax context are concatenated with a special token “<sep>”, e.g., “I ate an apple . <sep> <NP> <VP> .”. To effectively employ our method with BART, whose inputs are tokenized sequences byte-level, as same as Radford et al., we make several modifications. In the pre-processing, we make sure our special tokens (e.g., <sep>, <c>, <NP>, <VP>) are not split and add extra byte-level spaces before and after the special token. Thanks to the unused tokens in BART embeddings, we do not need to modify the embedding matrix. Instead, we assign our special tokens to unused token indexes.
‡https://github.com/facebookresearch/ fairseq/tree/main/examples/bart
14.5
15.5
16.5
17.5
18.5
19.5
31.5
32.0
32.5
33.0
33.5
34.0
34.5
0.0 0 0.0 1 0.0 2 0.0 4 0.0 8 0.1 6 0.3 2 0.6 4 1.2 8 2.5 6
D _s yn BL EU
Reward
BLEU D_syn
Figure 5: Effects of reward ratio on the WMT14’De-En validation set.
Finally, in the inference stage, we find the constituency expansion causes a discrepancy between inputs of train and test. Thus, we first detokenize each layer’s outputs and then tokenize them back with the same procedure in the preprocessing to avoid such a gap.

heading: E Generating Linearized Trees Directly
text: A baseline method to induce grammar simultaneously during generation is generating linearized parse trees directly, i.e., training a seq2seq model which takes in source sequences and outputs linearized parse trees. We compare it with our method on WMT’16 Ro-En. Specifically, the BLEU score for WMT’16 Ro-En is only 27.6 compared to the seq2seq baseline (34.1) and our method (34.9). This can be because the additional parentheses and constituency tags in linearized trees may deteriorate sequence coherence, making learning more difficult. Our method, on the other hand, breaks down syntax trees into level pieces to create a better learning curriculum. Furthermore, Generating linearized parse trees is much slower than the seq2seq counterpart, since the average sequence length of linearized tree sequences is longer (152.3 vs 28.4). As a result, the average speed for generating linearized parse trees is only 0.8 sentences/s compared to 3.6 sentences/s for the seq2seq baseline. Our method achieves an inference speed of 1.7 sentences/s under the same computing condition (V100 GPU). Additionally, generating a linearized parse tree is not easily interpretable or controllable, due to the black-box nature of the sequence-to-
sequence paradigm.

heading: F Effects of Control Reward
text: The magnitude of the reward γ determines how much priority is given to beam candidates that match the syntax exemplar. We experiment with different reward values to give a quantitative demonstration, shown in Figure 5. It can be seen that the control effectiveness grows with the increase of the reward value until 0.64, which suggests that all possible matched beam candidates are re-ranked to the top in the search space.

heading: G Control with Partial Syntax Template
text: We present 3 sample cases to demonstrate finegrained controls over the generation process, shown in Figure 6. Each Chinese source sentence is paired with 3 manual controls from three annotators. The model takes in the annotated syntax context and proceeds to obtain the respective translations.

heading: H Human Evaluation for Paraphrase Generation
text: We ask three annotators to conduct side-by-side human evaluations and report averaged results of their annotations. For each instance, the annotators vote for one of the two outputs by the baseline
and our model. The outputs contain top-5 beam candidates under beam search. The annotators are asked to evaluate both the best candidate and the beam results as a whole, based on the following three aspects:
• Fidelity: Whether the best candidate is semantics-equivalent with the input.
• Novelty: Whether the best candidate modifies the input sentence structure.
• Diversity: Whether the generated five candidates are different from each other given the input.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.
C 3 Did you run computational experiments? Section 4 & 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B & C.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4; Appendix B & C.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6.4 & 6.4.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix G & H.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section Ethics Consideration.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix G & H.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section Ethics Consideration.

heading: Future Work
text: (5) Future work should include experiments on large language models (Brown et al. , 2020; OpenAI, 2023; Zeng et al. , 2022; Touvron et al. , 2023; Taori et al. , 2023). to further demonstrate the effectiveness of our method beyond pretrained language models