Non-autoregressive Machine Translation with Probabilistic Context-free Grammar
Authors: Shangtong Gui
Chenze Shao
Zhengrui Ma
Xishan Zhang
Yunji Chen
Yang Feng
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine translation
Non-autoregressive generation
Probabilistic Context-free Grammar
TL;DR: We enhance NAT with PCFG to capture the dependency among output tokens in a hierarchical manner, which improves the translation quality and interpretability of NAT.

Abstract: Non-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages a specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates a deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation. Code is publicly available at https://github.com/ictnlp/PCFG-NAT.

Non-autoregressive Machine Translation with Probabilistic Context-free Grammar
Shangtong Gui Chenze Shao Zhengrui 2. Xishan Zhang Yunji Chen Yang Feng
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences Cambbricon Technologies
Abstract
con-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation!"
Introduction
Autoregressive machine translation models rely on sequential generation, leading to slow generation speeds and potential errors due the cascading nature of the process. In contrast, non-autoregressive Transformer (NAT) models [13] have emerged as promising alternatives for machine translation. The Vanilla NAT model assumes that the target tokens are independent of each other given the source sentence. This assumption enables the parallel generation of all tokens. However, it neglects the contextual dependencies among target language words, resulting a severe multi-modality problem caused by the rich expressiveness and diversity inherent in natural languages. Consequently, Vanilla NAT models exhibit significant performance degradation compared their autoregressive counterparts.
13
Several approaches have been proposed alleviate the multi-modality problem in NAT models while maintaining their computational efficiency. CTC-based NAT [1] (22) depicted in Figure Tb)
addresses this issue by implicitly modeling linear adjacent dependencies through token repetition and blank probability. This approach helps alleviate the each position compared to Vanilla NAT. Similarly, DA-Transformer [14] 37 26) shown Figure [Tc) introduces a hidden Markov model and employs directed acyclic graph to model sentence probability, enabling the model to capture dependencies between adjacent tokens and reduce the multi-modal nature of the distribution.
of token distributions at
"Corresponding author: Yang Feng "Code is publicly available
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

APD
GOIYT
bigger
GOTTA
naed
(a) Vanilla NAT
(b) CTC-NAT
10
You
arc
Reed
one
bigger
boot
You
will
are
gonna
need
one
bigger
best
(c) DA-Transfomee Transformer
(d) PCFG-NAT
Figure 1: Illustration of Vanilla NAT, CTC-NAT, DA MA-Transformer and PCFG-NAT
However, despite these enhancements, the modeling of only adjacent and unidirectional dependencies limits the model's 's ability to capture the rich semantic structures with non non-adjacent and bidirectional dependencies present in natural language. For example, considering the token need as the previous token, the distribution of the next token is still affected by the issue of multi-modality, as the word boat can be modified by various rich prefixes the corpus.
To overcome this constraints, we propose PCFG-NAT, a method that leverages context-free grammar endow the model with the capability capture the intricate syntactic and semantic structures present in natural language. For instance, as illustrated in Figure d)), PCFG-NAT can directly model the robust and deterministic correlation between non-adjacent tokens, such You need boat. In the parse tree generated by PCFG-NAT for the target sentence, the nodes connected to the tokens You need boat constitute the main chain of the tree, capturing the left-to-right semantic dependencies through linear connections. Additionally, PCFG-NAT can learn relationships between modifiers, such as are gonna and bigger, and the words they modify by constructing a local prefix tree for the nodes on the main chain. This allows for the modeling of right-to-left dependencies between the modified words and their modifiers.
We conduct experiments on major WMT benchmarks for NAT (WMT14 En++De, WMT17 Zh+=En, W En++Ro), which shows that our method substantially improves the translation performance and achieves comparable performance to autoregressive Transformer 40) with only one-iteration par- allel decoding. Moreover, PCFG-NAT allows for the generation of sentences more interpretable manner, thus bridging the gap between performance and explainability in neural machine translation.
2 Background
In this section, we will present the necessary background information to facilitate readers' com- prehension of the motivation and theoretical underpinnings of PCFG-NAT. In Section 2.1] we will provide formalization of the machine translation task and introduce the concepts of autoregressive Transformer (40) and Vanilla NAT [13]. which will serve as the basis for our proposed approach. Subsequently, in Section 2.2 will provide precise and rigorous definition of PCFG, accompanied by illustrative examples aimed at enhancing clarity and comprehension.
2.1 Non-autoregressive Translation
Machine translation can be formally defined eequence-to-sequenee generation problem. Given source sentence X = { T2, Fs. the translation model is required to generate target sentence Y {U1, 3/ ..., }. ). The autoregressive translation model predicts each token the condition of all previous tokens:
T P(Y = II p(u X).
(I)
1=1

---

Applied Rules
Sequence
Start
Vi
Vi
You V5.Vo
You
V.
Vi need Vo
You Va need V.
V3
Voure Vo.Vo +>
are V2 need Vg
14
You gonna need Vú
16
1G boat Vio
You are gonna need V; boat Vio
14 V6a Va.Vo -
are gonna need a boat Vi0 You gunna need higger hoat V10 You are gonna need a bigger boat
V.
bigger
Viu
Figure 2: An example parse tree for You are
Table I: Derivation of the parse tree. Used produc- tion rules for every step are listed, where stands for an empty string.
gonna need bigger boat.
The autoregressive inference is tine-consuming as it requires generating target tokens from left to right. To reduce the decoding latency, Gu al. [13 proposes non-autoregressive Translation, which discards dependency among target tokens and predicts all the tokens independently:
P(Y = II p(u
Since the prediction does not rely previous translation history, the decoder inputs of NAT purely positional embeddings or the copy of source embeddings. Vanilla NAT trains a length predictor to determine the sentence length when decoding. During the training, the golden target length is used. During the inference, the predictor predicts the target length and the translation of the given length is obtained by argmax decoding.
2.2 Probabilistic Context-free Grammar
Probabilistic Context-free Grammar is built upon Context-Free Grammar (CFG). A CFG is defined as 4-tuple:
G (N, R),
where N is the set of non-terminal symbols, T is the set of terminal symbols, R. is the set of production rules and S is the start symbol. PCFG extends CFG by associating each production rule r E with probability P(r). Production rules are in the form:
a.
where must be a single non-terminal symbol, and a is string of terminals and/or nonterminals, also be empty. Starting with the sequence that only has start symbol S, replace any non-terminal symbol A with a based on production rule + a. The derivation process repeats until the sequence only contains terminal symbols and the derived hierarchical structure can be seen Parse Tree. Those production rules and their probabilities can be applied regardless of the contexts, which provides Markov property for the joint probability of the parse tree. For every rule, A applied in derivation, all of the symbols in a are the children nodes of A in order. The probability of parse tree t formulated as:
P(t) II (P(r))", rER(L)
where R(() is the set of production rules applied in and n, is the number of times that r is applied. For sentence You are gonna need a bigger boat., Figure Souureessorr shows an example parse tree and how the sentence is derived from the start symbol V1.
3 The Proposed Method
In the following sections, we present the formal definition of novel variant of PCFG designed for PCFG-NAT in Section and the architecture of the PCFG-NAT model in Section 3.2 Finally, we
describe the training algorithm and decoding algorithm for PCFG-NAT Section 3.3 and Section
3.4

---

3.1 Right Heavy PCEG
Despite the strong expressive capabilities of PCFGs, the parameter induction process becomes challenging due to the vast latent variable space encompassing the possible parse tree structures of single sentence. Moreover, training PCFGs incurs significant time complexity [35] To address these challenges, we propose novel variant of PCFG called Right-Heavy PCFG (RH-PCFG). RH- PCFG strikes balance between expressive capabilities and computational complexity, mitigating the difficulties associated with parameter induction and reducing training time.
3.1.1 Derived Parse Tree of RH-PCFG
RH-PCFG enforces distinct right-heavy binary tree structure in the generated parse tree. Specifically, for each non-terminal symbol in the tree, the height of the right subtree increases proportionally with the length of the derived sub-string of the non-terminal symbol, while the height of the left subtree remains constant. This characteristic significantly reduces the number of possible parse trees for given sentence, thereby reducing training complexity.
In the context of a right-heavy binary tree, the traversal process begins with the root node and proceeds recursively by selecting the right child of the current node as the subsequent node visit. This sequential path, known as the Main Chain, is guided by design principle inspired by linguistic observations in natural language. Specifically, reflects the tendency for sentence structures to display left-to-right connectivity, with local complements predominantly positioned to the left of the central core structure. Within this framework, the left subtree of each node along the main chain is referred to as its Local Prefix Tree. The parse tree in Figure @rrreesenttst right heavy tree. In this tree, Vi, Vo, Vg, V13 form the main chain, and their left subtrees correspond their respective local prefix trees.
3.1.2 Support Tree of RH-PCFG
To make sure that all the parse trees are the right heavy tree, we first build up Support Tree the backbone for the parse trees derived from RH-PCFG. The construction of the support tree follows two steps: At first, given source sentence with length of Lg and upsample ratio of X, sequentially connect AL + nodes with right links, which become the candidates for the main chain nodes in parse trees. Next, excluding the root node, we append complete binary tree of depth I as the left subtree to each node. For the root node, only one left child is added. 'hee added nodes represent local prefix tree candidates the attached main chain candidates. Consequently, for source sentence of length Ly, the number of non-terminal symbols m of the RH-PCFG is
m Alz 2 + 2.
(5)
The nodes within the support tree assigned sequential indexes through in-order traversal, and these indexes correspond to the corresponding non-terminal symbols sharing the same index. Detailed construction algorithm can be found in Appendix A.I Figure 3 shows example of PCFG parse tree with L 3, 1.
Main Chain Candidates
Local Prefix Tree Candidates
Figure 3: An example support tree of Right Heavy PCFG
In RH-PCFG, the set of non-terminals is N = (V,, .... Vn-1) and the set of terminals T equivalent to the vocabulary. The collection of production rules is represented by R, and the starting symbol is Vi. The production rules RH-PCFG have the following form:
VO
(6)
V, a T,i 0, IsLeaf(1)
(7)
V, + V.VV E T, (
(Iwneeeteeahhhhe vjj
= 0) A InRightReach(!:.))
(8)

---

stands for an empty string. [sLeaf(i) indicates whether the non-terminal symbol V; corresponds leaf node in the support tree. InLeftReach(j. denotes that, in the support tree, the node corresponding to Vj is in the left subtree of the node corresponding to V, InRightReach(A, signifies that the node corresponding to Vj in the right subtree of the node corresponding to V; in the support tree, and if V, corresponds to main chain node, V2 has to correspond to main chain node at the same time.
3.2 Architecture
The main architecture of PCFG-NAT is identical to that of the Transformer 44), with the additional incorporation structures to model the probabilities of the PCFG. The input sentence to be translated is fed into the encoder of the model. Additionally, based the length Lg of the source sentence, we compute the number of corresponding non-terminal symbols m in the RH-PCFG. Then m positional embeddings are used as inputs to the decoder. Through multiple layers of self-attention modules with a global attention view, cross-attention modules that focus on distant information, and subsequent feed-forward transformations, states are associated with non-terminal symbols of the same index and are utilized to calculate the rule probabilities related to each corresponding non-terminal symbol.
obtain hidden states ho: hm-1- These hidden
Decoder Ourpu:
Attention 7ron Knooder Outgut
Linear Softmax
Oucpuz Tayer
Right Reavy PCPG
Stacked Encoder Layers
Stacked Decoder Layers
Fost:tional Enoediting
Figure 4: The architecture of PCFG-NAT model. PCFG is inserted between the decoder layers and the output layer, which captures the dependency between different tokens. parse is derived to generate the sentence You are gonna need a bigger boat. and is highlighted with bold black lines.
The probability of the rule P(V; a) can be regarded as the word prediction probability each position, and we can directly calculate it with the softmax operation. We use h; to denote the hidden state representation of the non-terminal symbol V, and use W, to denote the weight matrix of the output layer. Formally, we have:
P(V a) Softmax oh.).
For the rule P(V: + VjaV)), we can independently consider the prediction probability of word and the transition probability between the three non-terminal symbols. Formally, decompose it into two parts:
P(V + ViaVi. P(< V. P(alV).
(10)
The prediction probability P(a] VV,) can be calculated with the softmax operation like in Equation 9 To normalize the distribution of P(< < V,, V2. [V.), we have to enumerate all valid combinations of k. Let (i) {< o, VV, + V.VV, E R} denotes the set of children combinations of the non-terminal symbol Vú, we can formally define the transition probability as:
q. Wyhs,
9 Wihg, 9b. W, expla 9j + oo gx 17 9t)
(11)
P V V.
(12)
Σ Ecausechio exply" 4o + of 9n + 46 qn))
where Wg. Wi, W, are learnable model parameters.
5

---

3.3 Training
3.3.1 CYK Algorithm for RH-PCFG
We train PCFG-NAT with the maximum likelihood estimation, which sums up all the possible parse trees for the given sentence Y conditioned source sentence X and model parameters 0:
( = - log P(Y = log P(V yo3/1..un-1).
(13)
In PCFG, the CYK algorithm [35]
is employed to compute the probability of given sentence derived from the start symbol. The CYK algorithm follows bottom-up dynamic programming approach to fill in the table S. Each element Si in the table represents the sum of probabilities of that Va derives the consecutive sub-string ys....jj of the target sentence, denoted as P(va = yy...yz). Therefore the size of table S is (m Based on the definition of Si we have recursion formula
j-1
Σ E
P(V. 1
(14)
k=i+1 <b.c>EChild())
For every non-terminal symbol Va. the complexity filling all the Si is the product of the choice of span j.j 2, splitting point k. and children symbols b,
Local Prefix Tree For the non-terminal symbol V, in a local prefix tree, the rules specify that all values of b and are strictly fewer than the size of the left attached tree in the support structure,
denoted as 2 - 1 < d = 2'. Similarly, the variables i, j, and k are also strictly limited to values fewer than d, since it the max length of sub-strings that Va can derives. Please note that represents the max depth of the complete binary tree added as the left subtree to each node. Therefore the complexity filling all the S is Odd) for Va and all Va in Local Prefix Tree should be (dd m) where m is the number of non-terminal symbols.
Main Chain For the non-terminal symbol V, that serves main chain, based the rules of the model, the variable has strictly fewer than d choices, while e has maximum of m/d choices. In right-heavy parse tree, the symbols in the main chain are responsible for deriving a complete suffix of the target sentence as they are the right offspring of the root. Consequently, is always equal to and variable i has choices. However, k has only d choices since the the length of derived sub-string of Vo is less than d. Considering the filling of all S entries for the main chain symbols, the time complexity is ((mdn). Summing up the complexities for all m/ d main chain symbols, the total complexity is Omm?
Based on the above analysis, the overall time complexity of the CYK algorithm for RH-PCFG is O(d"n + m m2n). Such complexity is within the acceptable range, while the training complexity of general form PCFG, such as Chomsky normal form [5], is ( Detailed pseudocode for the training algorithm can be found in the Appendix A.2
3.3.2 Glancing Training
Glancing training, as introduced in the work of Qian et al. 330. has been successfully applied in various NAT architectures 12 14 In the case of PCFG-NAT, we also incorporate glancing training. This training approach involves replacing portion of the decoder inputs with the embeddings of the target sentence Y. To determine the ratio of decoder inputs be replaced with target embeddings, we employ the masking strategy proposed by Qian et al. [30]. There is mismatch between the decoder length m and target length ni, so we have to derive an alignment between them to enable the glancing. To incorporate Glancing Training into PCFG-NAT, we adapt the approach by finding the parse tree T = arg (aaXyve((Y)) P(Y,TY Ty X)) with the highest probability that generates the target sentence Y. where r(Y) stands for all parse trees of Y. Tу can traced by repalcing the sum opearation with max in the CYK algorithm. We provide the pseudocode in the Appendix A.3
3.4 Inference
We propose a Viterbi 41 decoding framework for PCFG-NAT to find the optimal parse tree T* arg maxy P(T]X,L) under the constraint of target length L. For each non-terminal symbol Va,

---

maintain a record Mł that represents the highest probability among sub-strings of length that can be derived by V Similar the CYK algorithm, we can compute Mł bottom-up manner using dynamic programming, considering all non-terminal symbols Va and lengths L.
MI
max <b.0> EChild (Cheldllh......T... ()).))). P(V. VooV))M21--4M.
(15)
To recover the parse tree with the maximum probability given L, we need to keep track the values of b, c, and & that lead Mr and start the construction from Mt. The detailed pseudocode is provided in the Appendix [A.4] According to Shao et al. [37], we rerank the output sentences with probability and length factors and then output the sentence with the highest score.
4 Experiments
4.1 Setup
Dataset We conduct our experiments on WMT14 English--German (En-De, 4.5M sentence pairs), WMT17 Chinese+>English (Zh-En, 20M sentence pairs) and WMT16 English+-Romanian (En-Ro, 610k sentence pairs). We use the same preprocessed data and train/dev/test splits as Kasai et al. [18]. For all the datasets, we learn joint BPE model [36] with 32K merge operations to tokenize the data and share the vocabulary for source and target languages. The translation quality
evaluated with sacreBLEU 29 for WMT17 En-Zh and tokenized BLEU (8
for other benchmarks.
WMTT4
WMTT7
WMT16
Models
Iter
Average Gap
Speedup
En-De
De-En
Eh-Zh
Zh-En
En-Ro
Ro-En
Transformer
40
27.66
31.59
34,89
23.89
34.26
33.87
0
1.0x
32.86
32.87
SMART
10
10
25.10
29.58
32.71
32.86
2.2x
DisCo
25.64
32.25
3.5x
Imputer 34
25.0
2.7x
CMLMC 15
26.40
30.92
34.14
34.13
1.7x
Vanilla
18.92
19.93
24.71
15.3x
CTC 21
17.68
19.80
26.
12.23
14.6x
FlowSeq
18.55
23.36
29.26
30.16
1.1x
AXE 9
20.40
24,90
30.47
31.42
14.2x
OAXE 6
22.4
26.8
14.2x
CTC
30
14.6x
DA-Transformer Lookahead 14
26.55
30.81
33.54
22.68
32.31
32.73
1.27
14.0x
DA- Ta-Taafsforrr- + Viterbi 37
26.89
31.10
33.65
23.24
32.46*
32.84*
1.00
13.2x
FA-DAT 26
27.47
31.44
34.49
24.22
0.41
13.2x
KCFG-NAT
27.02
31.29
33.60
23.40
32.72
33.07
0.84
12.6%
Table 2: Results raw WMT14 En-De, WMT17 Zh-En and 6 En-Ro. 'Iter' means the number of decoding iterations, and is the length of the target sentence. The speedup is evaluated on the WMT14 En-De test set with batch size of 1. *** indicates the results of our re-implementation.
Implementation Details For method, choose the = 1, 4 settings for Support Tree of RH-PCFG and linearly anneal r from 0.5 to 0.1 for the glancing training. For DA- Transformer [4), models. All models
use À = 8 for the graph size, which has comparable hidden states optimized with Adam 20 with 3 (0.9, 0.98) and = 10-8 For models, each batch contains approximately 64K source words. All models are trained for 300K steps. For WMT14 En- On-De and WMT17 Zh-En, we measure validation BLEU for every epoch and average the 5 best checkpoints as the final model. For W En-Ro, we just use the best checkpoint NVIDIA Tesla VI00S-PCIE-32GB GPU to measure the translation latency on the WMT14 14 En-De test set with a batch size of 1. We implement our models based on the open-source framework of fairseq 27
the valid dataset. We the
4.2 Main Results
Translation Quality
As demonstrated in Table PCFG-NAT outperforms CTC-GLAT 30 and DA-Transformer [14 in
terms of translation quality, effectively reducing the performance gap between NAT and AT models. This indicates its ability to capture the intricate translation distribution inherent the raw training
data and efficiently leamn from dependencies with reduced multi multi-modality. FA-DAT (26) builds upon
7

---

WM
WMT17
WMT16
Decoding Strategy
Speedup
En-De
De-En
En-Zh
Zh-En
En-Ro
Ro-En
Greedy Decoding
26.01
31.38
32.82
22.50
32.05
32.65
1.00x
Viterbi Decoding
27.02
32.29
33.60
23.40
32.72
33.07
0.91
Table 3: Ablation Study on the effect of decoding strategies. We use the NVIDIA Tesla V100S-PCII 32GB GPU measure the speedup on the WMT14 En-De test set with a batch size of 1.
DA-Transformer and trains the model to maximize a fuzzy alignment score between the graph and reference, taking captured translations in all modalities into account. The same training objective can also be implemented in PCFG-NAT. Experimental results show that FA-DAT (26) achieves state-of-the-ar NAT performance. the future, we plan incorporate the ideas from FA-DAT [26] into PCFG-NAT, enabling a fairer comparison and demonstrating the superiority of PCFG modeling.
Inference Latency
As shown Table 2 PCFG-NAT demonstrates a decoding speedup that slightly lower than Vanilla NAT but significantly higher than both AT models and iterative NAT models. This observation indicates that PCFG-NAT effectively strikes a favorable balance between translation quality and decoding latency, achieving desirable trade-off between the two factors.
4.3 Ablation study
4.3.1 Structure of Support Tree
In this section, we examine the impact of the max depth I of the local prefix tree and the upsample ratio in the support tree of RH-PCFG. Figure presents the results. When the total number of non-terminal symbols is kept constant, incorporating a local prefix tree with max two layers leads to significant decline in performance. We posit that an excessively large value of leads to a significant increase in the number of potential parse trees, thereby making it more challenging for the model to learn complex structures from the data. As a result, the translation performance of the model may decline.
Cfffeent Local Freet The Nox
272
Nas Lapers
204
264
Ablation study on the max depth I of local prefix tree and upsample rample ration X. Notice the number of the non- terminal symbols m equals to XL 2 2, where Ly is the length of the source sen- tence.
Figure 5:
4.4 Decoding Strategy
To determine the optimal parse tree for given sentence length, employ Viterbi decoding. However, this approach incurs a certain computational cost in terms of decoding speed. To demonstrate the necessity of Viterbi decoding, we compare with a greedy decoding strategy that always selects the rule with the highest probability. The experimental results, presented in Table [ indicate that while Viterbi decoding only slightly slower than greedy decoding, yields significantly better performance. Based on these findings, we assert that Viterbi decoding is necessary and do not recommend the use of greedy decoding.
4.5 Ablation Study for Distilled Data and GLAT
We conducted ablation studies to examine the impact of Glancing Training and Knowledge Distillation on our model's 's performance. .a..aneee....TT consistently improves the translation performance of all models, while the gains from KD are more limited.
8

---

WMT16 En-Ro
BLEU
Baseline
34.26
Transformer 40
+KD
34.72
Baseline
31.98
DA Transformer 14
+GLAT
32.46
+GLAT & KD
32.40
Baseline
31.94
PCFG-NAT
+GLAT
32.72
+GLAT & KD
32.75
Table 4: Comparison of WMT16 En-Ro translation performance (BLEU scores) for Transformer, DA Transformer, and PCFG-NAT models with and without Glancing (GLAT) 30 and Knowledge
Distillation (KD) techniques. Baseline means the models are trained train without GLAT
30
0.3
0.5
AM
855
180
14
162
166
78
Figure 6: The frequency of cart-of-speech and subwords generated from the main chain and local prefix tree respectively.
4.6 Syntax Analysis for Generated Parse Tree
To analyze the syntax information captured by PCFG-NAT, we conducted an analytical experiment on the WMT14 De-En test set. We utilized the averaged perceptron tagger, a part-of-speech tagging tool available in the nltk library [24), to assign part-of-speed tags to the translations. Subsequently, calculated the frequency of part-of-speech tags for tokens directly generated from non-terminal symbols in both the main chain and the local prefix tree. Additionally, we counted the frequency of unfinished subwords ending with the symbol @a. To ensure comparability, the frequency scores were normalized by the number of tokens in the main chain and local prefix tree, respectively. The results are presented Figure
As depicted in Figure the majority of tokens generated by non non-terminal symbols in the local prefix tree correspond to the pfrt-of-speech tags RB (adverb). JJ (adjective). NNP (proper noun, singular), and comma. Furthermore, the frequency of unfinished subwords the local prefix tree symbols is notably higher compared that in the main chain. It is well-known that RB, JJJ. NNP, comma, and unfinished subwords are closely related to the local adjacent tokens. Hence, PCFG-NAT effectively captures the connection between these elements with the local prefix tree serving complementary information to the main chain.
4.7 Probability Analysis of the Maximum Likelihood Path
calculated the proportion of the maximum probability parse tree (path) that generates the ref- erences among all parse trees (paths) to investigate the benefits of the syntax tree learned by the RH-PCFG in comparison to the path in the Directed Acyclic Graph (DAG). We observed that the proportion of the highest probability parse tree for PCFG-NAT is generally higher than the proportion

---

WMT14
WMT17
WMT16
Average
En-De
De-En
En-Zh
Zh-En
En-Ro
Ro-En
.M-Transformer
0.6026
0.7565
0.4899
0.5611
0.7948
0.7124
0.6529 0.6605
PCFG-NAT 0.6421
0.7718
0.4681
0.5886
0.7764
0.7160
Table 5: In the Valid Set, for the DA- DrA-Tranfformer model, we calculated X(Y|X, A*)/P((XX), where X, Y represents the source text and the target text the dataset, A (ogggrraP(AA.. Y). and A represents hidden state path. For the PCFG-NAT model, we calculated (T)/XX))//((((X))) where T (rggnaryPPT.... Y), and represents parse tree.
of the highest probability path for DA- DA-Transformer [4] This demonstrates that in the trained RH-PCFG, the probabilities of parse trees are more concentrated. Under the same training set and learnable parameters, this indicates that the modeling approach of RH-PCFG is more in line with the intrinsic structure of sentences compared to the flattened assumption of Directed Acyelic Graph.
Related Works
al. [3] accelerates neural machine translation with a non-autoregressive Transformer, which comes at the cost of translation quality. The major reason for the performance degradation is the multi-modality problem that there may exist multiple translations for the same source sentence. Many efforts have been devoted enhancing the representation power of NAT. One thread of research introduces latent variables to directly model the uncertainty in the translation process, with techniques
like vector quantization [16
33 3 generative flow 255 and variational inference 38 12 Another
thread of research explores some statistical information for intermediate prediction, including word
fertility (I3), word alignment 391, and syntactie structures 22 [23]. Zhang et al. [43] explores
the syntactic multi-modality problem in non-autoregressive machine translation The most relevant works to PCFG-NAT are CTC-based NAT I 22 and DA-Transformer 14
37 26], which enriches
the representation power of NAT with a longer decoder. They can simultaneously consider multiple translation modalities by ignoring some tokens [T modeling transitions
[4] Fang al. 7 proposes non autoreerresiive direct speech-to-speede translation model, which achieves both high- quality translations and fast decoding speeds by decomposing the generation process into two steps. Compared to previous NAT approaches, PCFG-NAT mitigates the issue of multi-modality in NAT by capturing structured non-adjacent semantic dependencies in the target language, leading to improved translation quality. Our work is also related to the line of work on incorporating structured semantic information into machine translation models [2 19 32 J]. contrast to these works, PCFG-NAT learns structured information unsupervised from data and can generate target tokens in parallel.
Conclusion
Non-autoregresssvve Transformer has higher inference speed but weaker expression power due to the lack of dependency modeling. In this paper, propose PCFG-NAT to model target-side dependency by capturing the hierarchical structure of the sentence. Experimental results show that PCFG-NAT further enhances the translation performance of NAT models while providing a more interpretable approach for generating translations.
7 Limitations
Further research and exploration are needed to overcome the limitations encountered when attempting to induce complex PCFGs from data. Developing more advanced learning algorithms or exploring alternative approaches for modeling and capturing the rich syntactic structures of language may be necessary to unlock the full potential of PCFG-NAT and further enhance its translation capabilities.
10

---

8 Acknowledgement
This work is partially supported by the National Key R&D Program of China(under Grant 101022, the NSF of China(under Grants 61925208), CAS Project for Young Scientists in Basic Research SSR-0299 and Xplore Prize. We thank all the anonymous reviewers for their insightful and valuable comments.
References
[1] R. Aharoni and Y. Goldberg. Towards string-to-tree neural machine translation. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pages 32-140, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.18653.1 1/P17-2021. BRLLittpee//aellaachoooogg
[2] N. Akoury, K. Krishna, and M. lyyer. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Compula- tional Linguistics, pages 1269-1281 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/P19- 122. htLhttps:: httpps:://accaaatll..o... hww.ilogg.orggg///. org/P19- 1122
[3] Y. Bao, S. Huang, T. Xiao, D. Wang, X. Dai, and Chen. Non-autoregressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3749-5759 Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 www.l-mainnna-.---- URL https://aclanthollo.
org 2021 nwwcl-acllaaooooaa main. 458
[4] Y. Bao, H. Zhou, S. Huang, D. Wang, L. Qian, X. Dai, Chen, and L. Li. latent-GLAT. Glancing at latent variables for parallel text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 8398 8489, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18663ww112222.ccllooooo 575. URL
org/ 2022 acl acl-long..00. 575
[5] N. Chomsky. On certain formal properties of grammars. Information and Control, 2(2):137- 167, 1959. ISSN 0019-9958. doi: http:://w..oonmm//ooo/oo//999999
URL
//t::////wwww. http://wwwccaceecccccceeeeeeeeas
[6] C. Du, Z. Tu, and J. Jiang. Order-agnostic cross entropy for non-autoregreessive machine translation. CoRR, abs/2106.05099. 2021. URL
[7] Q. Fang, Y. Zhou, and Y. Feng. Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. In Advances in Neural Information Processing Systems, 2023.
[8] M. Ghazvininejad, O. Levy, Y. Liu, and L.. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) pages 61 12-6121, Hong Kong, China, Nov. 2019, URL https:/7
Association for Computational Linguisties. doi: 10. 10.18653/1/DD19-163..
aclanthology -tw.rhoolooy.oooo/1---9.5--
[9] M. Ghazvininejad, V. Karpukhin, L.. Zettlemoyer, and O. Levy. Aligned cross entropy for non-autoregressive machine translation. CoRR, abs/2004.01655, 2020. URL https://arxiv org hrg/abs/2004.01655
[10] M. Ghazvininejad, O. Levy, and L. Zettlemoyer. Semi-autoregressive training improves mask- predict decoding. CoRR, abs/2001.08785, 2020. URL hhttps:/arxii. http://aaxvvvogg////// sttpivv.orr/abs//00000 08785
[11] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural nets. In ICML '06; Proceedings of the International Conference on Machine Learning, 2006.
[12] Gu and X. Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. CoRR, abs/2012, 15833, 2020. URL htthhtps::. //aarii org/ fabs/2012..o 15833
11

---

[13] J. Gu, J. Bradbury, C. Xiong, V. Li, and R. Socher. Non-autoregresssvve neural machine translation. In International Conference on Learning Representations, 2018. URL https
//openreview net/forum?inn Bit.mmmddddBBBBBeeee
[14] F. Huang, H. Zhou, Y. Liu, H. Li, and M. Huang. Directed acyclic transformer for non- autoregressive machine translation. In Proceedings of the 39th International Conference on Machine Learning, ICML 2022, 2022.
[15] X. S. Huang, F. Perez, and M. Volkovs. Improving non-autoregressive translation models without distillation. In International Conference Learning Representations, 2022. URL
[16] L. Kaiser, S. Bengio, A. Roy, A. Vaswani, Parmar, Uszkoreit, and N. Shazeer. Fast decoding sequence models using discrete latent variables. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 2390-2393 2399. PMLR, 10-15 Jul 2018. URL
mlr, preas//800kaiierr88.. www./eiisseeaarere htmmmmmm11.mmmm11111..mmmm html
[17] Kasai, J. Cross, M. Ghazvininejad, and J. Gu. Parallel machine translation with disentangled context transformer. CoRR, abs/2001.051133 2020. URL htttts://axxiii.orrrr ab0/ 05136
2001
[18] Kasai, J. Cross, M. Ghazvininejad, and Gu. Non-autoreerrssive machine translation with disentangled context transformer. In International conference on machine learning, pages PMLR, 2020.
[19] Y. Kim. Sequence-to-sequencc learning with latent neural grammars. Advances in Neural Information Processing Systems, 34:26302-26317, 2021.
[20] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego CA. USA. May 7-9, 2015, Conference Track Proceedings, 2015. URL http
//arxiv. org/ ///////////// 1412. 6980
[21] J. Libovický and J. Helel. End-to-end non-autoregreessvve neural machine translation with connectionist temporal classification. CoRR, abs/1811.04719, 2018. URL http://arxiv org/abs/1811.04
[22] J. Libovický and J. Helcl. End-to-end non-autoregressivve neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-3021. Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18-11 1336. URL https 7/
org/ anthology/DD1 1336
[23] Y. Liu, Y. Wan, J. Zhang, W. Zhao, and P. Yu. Enriching non-autoregressive transformer with syntactic and semantic structures for neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1235-1244, Online, Apr. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 1/2022
URL
maa.o/ooo..o./0.....eemmaaaaaaa. eacl-main
105
[24] E. Loper and S. Bird. Nitk: The natural language toolkit, 2002. URL httts:///axii...t.
[25] X. Ma, C. Zhou, X. Li, G. Neubig, and E. Hovy. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4282-4292, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-14433 aclweb. http://attolooyD1190
httpss/t::///w
[26] Z. Ma, C. Shao, S. Gui, M. Zhang, and Y. Feng. Fuzzy alignments in directed acyelic graph for non-autoregressive machine translation. The Eleventh International Conference on Learning Representations, 2023. URL http:p:://opeeeveee.eeees/// - gQyd0zE
12

---

[27] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng. D. Grangier, and M. Auli. fairseq: fast, extensible tooliit for sequence modeling. CoRR, abs/1904.01038, 2019. URLhttp 7/arxiv. orgo hbse//bs//// abs//9/s/111 01038
[28] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evalua- tion of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 1031115/107303310731155 URL attps://aclanthologgs wrg/ 1040
[29] M. Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels, Oct. 2018. Association for Computational Linguistics. URL https://wwwt awwweb. org/ www.orggggggggo///oo/ooo 6319
[30]
L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2003, Online, Aug. 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2022 acl--ong..555 155. URL attps://aclantholee
org/ 2021 wacl-llog..11. .55
[31] Q. Ran, Y. Lin, P. Li, and J. Zhou. Guiding non-autorrerressive neural machine translation decoding with reordering information. In Thirty-Fifth AAA/ Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pages 13727-13735, AAAI Press, 2021. URL
https://ojs. aaai. .trgii.rg/i/i. ind/x.ndexxx/.. php/ httpp//AA//aaricceeeee/////////a
[32] P. Rastogi, R. Cotterell, and J. Eisner. Weighting finite-state transductions with neural context. Association for Computational Linguistics, 2016.
[33] A. Roy, A. Vaswani, A. Neelakantan, and N. Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1805. 11663, 2018.
[34] C. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoregressive machine translation with latent alignments. CoRR, abs/2004.07437, 2020. URL UtLthtts::
org wttp://wwwwi.r///////////// 200/
07437
[35] I. Sakai. Syntax in universal translation. In EARLYMT, 1961.
[36] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, Aug.
2016. Association for Computational Linguistics. doi: 10.18653/1/P11-1110 1162. URL httts:///cclatthology wwwphhhooooyyoooPP------- org/P16-1162
[37] C. Shao, Z. Ma, and Y. Feng. Viterbi decoding of directed acyelic transformer for non- autoregressive machine translation. In Findings of EMNLP 2022, 2022.
[38] R. Shu, J. Lee, H. Nakayama, and K. Cho. Latent-variable non-autoreeressive neural machine translation with deterministic inference using delta posterior. In AAAI, 2020.
[39] Song, S. Kim, and Yoon. AligNART: Von-autoregresssve neural machine translation by jointly learning to estimate alignment and translate. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1-14, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v 2021.emmlp-mane 1. URL ttttps://aclaattholooo
org/2021. 2021. emnlp- main.
org/
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and Polosukhin. Attention all you need. In I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:::/t///tttt httetp:s//proceednnggg neurips
13

---

[41] A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):200-269, 1967. doi: 10.1 1967.1054010.
[42] B. Wang, I. Titov, J. Andreas, and Y. Kim. Hierarchical phrase-based sequence-fo-sequeecc learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8211-8229, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https: hatp::///caaattoooogoee org/2022. 2022. emnlp emnlp-main 563
[43] K. Zhang, R. Wang, X. Tan, Guo, Y. Ren, T. Qin, and T.-Y. Liu. A study of syntactic multi- modality in non-autoregrressvv machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1747-1757, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ http:/2222222aaeeeme 1a1o22.nacllmaatn 126. URL https://
aclanthology.orge wwwp:/logogorg///2222//// nww.l main.. 126
14

---

A Appendix
A.1 Construction of Support Tree
src_length stands for the length of source sentence D.-
layer_ stands for max depth of local prefix tree I.
upsample_rathe stands for hyperparameter X.
def build_support, tree wet.tttreeescreee (src__ennth, layer_ upsample_lambda) main_ saan_cchinnssiie size = upsample_lambda lambda src_length
root Node root leftchild Node
now root
for i in range (hain__hhiinsize)
now rightChild Node ()
now rightChild
now leftChild full_ biilbbiary__teee (layer_ (1ayer__e_ _izee
10
return root
Listing 1: Code for Constructing Support Tree in Section 3.1.2
def full_ Salll_binar_-see tree _azeer_iiz)::
if layer _ size <= 0:
None
root N)de
root foot.leften lefthhidd = full seal_iinrryttrere (layerrssssiieee -1)
root.rightChild - full_binary. tree [layer_size- return root
Listing 2: Code for Constructing a Constructing a Full Binary Tree with deepth layer_size
The function build_ mailddsupport_trre teree returns the root node of the support tree.
A.2 CYK Algorithm for RH-PCFG
m, stands for the numbers of non-terminal symbols and target sentence length.
layer_size stands for max depth of local prefix tree
OpProbs [a] [i] represents Va).
RuleProb [a] [b] [c] represents P(< Vo, V. (V.).
S[a] [i] [j] represents P(V, = V/ ..... Uj). are initialized
def local_prefix: cerefix_tree_ceee (y OpProbs, RuleProb, n, layer_size)
for i in range (0, n):
for in range (0,
[[a][i][1]
OpProbs [i]
for apan in range (2, @*llaye__izen for a in (0nge(0,m))
for in ronge for b, Child(a): for k range i+span):
10
S[a] [i] += [uleProb [b] [c]
II
OpProbs [a] [k]
s[b] [k-1] [[c] [i+span]
Listing 3: Code for CYK Training Local Prefix Tree in Section 3.3.1.
def main chainhainncc. c (S, OpProbs, RuleProb, m, n, layer size)
a range (0, m):
[[a] [n-1] [n-1] OpProbs [a] [n-1] for start in range (n-2, -1, -1); for in range (0, m):
15

---

for in range(0,n)
for b,c Child(a):
for k range (start, start=2* s [ [start] [n-1] +- [M]eePro([a](b]]]]
10
OpProbs [k] \
[star] [k-1] [[c][k+] [n-1]
Listing 4: Code for CYK Training of Main Chain Section 3.3.1.
After applying local local_prefix_teee wwwppreffxtteeeeppeeee trke_cyk and main_chain_ S[1] [0] [n-1] contains the
value of P(Vi Vo...., -n-))-
A.3 Best Parse Tree for Glancing Training
m, stands for the numbers of non-terminal symbols and target sentence length.
layer_size stands for max depth of local prefix tree
OpProbs [a] [i] represents P(yi(Va).
RuleProb [a] [b] [c] represents P(< Vo, Ve [Va).
S[a] [i] [j] represents
P(V. y4.-. Uy)-
Trace [a] [i] [j]=True means that derivation P(V. = Yis..., B;) ) is part of the maximum derivation of target sentence. Trace are initialized to False.
[k] a means that the token position in target sentence should be aligned to non con-terminal symbol Vg. A are initialized to O.
def best pest_p www.tppareeeteeeeeeeeeeeeeeeeeee tree (s, A, Trace, OpProbs, RuleProb, m, n, layer_ size) [m-1] (0]
Trace [1] for start in range (0, n-2)
True
for in range (0, m) if Trace [star]]]]]]
max_b, max_c, max_k argmax (
RuleProb [a] [b] [a] [pProbs[a]]s \
\
[tb] [[c] \
10
for (b,c,k) in \
11
zip ((Child(a), range (start, 2
12
14
Trace [max_ b] [start] [max_k-1] True Trace [max [max km]] True
15
16
k] =
for span in range (2, 2** 2=*1ayrrsizze] for in range m): for in range(0,n):
19
20
21
if Trace (aa)
max_b, max_c, max_k argmax (
23
RuleProb[a] \
24
[a]]
25
* s[b][i][k-1] S[c] [k+1] [i+apan] \
for (b,c,k) in (Ch(ldid(aa) range(i, i+span))
28
29
Trace [max_ba b] []] mma._--i True Trace [max_a cl [max_k+1] [i+span] True [mmxx_ k] =
30
Listing 5: Code for Glancing Training Best Parse Tree Section 3.3.2.
After applying the algorithm, the tensor contains the best alignment of non-terminal symbols and target tokens.
16

---

A.4 Algorithm for Inference
m, stands for the numbers of non-terminal symbols and target sentence length.
layer_size stands for max depth of local prefix tree
S[a] [i] [j] represents P(Va Vi: U.).)
MaxOpProbs [a] represents max mox: (a))V.)
MaxOpTokens [a] represents argmary P (yy Va).
RuleProb [a] [b] [c] represents P(< Vo, Ve > IVa).
MAX_ [A]_ [L] are the probability of Mf, of
Mf.- <b,c> (ccc)EChh())).))Tkkkk. ktl.--
max
P(V. - VooV.)
(16)
and [a]] [L] MAA_ [L] -MA_C[aa [L] are corresponding value of k. b,
output is the target sentence generated by viterbi decoding algorithm.
def viterbi_dde _eceeiddeodnngggge (S, Max0pProbs, RuleProb,
layer sazer_size, MAX_P,
MAX_K, MAX_B, MA
start in range (n-2, -1, -
for a in range (0, m)
max value, max_ max_c, max_k argmax RuleProb [c]]
MaxOpProbs [a]
[b] for (b,c,k) in zip (a) range (start, start+ lazer_ize)))
[k+1] [n-1]
11
12
MAX_P [a] [n-start] max value
13
MAX X [a] [n-start] max_k
MAX B [a] [n-start] max_b
15
MAX_ [a] [n-start] max_c
16
17
for span in range (2, 2*+layer_size
18
for in range (0, m)
max_value, max_b, max_c, max_k argmax RuleProb [b]
21
MaxOpProbs [a] \
22
[MX_P[b[(k]] * MAX_P[c] [span for (b,c,k)
(Ch(ilid((a), range i+span))
25
)
26
MAX P [a] [span] max_value
27
MAX_ [a] [span] max_k
MAX [a] [span] max_b
MAX [a] [span] max_
Listing 6: Code for Viterbi Decoding Algorihtm in Section 3.4.
def parse_ (a, output, prefix_length, length, MaxOpTokens, MAX_ B_ MAX_C, MAX_K):
max_b, max_c, max_k MAX_B[a] [length], MAX_C[a] [length], MAX_K [a [[lenghh] output len [1enthh)]] MaxOpTokens [a]
(max_b,, prefix_length, max_k-1) meefix.leggh--mat length max_k)
First, we utilize the viterbi _earbbd_deodinnggggo method to obtain the values of MAX_B, MAX_C, MAX_K. Subsequently, considering the length constraint denoted as L,, we employ the parse_ tree func-
tion with parameters set as follows: starting index of Vi. output to store the gener-
17

---

ated target sentence, prefix_length with an initial value of O, maximum limit of L tokens, MaxOpTokens, Baappoooess,MA_BBBBB.. MAX__B,MAX_,,MML MAX_KK This process effectively generates the desired target sentence.
A.5
Case Study
8-2434 mich meisten.
5-1254 Manchral bearchen aie such aintach nur sahr schnoel Gold.
Sometimee mhy jue the money vey quickly.
they just money very queckty
T-2434 That what troubles me most
H-2434 1,4299921623339
And that bothers me most.
Ard
Ev0N0
(a)
(b)
Figure 7: Two translation cases with generated parse tree from WMT14 De-En test set
To better understand the ability of PCFG-NAT to model structured semantic information, we select two cases from the WMT14 De-En test set and take close look at the generated parse tree of PCFG-NAT. In Figure 7 a), we can see that in the sentence generated by PCFG-NAT, the content words lie in the main chain of the generated parse tree, and the words like just, very, which supplements to words in the main chain, lie in the local prefix tree of the generated parse tree. In the generated parsing tree in Figure bb), the unfinished subword @ @ becomes the left node of its suffix thers, which demonstrates that PCFG-NAT can learn the local structure natural language and enhances the interpretability of NMT certain extent.
18


Reviews:
[{"review_id": "zFIpGK4Mtf", "reviewer": "Reviewer_FGhz", "summary": "This article proposes a new NAT training method based on PCFG. To address the limitations of critical and unidirectional dependencies, the authors provide a new way of establishing dependencies from the perspective of CFG, hoping to capture both grammatical and semantic information. Therefore, this article simplifies PCFG into Right Heavy PCFG and, in the process of training and decoding generation in NAT, achieves a further reduction in the translation gap between NAT and autoregressive models by adding a derivation process based on Right Heavy PCFG.", "questions": "1.\tHow does glancing training strategies influence the PCFG-NAT training?\n2.\tWhich languages does the Right Heavy PCFG proposed in this paper suitable for?\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "An interesting algorithm based on PCFG is proposed, which reduces many limitations on previous methods and provides more semantic and grammatical dependencies.", "weaknesses": "1.\tThis paper claims to endow the capability in capture more complex dependencies, but the experimental part does not give a comparison and analysis of this aspect.\n2.\tThe training loss function is not clear.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback on our paper and would like to respond to your comments and concerns.\n\n#### **Capturing complex dependencies:**\nWe acknowledge that our experimental section could be more explicit in comparing and analyzing the ability of our proposed PCFG-NAT method to capture complex dependencies. In the updated version of our manuscript, we will conduct in-depth experiments to analyze this aspect. \n\nOur syntax analysis experiment shows that our model generates more adverbs, adjectives, proper nouns, commas, and unfinished words from the local prefix tree. The semantic differences between the tokens generated by the main chain and local prefix tree indicate the ability of PCFG-NAT models to capture complex dependencies. \n\nIn our global rebuttal, we have added a new set of experiments that demonstrate, in comparison to the non-hierarchical DAG model, RH-PCFG learns more concentrated probabilities for sentence structures. This indirectly shows that RH-PCFG possesses a stronger ability to model dependencies between tokens.\n\n#### **Glancing training strategies:**\n\nIncorporating glancing training into PCFG-NAT encourages the model to capture more accurate dependencies between target tokens, resulting in improved translation quality compared to training without glancing, as shown in the table in the global rebuttal. We will include additional experiments on larger datasets in the final version of our paper.\n\n#### **Suitability of Right Heavy PCFG for different languages:**\n\nThe design of RH-PCFG is based on the assumption of left-to-right semantics in natural languages. To the best of our knowledge, left-to-right languages account for the largest proportion of the world's population, and the languages used in our experiments with the WMT Benchmark are also left-to-right. We acknowledge that there are languages that do not conform to the left-to-right assumption. However, our method can be easily adapted to other modes like right-to-left. In the future, we will conduct additional experiments to demonstrate this adaptability for a wider range of languages in the context of non-autoregressive translation.\n\nWe appreciate your feedback and will incorporate these clarifications and improvements into the revised version of our paper.\n"}, {"review_id": "FK0zoXFOiZ", "reviewer": "Reviewer_EKPB", "summary": "The paper proposes a new approach called PCFG-NAT to address the limitations of conventional Non-autoregressive Transformer (NAT) models. PCFG-NAT uses a specially designed Probabilistic Context-Free Grammar (PCFG) to capture complex dependencies among output tokens and enhance the expression power of NAT models. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and autoregressive (AT) models. Additionally, PCFG-NAT facilitates a deeper understanding of the generated sentences and addresses the lack of satisfactory explainability in neural machine translation.", "questions": "N/A", "limitations": "See weakness", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a very novel NAT model that performs comparably or even better than state-of-the-art NAT models.", "weaknesses": "1. The analysis of training complexity is not thorough enough. \na. It does not show the training inference of the proposed model compared with baselines.\nb. Although the paper mentions that RH-PCFG is faster than PCFG, there is no quantitative analysis and no empirical evidence to support this claim. It would be helpful to compare the training complexity with the baseline and provide a more in-depth analysis.\nc. How is d set in experiments? what is the relationship between d and m (or n)?\n\n2. There is a lack of an in-depth explanation of the generated sentences from the main chain and prefix tree in the syntax analysis experiment. Only the analysis based on the frequency is not intuitive.  Additionally, the analysis of the syntax tree of the generated sentences in the experiment is not adequate, because some of such sentences suffer from grammar issues. It would be better to analyze the syntax tree of the reference translation based on the given <X, R> and calculate p(T|X,R).\n\n3. Although RH-PCFG has strong expression power, the performance improvement is not significant. It would be helpful to explain the main reasons for this and whether there is potential to further explore this framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback on our paper and would like to respond to your comments and suggestions.\n\n#### **Training complexity analysis:**\n\nIn our paper, we provided the algorithm and complexity analysis for both training and inference of PCFG-NAT, along with detailed pseudocode in the supplementary material's appendix. We would like to clarify that the training complexity of our PCFG-NAT model is $O(d^5n+m^2n)$, where $m$ is the number of non-terminal symbols, $n$ is the sentence length, and $d=2^l$, where $l$ is the maximum height of the local prefix tree. In comparison, the complexity of a PCFG without rule pruning of Right Heavy Support Tree is $O(n^3m^3)$. And the training complexities of CTC-NAT and DA-Transformer are $O(mn)$ and $O(m^2n)$ respectively. In future work, we plan to provide quantitative analysis to support our claim that RH-PCFG is faster than PCFG without rule pruning and compare RH-PCFG with other baselines.\n\n#### **Analysis of the syntax tree:**\n\nWe agree that our syntax analysis experiment could be more comprehensive and intuitive. Analyzing the syntax tree of the reference translation based on the given <X, R> and calculating p(T|X,R) will provide a more in-depth understanding of the generated sentences from the main chain and prefix tree. \nIn our global rebuttal, we have added a new set of experiments that show, in comparison to the non-hierarchical DAG model, RH-PCFG learns more concentrated probabilities for sentence structures. This indirectly demonstrates that RH-PCFG possesses a stronger ability to model dependencies between tokens. In the updated version of our manuscript, we will undertake in-depth experiments to analyze this aspect.\n\n\n#### **Performance improvement:**\n\nWhile our PCFG-NAT model demonstrates a performance improvement compared to other NAT models, we agree that there is still potential for further exploration and enhancement. The limited performance improvement could be due to the simplifications made in the RH-PCFG structure and the trade-offs between model complexity and performance. In future work, we plan to investigate alternative approaches to capture more complex dependencies and further improve the performance of our model.\n\nWe appreciate your feedback and will incorporate these clarifications and improvements into the revised version of our paper.\n"}, {"review_id": "9E1Rq2ZDPf", "reviewer": "Reviewer_8FYH", "summary": "This paper proposes to rely on PCFG to incorporate the most likely parsing tree into the non-autoregressive transformer model as a way to mitigate multi-modality problem. Since considering all possible parsing trees is computationally expensive, the authors propose a right-heavy PCFG based on the assumption that language tends to show left-to-right connectivity. The experiments on the three well-known WMT datasets show their method outperforms others while keeping the speed-up provided by NAT.", "questions": "- What is your full objective function? During training, you rely on the target length, but you need to obtain the length from the model during decoding. Also, it is not clear to me if you only use the sum of LL of all possible trees as your only objective to train the translation model.\n- Why do you use full vocabulary and not `<eos>` symbol as terminal states?\n- Figure 5: it's unclear what contributes more, $\\lambda$ or $l$. And in your plot, there is no plateau, meaning it could be better if you increase it further.\n- I assume you do not use KD for your experiments. However, it is not explicitly stated. That is a valuable result that can strengthen your paper. Also, since KD is widely used, it would be interesting to see if KD can further improve your model\n\nTypos & Grammar & Style:\n- line 243: Figure instead of Table\n- line 23: to mitigate.... to alleviate: duplication of the same meaning\n- line 97: `know as` - I am confused here, I assume you propose this method, so it's not known before\n- line 294: space after `quality.`", "limitations": "- Assumption that left-to-right dependency is more valuable for languages (which potentially false for some languages) is not addressed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel approach to capture dependencies between tokens for NAT using PCFG \n- Experimental results show that method allows to improve NAT translation in terms of BLEU score", "weaknesses": "- Limited analysis:\n  - method is based on the assumption that left-to-right connectivity is the best for language and that \"RH-PCFG strikes a balance between expressive capabilities and computational complexity\"; however, no experiments support this claim. \n  - Also, Figure 5 shows that increasing the local prefix tree helps in terms of BLEU. \n- Missing references and comparison:\n  - Follow-up work on DAG Ma, Zhengrui et al. \"Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation.\" (ICLR 2023, published Feb 1st). I understand this work appeared a few months before your submission, but it would be fair to compare.\n- Lack of clarity\n  - I have a hard time understanding the central concept of training and evaluating NAT model, which hurts reproducibility (please see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback on our paper and would like to respond to your comments and suggestions.\n\n#### **Balancing between expressive capabilities and computational complexity:**\n\nWe acknowledge that our paper could have provided more experimental evidence to support our claim that RH-PCFG strikes a balance between expressive capabilities and computational complexity. RH-PCFG maintains the expressive power of PCFG while reducing the training complexity. From the perspective of RH-PCFG improving translation performance, it can be understood that the expressive capacity of RH-PCFG enhances the translation quality of NAT, while the training complexity of RH-PCFG is lower than that of the general type of PCFG without rule pruning.\n\n#### **Questions about Figure 5:**\n\nGiven that $m = \\lambda L_x \\cdot 2^l + 2$, under the condition of constant $l$, increasing $\\lambda$ does enhance the BLEU score. Subsequently, we will continue to conduct experiments to investigate the impact of larger $\\lambda$ values on the results. However, when the total number of non-terminal symbols $m$ remains constant, elevating $l$ results in considerable deterioration in performance. We hypothesize that an excessively large value of $l$ induces a significant increment in the number of potential parse trees, thereby posing increased difficulty for the model to discern complex structures from the data. Consequently, the translation performance of the model may experience a decline.\n\n#### **Missing references and comparison:**\n\nThank you for pointing out the relevant work by Ma, Zhengrui et al. (ICLR 2023). We will include this reference in our revised paper and conduct a comparison with their work to provide a more comprehensive evaluation of our method.\n\n#### **Lack of clarity and objective function:**\n\nWe apologize for any confusion regarding our objective function. During training, we maximize the likelihood of the target sentence by summing up all the possible parse trees for the given sentence conditioned on the source sentence and model parameters (Equation 6 in the paper) and the pseudocode is provided in supplementary material's Appendix A.2. During the decoding process, for every potential target length $L$, we use Viterbi decoding to find the optimal parse tree under the constraint of the target length $L$, as exhibited in Appendix A.4. Following Shao et al. [36], we reorder the generated sentences with different lengths considering probability and length factors, ultimately outputting the sentence with the highest score.\n\nRegarding the use of the full vocabulary and not the <eos> symbol as terminal states, terminal states are the symbols that will not derive a subtree. Consequently, for any PCFG responsible for generating sentences, the terminal states encompass the entirety of the vocabulary of the language.\n\n#### **Knowledge distillation:**\n\nIn our experiments, we did not use KD for training our PCFG-NAT model. We agree that it is a valuable result that our method achieves competitive performance without relying on KD, which is widely used in other NAT models. This finding highlights the effectiveness of our proposed method in capturing hierarchical dependencies and mitigating the multi-modality problem in NAT.\n\nIn our global rebuttal, we acknowledge that the effect of KD on performance is limited on the WMT16 EnRo dataset. We will include additional experiments on larger datasets in the final version of our paper.\n\nIn the revised version of our paper, we will clarify the non-use of KD in our experiments and discuss the benefits of incorporating KD.\n\n#### **Typos, grammar, and style:**\n\nWe appreciate your careful review and identification of these issues. We will correct the typos, grammar, and style issues in the revised version of our paper.\n\n#### **Limitations:**\n\nWe acknowledge the limitation regarding our assumption that left-to-right dependency is more valuable for languages. To the best of our knowledge, left-to-right languages account for the largest proportion of the world's population, and the languages used in our experiments with the WMT Benchmark are also left-to-right. For the languages that conform to the right-to-left assumption, our method can be easily adapted to a right-to-left mode. In the future, we will conduct additional experiments to demonstrate this adaptability for a wider range of languages.\n\nWe hope our responses have addressed your concerns and questions. We will incorporate these clarifications and improvements into the revised version of our paper. Thank you once again for your valuable feedback.\n"}, {"review_id": "Gqak7MBHlI", "reviewer": "Reviewer_sYwF", "summary": "This paper proposes PCFG-NAT, which alleviates the issue of conditional independent modeling through probabilistic context-free grammar. PCFG-NAT leverages syntax information to capture bidirectional dependencies among predicted tokens. In this way, this method also provides a better explanation of the translation process. Experiments on several benchmarks demonstrate the effectiveness of the proposed method in bridging the performance gap with autoregressive counterparts.", "questions": "Can the authors elaborate the effect of each technique by detailed ablation studies? I am curious about the performance and speedup without glancing training or RH-PCFG.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a novel PCFG-NAT method that introduces context-free grammar to capture dependencies in NAT. The authors develop systematic designs for it, including a lightweight variant called RH-PCFG, architecture support, training strategy, and decoding algorithms.\n\nIn addition, popular NAT techniques, such as glancing training and Viterbi decoding, can be integrated into PCFG-NAT.\n", "weaknesses": "The main concern for me is the wrong comparison. Table 2 shows the main results of the proposed method and related work. However, I notice that the authors omit some of the earlier methods that achieve better performance, such as [1][[2].\nIn addition, some reported results are inconsistent with the original paper, like the CTC+GLAT method reported in the paper [3] achieves 26.39 and 29.54 on WMT 14 En-De and De-En datasets, but 25.02 and 29.14 in this paper. If the report is reasonable, the author must explain it for clarification.\n \nI admit that the proposed method is novel, but it is complex. Simple methods, like glancing training [3] and DSLP [2], achieve comparable performance without complex modifications.\nAccording to Figure 2, PCFG is used only before the final prediction. I suspect that this design only helps slightly because the decoder layers are unchanged. Due to the complex design, it is difficult to design the ablation study to examine the role of PCFG.\n \n \n[1] Fully non-autoregressive neural machine translation: tricks of the trade.\n\n[2] Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision.\n\n[3] Glancing Transformer for Non-Autoregressive Neural Machine Translation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback on our paper and would like to respond to your comments and suggestions.\n\n#### **Comparison with earlier methods and reported results:**\n\nRegarding the discrepancy in the reported results for the CTC+GLAT method, we would like to clarify that the difference is due to the fact that the original paper [3] used a knowledge-distilled training set, while our experiments were conducted using the undistilled training set. If this misunderstanding was your primary concern, we hope that you can reconsider the evaluation of our work after our clarification.\n\nWhile we agree that simple and effective methods are of great value, we believe that the complexity of our proposed PCFG-NAT method also has its merits. Our method explores a new probabilistic model for capturing natural language sequences. Although the performance improvement is limited, PCFG-NAT learns how to generate sentences in a structured and interpretable manner without supervision, which could provide valuable insights for future work. Indeed, the PCFG is only utilized before the final prediction and the decoder layers remain unchanged, but our main contribution lies in designing a new and efficient probabilistic model for capturing target-side dependencies in NAT. In future work, we will consider investigating more suitable decoder layer architectures to further enhance the performance of our model.\n\n\n#### **Ablation studies to examine the effect of each technique:**\n\nWe acknowledge the importance of conducting detailed ablation studies to understand the impact of each technique, such as glancing training and RH-PCFG, on the performance and speedup of our method. Incorporating glancing training into PCFG-NAT encourages the model to capture more accurate dependencies between target tokens, resulting in improved translation quality compared to training without glancing, as shown in our global rebuttal. We will include additional experiments on larger datasets in the final version of our paper.\n\n\nWe hope our responses have addressed your concerns and questions. We will incorporate these clarifications and improvements into the revised version of our paper. Thank you once again for your valuable feedback.\n"}, {"review_id": "sdSTAK2QSs", "reviewer": "Reviewer_8Jr4", "summary": "This work attempts to learn hierarchical structure in form of a PCFG as part of a non-autoregressive translation (NAT) system. The intuition is that the PCFG relaxes the conditional independence assumption between target tokens in vanilla NAT. The authors argue that related approaches with the same objective focus on local dependencies, whereas their hierarchical approach can also represent long range dependencies. Results seem to indicate a good balance between latency and translation quality.", "questions": "- Could you provide more details about the target length / number of non-terminal estimation in your system?\n- Do you think there are any lessons learnt in hierarchical SMT that would be applicable here? For example, the constraints on production rules in Hiero?\n- NAT and AT greedy/beam search lends itself particularly well to GPU/TPU decoding, whereas the Viterbi decoder seems harder to support efficiently on accelerators. Can you comment on that? Are the Table 2 speed-ups measured on the CPU, GPU, or TPU?", "limitations": " Address last question above, if necessary.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and reproducible. The motivation for learning hierarchical dependencies in NAT is strong. The results are a bit on the slim side, but the novelty (as far as I am aware) of using hierarchy in NAT compensates for that. It is nice to see that a PCFG can be learned in that way. I appreciate the well designed ablation studies. The simplifying assumptions for the PCFG (binary and right-heavy) make sense.", "weaknesses": "I felt that the architecture under-delivers slightly on the claim of introducing PCFG-style dependencies to NAT, because production rule probabilities with a mix of non-terminals and terminals are still decomposed to the transition and terminal probabilities. The conditional independence assumption underlying Eq. 10 means that transition probabilities are independent of the terminal. Similarly, even if the PCFG introduces a link from y_i to y_j token, it doesn't mean that y_j really depends on y_i (because Eq. 9 is still used), just that the transition probability depends on h_i and h_j. \n\nThe connection to hierarchical SMT systems like Hiero is not made - please add citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and comments on our paper. We appreciate your recognition of the novelty of our proposed PCFG-NAT method and its potential to learn hierarchical dependencies in NAT. We would like to address your concerns and questions in the following rebuttal.\n\n#### **PCFG-style dependencies in NAT:**\n\nWe acknowledge that the decomposition of production rule probabilities with a mix of non-terminals and terminals into transition and terminal probabilities might not fully capture the PCFG-style dependencies in NAT. However, our method still provides a more flexible way to model dependencies between target tokens compared to vanilla NAT, as it allows the model to learn hierarchical structures and represent long-range dependencies. In future work, we will explore alternative approaches to better capture PCFG-style dependencies in NAT.\n\n#### **Connection to hierarchical SMT systems:**\n\nThank you for pointing out the connection to hierarchical SMT systems like Hiero. Similar to the idea of Hiero, we control the complexity of parsing PCFG by fixing the number of non-terminal symbols in the production rules and the size of the production set. We also consider the alignment of terminal symbols to reduce the uncertainty of parse trees. We will include relevant citations in our revised paper and discuss the potential lessons learned from hierarchical SMT that could apply to our method, such as constraints on production rules. \n\n#### **Target length estimation and Viterbi decoder efficiency:**\n\nIn our method, we use a upsample strategy to estimate the number of non-terminal symbols. $m = \\lambda L_x \\cdot 2^l + 2$.(Eq 5)\n\nDuring the decoding process, for every potential target length $L$, we use Viterbi decoding to find the optimal parse tree under the constraint of the target length $L$, as exhibited in Appendix A.4 of supplementary material. In accordance with Shao et al. [36], we reorder the output sentences of different lengths considering probability and length factors, ultimately outputting the sentence with the highest score.\n\nFor the Viterbi decoder, we agree that it might be less efficient on accelerators like GPUs and TPUs compared to greedy/beam/Viterbi decoding. For now, we write a custom CUDA Kernal to complete the computation of Viterbi decoding, and it still has room for improvement. The speed-ups reported in Table 2 are measured on a GPU (NVIDIA Tesla V100S-PCIE-32GB). Despite the potential efficiency concerns, our Viterbi decoder still achieves a good balance between translation quality and decoding latency, as shown in our experimental results.\n\nWe hope our responses have addressed your concerns and questions. We will incorporate these clarifications and improvements into the revised version of our paper. Thank you once again for your valuable feedback.\n"}]

Author rebuttal:
In response to the reviewers' suggestions, we have incorporated two sets of analysis experiments during the rebuttal stage. 

First, we conducted ablation studies to examine the impact of Glancing Training and Knowledge Distillation on our model's performance.GLAT consistently improves the translation performance of all models, while the gains from KD are more limited.
 
Secondly, we calculated the proportion of the maximum probability parse tree (path) that generates the references among all parse trees (paths) to investigate the benefits of the syntax tree learned by the RH-PCFG in comparison to the path in the Directed Acyclic Graph (DAG). We observed that the proportion of the highest probability parse tree for PCFG-NAT is generally higher than the proportion of the highest probability path for DA-Transformer. This demonstrates that in the trained RH-PCFG, the probabilities of parse trees are more concentrated. Under the same training set and learnable parameters, this indicates that the modeling approach of RH-PCFG is more in line with the intrinsic structure of sentences compared to the flattened assumption of Directed Acyclic Graph.


Meta-review: The paper proposes a new approach called PCFG-NAT to address the limitations of conventional Non-autoregressive Transformer (NAT) models. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and autoregressive (AT) models.

In all, the method is novel and reasonable. The main concerns are the empirical analyses. The author's response addressed the reviewers' concerns.
Decision: Accept (poster)