Title: Non-autoregressive Machine Translation with Probabilistic Context-free Grammar
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine translation

Non-autoregressive generation

Probabilistic Context-free Grammar
Abstract: Non-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages a specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates a deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation. Code is publicly available at https://github.com/ictnlp/PCFG-NAT.

Non-autoregressive Machine Translation with Probabilistic Context-free Grammar

Abstract

Non-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages a specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates a deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation. 2

1 Introduction

Autoregressive machine translation models rely on sequential generation, leading to slow generation speeds and potential errors due to the cascading nature of the process. In contrast, non-autoregressive Transformer (NAT) models [13] have emerged as promising alternatives for machine translation. The Vanilla NAT model assumes that the target tokens are independent of each other given the source sentence. This assumption enables the parallel generation of all tokens. However, it neglects the contextual dependencies among target language words, resulting in a severe multi-modality problem [13] caused by the rich expressiveness and diversity inherent in natural languages. Consequently, Vanilla NAT models exhibit significant performance degradation compared to their autoregressive counterparts.

Several approaches have been proposed to alleviate the multi-modality problem in NAT models while maintaining their computational efficiency. CTC-based NAT [11, 22] depicted in Figure 1(b) addresses this issue by implicitly modeling linear adjacent dependencies through token repetition and blank probability. This approach helps alleviate the multi-modality of token distributions at each position compared to Vanilla NAT. Similarly, DA-Transformer [14, 37, 26] shown in Figure 1(c) introduces a hidden Markov model and employs a directed acyclic graph to model sentence probability, enabling the model to capture dependencies between adjacent tokens and reduce the multi-modal nature of the distribution.

(b) CTC-NAT

(c) DA-Transformer

(d) PCFG-NAT

You

are

gonna

need

bigger

boat

You

<bnk>

are

gonna

need

<bnk>

a

bigger

boat

1

2

3

4

5

6

7

8

1

2

3

4

5

6

8

9

10

1

5

9

10

3

You

will

are

gonna

need

one

a

bigger

boat

2

4

6

8

1

2

3

4

5

6

8

9

10

You

will

are

gonna

need

one

bigger

boat

However, despite these enhancements, the modeling of only adjacent and unidirectional dependencies limits the model's ability to capture the rich semantic structures with non-adjacent and bidirectional dependencies present in natural language. For example, considering the token need as the previous token, the distribution of the next token is still affected by the issue of multi-modality, as the word boat can be modified by various rich prefixes in the corpus.

To overcome this constraints, we propose PCFG-NAT, a method that leverages context-free grammar to endow the model with the capability to capture the intricate syntactic and semantic structures present in natural language. For instance, as illustrated in Figure 1(d), PCFG-NAT can directly model the robust and deterministic correlation between non-adjacent tokens, such as You need boat . In the parse tree generated by PCFG-NAT for the target sentence, the nodes connected to the tokens You need boat constitute the main chain of the tree, capturing the left-to-right semantic dependencies through linear connections. Additionally, PCFG-NAT can learn relationships between modifiers, such as are gonna and a bigger , and the words they modify by constructing a local prefix tree for the nodes on the main chain. This allows for the modeling of right-to-left dependencies between the modified words and their modifiers.

We conduct experiments on major WMT benchmarks for NAT (WMT14 En ↔ De, WMT17 Zh ↔ En, WMT16 En ↔ Ro), which shows that our method substantially improves the translation performance and achieves comparable performance to autoregressive Transformer [40] with only one-iteration parallel decoding. Moreover, PCFG-NAT allows for the generation of sentences in a more interpretable manner, thus bridging the gap between performance and explainability in neural machine translation.

2 Background

In this section, we will present the necessary background information to facilitate readers' comprehension of the motivation and theoretical underpinnings of PCFG-NAT. In Section 2.1, we will provide a formalization of the machine translation task and introduce the concepts of autoregressive Transformer [40] and Vanilla NAT [13], which will serve as the basis for our proposed approach. Subsequently, in Section 2.2, we will provide a precise and rigorous definition of PCFG, accompanied by illustrative examples aimed at enhancing clarity and comprehension.

2.1 Non-autoregressive Translation

Machine translation can be formally defined as a sequence-to-sequence generation problem. Given source sentence X = { x 1 , x 2 , ..., x S } , the translation model is required to generate target sentence Y = { y 1 , y 2 , ..., y T } . The autoregressive translation model predicts each token on the condition of all previous tokens:

0

3

1

5

0

4

7

9

13

8

0

<blank>

You

<blank>

are

gonna

need

<blank>

a

bigger

boat

.

The autoregressive inference is time-consuming as it requires generating target tokens from left to right. To reduce the decoding latency, Gu et al. [13] proposes non-autoregressive Translation, which discards dependency among target tokens and predicts all the tokens independently:

Since the prediction does not rely on previous translation history, the decoder inputs of NAT are purely positional embeddings or the copy of source embeddings. Vanilla NAT trains a length predictor to determine the sentence length when decoding. During the training, the golden target length is used. During the inference, the predictor predicts the target length and the translation of the given length is obtained by argmax decoding.

2.2 Probabilistic Context-free Grammar

Probabilistic Context-free Grammar is built upon Context-Free Grammar (CFG). A CFG is defined as a 4-tuple:

where N is the set of non-terminal symbols, T is the set of terminal symbols, R is the set of production rules and S is the start symbol. PCFG extends CFG by associating each production rule r ∈ R with a probability P ( r ) . Production rules are in the form:

where A must be a single non-terminal symbol, and α is a string of terminals and/or nonterminals, α can also be empty. Starting with the sequence that only has start symbol S , we can replace any non-terminal symbol A with α based on production rule A → α . The derivation process repeats until the sequence only contains terminal symbols and the derived hierarchical structure can be seen as a Parse Tree . Those production rules and their probabilities can be applied regardless of the contexts, which provides Markov property for the joint probability of the parse tree. For every rule, A → α applied in a derivation, all of the symbols in α are the children nodes of A in order. The probability of a parse tree t is formulated as:

where R ( t ) is the set of production rules applied in t and n r is the number of times that r is applied. For sentence You are gonna need a bigger boat. , Figure 2 shows an example parse tree and how the sentence is derived from the start symbol V 1 .

3 The Proposed Method

In the following sections, we present the formal definition of a novel variant of PCFG designed for PCFG-NAT in Section 3.1 and the architecture of the PCFG-NAT model in Section 3.2. Finally, we describe the training algorithm and decoding algorithm for PCFG-NAT in Section 3.3 and Section 3.4.

3.1 Right Heavy PCFG

Despite the strong expressive capabilities of PCFGs, the parameter induction process becomes challenging due to the vast latent variable space encompassing the possible parse tree structures of a single sentence. Moreover, training PCFGs incurs significant time complexity [35]. To address these challenges, we propose a novel variant of PCFG called Right-Heavy PCFG (RH-PCFG). RHPCFG strikes a balance between expressive capabilities and computational complexity, mitigating the difficulties associated with parameter induction and reducing training time.

3.1.1 Derived Parse Tree of RH-PCFG

RH-PCFG enforces a distinct right-heavy binary tree structure in the generated parse tree. Specifically, for each non-terminal symbol in the tree, the height of the right subtree increases proportionally with the length of the derived sub-string of the non-terminal symbol, while the height of the left subtree remains constant. This characteristic significantly reduces the number of possible parse trees for a given sentence, thereby reducing training complexity.

In the context of a right-heavy binary tree, the traversal process begins with the root node and proceeds recursively by selecting the right child of the current node as the subsequent node to visit. This sequential path, known as the Main Chain , is guided by a design principle inspired by linguistic observations in natural language. Specifically, it reflects the tendency for sentence structures to display left-to-right connectivity, with local complements predominantly positioned to the left of the central core structure. Within this framework, the left subtree of each node along the main chain is referred to as its Local Prefix Tree . The parse tree in Figure 2 represents a right heavy tree. In this tree, V 1 , V 5 , V 9 , V 13 form the main chain, and their left subtrees correspond to their respective local prefix trees.

3.1.2 Support Tree of RH-PCFG

To make sure that all the parse trees are the right heavy tree, we first build up a Support Tree as the backbone for the parse trees derived from RH-PCFG. The construction of the support tree follows two steps: At first, given a source sentence with a length of L x and an upsample ratio of λ , we sequentially connect λL x +1 nodes with right links, which become the candidates for the main chain nodes in parse trees. Next, excluding the root node, we append a complete binary tree of depth l as the left subtree to each node. For the root node, only one left child is added. The added nodes represent local prefix tree candidates to the attached main chain candidates. Consequently, for a source sentence of length L x , the number of non-terminal symbols m of the RH-PCFG is

The nodes within the support tree are assigned sequential indexes through an in-order traversal, and these indexes correspond to the corresponding non-terminal symbols sharing the same index. Detailed construction algorithm can be found in Appendix A.1. Figure 3 shows an example of PCFG parse tree with L x = 3 , λ = 1 , l = 2 .

1

5

9

13

0

3

7

11

2

6

10

4

8

12

Main Chain

Candidates

Tree Candidates

In RH-PCFG, the set of non-terminals is N = { V 0 , ..., V m -1 } and the set of terminals T is equivalent to the vocabulary. The collection of production rules is represented by R , and the starting symbol is V 1 . The production rules in RH-PCFG have the following form:

̸

ϵ stands for an empty string. IsLeaf ( i ) indicates whether the non-terminal symbol V i corresponds to a leaf node in the support tree. InLeftReach ( j, i ) denotes that, in the support tree, the node corresponding to V j is in the left subtree of the node corresponding to V i . InRightReach ( k, i ) signifies that the node corresponding to V k is in the right subtree of the node corresponding to V i in the support tree, and if V i corresponds to a main chain node, V k has to correspond to a main chain node at the same time.

3.2 Architecture

The main architecture of PCFG-NAT is identical to that of the Transformer [40], with the additional incorporation of structures to model the probabilities of the PCFG. The input sentence to be translated is fed into the encoder of the model. Additionally, based on the length L x of the source sentence, we compute the number of corresponding non-terminal symbols m in the RH-PCFG. Then m positional embeddings are used as inputs to the decoder. Through multiple layers of self-attention modules with a global attention view, cross-attention modules that focus on distant information, and subsequent feed-forward transformations, we obtain m hidden states h 0 , ..., h m -1 . These hidden states are associated with non-terminal symbols of the same index and are utilized to calculate the rule probabilities related to each corresponding non-terminal symbol.

8

9

10

11

12

13

14

15

0

1

2

3

4

5

6

7

16

17

bigger

ship

one

a

bigger

boat

.

.

<blank>

You

will

are

gonna

need

one

a

.

.

8

9

10

11

12

13

14

15

0

1

2

3

4

5

6

7

16

17

Linear + Softmax

Output

Layer

Right

Heavy PCFG

Attention From

Encoder Output

Positional

Embedding

Decoder

Output

Stacked Decoder Layers

8

0

1

2

3

4

5

6

7

。

你

需要

⼀

艘

更

⼤

的

船

Stacked Encoder Layers

The probability of the rule P ( V i → a ) can be regarded as the word prediction probability at each position, and we can directly calculate it with the softmax operation. We use h i to denote the hidden state representation of the non-terminal symbol V i and use W o to denote the weight matrix of the output layer. Formally, we have:

For the rule P ( V i → V j aV k ) , we can independently consider the prediction probability of word a and the transition probability between the three non-terminal symbols. Formally, we decompose it into two parts:

The prediction probability P ( a | V i ) can be calculated with the softmax operation like in Equation 9. To normalize the distribution of P ( <V j , V k > | V i ) , we have to enumerate all valid combinations of j, k . Let Child ( i ) = { < o, u > | V i → V o aV u ∈ R} denotes the set of children combinations of the non-terminal symbol V i , we can formally define the transition probability as:

where W q , W l , W r are learnable model parameters.

3.3 Training

3.3.1 CYK Algorithm for RH-PCFG

We train PCFG-NAT with the maximum likelihood estimation, which sums up all the possible parse trees for the given sentence Y conditioned on source sentence X and model parameters θ :

In PCFG, the CYK algorithm [35] is employed to compute the probability of a given sentence derived from the start symbol. The CYK algorithm follows a bottom-up dynamic programming approach to fill in the table S . Each element S a i,j in the table represents the sum of probabilities of that V a derives the consecutive sub-string y i ...y j of the target sentence, denoted as P ( V a ⇒ y i ...y j ) . Therefore the size of table S is O ( mn 2 ) . Based on the definition of S a i,j , we have recursion formula

For every non-terminal symbol V a , the complexity filling all the S a i,j is the product of the choice of span < i, j > , splitting point k , and children symbols < b, c > .

Local Prefix Tree For the non-terminal symbol V a in a local prefix tree, the rules specify that all values of b and c are strictly fewer than the size of the left attached tree in the support structure, denoted as 2 l -1 < d = 2 l . Similarly, the variables i , j , and k are also strictly limited to values fewer than d , since it is the max length of sub-strings that V a can derives. Please note that l represents the max depth of the complete binary tree added as the left subtree to each node. Therefore the complexity filling all the S a i,j is O ( d 5 ) for V a and all V a in Local Prefix Tree should be O ( d 5 m ) where m is the number of non-terminal symbols.

Main Chain For the non-terminal symbol V a that serves as a main chain, based on the rules of the model, the variable b has strictly fewer than d choices, while c has a maximum of m/d choices. In a right-heavy parse tree, the symbols in the main chain are responsible for deriving a complete suffix of the target sentence as they are the right offspring of the root. Consequently, j is always equal to n -1 , and variable i has n choices. However, k has only d choices since the the length of derived sub-string of V b is less than d . Considering the filling of all S a i,j entries for the main chain symbols, the time complexity is O ( mdn ) . Summing up the complexities for all m/d main chain symbols, the total complexity is O ( m 2 n ) .

Based on the above analysis, the overall time complexity of the CYK algorithm for RH-PCFG is O ( d 5 n + m 2 n ) .Such complexity is within the acceptable range, while the training complexity of general form PCFG, such as Chomsky normal form [5], is O ( n 3 |R| ) . Detailed pseudocode for the training algorithm can be found in the Appendix A.2.

3.3.2 Glancing Training

Glancing training, as introduced in the work of Qian et al. [30], has been successfully applied in various NAT architectures [12, 14]. In the case of PCFG-NAT, we also incorporate glancing training. This training approach involves replacing a portion of the decoder inputs with the embeddings of the target sentence Y . To determine the ratio of decoder inputs to be replaced with target embeddings, we employ the masking strategy proposed by Qian et al. [30]. There is a mismatch between the decoder length m and target length n , so we have to derive an alignment between them to enable the glancing. To incorporate Glancing Training into PCFG-NAT, we adapt the approach by finding the parse tree T ∗ Y = arg max T Y ∈ Γ( Y ) P ( Y, T Y | X ) with the highest probability that generates the target sentence Y , where Γ( Y ) stands for all parse trees of Y . T ∗ Y can be traced by repalcing the sum opearation with max in the CYK algorithm. We provide the pseudocode in the Appendix A.3.

3.4 Inference

We propose a Viterbi [41] decoding framework for PCFG-NAT to find the optimal parse tree T ∗ = arg max T P ( T | X,L ) under the constraint of target length L . For each non-terminal symbol V a , we

maintain a record M a L that represents the highest probability among sub-strings of length L that can be derived by V a . Similar to the CYK algorithm, we can compute M a L in a bottom-up manner using dynamic programming, considering all non-terminal symbols V a and lengths L .

To recover the parse tree with the maximum probability given L , we need to keep track of the values of b , c , and k that lead to M a L and start the construction from M 1 L . The detailed pseudocode is provided in the Appendix A.4. According to Shao et al. [37], we rerank the output sentences with probability and length factors and then output the sentence with the highest score.

4 Experiments

4.1 Setup

Dataset We conduct our experiments on WMT14 English ↔ German (En-De, 4.5M sentence pairs),WMT17 Chinese ↔ English (Zh-En, 20M sentence pairs) and WMT16 English ↔ Romanian (En-Ro, 610k sentence pairs). We use the same preprocessed data and train/dev/test splits as Kasai et al. [18]. For all the datasets, we learn a joint BPE model [36] with 32K merge operations to tokenize the data and share the vocabulary for source and target languages. The translation quality is evaluated with sacreBLEU [29] for WMT17 En-Zh and tokenized BLEU [28] for other benchmarks.

Table 2: Results on raw WMT14 En-De, WMT17 Zh-En and WMT16 En-Ro. 'Iter' means the number of decoding iterations, and N is the length of the target sentence. The speedup is evaluated on the WMT14 En-De test set with a batch size of 1. '*' indicates the results of our re-implementation.

Implementation Details For our method, we choose the l = 1 , λ = 4 settings for Support Tree of RH-PCFG and linearly anneal τ from 0.5 to 0.1 for the glancing training. For DA-Transformer [14], we use λ = 8 for the graph size, which has comparable hidden states to our models. All models are optimized with Adam [20] with β = (0 . 9 , 0 . 98) and ϵ = 10 -8 . For all models, each batch contains approximately 64K source words. All models are trained for 300K steps. For WMT14 En-De and WMT17 Zh-En, we measure validation BLEU for every epoch and average the 5 best checkpoints as the final model. For WMT16 En-Ro, we just use the best checkpoint on the valid dataset. We use the NVIDIA Tesla V100S-PCIE-32GB GPU to measure the translation latency on the WMT14 En-De test set with a batch size of 1. We implement our models based on the open-source framework of fairseq [27].

4.2 Main Results

Translation Quality

As demonstrated in Table 2, PCFG-NAT outperforms CTC-GLAT [30] and DA-Transformer [14] in terms of translation quality, effectively reducing the performance gap between NAT and AT models. This indicates its ability to capture the intricate translation distribution inherent in the raw training data and efficiently learn from dependencies with reduced multi-modality. FA-DAT [26] builds upon

DA-Transformer and trains the model to maximize a fuzzy alignment score between the graph and reference, taking captured translations in all modalities into account. The same training objective can also be implemented in PCFG-NAT. Experimental results show that FA-DAT [26] achieves state-of-the-art NAT performance. In the future, we plan to incorporate the ideas from FA-DAT [26] into PCFG-NAT, enabling a fairer comparison and demonstrating the superiority of PCFG modeling.

Inference Latency

As shown Table 2, PCFG-NAT demonstrates a decoding speedup that is slightly lower than Vanilla NAT but significantly higher than both AT models and iterative NAT models. This observation indicates that PCFG-NAT effectively strikes a favorable balance between translation quality and decoding latency, achieving a desirable trade-off between the two factors.

4.3 Ablation study

4.3.1 Structure of Support Tree

In this section, we examine the impact of the max depth l of the local prefix tree and the upsample ratio λ in the support tree of RH-PCFG. Figure 5 presents the results. When the total number of non-terminal symbols is kept constant, incorporating a local prefix tree with max two layers leads to a significant decline in performance. We posit that an excessively large value of l leads to a significant increase in the number of potential parse trees, thereby making it more challenging for the model to learn complex structures from the data. As a result, the translation performance of the model may decline.

4.4 Decoding Strategy

To determine the optimal parse tree for a given sentence length, we employ Viterbi decoding. However, this approach incurs a certain computational cost in terms of decoding speed. To demonstrate the necessity of Viterbi decoding, we compare it with a greedy decoding strategy that always selects the rule with the highest

4.0

4.5

5.0

5.5

6.0

6.5

7.0

7.5

8.0

* 2^l

26.2

26.4

26.6

26.8

27.0

probability. The experimental results, presented in Table 3, indicate that while Viterbi decoding is only slightly slower than greedy decoding, it yields significantly better performance. Based on these findings, we assert that Viterbi decoding is necessary and do not recommend the use of greedy decoding.

4.5 Ablation Study for Distilled Data and GLAT

Weconducted ablation studies to examine the impact of Glancing Training and Knowledge Distillation on our model's performance.GLAT consistently improves the translation performance of all models, while the gains from KD are more limited.

Frequency

NN

IN

DT

NNP

JJ

NNS

.

VB

,

VBD

RB

VBZ

CC

VBN

TO

CD

PRP

VBP

MD

VBG

PRP

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Main Chain

4.6 Syntax Analysis for Generated Parse Tree

To analyze the syntax information captured by PCFG-NAT, we conducted an analytical experiment on the WMT14 De-En test set. We utilized the averaged perceptron tagger, a part-of-speech tagging tool available in the nltk library [24], to assign part-of-speech tags to the translations. Subsequently, we calculated the frequency of part-of-speech tags for tokens directly generated from non-terminal symbols in both the main chain and the local prefix tree. Additionally, we counted the frequency of unfinished subwords ending with the symbol @@ . To ensure comparability, the frequency scores were normalized by the number of tokens in the main chain and local prefix tree, respectively. The results are presented in Figure 6.

As depicted in Figure 6, the majority of tokens generated by non-terminal symbols in the local prefix tree correspond to the part-of-speech tags RB (adverb), JJ (adjective), NNP (proper noun, singular), and comma . Furthermore, the frequency of unfinished subwords in the local prefix tree symbols is notably higher compared to that in the main chain. It is well-known that RB , JJ , NNP , comma , and unfinished subwords are closely related to the local adjacent tokens. Hence, PCFG-NAT effectively captures the connection between these elements with the local prefix tree serving as complementary information to the main chain.

4.7 Probability Analysis of the Maximum Likelihood Path

We calculated the proportion of the maximum probability parse tree (path) that generates the references among all parse trees (paths) to investigate the benefits of the syntax tree learned by the RH-PCFG in comparison to the path in the Directed Acyclic Graph (DAG). We observed that the proportion of the highest probability parse tree for PCFG-NAT is generally higher than the proportion

of the highest probability path for DA-Transformer [14]. This demonstrates that in the trained RH-PCFG, the probabilities of parse trees are more concentrated. Under the same training set and learnable parameters, this indicates that the modeling approach of RH-PCFG is more in line with the intrinsic structure of sentences compared to the flattened assumption of Directed Acyclic Graph.

5 Related Works

Gu et al. [13] accelerates neural machine translation with a non-autoregressive Transformer, which comes at the cost of translation quality. The major reason for the performance degradation is the multi-modality problem that there may exist multiple translations for the same source sentence. Many efforts have been devoted to enhancing the representation power of NAT. One thread of research introduces latent variables to directly model the uncertainty in the translation process, with techniques like vector quantization [16, 33, 3, 4], generative flow [25], and variational inference [38, 12]. Another thread of research explores some statistical information for intermediate prediction, including word fertility [13], word alignment [31, 39], and syntactic structures [2, 23]. Zhang et al. [43] explores the syntactic multi-modality problem in non-autoregressive machine translation The most relevant works to PCFG-NAT are CTC-based NAT [11, 22] and DA-Transformer [14, 37, 26], which enriches the representation power of NAT with a longer decoder. They can simultaneously consider multiple translation modalities by ignoring some tokens [11] or modeling transitions [14]. Fang et al. [7] proposes a non-autoregressive direct speech-to-speech translation model, which achieves both highquality translations and fast decoding speeds by decomposing the generation process into two steps. Compared to previous NAT approaches, PCFG-NAT mitigates the issue of multi-modality in NAT by capturing structured non-adjacent semantic dependencies in the target language, leading to improved translation quality. Our work is also related to the line of work on incorporating structured semantic information into machine translation models[42, 19, 32, 1]. In contrast to these works, PCFG-NAT learns structured information unsupervised from data and can generate target tokens in parallel.

6 Conclusion

Non-autoregressive Transformer has higher inference speed but weaker expression power due to the lack of dependency modeling. In this paper, we propose PCFG-NAT to model target-side dependency by capturing the hierarchical structure of the sentence. Experimental results show that PCFG-NAT further enhances the translation performance of NAT models while providing a more interpretable approach for generating translations.

7 Limitations

Further research and exploration are needed to overcome the limitations encountered when attempting to induce complex PCFGs from data. Developing more advanced learning algorithms or exploring alternative approaches for modeling and capturing the rich syntactic structures of language may be necessary to unlock the full potential of PCFG-NAT and further enhance its translation capabilities.

References

R. Aharoni and Y. Goldberg. Towards string-to-tree neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 132-140, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-2021. URL https://aclanthology.org/P17-2021 .

N. Akoury, K. Krishna, and M. Iyyer. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1269-1281, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1122. URL https://aclanthology.org/P19-1122 .

Y. Bao, S. Huang, T. Xiao, D. Wang, X. Dai, and J. Chen. Non-autoregressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5749-5759, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.naacl-main.458. URL https://aclanthology.org/2021.naacl-main.458 .

Y. Bao, H. Zhou, S. Huang, D. Wang, L. Qian, X. Dai, J. Chen, and L. Li. latent-GLAT: Glancing at latent variables for parallel text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8398-8409, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.575. URL https://aclanthology.org/2022.acl-long.575 .

N. Chomsky. On certain formal properties of grammars. Information and Control , 2(2):137167, 1959. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019-9958(59)90362-6. URL https://www.sciencedirect.com/science/article/pii/S0019995859903626 .

C. Du, Z. Tu, and J. Jiang. Order-agnostic cross entropy for non-autoregressive machine translation. CoRR , abs/2106.05093, 2021. URL https://arxiv.org/abs/2106.05093 .

Q. Fang, Y. Zhou, and Y. Feng. Daspeech: Directed acyclic transformer for fast and high-quality speech-to-speech translation. In Advances in Neural Information Processing Systems , 2023.

M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6112-6121, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1633. URL https:// aclanthology.org/D19-1633 .

M. Ghazvininejad, V. Karpukhin, L. Zettlemoyer, and O. Levy. Aligned cross entropy for non-autoregressive machine translation. CoRR , abs/2004.01655, 2020. URL https://arxiv. org/abs/2004.01655 .

M. Ghazvininejad, O. Levy, and L. Zettlemoyer. Semi-autoregressive training improves maskpredict decoding. CoRR , abs/2001.08785, 2020. URL https://arxiv.org/abs/2001. 08785 .

A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural nets. In ICML '06: Proceedings of the International Conference on Machine Learning , 2006.

J. Gu and X. Kong. Fully non-autoregressive neural machine translation: Tricks of the trade. CoRR , abs/2012.15833, 2020. URL https://arxiv.org/abs/2012.15833 .

J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations , 2018. URL https: //openreview.net/forum?id=B1l8BtlCb .

F. Huang, H. Zhou, Y. Liu, H. Li, and M. Huang. Directed acyclic transformer for nonautoregressive machine translation. In Proceedings of the 39th International Conference on Machine Learning, ICML 2022 , 2022.

X. S. Huang, F. Perez, and M. Volkovs. Improving non-autoregressive translation models without distillation. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=I2Hw58KHp8O .

L. Kaiser, S. Bengio, A. Roy, A. Vaswani, N. Parmar, J. Uszkoreit, and N. Shazeer. Fast decoding in sequence models using discrete latent variables. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2390-2399. PMLR, 10-15 Jul 2018. URL http://proceedings.mlr.press/v80/kaiser18a.html .

J. Kasai, J. Cross, M. Ghazvininejad, and J. Gu. Parallel machine translation with disentangled context transformer. CoRR , abs/2001.05136, 2020. URL https://arxiv.org/abs/2001. 05136 .

J. Kasai, J. Cross, M. Ghazvininejad, and J. Gu. Non-autoregressive machine translation with disentangled context transformer. In International conference on machine learning , pages 5144-5155. PMLR, 2020.

Y. Kim. Sequence-to-sequence learning with latent neural grammars. Advances in Neural Information Processing Systems , 34:26302-26317, 2021.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http: //arxiv.org/abs/1412.6980 .

J. Libovický and J. Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. CoRR , abs/1811.04719, 2018. URL http://arxiv. org/abs/1811.04719 .

J. Libovický and J. Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3016-3021, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1336. URL https: //www.aclweb.org/anthology/D18-1336 .

Y. Liu, Y. Wan, J. Zhang, W. Zhao, and P. Yu. Enriching non-autoregressive transformer with syntactic and semantic structures for neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 1235-1244, Online, Apr. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.105. URL https://aclanthology.org/2021.eacl-main. 105 .

E. Loper and S. Bird. Nltk: The natural language toolkit, 2002. URL https://arxiv.org/ abs/cs/0205028 .

X. Ma, C. Zhou, X. Li, G. Neubig, and E. Hovy. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4282-4292, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1437. URL https://www. aclweb.org/anthology/D19-1437 .

Z. Ma, C. Shao, S. Gui, M. Zhang, and Y. Feng. Fuzzy alignments in directed acyclic graph for non-autoregressive machine translation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=LSz-gQyd0zE .

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. CoRR , abs/1904.01038, 2019. URL http: //arxiv.org/abs/1904.01038 .

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040 .

M. Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186-191, Belgium, Brussels, Oct. 2018. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ W18-6319 .

L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1993-2003, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.155. URL https://aclanthology.org/2021.acl-long.155 .

Q. Ran, Y. Lin, P. Li, and J. Zhou. Guiding non-autoregressive neural machine translation decoding with reordering information. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021 , pages 13727-13735. AAAI Press, 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/17618 .

P. Rastogi, R. Cotterell, and J. Eisner. Weighting finite-state transductions with neural context. Association for Computational Linguistics, 2016.

A. Roy, A. Vaswani, A. Neelakantan, and N. Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv:1805.11063 , 2018.

C. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoregressive machine translation with latent alignments. CoRR , abs/2004.07437, 2020. URL https://arxiv.org/abs/2004. 07437 .

I. Sakai. Syntax in universal translation. In EARLYMT , 1961.

R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715-1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162 .

C. Shao, Z. Ma, and Y. Feng. Viterbi decoding of directed acyclic transformer for nonautoregressive machine translation. In Findings of EMNLP 2022 , 2022.

R. Shu, J. Lee, H. Nakayama, and K. Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In AAAI , 2020.

J. Song, S. Kim, and S. Yoon. AligNART: Non-autoregressive neural machine translation by jointly learning to estimate alignment and translate. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1-14, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.emnlp-main.1. URL https://aclanthology.org/2021.emnlp-main.1 .

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips. cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory , 13(2):260-269, 1967. doi: 10.1109/TIT. 1967.1054010.

B. Wang, I. Titov, J. Andreas, and Y. Kim. Hierarchical phrase-based sequence-to-sequence learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8211-8229, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.563 .

K. Zhang, R. Wang, X. Tan, J. Guo, Y. Ren, T. Qin, and T.-Y. Liu. A study of syntactic multimodality in non-autoregressive machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1747-1757, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.126. URL https:// aclanthology.org/2022.naacl-main.126 .

A Appendix

A.1 Construction of Support Tree

src_length stands for the length of source sentence L x .

layer_size stands for max depth of local prefix tree l .

upsample_ratio stands for hyperparameter λ .

The function build_support_tree returns the root node of the support tree.

A.2 CYK Algorithm for RH-PCFG

m,n stands for the numbers of non-terminal symbols and target sentence length.

layer_size stands for max depth of local prefix tree l .

OpProbs[a][i] represents P ( y i | V a ) . RuleProb[a][b][c] represents P ( < V b , V c > | V a ) .

S[a][i][j] represents P ( V a ⇒ y i , ..., y j ) . S are initialized to 0.

After applying local_prefix_tree_cyk and main_chain_cyk , S[1][0][n-1] contains the value of P ( V 1 ⇒ y 0 , ..., y n -1 )

A.3 Best Parse Tree for Glancing Training

m,n stands for the numbers of non-terminal symbols and target sentence length.

layer_size stands for max depth of local prefix tree l .

OpProbs[a][i] represents P ( y i | V a ) .

RuleProb[a][b][c] represents P ( < V b , V c > | V a ) .

S[a][i][j] represents P ( V a ⇒ y i , ..., y j ) .

Trace[a][i][j]=True means that derivation P ( V a ⇒ y i , ..., y j ) is part of the maximum derivation of target sentence. Trace are initialized to False .

A[k]=a means that the token at position k in target sentence should be aligned to non-terminal symbol V a . A are initialized to 0.

After applying the algorithm, the tensor A contains the best alignment of non-terminal symbols and target tokens.

A.4 Algorithm for Inference

m,n stands for the numbers of non-terminal symbols and target sentence length.

layer_size stands for max depth of local prefix tree l .

MaxOpProbs[a] represents maxP ( y | V a ) .

MaxOpTokens[a] represents argmax y P ( y | V a ) .

RuleProb[a][b][c] represents P ( < V b , V c > | V a ) .

MAX_P[a][L] are the probability of M a L of

and MAX_K[a][L],MAX_B[a][L],MAX_C[a][L] are corresponding value of k, b, c .

output is the target sentence generated by viterbi decoding algorithm.

First, we utilize the viterbi_decoding method to obtain the values of MAX_B, MAX_C, MAX_K . Subsequently, considering the length constraint denoted as L , we employ the parse_tree function with parameters set as follows: starting index of 1 as V 1 , output to store the gener-

ated target sentence, prefix_length with an initial value of 0, a maximum limit of L tokens, MaxOpTokens,MAX_B,MAX_C,MAX_K . This process effectively generates the desired target sentence.

A.5 Case Study

(a)

(b)

S-1954

Manchmalbrauchen sie aucheinfachnur sehrschnell Geld.

T-2434

S-2434

Thatiswhattroubles memostof all.

Unddasstortmichammeisten.

T-1954S

H-1954 1.4426950216293335

Sometimestheyjustneed themoneyveryquickly.

Sometimestheyjustneedmoneyveryquickly.

H-2434 1.4426950216293335

And thatbothersmemost.

1

1

3

3

23

15

26

27

57

32

33

57

87

61

86

95

79

97

81




Someti-

mes

they

just

need

money

very

quickly







And

that

bo@@

thers

me

most




To better understand the ability of PCFG-NAT to model structured semantic information, we select two cases from the WMT14 De-En test set and take a close look at the generated parse tree of PCFG-NAT. In Figure 7(a), we can see that in the sentence generated by PCFG-NAT, the content words lie in the main chain of the generated parse tree, and the words like just, very , which are supplements to words in the main chain, lie in the local prefix tree of the generated parse tree. In the generated parsing tree in Figure 7(b), the unfinished subword bo@@ becomes the left node of its suffix thers , which demonstrates that PCFG-NAT can learn the local structure of natural language and enhances the interpretability of NMT to a certain extent.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work attempts to learn hierarchical structure in form of a PCFG as part of a non-autoregressive translation (NAT) system. The intuition is that the PCFG relaxes the conditional independence assumption between target tokens in vanilla NAT. The authors argue that related approaches with the same objective focus on local dependencies, whereas their hierarchical approach can also represent long range dependencies. Results seem to indicate a good balance between latency and translation quality.\n\nSTRENGTHS:\nThe paper is well written and reproducible. The motivation for learning hierarchical dependencies in NAT is strong. The results are a bit on the slim side, but the novelty (as far as I am aware) of using hierarchy in NAT compensates for that. It is nice to see that a PCFG can be learned in that way. I appreciate the well designed ablation studies. The simplifying assumptions for the PCFG (binary and right-heavy) make sense.\n\nWEAKNESSES:\nI felt that the architecture under-delivers slightly on the claim of introducing PCFG-style dependencies to NAT, because production rule probabilities with a mix of non-terminals and terminals are still decomposed to the transition and terminal probabilities. The conditional independence assumption underlying Eq. 10 means that transition probabilities are independent of the terminal. Similarly, even if the PCFG introduces a link from y_i to y_j token, it doesn't mean that y_j really depends on y_i (because Eq. 9 is still used), just that the transition probability depends on h_i and h_j. \n\nThe connection to hierarchical SMT systems like Hiero is not made - please add citations.\n\nQUESTIONS:\n- Could you provide more details about the target length / number of non-terminal estimation in your system?\n- Do you think there are any lessons learnt in hierarchical SMT that would be applicable here? For example, the constraints on production rules in Hiero?\n- NAT and AT greedy/beam search lends itself particularly well to GPU/TPU decoding, whereas the Viterbi decoder seems harder to support efficiently on accelerators. Can you comment on that? Are the Table 2 speed-ups measured on the CPU, GPU, or TPU?\n\nLIMITATIONS:\nAddress last question above, if necessary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PCFG-NAT, which alleviates the issue of conditional independent modeling through probabilistic context-free grammar. PCFG-NAT leverages syntax information to capture bidirectional dependencies among predicted tokens. In this way, this method also provides a better explanation of the translation process. Experiments on several benchmarks demonstrate the effectiveness of the proposed method in bridging the performance gap with autoregressive counterparts.\n\nSTRENGTHS:\nThis paper proposes a novel PCFG-NAT method that introduces context-free grammar to capture dependencies in NAT. The authors develop systematic designs for it, including a lightweight variant called RH-PCFG, architecture support, training strategy, and decoding algorithms.\n\nIn addition, popular NAT techniques, such as glancing training and Viterbi decoding, can be integrated into PCFG-NAT.\n\nWEAKNESSES:\nThe main concern for me is the wrong comparison. Table 2 shows the main results of the proposed method and related work. However, I notice that the authors omit some of the earlier methods that achieve better performance, such as [1][[2].\nIn addition, some reported results are inconsistent with the original paper, like the CTC+GLAT method reported in the paper [3] achieves 26.39 and 29.54 on WMT 14 En-De and De-En datasets, but 25.02 and 29.14 in this paper. If the report is reasonable, the author must explain it for clarification.\n \nI admit that the proposed method is novel, but it is complex. Simple methods, like glancing training [3] and DSLP [2], achieve comparable performance without complex modifications.\nAccording to Figure 2, PCFG is used only before the final prediction. I suspect that this design only helps slightly because the decoder layers are unchanged. Due to the complex design, it is difficult to design the ablation study to examine the role of PCFG.\n \n \n[1] Fully non-autoregressive neural machine translation: tricks of the trade.\n\n[2] Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision.\n\n[3] Glancing Transformer for Non-Autoregressive Neural Machine Translation.\n\nQUESTIONS:\nCan the authors elaborate the effect of each technique by detailed ablation studies? I am curious about the performance and speedup without glancing training or RH-PCFG.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to rely on PCFG to incorporate the most likely parsing tree into the non-autoregressive transformer model as a way to mitigate multi-modality problem. Since considering all possible parsing trees is computationally expensive, the authors propose a right-heavy PCFG based on the assumption that language tends to show left-to-right connectivity. The experiments on the three well-known WMT datasets show their method outperforms others while keeping the speed-up provided by NAT.\n\nSTRENGTHS:\n- Novel approach to capture dependencies between tokens for NAT using PCFG \n- Experimental results show that method allows to improve NAT translation in terms of BLEU score\n\nWEAKNESSES:\n- Limited analysis:\n  - method is based on the assumption that left-to-right connectivity is the best for language and that \"RH-PCFG strikes a balance between expressive capabilities and computational complexity\"; however, no experiments support this claim. \n  - Also, Figure 5 shows that increasing the local prefix tree helps in terms of BLEU. \n- Missing references and comparison:\n  - Follow-up work on DAG Ma, Zhengrui et al. \"Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation.\" (ICLR 2023, published Feb 1st). I understand this work appeared a few months before your submission, but it would be fair to compare.\n- Lack of clarity\n  - I have a hard time understanding the central concept of training and evaluating NAT model, which hurts reproducibility (please see questions)\n\nQUESTIONS:\n- What is your full objective function? During training, you rely on the target length, but you need to obtain the length from the model during decoding. Also, it is not clear to me if you only use the sum of LL of all possible trees as your only objective to train the translation model.\n- Why do you use full vocabulary and not `<eos>` symbol as terminal states?\n- Figure 5: it's unclear what contributes more, $\\lambda$ or $l$. And in your plot, there is no plateau, meaning it could be better if you increase it further.\n- I assume you do not use KD for your experiments. However, it is not explicitly stated. That is a valuable result that can strengthen your paper. Also, since KD is widely used, it would be interesting to see if KD can further improve your model\n\nTypos & Grammar & Style:\n- line 243: Figure instead of Table\n- line 23: to mitigate.... to alleviate: duplication of the same meaning\n- line 97: `know as` - I am confused here, I assume you propose this method, so it's not known before\n- line 294: space after `quality.`\n\nLIMITATIONS:\n- Assumption that left-to-right dependency is more valuable for languages (which potentially false for some languages) is not addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach called PCFG-NAT to address the limitations of conventional Non-autoregressive Transformer (NAT) models. PCFG-NAT uses a specially designed Probabilistic Context-Free Grammar (PCFG) to capture complex dependencies among output tokens and enhance the expression power of NAT models. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and autoregressive (AT) models. Additionally, PCFG-NAT facilitates a deeper understanding of the generated sentences and addresses the lack of satisfactory explainability in neural machine translation.\n\nSTRENGTHS:\nThe paper proposes a very novel NAT model that performs comparably or even better than state-of-the-art NAT models.\n\nWEAKNESSES:\n1. The analysis of training complexity is not thorough enough. \na. It does not show the training inference of the proposed model compared with baselines.\nb. Although the paper mentions that RH-PCFG is faster than PCFG, there is no quantitative analysis and no empirical evidence to support this claim. It would be helpful to compare the training complexity with the baseline and provide a more in-depth analysis.\nc. How is d set in experiments? what is the relationship between d and m (or n)?\n\n2. There is a lack of an in-depth explanation of the generated sentences from the main chain and prefix tree in the syntax analysis experiment. Only the analysis based on the frequency is not intuitive.  Additionally, the analysis of the syntax tree of the generated sentences in the experiment is not adequate, because some of such sentences suffer from grammar issues. It would be better to analyze the syntax tree of the reference translation based on the given <X, R> and calculate p(T|X,R).\n\n3. Although RH-PCFG has strong expression power, the performance improvement is not significant. It would be helpful to explain the main reasons for this and whether there is potential to further explore this framework.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This article proposes a new NAT training method based on PCFG. To address the limitations of critical and unidirectional dependencies, the authors provide a new way of establishing dependencies from the perspective of CFG, hoping to capture both grammatical and semantic information. Therefore, this article simplifies PCFG into Right Heavy PCFG and, in the process of training and decoding generation in NAT, achieves a further reduction in the translation gap between NAT and autoregressive models by adding a derivation process based on Right Heavy PCFG.

STRENGTHS:
An interesting algorithm based on PCFG is proposed, which reduces many limitations on previous methods and provides more semantic and grammatical dependencies.

WEAKNESSES:
1.	This paper claims to endow the capability in capture more complex dependencies, but the experimental part does not give a comparison and analysis of this aspect.
2.	The training loss function is not clear.

QUESTIONS:
1.	How does glancing training strategies influence the PCFG-NAT training?
2.	Which languages does the Right Heavy PCFG proposed in this paper suitable for?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07