SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation Details For our method, we choose the  $l = 1$ ,  $\lambda = 4$  settings for Support Tree of RH-PCFG and linearly anneal  $\tau$  from 0.5 to 0.1 for the glancing training. For DA-Transformer [14], we use  $\lambda = 8$  for the graph size, which has comparable hidden states to our models. All models are optimized with Adam [20] with  $\beta = (0.9, 0.98)$  and  $\epsilon = 10^{-8}$ . For all models, each batch contains approximately 64K source words. All models are trained for 300K steps. For WMT14 En-De and WMT17 Zh-En, we measure validation BLEU for every epoch and average the 5 best checkpoints as the final model. For WMT16 En-Ro, we just use the best checkpoint on the valid dataset. We use the NVIDIA Tesla V100S-PCIE-32GB GPU to measure the translation latency on the WMT14 En-De test set with a batch size of 1. We implement our models based on the open-source framework of fairseq [27].
Translation Quality
As demonstrated in Table 2, PCFG-NAT outperforms CTC-GLAT [30] and DA-Transformer [14] in terms of translation quality, effectively reducing the performance gap between NAT and AT models. This indicates its ability to capture the intricate translation distribution inherent in the raw training data and efficiently learn from dependencies with reduced multi-modality. FA-DAT [26] builds upon
Decoding Strategy	WMT14	WMT17	WMT16	Speedup
En-De	De-En	En-Zh	Zh-En	En-Ro	Ro-En
Greedy Decoding	26.01	31.38	32.82	22.50	32.05	32.65	1.00×
Viterbi Decoding	27.02	32.29	33.60	23.40	32.72	33.07	0.91×
Table 3: Ablation Study on the effect of decoding strategies. We use the NVIDIA Tesla V100S-PCIE-32GB GPU to measure the speedup on the WMT14 En-De test set with a batch size of 1.
DA-Transformer and trains the model to maximize a fuzzy alignment score between the graph and reference, taking captured translations in all modalities into account. The same training objective can also be implemented in PCFG-NAT. Experimental results show that FA-DAT [26] achieves state-of-the-art NAT performance. In the future, we plan to incorporate the ideas from FA-DAT [26] into PCFG-NAT, enabling a fairer comparison and demonstrating the superiority of PCFG modeling.
Inference Latency
As shown Table 2, PCFG-NAT demonstrates a decoding speedup that is slightly lower than Vanilla NAT but significantly higher than both AT models and iterative NAT models. This observation indicates that PCFG-NAT effectively strikes a favorable balance between translation quality and decoding latency, achieving a desirable trade-off between the two factors.
4.3.1 Structure of Support Tree
In this section, we examine the impact of the max depth  $l$  of the local prefix tree and the upsample ratio  $\lambda$  in the support tree of RH-PCFG. Figure 5 presents the results. When the total number of non-terminal symbols is kept constant, incorporating a local prefix tree with max two layers leads to a significant decline in performance. We posit that an excessively large value of  $l$  leads to a significant increase in the number of potential parse trees, thereby making it more challenging for the model to learn complex structures from the data. As a result, the translation performance of the model may decline.
4.4 Decoding Strategy
To determine the optimal parse tree for a given sentence length, we employ Viterbi decoding. However, this approach incurs a certain computational cost in terms of decoding speed. To demonstrate the necessity of Viterbi decoding, we compare it with a greedy decoding strategy that always selects the rule with the highest
probability. The experimental results, presented in Table 3, indicate that while Viterbi decoding is only slightly slower than greedy decoding, it yields significantly better performance. Based on these findings, we assert that Viterbi decoding is necessary and do not recommend the use of greedy decoding.
Figure 5: Ablation study on the max depth  $l$  of local prefix tree and upsample rample ration  $\lambda$ . Notice the number of the non-terminal symbols  $m$  equals to  $\lambda L_x \cdot 2^l + 2$ , where  $L_x$  is the length of the source sentence.
4.5 Ablation Study for Distilled Data and GLAT
We conducted ablation studies to examine the impact of Glancing Training and Knowledge Distillation on our model's performance. GLAT consistently improves the translation performance of all models, while the gains from KD are more limited.
WMT16 En-Ro	BLEU
Transformer [40]	Baseline	34.26
+KD	34.72
DA Transformer [14]	Baseline	31.98
+GLAT	32.46
+GLAT & KD	32.40
PCFG-NAT	Baseline	31.94
+GLAT	32.72
+GLAT & KD	32.75
Table 4: Comparison of WMT16 En-Ro translation performance (BLEU scores) for Transformer, DA Transformer, and PCFG-NAT models with and without Glancing (GLAT) [30] and Knowledge Distillation (KD) techniques. Baseline means the models are trained on raw train sets without GLAT [30].
Figure 6: The frequency of part-of-speech and subwords generated from the main chain and local prefix tree respectively.
4.6 Syntax Analysis for Generated Parse Tree
To analyze the syntax information captured by PCFG-NAT, we conducted an analytical experiment on the WMT14 De-En test set. We utilized the averaged perceptron tagger, a part-of-speech tagging tool available in the nltk library [24], to assign part-of-speech tags to the translations. Subsequently, we calculated the frequency of part-of-speech tags for tokens directly generated from non-terminal symbols in both the main chain and the local prefix tree. Additionally, we counted the frequency of unfinished subwords ending with the symbol @@. To ensure comparability, the frequency scores were normalized by the number of tokens in the main chain and local prefix tree, respectively. The results are presented in Figure 6.
As depicted in Figure 6, the majority of tokens generated by non-terminal symbols in the local prefix tree correspond to the part-of-speech tags RB (adverb), JJ (adjective), NNP (proper noun, singular), and comma. Furthermore, the frequency of unfinished subwords in the local prefix tree symbols is notably higher compared to that in the main chain. It is well-known that RB, JJ, NNP, comma, and unfinished subwords are closely related to the local adjacent tokens. Hence, PCFG-NAT effectively captures the connection between these elements with the local prefix tree serving as complementary information to the main chain.
4.7 Probability Analysis of the Maximum Likelihood Path
We calculated the proportion of the maximum probability parse tree (path) that generates the references among all parse trees (paths) to investigate the benefits of the syntax tree learned by the RH-PCFG in comparison to the path in the Directed Acyclic Graph (DAG). We observed that the proportion of the highest probability parse tree for PCFG-NAT is generally higher than the proportion
WMT14	WMT17	WMT16	Average
En-De	De-En	En-Zh	Zh-En	En-Ro	Ro-En
DA-Transformer	0.6026	0.7565	0.4899	0.5611	0.7948	0.7124	0.6529
PCFG-NAT	0.6421	0.7718	0.4681	0.5886	0.7764	0.7160	0.6605
Table 5: In the Valid Set, for the DA-Transformer model, we calculated  $P(Y|X,A^{*}) / P(Y|X)$ , where  $X,Y$  represents the source text and the target text in the dataset,  $A^{*} = \text{argmax}_{A}P(A|X,Y)$ , and  $A$  represents a hidden state path. For the PCFG-NAT model, we calculated  $P(Y|X,T^{*}) / P(Y|X)$ , where  $T^{*} = \text{argmax}_{T}P(T|X,Y)$ , and  $T$  represents a parse tree.
of the highest probability path for DA-Transformer [14]. This demonstrates that in the trained RH-PCFG, the probabilities of parse trees are more concentrated. Under the same training set and learnable parameters, this indicates that the modeling approach of RH-PCFG is more in line with the intrinsic structure of sentences compared to the flattened assumption of Directed Acyclic Graph.
Gu et al. [13] accelerates neural machine translation with a non-autoregressive Transformer, which comes at the cost of translation quality. The major reason for the performance degradation is the multi-modality problem that there may exist multiple translations for the same source sentence. Many efforts have been devoted to enhancing the representation power of NAT. One thread of research introduces latent variables to directly model the uncertainty in the translation process, with techniques like vector quantization [16, 33, 3, 4], generative flow [25], and variational inference [38, 12]. Another thread of research explores some statistical information for intermediate prediction, including word fertility [13], word alignment [31, 39], and syntactic structures [2, 23]. Zhang et al. [43] explores the syntactic multi-modality problem in non-autoregressive machine translation The most relevant works to PCFG-NAT are CTC-based NAT [11, 22] and DA-Transformer [14, 37, 26], which enriches the representation power of NAT with a longer decoder. They can simultaneously consider multiple translation modalities by ignoring some tokens [11] or modeling transitions [14]. Fang et al. [7] proposes a non-autoregressive direct speech-to-speech translation model, which achieves both high-quality translations and fast decoding speeds by decomposing the generation process into two steps. Compared to previous NAT approaches, PCFG-NAT mitigates the issue of multi-modality in NAT by capturing structured non-adjacent semantic dependencies in the target language, leading to improved translation quality. Our work is also related to the line of work on incorporating structured semantic information into machine translation models[42, 19, 32, 1]. In contrast to these works, PCFG-NAT learns structured information unsupervised from data and can generate target tokens in parallel.
Non-autoregressive Transformer has higher inference speed but weaker expression power due to the lack of dependency modeling. In this paper, we propose PCFG-NAT to model target-side dependency by capturing the hierarchical structure of the sentence. Experimental results show that PCFG-NAT further enhances the translation performance of NAT models while providing a more interpretable approach for generating translations.
Further research and exploration are needed to overcome the limitations encountered when attempting to induce complex PCFGs from data. Developing more advanced learning algorithms or exploring alternative approaches for modeling and capturing the rich syntactic structures of language may be necessary to unlock the full potential of PCFG-NAT and further enhance its translation capabilities.
This work is partially supported by the National Key R&D Program of China(under Grant 2021ZD0110102), the NSF of China(under Grants 61925208), CAS Project for Young Scientists in Basic Research(YSBR-029) and Xplore Prize. We thank all the anonymous reviewers for their insightful and valuable comments.
[41] A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13(2):260-269, 1967. doi: 10.1109/TIT.1967.1054010.
[42] B. Wang, I. Titov, J. Andreas, and Y. Kim. Hierarchical phrase-based sequence-to-sequence learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8211–8229, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.563.
[43] K. Zhang, R. Wang, X. Tan, J. Guo, Y. Ren, T. Qin, and T.-Y. Liu. A study of syntactic multimodality in non-autoregressive machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1747-1757, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.nacl-main.126. URL https://aclanthology.org/2022.nacl-main.126.
A.1 Construction of Support Tree
src_length stands for the length of source sentence  $L_{x}$ .
layer_size stands for max depth of local prefix tree  $l$ .
upsample_ratio stands for hyperparameter  $\lambda$ .
Listing 1: Code for Constructing a Support Tree in Section 3.1.2
def build_support_tree(src_length, layer_size, upsample_lambda):
    main_chain_size = upsample_lambda * src_length + 1
    root = Node()
    root.leftChild = Node()
    now = root
    for i in range(main.chain_size):
        now.rightChild = Node()
        now = now.rightChild
        now.leftChild = full_binary_tree(layer_size)
    return root
Listing 2: Code for Constructing a Constructing a Full Binary Tree with depth layer_size
def full_binary_tree(layer_size):
    if layer_size <= 0:
        return None
    root = Node()
    root.leftChild = full_binary_tree(layer_size - 1)
    root.rightChild = full_binary_tree(layer_size - 1)
    return root
The function build_support_tree returns the root node of the support tree.
A.2 CYK Algorithm for RH-PCFG
$m, n$  stands for the numbers of non-terminal symbols and target sentence length.
OpProbs[a][i] represents  $P(y_{i}|V_{a})$ .
RuleProb [a] [b] [c] represents  $P(<V_b, V_c > |V_a)$ .
S[a][i][j] represents  $P(V_{a}\Rightarrow y_{i},\dots,y_{j})$  . S are initialized to 0.
Listing 3: Code for CYK Training of Local Prefix Tree in Section 3.3.1.
def local_prefix_tree_cyk(S, OpProbs, RuleProb, m, n, layer_size):  
    for i in range(0, n):  
        for a in range(0, m):  
            S[a][i][i] = OpProbs[a][i]  
    for span in range(2, 2**layer_size):  
        for a in range(0, m):  
            for i in range(0, n):  
                for b, c in Child(a):  
                    for k in range(i, i+span):  
                        S[a][i][i+span] += RuleProb[a][b][c] \  
                        * OpProbs[a][k] \  
                        * S[b][i][k-1] * S[c][k+1][i+span]
def main_chain_cyk(S, OpProbs, RuleProb, m, n, layer_size):
    for a in range(0, m):
        S[a][n-1][n-1] = OpProbs[a][n-1]
    for start in range(n-2, -1, -1):
        for a in range(0, m):
for i in range(0,n):
    for b,c in Child(a):
        for k in range(start, start+2**layer_size):
            S[a][start][n-1] += RuleProb[a][b][c] \
            * OpProbs[a][k] \
            * S[b][start][k-1] * S[c][k+1][n-1]
Listing 4: Code for CYK Training of Main Chain in Section 3.3.1.
After applying local_prefix_tree_cyk and main_chain_cyk, S[1][0][n-1] contains the value of  $P(V_{1} \Rightarrow y_{0}, \dots, y_{n - 1})$
A.3 Best Parse Tree for Glancing Training
S[a][i][j]represents  $P(V_{a}\Rightarrow y_{i},\dots,y_{j})$
Trace [a] [i] [j] = True means that derivation  $P(V_{a} \Rightarrow y_{i}, \dots, y_{j})$  is part of the maximum derivation of target sentence. Trace are initialized to False.
A  $[\kappa] = \mathsf{a}$  means that the token at position  $k$  in target sentence should be aligned to non-terminal symbol  $V_{a}$ . A are initialized to 0.
def best.parse_tree(S, A, Trace, OpProbs, RuleProb, m, n, layer_size):
    Trace[1][0][n-1] = True
    for start in range(0, n-2):
        for a in range(0, m):
            if Trace[a][start][n-1]:
                max_b, max_c, max_k = argmax(
                    RuleProb[a][b][c] \[
                    * OpProbs[a][k] \[
                    * S[b][start][k-1] * S[c][k+1][n-1] \]
                    for (b,c,k) in \
                        zip((Child(a), \)
                        range(start, start+2**layer_size)))
        )
    Trace[max_b][start][max_k-1] = True
    Trace[max_c][max_k+1][n-1] = True
    A[max_k] = a
    for span in range(2, 2**layer_size):
        for a in range(0, m):
            for i in range(0,n):
                if Trace[a][i][i+span]:
                    max_b, max_c, max_k = argmax(
                        RuleProb[a][b][c] \[
                        * OpProbs[a][k] \[
                        * S[b][i][k-1] * S[c][k+1][i+span] \)
                        for (b,c,k) \)
                        in zip(Child(a), range(i, i+span))
            )
    Trace[max_b][i][max_k-1] = True
    Trace[max_c][max_k+1][i+span] = True
    A[max_k] = a
Listing 5: Code for Glancing Training Best Parse Tree in Section 3.3.2.
After applying the algorithm, the tensor  $A$  contains the best alignment of non-terminal symbols and target tokens.
6   
7   
8   
9   
10   
11
A.4 Algorithm for Inference
S[a][i][j] represents  $P(V_{a}\Rightarrow y_{i},\dots,y_{j})$
MaxOpProbs [a] represents  $\max P(y|V_a)$ .
MaxOpTokens [a] represents  $\text{argmax}_y P(y|V_a)$ .
MAX_P [a] [L] are the probability of  $M_L^a$  of
$$
M _ {L} ^ {a} = \max  _ {<   b, c > \in \mathbf {C h i l d} (a), o \in \mathcal {T}, k = 1.. L - 1} P \left(V _ {a} \rightarrow V _ {b} o V _ {c}\right) M _ {L - 1 - k} ^ {b} M _ {k} ^ {c} \tag {16}
$$
and MAX_K [a] [L], MAX_B [a] [L], MAX_C [a] [L] are corresponding value of  $k, b, c$ .
output is the target sentence generated by viterbi decoding algorithm.
def viterbi_decoding(S, MaxOpProbs, RuleProb, m, n, layer_size, MAX_P, MAX_K, MAX_B, MAX_C):
    for start in range(n-2, -1, -1):
        for a in range(0, m):
            max_value, max_b, max_c, max_k = argmax(
                RuleProb[a][b][c] \[
                * MaxOpProbs[a] \[
                * MAX_P[b][k-1] * MAX_P[c][k+1][n-1] \]
            for (b,c,k) in \
                zip((Child(a), \)
                range(start, start+2**layer_size)))
        )
        MAX_P[a][n-start] = max_value
        MAX_K[a][n-start] = max_k
        MAX_B[a][n-start] = max_b
        MAX_C[a][n-start] = max_c
    for span in range(2, 2**layer_size):
        for a in range(0, m):
            max_value, max_b, max_c, max_k = argmax(
                RuleProb[a][b][c] \[
                * MaxOpProbs[a] \[
                * MAX_P[b][k-1] * MAX_P[c][span-k] \)
            for (b,c,k) \)
                in zip(Child(a), range(i, i+span))
        )
        MAX_P[a][span] = max_value
        MAX_K[a][span] = max_k
        MAX_B[a][span] = max_b
        MAX_C[a][span] = max_c
Listing 6: Code for Viterbi Decoding Algoihtm in Section 3.4.
def parse_tree(a, output, prefix_length, length, MaxOpTokens, MAX_B, MAX_C, MAX_K):
    max_b, max_c, max_k = MAX_B[a][length], MAX_C[a][length], MAX_K[a][length]
    output[prefix_length + max_k.length] = MaxOpTokens[a]
    parse_tree(max_b, prefix_length, max_k-1)
    parse_tree(max_c, prefix_length + max_k, length-max_k)
First, we utilize the viterbi_decoding method to obtain the values of MAX_B, MAX_C, MAX_K. Subsequently, considering the length constraint denoted as  $L$ , we employ the parse_tree function with parameters set as follows: starting index of 1 as  $V_{1}$ , output to store the gener-
ated target sentence, prefix_length with an initial value of 0, a maximum limit of  $L$  tokens, MaxOpTokens, MAX_B, MAX_C, MAX_K. This process effectively generates the desired target sentence.
A.5 Case Study
S-1954 Manchmal brauchen sie auch einfach nur sehr schnell Geld.
T-1954 Sometimes they just need the money very quickly
H-1954 1.4426950216293335 Sometimes they just need money very quickly.
S-2434 Und das stört mich am meisten
T-2434 That is what troubles me most of all.
H-2434 1.4426950216293335 And that bothers me most .
(b)
Figure 7: Two translation cases with generated parse tree from WMT14 De-En test set
To better understand the ability of PCFG-NAT to model structured semantic information, we select two cases from the WMT14 De-En test set and take a close look at the generated parse tree of PCFG-NAT. In Figure 7(a), we can see that in the sentence generated by PCFG-NAT, the content words lie in the main chain of the generated parse tree, and the words like just, very, which are supplements to words in the main chain, lie in the local prefix tree of the generated parse tree. In the generated parsing tree in Figure 7(b), the unfinished subword bo@@ becomes the left node of its suffixthers, which demonstrates that PCFG-NAT can learn the local structure of natural language and enhances the interpretability of NMT to a certain extent.