Prior Knowledge Representation for Self-Attention Networks
Authors: Kehai Chen
Rui Wang
Masao Utiyama
Eiichiro Sumita
Venue:  (2021)
Keywords: Prior Knowledge
Universal Representation
Self-Attention Networks
Neural Machine Translation

Abstract: Self-attention networks (SANs) have shown promising empirical results in various natural language processing tasks. Typically, it gradually learning language knowledge on the whole training dataset in parallel and stacked ways, thereby modeling language representation. In this paper, we propose a simple and general representation method to consider prior knowledge related to language representation from the beginning of training. Also, the proposed method allows SANs to leverage prior knowledge in a universal way compatible with neural networks. Furthermore, we apply it to one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data, respectively, thereby enhancing the language representation. Experimental results on WMT14 English-to-German and WMT17 Chinese-to-English translation tasks demonstrate the effectiveness and universality of the proposed method over a strong Transformer-based baseline. 

Under review as a conference paper at ICLR 2021
PRIOR KNOWLEDGE REPRESENTATION FOR SELF-ATTENTIO NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Self-attention networks (SANs) have shown promising empirical results in various natural language processing tasks. Typically, it gradually learning language knowledge on the whole training dataset parallel and stacked ways, thereby modeling language representation. In this paper, we propose a simple and general representation method to consider prior knowledge related to language representation from the beginning of training. Also, the proposed method allows SANs to leverage prior knowledge in universal way compatible with neural networks. Furthermore, we apply it to one prior word frequeney knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data, respectively, thereby enhancing the language representation. Experimental results on WMT14 English-to-Genn to-Gemmnn and WMT17 Chinese-to English translation tasks demonstrate the effectiveness and universality of the proposed method over a strong Transformer-based baseline.
INTRODUCTION
Self-attention networks (SANs) have attracted increasing attention the natural language processing community.
Instead of using complex recurrent or convolutional neural 2014 Bahdanau et al., 2015). SANs first use the positional encoding mechanism /ehhring al. positional embedding is then added to corresponding word embedding to obtain an input representation, based on which SANs perform (multi-head) and stack (multi-layer) self-attentive functions (Vaswani et al. 2017) in parallel to learn language representation. The SAN-based models are iteratively optimized to model language knowledge on the whole training dataset, which has achieved Nate-of-the-an performance in many natural language processing tasks pairs (Barrault
networks (Sutskever et al.
(2017) to encode order dependencies in the language. The learned
al.) 2019 Oepen et al. 2019. Weissenbacher Gonzalez- Herrandez) 2019 Demner-Fushman al. 2019)
Despite the success, SANs gradually model language knowledge on the batch-level datasets and does not consider the prior knowledge on the whole dataset from the beginning of training, which may decrease its language representation capability. For example, the SAN-based neural machine translation (NMT) model often mistranslate into words that seem be natural in the target language sentence, but do not reflect the original meaning of the source language sentence (Arthur et al.) 2016 Wang al. 201 7a). As result, the NMT model produces fluent yet sometimes inadequate translations (Tu et al. 2016 2017). To address this issue, recent studies explored the prior knowledge which has the stringer ability model the fluency of translation in traditional SMT (Koehn al.) 2003 Liu et al., 2006] | 2007 Liu et al.) 2007). Take the prior translation lexicon knowledge as an example. Arthur al. (2016) directly biased or interpolated the bilingual lexicon translation distribution with the output of the softmax layer of NMT to improve the translations of infrequency words. An auxiliary classifier (Zhao et al.] 2018 Wang al. 2018) were employed to integrate the SMT recommendations with NMT generations to generate the faithful translation. In addition, the phrase translation rules was as the recommendation memory to make better predictions in NMT (Wang al. 2017b Zhao et al. 2018). Although these studies successfully improved the issue of inadequate translations in NMT, they tended to focus on exploring the prior translation lexicon knowledge by using their specific methods. In other words, these unique methods make it difficult to explore other prior knowledge in a universal way and to determine which of the prior knowledge and the unique method this improvement comes from. Meanwhile, these studies directly

---

Under review as a conference paper at ICLR 2021
utilized the probability distribution of the prior knowledge and lacked the neural network's ability to semantically generalize, while will further hinder the language representation ability of SANs.
In this paper, propose simple and general representation method to introduce the prior knowledge into SANs. particular, we package the prior knowledge related with one source sentence to continuous space matrix, which allows SANs utilize the prior knowledge from the beginning of training, thereby better performing language representation in a universal way compatible with neural networks. To maintain the simplicity and flexibility of the SANs, we use the prior knowledge representation in parallel and stacked ways to learn the representation of the input sentence. Furthermore, use the proposed method explore one prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data, respectively. Empirical results on two widely used translation data sets, including WMT14 Geemmnn and WMT17 Chinese->English, to verify the effectiveness and universality of the proposed method over strong Transformer-bast baseline.
SELF-ATTENTION NETWORKS
The self-attentio networks (SANs) (Vaswani al.] 2017) is composed of a stack of N identical layers, each of which includes two sub-layers. Formally, given input sentence with the length J, X=(x). 22 ... x.). , the positional encoding mechanism (Gehring et al. 2017 used to compute positional embedding of each word based on its position index. The positional embedding is then added to the corresponding word embedding as an combined embedding. thereby gaining a sequence of input representation HJ = V2,
V2). Moreover, the stacked SANs is organized follows:
H" LN(SelfAut (Q" K"-1 Vm-1) + HH""),
Hº LN(FFN" H
where SelfAtt(()), LN(-), and FFN" (.) are self-attention module, layer normalization (Ba et al. 2016), and feed-forward network for the n-th identical layer, respectively. Q" Qa-1_--- kn=1 and V are query, key, and value matrices that are transformed from the (n-1)-th layer H For example, Qa K and V packed from the Hº learned by the positional encoding mechanism (Gehring al. 2017). In particular, SelfAtt,. (-) is applied on the (Q"-1 K"-1 vn-1) of the n-1 layer:
SelfAn" (Q K v=-1) softmax Km-1
dmodel Vn-1
V"
where dmodet is the dimension size of the query and key vectors. As a result, the output of the N-th layer H N is the representation of the input sentence. Moreover, the self-attention mechanism can be further refined as multi-head self-attention to jointly attend the information from different representation sub-spaces at different positions.
3 PRIOR KNOWLEDGE REPRESENTATION
In this section, we propose simple and general representation method to encode the prior knowledge, which allows SANs to model the prior knowledge in manner compatible with neural networks. Given input sentence X= 22, ,I associated prior knowledge as a matrix M:
,JJ with the length J, we represent the
m
M
(3)
where each row denotes the prior knowledge related with word xy and each element m fixed size vector. Also, M is packed into key and value matrix pair {K, V) for the prior knowledge. The prior {K, V} and the current Q are the input to the self-attention mechanism (see 2 to learn prior knowledge representation PK for the input sentence X:
PK LN(SeIfAut Q, IK, V) H),
2

---

Under review as a conference paper at ICLR 2021
where Q is transformed from the current sentence representation H. for example, the n-th layer output H of the stacked SANs in Eq. ( Finally, PK is the expected prior knowledge representation (PKR). Later, we will apply the proposed method to prior word frequency knowledge for the monolingual data and other prior translation lexicon knowledge for the bilingual data, respectively, thereby enhancing the language representation.
3.1 WORD FREQUENCY BASED PRIOR KNOWLEDGE REPRESENTATION
Recent studies have shown the effectiveness of the prior word frequency information for improving
the translations of content words (Arthur al. 2016 Wang al. 2018 He et al. 2019 Chen al.,
2020). Intuitively, these content words generally have more meanings and lower word frequency than function words in one sentence. To capture the prior knowledge, we use word frequency information to enhance the importance of content words in the sentence representation. Inspired by the previous studies (Setiawan et al., [007]] 2009 Zhang Zhao 2013] Chen et al., 22020) we use the frequency of each word w on the monolingual corpus F to distinguish content words and function words in the input sentence. To this end, let the B most frequent words in the monolingual corpus denote the function words while the remaining words are regarded content words for the input sentence. In other words, let the prior word frequency knowledge of each word denote a binary mask:
0, 3y zy
fg
=
(5)
1,
B B
Furthermore, the prior word frequency knowledge of the input sentence is a diagonal matrix according to E3.
fi
0
f2
M
(6)
frJ
Finally, My to mask the representation of function words in H. and thereby gain key and
value matrices {Ky, Vr} } to learn a prior word frequency based PKR PK for the input sentence:
PK LN(SelfAut QQ, Ky, Vp) H),
where Q is transformed from the current sentence representation H, for example, H in Eq
3.2 BILINGUAL TRANSLATION LEXICON BASED PRIOR KNOWLEDGE REPRESENTATION
We apply the proposed PKR method into the prior bilingual translation lexicons in SMT (Brown jet al. 1993). Discretely, each source word Σÿ has multiple associated target translation options retrieved from the bilingual translation lexicon table. For simplicity, translation options as the prior translation knowledge of word 2y are remained according to their lexicon translation probabilities and the entry corresponds to the target word. Formally, all retrieved lists for the input sentence X are constructed as a J x K x dmodel prior translation knowledge matrix My according to Eq. (3):
select the top T, target
t
t
t
M
(8)
where each row corresponds to the retrieved list of the word Iy, and the entry corresponds to the embedding of the target word. particular, when the number of the associated target words for one source word n, is less than L, we use the embedding of special placeholder "y" to fill the j-th row in My up to L. Also, the My is packed into a key and value matrix pair {Kr, V (Q, Ky, and Wr} are the input to Eq. (4) to learn a prior translation knowledge representation PK7 for the source sentence:
PKr LN(SelfAut( Q, Ky, VT) H).
(9)
where Q is transformed from the current sentence representation H, for example, H in Eq.
4 SANS WITH PRIOR KNOWLEDGE REPRESENTATION
3

---

Under review as a conference paper at ICLR 2021
Intuitively, the prior knowledge assists SANs to enhance the language rep- resentation in the SAN-based models of NLP tasks. In other words, the prior knowledge plays auxiliary role during the processing of learning language representation. Therefore, we compute gate scalar g" [0, 1] to weight the expected importance of the prior knowledge representation PK at the n-th layer:
Add Norm
Feed Forward
PK"
Gated Sum
1
Add & Norm
Add & Norm
Multi-head Attention
Multi- head Attention
Positional
Encoding
g" = Sig WTPK), (10)
Prior knowledge
Input Embedding
where the Sigmoid is an active function, and W E"" x1 and 0 ERdmotd X1 are model parameters. We then fuse H and PK through the gate scalar g" learn the sentence representation at the n-- th layer:
Representation
Source
Figure 1:
SANs with prior knowledge representation. Dotted boxes denote the shared modules.
Htd H g" PK.
(11)
Also, the new stacked SAN is formally denoted as the modified version of Eq. I
H LN((SefAuu (Q"-1, "0"11K""_ KK"= vo-1)= (--1))++-1 H²-1)
PK n = LN(SelfAttt (Q" K, + H"),
g" Sigmoid (U"" + PK"),,
(12)
H n = H" + - PKK,
H = LN(FFN" (H") +
where H. is the initialized sentence representation HD in the Section 2 As result, there is more effective sentence representation H. for the input sentence. Later, H will be fed into the Decoder of the SAN-based model to model NLP tasks, for example, machine translation task.
EXPERIMENTS
5.1 EXPERIMENT SETUP
In this paper, we selected the machine translation task to evaluate the effectiveness of the proposed method, that is, two widely-used datasets: The WMT14 En-De includes 4.43 million bilingual sentence pairs, and the newstest2013 and newstest2014 datasets were used as the validation set and test set, respectively; The WMT17 Zh-En includes 22 million bilingual sentence pairs, and the newsdev2017 and newstes12017 datasets were used the validation set and the test set, respectively. In the experiment, we first used the fast_align toolkit Dyer et al. (2013) to obtain word alignments from the source language the target language. We then learned bilingual translation lexicon table between each word pair from the bilingual parallel training data with word alignments.
The byte pair encoding algorithm (Sennichh al. 2016) was adopted, and the vocabulary size was set to 40K. For the Transformer base NMT, the dimension of all input and output layers was 512, the dimension of the inner feedforward neural network layer was set to 2048, and the total heads of all multi-head modules were set to 8 in both the encoder and decoder layers. In each training batch, there was a set of sentence pairs containing approximately 4096x 8 source tokens and 4096x8 target tokens. During training, the value of label smoothing was set to 0.1, and the attention dropout and residual dropout were p == 0.1. The learning rate was varied under a warm-up strategy with warmup steps of 8,000. For evaluating the test sets, used a beam size of 4 for decoding, and evaluated tokenized case- sese-sensitive BLEU with the averaged model of the last 5 checkpoints for Transformer base model and 20 checkpoints for Transformer big model saved with interval of 2,000 training steps. Following the training of 300,000 batches, the model with the highest BLEU selected to evaluate the test sets. For the other setting mentioned,
score for the validation
4

---

Under review as a conference paper at ICLR 2021
followed the setting in Vaswani al. (2017). ). All models were trained eight V100 GPUs and evaluated on single V100 GPU. The multi-bue caaliibbeeupper bleu,perl script was used as the evaluation metric for the three translation tasks, and signtest (Collins et al. 2005) used as the statistical significance test. We implemented the proposed NM'I models on the fairseq toolkit (Ott et al. 2019).
5.2 BASELINE SYSTEMS
branssbase/big: vanilla Transformer-based base/big models (Vaswani al. 2017). Both Trans, base and Trans.big models differ at the hidden size (512 vs. 1024), filter size (2048 vs. 4096), and the number of attention heads (8 vs. 16).
+Shared-pris - It proposes shared-pri give closer relationship between the source and target embeddings in the Transformer-bast based NMT.
bilingual word embeddings (Liu et al. 2019), which
+SoftPrototype: Each word in the input sequence is mapped into a distribution over the target vocabulary, and the weighted average of target word embeddings is treated as an "expected" word representation in the prototype R for the Transformer-basee NMT model. It allows the Decoder to have indirect access to both past and future information (Wang et al. 2019).
+D2GPo: A kind of data-dependent Gaussian prior objective is proposed to build two probability distributions for the Transformer- based NMT, the first of which from the detailed model training prediction and the second of which is from ground-truth word-wise distribution (Li al. 2020).
+BCWAConLLos: used word frequency information to learn a sequence of content words for source and target sentences. A content word-aware Transformer-based NMT model was designed to learn an additional content word-aware source representation and to utilize target content words to compute additional loss during the training (Chen et al., 2020).
+Soft Template: used extracted templates from tree structures as soft target templates to guide the translation procedure. We incorporate the prior template information into the encoder- decoder framework to jointly utilize the templates and source text to enhance the Transformer-base NMT (Yang et al. 2020
5.3
TRANSLATION RESULTS
En-De #Speed.
Zh-En
Systems
BLEU
#Para.
BLEU
#Para.
Existing NMT systems
Trans.base Vaswani et al. 2017)
27.3
N/A
65.0M
N/A
N/A
+Shared- private (Liu et al. 2019
-Shared-private
28.06
N/A
65.0M
N/A
N/A
+D2GPo (Li et af.
2020)
27.93
N/A
N/A
N/A
N/A
#BCWACon Chen et al.
2020
28.51
13.1k
72.8M
24.94
81.MM
Tigns.big (Vaswani et al.
2017
28.4
N/A
213.0M
N/A
N/A
+SoftPrototype [Wang el al.
2019)
29.46
N/A
200.2M
N/A
N/A
+D2GPo (Li et al.
2020)
29.10
N/A
N/A
N/A
N/A
+BCWAComtLoss CChen et al.
2020
29.14
10.
246.3M
25.12
262.7M
+Soft Template Yang al.
2020
29.68
N/A
N/A
N/A
N/A
Our NMT systems
Trans.base
27.67
13.2k
66.5M
24.28
74.7M
+PKF
28.41+
12.1k
67.5M
25.03++
75.7M
+PKr
28.66++
11.6k
67.5M
25.32++
75.7M
Trans. big
28.65
11.2k
22T.2M
24.84
237.5M
+ +PKr
29.21++
10.1k
225.4M
25.26
241.7M
+PK,
29.58++
9.1k
225.4M
25.54+
241.7M
Table 1: Comparison of the proposed method with existing NMT systems on the three translation tasks. "#Speed." and "#Para." denote the training speed (tokens/second) and the size of model parameters, respectively. "+/++" after the score indicates that the proposed method was significantly better than the corresponding baseline Trans. at significance pc0.0500...

---

Under review as a conference paper at ICLR 2021
Main Results: Table showed the main results of WMT14 En-De and WMT17 Zh-En translation tasks. The BLEU scores of re-implemented Trans, base/big models were better than that of the original Trans.base/big models /Vaswani et al. 2017), which makes the evaluation convincing. As seen, our models (i.e., +PKr F and + +PKr) and the comparison methods (tSSoftProooyype, +Shared-private, +D2GPo, BCWAContL Loss, and +Soft Template) were superior the baseline Trans. base/bb models. This indicates that introducing the prior knowledge consistently improved translation performance, demonstrating the effectiveness of the proposed prior knowledge representation approach.
Evaluating Prior Knowledge: Based on the proposed prior knowledge representation approach, the prior word frequency knowledge (+PKF) and the prior translation lexicon knowledge (+PKr) gained the improvement of BLEU scores on the Trans. base/bb models. This indicates that the proposed approach provided universal setting to better verify the effectiveness of two prior knowledge compare the comparison models (i.e., +SootPrototype, +Shared-private, +D2GPo, BCWAContLoss, and +Soft Template) with their specific methods. Furthermore, BLEU scores of +PK, were higher than that +PKr, that is, two types of prior knowledge have different improvements. The difference means that the improvement comes more from the prior knowledge itself rather than the method.
Comparison with Previous Works: Trans +PK₁ superior to +Shared- and +D2GPo while it was slightly inferior to Trans. +BCWAContL.os PKr only focused on exploring the importance of source content words while BCWAContLoss captured the importance of source and target content words. In comparison, Trans.big +PKr was slightly superior to Trans. base +BCWAContLoss, The reason may be that the proposed approach can make full use of the prior word frequency knowledge on the Trans. big setting. +PK, outperformed +SoftPrototype, +Shared- private and +D2GPo and +BCWAComLLoss on the Trans. base/big settings while it was slightly inferior to +Soft Template on the Trans big seting. The reason may be that +Soft Template encoded syntax constraint in addition to the prior translation knowledge.
Model Parameters and Training Speed: Take the En-De translation task an example in Table 1 base +PK and +PKr models contained approximately 1.5% additional parameters while both they decreased 8.3% % the training speeds, compared to the baseline Trans.base model. Meanwhile, Trans.base + +PKr model achieved a comparable performance compared to the baseline Trans. big model which has many more parameters. This indicates that the improvement is indeed from prior knowledge rather than more parameters.
5.4 MODEL CONVERGENCE
In this section, we tried to verify whether the introduction of prior knowledge has impact on the training process. Specifically, for Trans.base, +PKp, and +PKr mod- els, we observed their BLEU scores on the En-De valid set every five epoch intervals as shown Figure As seen, the learning curves of +PK and +PK7 are that of Trans.base confirming the effectiveness of the prior translation lexicons and word frequency knowledge. Also, BLEU scores of +PKr and +PKr had been improving from the 5-th Epoch the 55-th and 45-th Epochs, respectively. Thus, +PKF and +PKr reached the highest BLEU scores at the 55-th and 45-th Epochs. This means that +PK, converged faster than +PKp during the training. This may be that the target language knowledge is advanced to the Encoder of NMT to learn effective representation of the source sentence.
28
BLEU scores
26
Trans.base
24
+PKp
+PKr
10,
2000000000 1 1 Soo d s G5
Training Epochs
Figure 2: BLEU scores of different training epochs the En-De valid set.

---

Under review as a conference paper at ICLR 2021
5.5 PROBING EXPERIMENT
Following the word prediction ex- periment in (Weng et al. 2017)'s work, we also did similar prob- ing experiment here to evaluate the source representation learned by the proposed SANs with PKr and Pky. In detail, for each sentence in the test set, we use the final source representation to make a prediction about the possible source words. Given the set top C' words the target vocabulary and the of words in all the references as R, and the precision of the word prediction: Precision = |C n R R|/|C/ (C) x 100. We then trained a ong-oo-words predictor by maximizing P(yhono | where Yhow is an unordered set containing target words in the output sentence. In addition to the structure SANs with PK and PK7, the predictor included an additional feed-forward network layer which maps the final source sentence to the target word embedding matrix. Then, we compare the precision of target words in the top-N words which are chosen through the predicted probabilities. As shown in Table 2. +PKr and +PKr gained higher precision in all conditions, especially +PK, was higher than +PKr, which consistent with the analysis that +PKr allows the target language knowledge be advanced to the Encoder compare to +PK p. This shows that the proposed method can obtain more information about the target language sentence and partial answers why the proposed NMT models could improve generation quality.
Precision
System
Top-200)
Top-500
Top-1000
Predictor of Trans.base
45%
55%
66%
+PKF
52%
61%
72%
+PKT
61 %
72%
79%
Table 2: Precision of hag-of-word. predictor.
5.6 EVALUATING HYPER PARAMETERS PKT AND PKF
28
BLEU scores
28
BLEU scores
En-De Zh-En
En-De
26
26
Zh-En
24
24
64
128
256
512
1024
2048
5
10
Hyper-parameter K for PKT
hyper-parameter B PK
Figure 3: The left sub-figure shows BLEU scores of Trans.base+PK. with the different hyper- parameter K; the right figufigure shows BLEU scores of baans.base+PPt with the different hyper- parameter B. The dashed lines denote the results of the brse model.
The left sub-figure of Figure showed the results of Trans.base model (black dotted line) and +PK, model with different dyper-paramete K for learning prior translation knowledge representation (colorful line) on the En-De and Zh-En test sets. When K one of (0, 5, 10, 15, 20. 30), the result of Trans.base+PK. outperformed the Trans.base on the two test sets. Furthermore, reached the point of highest BLEU score K=10, K=15, and K=10 on the En-De and Zh-En test sets, respectively. Finally, trained the proposed NMT models in the two translation tasks (Sec Table according to this optimized hyper-parameter K.
Furthermore, masked the function words in the source sentence according list of the B function words, thereby gaining the Mp in Eq. The right sub-figure of Figure 3 showed the results of Trans.based bass.ba with the different number of the top B function words the En-De obtained the highest BLEU scores on both test sets over the Tranns base modeling B 256. The trained +PK, was shown in Table
and Zh-En test sets.

---

Under review as a conference paper at ICLR 2021
5.7 TRANSLATING DIFFERENT LENGTH SENTENCES
To further evaluate our method, we showed the translation performance of source sentences with different sentence lengths. Specifically, divided each test into six groups according to the length of the source sentence, for example, "40" indicates that the length of sentences is between 30 and 40. Figure shows the results of Trans. base, +PKF. and +PKr models on the translation tasks. As seen, +PKr and +PKr were superior to the baseline Trans.base in almost every length group on the two tasks, confirming the effectiveness of our method. Also, BLEU scores of all models decreased when the length was greater than thirty over the Zh-En task. BLEU scores of all models decreased when the length was greater than forty over the En-De task. The reason may be more diverse vocabulary in Chinese than English and German. Thus, target word was mapped more source words the prior translation lexicons, that is, increasing the ambiguity the target word.
29
26
EU Scores
BLEU Scores
27
24
Trans base
Trans.best
+PKp
+PK
+
22
10
20
30
40
50
50+
10
20
30
40
50
50+
Source Sentence Length (En-De)
Source Sentence Length (Zh-En)
Figure Trends of BLEU scores with different source lengths on the two translation tasks.
RELATED WORK
There were a variety of methods to combine the advantages of both the dominant NMT and traditional SMT models. Typically, the prior knowledge was used to enhance the NMT, for example, discrete dictionary (Luong et al. 2015 Jean et al.
2015 Arthur et al. [0016] Gu et al.) 2016), the 2016 Indurthi
limitation of vocabulary (Mi al. 2016 He et al.)
2019), translation of certain terminology (Alkhouli et al. 2018), and translations and interpretable al alignments (Garg et al. 2019).
Wang et al.] (2017a) combined NMT posteriors with SMT word recommendations through linear interpolation implemented by gating function. Nichues et al. (2016) utilized the SMT model to pre-translate the inputs into target translations sequences in NMT. The combination of NMT and SMT had been also introduced in interactive machine translation to improve the system's ' suggestion quality (Wuebker et al. (0166). Zhou et al. (017) presented a neural system combination framework to directly combine NMT and SMT outputs. Different from the direct combination SMT (Wang et al. 201 Wuebker et al., 2016 Niehues al.) 2016 Zhou al. 2017), the proposed method allows the prior knowledge to be introduced into NMT in a universal neural way. Meanwhile, the prior knowledge representation enables the NMT semantically generalize the prior knowledge instead of directly combining the probability distribution of the prior translation knowledge.
employed the target pre-translations as input
7 CONCLUSION
This article explored universal representation method to introduce the prior knowledge into the Transformer-based NMT in a universal way. In particular, the proposed method can represent the prior knowledge a continuous space matrix to semantically generalize instead of directly combining the probability distribution of the prior translation knowledge, thereby enhancing the representation of the input sentence. Experimental results verified the effectiveness of our method on the two wildly used translation tasks. In the future, we will adopt the proposed method to other natural language processing tasks and explore more effective prior knowledge.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. On the alignment problem in multi-head attention-based neural machine translation. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 177-185, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/11866311
Philip Arthur, Graham Neubig, and Satoshi Nakamura. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1557-1567, Austin, Texas, November 2016. Association for Computational Linguistics. doi:
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, May 2015.
Loic Barrault, Ondiej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference machine translation. In WMT, pp. 1-61, Florence, Italy, August 2019.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pictra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311 1993.
Kehai Chen, Rui Wang, Masao Utiyama, and Eicchiro Sumita. Content word aware neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 358-364, Online, July 2020. Association for Computational Linguistics. doi: 10118633/11/222.aa/-mm
David Chiang. Hierarchical phrase-basc based translation. Computational Linguistics, 33(2):201-228, 2007. doi: 10. 00.1162/co1..200.333...2
Michael Collins, Philipp Kochn, and Ivona Kučerová. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL '05). pp. 531-540, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 10.3 00.1115/121980.1222 121999.
Dina Demmer-Fushhmn,, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujüi (eds.). Proceedings of the 18th BioNLP Workshop and Shared Task, Florence, Italy, August 2019.
Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644-648, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. pp. 4453-4462, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D199
Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. A convolutional encoder model for neural machine translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 123-135, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in sequence-to-seueeee sequence learning. In Proceedings the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1631-1640, Berlin, Germany, August 2016. A Association for Computational Linguisties. doi: 10. 18653v 11885//1/P16-111111 1/P16-1154,

---

Under review as a conference paper at ICLR 2021
Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. Towards understanding neural machine translation with word importance. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCNI.P), pp. 953-962, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. (0.18653/v1/DD)) 1088.
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. Improved neural machine translation with smt features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 151-157.
AAAI Press, February 2016.
Sathish Reddy Indurthi, Insoo Chung, and Sangha Kim. Look harder: A neural machine translation model with hard attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3037-3043, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1//19111
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 1-10, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.31 1/v1/115-1001.
Philipp Koehn, Franz J. Och, and Daniel Marcu. Statistical phrase- based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 127-133, May-June 2003.
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. In International Conference Learning Representations, April 2020.
Xuebo Liu, Derek F. Wong, Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu. Shared-private bilingual word embeddings for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3613-3621 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1w.11665//1//11...
Yang Liu, Qun Liu, and Shouxun Lin. Tree-to-string alignment template for statistical machine translation. In Proceedings of the 2 International Conference on Computational Linguisties and 44th Annual Meeting of the Association for Computational Linguistics, pp. 609-616, Sydney, Australia, July 2006. Association for Computational Linguistics. doi: 21111////2111522225555
Yang Liu, Yun Huang, Qun Liu, and Shouxun Lin. Forest-to-string statistical translation rules. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp- 704-711, Prague. Czech Republic, June 2007. Association for Computational Linguisties.
Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojeiech Zaremba. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 11 11-19, Beijing, China, July 2015. Association for Computational Linguisties. doi: 10.3115/v1/115-100
Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. Vocabulary manipulation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers). pp. 124-129, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18653M 1/P16/20111
Jan Niehues, Eunah Cho. Thanh-Le Ha, and Alex Waibel. Pre-translation for neural machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1828-1836, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.
Stephan Oepen, Omri Abend, Jan Hajic, Daniel Hershcovich, Marco Kuhlmann, Tim O'Gorman, and Nianwen Xue (eds.). Proceedings of the Shared Task on Cross-Framt Fransewrr Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, Hong Kong, November 2019. Association for Computational Linguistics.
10

---

Under review as a conference paper at ICLR 2021
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier. and Michael Auli. fairseq; A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference the North American Chapter the Association for Computational Linguisties (Demonstrations), pp. 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 1/N19-099/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.11863//1/P11-1116
Hendra Setiawan, Men Kan, and Haizhou Li. Ordering phrases with function words. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp- 712-719, Prague. Czech Republic, June 2007. Association for Computational Linguistics.
Hendra Setiawan, Min- Yen Kan, Haizhou Li, and Philip Resnik. Topological ordering of function words in hierarchical phrase-based translation. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 324-332 Suntec, Singapore, August 2009. Association for Computational Linguistics.
Ila Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1004-112. Curran Associates, Inc., 2014.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 76-85, Berlin, Germany, August 2016. Association for Computational Linguistics. Lingusssss., doi: 10. 1111866//11116-1000
Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics, 5:87-99, 2017. doi: 10. 10.1162/taclaa.000 aacacla_0000 a00048.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-6090 Curran Associates, Inc., December 2017.
X. Wang, Z. Tu, and M. Zhang. Incorporating statistical machine translation word knowledge into neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(12):2255 2266, Dec 2018. ISSN 2329-9290.
Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. Neural machine translation advised by statistical machine translation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, pp. 3330-3336. AAAI Press, 2017a.
Xing Wang, Zhaopeng Tu, Deyi Xiong, and Min Zhang. Translating phrases in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1421-1431, Copenhagen, Denmark, September 2017b. Association for Computational Linguistics. doi: 10. 10.11863/1/DDI--11499
Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Cheng Xiang Zhai, and Tie-Yan Liu. Neural machine translation with soft prototype. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 6316-6325. Curran Associates, Inc., 2019.
Davy Weissenbacher and Graciela Media Mining for Health Applications (#SMM4H) Workshop & Shared Task, Florence, Italy, August 2019. Association for Computational Linguistics.
(eds.). Proceedings of the Fourth Social

---

Under review as a conference paper at ICLR 2021
Rongxiang Weng, Shujian Huang, Zaixiang Zheng, Xinyu Dai, and Jiajun Chen. Neural machine translation with word predictions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 336-145, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 11855//11/DD7-1003.
Joern Wuebker, Spence Green, John DeNero, Saša Hasan, and Minh- Minh-Thang Luong. Models and inference for prefix conssraind machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 66-75, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 10.18653/v1/P16 1007.
Jian Yang, Shuming Ma, Dongdong Zhang, Zhoujun Li, and Ming Zhou. Improving neural machine translation with soft template prediction. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5979-5989, Online, July 2020. Association for Computational Linguistics. doi: 10.18663////2220aac- m31.m....31
Jingyi Zhang and Hai Zhao. Improving function word alignment with frequency and syntactic information. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI 13, pp. 2211-2217. AAAI Press, 2013. ISBN 978-1-57735-633-2.
Yang Zhao, Yining Wang, Jiajun Zhang, and Chengqing Zong. Phrase table as recommendation memory for neural machine translation. In Proceedings of the Twenty-Sevnnth International Joint Conference on Artificial Intelligence, IJCAI-18, pp. on Artificial Intelligence Organization, 7 2018. doi: 10. 10.24963/jcai.222 201.////...221/14
International Joint Conferences
Long Zhou, Wenpeng Hu, Jiajun Zhang, and Chengqing Zong. Neural system combination for machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computationnl Linguisties (Volume 2: Short Papers), pp. 378-384, Vancouver. Canada, July 2017. Association for Computational Linguistics. doi: 1/111666//1/1172200000
A BILINGUAL TRANSLATION LEXICONS IN SMT
The goal of machine translation is to translate a source sentence X X=r/ into a target sentence Y=yl- where z and yi belong to the source vocabulary Vz. and the target vocabulary Vy, respectively. In the traditional SMT systems, bilingual lexical translation rules are generally learned directly from the large-scale bilingual parallel data in unsupervised fashion using the IBM models (Brown et al. 1993 Koehn et al. 2003). These models can be used to estimate the word alignments a and lexical translation probabilities P(a) (y/z) between the words of the two language through the expectation maximization algorithm. First, in the expectation step, the algorithm estimates the expected count c(x; y) In the maximization step, lexical translation probabilities are calculated by dividing the expected count by all possible counts:
(ula) clx, v)/ Σ c(x, (). $
(13)
As a result, there is a list table in which each source word is linked to set of possible target candidate translations, called bilingual lexicon translation table T {z, y. P(a) (g/l²} from source language to target language.
12


Reviews:
[{"review_id": "onnodFZ8t1v", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for introducing prior knowledge into Transformer models. More specifically, the authors propose to use an additional self-attention block to incorporate prior knowledge about the word frequency and translation lexicon and use a gating mechanism to combine its output with that of the standard sefl-attention block. Experiments are conducted using English-to-German and Chinese-to-English translation datasets, and the results show the effectiveness of the proposed approach.\n\nThe details of the proposed approach are not very clear. The authors state that matrix M is packed into a key and value matrix pair {K, V}, but it is not really clear how exactly these two matrices are produced from M. I think they should describe this process more concretely in Sections 3.1 and 3.2.\n\nI also think that the authors should test simpler baselines as well. It seems to me that such prior knowledge can be incorporated into the standard Transformer model in the same manner as positional encoding or expanded input embeddings. What is the advantage of the proposed approach over these baseline methods?\n\nIn Equation (5), $B$ is treated as a set, but it is treated as an integer in the previous paragraph.\n\nThe manuscript contains many grammatical errors and is not ready for publication. I suggest that the authors proofread their manuscript much more carefully before submission. Here are some of the errors in the first page:\n - gradually learning -> gradually learns?\n - on WMT14 -> on the WMT14?\n - Gehring et al. (2017) -> (Gehring et al., 2017)?\n - then added to -> then added to the?\n - mistraslate -> mistraslates?\n - infrequency -> infrequent?\n - were employed -> was employed?\n - rules was -> rules were used?", "extra_scores": {}, "extra_text": {}}, {"review_id": "l5KFz5kL7Dd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis submission works on the neural machine translation problem. The authors extend the previous works on leveraging language statistics or prior knowledge (SMT model or whatever) in LSTM based NMT models in self-attention based NMT models, Transformer model. The authors propose two alternatives to incorporate prior knowledge, which are the word frequency information for the monolingual data and the prior translation lexicon information for the bilingual data. These resources are integrated into the hidden representations from the self-attention computations and then the two output hidden representations are gated together for upper computations. The experiments are conducted on two typical NMT datasets: WMT14 En->De and WMT17 Zh->En, the results show that the proposed method can improve the NMT model performances. \n\nGeneral comments:\n\nThe submission works on the neural machine translation tasks, which are important in recent years and have achieved great breakouts. This work extends the previous work on how to leverage the information from the statistical data knowledge into the NMT model. Overall speaking, there is no big tech flaws or incorrect points, but this paper is not suitable for ICLR conference, better for NLP related conferences. The main pros and cons of my side are listed as follows.\n\nStrengths:\n* The paper writing is good in an overall reading, no big mistakes existed in this submission.\n* The experiments are conducted on two different large datasets, with extensive ablation studies to show the effectiveness of each part of the model. \n\nWeaknesses:\n* The motivation of this work is not presented in a very clear way. The authors claim that previous works can not explore other prior knowledge in a universal way. This is somehow too overwhelmed, without clear examples or explanations, I feel hard to be convinced of this point. Besides, the authors do not present that the methods utilized in this submission are much better than previous works in terms of the differences and contributions. \n* The contribution from both technique insights and the studying problem is limited. First, for the studying problem, how to leverage the information from the statistic data knowledge is widely studied in the MT committee previously and in recent years. The authors must be very familiar with the works conducted on LSTM models about incorporating SMT knowledge or others. Second, as for the technique contribution, the authors propose word frequency information and the translation lexicon knowledge. These two are typical methods used in MT research, which makes the submission not so insightful from the different prior knowledge. Therefore, the contributions of the submission are not important or enough. \n* As for the specific utilization of the information, there are also several concerns. The authors seem to emphasize the prior word frequency from the \"monolingual\" data, which looks unnecessary. In current modeling, there is nothing related to extra monolingual data, but only the source data. The authors want to compare with the lexicon information from the bilingual dataset, but this will mislead the readers to think of the extra monolingual data. As for the bilingual translation lexicon, the extra costs increase and the authors should discuss this. \n* There are several unclear parts of the presentation. First, the dimension of the matrix $M$ is not clear. For example, $M_F$ seems to be a $J\\times J$, but for a bilingual lexicon, it is $J\\times K\\times d_{model}$ for $M_T$. Therefore, what is induced $K_F, V_F$ and $K_T, V_T$, and how are they converted to $K$  and $V$. By the way, in the bilingual lexicon, $L$ is leveraged, but not show in $M_T$, this also makes it to be unclear. It supposes that $M$ should be a similar size as $H$, but seems not.\n* In current modeling, the prior knowledge is only incorporated in the encoder-side and the training phase, if I understand correctly. This is also limited. I acknowledge that the word frequency information is from the training data, therefore only the training phase is influenced, but the authors should clearly talk and discuss this. By the way, the encoder-side only modeling again makes the contribution to be limited. More efforts are encouraged than this submission. \n* One another point is about the experiment results. The authors said that they reimplement the baselines of the Transformer base model and Transformer large model. However, according to the settings they described in the submission, the baselines are reported at a low level, not a very convincing score. The authors used attention dropout and residual dropout as $0.1$, and also they use $8$ GPU cards (since $4096\\times 8$ batch size), and checkpoints are averaged. Therefore, according to the experiences, the baseline results should be higher, and it makes me not be convinced about \"which makes the evaluation convincing\". However, I acknowledge that the method proposed is effective and the results of the proposed method are in reasonable scores, what I want to mention is only the baseline results. \n\nMinor suggestions:\n* Section 5.7 about length experiments is not necessary, it is not related to the claim, motivation, contribution of the submission. So it is recommended to remove this part. \n\nIn a word, this paper is okay for NLP related conferences, but maybe not enough for ICLR. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UZgoDCRtMzh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to introduce **prior knowledge** into Transformer-based sentence encoders, here in the context of neural machine translation (NMT). More concretely, the prior knowledge is represented in the form of a matrix $\\boldsymbol{M}$, where each row denotes a vector of prior knowledge associated with each word $x_i$. The prior knowledge matrix $\\boldsymbol{M}$ is then represented as a (key, value) pair that can be attended by the query matrix $\\boldsymbol{Q}$ (the same query matrix as used in the main NMT component) using a standard Transformer self-attention mechanism. This procedure results in a prior knowledge representation matrix $\\boldsymbol{PK}$, which is then combined with the standard Transformer encoder output using a simple gating mechanism. \n\nHere the paper explores two kinds of prior knowledge: \n1. a binary mask that zeroes out frequent yet semantically uninformative function words (which here is defined as the top $B$ most frequent words on the training corpus) whilst keeping the content words the same, and \n2. a word type-level translation lexicon that defines the **L** most frequent target words for each source token, as obtained from word alignment methods used in statistical machine translation (SMT) systems. \n\nExperiments on WMT English-German and Chinese-English translation datasets suggests that the prior knowledge-augmented model outperforms the standard Transformer models without any prior knowledge representation, and compares favourably to other alternative methods of injecting prior knowledge as proposed in prior work.\n\n**Pros:**\n1. The question of how we can inject prior knowledge into neural sequence-to-sequence models is an important and interesting research question. Rather than learning everything from scratch, injecting prior knowledge may lead to more data-efficient models, and can also encourage better adequacy (since neural seq2seq models often produce fluent yet inadequate generations).\n2. The proposed approach is empirically validated on two machine translation benchmarks, and obtains improvements in both datasets. Furthermore, the improvements are consistent across both base and large Transformer setups.\n\n**Cons:**\n1. Despite the positive points above, my main concern with this paper is regarding the clarity. In particular, the introduction does not clearly explain **where exactly** previous work on injecting prior knowledge falls short. For instance, the last paragraph of page 1 mentions \"..these unique methods make it difficult to explore other prior knowledge in a universal way and to determine which of the prior knowledge and the unique method this improvement comes from\". This sentence is quite hard to understand---the paper does not really explain how the proposed approach is more universal than previous ones. In particular, the paper only explores two kinds of prior knowledge, and only for the task of machine translation---any extensions to other prior knowledge seem to need careful manual design. Furthermore, the beginning of page 2 mentions that \"Meanwhile, these [previous] studies directly utilized the probability distribution of the prior knowledge and lacked the neural network's ability to semantically generalize ...\". I fail to see what exactly the proposed approach has to do with semantic generalisation. Overall, the introduction could benefit from a substantial reframing to better motivate why we need better ways of injecting prior knowledge. Furthermore, the paper can also benefit from *substantial copy-editing*, as there are many grammatical errors that make the paper a bit hard to read.\n\n2. The analysis section does not seem particularly insightful. In particular, the probing experiment is limited to predicting bag-of-words (which seems like a very artificial task), while the analysis on the choice of hyper-parameters (Section 5.6) does not strike me as particularly interesting (for instance, in Figure 3, *any* hyper-parameters above 0 do better than 0, and the differences between different hyper-parameters are fairly minimal). The fact that NMT systems do worse on longer sentences has also been established before in prior work, and is thus not particularly insightful. I have some suggestions for analysis as detailed in the **Suggestions** section below.\n\n3. The paper has some errors in terms of notational inconsistencies and grammatical mistakes that make it a bit hard to read. For instance, the prior knowledge hyper-parameter for bilingual translation lexicon is denoted as $L$ in page 3 (between equations 8 and 9), but the same hyper-parameter is denoted as $K$ in the caption of Figure 3 (page 7). Some of the grammatical errors are detailed below.\n\n**Suggestions:**\nRather than the current set of analysis experiments, I have some suggestions for further analysis that may be more informative:\n1. It would be interesting to analyse the translation performance on **rare words**. The prior knowledge in terms of word translation lexicon may help translate rare words better, since the standard NMT systems tend to hallucinate fluent yet inadequate outputs.\n2. It would also be interesting to see the effects of injecting prior knowledge on **adequacy**. Does the prior-injected model suffer from less hallucination? Or does it more faithfully translate all the content words in the source sentence (as opposed to skipping and not translating some of the content words)?\n3. It would also be interesting to qualitatively show some examples where the model outputs differ between the baseline and the proposed approach. \n\n**Questions:**\n1. The frequency prior assumes that the top $B$ most frequent words correspond to function words. Did you verify whether this is really the case?\n2. What happens if you inject both prior knowledge into the model, rather than just choosing one?\n\n**Typos / Grammatical Mistakes / Minor Clarification:**\n1. Abstract: \"... it gradually *learning* ...\" -> should be \"it gradually learns\".\n2. The first paragraph in page 1 mentions \"... in many natural language processing tasks pairs\". Not sure what is meant by tasks pairs here.\n3. Page 1: \"... explored the prior knowledge which has the *stringer* ability\" -> should be \"stronger\".\n4. Page 1: \"... the phrase translation rules *was*\" -> should be \"were\".\n5. Page 1: \"... translations of *infrequency* words\" -> should be \"infrequent\".\n6. The last paragraph of page 2 mentions \"Also, $\\mathbf{M}$ is packed into a key and value ...\". What exactly is meant by \"packed into a key and value matrix pair\"?\n7. Page 4: \"where the sigmoid is an *active* function\" -> \"activation\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to incorporate additional prior knowledge into transformer architectures for machine translation tasks. The definition of problem is reasonable,  despite the fact that there is a long thread of work on adding knowledge of different types into neural architectures of NMT. The proposed model, however, needs to be better motivated, as to why the same thing cannot be done in a simpler way in the framework of transformers.  Judging from the exposition and the experiments, the proposed model is neither novel or empirically significant enough. The writing needs to be greatly improved to get rid of the grammatical errors and notational inconsistency.  

I’d suggest to reject this paper 

Decision: Reject