Neural Phrase-to-Phrase Machine Translation
Authors: Jiangtao
Feng
Lingpeng Kong
Po-sen Huang
Chong
Wang
Da
Huang Jiayuan
Mao
Kan
Qiao
Dengyong
Zhou
Venue:  (2020)

Abstract: We present Neural Phrase-to-Phrase Machine Translation (\nppmt), a phrase-based translation model that uses a novel phrase-attention mechanism to discover relevant input (source) segments to generate output (target) phrases. We propose an efficient dynamic programming algorithm to marginalize over all possible segments at training time and use a greedy algorithm or beam search for decoding. We also show how to incorporate a memory module derived from an external phrase dictionary to \nppmt{} to improve decoding. %that allows %the model to be trained faster %\nppmt is significantly faster %than existing neural phrase-based %machine translation method by \cite{huang2018towards}. Experiment results demonstrate that \nppmt{} outperforms the best neural phrase-based translation model \citep{huang2018towards} both in terms of model performance and speed, and is comparable to a state-of-the-art Transformer-based machine translation system \citep{vaswani2017attention}.

Under review as a conference paper at ICLR 2020
NEURAL PHRASE-TO-PHRASE MACHINE TRANSLA- TION
Anonymout monnymouss.. authors Paper under double-blind review
ABSTRACT
We present Neural Thrase-to-Phrass Machine Translation (NP²MT), a phrase- based translation model that uses a novel phrase-attention mechanism to discover relevant input (source) segments to generate output (target) phrases. We propose efficient dynamic programming algorithm marginalize over all possible segments at training time and use a greedy algorithm or beam search for decoding. We also show how to incorporate a memory module derived from an external phrase dictionary to NP to improve decoding. Experiment results demonstrate that NP outperforms the best neural phrase-based translation model (Huang et al. 2018) both in terms model performance and speed, and comparable to state-of-the-art Transformer-basee machine translation system (Vaswani et al. 2017
INTRODUCTION
Segmental structures (i.e., phrasal information) are abstractions that have been shown to be useful in machine translation. State-of-the-art machine translation systems prior to the deep learning revolution
were dominated by phrase-based models (Koehn et al. 2003 2007: Lopez 2008 Koehn (209).. In contrast word-based approaches, these methods consider explicit phrase structures in both source and target sentences and model their alignments.
Recent advances in segueencett-sequeenee learning (Sutskever et al. 2014) and attention-based mechanisms (Bahdanau al. 2014) have driven impressive progress new generation of machine translation models. They typically consist of three main components: encoder that summarizes the source sentence into vectors, a decoder that generates the translation word by word, and a word- based attention module that used by the decoder capture relevant source information when decoding. Innovations in terms of model architectures (Vaswani al. 2017] Gehring et al.) (017a) and optimization algorithms (Ba et al. 2016) push these models to become the de facto sfate-of-the-arr machine translation systems. These models operate at the word-level, since discovering phrases for both the source and target sentences and incorporating this phrasal information into neural sequence sequence model is not straightforward (i.e., it requires marginalizing over all possible phrase alignments in a neural model).
Huang et al. (2018) show first-step toward neural phrase- based machine translation model by designing model that is aware of phrase structures in the target sentence. They show that such an approach leads to better results compared to baseline sequence to sequence models on several language pairs.
In this paper, we present the first neural phrase-based machine translation model (NP2MT) that considers phrasal information in both the source and target sentences. Our model uses segmental recurrent neural networks (Kong et al. 2015) to discover phrases on the source sentence and a novel phrase-atteninn model to align them generated target phrases ($2.1). We design dynamic programming algorithm to efficiently marginalize over all possible segments (phrases) in the source and target sentences in our model for training ($..)) and use a greedy algorithm or beam search for decoding ($2.4). We also show how to augment this model with a memory module consisting of phrase-to-pheet phrase translation (i.e., phrase dictionary) in setup when there are many unseen

---

Under review as a conference paper at ICLR 2020
gk
decoder
phrase-lovel enooler
R1
word-level enceder
largei envoder
Figure 1: The architecture of NP model. The example shows how the target phrase ya:4 is translated conditioning directly on the source phrase 72:3, using the phrase-level attention. Note that for brevity, the phrase-level encoder, we only show one possible segmentation of the source sentence. We "... indicate all the possible segments Zyz Eq.
words (e.g.. proper nouns, cross-domain translation) to improve performance (2..) Experiments on benchmark datasets show that our model outperforms existing phrase- based machine translation models both in terms of performance and speed and is competitive with state-of-ther word-based models ($3).
MP
Consider source sentence #1.5 = { T2, T2,..., rs} and a target sentence y1-7 (y1, 9/2 Ur }-
Our model is based on the sequence-to- to-sequence model (Sutskever 2014) with two main components: source sentence encoder and a target sentence decoder. Figure shows an overview of NPºMT, we describe each component details below.
2.1 ENCODER
We represent each word in $1.5 as a vector and use a bidirectional LSTM to obtain token representa- tions ri...., Fs by concatenating the forward and backward hidden states.
We then obtain phrase representations using another bidirectional LSTM that takes token representa- tions above as inputs based on Segmental RNNs (Kong et al. 2015). Specifically, in order to compute the representation
phrase spanning from the i-th token bidirectional LSTM with r,. ,,....,Fy ry inputs. Denote the hidden states of the forward LSTM by h and the hidden states of the backward LSTM by h. We have Say concat( } h h.).
j-th token, denoted by 8t.j. we run
Given the maximum segment length (a hyperparameter of the model), we compute representations for all phrases:
S si (i,,)) ((i,)) :1 ((it)::isissst----
Since bidirectional LSTM the phrase encoder and all phrases contiguous segments, we can efficiently compute these representations in ((SP).
2.2 DECODER
Let E Zyy.r be a valid of segmentation of yı.r {V1, .2...., ,. Denote each segment in by Zi and the number segments |zx]-
The generative process of the target sentence is:
For each segment à: 1,..., /==]:
Sample word of until we get the end segment symbol (i.e., of 8) or the end of sentence symbol in the last segment (i.e., y $8).
2

---

Under review as a conference paper at ICLR 2020
Our decoder consists of two main components:
A unidirectional (forward) LSTM to compute contextual representation gzc., of previously generated segments Zck- In training time, 8zC* can be obtained simply by running the LSTM on all target words from previous segments token by token.
A segment decoder that generates words the current segment. The segment decoder an attention-based model over all possible source phrases Si.j. 8z and previously generated words in the current segment y
We compute the phrase-attention
a once for every segment 2k, using 8zca query all possible source phrase representations St.) attn aattn(8r..8z...
at gives soft alignmen between the target phrase Ey and the source The phrase-attention score [.j phrases St.. The attention value is then computed as the sum over $. weighted by the attention score a.j" We use it as the initial state of the segment decoder.
The probability generating the target sentence is computed as:
plyeT an:s)
Σ
II
P(vē"
Si. 8xc. 922
=EZ 1E2k
2.3 TRAINING
For given pair of source and target sentence, the objective function that would like maximize in Eq. is intractable due to the summation over Zyur (i.e., all possible target segmentations). Here, we show a dynamic programming algorithm to compute it efficiently.
Denote the probability of generating all valid segmentations ending in the n-th target words by a(n). The probability of generating the entire sentence is therefore oz (xT)).
We have the following recursive function:
(n))
Σ (mm)
II
plvi (m.n)
Sj.j+ 8z com.n)) D (m.n)
(E2(m..)
where (<))) 1, and (m, denotes a segment from m to n. The computational complexity of this dynamic programming algorithm is O(TP), where P is the maximum (target) segment length.
2.4 DECODING
For inference at test time (i.e. decoding), we cannot use dynamic programming in since target sentence is unknown. We either use greedy algorithm or beam search for decoding.
since the
Greedy algorithm For the greedy method, we follow the generative process in $2.2 and greedily pick the word v/. until we reach the end sentence symbol (i.c. y = $$).
Beam search Our beam search algorithm different from the standard left-to-right beam search algorithm used in vanilla sequence- pepuencce-t-sequaeee io-sequence models since the probability for the next word is conditioned not only on the local context with in the segment, but also on all previously generated segments. The main idea our algorithm is to keep track of both during the beam search process. We show our Algorithm in Appendix A.
We maintain set ya to store the incomplete (open) sentences during the search. An open sentence represented as a tuple consisting of three elements: previously generated segments, previously
'Effrriin hard attention similar as in Raffel et al. 2017 may be beneficial in the context of phrase-level attention. We leave that for future work.

---

Under review as a conference paper at ICLR 2020
generated words in the current segment, and the probability of generating the open sentence far. This tuple contains sufficient information to compute the probability distribution of the next output word. We update each element (each open sentence) in yo after we generate new word, so the open sentences in y, will always have the same length.
2.5 DECODING WITH MEMORY MODULE
In language modeling, augmenting a recurrent neural network with memory module at evaluation time has been shown to improve performance (Grave et al. 2016 Krause et al. [0177] Grave et al.) 2017), especially for rare word predictions and in an open vocabulary setup. We explore the possibility of augmenting a neural machine translation model with persistent memory module derived from external sources (i.e., phrase-to-phrase dictionary) to improve decoding.
In this setup, we first build phrase-to-phrase dictionary using a phrase extractor from Moses (Koehn) et al. 2007). During decoding, each timestep, we compute attention scores forr possible source segments " We then choose the source phrase with the highest score be translated. Given the attended source phrase, we decide whether to use a dictionary to translate the phrase or not. We perform a memory lookup and check whether the phrase is in the dictionary. If it is, we directly translate using the dictionary; otherwise, we proceed in the greedy decoding case described above. When there multiple translations for a phrase in the dictionary, candidates, and then choose the best one as follows:
the model to score all
y
<- arg max
II p(dht Smmn, Szcap
(2)
d Ed KEZR
We summarize our memory-augmented decoding method in Algorithm I and illustrate in Figure
We note that word-based models can also benefit from the augmented memory module. However, it is more straightforward to use the phrase-level information in NP"MT. The word-based models only lookup phrases with source with length using attention weights. Experiments show that when compared models not augmented with memory modules, 2P improves significantly when there is larger amount of OOV (See Appendix Table 6.
In addition, it also possible to augment beam search decoding with a memory module. In this case, we perform lookup and we add the translated phrase yo after the bean reaches the length of the translated phrase. We present the greedy case for simplicity. It should be straight-forward for the readers to extend the algorithm into beam search cases by combining algorithms and 2.
We leave exploration of beam search and memory future work.
you see the colosseum in the middle, the river tiber
sie sehen in der mitte das kolosseum den fluß liber..
Figure 2: The decoding process of NP model with dictionary extension. The example shows how a German sentence is translated into the English sentence. Different colors are used to represent the aligned phrases. The system found the translations for the underlined OOV words German. We solid lines indict phrase translated using the segment decoder, and dashed lines to indict dictionary lookup.
EXPERIMENTS
3.1 DATASETS
We use IWSLT 2014 German-English (Cettolo al. 2014) and IWSLT 2015 English-Vietnande (Cettolo et al. 2015) as our benchmark datasets. We use "word" as a basic unit as words are better atoms for phrases, while many other papers use BPE (Sennrich et al., 2015). We think word level experiments fit more into the purpose of examining our model, in discussion and analysis,
4

---

Under review as a conference paper at ICLR 2020
Data: source sentence: 11,7, dictionary: D, maximum segment length P
Result: a target sentence:
Compute the representations Si,j for phrases given the maximum segment length P:
y =
while nor ends with $$ do
Compute phrase-attention score:
a ( Ex-ca)
m, = arg max St
UNK in Xm:: then
// Retrieve all translations for Xmen from D;
// The function sre retrieves the original text for segment X
d = -Lookup(D, sre sxc(x))));
// Select the best candidate ranked by the segment decoder, fixing the attention on Sya:
y arg max Irr p(dp* d'eed%
Smmm Sxcey" d
else end
Generate n following the greedy algorithm 2.4
Append y² to output y:
end
Algorithm 1: NP greedy decoding with dictionary
use the word level models. We run the BPE level experiment for the German-Englist dataset reference.
The IWSLT14 German Eannlsh dataset contains approximately 153,000 training pairs, 7,000 develop- ment pairs, and 7,000 test pairs from translated TED talks. We apply the same preprocessing as Ott ct al. 2018) and remove sentences longer than 175 words.
The IWSLTIS English-Vietnames Vietnamese dataset contains approximately 133,000 sentence pairs. follow Luong al. (2017) and use 1,553 sentence pairs in tst2012 the development set and 1,268 sentence pairs in tst2013 as the test set.
3.2
BASELINES
We compare 2 with two baseline models in experiments:
Seq2Seq-Alt sequence to sequence model with attention module (Bahdanau al.) 2014).
Transformer: a state-of-the-ar machine translation model based on Transformer Vaswani et al. 2017). use the model implemented in fairsed Gehring et al. 2017b).
NPMT: a baseline neural phrase- based machine translation model that only explicitly considers target segments (Huang al. 2018). We use the original implementation
Appendix. contains the implemenation details of the baselines and our model.
3.3
RESULTS
The IWSLT 2014 German- -nngishh and IWLST 2015 English-Vietnamess Vietnamese test results are shown in Tables 1 2P2MT achieves comparable results Transformer (Vaswani al. 2017). and outperforms other baselines in both datasets. NP²MT is not only better but also significantly faster than NPMT (Huang et al.) 2018). The proposed dynamic programming ($ (.3) reduces the complexity from O(SPP) in NPMT O(TP) NP"MT, where S and T' denote the length of source and target

---

Under review as a conference paper at ICLR 2020
sentence respectively, and P is the maximum segment length. Hence it results in approximately 10 times speedup at training time (2 hours vs 24 hours on an Nvidia V100 GPU).
BLEU
Model
Greedy
Beam
BLEU
Model
Greedy
Beam
BSO
23.83
25.48
Actor-Crite
27.49
28.53
Stanford NMT
23.30
DenseNMT
29.11
30.33
Hard monotonic attention
23.00
Graph2Seq
29.06
30.66
DeconvDec
28.47
SACT
29. 12.12
Seq2Seq with attention
26.17
27.61
NPMT
28.57
29.92
Seq2Seq with attention
25.50
26.10
NPMT
26.91
27.69
Transformer,
31.27
32.30
Transformer,
33.90
34.63
Transformer NP
29.72
30.74
MT
30.99
31.70
29.93
30.60
MT²
34.75
35.06
Table 1: Performance on IWSLT14 German-English (left-side) and IWSLT15 English-Vietnaneee Vietnamese (right-side) test set. For BSO (Wiseman & Rush, 2016). Actor-Critic (Bahdanau al. (016). DenseNMT (Shen et al. 2018 and Graph2Seq (Xu al. 2018), Stanford NMTnLuog & Manning (2013), Hard monotonic (Raffel et al.) 2017). DeconvDec (Lin et al., [20 80) and SACT (Lin et al. 2018b), we take the numbers from their papers. For the Seq2Seq with attention baseline (Bahdanau al., 2014) and NPMT (Huang et al., 20188. we take the numbers from Huang et al. (2018). We use the subscript 'w' and '6' to denote the word level and BPE level for our model and Transformer.
AP"M SLEU Imp.
APMM BLEU Ins.
Traneborer ILE
25
Tranetme BLCU Imp.
-00-
#P*MT UPR
-40-
AFTMP
-40- Transfomer LFR
Lockue in 1e
-d0-
Transfimar LPR
fasto
Lookue Pianees bakko
2.0
1.5
0.5
0.5
Source 0OV
Sourse 00V 5
(a) IWSLT14 DE-EN
(b) IWSLT15 EN-VI
Figure 3: Comparison on BLEU score improvement and lookup phrase ratio under different source language OOV rates both under in-domain dictionaries. NP and Transformer start from BLEU score 30.99 and 31.27 on DE-EN, 29.93 and 29.72 on EN-VI, respectively.
3.4 MEMORY- AUGOENEED TRANSLATION
In this subsection, we perform two experiments to understand the effectiveness of augmenting translation model with memory module such dictionary.
Rare Words We first consider setup where for every word (for Transformer) or every phrase (for NP²MT) that appears in the training data below a certain threshold (e.g., 3 times, 5 times, etc.), we deterministically look up the translation in the dictionary instead of using the segment decoder to generate target phrase.
We consider six lookup thresholds: 3, 5, 10, 20, 50, 100. For this experiment, we explore two ways to construct the dictionary: using in-domain dataset and out-of-domain dataset (denoted by D, and Do respectively). For the in-domain dictionaries, we have 97,399 and 84,817 entries for the IWSLT14

---

looked up from the external dictionary, and represents the frequency of the word replaced by the UNK token in the neural network model. The subscript represents the corresponding phrases alignments discovered by the phrase-attention mechanism ($..) Under review as a conference paper at ICLR 2020
German-Enge English and IWSLT15 English- Vietnamese respectively. For the German- Englishh translation task, we construct Do from the training set of WMT14
that has 1,243,722 entries.
We show the results in Table and Table Transformer benefit from the addition of dictionary, and that the increase in performance is more significant the higher the threshold is. There is no significant difference between using the in- in-domiin dictionary and the out-of-domain dictionary. However, we observe that when the lookup threshold is low, using a bigger dictionary (Do) is consistently better, whereas when the threshold is high Dy is better.
Our experiments demonstrate that both NP and
Figure 3 shows the dictionary usage ratio as a percentage of the number of translated words (Trans- former) or phrases (NP*MT). 3 shows the statistics on the length of the phrases on the IWSLT14 Ferman-Englise data using threshold 10 model as example. We can see that the model learns to capture phrases of various length automatically in both the source and the target side.
Source
[dies]] ist [die maschine]2 [unterhalb]s (von genf), 4 [.l [this is]; [the machinel2 [below]3 (geneva); [-]g
Greedy decoding
Target ground truth
this is the machine below geneva.
Source
[dies]] ist (eine bescheidene), [kleine]s (app), [-ls [this is]; (a modest), [little], (app), [-]s this is modest little app
Greedy decoding
Target ground truth
Source Greedy decoding
[unsere]; (zeitschriften), [werden von]3 [millionen], [gelesen]; [.le [our], (journals), [are]g [read byls [millions], [.le magazines are read by millions.
Target ground truth
Table 2: DE-EN translation examples, where "[-]" denotes the phrase boundary, "(-)" indicts phrase
We show translation examples from NP and lookup threshold 50 in Table 2 2P discovers meaningful source phrases such as "this is", "the machine" and "read by""
Length
2
3
4
5
German
25.5
30.1
21.9
11.0
5.3
6.1
English
65.0
24.3
7.9
2.2
0.4
0.6
Table 3: Phrase length statistics. The ratio (%) of the length of the phrases learned by NP2 the IWSLT14 German English when we set the threshold as 10.
Threshold
3
5
10
20
50
100
Transformer
29.72
29.71
28.97
28.40
25.68
23.00
Transformer + D,
29.72
29.72
29.01
28.71
26.92
25.02
NP
29.93
29.78
29.27
27.87
25.89
23.63
NP*MM+D, 29.94
29.79
29.36
28.38
27.65
26.31
Table 4: IWSLTIS English-
English-Vietamess Vietnamse Translation with an in-domain dictionary. We use Dj to denote in-domain dictionary.
Cross-Domain Translation In this experiment, evaluate our best two models: Transformer and NP on a cross domain machine translation task with and without memory modules. We train both models on the IWSLT14 German-English dataset (TED talks) and test them on the WMT14 German-English dataset (news articles). The goal of this experiment evaluate the effectiveness
7

---

Under review as a conference paper at ICLR 2020
Model
BLEU
Transformer
14.69
Transformer + DWMT
15.60
NT MP
14.86
16.11
Table 5: Results on cross-domain translation by training the model on IWSLT14 and testing WMT14, both with German-English data. In the test data from WMT14, OOV rates are 12.8% for German and 6.7% for English.
of the memory module in cases when there are many out of vocabulary words in more realistic setting. We construct a phrase dictionary using the training set of WMT14, although we do not train our models on this. Future work can explore better setups such as expert-curated dictionaries for cross-domain translation.
We show our results in Table While the addition of a dictionary improves the performance of both models, phrase-but -bsse model such as NP can use this memory module more effectively, as demonstrated by higher absolute gain (+1.25 vs. 0.91) in terms of BLEU score.
RELATED WORK
Huang al. (2018) introduced a variant of neural phrase-based machine translation that only considers target segments. Their model builds upon Sleep- WAke Networks (SWAN), a segmentation based sequence modeling technique described Wang et al. (2017a), and uses a new layer to perform (soft)
local reordering input sequences.
Another related work is the segment-to-segment neural transduction model (SSNT) Yu et al. 2016b Their model assumes monotonic alignment, which is often inappropriate in many language pairs. In contrast, model relies phrasal attention mechanism instead of marginalizing out monotonic alignments using dynamic programming.
There have been other works that propose different ways to incorporate phrases into attention based neural machine translation. Wang et al. (2017)), (Tagg al. (206)) and /Zha et al. (0018 incorporate the phrase table as memory in neural machine translation architectures. Hasler et al. (2018) use user-provided phrase table terminologies into NMT system by organizing the beam search into multiple stacks corresponding subsets of satisfied constraints as defined by FSA states. Dahlmann et al. (2017) divide the beams into the word beam and the phrase beam of fixed size. He al. (2016) uses statistical machine translation (SMT) as features in the NMT model under the log-linear framework, Yang et al. (2018) enhance the self-attention networks to capture useful phrase patterns by imposing learned Gaussian biases. Nguyen & Joty (018)) also incorporates phrase-level attention with transformer by encoding a fixed number of n-grams (e.g. unigram, bigram). However, Nguyen & Joty (2018) only focuses on phrase-level attention on the source side, whereas our model focuses on phrase-to-phrase translation by attending on phrase at the source side and generating a phrase the target side. Our model further integrates external dictionary during decoding which is important in open vocabulary and cross domain translation settings.
5 CONCLUSIONS
We proposed Neural Phrase-to-Phras Machine Translation (NP2MT) that uses a phrase-level attention mechanism to enable phrase-to-phrase level translation in neural machine translation system. We showed how to marginalize over all possible segments at training time and presented a greedy algorithm or beam search for decoding. We also proposed to incorporate a memory module derived from an external phrase dictionary to NP to improve decoding. Our experiments showed that MT outperforms the best neural phsese-based translation model (Huang al. 2018) both in
8

---

Under review as a conference paper at ICLR 2020
terms of model performance and speed, and is comparable to Stateoo--the-an Transformer-basee machine translation system (Vaswani et al. 2017)
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 149.0413, 2014.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.070066 2016.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign, IWSLT 2014. Proceedings of /WSLT, 2014.
Mauro Cettolo, Jan Nichues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. The IWSLT 2015 evaluation campaign. In International Conference Spoken Language, 2015.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. The best of both worlds: Combining recent advances in neural machine translation. arXiv preprint vrXiv:180400984 2018.
Leonard Dahlmann, Evgeny Matusov, Pavel Petrushkov, and Shahram Khadivi. Neural machine translation leveraging phrase-based models in hybrid search. Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1422-1431, 2017.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning, arXiv preprint arXiv:1705.03122 2017a.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional Sequence to Sequence Learning. In Proc. of ICML. 2017b.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with continuous cache. CoRR, abs/161 htt//611.000 1612.04426 2016.
Edouard Grave, Moustapha Cissé, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary. CoRR, 0tss/1711.22660 2017.
Eva Hasler, Adria de Gispert, Gonzalo Iglesias, and Bill Bymee Neural machine translation decoding with terminology constraints. In NAACL, pp. 506-512, 2018.
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. Improved neural machine translation with smt features. In AAAI, pp. 151-157, 2016.
Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. Towards neural phrase- based machine translation. International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Philipp Koehn. Statistical Machine Translation. Cambridge University Press, 2009.
Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation. In Proceed- ings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology Volume 1, pp. 48 48-54, 2003.

---

Under review as a conference paper at ICLR 2020
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACI. Interactive Poster and Demonstration Sessions, pp- 177-180. Association for Computational Linguistics, 2007.
Lingpeng Kong, Chris Dyer, and Noah A Smith. Segmental recurrent neural networks. arXiv preprint arXiv:1511.0660 2015.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. CoRR, abs/1709.07432, 2017.
Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2985-2990, 201 Sa.
Junyang Lin, Xu Sun, Xuancheng Ren, Shuming Ma, Jinsong Su, and Qi Su. Deconvolution- based global decoding for neural machine translation. In Proceedings of the 27th International Conference Computational Linguistics, pp. -260-3271, 2018b.
Adam Lopez. Statistical machine translation. ACM Computing Surveys (CSUR), 40(3) 2008.
Minh- Minh-Thang Luong and Christopher D Manning. Stanford neural machine translation systems for spoken language domains. In Proceedings of the International Workshop on Spoken Language Translation, 2015.
Minh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural machine translation (seq2seq) tutorial. 2017.
https//ggghuu..cco////ns///////
Phi Xuan Nguyen and Shafiq Joty. Phrase-based attentions. arXiv preprint arXiv:1810.03444 2011 2018.
Myle Ott. Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In EMNLP, 2018.
Colin Raffel, Minh- Kinh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. arXiv preprint 17X4:1170400784,, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:/550e 079009 2015.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie- Tie-Yan Liu. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long
Papers). pp. 1294-1225 1303, 2018
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. Neural Information Processing Systems, pp. 3104-3112, 2014.
Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, and Philip LH Yu. Neural machine translation with external phrase memory. arXiv preprint arXiv: 017600 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp.
2017.
Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. Sequence modeling via segmentations. International Conference on Machine Learning, pp. 3674- 3683, 2017a
Xing Wang, Zhaopeng Tu, Deyi Xiong, and Min Zhang. Translating phrases in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 1432- 1442. Association for Computational Linguistics, 2017b.
10

---

Under review as a conference paper at ICLR 2020
Sam Wiseman and Alexander M Rush. arXiv preprint arXiv:1606.0296)) 2016.
learning as beam-search optimization.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, and Vadim Sheinin. Graph2seq: Graph to sequence learning with attention-basee neural networks. CoRR, abs/1804.00823, 2018.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng. Lidia S. Chao, and Tong Zhang. Modeling localness for self-attention networks, In EMNLP, 2018.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. The neural noisy channel, arXiv preprint arXiv:/611.02554 2016a.
Lei Yu, Jan Buys, and Phil Blunsom. Online segment segment neural transduction. arXiv preprint arXiv:1609.08194, 2016b.
Yang Zhao, Yining Wang, Jiajun Zhang, and Chengqing Zong. Phrase table as recommendation memory for neural machine translation. arXiv preprint arXiv:180..09960, 2018.

---

Under review as a conference paper at ICLR 2020
A MT BEAM SEARCH ALGORITHM
Data: source sentence: T1.S, beam size B, maximum segment length P, maximum sentence length
N
Result: a target sentence:
Compute the representations Si,j for phrases given the maximum segment length P:
// Ye stores completed target sentences, represented tuples (SEN, PROB).;
ye 1-=13 (
// Vo stores open target sentences, represented as tuples (SEN, SEG, PRO3).
// The tuple tracks previously generated segments (gx.., ). previously generated words in the current segment (y2r) and the probablity of generating the target sentence so far.
y. = {(. 1)};
len 0;
while len < N do
// y. stores the open sentences of len + 1;
9. = 0:
for in yo do
for 1 = 0 to P do
Compute the distribution of the next output and select the top words Par from it: for P do
p PROB(m) Pie
if w not in {8, $$} then
// w continues the current segment."
Oseg SEG(m) w:
append (Osen: Ozeg, p) to yo;
else if w = $ then
// w ends the current segment.
Osen + SEN(m) + SEG(m);
Oseg +;
append (Osen: Oseg, p) yo:
else
// w $$ completes a sentence.
c - SEN(m) SEG(m)
append (c,p) Ve:
end
end end end
Retain only the top B candidates in 9o
y. + Vo: len len 1
end
return arg max(Y.)
Algorithm 2: NPºMT Beam Search.
12

---

Under review as a conference paper at ICLR 2020
B GERMAN-ENGLIS! TRANSLATION WITH IN-DOMAIN AND OUT-OF-DOMAIN DICTIONARIES
Threshold
5
10
20
50
100
Transformer
31.27
30.92
30.35
28.88
26.27
23.73
Transformer + D
31.27
30.97
30.65
29.34
27.78
26.01
Transformer Do
31.67
31.40
31.04
29.43
27.42
25.20
MT
30.99
30.92
29.86
28.29
25.81
23.08
NT Dr
30.99
31.01
30.33
29.52
28.02
26.61
2T + Do
31.48
31.75
31.06
30.04
27.86
25.85
Table IWSLTI4 German-Englisi Translation with in-domain and out-of-domain dictionaries. D, denote in-domain dictionary and Do denote out-of-domain dataset.
C IMPLEMENTATION DETAILS
Seq2Seq-AAteention We use the model architectures and results reported in Huang et al. (2018) as the baselines for Seq2Seq with attention. The Seq2Seq with attention baselines they built consist of 2-layer BiGRU encoder with 256 hidden units and 2-layer GRU decoder with 512 hidden units for the IWSLT14 German-English dataset, and 2-layer BiLSTM encoder with 512 hidden units and 3-layer LSTM decoder with 512 hidden units for IWSLT15 English- -iggiss-Vietnamese dataset. Adam algorithm Kingma & Ba 2014) is used for optimization with initial learning rate of 1e
We also use the model architectures and results reported in Huang al. (2018) for the NPMT baseline. The NPMT models consist of a reordering layer, a 2-layer BiGRU encoder with 256 hidden units and 2-layer GRU decoder with 512 hidden units for the IWSLT114 German-English dataset, and reordering layer, 2-layer BiLSTM encoder with 512 hidden units and 3-layer LSTM decoder with 512 hidden units for the IWSLT15 English- Vietd---eee dataset. The models are optimized using the Adam algorithm with an initial learning rate of 13
NPMT
We the number of layers in both encoder and decoder 6. Our preliminary experiments suggest that increasing the number of layers does not improve performance further.
Transformer
We use Adam Kingma & Ba 2014) our optimization method and search the best hyperparameters using a validation set. We use a three-stage learning rate scheduler by replacing the exponential decay with linear one for fast convergence, similar the RNMT+ learning rate scheduler (Chen al.) 2018 Ott et al. 2018). Specifically, our learning rate is quickly warmed up to the maximum, kept at the maximum value until 50% training iterations, and then decayed to zero. The maximum learning rate Transformer set to 2e
MT We l-yerer BiLSTMs encode both words and segment level in the source encoder, 6-layer LSTMs as the target encoder, and 6-layer transformer the segment decoder We also use Adam (Kingma & Ba 2014) as our optimization method. We set the word embedding dimension 256 In our experiments, we set the maximum learning rate to le -3 weight decay to le 1 and the dropout rate to 0.4. We set the word embedding dimension to 256. For 6 layer encoder/decoder,, 2P2MT and Transformer use 442M and ~75M parameters, respectively.
'In our preliminary experiments, reran the NPMT open source code. We found the NPMT docsn't improve with deeper layers. We also experiment L.STM segment decoder variant, hut the performance is inferior.
13


Reviews:
[{"review_id": "SJlq8XkRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a phrase-based encoder-decoder model for machine translation. The encoder considers all possible phrases (i.e. word sequences) up to a certain length and compute phrase representations using bidirectional LSTMs from contextual word embeddings computed with another bidirectional LSTM layer. The decoder also considers possible segmentations and computes contextual representations for the previously generated segments. Each word in the current segment is generated by a Transformer model by attending to all phrases in the source sentence. The authors present a dynamic programming method for considering all possible segmentations in decoding. They also present a method for incorporating a phrase-to-phrase dictionary built by Moses into the decoding process.\n\nI like the idea of phrase-to-phrase translation and the relatively simple architecture proposed in the paper. At the moment, however, I am not quite sure how practical their approach is. One reason is the experimental setting. Both of the datasets used in the experiments are quite small and it is not clear how the proposed model performs when several millions of sentence pairs are available for training. \n\nAnother reason is that the computational cost of the proposed model is not really clear. The authors state that it is much more efficient than NPMT but it is not clear how it compares to the standard Transformer approach. It seems to me that the computational cost of their model is highly dependent on the value of P (maximum length of phrases). \n\nAt first, I thought the decoder was implemented with LSTMs, but I realized that it was actually implemented with a Transformer by reading the appendix. I think this should be explained in the main body of the paper. I am also wondering how the authors’ model compares to a standard seq-to-seq model whose decoder is implemented with a Transformer.\n\nThe equation in section 2.2 seems to suggest that the model prefers segmentations with small numbers of segments. I am wondering if there is any negative effect on the translation quality.\n\nHere are some minor comments:\n\np.2 valid of -> valid\np.4 lookup -> look up?\np.4 forr -> for\np.4 indict -> indicate?\np.5 Table 1 -> Table 1", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx5Mz-xiH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an end-to-end phrase-to-phrase NMT model (NP2MT). I think the contribution of this paper is incremental and the idea is of less novelty. In general, the model is largely based on the NPMT model, where modification is the introduce of phrases in the source sentences. Then the author proposed the memory module strategy. In the experiments, the performance improves significant when using out of domain dictionary, but less significant for in-domain dictionary. I also have a concerns about the experiments. The dataset used in this paper seems not convincing to me. By my own experience, the performance on small dataset for either LSTM or Transformer is not stable. The authors just tested the model performance on WMT test set. I think at least the WMT training data should be used for training as well. Another question is the details about the training time and decoding time, since the dynamic programming is used.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgZqngyqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the field of machine translation. In particular, it looks at the problem of phrase-to-phrase translation (previously used state of the art approach) using neural network approaches. The main idea behind this paper is to use a segmental (analogue of phrases) form of neural networks on the source side and attention mechanism to align those segments with segments generated on the target size. This submission additionally describes how an external dictionary can be incorporated using a heuristic approach. I believe this submission could be of wide interest to the machine learning community. I find experimental validation to be satisfactory whilst presentation unsatisfactory for the following reasons:\n\n1) notation\n\nGiven that you are dealing with two sets of sequences (source, target side), segments on both sides, attention linking both sides I find it strange that the notation used is not carefully introduced and clearly explained. What is ${\\bf g}_{{\\bf z}<z_k}$ using precise mathematical language? How it is different from ${\\bf g}_{{\\bf z}<k}$ and what that is? The same for ${\\bf y}_{<t}^{z_k}$, a*, d* and all other variables. In order to help the reader understand your approach it is fundamental to be precise and not ambiguous about each (!) symbol you are using. \n\n2) Algorithm 1, 2 and Figure 1\n\nThe algorithmic description was meant to help the reader to understand the process. Unfortunately I have to disagree that is has accomplished this purpose. Please make sure you have unambiguously explained every single term, explicitly say what you are running argmax over, etc. Please also make sure you are discussing/describing the algorithms/figures in your submission. Given the non-trivial nature, lack of proper introduction into the notation used, you cannot simply point the reader to it and not discuss it. \n\nMinor comments:\n\nPlease refrain from using \"vanilla model\" unless you can cite a publication defining exactly what that is.\nPlease explain how did you derive dictionary. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes how they extend a previous phrase-based neural machine translation model to incorporate external dictionaries. The reviewers mention the small scale of the experiments, and the lack of clarity in the writing, and missing discussion on computational complexity. Even though the method seems to have the potential to impact the field, the paper is currently not strong enough for publication. The authors have not engaged in the discussion at all. 
Decision: Reject