Enhancing Attention with Explicit Phrasal Alignments
Authors: Xuan-Phi Nguyen
Shafiq Joty
Thanh-Tung Nguyen
Venue:  (2020)
Keywords: NMT
Phrasal Attention
Machine Translation
Language Modeling

Abstract: The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system. However, existing attention methods are often token-based and ignore the importance of phrasal alignments, which are the backbone of phrase-based statistical machine translation. We propose a novel phrase-based attention method to model n-grams of tokens as the basic attention entities, and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Our approach yields improvements in English-German, English-Russian and English-French translation tasks on the standard WMT'14 test set. Furthermore, our phrasal attention method shows improvements on the one-billion-word language modeling benchmark.


Under review as a conference paper at ICLR 2020
ENHANCING ATTENTION WITH EXPLICIT PHRASAL ALIGNMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
The attention mechanism is an indispensable component of any state-of-the-art neural machine translation system. However, existing attention methods are often token-based and ignore the importance of phrasal alignments, which are the back- bone of phrase-basee statistical machine translation. We propose a novel phrase- based attention method to model n-grams of tokens the basic attention entities, and design multi-headed phrasal attentions within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Our approach yields im- English-Russe Russian and English- French translation
provements in English-German tasks on the standard WMT' 14 test set. Furthermore, our phrasal attention method shows improvements on the one-billion www.bbiilionwwoeee -orld language modeling benchmark.
INTRODUCTION
The encoder-decoder neural architectures have established breakthroughs in many natural language processing (NLP) tasks including machine translation (Luong et al. 2015). summarization (See et al., 2017), and parsing (Vinyals et al. 20115). Particularly for machine translation, most state of the art neural machine translation (NMT) models possess attention mechanisms to perform align- ments of the target tokens to the source tokens. The attention module therefore plays a role anal- ogous the word alignment model in Statistical Machine Translation
SMT (Koehn 2010). In fact, the Transformer network introduced recently by Vaswani et al. (2017) achieves -tate-of-the-art performance both speed and BL.U scores (Papineni et al. 2002) by using only attention modules. Furthermore, its proposed self celf-attention layer extends the application of attention way beyond the scope of machine translation and achieves tremendous success in other NLP tasks such as contextual representation learning (Devlin al. 2018) and machine reading comprehension YYu al. 2018
On the other hand, phrasal interpretation is an important aspect for many NLP tasks, and forms the basis of Phrase-Baas -asse Machine Translation (Koehn) 2010). Phrasal (n (n-gram based) alignments (Koehn et al. (200)) can model one-to-one, one-to-many, many-to-one, and many-to-many rela- tions between target and source tokens while exploiting local context. They also robust compositional phrases. Despite the advantages, the concept of explicit phrasal attentions has largely been neglected in neural NLP. In fact, most language generation models produce sentences token- by-token autorcgressively, and tend to use the token-based attention method which is order invariant.
Therefore, the intuition of phrasal alignments is vague in existing systems that solely depend on the underlying neural architectures (recurrent, convolutional, or self-attention) to incorporate contextual information. However, the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues to represent only the current token, and do not explicitly model phrasal alignments. We argue that having an explicit inductive bias for phrases and phrasal alignments is beneficial for neural sequence transduction models to exploit the strong correlation between source and target language phrases. This would also make self-attention more capable of tackling other sequence modeling tasks such as language modeling.
In this paper, we propose a novel n- n-gram-level attention technique to leverage phrasal alignments in various NLP tasks. The technique designed to assign attention scores directly phrases in the source and compute phrase-leve attention vectors for the target token. It is then applied in a new attention structure to conduct token-to-token and token-to-phrase mappings.

---

Under review as a conference paper at ICLR 2020
To show the effectiveness of our approach, we apply phrase-based attention method all multi- head attention layers of the Transformer network. Our experiments on WMT 14 translation tasks show improvements of up to 0.69, 1.84 and 1.65 BLEU in English-to-German English-to-Ruee Russian and English-to-Frende translation tasks respectively, compared to the baseline Transformer trained in identical settings. Furthermore, when evaluated on the language modeling task, our phrasal mod- els outperform the Transformer base model by up to 4,6 points in perplexity on the one-billion-orr language modeling task. We make our code available at annoyymou for research purposes.
BACKGROUND
Most NMT models adopt an encoder-decoder framework, where the encoder network first trans- forms an input sequence of symbols x = (x1,T2, 22, (.n) sentations Z (11. 22...., .n.)...................... From this, the decoder generates a target sequence of symbols = (31. ....... Vm autoregressively, one element at a time. Recurrent seq2seq models with di- verse structures and complexity (Sutskever et al. 2014 Bahdannuu et al., 2014, Luong et al., 2015] Wu et al. 2016) were the first to yield state-of-the-an results. Convolutional seq2seq models (Kalch- brenner et al. 2016; Gehring et al., 2017; Kaiser et al. 2018) alleviate the drawback of sequential computation of recurrent models and leverage parallel computation to reduce training time. WW et al. 20199) recently proposed a light-weight convolution, which shows very promising results.
to a sequence of continuous repre-
The Transformer network (Vaswani et al. 2017) structures the encoder and the decoder entirely with stacked self-attentions and cross-attentions (only in the decoder). In particular, it uses a multi- headed, scaled multiplicative attention defined as follows:
Attention (qaution(QQu0 K, V, Wg. WR, W.) = softmax (QW,) (WW)) (VW,)
(1)
Head' Attention (Q, K, V, W W W for i 1.... AttentionOOutuu(Q.... V, W) concat(Head Head? ...... Head")
(2)
where Q E RR K E RR and V E RR are the matrices with query, key, and value vectors respectively, with d being the number of dimensions; W W W are the head-specifie weights for query, key, and value vectors respectively, and W is the weight matrix that combines the outputs of the heads. To encode a source sequence, the encoder applies self-attention, where Q, K and V identically contain the same vectors coming from the output of the previous layer. In the decoder, each decoder layer first applies the masked-seff-attenn previous layer. The results are then used as queries to compute cross-attentions (encoder-decodder attentions) over the encoder states. For cross-attention, Q comprises of the decoder self-attentio states, while K and V comprise of the encoder states. We refer the reader to /Vaswani et al.) 2017) for further details. Since the self-attention based encoder (or decoder) considers the input as fully- connected directed graph, it can model long-range dependencies by explicitly attending to all tokens, and the maximum path length that signals need to travel is O(1). The non-sequential computation also makes it highly parallclizable to multiple threads.
over the outputs from the
However, one crucial issue with the attention mechanisms employed in the Transformer as well as other NMT architectures is that they are order invariant locally and globally. That is, changing the order of the vectors in Q, K and V does not change the resulting attention weights and vectors. If this problem is not tackled properly. the model may not learn the sequential characteristics of the sentences. RNN-based models (Bahdanau et al. 2014 Luong et al.] 2013) tackle this issue with a recurrent encoder and decoder, CNN-based models like (Gehring et al. (017) use position embeddings, while the Transformer uses positional encoding. [Shaw et al. (2018) further encode relative positions inside the self-attention layers.
Another limitation is that these attention methods attend to tokens, and play a role analogous to word alignment models in traditional SMT. It is, however, well admitted in SMT that phrases are better as translation units than words (Kochn) 2010). Without explicit attention to phrases, a particular attention function has to depend entirely the token-level scores of phrase for phrasal alignment, which is not robust or reliable, thus making it difficult for the model to learn the required mappings. For example, the attention heatmaps of the Transformer (Vaswani al. 2017) show concentration
2

---

Under review as a conference paper at ICLR 2020
of the scores on individual tokens even if it uses multiple heads concurrently in multiple layers. Our main hypothesis is that in order to exploit the strong correlation between source and target phrases, the NMT models should have explicit inductive biases for phrases.
There exists some research on phrase-basee decoding in the NMT framework. For example, Huang et al. 2018) proposed phrase-basee based decoding approach based soft reordering layer and sequence model proposed by [Wang et al. 20177). Their decoder uses a recurrent architecture without any attention the source. [ang al. 2016) and Wang et al. (2017)) used an external phrase memory to decode phrases for Chinese- to-English translation. In addition, hybrid beam search with phrase translation features from the statistical phrase table of a phrase-based SMT system was used to perform phrasal translation in (Dahlmann et al., 2017). Nevertheless, to the best of our knowledge, our work is the first to embed phrases into attention modules, which then propagate the information through the entire end-to-end Transformer network, including the encoder, decoder, and the cross-attention.
Sleep- WAke Network (SWAN), a segmentation-
3 PHRASAL ATTENTION MODEL
In this section, we present our phrase- based attention model. In Subsection 3.1 we first describe our proposed module to compute attention weights and attention vectors based n-grams of queries, keys, and values. Then, Subsection (.2, we describe how we use this module to compute different kinds of phrasal attentions. We describe methods in the context of the Transformer (Vaswani et al.] 2017). However, is straight-forward to adopt them into other architectures such as the RNN- or CNN-based seq2seq models.
Key Operations and Notations. The core element in method is temporal (or one- dimensional) convolutional operation that is applied to a sequence of vectors representing the tokens sequence. Formally, we define the convolutional operator applied to each token 2; with the cor- responding vector representation x, RR as:
aij w (e
where n the convolution window size, w, E IRdd pd is the kernel weight taken from the weight matrix W E Ix" and @ denotes vector concatenation that produces a vector in IR." ad The convolution over the entire sequence X E !xd of length gives a feature map aj. By repeating this process with different weight vectors from W, we get d-dimensional representation for each token 2g. In the rest of the paper, we will use the following notations.
Conv, (X, W) to denote a convolution operation with window size n and kernel weights W E nxxd over an input sequence X E Idd based on the definition in Equation 4 The result of this convolution is matrix in IR (I-++))xd whose columns represent the feature maps corre- sponding to the kernels, and rows represent feature representations corresponding to the tokens in the sequence.
SConv, (X, Y) to denote a serial convolution operation with window size n over an imput R!xd for all the rows W, E IR" axdx1 in the kernel tensor (WI.... W,) e xxxxxxx1 In other words, SConv,, (X, Y) o Conv, X, W;) that produces matrix in IR. (1==++1)
(x) to denote a reshape operation on the vector E IRnd to reshape it to a matrix X E mxxxx1 and v/n, (X) to denote a serial reshape operation reshape the matrix X E IR (x((d))) to tensor X IR* xxxxdx1
softmax( A) to denote a softmax operation over each row of the matrix A. Formally, for A' exp(A(i.f)) softmax( A), each entry ji )) is computed as: jii Σ, expl A(t.)))
TTessorflow: tf.reshape(; Pytorch: .vie(()
3

---

Under review as a conference paper at ICLR 2020
3.1 n-GRAM-LEVEL ATTENTION
Our n-gram based attention method computes the attention weights and vectors based on 7-gram representations of the tokens. In particular, when computing the attention scores a, E IR for each query qi in Q = (q I1,)) E IR xd we use the query vector qi as the kernel in the convolution operation applied to the set of keys K (k1....,ki,)
kin) E IR xd More formally,
a; Conv W ks Pn (q W)
(5)
vd n
where We E IR Rd and W, E Iπdx(nd) are trainable weights (head weights) to linearly transform K and respectively, and P. is the required reshaping operation make the dimensions compat- ible. Applying Equation [ for all the queries in Q gives the score matrix A = (a1,..., at,)) E
R where each vector a, A contains the attention scores for the query vector q: Q.
Next, we compute the attention (or context) vectors as follows.
C = softmax A) Conv,, (V. W.)
(6)
where W. E RRdxxx are the kernel parameters to achieve the n-gram representations by convoly- ing over the value vectors V (V),.... 0%,)) RRlexd which are in turn linearly combined using the attention weights computed by the softmax.
By using the queries as the kernel parameters, we allow the queries to dynamically (kernels vary for different queries) and directly interact with the window of key vectors and compute the n-gram based attention scores. An alternative way achieve phrasal attentions would be use A
(QW,) Conv,, (K.W.) to compute the attention scores in Equation In contrast to approach, Vd. this method is static and indirect in the sense that the convolution uses static kernel and the queries do not interact directly with the keys; instead the model relies on the kernel weights (W&) to learn n-gram patterns. In our initial experiments, we also found the direct approach to perform better.
3.2 MULTI-HEADED PURASAL ATTENTION
Having presented our core method to perform attentions based on n-grams, we now introduce our novel extension to the multi-headed attention framework of the Transformer to enable it to pay attention not only to tokens but also to other ri-grams across many sub-spaces and locations. In particular, each head uses the token (unigram) representation of the query to attend to all n-gram types (e.g., n ..2..... .N) simultaneously. To achieve this, we first the query vectors to compute the attention scores for each n-gram type separately by performing convolution over the key vectors with the respective window sizes. Áll the n-gram scores are then concatenated before passing them through a softmax to compute the attention weights over all n-grams. Similarly, the value vectors for the n-gram types are concatenated produce the overall attention output. Figure I exemplifies the process for attentions over unigrams and bigrams. For self-attention, the query vectors in Q represent the unigrams ('India', 'and'. 'Japan') of the input sequence, whereas for cross-attention, they represent the decoder states of the target side tokens. The overall attention output C₂ can formally described by the following sequence of equations:
Q QW = 9.2
A = (KW,)
(8)
Vd
A. SConv,, (K WK, 2/ (Qn))
vd
S, softmax ([amaax(AAA;AAA) A2: ...; AA,))
(10)
V, (VW (VW1));Com (V, Wo,2);.... Conv, (V, Won))
(11)
C, S.
(12)
4

---

Under review as a conference paper at ICLR 2020
India
and
Japan
ministers
meet
Tokyo
India and
and
Japan
prime
ministers meet in
Japan
prime
ministers
meet
in Tokyo
Figure I: Multi-Headed phrasal attention for each head. Each query token attends to uni- and bi- grams of key/value tokens.
where W E (nd) and W v.a E Raxdx are the respective weight matrices for the queries and values, and W E IRaxd d is weight matrix for the keys that shared across all n-gram types.
Note that the serial convolution SConv, and the serial reshape v.. functions in Equation implies the computation of all attention scores a, E A, (i.e., applying Conv,, and Pr. Equation 5 for each row of Q). We can aggregate multiple n-gram types within an attention module (e.g.. T-2-3 grams). our method, we not need to pad the input sequences before the convolution operation to ensure identical sequence length. The key/value sequences that are shorter than the window size do not have any valid phrasal component to be attended.
4 EXPERIMENTS
To demonstrate the effectiveness of our phrase-based attention method, experiment with two different tasks: machine translation (MT) and language modeling (LM). In the following, we present the training settings, experimental results and analysis of models on these tasks.
4.1 TRAINING SETTINGS
We replicate most of the training settings from Vaswani et al. 2017) for our models, to enable fair comparison with the original Transformer and the Transformer with relative positioning (Shaw ct al. 2018 Specifically,
use the Adam optimizer (Kingma & Ba 2014) with 81 0.9, 3g 0.98, and c 10-9. We follow a similar learning rate schedule with a warmup steps of 16000 updates: Learning Rate 2 d 0.5 min(step "miinstpnnumm
"_mmm -0.5 step_numm
warmup_stegg -1.5.
Similar to Vaswani et al. (2017), we also applied residual dropout with 0.1 probability and label smoothing with 6ts 0.1. Our models are implemented in the tensor2tensof library (Vaswani et al. 2018), on top of the original Transformer codebase.
We conducted all the experiments with models and the original Transformer in identical setup for a fair comparison. While Vaswani et al. (2017) trained their base and big models at massive scale with GPUs, could only train our models and the baselines a single GPU because of limited GPU facilities. However, we could (virtually) replicate the 8-GPU setup with a single GPU following the gradient aggregation method proposed recently by Ott al. (2018).
Setup for MT Experiments. To compare models with state-of-the-art models, train all the models with the identical 8-GPU (by gradient aggregation) setup on WMT' 16 English-German (En-De), WMT' 17 English-Russian (En-Ru) and WMT' 14 English-Frende French (En-Fr) datasets. The training datasets contain about 4.5, 25, and 35 million sentence pairs for En-De, En-Ru, and En-Fr. respectively. The effective batches were formed by sentence pairs containing approximately 32,768 source and 32,768 target tokens.
All translation tasks are evaluated in case-sensitive tokenized BLEU. For validation (development) purposes, we use newstesf2013 for En-De, newstest2016 for En-Ru, and a random split from the
5

---

Under review as a conference paper at ICLR 2020
Model
#- Params
N-grams
En->De
En-
En->Fr
Base size
Vaswani et al.
(0177)
63M 63M
27.16
34.37
39.21
Shaw et al. 2018)
27.20
33.59
39.37
Big size
Vaswani et al. 2017
214M
27.77
35.54
40.61
Ours, Base size
Ours Ours
8IM
1-2 1-2-3
27.70
35.44
40.57
110M
27.85
36.21
40.86
Table 1: BLEU (cased) scores on WMT" 14 testsets for En-+ee, En->Ru and En-+Fr translation tasks. All the models were trained with gradient aggregation to replicate a 8-GPU setup on a single physical GPU.
training set for En-Fr. We evaluate all our models on WMT' 14 translation tasks (newstest2014 test sets). We use Byte-Pair Encoding BPE (Sennrich al. 2016) with combined (source and target) vocabulary of 37,000 subwords for En-De, 40,000 subwords for En-Ru and En-Fr. We take the average of the last 5 checkpoints (saved at 5000-update intervals) for evaluation, and use beam search size of and length penalty of 0.6 (Wu et al. 2016).
Setup for LM Experiments. For our LM experiments, we use the One Billion Word Benchmark dataset (Chelba et al. 2013), which contains 768 million words of data compiled from WMT 2011 News Crawl data with a vocabulary of 32,000 words. We use its held-out data as the test set. We train the base-size models (monolingual decoders) on virtually GPUs for 100,000 updates by using gradient aggregation technique on single GPU. The effective batch size is 16,384 tokens.
4.2 MACHINE TRANSLATIIO RESULTS
Comparison with State-of-the-art. In Table we compare our models on the En-*De, En Ru and En-tFr translation tasks with the base and big size Transformer models (Vaswani et al., (017), and the base size Transformer with relative position model (Shaw et al. 2018). All the models are trained in the identical 8-GPU setup using gradient aggregation for 100K updates for En-*De, 120K updates for En-Ru, and 150K updates for En En-FFr.
Generally,
that models outperform the base models. Some of them also surpass the Transformer big model trained in identical settings, while having only half the number parameters.
More specifically, the En-De translation task, base-size model with 1-2 grams achieves a BLEU of 27.70, exceeding the Transformer base by 0.6 BLEU. Including higher order n-grams contributes further improvements, reaching up 27.5 .88 BLEU for 1-2-3 grams. It also performs par with the big Transformer, while requiring less than half of the number of parameters.
Likewise, the En- translation task, models boost the performances significantly over the Transformer base. They achieve 36.21 BLEU with 1-2-3 grams, which is 1.84 points higher than that of the Transformer base. They also outperform the Transformer big by 0.67 points.
Similar trends can also be seen in the En- translation task, where our model with 1-2-3 grams 40.86 BLEU, exceeding the Transformer base by 1.65 points. It also outdoes the Trans- former big by 0.2, even though it has considerably fewer parameters. These results demonstrate the effectiveness of our approach over the existing methods.
Effect of Higher-Orded n n-grams and Training Batch Size. To analyze our models further with respect to the effect of higher-order n-grams and the impact of training batch size, we conducted another set of experiments on En-De and En-Ru translation tasks (both directions) in a single-GPU setup, and compared with the Transformer base and big models on the same setup. The effective
http://www..saam..oo///mmeeeeaa

---

Under review as a conference paper at ICLR 2020
Model
N-grams
En->De
De-+En
En-+Ru
Ru-*En
Transformer big
26.62
32.31
Transformer base
26.31
29.76
33.12
32.87
Ours
1-2
27.11
30.16
34.18
33.12
Ours
1-2-3
27.37
30.55
34.72
33.53
Ours
1-2-3-4
27.37
30.04
34.90
33.70
Table 2: BLEU (cased) scores WMT 14 testsets for English- Gerimnn (En-De) and English-
Russian (En-Ru) language pairs (in both directions). All models were trained with 1 GPU (no gradient aggregation). The decrease in scores in this table compared to the ones in Table i due to the number of GPUs used (I vs. 8).
Model
N-grams
Perplexity
Vaswani al. (017))
46.37
Shaw et al. 2018
46.13
Ours
1-2
41.77
Table 3: Perplexity scores on one-billion-woe word language modeling benchmark. All models are of base-size and were trained for 100K updates with gradient aggregation to produce virtual 4-GPU setup on a single GPU.
batch size in this experiment 4096 tokens. En-De models were trained for 500K updates while En-Ru ones were trained for 900K updates. The results are shown in Table 2
It is evident that almost all of models achieve higher BLEU scores than the Transformer base and big models. For the En-De pair, our model with 1-2 grams achieves 27.11 and 30.16 BLEU for En->De and De-+En translation tasks, giving improvements of 0.8 and 0.4 points, respectively. Including higher-order n-grams does improve the performance (for 1-2-3 grams), but diminishing patterns are also observed (for 1-2-3-4 grams) these tasks.
On the other hand, for the En-Ru pair, the models with higher-order n-grams excel in BLEU per- formances. Specifically, with 1-2-3-4 grams it yields up 34.90 and 33.70 BLEU for En and Ru-FEn surpassing the baselines by 1.78 and 0.83 points, respectively.
If compare the corresponding translation results for En-*De and En in Table vs. Table 2. we notice significant drop in BLEU for the 1-GPU setup across all the models, though the 1- GPU models were offset by longer training (more updates). The performance of the Transformer big is outstandingly low in the 1-GPU setup. There has been evidence that practical training of the Transformer (theirs and ours) is significantly susceptible to the batch size (which increases with the number of GPUs used), and training does not produce similar results as with or more GPU settings; please see Popel & Bojar (018); Ott al. 2018 and the diseussions" for details on this issue.
single GPU with lower batch size for sufficiently long
4.3 LANGUAGE MODELING RESULTS
Table presents the results on the language modeling task, where compare our model with uni- and bi-gram attentions against the three baseline models in terms of perplexity. Our model outperforms the Transformer base by 4.6 perplexity points. It also surpasses the Transformer with relative positional encoding (Shaw et al., 2018) by comparable margin. These results demonstrate that our models are superior the baselines in the language modeling task, well.
4.4 MODEL INTERPRETATION
To interpret our phrasal attention models, we now discuss how they learn the alignments. Figure 2 and show attention heatmaps for an En-De sample in newstest2014; Figure 2 displays the heatmap layer (mid layer), while Figure shows the one in layer 6 (top layer) within a 6-layer

---

Under review as a conference paper at ICLR 2020
Pote
Landgo
110000
ate contrane
by Plier
Ainthe ond
Frome fower coltrrghe
Zhanc
Figure 2: Attention heat maps at layer of our model for a sample sentence pair in English-German German
newstest2014 14 test set. The left half in each figure indicates token-to-token mappings, while the right half indicates token-to- ooken-to-phrase mappings.
bastagg
Peter
42
Larop
Deserd
ahe conteredd controned
nore Arrapg amate ood
neree
foor tte nee ortonaai creenttt Airp her piorst
outerr cotice
cefice
Figure 3: Attention heat maps at layer 6 of our model for sample sentence pair in English- German newstest2014 test set. The left half in each figure indicates token-to-tts to-token mappings, while the right half indicates token-to-phin to-paasseeeeeeeee mappings.
Transformer with our phrasal attention. Each figure shows two squares representing token-to- to-keen (left square) and token-to- token-to-phrase (right square) attentions, respectively. We can see in the two figures that phrasal bakken-to-phrasee attentions are activated strongly in the mid-layers. token-token attentions are activated the most in the top layer, whose final representations are used to predict translated tokens. Although the distribution of attentions can vary depending on model initialization, we observed that 50%-60% of the attentions are concentrated on phrases.
the other hand,
CONCLUSION
We have presented novel approaches to incorporating phrasal alignments into the attention mecha- nism of state-of-the-art sequence transduction models. Our methods assign attentions both tokens and phrases of the source sequences. While we have applied our attention mechanism to the Trans- former network, effectiveness of our approach on two NLP tasks. On machine translation, our models show signif- icant gains on WMT' 14 English-German, English-Ruu Russian, and English-Fres -renchh translation tasks. Our phrasal models also outperform the Transformer in the one-billion-we wordd language modeling task. We are planning future extensions of our techniques to other tasks, such as summarization and question answering. We also plan to improve models with phrase-based decoding procedure.
generic and can be implemented in other architectures. We have shown the
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1et.:.449..43 2014.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. Technical report, Google, 2013. URL
awaaaass//// 3005
Leonard Dahlmann, Evgeny Matusov, Pavel Petrushkov, and Shahram Khadivi. Neural machine translation leveraging phrase- based models in hybrid search. In Proceedings of the 2017 Con- ference Empirical Methods in Natural Language Processing, pp. 1411-1420. Association for antho wtt.r////anhooollg//-------11111
Computational Linguistics, 2017. htR.ttt.::/// //tp:://cct http::///wwwb.....
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18.00400 2018.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional Sequence to Sequence Learning. In Proc. of ICML, 2017.
Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. Towards neural phrase- based machine translation. ICLR, 2018
Lukasz Kaiser, Aidan N. Gomez, and Francois Chollet. Depthwise separable convolutions for neu- ral machine translation. In International Conference on Learning Representations, 2018. URL
net https::/openrew// wwwp:/eeeeeeeooooneeeeeeee Forumiid=S11Bcuue.]
Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron den Oord, Alex Graves, and Koray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv: 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1:Xiv::412.6990 2014.
Philipp Koehn. Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition, 2010. ISBN 0521874157, 9780521874151.
Philipp Koehn, Josez Och, and Daniel Marcu. Statistical phrase-based translation. In Proceed- ings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technology Volume 1, NAACL '03, pp. 48-54, Stroudsburg, PA, USA, 2003, Association for Computational Linguistics. doi: 00.3115/1073445.00730 URL / org /10./10..000 31.5/111111771311 34400 1073462
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1412-1421. ACI., 2015.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT), 2018
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Martin Popel and Ondłej Bojar Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1):43-70, 2018
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18663//1//11-10000 URL https: 7 7/www. c web. www.wwww.........weeeeeeoe web..oorg/antttooooo///1111111--

---

Under review as a conference paper at ICLR 2020
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725 Association for Computational Linguis- tics, 2016. doi: (0.18653/11/11--1166 URL http: /wwp:///www./o/.ooo/////////. ac www..w..........................
P16 1162
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pp- 464 468. Association for Computational Linguistics, 2018. doi: 10.18653/11//18-2077 URL http://aclweb: org/ http://wwww.gg/tohooooooooo orggannoologyy////oooooooooooo// / N1182074
Ily Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In 7. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp- 3104-3112. Curran Associates, Inc., 2014. URL http::/rappess.niss... wwtpppp....pppprr.pppppppppppppe 5346 sequence to sequence Secnnnee-eeaannn learning with neural -eell-nnewwwssss pdf
Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, and Philip LH Yu. Neural machine transla- tion with external phrase memory. arXiv preprint / (06.000772222 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer. and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs//803.00741, 2018. URL httt::tt//aaaae wttpaaavvv.orr/ab///////// http:/bbs/11033
org/
07416
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, llya Sutskever, and Geoffrey Hinton. Gram- 7449.
mar as foreign language. In NIPS, 2015.
Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng- Sequence modeling via segmentations. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 3674-3683, 2017a.
Xing Wang. Zhaopeng Tu, Deyi Xiong, and Min Zhang. Translating phrases in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 1421-1431. Association for Computational Linguistics, 2017b. URL http: //aclweb.org//ne htwpee...orr.aaaooooool//111111-
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference Learning Representations, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 011.66609946440 2016.
Adams Wei Yu, David Dohan, Minh- Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe V Le. Qanet: Combining local convolution with global self-attention for reading compre- hension, arXiv preprint arXiv:1804.000 2018.
10


Reviews:
[{"review_id": "rkxr4LdpYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims to incorporate phrase representation into attention mechanism. The proposed method is straightforward (which is good): a convolution window with size n is used to calculate representation for an n-gram, which then replaces the token representation in a standard attention model. The paper implements the multihead version of the proposed phrase-based attention, more specifically, in a transformer model. Experiments with machine translation and language modeling show that it outperforms the token attention counterpart.\n\nMy main concerns are about the experiments: \n\n- For the translation experiments, the transformer numbers are lower than those reported by Vaswani et al. (2017) across the board, for both the \"base\" and the \"big\" settings. I didn't find convincing reason for this. Could the authors comment on this, and also on why they do not directly compare against Vaswani et al. (2017). The same for the language modeling experiments.\n\n- Table 3. To the best of my knowledge, neither Vaswani et al. (2017) or Shaw et al. (2018) report language modeling results. I'm guessing the authors use their implementation and establish their own baselines, which is okay. But please indicate this instead of putting a citation in the table, which can be misleading.\n\n- Minor: the start of Section 3 reads like source code and comment, and is a bit hard for me to follow.\n\nI do not recommend the that the paper is accepted, until the authors address my concerns on the baselines. \n\n\nMissing references:\nPaLM: A Hybrid Parser and Language Model. https://arxiv.org/abs/1909.02134", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byl2Sx7aYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an extension of the attention module that explicitly incorporates phrase information. Using convolution, attention scores are obtained independently for each n-gram type, and then combined. Transformer models with the proposed phrase attention are evaluated on multiple translation tasks, as well as on language modelling, generally obtaining better results than by simply increasing model size.\n\nI lean towards the acceptance of the paper. The approach is fairly well motivated, likely easy to implement and results are mostly convincing. However, some claims may be too strong and I had difficulty understanding some parts of the approach.\n\nI find the idea interesting. Standard attention is unbiased to distance (but sensitive to it because of positional embeddings). Phrasal attention may be a useful learning bias, giving particular importance to nearby words.\n\nOn 3 WMT'14 translation tasks, the proposed approach leads to improvements between 0.7 and 1.8 BLEU with respect to Transformer Base. Running each model with different random seeds and presenting statistical significance results would be ideal, but such runs can be expensive given the size of the datasets. Using phrasal attention appears to be more efficient than simply increasing model size. In addition to the number of parameters, the number of FLOPs per update might also be useful to know. Phrasal attention also leads to lower perplexity on a large-scale modeling task, although I can't confidently evaluate the importance of this result.\n\nWhile results in the model interpretation section are cherry-picked, they illustrate that the model can use the additional capacity provided by phrasal attention. There is also clear qualitative differences between layers.\n\nSome equations are confusing. For example, in Eq. 5, the right-most argument of Conv_n() appears to be of dimension nxdx1, but convolutions are defined for dimension nxdxd. I would suggest going over the presentation of phrasal attention carefully (or correct me if I interpreted the notation wrongly).\n\nSome claims made in the paper may be too strong. While there are similarities between alignment and attention, they are not necessarily interchangeable in neural models. For example, (Koehn and Knowles. Six Challenges for Neural Machine Translation) show that they can be mismatched (Fig. 9).\n\nMoreover, while input embeddings (and arguably the last decoder hidden layer) mostly contain token-level information, intermediate representations merge information from multiple positions. As such, at a given layer, it is not guaranteed the the i^{th} vector is a representation i^{th} token. For example, (Voita et al. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives) show that the mutual information between an input token and its corresponding encoder representation diminishes as depth increases. As such, neighbouring representations may not represent n-grams.\n\nIt would be appropriate to compare the proposed approach to (Hao et al. Multi-Granularity Self-Attention for Neural Machine Translation). However, this is very recent work (September 5 on ArXiv), so it would be understandable for the authors not to know about it.\n\nQuestions:\n\nWhile searching for related work, I found an earlier submitted version of this paper (\"Phrase-Based Attentions\", submitted to ICLR 2019). The reported numbers differ from the current version. Why?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxB9WY0KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an attention mechanism that directly reflects the phrasal correspondence by employing convolution. The n-gram aware attention is incorporated into Transformer and shows gains in translation and language modeling tasks.\n\nIt is a novel and straightforward way to incorporate phrasal relation in the attention mechanism. The gains reported in this paper are meaningful, thought might not be SOTA. The analysis on attention is also interesting in that the phrasal relation is employed in lower layers, but not in the higher layer.\n\nOther comment:\n\n- It is not clear to me how the n-gram aware phrasal attention is incorporated into multi-headed attention described in section 3.2. Did you completely remove the multi-head attention, but used only n-gram attentions? Or, did you keep multi-head attention and incorporated phrasal attention for each head?\n\n- It is unfortunate that this work does not report empirical results when applied to a big model configuration. Did you encounter OOM when running on a big model?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a phrase-based attention method to model word n-grams (as opposed to single words) as the basic attention units. Multi-headed phrasal attentions are designed within the Transformer architecture to perform token-to-token and token-to-phrase mappings. Some improvements are shown in English-German, English-Russian and English-French translation tasks on the standard WMT'14 test set, and on the one-billion-word language modeling benchmark.

While the proposed approach is interesting and takes inspiration in the notion of phrases used in phrase-based machine translation, with some positive empirical results, the technical novelty of this paper is rather limited, and the experiments could be more solid. While it is understandable that lack of computational resources made it hard to experiment with larger models (e.g. Transformer-big), perhaps it would be interesting to try on language pairs with fewer resources (smaller datasets), where base models are more competitive.
Decision: Reject