SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhihao Fan $^{1*}$ , Yeyun Gong $^{2}$ , Dayiheng Liu $^{3}$ , Zhongyu Wei $^{1,6\dagger}$ , Siyuan Wang $^{1}$ , Jian Jiao $^{4}$ , Nan Duan $^{2}$ , Ruofei Zhang $^{4}$ , Xuanjing Huang $^{5}$
$^{1}$ School of Data Science, Fudan University, China
$^{2}$ Microsoft Research Asia,  $^{3}$ DAMO Academy,  $^{4}$ Microsoft
$^{5}$ School of Computer Science, Fudan University, China
$^{6}$ Research Institute of Intelligent and Complex Systems, Fudan University, China {fanzh18,zywei,wangsy18,xjhuang} @fudan.edu.cn, liudayiheng. ldyh@alibaba-inc.com {yegong,Jian.Jiao,nanduan,bzhang} @microsoft.com
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and FeedForward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.
Recently, Transformer (Vaswani et al., 2017) has been widely applied in various natural language processing tasks, such as neural machine translation (Vaswani et al., 2017) and text summarization (Zhang et al., 2019). To further improve the performance of the text representation, Transformer-based variants have attracted a lot of attention (Lu et al., 2019; Sukhbaatar et al., 2019a,b; Bugliarello and Okazaki, 2019; Ma et al., 2020).
Each building block of Transformer has two sublayers: Self-Attention Network (SAN) and Feed-Forward Network (FFN). Shaw et al. (2018)
(a) SAN
(b)DMAN
(c) FFN
Figure 1: The mask matrices of (a) SAN, (b) DMAN and (c) FFN in Mask Attention Networks. Color that fades from black to white means the values in mask matrices decrease from 1 to 0.
presents an extension to SAN which incorporates the relative positional information for the sequence. Sukhbaatar et al. (2019a) proposes attention span to control the maximum context size used in SAN and scales Transformer to long-range ( $\sim$  8192 tokens) language modeling. Recently, some works targeting on FFN have been proposed. Lu et al. (2019) gives a new understanding of Transformer from a multi-particle dynamic system point of view and designs a macaron architecture following Strang-Marchuk splitting scheme. Sukhbaatar et al. (2019b) regards the FFN as the persistent memory in SAN to augment SAN. These works focus on enhancing SAN or FFN, but neglect the inner relationship between SAN and FFN that hinders further improvement.
In this work, we present a more systematic analysis for both SAN and FFN to reveal their connections. We introduce Mask Attention Networks(MANs), in which each network has a mask matrix that element-wise multiplies a key-query attention matrix. We show that SAN and FFN are two special cases in MANs with static mask matrices. The mask matrix of SAN is an all-ones matrix, while that of FFN is an identity matrix, which is shown as (a) and (c) in Figure 1. Since the mask matrix of SAN has no restriction on relationship modeling with other tokens, SAN is expert in long-range dependency modeling and capture the global
*Work is done during internship at Microsoft Research Asia.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1692-1701 June 6-11, 2021. ©2021 Association for Computational Linguistics
semantics. In contrast, mask of FFN disables it to perceive the information of other tokens and forces it into self-evolution. We believe that these two specialties endowed by two mask matrices make the success of Transformer in text representation.
Although positive results of Transformer have been reported, recent works (Shaw et al., 2018; Yang et al., 2018; Guo et al., 2019) have shown that modeling localness would further improve the performance through experiments. We argue that deficiency of Transformer in local structure modeling is caused by the attention computation with static mask matrix. In the framework of MANs, we find a problem that irrelevant tokens with overlapping neighbors incorrectly attend to each other with relatively large attention scores. For example "a black dog jump to catch the frisbee", though "catch" and "black" are neither relevant nor neighbors, for the reason that both of them are highly related to their common neighbor "dog" in attention, we demonstrate that the attention score from "catch" to "black" would be large, which also decreases the attention score from "catch" to "frisbee". The issue in self-attention not only introduces noise to the semantic modeling, but also mislead query tokens to overlook these neighbor tokens. This reveals that self-attention is insufficient in localness modeling and inspires us to mask tokens that not appear in neighborhood.
To strengthen Transformer in localness modeling with better keeping the advantage of SAN and FFN, we propose a Dynamic Mask Attention Network (DMAN) as shown in Figure 1(b), which originates from MANs. Observations reveal that tokens have different ranges of neighbors, for example, that of "dog", which is also connected with "frisbee", is larger than "black" and "catch". Instead of being static that determined in advance, the mask matrix of DMAN is dependent on the query context and relative distance. In DMAN, the tokens in a specific neighborhood are able to receive more attention beyond the normal self-attention mechanism. The dynamic endows DMAN with text representation in different scales, and we validate the superiority through experiments. In Transformer (Vaswani et al., 2017), SAN and FFN cooperate in a sequential layered structure SAN  $\rightarrow$  FFN. Considering SAN, FFN, and DMAN all belong to MANs and have different advantages in text representation, instead of directly replacing SAN in previous works (Shaw et al., 2018; Yang et al., 2018; Guo
et al., 2019), we propose to incorporate them with the architecture DMAN  $\rightarrow$  SAN  $\rightarrow$  FFN.
The main contributions of this work are three-fold:
- We introduce Mask Attention Networks and reformulate SAN and FFN to point out that they are two special cases with static mask in MANs. We analyze the advantages of SAN and FFN in text representation learning and demonstrate that they are insufficient for localness modeling.
- Inspired by the different specialities of SAN and FFN, we propose Dynamic Mask Attention Network (DMAN) to model localness more effectively. We investigate the different collaboration methods of SAN, FFN, and DMAN, and propose a sequential layered structure DMAN  $\rightarrow$  SAN  $\rightarrow$  FFN.
- We conduct experiments on machine translation and abstract summarization. Experimental results show that our method outperforms original Transformer. We also perform ablation study to verify the effectiveness of different modules of our proposed model.
In § 2.1, we review the Transformer architecture. We introduce Mask Attention Networks and reformulate SAN and FFN to point out they are two special cases in § 2.2, and analyze their deficiency in localness modeling in § 2.3. Then, in § 2.4, we describe Dynamic Mask Attention Network (DMAN) in detail. At last, in § 2.5, we discuss the collaboration of DMAN, SAN and FFN.
Transformer has two sublayers: Self-Attention Network (SAN) and Feed-Forward Network (FFN).
As discussed in Vaswani et al. (2017), an attention function maps a query and a set of key-value pairs to an output shown in Equation 1.
$$
\mathcal {A} (Q, K, V) = \mathcal {S} (Q, K) V
$$
$$
\mathcal {S} (Q, K) = \left[ \frac {\exp \left(Q _ {i} K _ {j} ^ {T} / \sqrt {d _ {k}}\right)}{\sum_ {k} \exp \left(Q _ {i} K _ {k} ^ {T} / \sqrt {d _ {k}}\right)} \right] \tag {1}
$$
where the queries  $Q$ , keys  $K$  and values  $V \in \mathbb{R}^{T \times d_k}$  are all matrices.
SAN produces representations by applying attention function to each pair of tokens from the input sequence. It is beneficial to capture different contextual features with multiple individual attention functions. Given a text representation sequence  $H^{l}\in \mathbb{R}^{T\times d}$  in the  $l$  -the layer.
$$
H ^ {l} = \left[ A ^ {1}, \dots , A ^ {I} \right] W _ {H} \tag {2}
$$
$$
A ^ {i} = \mathcal {A} \left(H ^ {l} W _ {Q} ^ {i}, H ^ {l} W _ {K} ^ {i}, H ^ {l} W _ {V} ^ {i}\right)
$$
where  $\{W_{Q}^{i}, W_{K}^{i}, W_{V}^{i}\} \in R^{d \times d_{k}}$  are trainable parameters,  $i$  denotes the attention head and  $d$  is the hidden size.
In FFN, the computation of each  $h_t^l$  in  $H^l$  is independent of others. It consists of two affine transformations with a pointwise non-linear function:
$$
H ^ {l + 1} = \operatorname {R e L U} \left(H ^ {l} W _ {1}\right) W _ {2} \tag {3}
$$
where  $W_{1}$  and  $W_{2}$  are matrices of dimension  $d \times d_{f}$  and  $d_{f} \times d$ , respectively. Typically,  $d_{f}$  is set to be 4 times larger than  $d$ .
2.2 Mask Attention Networks
On the basis of attention function in Equation 1, we define a new mask attention function:
$$
\begin{array}{l} \mathcal {A} _ {M} (Q, K, V) = S _ {M} (Q, K) V \\ \mathcal {S} _ {M} (Q, K) = \left[ \frac {M _ {i , j} \exp \left(Q _ {i} K _ {j} ^ {T} / \sqrt {d _ {k}}\right)}{\sum_ {k} M _ {i , k} \exp \left(Q _ {i} K _ {k} ^ {T} / \sqrt {d _ {k}}\right)} \right] \tag {4} \\ \end{array}
$$
where  $M \in \mathbb{R}^{T \times T}$ ,  $M_{i,j} \in [0,1]$  is a mask matrix and can be static or dynamic. Intuitively, the value in each position of  $M$  can be viewed as the color shade in Figure 1.
With the knowledge of mask attention function, we introduce Mask Attention Networks(MANs), in which each network can be written as Equation 5.
$$
\begin{array}{l} H ^ {l + 1} = \mathcal {F} \left(\left[ A _ {M ^ {1}} ^ {1}, \dots , A _ {M ^ {I}} ^ {I} \right]\right) W _ {H} \tag {5} \\ A _ {M ^ {i}} ^ {i} = \mathcal {A} _ {M ^ {i}} \left(H ^ {l} W _ {Q} ^ {i}, H ^ {l} W _ {K} ^ {i}, H ^ {l} W _ {V} ^ {i}\right) \\ \end{array}
$$
where  $\mathcal{F}$  is the activation function,  $M^i$  is the mask matrix for the  $i$ -th attention head.
Next, we show that SAN and FFN both belong to the Mask Attention Networks.
For SAN, let  $M = [1] \in \mathbb{R}^{T \times T}$  be an all-ones matrix and  $\mathcal{F} = \mathcal{F}_{id}$  be the identity function, its mask attention function would be formalized:
$$
\begin{array}{l} \mathcal {S} _ {[ 1 ]} (Q, K) = \left[ \frac {1 \cdot \exp \left(Q _ {i} K _ {j} ^ {T} / \sqrt {d _ {k}}\right)}{\sum_ {k} \exp \left(Q _ {i} K _ {k} ^ {T} / \sqrt {d _ {k}}\right)} \right] = \mathcal {S} (Q, K) \\ \mathcal {A} _ {[ 1 ]} (Q, K, V) = \mathcal {S} _ {[ 1 ]} (Q, K) V = \mathcal {A} (Q, K, V) \tag {6} \\ \end{array}
$$
Figure 2: Overview of our proposed model. Left is the Transformer architecture, right is our DMAN  $\rightarrow$  SAN  $\rightarrow$  FFN one.
Then, the MAN degenerates into SAN.
$$
\begin{array}{l} H ^ {l + 1} = \mathcal {F} _ {i d} \left(\left[ A _ {[ 1 ]} ^ {1}, \dots , A _ {[ 1 ]} ^ {h} \right]\right) W _ {H} \tag {7} \\ = \left[ A ^ {1}, \dots , A ^ {h} \right] W _ {H} \\ \end{array}
$$
For FFN, let  $M = \mathbb{I} \in \mathbb{R}^{T \times T}$  be the identity matrix,  $\mathcal{F} = \mathrm{ReLU}$  and head number  $I = 1$ .
$$
\begin{array}{l} \mathcal {S} _ {\mathbb {I}} (Q, K) = \left[ \frac {1 _ {i} (j) \cdot \exp \left(Q _ {i} K _ {j} ^ {T} / \sqrt {d _ {k}}\right)}{\sum_ {k} 1 _ {i} (k) \cdot \exp \left(Q _ {i} K _ {k} ^ {T} / \sqrt {d _ {k}}\right)} \right] = \mathbb {I} \\ \mathcal {A} _ {\mathbb {I}} (Q, K, V) = \mathcal {S} _ {\mathbb {I}} (Q, K) V = \mathbb {I} V = V \tag {8} \\ \end{array}
$$
where  $1_{i}(x)$  is an indicator function that equal to 1 if  $x = i$ , otherwise 0.
The MAN degenerates into FFN.
$$
H ^ {l + 1} = \operatorname {R e L U} \left(\left[ A _ {M} ^ {1} \right]\right) W _ {H} = \operatorname {R e L U} \left(H ^ {l} W _ {V} ^ {1}\right) W _ {H} \tag {9}
$$
In summary, SAN and FFN are two special cases in MANs with different static mask matrices.
2.3 Deficiency of SAN and FFN in Localness Modeling
The mask matrix of SAN is an all-ones matrix and that of FFN is an identity matrix, they are two extreme cases in MANs. We analyze that these two static MANs are deficient in localness modeling. Intuitively, through blocking other tokens in advance, FFN focuses on its own information and is unable to perceive the information except itself, let alone its neighbors. In SAN, each token is equally accessible to any other ones. As the example in Introduction shows, we find that tokens not in neighborhood are also likely to attend to each
other with relatively large scores. Therefore, SAN might introduce noises to semantic modeling and overlook the relation of neighboring signals.
We demonstrate the issue of self-attention. Generally assuming that  $[a, b, c]$  appear in sequence, and  $(a, b)$ ,  $(b, c)$  are two neighbor pairs, but  $a, c$  are not neighbors.
First, to explicitly define the relationship of tokens, we introduce  $U_{\delta}(h)$  as the set of tokens at the distance of  $\delta$  from  $h$  with key and query linear transformation in SAN, in other words,  $u \in U_{\delta}(h) \Leftrightarrow ||hW_{Q} - uW_{K}||_{2}^{2} \leq \delta$ . For example, if  $(a,b)$  is a neighbor pair, there would exist some small  $\delta \geq 0$  such that  $a \in U_{\delta}(b)$  and  $b \in U_{\delta}(a)$ .
Second, we know that the larger the inner product is, the smaller the Euclidean distance is, and vice versa. With the awareness of the relationships between  $[a,b,c]$ , we have  $a,b\in U_{\delta}(a)$ ,  $b,c\in U_{\delta}(c)$  and  $a,b,c\in U_{\delta}(b)$  for some small  $\delta \geq 0$ .
Third, we are able to estimate the semantic distance between  $a$  and  $c$  as the Equation 10 shows.
$$
\begin{array}{l} \left| \left| a W _ {Q} - c W _ {K} \right| \right| _ {2} ^ {2} \\ = \left\| a W _ {Q} - b W _ {K} + b W _ {K} - b W _ {Q} + b W _ {Q} - c W _ {K} \right\| _ {2} ^ {2} \\ \leq 3 \left| \left| a W _ {Q} - b W _ {K} \right| \right| _ {2} ^ {2} + 3 \left| \left| b W _ {K} - b W _ {Q} \right| \right| _ {2} ^ {2} \\ + 3 \left\| b W _ {Q} - c W _ {K} \right\| _ {2} ^ {2}) \leq 9 \delta \tag {10} \\ \end{array}
$$
Thus, though  $a$  and  $c$  are not neighbors, no matter how irrelevant the semantics of  $a$  and  $c$ ,  $c \in U_{9\delta}(a)$  that  $c$  would play an important role in modeling semantics of  $a$ .
The upper phenomenon illustrates following normal attention function in Equation 1, some tokens not in neighborhood not are still likely to occupy an important position in attention weight that can not be ignored.
2.4 Dynamic Mask Attention Network
With the knowledge of MANs, we propose to mask other tokens that not in neighborhood of the target token for better local semantic modeling.
For example, we build a distance-dependent mask matrix SM. If each token only model the relationship with those tokens within  $b$  units of itself, we can set
$$
\operatorname {S M} [ t, s ] = \left\{ \begin{array}{l l} 0, & | t - s | > b \\ 1, & | t - s | \leq b \end{array} \right. \tag {11}
$$
where  $t, s$  are the positions of query and key, and
$\mathbf{SM}[t,s]$  is the value of the  $t$  -th row and  $s$  -th column of SM.
By means of SM, we take those tokens within  $b$  units into account and ignore others. The static mask does assign more weights to a specific neighborhood, but lacks flexibility. Considering the neighborhood size varies with different query tokens, number of tokens that benefit for different query tokens' local semantic representation are different. Moreover, their mask matrices should match different attention heads and layers in MANs.
We propose Dynamic Mask Attention Network (DMAN) that replaces the static mask matrix. Incorporating query tokens, relative distance, attention head and layer, we build a dynamic mask function which replaces the hard  $0/1$  mask gate in Equation 11 with a soft one through sigmoid activation function in Equation 12.
$$
\mathrm {D M} _ {i} ^ {l} [ t, s ] = \sigma \left(h _ {t} ^ {l} W ^ {l} + P _ {t - s} ^ {l} + U _ {i} ^ {l}\right) \tag {12}
$$
where  $s, t$  are the positions of query and key,  $i$  is the attention head,  $l$  is the layer.  $P_{t - s}^{l}$  is parameterized scalar for the positions  $t$  and  $s$ ,  $U_{i}^{l}$  is for the  $i$ -th head, and  $W^{l} \in \mathbb{R}^{d \times 1}$ .  $W^{l}$ ,  $P_{t - s}^{l}$  and  $U_{i}^{l}$  are trainable parameters.
2.5 Collaboration of Mask Attention Networks
Until here, we have three sub-networks of MANs, namely, SAN, FFN and DMAN. SAN that does not mask any tokens and specializes in global semantic modeling. FFN that masks all tokens except itself and focuses on self-processing. DMAN masks the tokens not in neighborhood and is able to model local structure more effectively.
Transformer is composed of SAN and FFN that achieves positive results in various NLP tasks, the stacking method of Transformer inspires us to stack DMAN, SAN and FFN to incorporate their advantages. We insert DMAN in the manner of DMAN  $\rightarrow$  SAN  $\rightarrow$  FFN, which is shown in Figure 2. With this architecture, we first model the localness then globalness, and take the step for self-evolution in the end.
In this section, we introduce our experiments. We first describe the experimental details in  $\S 3.1$ . Then we show the experimental results in  $\S 3.2$ .
Model	IWSLT14 De-En	WMT14 En-De
small	params	base	params	big	params
Transformer (Vaswani et al., 2017)	34.4	36M	27.3	62M	28.4	213M
Convolutional Transformer (Yang et al., 2019b)	-	-	28.2	88M	28.7	-
Weighted Transformer (Ahmed et al., 2017)	-	-	28.4	65M	28.9	213M
Local Transformer (Yang et al., 2018)	-	-	28.5	89M	29.2	268M
Relative Transformer (Shaw et al., 2018)	-	-	26.8	-	29.2	-
Scaling NMT (Ott et al., 2018)	-	-	-	-	29.3	213M
Dynamic Conv (Wu et al., 2019)	35.2	-	-	-	29.7	213M
Ours	36.3	37M	29.1	63M	30.4	215M
Table 1: Translation performance (BLEU) on IWSLT14 De-En and WMT14 En-De testsets.
Finally we conduct the ablation study and analysis in  $\S 4$ .
3.1.1 Machine Translation
Machine translation is an important application of natural language processing (Vaswani et al., 2017). We evaluate our methods on two widely used public datasets: IWSLT14 German-to-English (De-En) and WMT14 English-to-German (En-De). IWSLT14 De-En dataset consists of about  $153\mathrm{K} / 7\mathrm{K} / 7\mathrm{K}$  sentence pairs for training/validation/testing. WMT14 En-De dataset consists of about  $4.5\mathrm{M}$  sentence pairs, and the models were validated on newstest2013 and examined on newstest2014.
Our data processing follows Lu et al. (2019). For IWSLT2014, we set our model into the small one, the hidden size, embeddings and attention heads to 512, 512, and 4 respectively. For the WMT14 dataset, following the Transformer setting of Vaswani et al. (2017), we set our model into the base and big ones which both consist of a 6-layer encoder and 6-layer decoder, the hidden nodes are set to 512 and 1024, and the number of attention heads are 8 and 16. For each setting (small, base and big), we replace all layers in Transformer by our MAN layer. To make a relatively fair comparison, we set the dimensionality of the inner-layer of the FFN in the MAN layers to two times of the dimensionality of the hidden states.
We train our proposed model with cross-entropy with 0.1 label smoothing rate. Inverse-sqrt learning rate scheduler are employed, the peak learning rates are  $1.5\mathrm{e} - 2$ , 1e-2 and 7e-3 with 8k warmup, 50k update, 80k update and 80k update for transformer
big, base and small model with max-tokens 4096, 12288 and 8192 per batch. The dropout rates are 0.3, 0.1 and 0.3 for small, base and big models. The optimizer of model is Adam with (0.9,0.98). The beam size and length penalty for base and big models are 4 and 0.6, for small model is 5 and 1.0. The base and large model are trained on 8 V100 GPUs, and the small model is trained on 2 P40.
3.1.2 Abstract Summarization
Automatic summarization aims to produce a concise and fluent summary conveying the key information in the input text. We focus on abstractive summarization, a generation task where the summary is not limited in reusing the phrases or sentences in the input text. We use the CNN/Daily Mail (See et al., 2017) and Gigaword (Rush et al., 2015) for model evaluation.
Following Song et al. (2019), we set the hidden size, embeddings and attention heads to 768, 768, and 12 respectively. Our model consists of a 6-layer encoder and 6-layer decoder. For the convenience of comparison, the training follows classic seq2seq model without copy, converge or RL. We remove duplicated trigrams in beam search (Paulus et al., 2018). Moreover, the dimensionality of the inner-layer of the FFN in the MAN layers is set to two times of the dimensionality of the hidden states.
In training, inverse-sqrt learning rate scheduler is employed. The peak learning rates are 1e-3 and 8e-4, max-tokens per batch are 8192 and 12288 for CNN/Daily Mail and Gigaword, respectively. The warmup steps is 8k and the total updates is 50k. The optimizer of model is Adam with (0.9,0.98). The dropout and clip-norm are both 0.1. During decoding, the beam size are both 5, the max length and length penalty are 50 and 2.0 for CNN/Daily
Mail, 30 and 1.0 for Gigaword. The models are trained on 4 P40 GPUs.
3.2.1 Machine Translation
In machine translation, BLEU (Papineni et al., 2002) is employed as the evaluation measure. Following common practice, we use tokenized case-sensitive BLEU and case-insensitive BLEU for WMT14 En-De and IWSLT14 De-En, respectively. We take Transformer (Vaswani et al., 2017) as the baseline and compare with other concurrent methods. Convolutional Transformer (Yang et al., 2019b) restricts the attention scope to a window of neighboring elements in order to model locality for self-attention model. Local Transformer (Yang et al., 2018) casts localness modeling as a learnable Gaussian bias, which indicates the central and scope of the local region to be paid more attention.
The results for machine translation are shown in Table 1. Our model exceeds the baseline Transformer and other models. For the IWSLT14 dataset, our small model outperforms the Transformer small by 1.6 points in terms of BLEU. For the WMT14 dataset, our base model exceeds its Transformer counterpart by 1.8 BLEU points. Furthermore, the performance of our base model is even better than that of the Transformer big model reported in (Vaswani et al., 2017), but with much less parameters. Our big model outperforms the Transformer big by 2.0 BLEU points.
Compare with Convolutional Transformer and Local Transformer, our model also achieve 1.7 and 1.2 points improvement in BLEU, respectively. This validates that the superiority of our model to systematically solve the localness modeling problem in Transformer.
3.2.2 Abstractive Summarization
We use the F1 score of ROUGE (Lin and Hovy, 2003) as the evaluation metric<sup>1</sup>. In Table 2, we compare our model against the baseline Transformer (Vaswani et al., 2017) and several generation models on CNN/Daily Mail and Gigaword. LEAD3 (Nallapati et al., 2016) extracts the first three sentences in a document as its summary. PTGEN+Convergence (See et al., 2017) is a sequence-to-sequence model based on the pointer-generator network. As shown in Table 2, our model outperforms Transformer by 1.4 in ROUGE-1, 2.2 in
ROUGE-2 and 1.2 in ROUGE-L in CNN/Daily Mail. In Gigaword dataset, ours exceeds the baseline by 0.7 in ROUGE-1, 0.5 in ROUGE-2 and 0.7 in ROUGE-L.
As a summary, in machine translation and abstractive summarization our proposed model achieves better results than the Original Transformer (Vaswani et al., 2017).