This indicates that the compression knowledge as an additional context can enhance NMT better. 4) BBFNMT (based) is comparable to the +global-deep context, the best comparison system, while BBFNMT (big) slightly outperformed +global-deep context by $0.16$ BLEU scores. In particular, the parameters of BBFNMT (base/big) model, which just increased $12.1/7.9$M over the Transformer (base/big), were only 70% of the +global-deep context model. This denotes that the BBFNMT model is more efficient than the +global-deep context model. In addition, the training speed of the proposed models slightly decreased ($8\%$), compared to the corresponding baselines. 5) The proposed BBFNMT (base) slightly outperformed the Transformer (big) which contains much more parameters than BBFNMT (base). This indicates that our improvement is not likely to be due to the increased number of parameters. For the EN-FR translation task, the proposed models gave similar improvements over the baseline systems and comparing methods (except that the Transformer (big) performed much more better than Transformer (base)).

Experiments ::: Main Results ::: Machine Translation	According to the results in Table TABREF34, we chose the semi-supervised ESC model (which performed the best) to generate compressed sentences for the machine translation task. The main results on the WMT14 EN-DE and EN-FR translation tasks are shown in Table TABREF35. In the EN-DE task, we made the following observations: 1) The baseline Transformer (base) in this work achieved a performance comparable to the original Transformer (base) . This indicates that it is a strong baseline NMT system. 2) All BSFNMT, BTFNMT, and BBFNMT significantly outperformed the baseline Transformer (base/big) and only introduces a very small amount of extra parameters. This indicates that the learned compressed backbone information was beneficial for the Transformer translation system. 3) Among the proposed three methods, BTFNMT performed better than BSFNMT. This indicates that the backbone fusion at the target-side is better than at the source-side. In addition, BBFNMT (base/big) outperformed the comparison systems +Localness and +Context-Aware SANs.

Experiments ::: Ablation Study ::: Effect of Encoder Parameters	In our model, representations of the original sentence and its compressed version were learned by a shared encoder. To explore the effect of the encoder parameters, we also designed a BBFNMT with two independent encoders to learn representations of the original sentence and its compressed version, respectively. Table TABREF41 shows results on the newstest2014 test set for the WMT14 EN-DE translation task. The BBFNMT (w/ independent params) slightly outperformed the proposed shared encoder model by a BLEU score of 0.15, but its parameters increased by approximately 30%. In contrast, the parameters in our model are comparable to the baseline Transformer (base). Considering the parameter scale, we took a shared encoder to learn source representation, which makes it easy to verify the effectiveness of the additional translation knowledge, such as our backbone knowledge.

Experiments ::: Ablation Study ::: Evaluating Sentence Compression	To demonstrate the effectiveness of sentence compression, we compared the compressed sentences ($\gamma = 0.6$) generated in the Transformer translation system (BBFNMT) under different settings: AllText, F8W, RandSample (random sampling), supervised ESC, Unsupervised ESC and semi-supervised ESC. Table TABREF39 shows the results on newstest2014 for the EN-DE translation task. We made the following observations: 1) Simply introducing AllText and F8W achieved few improvement, and RandSample is lower than the baseline. In comparison, all the +supervised ESC, +unsupervised ESC, and +semi-supervised ESC models substantially improved the performance over the baseline Transformer (base). This means that our ESC method provides a richer source information for machine translation tasks. 2) +Unsupervised ESC can gain better improvements over the +supervised ESC although supervised ESC model can achieve higher quality than the unsupervised ESC model in the benchmark test dataset. This may be due to that the annotated sentence compression training data is in different domain with the WMT EN-DE traing data.

Experiments ::: Main Results ::: Sentence Compression	To evaluate the quality of our sentence compression model, we conducted a horizontal comparison between the proposed sentence compression model and other sentence compression models in different settings. Table TABREF34 shows the comparison results. We observed that the proposed unsupervised ESC model performed substantially better than Fevry and 's unsupervised method. The proposed supervised ESC model also substantially outperformed the RNN-based Seq2seq and 's baseline method. That is, our supervised model gave +2.0 improvements on R-1, R-2, and R-L scores over the RNN-based Seq2seq. This means that the proposed Transformer-based approaches can generate compressed sentences of high quality. We further compared our semi-supervised model with the semi-supervised pretraining methods of MLM , DAE , and MASS . Our unsupervised pretrainining method outperformed the other unsupervised pretrainining ones on the sentence compression task consistently.

For the semi-supervised setting, in order to make the results comparable to , we used the same 190M English monolingual unpaired data from WMT News Crawl datasets for pre-training (unsupervised training). We included the other pretraining methods: masked language modeling (MLM, BERT) , denoising auto-encoder (DAE) , and masked sequence to sequence (MASS)  to compare with our unsupervised pretraining method in the semi-supervised setting.

showed that the spaces of original and simplified translations can be effectively combined using translation lattices and compare two decoding approaches to process both inputs at different levels of integration. Different from these work, our proposed sentence compression model does not rely on any known linguistics motivated (such as syntax) skeleton simplification, but directly trains a computation motivated sentence compression model to learn to compress sentences and re-paraphrase them directly in seq2seq model. Though with a pure computation source, our sentence compression model can surprisingly generate more grammatically correct and refined sentences, and the words in the compressed sentence do not have to be the same as the original sentence. In the meantime, our sentence compression model can stably give source backbone representation exempt from unstable performance of a syntactic parser which is essential for syntactic skeleton simplification. Our sentence compression model can perform unsupervised training on large-scale data sets, and then use the supervised data for finetune, which is more promising from the results.

Explicit Sentence Compression for Neural Machine Translation	State-of-the-art Transformer-based neural machine translation (NMT) systems still follow a standard encoder-decoder framework, in which source sentence representation can be well done by an encoder with self-attention mechanism. Though Transformer-based encoder may effectively capture general information in its resulting source sentence representation, the backbone information, which stands for the gist of a sentence, is not specifically focused on. In this paper, we propose an explicit sentence compression method to enhance the source sentence representation for NMT. In practice, an explicit sentence compression goal used to learn the backbone information in a sentence. We propose three ways, including backbone source-side fusion, target-side fusion, and both-side fusion, to integrate the compressed sentence into NMT. Our empirical tests on the WMT English-to-French and English-to-German translation tasks show that the proposed sentence compression method significantly improves the translation performances over strong baselines.

Explicit Sentence Compression ::: Compression Rate Control	Explicit compression rate (length) control is a common method which has been used in previous sentence compression works.  examined several methods of introducing target output length information, and found that they were effective without negatively impacting summarization quality.  introduced a length marker token that induces the model to target an output of a desired length, coarsely divided into discrete bins.  augmented the decoder with an additional length countdown input which is a single scalar that ticks down to 0 when the generation reached the desired length. Different with the length marker or length countdown input, to induce our model to output the compression sequence with desired length, we use beam search during generation to find the sequence $S^{^{\prime }}$ that maximizes a score function $s(S^{^{\prime }}, S)$ given a trained ESC model. The length normalization is introduced to account for the fact that we have to compare hypotheses of different length. Without some form of length-normalization regular $ln$, beam search will favor shorter sequences over longer ones on average since a negative log-probability is added at each step, yielding lower (more negative) scores for longer sentences.

Experiments ::: Setup ::: Sentence Compression	To evaluate the quality of our sentence compression model, we used the Annotated Gigaword corpus  as the benchmark . The data includes approximately 3.8 M training samples, 400 K validation samples, and 2 K test samples. The byte pair encoding (BPE) algorithm  was adopted for subword segmentation, and the vocabulary size was set at 40 K for our supervised, unsupervised and semi-supervised settings . Baseline systems include AllText and F8W , . F8W is simply the first 8 words of the input, and AllText uses the whole text as the compression output. The $F_1$ score of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) was used to evaluate this task . We use beam search with a beam size of 5, the length length normalization of 0.5, and the coverage penalty of 0.2.

Conclusion and Future work	To give a more focused source representation, this paper makes the first attempt to propose an explicit sentence compression method to enhance state-of-the-art Transformer-based NMT. To demonstrate that the proposed sentence compression enhancement is indeed helpful for the neural machine translation, We evaluate the impact of the proposed model on the large-scale WMT14 English-to-German and English-to-French translation tasks. The experimental results on WMT14 EN-DE and EN-FR translation tasks show that our proposed NMT model can yield significantly improved results over strong baseline translation systems. In the future work, we will release a pre-trained language model that uses unsupervised sentence compression as the pre-training objective to demonstrate the performance of unsupervised sentence compression in representation learning.

Explicit Sentence Compression ::: Supervised ESC	Sentence compression usually relies on large-scale raw data together with their human-labeled data, which can be viewed as supervision, to train a sentence compression model , , , , , . For example,  proposed an attentive encoder-decoder recurrent neural network (RNN) to model abstractive text summarization.  furture proposed MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder sentence compression framework which reported state-of-the-art performance on both the Gigaword Corpus and DUC Corpus. Sentence compression can be conducted by a typical sequence-to-sequence model. The encoder represents the input sentence $S$ as a sequence of annotation vectors, and the decoder depends on the attention mechanism to learn the context vector for generating a compressed version $S^{^{\prime }}$ with the key meaning of the input sentence. Recently, the new Transformer architecture proposed by , which fully relies on self-attention networks, has exhibited state-of-the-art translation performance for several language pairs. We follow this practice and attempt to apply the Transformer architecture to such a compression task.

To gain a better quality for the compressed sentences, we transfer the method of  into the Transformer architecture instead of their suggested RNN architecture, which makes it conducive to deeper network training and a larger corpus.

We argue that such backbone information is also helpful for learning source representation, and is not explicitly considered by the existing NMT system to enrich the source sentence representation. In this paper, we propose a novel explicit sentence compression approach to enhance the source representation for NMT. To this end, we first design three sentence compression models to accommodate the needs of various languages and scenarios, including supervised, unsupervised, and semi-supervised ways, to learn a backbone information words sequence (as shown in Table TABREF2) from the source sentence. We then propose three translation models, including backbone source-side fusion based NMT (BSFNMT), backbone target-side fusion (BTFNMT), and both-side fusion based NMT (BBFNMT), to introduce this backbone knowledge into the existing Transformer NMT system for improving translation predictions. Empirical results on the WMT14 English-to-German and English-to-French translation tasks show that the proposed approach significantly improves the translation performance over the strong even state-of-the-art NMT baselines.

Explicit Sentence Compression ::: Semi-supervised ESC	As pointed out in , sequence to sequence framework has attracted much attention recently due to the advances of deep learning by using large-scale data. Many language generation tasks have only a small scale of pair data which can't support to train a deep model with good generalization ability. In comparison, there is a lot of unpaired data which is earier to obtain. We observe a performance degradation caused by different domains in the supervised ESC. According to the experimental results of , the accuracy of the unsupervised ESC is currently lower than the supervised one. Therefore, we have further adopted the semi-supervised explicit sentence compression model to alleviate this problem. Specifically, the unsupervised training (often referred to as pre-training) is performed on the unpaired data first and fine-tuning with the small scale paired data (supervised training) to obtain the ESC model with good performance and generalization ability.

Introduction	Neural machine translation (NMT) is popularly implemented as an encoder-decoder framework , in which the encoder is right in charge of source sentence representation. Typically, the input sentence is implicitly represented as a contextualized source representation through deep learning networks. By further feeding the decoder, the source representation is used to learn dependent time-step context vectors for predicting target translation . In state-of-the-art Transformer-based encoder, self-attention mechanisms are good at capturing the general information in a sentence , , . However, it is difficult to distinguish which kind of information lying deeply under the language is really salient for learning source representation. Intuitively, when a person reads a source sentence, he/she often selectively focuses on the basic sentence meaning, and re-reads the entire sentence to understand its meaning completely. Take the English sentence in Table TABREF2 as an example. We manually annotate its basic meaning as a shorter sequence of words than in the original sentence, called backbone information. Obviously, these words with the basic meaning contain more important information for human understanding than the remaining words in the sentence.

Related Work	To let the translation have more focus over the source sentence information, efforts have been initiated on exploiting sentence segmentation, sentence simplification, and sentence compression for machine translation.  presented a approach to integrating the sentence skeleton information into a phrase-based statistic machine translation system.  proposed an approach to modeling syntactically-motivated skeletal structure of source sentence for statistic machine translation.  describe an early approach to skeleton-based translation, which decomposes input sentences into syntactically meaningful chunks. The central part of the sentence is identified and remains unaltered while other parts of the sentence are simplified. This process produces a set of partial, potentially overlapping translations which are recombined to form the final translation.  describe a “divide and translate” approach to dealing with complex input sentences. They parse the input sentences, replace subclauses with placeholders and later substitute them with separately translated clauses. Their method requires training translation models on clause-level aligned parallel data with placeholders in order for the translation model to deal with the placeholders correctly.  experimented with automatically segmenting the source sentence to overcome problems with overly long sentences.

Explicit Sentence Compression	Generally, sentence compression is a typical sequence generation task which aims to maximize the absorption and long-term retention of large amounts of data over a relatively short sequence for text understanding , . To distinguish the importance of words in the sentence and, more importantly, to dig out the most salient part in the sentence representation, we utilize the sentence compression method to explicitly distill the key knowledge that can retain the key meaning of the sentence, termed explicit sentence compression (ESC) in this paper. Depending on whether or not the sentence compression is trained using human annotated data, the proposed method can be implemented in three ways: supervised ESC, unsupervised ESC, and semi-supervised ESC.

NMT with ESC ::: Transformer Networks	A Transformer NMT model consists of an encoder and a decoder, which fully rely on self-attention networks (SANs), to translate a sentence in one language into another language with equivalent meaning. Formally, one input sentence $x$=$\lbrace x_1, \cdots , x_J\rbrace $ of length $J$ is first mapped into a sequence of word vectors. Then the sequence and its position embeddings add up to form the input representation $v_x=\lbrace v^x_1, \cdots , v^x_J\rbrace $. The sequence $\lbrace v^x_1, \cdots , v^x_J\rbrace $ is then packed into a query matrix $\textbf {Q}_x$, a key matrix $\textbf {K}_x$, and a value matrix $\textbf {V}_x$. For the SAN-based encoder, the self-attention sub-layer is first performed over $\textbf {Q}$, $\textbf {K}$, and $\textbf {V}$ to the matrix of outputs as: where $d_{model}$ represents the dimensions of the model.

Experiments ::: Setup ::: Machine Translation	The proposed NMT model was evaluated on the WMT14 English-to-German (EN-DE) and English-to-French (EN-FR) tasks, which are both standard large-scale corpora for NMT evaluation. For the EN-DE translation task, 4.43 M bilingual sentence pairs from the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7. The newstest2013 and newstest2014 datasets were used as the dev set and test set, respectively. For the EN-FR translation task, 36 M bilingual sentence pairs from the WMT14 dataset were were used as training data. Newstest12 and newstest13 were combined for validation and the newstest14 was the test set, following the setting of . The BPE algorithm  was also adopted, and the joint vocabulary size was set at 40 K. For the hyper-parameters of our Transformer (base/large) models, we followed the settings used in 's work.

It is lossy from sentence $S$ to sentence $S^{^{\prime }}$, which makes it difficult to restore from the compressed sentence $S^{^{\prime }}$ to the original sentence $S$.  added noises to extend the original sentences and trained a denoising auto-encoder to recover the original, constructing an end-to-end training network without any examples of compressed sentences in sequence to sequence framework. In doing so, the model has to exclude and reorder the noisy sentence input, and hence learns to output more semantic important, shorter but grammatically correct sentences. There are two types of noise used in the model: Additive Sampling Noise and Shuffle Noise. Additive Sampling Noise: To extend the original sentence, we sample additional sentence from the training dataset randomly, and then sub-sample a subset of words from each without replacement. The newly sampled words are appended to the original sentence. Shuffle Noise: In order for the model to learn to rephrase the input sentence to make the output shorter, we shuffle the resultant additive noisy sentence.

Experiments ::: Ablation Study ::: Evaluating Compression Ratio	In order to verify the impact of different compression ratios on translation quality, we conducted experiments on EN-DE translation task with semi-supervised sentence compression in BBFNMT model. We controled the compression ratio $\gamma $ from 0 to 1.0. Consider two boundary conditions, when the compression ratio $\gamma = 0$, it means no compression sequence generated, which is the same as the vanilla Transformer. When the compression ratio $\gamma = 1.0$, it is equivalent to re-paraphrasing the source sentence using the sentence compression model (maintaining the same length) as the additional input for BBFNMT. The experimental results are shown in Fig. FIGREF43. As can be seen from the results, in our experiments, sentence compression (re-paraphrasing) can bring performance improvement, even when the compression ratio $\gamma =1.0$ and the sentence length is not shortened, re-paraphrasing can still bring slight improvement of translation quality. On the wmt14 EN-DE translation task, the compression ratio $\gamma $ was set to 0.6 to get the best results.

Moreover, a coverage penalty $cp$ is also added to favor the sequence that cover the source sentence meaning as much as possible according to the attention weights . where $p_{i,j}$ is the attention probability of the $j$-th target word on the $i$-th source word. Parameters $\alpha $ and $\beta $ control the strength of the length normalization and the coverage penalty. Although $\alpha $ can be used to control the compression ratio softly, we use the compression ratio $\gamma $ to control the maximum length of decoding generation by hard requirements. When the decoding length $|S^{^{\prime }}|$ is greater than $\gamma |S|$, the decoding stops.

Acknowledgment	The corresponding authors are Rui wang and Hai Zhao. Zuchao Li and Zhuosheng Zhang were internship research fellows at NICT when conducting this work. Hai Zhao was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100) and Key Projects of National Natural Science Foundation of China (No. U1836222 and No. 61733011). Rui Wang was partially supported by JSPS grantin-aid for early-career scientists (19K20354): “Unsupervised Neural Machine Translation in Universal Scenarios” and NICT tenure-track researcher startup fund “Toward Intelligent Machine Translation”.

NMT with ESC ::: Backbone Target-side Fusion based NMT	In the backbone target-side fusion based NMT (BTFNMT) model, both the original sentence and its compressed version are also represented as $H_x$ and $H_c$ respectively by the shared SANs. We then use a tuple ($H_x,H_c$) instead of the source-side fusion representation $H_x^{^{\prime }}$ as the input to the decoder. Specifically, we introduce an additional “encoder-decoder attention" module into the decoder to learn the compressed sequence context $b_i$ at the current time-step $i$: Since we are here to treat the original sentence and the compressed sentence as two independent source contexts when encoding at the source side, we use a context gate $g_c$ for integrating two independent contexts of the source: original context $c_i$ and compressed context $b_i$. The gate $g_i$ is calculated by: Therefore, the final target fusion context $c_i^{\prime }$ is: where $\sigma $ is the logistic sigmoid function, $\otimes $ is the point-wise multiplication, and $[\cdot ]$ represent the concatenation operation.

NMT with ESC	In this section, we first introduce the Transformer networks for machine translation. Then based on the fusion position of the backbone knowledge sequence, we propose three novel translation models: the backbone source-side fusion based NMT model (as shown in Figure FIGREF11), the backbone target-side based NMT model (as shown in Figure FIGREF12), and the backbone both-side based NMT. All of these models can make use of the source backbone knowledge generated by our sentence compression models.

Similarly, the translated target words are used to generate the decoder hidden state $\textbf {s}_i$ at the current time-step $i$. Generally, the self-attention function is further refined as multi-head self-attention to jointly consider information from different representation subspaces at different positions: where the projections are parameter matrices $\textbf {W}_{h}^{Q}$$\in $$\mathbb {R}^{d_{model}\times d_k}$, $\textbf {W}_{h}^{K}$$\in $$\mathbb {R}^{d_{model}\times d_k}$, $\textbf {W}_{h}^{V}$$\in $$ \mathbb {R}^{d_{model}\times d_v}$, and $\textbf {W}^{O}$$\in $$\mathbb {R}^{hd_{v}\times d_{model}}$. For example, there are $H$=8 heads, $d_{model}$ is 512, and $d_k$=$d_v$=512/8=64.

A position-wise feed-forward network (FFN) layer is applied over the output of multi-head self-attention, and then is added with the matrix $\textbf {V}$ to generate the final source representation $H_{x}$=$\lbrace H^{x}_1, \cdots , H^{x}_J\rbrace $: The SAN of decoder then uses both $H_x$ and target context hidden state $H_{tgt}$ to learn the context vector $o_i$ by “encoder-decoder attention": Finally, the context vector $o_{i}$ is used to compute translation probabilities of the next target word $\textit {y}_i$ by a linear, potentially multi-layered function: where $\textbf {L}_{o}$ and $\textbf {L}_{w}$ are projection matrices.

NMT with ESC ::: Backbone Source-side Fusion based NMT	In the backbone source-side fusion based NMT (BSFNMT) model, given an input sentence $x$=$\lbrace x_1, \cdots , x_J\rbrace $, there is an additional compressed sequence $x_c$=$\lbrace x^c_1, \cdots , x^c_K\rbrace $ of length $K$ generated by the proposed sentence compression model. This compressed sequence is also input to the SAN shared with the original encoder with word vectors $v_c = \lbrace v^c_i, \cdots , v^c_K\rbrace $ in shared vocabulary to learn its final representation $H_{c}$=$\lbrace H^{c}_1, \cdots , H^{c}_K\rbrace $. In the proposed SFNMT model, we introduce an additional multi-head attention layer to fuse the compressed sentence and the original input sentence for learning a more effective source representation.

Specifically, for the multi-head attention-fusion layer, a compressed sentence-specific context representation $H_x^c$ is computed by the multi-head attention on the original sentence representation $H_x$ and the compressed sentence representation $H_c$: $H_x^c$ and $H_x$ are added to form a fusion source representation $H_{x}^{^{\prime }}$: Finally, the $H_{x}^{^{\prime }}$ instead of $H_{x}$ is input to the Eq. (DISPLAY_FORM17) in turn for predicting the target translations word by word.

The context $c_i^{\prime }$ is input to replace the $c_i$ the Eq. (DISPLAY_FORM18) to compute the probabilities of next target word.

NMT with ESC ::: Backbone Both-side Fusion based NMT	In the backbone both-side fusion based NMT (BBFNMT) model, we combine BSFNMT and BTFNMT. Both the original representation $H_x$ and its compressed enhanced representation $H_x^{^{\prime }}$ are as the input to the decoder. Similarly, we introduce an additional “encoder-decoder attention" module into the decoder to learn the compressed sequence enhanced context $b_i^{^{\prime }}$ at the current time-step $i$: Then, the context gate $g_i$ consistent with BTFNMT is applied to combine the two context information $c_i$ and $b_i^{^{\prime }}$.

In addition, we also reported the state-of-the-art results in recent literatures, including modelling local dependencies (Localness) , fusing multiple-layer representations in SANs (Context-Aware) , and fusing all global context representations in SANs (global-deep context) . MultiBLEU was used to evaluate the translation task.

These results show that our method is robust for improving the translation of other language pairs.

Meanwhile, +Semi-supervised ESC with annotated data fine-tuning outperformed both +Unsupervised and +supervised ESC.

Explicit Sentence Compression ::: Unsupervised ESC	A major challenge in supervised sentence compression is the scarce high quality human annotated parallel data. In practice, due to the lack of parallel annotated data, the supervised sentence compression model cannot be trained or the annotated data domain is different, resulting in the sentence compression model trained on the in-domain performing poorly on the out-of-domain. Supervised sentence compression models have achieved impressive performances based on large corpora containing pairs of verbose and compressed sentences with human annotation , . However, the effectiveness relies heavily on the availability of large amounts of parallel original and human-annotated compressed sentences. This hinders the sentence compression approach from further improvements for many low-resource scenarios. Recently, motivated by recent progress in unsupervised cross-lingual embeddings, the unsupervised NMT , ,  opened the door to solving the problem of sequence-to-sequence learning without any parallel sentence pairs. It takes advantage of the lossless (ideal situation) nature of machine translation between languages; i.e., it can translate language $L_1$ to language $L_2$ and back translate $L_2$ to language $L_1$. However, sentence compression does not have this feature.

Question: Which baselines to they compare to?
Answer: AllText, F8W