SlowGuess/ACL_2020_OCR huggingface dataset:
Yue Cao, Hui Liu, Xiaojun Wan
{yuecao,xinkeliuhui,wanxiaojun}  $@$  pku.edu.cn
Cross-lingual summarization is the task of generating a summary in one language given a text in a different language. Previous works on cross-lingual summarization mainly focus on using pipeline methods or training an end-to-end model using the translated parallel data. However, it is a big challenge for the model to directly learn cross-lingual summarization as it requires learning to understand different languages and learning how to summarize at the same time. In this paper, we propose to ease the cross-lingual summarization training by jointly learning to align and summarize. We design relevant loss functions to train this framework and propose several methods to enhance the isomorphism and cross-lingual transfer between languages. Experimental results show that our model can outperform competitive models in most cases. In addition, we show that our model even has the ability to generate cross-lingual summaries without access to any cross-lingual corpus.
Neural abstractive summarization has witnessed rapid growth in recent years. Variants of sequence-to-sequence models have shown to obtain promising results on English (See et al., 2017) or Chinese summarization datasets. However, Cross-lingual summarization, which aims at generating a summary in one language from input text in a different language, has been rarely studied because of the lack of parallel corpora.
Early researches on cross-lingual abstractive summarization are mainly based on the summarization-translation or translation-summarization pipeline paradigm and adopt different strategies to incorporate bilingual features (Leuski et al., 2003; Orasan and Chiorean, 2008; Wan et al., 2010; Wan, 2011) into the pipeline model.
Recently, Shen et al. (2018) first propose a neural cross-lingual summarization system based on a large-scale corpus. They first translate the texts automatically from the source language into the target language and then use the teacher-student framework to train a cross-lingual summarization model. Duan et al. (2019) further improve this teacher-student framework by using genuine summaries paired with the translated pseudo source sentences to train the cross-lingual summarization model. Zhu et al. (2019) propose a multi-task learning framework to train a neural cross-lingual summarization model.
Cross-lingual summarization is a challenging task as it requires learning to understand different languages and learning how to summarize at the same time. It would be difficult for the model to directly learn cross-lingual summarization. In this paper, we explore this question: can we ease the training and enhance the cross-lingual summarization by establishing alignment of context representations between two languages?
Learning cross-lingual representations has been proven a beneficial method for cross-lingual transfer for some downstream tasks (Klementiev et al., 2012; Artetxe et al., 2018; Ahmad et al., 2019; Chen et al., 2019). The underlying idea is to learn a shared embedding space for two languages to improve the model's ability for cross-lingual transfer. Recently, it has been shown that this method can also be applied to context representations (Aldarmaki and Diab, 2019; Schuster et al., 2019). In this paper, we show that the learning of cross-lingual representations is also beneficial for neural cross-lingual summarization models.
We propose a multi-task framework that jointly learns to summarize and align context-level representations. Concretely, we first integrate monolingual summarization models and cross-lingual summarization models into one unified model and then
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6220-6231 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
build two linear mappings to project the context representation from one language to the other. We then design several relevant loss functions to learn the mappers and facilitate the cross-lingual summarization. In addition, we propose some methods to enhance the isomorphism and cross-lingual transfer between different languages. We also show that the learning of aligned representation enables our model to generate cross-lingual summaries even in a fully unsupervised way where no parallel cross-lingual data is required.
We conduct experiments on several public crosslingual summarization datasets. Experiment results show that our proposed model outperforms competitive models in most cases, and our model also works on the unsupervised setting. To the best of our knowledge, we are the first to propose an unsupervised framework for learning neural crosslingual summarization.
In summary, our primary contributions are as follow:
- We propose a framework that jointly learns to align and summarize for neural cross-lingual summarization and design relevant loss functions to train our system.
- We propose a procedure to train our crosslingual summarization model in an unsupervised way.
- The experimental results show that our model outperforms competitive models in most cases, and our model has the ability to generate cross-lingual summarization even without any cross-lingual corpus.
We show the overall framework of our proposed model in Figure 1. Our model consists of two encoders, two decoders, two linear mappers, and two discriminators.
Suppose we have an English source text  $\boldsymbol{x} = \{x_1, \ldots, x_m\}$  and a Chinese source text  $\boldsymbol{y} = \{y_1, \ldots, y_n\}$ , which consist of  $m$  and  $n$  words, respectively. The English encoder  $\phi_{E\chi}$  (res. Chinese encoder  $\phi_{E\mathcal{V}}$ ) transforms  $\boldsymbol{x}$  (res.  $\boldsymbol{y}$ ) into its context representation  $z_x$  (res.  $z_y$ ), and the decoder  $\phi_{D\chi}$  (res.  $\phi_{D\mathcal{V}}$ ) reads the memory  $z_x$  (res.  $z_y$ ) and generates the corresponding English summary  $\tilde{\boldsymbol{x}}$  (res. Chinese summary  $\tilde{\boldsymbol{y}}$ ).
The mappers  $M_{\mathcal{X}}:\mathcal{Z}_x\to \mathcal{Z}_y$  and  $M_{\mathcal{Y}}:\mathcal{Z}_y\rightarrow$ $\mathcal{Z}_x$  are used for transformations between  $\pmb {z}_x$  and
Figure 1: The overall framework of our proposed model.
$z_{y}$ , and the discriminators  $D_{\mathcal{X}}$  and  $D_{\mathcal{Y}}$  are used for discriminating between the encoded representations and the mapped representations.
Taking English-to-Chinese summarization for example, our model generates cross-lingual summaries as follows: First we use the English encoder to get the English context representations, then we use the mapper to map English representations into Chinese space. Lastly the Chinese decoder is used to generate Chinese summaries.
In Section 3, we describe the techniques we adopt to enhance the cross-lingual transferability of the model. In Section 4 and Section 5, we describe the unsupervised training objective and supervised training objective for cross-lingual summarization, respectively.
3 Model Adjustment for Cross-Lingual Transfer
3.1 Normalizing the Representations
In our model, we adopt Transformer (Vaswani et al., 2017) as our encoder and decoder, which is the same with previous works (Duan et al., 2019; Zhu et al., 2019). The encoder and decoder are connected via cross-attention. The cross-attention is implemented as the following dot-product attention module:
$$
\text {A t t e n t i o n} (S, T) = \operatorname {s o f t m a x} \left(\frac {T S ^ {\top}}{\sqrt {d _ {k}}}\right) S \tag {1}
$$
where  $S$  is the packed encoder-side contextual representation,  $T$  is the packed decoder-side contextual representation and  $d_{k}$  is the model size.
6221
In the dot-product module, it would be beneficial if the contextual representations of the encoder and decoder have the same distributions. However, in the cross-lingual setting, the encoder and decoder deal with different languages and thus the distributions of the learned contextual representations may be inconsistent. This motivates us to explicitly learn alignment relationships between languages.
To make the contextual representations of two languages easier to be aligned, we introduce the normalization technique into the transformer model. Normalizing the word representations has been proved an effective technique on word alignment (Xing et al., 2015). After normalization, two sets of embeddings are both located on a unit hypersphere, which makes them easier to be aligned.
We achieve this by introducing the pre-normalization technique and replacing the LayerNorm with ScaleNorm (Nguyen and Salazar, 2019):
$$
\begin{array}{c} o _ {\ell + 1} = \text {L a y e r N o r m} \left(o _ {\ell} + F _ {\ell} \left(o _ {\ell}\right)\right) \\ \Downarrow \end{array}
$$
$$
o _ {\ell + 1} = o _ {\ell} + F _ {\ell} \left(\operatorname {S c a l e N o r m} \left(o _ {\ell}\right)\right)
$$
where  $F_{\ell}$  is the  $\ell$  -th layer and  $o_{\ell}$  is its input. The formula for calculating ScaleNorm is:
$$
\operatorname {S c a l e N o r m} (x; g) = g \cdot x / \| x \| \tag {2}
$$
where  $g$  is a hyper-parameter.
An additional benefit of ScaleNorm is that after being normalized, the dot-product of two vectors  $u^{\top}v$  is equivalent to their cosine distance  $\frac{u^{\top}v}{\|u\|\|v\|}$ , which may benefit the attention module in Transformer. We will conduct experiments to verify this.
3.2 Enhancing the Isomorphism
A key assumption of aligning the representations of two languages is the isomorphism of learned monolingual representations. Some researchers show that the isomorphism assumption weakens when two languages are etymologically distant (Søgaard et al., 2018; Patra et al., 2019). However, Ormazabal et al. (2019) show that this limitation is due to the independent training of two separate monolingual embeddings, and they suggest to jointly learn cross-lingual representations on monolingual corpora. Inspired by Ormazabal et al. (2019), we take the following approaches to address the isomorphism problem.
First, we combine the English and Chinese summarization corpora and build a unified vocabulary.
Second, we share encoders and decoders in our model. Sharing encoders and decoders can also enforce the model to learn shared contextual representations across languages. For the shared decoder, to indicate the target language, we set the first token of the decoder to specify the language the module is operating with. Third, we train several monolingual summarization steps before cross-lingual training, as shown in the first line in Alg. 1. The pre-trained monolingual summarization steps also allow the model to learn easier monolingual summarization first, then further learn cross-lingual summarization, which may reduce the training difficulty.
4 Unsupervised Training Objective
We describe the objective of unsupervised crosslingual summarization in this section. The whole training procedure can be found in Alg. 1.
Summarization Loss Given an English text-summary pair  $\pmb{x}$  and  $\pmb{x}'$ , we use the encoder  $\phi_{E\chi}$  and the decoder  $\phi_{D\chi}$  to generate the hypothetical English summary  $\tilde{\pmb{x}}$  that maximizes the output summary probability given the source text:  $\tilde{\pmb{x}} = \arg \max_{\bar{\pmb{x}}} P(\bar{\pmb{x}} \mid \pmb{x})$ . We adopt maximum log-likelihood training with cross-entropy loss between hypothetical summary  $\tilde{\pmb{x}}$  and gold summary  $\pmb{x}'$ :
$$
\boldsymbol {z} _ {\boldsymbol {x}} = \phi_ {E _ {\boldsymbol {\chi}}} (\boldsymbol {x}), \quad \tilde {\boldsymbol {x}} = \phi_ {D _ {\boldsymbol {\chi}}} (\boldsymbol {z} _ {\boldsymbol {x}})
$$
$$
\mathcal {L} _ {\operatorname {s u m m} _ {\mathcal {X}}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = - \sum_ {t = 1} ^ {T} \log P \left(x _ {t} ^ {\prime} \mid \tilde {x} _ {<   t}, \boldsymbol {z} _ {\boldsymbol {x}}\right) \tag {3}
$$
where  $T$  is the length of  $\pmb{x}^{\prime}$ . The Chinese summarization loss  $\mathcal{L}_{\mathrm{summ}_y}$  is similarly defined for the Chinese encoder  $\phi_{E_y}$  and decoder  $\phi_{D_y}$ .
Generative and Discriminative Loss Given an English source text  $\pmb{x}$  and a Chinese source text  $\pmb{y}$ , we use the encoder  $\phi_{E\mathcal{X}}$  and  $\phi_{E\mathcal{Y}}$  to obtain the contextual representations  $z_{x} = \{z_{x_{1}},\ldots ,z_{x_{m}}\}$  and  $z_{y} = \{z_{y_{1}},\dots ,z_{y_{n}}\}$ , respectively. For Zhto-En summarization, we use the mapper  $M_y$  to map  $z_{y}$  into the English context space:  $z_{y\rightarrow x} = M_{\mathcal{Y}}(z_{y})$ . We hope the mapped distribution  $z_{y\rightarrow x}$  and the real English distribution  $z_{x}$  could be as similar as possible such that the English decoder can deal with cross-lingual summarization just like dealing with monolingual summarization.
To learn this mapping, we introduce two discriminators and adopt the adversarial training (Goodfellow et al., 2014) technique. We optimize the
mappers at the sentence-level $^{1}$  rather than word-level, which is inspired by Aldarmaki and Diab (2019) where they found learning the aggregate mapping can yield a more optimal solution compared to word-level mapping.
Concretely, we first average the contextual representations:
$$
\tilde {z} _ {y \rightarrow x} = \frac {1}{n} \sum_ {i = 1} ^ {n} (z _ {y \rightarrow x}) _ {i}, \quad \tilde {z} _ {x} = \frac {1}{m} \sum_ {i = 1} ^ {m} z _ {x _ {i}} (4)
$$
Then we train the discriminator  $D_{\mathcal{X}}$  to discriminate between  $\tilde{z}_{y\rightarrow x}$  and  $\tilde{z}_x$  using the following discriminative loss:
$$
\begin{array}{l} \mathcal {L} _ {d i s \chi} \left(\tilde {z} _ {y \rightarrow x}, \tilde {z} _ {x}\right) = - \log P _ {D \chi} \left(\mathrm {s r c} = 0 | \tilde {z} _ {y \rightarrow x}\right) \\ - \log P _ {D _ {\mathcal {X}}} (\operatorname {s r c} = 1 | \tilde {z} _ {x}) \tag {5} \\ \end{array}
$$
where  $P_{D\mathcal{X}}$  (src  $|\tilde{z})$  is the predicted probability of  $D\mathcal{X}$  to distinguish whether  $\tilde{z}$  is coming from the real English representation (src  $= 1$  ) or from the mapper  $M_y$  (src  $= 0$  ).
In our framework, the encoder  $\phi_{E_X}$  and mapper  $M_y$  together make up the generator. The generator tries to generate representations which would confuse the discriminator, so its objective is to maximize the discriminative loss in Eq. 5. Alternatively, we train the generator to minimize the following generative loss:
$$
\begin{array}{l} \mathcal {L} _ {g e n y} \left(\tilde {z} _ {y \rightarrow x}, \tilde {z} _ {x}\right) = - \log P _ {D \chi} \left(\operatorname {s r c} = 1 | \tilde {z} _ {y \rightarrow x}\right) \\ - \log P _ {D _ {\chi}} (\operatorname {s r c} = 0 | \tilde {z} _ {x}) \tag {6} \\ \end{array}
$$
The discriminative loss  $\mathcal{L}_{disy}(\tilde{z}_{x\rightarrow y},\tilde{z}_y)$  for  $D_{\mathcal{V}}$  , generative loss  $\mathcal{L}_{gen\chi}\left(\tilde{z}_{x\rightarrow y},\tilde{z}_y\right)$  for  $\phi_{E\gamma}$  and  $M_{\mathcal{X}}$  are similarly defined.
Notice that since we use vector averaging and adopt the linear transformation, it does not matter whether we apply the linear mapping before or after averaging the contextual representations, and the learned sentence-level mappers can be directly applied to word-level mappings.
Cycle Reconstruction Loss Theoretically, if we do not add additional constraints, there exist infinite mappings that can align the distribution of  $\tilde{z}_x$  and  $\tilde{z}_y$ , and thus the learned mappers may be invalid. In order to learn better mappings, we introduce the cycle reconstruction loss and back-translation loss to enhance them.
Given  $z_{x}$ , we first use  $M_{\mathcal{X}}$  to map it to the Chinese space, and then use  $M_{\mathcal{Y}}$  to map it back:
$$
z _ {x \rightarrow y} = M _ {\mathcal {X}} (z _ {x}), \quad \hat {z} _ {x} = M _ {\mathcal {Y}} (z _ {x \rightarrow y}) \tag {7}
$$
We force  $z_{x}$  and  $\hat{z}_{x}$  to be consistent, constrained by the following cycle reconstruction loss:
$$
\mathcal {L} _ {\text {c y c} _ {\chi}} \left(\boldsymbol {z} _ {\boldsymbol {x}}, \hat {\boldsymbol {z}} _ {\boldsymbol {x}}\right) = \left\| \boldsymbol {z} _ {\boldsymbol {x}} - \hat {\boldsymbol {z}} _ {\boldsymbol {x}} \right\| \tag {8}
$$
The cycle reconstruction loss  $\mathcal{L}_{\mathrm{cyc}_y}$  for  $z_{y}$  and  $\hat{z}_y$  is similarly defined.
Back-Translation Loss The cycle-reconstructed representation  $\hat{z}_x$  in Eq. 8 can be regarded as augmented data to train the decoder, which is similar to the back-translation in the Neural Machine Translation area.
Concretely, we use the decoder  $\phi_{D_x}$  to read  $\hat{z}_x$  and generate the hypothetical summary  $\hat{x}$ . The back-translation loss is defined as the cross-entropy loss between  $\hat{x}$  and gold summary  $x'$ :
$$
\hat {\boldsymbol {x}} = \phi_ {D _ {\mathcal {X}}} (\hat {\boldsymbol {z}} _ {\boldsymbol {x}})
$$
$$
\mathcal {L} _ {\text {b a c k} \chi} (\hat {\boldsymbol {z}} _ {\boldsymbol {x}}) = - \sum_ {t = 1} ^ {T} \log P \left(x _ {t} ^ {\prime} \mid \hat {x} _ {<   t}, \hat {\boldsymbol {z}} _ {\boldsymbol {x}}\right) \tag {9}
$$
The back-translation loss enhances not only the generation ability of the decoder but also the effectiveness of the mapper. The back-translation loss  $\mathcal{L}_{\mathrm{back}_y}$  for  $\hat{z}_y$  is similarly defined.
Total Loss The total loss for optimizing the encoder, decoder, and mapper of the English side is weighted sum of the above losses:
$$
\mathcal {L} _ {\mathcal {X}} = \mathcal {L} _ {\text {s u m m} \mathcal {X}} + \lambda_ {1} \mathcal {L} _ {\text {g e n} \mathcal {X}} + \lambda_ {2} \mathcal {L} _ {\text {c y c} \mathcal {X}} + \lambda_ {3} \mathcal {L} _ {\text {b a c k} \mathcal {X}} \tag {10}
$$
where  $\lambda_1, \lambda_2$ , and  $\lambda_3$  is the weighted hyperparameters.
The total loss of the Chinese side is similarly defined, and the complete loss of our model is the sum of English loss and Chinese loss:
$$
\mathcal {L} = \mathcal {L} _ {\mathcal {X}} + \mathcal {L} _ {\mathcal {Y}} \tag {11}
$$
The total loss for optimizing the discriminators is:
$$
\mathcal {L} _ {d i s} = \mathcal {L} _ {d i s x} + \mathcal {L} _ {d i s y} \tag {12}
$$
5 Supervised Training Objective
The supervised training objective contains the same summarization loss in unsupervised training objective (Eq. 3). In addition, it has X-summarization loss and reconstruction loss.
1The "sentence" in this paper can refer to the sequence containing multiple sentences.
6223
Algorithm 1 Cross-lingual summarization
Input: English summarization data  $\mathcal{X}$  and Chinese summarization data  $\mathcal{Y}$ .
1: Pre-train English and Chinese monolingual summarization several epochs on  $\mathcal{X}$  and  $\mathcal{Y}$ .
2: for  $i = 0$  to max_iters do
3: Sample a batch from  $\mathcal{X}$  and a batch from  $\mathcal{Y}$
4: if unsupervised then
5: for  $k = 0$  to dis_iters do
6: Update  $D_{\mathcal{X}}$  and  $D_{\mathcal{Y}}$  on  $\mathcal{L}_{dis}$  in Eq.5.
7: (a) Update  $\phi_{E\mathcal{X}},\phi_{E\mathcal{Y}},\phi_{D\mathcal{X}}$  , and  $\phi_{D\mathcal{Y}}$  8: on  $\mathcal{L}_{\mathrm{summ}}$  in Eq. 3.
9: (b) Update  $\phi_{E\mathcal{X}},\phi_{E\mathcal{Y}},M_{\mathcal{X}}$  , and  $M_y$  10: on  $\mathcal{L}_{\mathrm{gen}}$  in Eq. 6.
11: (c) Update  $\phi_{E\chi},\phi_{E\mathcal{V}},M_{\mathcal{X}}$  , and  $M_y$  12: on  $\mathcal{L}_{\mathrm{cyc}}$  in Eq. 8.
13: (d) Update  $M_{\mathcal{X}}, M_{\mathcal{Y}}, \phi_{D_{\mathcal{X}}}$ , and  $\phi_{D_{\mathcal{Y}}}$   
14: on  $\mathcal{L}_{\mathrm{back}}$  in Eq. 9.
15: else if supervised then
16: (a) Upate  $\phi_{E\mathcal{X}},\phi_{E\mathcal{Y}},\phi_{D\mathcal{X}}$  , and  $\phi_{Dy}$  17: on  $\mathcal{L}_{\mathrm{summ}}$  in Eq. 3.
18: (b) Update  $\phi_{E\chi},\phi_{E\gamma},\phi_{D\chi}$  , and  $\phi_{Dy}$  19: on  $\mathcal{L}_{\mathrm{xsumm}}$  in Eq. 13.
20: (c) Update  $\phi_{E\chi},\phi_{E\mathcal{V}},M_{\mathcal{X}}$  , and  $M_y$  21: on  $\mathcal{L}_{\mathrm{rec}}$  in Eq. 14.
X-Summarization Loss Given a parallel English source text  $\mathbf{x}$  and Chinese summary  $\mathbf{y}'$ . We use  $\phi_{E\chi}$ ,  $M_{\chi}$ , and  $\phi_{D\mathcal{Y}}$  to generate the hypothetical Chinese summary  $\tilde{\mathbf{y}}$ , then train them with cross-entropy loss:
$$
\begin{array}{l} \boldsymbol {z} _ {\boldsymbol {x}} = \phi_ {E _ {\mathcal {X}}} (\boldsymbol {x}), \boldsymbol {z} _ {\boldsymbol {x} \rightarrow \boldsymbol {y}} = M _ {\mathcal {X}} (\boldsymbol {z} _ {\boldsymbol {x}}), \tilde {\boldsymbol {y}} = \phi_ {D _ {\mathcal {Y}}} (\boldsymbol {z} _ {\boldsymbol {x} \rightarrow \boldsymbol {y}}) \\ \mathcal {L} _ {\mathrm {x s u m m} _ {\mathcal {X}}} \left(\boldsymbol {x}, \boldsymbol {y} ^ {\prime}\right) = - \sum_ {t = 1} ^ {T} \log P \left(y _ {t} ^ {\prime} \mid \tilde {y} _ {<   t}, \boldsymbol {x}\right) \tag {13} \\ \end{array}
$$
The X-summarization loss for a Chinese text  $y$  and English summary  $x^{\prime}$  is similarly defined.
Reconstruction Loss Since the cross-lingual summarization corpora are constructed by translating the texts to the other language, the English texts and the Chinese texts are parallel to each other. We can build a reconstruction loss to align the sentence representation for the parallel English and Chinese texts.
Specifically, supposing  $\mathbf{x}$  and  $\mathbf{y}$  are parallel source English and Chinese texts, we first use  $\phi_{E\chi}$  and  $\phi_{E\gamma}$  to obtain contextual representations  $z_{x}$
and  $z_{y}$ , respectively. Then we average the contextual representations to get their sentence representations and use the mappers to map them into the other language. Since the English and Chinese texts are translations to each other, the semantics of their sentence representations should be the same. Thus we design the following reconstruction loss:
$$
\tilde {z} _ {x} = \frac {1}{m} \sum_ {i = 1} ^ {m} z _ {x _ {i}}, \quad \tilde {z} _ {y \rightarrow x} = \frac {1}{n} \sum_ {i = 1} ^ {n} (z _ {y \rightarrow x}) _ {i}
$$
$$
\mathcal {L} _ {\operatorname {r e c} \chi} \left(\tilde {z} _ {x}, \tilde {z} _ {y \rightarrow x}\right) = \| \tilde {z} _ {x} - \tilde {z} _ {y \rightarrow x} \| \tag {14}
$$
and  $\mathcal{L}_{\mathrm{rec}_y}$  is similarly defined.
Notice that the generative and discriminative loss, cycle-construction loss, and back-translation loss are unnecessary here because we can directly use aligned source text with objective 14 to align the context representations.
Total Loss The total loss for training the English side is:
$$
\mathcal {L} _ {\mathcal {X}} = \mathcal {L} _ {\text {x s u m m} \mathcal {X}} + \lambda_ {1} \mathcal {L} _ {\text {s u m m} \mathcal {X}} + \lambda_ {2} \mathcal {L} _ {\text {r e c} \mathcal {X}} \tag {15}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  is the weighted hyper-parameters. The total loss of the Chinese side is similarly defined.
We conduct experiments on English-to-Chinese (En-to-Zh) and Chinese-to-English (Zh-to-En) summarizations. Following Duan et al. (2019), we translate the source texts to the other language to form the (pseudo) parallel corpus. Since they do not release their training data, we translate the source text ourselves through the Google translation service. Notice that Zhu et al. (2019) translate the summaries rather than source texts.
Since Duan et al. (2019) use Gigaword and DUC2004 datasets for experiments while Zhu et al. (2019) use LCSTS and CNN/DM for experiments, we conduct experiments on all the 4 datasets. When comparing with Duan et al. (2019) and Zhu et al. (2019), we use the same number of translated parallel data for training. Due to limited computing resources, we only do unsupervised experiments on gigaword and LCSTS datasets.
Notice that the test sets provided by Zhu et al. (2019) are unprocessed, therefore we have to process the test samples they provided ourselves.
6224
6.2 Dataset
Gigaword English Gigaword corpus (Napolese et al., 2012) contains 3.80M training pairs, 2K validation pairs, and 1,951 test pairs. We use the human-translated Chinese source sentences provided by (Duan et al., 2019) to do Zh-to-En tests.