Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Year: 2023

Abstract:
This paper presents Z-Code++, a new pretrained language model optimized for abstractive text summarization. The model extends the state of the art encoder-decoder model using three techniques. First, we use a two-phase pre-training process to improve model’s performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. Our model is parameterefficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.

heading: 1 Introduction
text: Text summarization aims at producing a concise and fluent summary while preserving salient content and overall meaning of the source documents. It has been applied in a wide range of real-world applications, e.g., summarizing Web search results for interactive information retrieval (Gao et al., 2022) and generating medical summaries from doctor-patient conversation transcripts (Zhang et al., 2021).
While the extractive approach is the dominant approach in commercial systems due to its simplicity and effectiveness (Allahyari et al., 2017), the abstractive approach is getting more attention in the research community as neural language models are used (e.g., Rush et al., 2015; Nallapati et al.,
2016; Chopra et al., 2016; Liu and Lapata, 2019b,a; Pasunuru et al., 2021). Compared to the extractive approach where a summary is constructed using extracted sentences, abstractive summarizers paraphrase the idea of the source documents in a new form, and have a potential of generating more concise and coherent summaries.
However, good abstractive summarizers are harder to develop since we have to deal with problems like semantic representation, inference and low-resource text generation, which are more challenging than sentence extraction. Recently, largescale pre-trained language models (PLMs) such as PEGASUS (Zhang et al., 2020), GPT (Radford et al., 2019; Brown et al., 2020), T5 (Raffel et al., 2020), have been applied for abstractive summarization. While these models can produce surprisingly fluent text, the generated summaries often contain factual inconsistencies, caused by distorted or fabricated facts about the source documents, which is known as the hallucination problem (Kryściński et al., 2019; Celikyilmaz et al., 2020; Ji et al., 2022). In addition, since the amount of text in the source documents can be very large, it is expensive to train an end-to-end abstractive model (e.g., an encoder-decoder transformer model) given the memory constraints of current hardware and the latency constraints of applications such as online document summarization for interactive information retrieval. Therefore, a two-stage approach is widely used, where a subset of document sentences is coarsely selected using an extractive summarizer, and an abstractive summarizer generates the summary conditioning on the extraction (Liu and Lapata, 2019b). This approach is sub-optimal in that salient information might be missed in the extraction.
In this paper, we propose a new encoderdecoder PLM optimized for abstractive summarization, Z-Code++, which significantly extends Z-Code (Wang et al., 2020), a state-of-the-art PLM
5095
developed for machine translation, as follows.
First, Z-Code++ is pre-trained on web text using two tasks, replaced token detection (RTD) and corrupted span prediction (CSP). RTD uses a generator to generate ambiguous corruptions and a discriminator to distinguish the ambiguous tokens from the original inputs (Clark et al., 2020). RTD is proved to be more sample-efficient than the classic mask language modeling (MLM) task in learning text representations for language understanding (Bajaj et al., 2022; Hao et al., 2021). In CSP, a consecutive segment of tokens are corrupted and the model is learned to predict the corrupted spans using all the uncorrupted tokens in the original input (Raffel et al., 2020; Joshi et al., 2020). CSP can be viewed as a generalized form of gap sentences generation (GSG), a pre-training task tailored to abstractive summarization (Zhang et al., 2020), where the spans are entire sentences. CSP outperforms GSG in our experiments. In the second phase of grounded pre-training (Peng et al., 2022), the model is continually trained on summarization corpora of documents-summary pairs to better support low-resource fine-tuning to downstream summarization tasks that require the model to produce summaries grounded in source documents. We find in our experiments that grounded pre-training significantly boosts the results on downstream tasks in low-resource settings.
To handle the large input documents, we use fusion-in-encoder (FiE), a simple yet effective method of encoding long sequences in a hierarchical manner. It works by first splitting the input sequence into small chunks, applying attention on each chunk locally to get the chunk representation, and applying attention globally on the concatenated chunk representations to get the representation of the original input.
In addition, we replace the self-attention layer in the encoder with the disentangled attention (DA) layer (He et al., 2020, 2021), where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. DA is motivated by the observation that the attention weight of a word pair depends on not only their contents but their relative positions. For example, the dependency between the words “deep” and “learning” is much stronger when they occur next to each other than when they
occur in different sentences. We show in our experiments that DA leads to a more effective abstractive summarizer.
For evaluation, we have pre-trained two ZCode++ models on English data and multi-lingual data, respectively. The English model is trained using 160G English text data and the vocabulary of DeBERTaV2 (He et al., 2020). The multilingual model is trained on mC4 corpus which is the same as mT5. These models are evaluated on 13 text summarization tasks across 5 languages, and create new state of the art on 9 tasks. As of May 6th, 2022, Z-Code++ sits atop of the XSum leaderboard, surpassing UL220B, T511B and PEGASUS. It is worth noting that our models are very parameter-efficient. For example, Z-Code++ outperforms PaLM540B, which is 600x larger in model parameters, on XSum, and outperforms a fine-tuned, 200x larger, GPT3175B on SAMSum. In zero-shot and few-shot settings, our models outperform more substantially the competing models.

heading: 2 Z-Code++
text: This section describes three modeling techniques we have exploited to optimize Z-Code++ for abstractive summarization, including two-phase pretraining, disentangled attention, and long sequence encoding.

heading: 2.1 Two-Phase Pre-Training
text: The two-phase pre-training, which includes the language model pre-training and grounded pretraining phases, is inspired by the GODEL recipe (Peng et al., 2022) that has been proposed to pretrain language models for grounded text generation tasks, such as dialog response generation and abstractive question-answering.
In the language model pre-training phase, ZCode++ is pre-trained using two language modeling tasks, replaced token detection (RTD) (Clark et al., 2020) and corrupted span prediction (CSP) (Raffel et al., 2020; Joshi et al., 2020). As illustrated in Figure1 (Left), RTD uses a generator trained with MLM to generate ambiguous tokens to replace tokens in the original input X , and a discriminator to determine whether a token is from X or generated by the generator. Let θG and θD be the parameters of the generator and the discriminator, respectively. The MLM loss of the generator is written as
LMLM “ E ´ ´ řiPC log pθG ´ x̃i,G “ xi|X̃G ¯¯ , (1)
where X̃G is the input to the generator by randomly masking 15% tokens in original input X . The input sequence of the discriminator is constructed by replacing the masked tokens, xi, i P C, with the tokens, x̃i, sampled by the generator as
x̃i,D “ # x̃i „ pθG ´ x̃i,G “ xi|X̃G ¯ , i P C
xi, i R C. (2)
Then the discriminator is trained using the loss LRTD “ E ´ ´ ři log pθD ´ 1 px̃i,D “ xiq |X̃D, i ¯¯ , (3)
where 1p¨q is the indicator function and X̃D is the input to the discriminator constructed via (2). In ELECTRA (Clark et al., 2020), the discriminator and generator share token embeddings and their parameters are optimized via MLM and RTD jointly as L “ LMLM ` λLRTD. However, as pointed out in (He et al., 2021), such embedding sharing makes training highly inefficient since MLM and RTD pull token embeddings into very different directions, creating the “tug-of-war” dynamics. MLM tries to map the tokens that are semantically similar to the embedding vectors that are close to each
other. RTD, on the other hand, tries to discriminate semantically similar tokens, pulling their embeddings as far as possible to optimize the classification accuracy. Thus, we use the method of gradientdisentangled embedding sharing (He et al., 2021) by re-parameterizing the token embeddings of the discriminator as
ED “ sgpEGq ` E∆, (4)
where ED and EG are the embedding parameters of the discriminator and generator, respectively, sg is the stop gradient operator which only allows gradients propagation through E∆. E∆ is initialized as a zero matrix. In each training pass, we first run a forward pass of the generator to generate inputs for the discriminator, and then a backward pass to update EG with respect to MLM. After that, we run a forward pass for the discriminator using the inputs produced by the generator and run a backward pass with respect to the RTD loss to update ED by propagating gradients only through E∆. After model training, E∆ is added to EG and the sum is saved as ED in the discriminator, as Equation 4.
The CSP is widely used to optimize the encoderdecoder PLMs such as T5 (Raffel et al., 2020). As illustrated in Figure 1 (Right), given input string X , we first select a continuous span Yi by first
randomly selecting a start position in X and a span with an average length of 3. Then we replace the selected span Yi with a sentinel token [Mi]. We repeat the process until the replaced tokens amount to 15% of all tokens in X . Then, we feed the corrupted input X̃CSR to the encoder. The encoderdecoder model is then trained to recover the Yi from the context. The CSP loss is written as
LCSP “ E ¨ ˝´ |Y |ÿ
i“1 log pθ
´ Yi|X̃CSP,Yăi ¯ ˛ ‚ (5)
If we restrict the corrupted span Yi to a complete sentence, CSP is equivalent to the GSG task which simulates the process of extractive summarization and is shown to be effective for training abstractive summarizers (Zhang et al., 2020). In this study, we find the that CSP, as a more general form of GSG, works better across many natural language understanding and generation tasks, including summarization, as to be discussed in Section 3.
Combining the pre-training tasks of MLM, RTD and CSP, in the language model pre-training phase, Z-Code++ is optimized using the joint loss as L “ λ1LMLM ` λ2LRTD ` λ3LCSP, where we set λ1 “ 1, λ2 “ 30, λ3 “ 1 in our experiment.
In the second phase of grounded pre-training, ZCode++ is continually pre-trained on a collection of summarization datasets, as shown in Table 1, which consist of documents-summary pairs pX,Y q, to better support low-resource finetuning for downstream summarization tasks that require the model to generate target summaries Y grounded in source documents X , as
ppY |Xq “ Nź
n“1 ppyn|y1, ¨ ¨ ¨ , yn´1,Xq (6)
Following T0 (Wei et al., 2021), FLAN (Sanh et al., 2022), and GODEL (Peng et al., 2022), we add for each training pair pX,Y q a natural language instruction of the summarization task, as illustrated in the below example and in Table 1. In our experiment, we only apply grounded pre-training for low-resource summarizations. Unless specified, we apply the first phase Z-Code++ to downstream task adaptation.

heading: 2.2 Disentangled Attention
text: Disentangled Attention (DA) is first used in DeBERTa (He et al., 2020, 2021). DA is an extension of the classic self-attention (SA) mechanism in that
DA represents each input word using two separate vectors: one for the content and the other for the position. Meanwhile, its attention weights among words are computed via disentangled matrices on both their contents and relative positions. The experiments of DeBERTa shows that DA is more efficient than SA to encode the positional dependency in Transformer models. Z-Code++ adopts DA in modeling. Our experiments show that DA leads to a more effective abstractive summarizer.

heading: 2.3 Long Sequence Encoding
text: It is challenging to encode long sequence given the OpN2q memory and computation complexity of self-attention and DA. Various sparse attention mechanisms have been proposed to alleviate the problem. However, sparse attention often hurts performance on short sequences due to the decrease of attention precision. Inspired by fusion-indecoder (Izacard and Grave, 2020) and hierarchical transformer (Liu and Lapata, 2019a), we propose fusion-in-encoder (FiE), a simple but effective mechanism to encode long sequences while retaining high attention precision on short sequences. FiE works by separating the L encoder layers of Z-Code++ into m local layers and n global layers. In each local layer, the hidden states of input sequence are split into small chunk of size l (e.g. 256 or 512), and self-attention (or DA) is only applied to those small chunks locally with a complexity of Opl2q. After local layer, the hidden states of those small chunks are concatenated together to form the representation of the long sequence. Global layers are the same as original self-attention (or DA) layers in encoder to fuse the local states of small chunks. With FiE, the complexity of encoder is reduced from OpLN2q to OpmNl`nN2q. Both the local layers and fusion layers are initialized with the corresponding weights of encoder
layers of Z-Code++. Please check Appendix A.3 for a graphic illustration of FiE. In experiment, we show that compared with LongT5 (Guo et al., 2021) which applies sparse attention that is specifically optimized for summarization, Z-Code++ achieves similar or better performance on long document summarization tasks.

heading: 3 Experiment

heading: 3.1 Experiment Setups
text: Datasets We validate the effectiveness of ZCode++ on 11 representative summarization tasks, which are detailed in Table 2. Among these datasets, XSum (Narayan et al., 2018), CNNDM (See et al., 2017), NewsRoom (Grusky et al., 2018), and MultiNews (Fabbri et al., 2019) are news article summarizations, while SAMSum (Gliwa et al., 2019), MediaSum (Zhu et al., 2021), and Reddit TIFU (Kim et al., 2018) are conversation-like summarization tasks. Following LongT5, we use MultiNews, MediaSum, arXiv (Cohan et al., 2018) and PubMed (Cohan et al., 2018) to assess the long document summarization capability. In addition, WikiLingua (Ladhak et al., 2020) and MLsum (Scialom et al., 2020) are used to evaluate the capacity of Z-Code++ on multilingual summarization.
Implementation Details We have built our models following the same setting as T5. For ZCode++LARGE, there are 24 layers for the encoder and 24 layers for the decoder with 1024 hidden dimension sizes and 16 self-attention heads. Following DeBERTaV3 (He et al., 2021), a 6-layer generator with the same structure as the encoder is employed during the pre-training stage. ZCode++LARGE is trained on 160G data with a vocabulary of size 128k. Our code is implemented based on open sourced pytorch1 and DeBERTa2.
1https://pytorch.org/ 2https://github.com/microsoft/DeBERTa
We pre-train Z-Code++LARGE for 1M steps with a batch size of 2048 in Azure Machine Learning cluster3 with 128 A-100 GPUS for 20 days. AdamW is used as the optimizer in all experiments. For tasks with an input length of more than 10k words, i.e., arXiv and PubMed, Fusion-in-Encoder is used to encode the document as described in 2.3. For the other standard summarization tasks with moderate input length (i.e., less than 4k words) we directly feed the input document to the encoder.
For multilingual summarization, we have built Z-Code++LARGE with the same architecture but different training data and vocabulary. Specifically, ZCode++LARGE is trained with mC4 data and a vocabulary of size 250k, which are the same as mT5 (Xue et al., 2021). Following XLM (Lample and Conneau, 2019), CCMatrix (Schwenk et al., 2019) and CCAligned (El-Kishky et al., 2019), parallel data is used to enhance the cross-lingual summarization of Z-Code++LARGE. Due to the limited computational resource, Z-Code++LARGE is trained with only 500B tokens instead of 1T tokens as that for mT5 training.
We use grid search to choose the grounded training and fine-tuning hyper-parameters based on validation set, the parameter search range are listed in appendix A.1.

heading: 3.2 Experiment Results

heading: 3.2.1 Results on Standard English Summarization Tasks
text: We first conduct experiments to compare the performance of Z-Code++LARGE with SOTA and PEGASUSLARGE on 7 representative standard public English summarization datasets with moderate document length, including AESLC, SAMSum, XSUM, WikiHow, NewsRoom, CNN/DailyMail(CNNDM), and Reddit TIFU. Following (Chowdhery et al., 2022; Gehrmann et al., 2022), for each dataset we re-
3https://ml.azure.com
port the average F-measure ROUGE-2 score of 5 runs. Detailed F-measure of ROUGE-1/ROUGE2/ROUGE-L scores can be found in Appendix ??.
As listed in Table 3 , Z-Code++LARGE achieves substantial improvements over PEGASUSLARGE, which is a PLM optimized for abstractive summarization, on 6 out of 7 tasks in terms of ROUGE-2 F-measure score. 5 Specifically, on SAMSum, a critical dialog summarization task, Z-Code++LARGE outperforms GPT-3175B that is extensively fine-
5The computation cost of the embedding layer is not factored in, so we only display the primary model parameters in the table, excluding those from the embedding layer. This approach is consistent across all subsequent experiments for comparison purposes.
tuned with LoRA(Hu et al., 2021) even though Z-Code++LARGE has less than 1/175 parameters of GPT-3175B. Furthermore, Z-Code++LARGE lifts SOTAs by 0.36 points on average. These results demonstrate the effectiveness of Z-Code++ on English document summarization tasks. Additionally, we observe that Z-Code++LARGE outperforms PEGASUSLARGE on WikiHow, SAMSum, Reddit TIFU, and AESLC by a much larger margin (ą 1%) than it does on XSum, CNNDM, and NewsRoom. We speculate that PEGASUS is biased to news-like tasks since it is heavily pre-trained on large amounts of news data. In contrast, Z-Code++ is pre-trained on diverse web data and thus is more adaptable for general-domain summarization tasks.

heading: 3.2.2 Results on Long Document Summarization
text: We compare Z-Code++ to PEGASUS and LongT5, which is optimized for long document summarization. Results in Table 4 show that Z-Code++LARGE exceeds all the strong competitors on all long document summarization datasets and lifts SOTA by 0.35 point on average. For FiE, which is used to generate summaries for arXiv and PubMed, we choose the chunk size l “ 256, and choose the last
5We have achieved 24.1 R2 score on CNNDM using exposure debiasing to address the mismatch between teacher forcing and student forcing learning, as we will describe in detail in a future publication.
layer of encoder as fusion layer based on the experiment results. Specifically, Z-Code++LARGE outperforms LongT53B with less than 1/3 of parameters. These results demonstrate both the effectiveness and flexibility of Z-Code++ by using DisentangledAttention to encode word dependencies.

heading: 3.2.3 Human Evaluation
text: As human evaluation is the most reliable measurement of the quality of natural language generation models, we submit the test results of XSum to the leaderboard (Khashabi et al., 2021) which requires human raters to compare the generated summaries side by side with human written references. Please check the paper of the leaderboad (Khashabi et al., 2021) to get more details of human evaluation process including instructions, dataset preparing, payments and demographics of the raters. We list the
human evaluation results in Table5. Z-Code++ outperforms all the other models, e.g., BARTLARGE, PEGASUSLARGE, T511B, UL220B(Tay et al., 2022), on the leaderboard in terms of human-overall score. As the human evaluation score is an average of sideby-side preference comparison scores, a score of 0.51 indicates that the annotators prefer the output of Z-Code++ to the human written references. Further more, while hallucination is one of the most critical problems for abstractive summarization, ZCode++ does not suffer much, i.e., 0.55, among the leaderbard. The human evaluation results validate that Z-Code++ produces higher quality summaries than other models.

heading: 3.2.4 Results on Multilingual Summarization
text: Following GEM-benchmark (Gehrmann et al., 2021), we evaluate the performance of ZCode++LARGE 6 on multilingual summarization with WikiLingua and MLSum. We compare ZCode++LARGE with mT5LARGE and mT5XLARGE. The results of PaLM540B, a state of the art PLM, are also listed in Table 6. Compared with mT5XLARGE, Z-Code++LARGE achieves substantially better performance across all the tasks with only 1/3 parameters and half training data. In addition, we observe a significant performance gap between Z-Code++LARGE and PaLM540B on WikiLingua, which is not surprising due to the sharp difference in model size and capacity. However, Z-Code++LARGE surpasses
6Note that Z-Code++LARGE for multilingual summarization is differently trained. Refer to 3.1 for more details.
PaLM540B on MLSum by a large margin, i.e., 3.7% on MLSum(de), 2.8% on MLSum(es), albeit ZCode++LARGE has less than 1/500 parameters. We believe that by scaling up Z-Code++ to a moderate size (e.g., 10B), the performance gap on WikiLingua would be mitigated. We leave it to future work.

heading: 3.2.5 Results on Low-Resource Summarization
text: We explore how well knowledge learned in different pre-training stages can generalize to lowresource summarization scenarios, i.e. zero/fewshot evaluation. For the grounded pre-training phase, we choose to include MediaSum, MultiNews, NewsRoom, and WikiHow datasets. Corresponding instructions are listed in Table 1. We reckon that incorporating diverse datasets and instructions is beneficial, which we leave it to future work. For the fine-tuning stage, following the setting in Zhang et al. (2020), we randomly select the number of training data to 0, 10, 100, and 1000, and sample examples from XSUM, CNNDM, and SAMSum, and then fine-tune Z-Code++ until no significant improvement on the validation set is observed. Note that 0 denotes zero-shot evaluation. Table 11 presents the results. By fine-tuning first-phase pre-trained model, Z-Code++LARGE outperforms T5LARGE by more than 3 points on average. PEGASUSLARGE exceeds Z-Code++LARGE when the number of training examples is less than 100, which is foreseeable as PEGASUSLARGE is pre-trained with a pseudo summarization objective. However, Z-Code++LARGE performs significantly better than them when it is trained with more than 100 examples, showing strong generalization in the few-shot setting. More importantly, with grounded pre-training, Z-Code++LARGE beats all the competing models by a large margin in both zero and few-shot settings, outperforming PEGASUSLARGE by 5.7/1.5/3.3 points on average. This suggests that instructions-grounded pre-training enables effective knowledge transfer to downstream lowresource tasks.

heading: 4 Conclusions
text: We present Z-Code++, an efficient and effective pre-trained language model optimized for abstractive text summarization. The model extends the encoder-decoder model using three techniques. The first is a two-phase pre-training process, where the model is first pre-trained using text corpora for language understanding, and then is continually
pre-trained on summarization corpora for grounded text generation. The second technique is the use of the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively. The third is the fusion-in-encoder method for encoding long sequence inputs. We present a comprehensive empirical study to validate the effectiveness of ZCode++. The model creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. In addition, we show that our model is parameter-efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. Z-Code++ also generalizes well to low-resource downstream tasks. For example, in zero-shot and few-shot settings, our model outperforms more substantially the competing models.
However, evaluation (Liang et al., 2022) and hallucinations are still two long-standing problems of summarizations that we do not touch with in this work, in the future we will 1) explore evaluation metrics that correlate well with human experience, 2) learn to summarize to better align with human preferences (Stiennon et al., 2020; Ouyang et al., 2022), and 3) ground summarization models on
world knowledge to largely reduce hallucinations (LeCun, 2022; Hafner et al., 2023).

heading: Limitations
text: In this paper, we introduce Z-Code++, a robust pre-trained model tailored for summarization tasks. However, it should be noted that there are certain limitations to our model. Firstly, the model is not versatile enough as it is specifically designed for summarization. It is unclear whether it performs well on other natural language tasks. Secondly, while FiE can handle document summarization, there are still significant potential for improving cost efficiency. Lastly, the evaluation of multilingual summarization is not thorough enough due to the limitations of available datasets. We intend to address these limitations in our future work.

heading: Ethics Statement
text: The same as all existing generative language models, the generated text of Z-Code++ raises various ethical considerations. One crucial consideration is the issue of potential hallucinations in the summaries generated by the model. The summaries produced by a generative model may not necessarily be faithful to the original article or entirely factual which may mislead the users to make incorrect decisions based on the summary without additional knowledge. In addition, another important consideration is the potential for bias in generated summaries, such as bias based on gender, race, and other factors.

heading: Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
text: Processing (EMNLP-IJCNLP), pages 540–551.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. NeurIPS.
Yann LeCun. 2022. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.
Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070– 5081.
Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730–3740.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
Bo Pang, Erik Nijkamp, Wojciech Kryściński, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2022. Long document summarization with topdown and bottom-up inference. arXiv preprint arXiv:2203.07586.
Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley, Chenyan Xiong, Yizhe Zhang, Mohit Bansal, and Jianfeng Gao. 2021. Data augmentation for abstractive query-focused multi-document summarization.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13666–13674.
Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Large-scale pretraining for goal-directed dialogue. Technical report, Microsoft Technical Report.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pretraining for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140–145.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zeroshot task generalization. In The Tenth International Conference on Learning Representations.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2019. Ccmatrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. Mlsum: The multilingual summarization corpus. arXiv preprint arXiv:2004.14900.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019.
Yiren Wang, ChengXiang Zhai, and Hany Hassan. 2020. Multi-task learning for multilingual neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1022–1034.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021. Primer: Pyramid-based masked sentence pre-training for multi-document summarization. arXiv preprint arXiv:2110.08499.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.
Longxiang Zhang, Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, and Matthew R Gormley. 2021. Leveraging pretrained models for automatic summarization of doctor-patient conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3693–3712.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. Mediasum: A large-scale media interview dataset for dialogue summarization. arXiv preprint arXiv:2103.06410.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906.

heading: A Appendix

heading: A.1 Hyper parameters
text: A.2 Rouge scores of summarization tasks
We list the rouge scores of summarizaiton tasks in table10

heading: A.3 Fusion-in-Encoder structure
text: In figure 3, we show the architecture of FiE.

heading: A.4 Ablation study
text: We conducted a comprehensive experiment to explore what is important for the encoder’s language understanding ability. Specifically, we experiment on the natural language inference task, e.g., MNLI (Williams et al., 2018), the question answering task,
e.g., SQuAD (Rajpurkar et al., 2016), the summarization tasks, e.g., XSum (Narayan et al., 2018) and CNNDM (See et al., 2017). The results in Table 12 show that using disentangled attention improves MNLI-matched/mismatched accuracy by 0.9%/1.2%, indicating an improvement in the encoder’s language understanding ability. This improvement is also reflected in the performance of two summarization tasks, which see an improvement in R2 scores by 0.39% and 0.22%. Removing RTD significantly decreased performance, indicating that it is essential for improving the model’s NLU capability.

heading: A.5 Evaluate on NLU tasks
text: In order to assess the model’s effectiveness on natural language understanding (NLU) tasks, we conducted experiments using the eight NLU tasks from
the GLUE dataset (Wang et al., 2019). These tasks are commonly used to evaluate sentence classification performance in machine learning. Our model, Z-Code++, was tested using two approaches: adapting only the encoder and fine-tuning with a classification head, similar to BERT, or adapting the encoder-decoder and treating the task as a generation task, similar to T5. We compared Z-Code++ to other encoder-based PLMs with similar structures, including BERT, RoBERTa, ELECTRA, DeBERTa, and DeBERTaV3, as well as T5 for the encoder-decoder comparison.
The results, shown in Table 13, demonstrate that Z-Code++ performs comparably or better than the
other models on all tasks. In particular, Z-Code++ outperformed the other encoder PLMs by an average of more than 1% and outperformed T5 on all tasks with an average improvement of 1.98% in test scores. These results demonstrate Z-Code++ as a strong universal language model with excellent performance on generation tasks and superior performance on NLU tasks.

heading: A.6 Evaluate on NLG tasks
text: We evaluated the language generation performance of Z-Code++ on a range of English tasks, including abstractive document summarization tasks (XSum, CNNDM, Wikilingual-en), a conversa-
tional summarization task (SAMSum), data-to-text tasks (WebNLG-en, E2ENLG) and a question answering task (SQuAD v1.1). We compared the performance of the Z-Code++ model to other stateof-the-art models with similar architectures and parameters, as shown in Table 14.
Results show that Z-Code++ outperforms all of the other models’ scores by a large margin in terms of ROUGE and BLEU scores. For example, Z-Code++ significantly outperformed T5XLARGE on CNNDM by 1% in terms of ROUGE-2 score, on the WebNLG-en task by 6.9%, and about 1% BLEU score on dialog response generation tasks. Even though it has less than 1/3 the parameters of T5XLARGE, Z-Code++ outperformed PEGASUS on SAMSum task by 4% in terms of ROUGE-2 score. We conjecture that PEGASUS is a model specifically optimized for summarization using 1500GB of news data, which may have introduced a domain mismatch with the conversational summarization task. We also compared Z-Code++ to other stateof-the-art models with extremely large parameters, including PaLM, GPT3, and UL2. Z-Code++ outperformed PaLM on three out of four tasks by a large margin, even though it has less than 1/600 the parameters of PaLM. Z-Code++ also outperformed UL220B on four out of five tasks, even though it has less than 1/20 the parameters of UL220B. These results demonstrate the efficiency of the Z-Code++ model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Instruction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 3, experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.1, experiment setup
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3, experiments and appendix A 1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3, experiments
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3, experiments
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3, experiments
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. We quote our results from public benchmark https://leaderboard.allenai.org/geniemt/submissions/public which run human evaluation from their backend.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. We quote our results from public benchmark https://leaderboard.allenai.org/geniemt/submissions/public which run human evaluation from their backend.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. We quote our results from public benchmark https://leaderboard.allenai.org/geniemt/submissions/public which run human evaluation from their backend.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. We quote our results from public benchmark https://leaderboard.allenai.org/geniemt/submissions/public which run human evaluation from their backend.

heading: Future Work
text: We intend to address these limitations in our future work