DiffuSum: Generation Enhanced Extractive Summarization with Diffusion
Year: 2023

Abstract:
Extractive summarization aims to form a summary by directly extracting sentences from the source document. Existing works mostly formulate it as a sequence labeling problem by making individual sentence label predictions. This paper proposes DiffuSum, a novel paradigm for extractive summarization, by directly generating the desired summary sentence representations with diffusion models and extracting sentences based on sentence representation matching. In addition, DiffuSum jointly optimizes a contrastive sentence encoder with a matching loss for sentence representation alignment and a multi-class contrastive loss for representation diversity. Experimental results show that DiffuSum achieves the new state-ofthe-art extractive results on CNN/DailyMail with ROUGE scores of 44.83/22.56/40.56. Experiments on the other two datasets with different summary lengths also demonstrate the effectiveness of DiffuSum. The strong performance of our framework shows the great potential of adapting generative models for extractive summarization. To encourage more following work in the future, we have released our codes at https://github.com/hpzhang94/

heading: 1 Introduction
text: Document summarization aims to compress text material while keeping its most salient information. It plays a critical role with the growing amount of publicly available text data. Automatic text summarization approaches can be divided into two streams: abstractive and extractive summarization. Although abstractive methods (Nallapati et al., 2016; Gupta and Gupta, 2019; Bae et al., 2019; Li et al., 2020) produce flexible and less redundant summaries, they suffer from problems of generating ungrammatical or even nonfactual contents (Kryściński et al., 2019; Zhang et al., 2022b). In
∗ equal contribution
contrast, extractive summarization forms a summary by directly extracting sentences from the source document. Thus, the extracted summaries are grammatically accurate and faithful.
We focus on extractive summarization in this work. Extractive summarization is commonly formulated as a sequence labeling problem, which predicts a 0/1 label for each sentence, indicating whether the sentence should be included in summary (Nallapati et al., 2017; Zhou et al., 2018; Liu and Lapata, 2019). Compared to individual sentence label prediction in the sequence labeling setting, generative models offer increased flexibility and attend to the entirety of input context. Recent works have also successfully applied generative models to wide-ranging token-level sequence labeling tasks (Athiwaratkun et al., 2020; Du et al., 2021; Yan et al., 2021). Nonetheless, how to apply generative models for sentence-level tasks like extractive summarization has not been explored.
Recently, continuous diffusion models have achieved great success in vision and audio domains (Ho et al., 2020; Kong et al., 2020; Yang et al., 2022; Rombach et al., 2022; Ho et al., 2022). Researchers have also attempted to apply diffusion models for text generation by converting the discrete token to continuous embeddings and mapping from embedding space to words with a rounding method (Li et al., 2022; Yuan et al., 2022; Strudel et al., 2022; Gong et al., 2022). However, these approaches are not applicable for sentence-level tasks like summarization: (1) Summarization has a relatively longer input context and larger generation length (around 3-6 sentences), while the above token-level diffusion-LM models are only applicable to short generation tasks like text simplification and question generation. Their performance tends to drop by a large margin when generating longer sequences; (2) The word embeddings generated by these models could be indistinguishable, resulting in ambiguous and hallucinated generation; (3)
13089
The rounding step in those existing diffusion models is less efficient and slows down the inference dramatically.
To address the above issues, we propose a novel extractive summarization paradigm, DiffuSum, which generates the desired summary sentence representations with transformer-based diffusion models and extracts summaries based on sentence representation matching. Instead of generating word by word, DiffuSum directly generates the desired continuous representations for each summary sentence and thus could process much longer text. DiffuSum is a summary-level framework since the transformer-based diffusion architecture generates all summary sentence representations simultaneously. Moreover, DiffuSum incorporates a contrastive sentence encoding module with a matching loss for sentence representation alignment and a multi-class contrastive loss (Khosla et al., 2020) for representation diversity. DiffuSum jointly optimizes the sentence encoding module and the diffusion generation module, and extracts sentences by representation matching without any rounding step. We validate DiffuSum by extensive experiments on three benchmark datasets and experimental results demonstrate that DiffuSum achieves a comparable or even better performance than state-of-the-art systems that rely on pre-trained language models. DiffuSum also shows a strong adaptation ability based on cross-dataset evaluation results.
We highlight our contributions in this paper as follows:
(i) We propose DiffuSum, a novel generationaugmented paradigm for extractive summarization with diffusion models. DiffuSum directly generates the desired summary sentence representations and then extracts sentences based on representation matching. To the best of our knowledge, this is the first attempt to apply diffusion models for the extractive summarization task.
(ii) We also introduce a contrastive sentence encoding module with a matching loss for representation alignment and a multi-class contrastive loss for representation diversity.
(iii) We conduct extensive experiments and analysis on three benchmark summarization datasets to validate the effectiveness of DiffuSum. DiffuSum achieves new extractive state-of-art results on CNN/DailyMail dataset with ROUGE scores of 44.83/22.56/40.56.

heading: 2 Related Work

heading: 2.1 Extractive Summarization
text: Recent advances in deep neural networks have dramatically boosted the progress in extractive summarization systems. Existing extractive summarization systems span an extensive range of approaches. Most works formulate the task as a sequence classification problem and use sequential neural models with different encoders like recurrent neural networks (Cheng and Lapata, 2016; Nallapati et al., 2016) and pre-trained language models (Egonmwan and Chali, 2019; Liu and Lapata, 2019; Zhang et al., 2023). Another group of work formulates extractive summarization as a node classification problem and applies graph neural networks to model inter-sentence dependencies (Xu et al., 2019; Zhang and Zhang, 2020; Wang et al., 2020; Zhang et al., 2022a). These formulations are sentence-level methods that make individual predictions for each sentence. Recently, Zhong et al. (2020) observed that a summary consisting of sentences with the highest scores is not necessarily the best. As a result, summary-level formulation like text matching (Zhong et al., 2020; An et al., 2023) and reinforcement learning (Narayan et al., 2018b; Bae et al., 2019) are proposed. Our proposed framework DiffuSum is also a novel summary-level extractive system with generation augmentation. Instead of sequentially labeling sentences, DiffuSum directly generates the desired summary sentence representations with diffusion models and extracts sentences by representation matching.

heading: 2.2 Diffusion Models on Text
text: Continuous diffusion models are first introduced in (Sohl-Dickstein et al., 2015) and have achieved great success in continuous domain generations like image, video, and audio (Kong et al., 2020; Yang et al., 2022; Rombach et al., 2022; Ho et al., 2022). Nevertheless, few works have applied continuous diffusion model to text data due to its inherently discrete nature. Among the initial attempts, Diffusion-LM (Li et al., 2022) first adapts continuous diffusion models for text by adding an embedding step and a rounding step, and designing a training objective to learn the embedding. DiffuSeq (Gong et al., 2022) proposes a diffusion model designed for sequenceto-sequence (seq2seq) text generation tasks by adding partial noise during the forward process and conditional denoising during the reverse pro-
cess. CDCD (Dieleman et al., 2022) is proposed for text modeling and machine translation based on variance-exploding stochastic differential equations (SDEs) on token embeddings. SeqDiffuSeq (Yuan et al., 2022) also proposes an encoderdecoder diffusion model architecture for conditional generation by combining self-conditioning and adaptive noise schedule technique. However, these works only focus on generating token-level embeddings for short text generation (less than 128 tokens). In order to adapt diffusion models to longer sequences like summaries, our DiffuSum directly generates summary sentence embeddings with a partial denoising framework. In addition, DiffuSum jointly optimizes the diffusion model with a contrastive sentence encoding module instead of using a static embedding matrix.

heading: 3 Preliminary

heading: 3.1 Continuous Diffusion Models
text: The continuous diffusion model (Ho et al., 2020) is a probabilistic model containing two Markov chains: the forward and the backward process. Forward Process Given a data point sampled from a real-world data distribution x0 ∼ q(x), the forward process gradually corrupts x0 into a standard Gaussian distribution prior xT ∼ N (0, I). Each step of the forward process gradually interpolates Gaussian noise to the sample, represented as:
q(xt+1|xt) = N ( xt+1; √ 1− βtxt, βtI ) , (1)
where βt ∈ (0, 1) adjusts the scale of the variance. Reverse Process The reverse process starts from xT ∼ N (0, I) and learns a parametric distribution pθ (xt−1|xt) to invert the diffusion process of Eq. 1 gradually. Each step of the reverse process is defined as:
pθ (xt−1|xt) = N ( xt−1;µθ (xt, t) , σ 2 θ (t) I ) , (2) where µθ (xt, t) and σ2θ(t) are learnable means and variances predicted by neural networks.
While there exists a tractable variational lowerbound (VLB) on log pθ (x0), Ho et al. (2020) simplifies the loss function of continuous diffusion to:
Lsimple = T∑
t=1
∥∥∥x0 − f̃θ (xt, t) ∥∥∥ 2 , (3)
where f̃θ (xt, t) is the reconstructed x0 at step t.

heading: 3.2 Problem Formulation
text: Given a document with n sentences as D = {sd1, sd2, ..., sdn}, extractive summarization system aims to form a m(m ≪ n) sentences summary S = {ss1, ss2, ..., ssm} by directly extracting sentences from the source document. Most existing work formulates it as sequence labeling and gives each sentence a {0, 1} label, where label 1 indicates that the sentence will be included in summary S. Since extractive ground-truth labels (ORACLE) are not available for human-written gold summary, it is common to use a greedy algorithm to generate an ORACLE consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary following (Nallapati et al., 2017).
In contrast, we propose a summary-level framework with generative model augmentation as shown in Figure 1. Formally, we train a diffusion model with the reverse process pθ(H̃st−1|H̃st ,Hd) to directly generate the desired summary sentence representations H̃st−1 = [h̃ s 1, h̃ s 2, ..., h̃ s m] ∈ Rm×h, where h̃sj is the vector representing the j-th summary sentence at diffusion step t− 1. The model then extracts summary sentences based on the matching between the generated summary sentence representations after T reverse steps H̃s0 = [h̃s1, h̃ s 2, ..., h̃ s m] and the document sentence embeddings Hd = [hd1,h d 2, ...,h d n]. The matching score for the j-th sentence in the output ssj with the document is defined as:
ỹj = softmax(h̃ s j ·Hd T ). (4)
Here we use dot product as similarity measurement and then extract the sentence with the highest matching score for each generated summary sentence.
Our framework operates on the summary level by generating all summary sentence representations simultaneously and adopts continuous diffusion models here for sentence embedding generation.

heading: 4 Method
text: In this section, we introduce the detailed design of DiffuSum. DiffuSum consists of two major modules: a sentence encoding module and a diffusion module, which will be introduced in Section 4.1 and Section 4.2, respectively. After that, we explain how we optimize our model and conduct inference in Section 4.3. The overall model architecture of DiffuSum is also illustarted in Figure 2.

heading: 4.1 Sentence Encoding Module
text: In order to generate desired summary sentence embeddings, we first build a contrastive sentence encoding module to transfer discrete text inputs D = {sd1, sd2, ..., sdn} into continuous vector representations Hd = [hd1,hd2, ...,hdn] ∈ Rn×h, where h is the dimension of the encoded sentence representations.
Specifically, we first obtain the initial representations of sentences Ed = [ed1, e d 2, ..., e d n] with Sentence-BERT (Reimers and Gurevych, 2019). Note that the Sentence-BERT is only used for initial sentence embedding, but is not updated during training. The initial representations are then fed into a stacked transformer layer followed by a projection layer to obtain contextualized sentence representations hdi :
hdi = MLP(Transformer(e d i )). (5)
The same encoding process is applied to the summary sentences S = {ss1, ss2, ..., ssm} to obtain en-
coded summary sentence representations Hs = [hs1,hs2, ...,hsm] ∈ Rm×h. The encoded document sentence representations Hd and summary sentence representations Hs are then concatenated as Hin = Hd∥Hs ∈ R(n+m)×h and will be passed to the diffusion generation module.
To ensure the sentence encoding module produces accurate and distinguishable representations, we introduce a matching loss Lmatch and a multiclass supervised contrastive loss Lcontra to optimize the module, which are defined as follows.: Matching Loss We first introduce a matching loss to ensure an accurate matching between the encoded document and summary sentence representations. Formally, for the j-th encoded summary sentence representation hsj , we generate its encoding matching scores ŷj by computing the dot product with document representations followed by a softmax function:
ŷj = softmax(h s j ·Hd T ). (6)
Then we have the encoding matching loss Lmatch as the cross-entropy between our encoding matching score ŷj and the ground truth extractive summarization label (ORACLE) yj :
Lmatch = m∑
j=1
CrossEntropy (yj , ŷj) . (7)
Contrastive Loss The sentence encoding module also needs to ensure the encoded summary sentence embeddings [hs1,hs2, ...,hsm] are diverse and distinguishable. Thus, we introduce the multi-class supervised contrastive loss (Khosla et al., 2020) to push the summary sentence representation closer
to its corresponding document sentence representation while keeping it away from other sentence embeddings.
Given the sentence contextual representations Hin = [h1,h2, ...,hn+m] ∈ R(n+m)×h, the contrastive label yc is defined as:
ycp =    q, if p ≤ n and sdp = ssq q, if p = n+ q 0, otherwise , (8)
where q ∈ {1, 2, · · · ,m} and ycp is the p-th element of yc. The contrastive loss Lcontra is defined as:
Lcontra = −1 2Nycp − 1
n+m∑
p=1
Lpcontra ,
Lpcontra = n+m∑
q=1;q ̸=p; ycq=y c p
log exp
( hp · hTq /τ ) ∑n+m
k=1;p̸=k exp ( hp · hTk /τ
) , (9)
where Nycp is the total number of sentences in the document that have the same label ycp (Nycp = 2 in our case) and τ is a temperature hyperparameter.
The overall optimizing objective for the sentence encoding module Lse is defined as:
Lse = Lmatch + γLcontra, (10)
where γ is a rescale factor that adjusts the diversity of the sentence representations.

heading: 4.2 Diffusion Generation Module
text: After obtaining the input encoding Hin = Hd∥Hs, we adopt the continuous diffusion model to generate desired summary sentence embeddings conditionally. As described in Section 3.1, our diffusion generation module adds Gaussian noise gradually through the forward process and fits a stacked Transformer to invert the diffusion in the reverse process.
We first perform one-step Markov transition q(x0|Hin) = N ( Hin, β0I ) for the starting state x0 = x d 0∥xs0. Note that the initial Markov transition is applied to both document and summary sentence embeddings.
We then start the forward process by gradually injecting partial noise to summary embeddings xs and leaving document embeddings unchanged xd similar to (Gong et al., 2022). This enables the diffusion model to generate conditionally on the source document. At step t of the forward process q (xst |xt−1), the noised representations is xt:
xt = x d 0||N ( xst ; √ 1− βtxst−1, βtI ) , (11)
where t ∈ {1, 2, · · · , T} for a total of T diffusion steps and ∥ represents concatenation.
Once the partially noised representations are acquired, we conduct the backward process to remove the noise of summary representations given the condition of the sentence representations of the previous step:
pθ ( xst−1|xt ) = N ( xst−1;µθ (xt, t) , σ 2 θ (t) I ) ,
(12)
where µθ (·) and σ2θ (·) are parameterized models (stacked Transformer in our case) to predict the mean and standard variation at diffusion step t− 1. The final output of the diffusion module is the generated summary sentence representations after T reverse steps H̃s0 = [h̃ s 1, h̃ s 2, ..., h̃ s m]. We optimize the diffusion generation module with diffusion loss Ldiffusion defined as:
Ldiffusion = T∑
t=2
∥∥∥x0 − f̃θ (xt, t) ∥∥∥ 2 +
∥∥∥Hin − f̃θ (x1, 1) ∥∥∥ 2 +R (x0) ,
(13)
where f̃θ (xt, t) is the reconstructed x0 at step t and R (x0) is a L-2 regularization term.

heading: 4.3 Optimization and Inference
text: We jointly optimize the sentence encoding module and the diffusion generation module in an end-toend manner. The overall training loss of DiffuSum can be represented as:
L = Lse + ηLdiffusion (14)
where η is a balancing factor of sentence encoding module loss Lse and diffusion generation module loss Ldiffusion.
For inference, DiffuSum first obtains encoded document representations Hd, followed by a onestep Markov transition q(xd0|Hd). Then we random sample m Gaussian noise embeddings as initial summary sentence representations xsT ∈ Rm×h and concatenate it with document representations to get the input xinT = x d 0||xsT for diffusion step T . Then DiffuSum applies the learned reverse process (generation process) to remove the Gaussian noise iteratively and get the output summary sentence representations H̃s0 = [h̃ s 1, h̃ s 2, ..., h̃ s m].
DiffuSum then calculates the matching between the generated summary representation h̃si and the document representation Hd to obtain prediction
label ỹpredi as in Eq. 4. We extract the sentence with the highest score for each generated summary sentence representation and form the summary.

heading: 5 Experiment

heading: 5.1 Experimental Setup
text: Datasets We conduct experiments on three benchmark summarization datasets: CNN/DailyMail, XSum, and PubMed. CNN/DailyMail (Hermann et al., 2015) is the most widely-adopted summarization dataset that contains news articles and corresponding human-written news highlights as summaries. We use the non-anonymized version in this work and follow the common training, validation, and testing splits (287,084/13,367/11,489). XSum (Narayan et al., 2018a) is a one-sentence news summarization dataset with all summaries professionally written by the original authors of the documents. We follow the common training, validation, and testing splits (204,045/11,332/11,334). PubMed (Cohan et al., 2018) is a scientific paper summarization dataset of long documents. We follow the setting in (Zhong et al., 2020) and use the introduction section as the article and the abstract section as the summary. The training/validation/testing split is (83,233/4,946/5,025). The detailed statistics of each dataset are shown in Table 1.
Baselines We compare DiffuSum with strong sentence-level baseline methods: the vanilla Transformer (Vaswani et al., 2017), hierarchical encoder model HIBERT (Zhang et al., 2019), PNBERT (Zhong et al., 2019) that combines LSTMPointer with pre-trained BERT, BERT-based extractive model BERTSum (Liu and Lapata, 2019), and BERTEXT (Bae et al., 2019) that augments BERT with reinforcement learning,.
We also compare DiffuSum with state-of-theart summary-level approaches: contrastive Learning based re-ranking framework COLO (An et al.,
2023) and summary-level two-stage text matching framework MATCHSUM (Zhong et al., 2020).

heading: 5.2 Implementation Details
text: We use Sentence-BERT (Reimers and Gurevych, 2019) checkpoint all-mpnet-base-v2 for initial sentence representations. The dimension of the sentence representations h is set to 128. We use an 8-layer Transformer with 12 attention heads in our sentence encoding module and a 12-layer Transformer with 12 attention heads in the diffusion generation module. The hidden size of the model is set to 768, and temperature τ is set to 0.07. The scaling factors γ and η are set to 0.001 and 100, where γ is searched in the range of [0.0001, 1] and η is searched within the range of [10, 1000]. We set the diffusion steps T to 500. Effects of hyperparameter T and h are discussed in section 6.2.
DiffuSum has a total of 13 million parameters and is optimized with AdamW optimizer (Loshchilov and Hutter, 2017) with a learning rate of 1e−5 and a dropout rate of 0.1. We train the model for 10 epochs and validate the performance by the average of ROUGE-1 and ROUGE-2 F-1 scores on the validation set.
Following the standard setting, we evaluate model performance with ROUGE1 F-1 scores (Lin and Hovy, 2003). Specifically, ROUGE-1/2 scores measure summary informativeness, and the ROUGE-L score measures summary fluency. Single-run results are presented in the following sections with the default random seed of 101.

heading: 5.3 Experiment Results
text: Results on CNN/DailyMail Experimental results on CNN/DailyMail dataset are shown in Table 2. The first block in the table contains the extractive ground truth ORACLE (upper bound) and LEAD that selects the first few sentences as a summary. The second block includes recent strong one-stage extractive baseline methods and our proposed model DiffuSum. The third section includes two-stage baseline methods that pre-select salient sentences. We follow the same setting and show the results of DiffuSum with the same pre-selection for a fair comparison.
According to the results, DiffuSum achieves new state-of-the-art performance under both onestage and two-stage settings, especially a large raise in the ROUGE-2 score. The supreme per-
1ROUGE: https://pypi.org/project/rouge-score/
formance of DiffuSum demonstrates the efficacy of our generation-augmented framework and the great potential to apply diffusion models in text representation generation. It is worth noting that most baseline methods contain pre-trained language model components, but our proposed framework DiffuSum trains Transformers from scratch and contains no pre-trained knowledge. We believe DiffuSum would achieve even better performance if combining pre-trained knowledge, and leave it to future work. We also notice that summary-level methods generally outperform sentence-level methods, proving the need to fill the inherent gap. Results on XSum and PubMed We also evaluate DiffuSum on PubMed and XSum datasets, representing datasets of different domains and summary lengths as shown in Table 3.
For data with longer summaries like PubMed, DiffuSum shows highly strong performance and outperforms state-of-the-art baselines. The strong performance proves that our model can tackle
longer input contexts and complex generations. Our summary-level setting also benefits data with longer summaries by considering summary sentence dependencies.
For data with shorter summaries like XSum, DiffuSum also achieves comparable performance to SOTA approaches, with a significantly higher ROUGE-2 score. Short-summary data tend to be simpler for matching-based methods like MatchSum since the candidate pool is much smaller.
Overall, DiffuSum achieves a comparable or even better performance compared to pre-trained language model-based baseline methods. The results demonstrate the effectiveness of DiffuSum on summarization data with different lengths.

heading: 6 Analysis

heading: 6.1 Ablation Study
text: To understand the strong performance of DiffuSum, we perform an ablation study by removing model components of the sentence encoding module and show the results in Table 4. The second row shows that performance drops when replacing the initial sentence representation from Sentence-BERT to BERT-base encoder(Devlin et al., 2018). The performance drop indicates sentence-level information is necessary for the success of DiffuSum. The third row shows that replacing ORACLE with abstractive reference summaries degrades performance. As for the sentence encoding loss, both the matching loss and contrastive loss benefit the overall model performance according to rows 4 and 5. The matching loss is critical to the model, and the performance drops dramatically by more than 40% without it. The results prove the importance of jointly training a sentence encoder that produces accurate and diverse sentence representations with the generation module..

heading: 6.2 Hyperparameter Sensitivity
text: We also study the influence of our diffusion generation module’s two important hyperparameters: diffusion steps T and the sentence representations dimension h in Table 5. The first row is our best model, and the second block shows the performance of DiffuSum with different sentence representation dimensions. The performance drops by a large margin when setting the dimension to 64, indicating severe information loss when shrinking the sentence dimension too much. The performance also drops a little when the dimension is set to 256, suggesting that a too-large dimension may bring in more noise. The third block shows the influence of diffusion steps, where we find that model performance first increases with more diffusion steps, then starts to decrease and oscillate if steps keep increasing. We argue that the noise injected in the forward pass cannot be fully removed if the steps are too small, and the model will introduce too much noise to recover if the steps are too big.

heading: 6.3 Cross-dataset Evaluation
text: We also notice that DiffuSum shows a strong crossdataset adaptation ability. As shown in Table 6, the model trained on the news domain (CNN/DM and XSum) achieves comparable performance (only 1.57 and 2.69 ROUGE-1 drops) when directly tested on the scientific paper domain. The crossdataset results demonstrate the robustness of our generation-augmented framework and the potential to build a generalized extractive summarization system.

heading: 6.4 Representation Analysis
text: We also analyze the generated sentence representation quality. We apply T-SNE (van der Maaten and Hinton, 2008) to reduce the sentence representation’s dimension to 2 and show the encoded sentence representations as well as the generated summary sentence representations in Figure 3. The blue dots in the figure represent non-summary sentences, and the red dots represent summary sentences (ORACLE) from our sentence encoding module. The green dots are summary sentence representations reconstructed by our diffusion generation module. We can find that most of the ORACLE sentences gather on the right. This finding proves that our contrastive encoder could distinguish ORACLE sentences from non-summary sentences. We also find that the sentence representations generated by the diffusion module (green) are very close to the original summary representations (red). The finding demonstrates that our diffusion generation module is powerful in reconstructing sentence representations from random Gaussian noise.

heading: 7 Conclusions
text: This paper proposes a new paradigm for extractive summarization with generation augmentation. Instead of sequentially labeling sentences, DiffuSum directly generates the desired summary sentence representations with diffusion models and extracts summary sentences based on representation matching. Experimental results on three benchmark datasets prove the effectiveness of DiffuSum. This work is the first attempt to adapt diffusion models for summarization. Future work could explore various ways of applying continuous diffusion models to both extractive and abstractive summarization.
Limitations
Despite the strong performance of DiffuSum, its design still has the following limitations. First, DiffuSum is only designed for extractive summarization, and the diffusion generation module only generates sentence embeddings instead of tokenlevel information. Thus, it is not applicable to the abstractive summarization setting. Moreover, DiffuSum is only tested on single document summarization datasets. How to adapt DiffuSum for multi-document and long document summarization scenarios need further investigation. In addition, our generative model involves multiple steps of noise injection and denoising, compared to discriminator-based extractive systems.

heading: Acknowledgement
text: This work is supported by NSF through grants IIS1763365 and IIS-2106972. We also thank anonymous reviewers for their helpful feedback.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The Limitations Section
7 A2. Did you discuss any potential risks of your work? Our paper proposes a summarization model and we experiment with public datasets. The model will only summarize documents and has no potential risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We only use public available data and models.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.1
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sections 5.2 and 6.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could explore various ways of applying continuous diffusion models to both extractive and abstractive summarization. Limitations
Despite the strong performance of DiffuSum, its design still has the following limitations. First, DiffuSum is only designed for extractive summarization, and the diffusion generation module only generates sentence embeddings instead of tokenlevel information. Thus, it is not applicable to the abstractive summarization setting. Moreover, DiffuSum is only tested on single document summarization datasets. How to adapt DiffuSum for multi-document and long document summarization scenarios need further investigation. In addition, our generative model involves multiple steps of noise injection and denoising, compared to discriminator-based extractive systems