Diffusion Language Model with Query-Document Relevance for Query-Focused Summarization
Authors: Shaoyao Huang
Luozheng Qin
Ziqiang Cao
Venue: EMNLP 2023 Findings (2023)
Keywords: Query-Focused Summarization
Diffusion Language Model
Query-Document Relevance

Abstract: Query-Focused Summarization (QFS) aims to generate summaries from source documents that can answer specific queries. 
Although the QFS task has gained increasing attention recently, its development is constrained by the fact that mainstream QFS models are BART variants,  which are autoregressive and suffer from long-term dependencies and exposure bias. 
To address these problems, we adopt a diffusion language model that performs well in non-autoregressive scenarios to effectively resolve issues related to autoregressive methods.
However, QFS requires guidance from queries to generate adequate summaries, while diffusion language models have limited sensitivity to queries. 
In this paper, we propose QFS-DLM, a non-autoregressive diffusion language model that incorporates query-document fragment relevance and query-document global relevance to enhance the adaptability of QFS tasks.
Firstly, we extract key fragments from documents based on queries and assign higher weights to them, thereby emphasizing crucial and continuous information within the document.
Secondly, we calculate global relevance scores between queries and documents, and then integrate these scores into the model's loss function, enabling the model to prefer high-quality data and distance itself from low-quality data.
Overall, our method achieves state-of-the-art performance on Debatepedia and PubMedQA datasets in ROUGE scores, GPT-4, and human evaluations.

Diffusion Language Model with Query-Document Relevance for Query-Focused Summarization
Shaoyao Huang, Luozheng Qin and Ziqiang Cao"
Institute of Artificial Intelligence, Soochow University
(20224227022, 20222227066) stu..uuda.edu.nn, zqcao @udd.eduu.dc
Abstract
High Query-Document Relevance Score Sample Query: do gasoline economy standards help re- ducing emissions and combat global warming? Document: the most basics function of fuel econ- omy standards that they help the average cars burn less gasoline so emit less # into the atmo- sphere. the net effect is a reduction in green- house gas emission into the atmosphere which lowers the net human contribution to global warming.
Query-Focused Summarization (QFS) aims to generate summaries from source documents that can answer specific queries. Although the QFS task has gained increasing attention re- cently, its development is constrained by the fact that mainstream QFS models BART variants, which are autoregressive and suffer from long-term dependencies and exposure bias. To address these problems, adopt diffusion language model that performs well non-autoregressive scenarios to effectively resolve issues related to autoregressive meth- ods. However, QFS requires guidance from queries to generate adequate summaries, while diffusion language models have limited sen- sitivity to queries. In this paper, we propose QFS-DLM, non-autoregressive diffusion lan- guage model that incorporates query- -ocument fragment relevance and query-document global relevance to enhance the adaptability of QFS tasks. Firstly, we extract key fragments from documents based on queries and assign higher weights to them, thereby emphasizing crucial and continuous information within the docu- ment. Secondly, calculate global relevance scores between queries and documents, and then integrate these scores into the model's loss function, enabling the model prefer high- quality data and distance itself from cuw-quality data. Overall, our method achieves state-of- the-art performance Debatepedia and Pub- MedQA datasets in ROUGE scores, GPT-4, and human evaluations.
Summary: fuel economy requirements reduce emissions fight globally warming.
Low Duery-Document Relevance Score Sample Query: does meeting unfriendly nations helping diplomacy?
Document: michigan democrat congressman dave hoekstra responding the notion of these requests predicted that would be unsupport- able positions for the president of the united states to be putting [ [#1. 1.
Summary: obama closed talks offer rogue lead- ers disinformation opportunities.
Table Examples are taken from training set Debatepedia. In the high relevance score sample, red represents the common keywords shared among the query, document, and summary. In the low relevance score sample, only find the words president and obama, which have slight connection.
to generate query-focused summary in an aoture- gressive(AR) manner (Su et al., 2021; Laskar et al., 2022; Park and Ko, 2022). However, traditional AR models used in QFS have limitations such long-term dependencies (Bengio al., 1994) and exposure bias (Bengio et al., 2015).
1 Introduction
Summarization (QFS) aims to gen- erate summaries that address specific queries by extracting crucial information from source doc- uments (Dang, 2005). Inspired by the signifi- cant generation capabilities of BART (Lewis et al., 2019), researchers manage adopt it into QFS
Recently, the diffusion language model GE- NIE (Lin al., 2023) has achieved remarkable per- formance without operating AR. In the case non- autoregressive (NAR) modeling, GENIE signifi- cantly outperforms BART. This performance gap is
Coorresponding Author.

---

because BART operates fully-NAR, while GENIE adopts an iter-NAR approach. Huang et al. (2022) shows gap called conditional total correlation between AR and fully-NAR learning paradigms because of the lossy decomposition of NAR mod- cls. However, when comparing iter-NAR with AR models, they both can be factorized into an initial prediction term and progressive prediction pro- cess based different contexts (i.e., left-context in AR and full-context in iter-NAR) (Gong al., 2022), and the discrepancy pointed out by Huang et al. (2022) is therefore closed in iter-NAR as- suming sufficient steps. By showing GENIE is an extension of the iter-NAR model, we offer justi- fication that it will not suffer from the conditional total correlation for the same reason.
fragment of the document. Secondly, we incorpo- rate the query-document global relevance into the model's loss function, enabling to prefer high- quality data and distance itself from low-quality data. Finally, to validate the effectiveness of our proposed approach, we conduct extensive experi- ments on popular QFS datasets, including Debate- pedia (Nema et al., 2017) and PubMedQA (Jin et al., 2019).
Our contributions are as follows:
We incorporate query-doeument fragment rel- evance into the diffusion language model, as- signing higher weights to the document frag- ment that the QA model identifies most relevant to the query, enabling the model focus more on the continuous information in the key fragment.
Despite these advantages, two problems hinder the diffusion language model from achieving op- timal performance when applied to the QFS task. Firstly, different fragments in the document have varying degrees of impact on the summary, and treating them equally does not reflect the specificity of query-guided summary generation. Secondly, the query- global relevance indirectly re- flects the data quality (see Table 1), and low-quality data can significantly impair the model's perfor- mance.
We incorporate the query-document global relevance into the model's loss function, abling it to prefer high-quality data and dis- tance itself from low-quality data.
We conduct experiments on the Debatepe- dia and PubMedQA datasets, achieving state- of-the-art performance in terms of ROUGE scores. Furthermore, our approach demon- strates promising results in the GPT-4 and hu- man evaluations.
To address the first problem, we referred to the work of QARTH et al., 2021), QFS-BART method utilizes a QA model to generate the an- swer relevance score for each word the docu- ment and adds this relevance score to the cross- attention. We believe that it is more effective to average the scores for the document fragment that the QA model identifies as most relevant to the query, as the document fragment recognized by the QA model has a high degree of correlation with the summary. To address the second prob- lem, we calculate the query-document global relc- vance score and assign higher/lower loss weights to high-quality/low -quulittyy data, enhancing the posi- tive impact of high-quality data and reducing the impairment of low-quality data.
2 Related Work
2.1 Query-focused Summarization
In the QFS task, evaluating query relevance is critical for generating a summary focused on the query. Early works (Lin et al., 2010; Shen and Li, 2011) focused on extracting query-related sen- tences summaries, while Li and Li (2014) im- proved this procedure by compressing the extracted Semte et al. (2017); Hasselqvist al. (2017) proposed neural abstraction models with additional query attention mechanisms to gener- ate query-focused summaries. Deng et al. (2020a) treated the relationship between query and source sentences as multi-hop reasoning process and generated summaries by integrating information from different reasoning steps. Additionally, searchers utilized question-answer models find potential query-related evidence in QFS. Xu and Lapata (2020) employed question-ansser models for ranking sentence-level or paragraph-lews an- swer evidence. Su et al. (2021) incorporated answer
In this study, we propose QFS-DLM !, a dif- fusion language model that incorporates query- document relevance to generate high-quality sum- maries. Firstly, we incorporate the query-document fragment relevance to make the model pay more attention to continuous information within key
'The code is available at: https://github com/ ShaoyaoHuang/QFS-oa DFDL//////////////aaaii

---

Symbols
Deseription
fragment relevance score sequence
time step
B. E 10, as different variance scales
continuous latent representation of the gold summary
Hs
souree text representation.
Q1
glohal relevance of the i-th sample between query the document
Emb(yy
embedding of gold summary
Mg (-1
predicted noise
actual noise
Z0
output of the demoising architecture
covariance matrix Gaussian distribution
Table 2: Parameter meaning explanation.
relevance scores generated by question-answer model as explicit tine-grained query relevance into a transformer- based abstract summarization model. Yuan et al. (2022) integrated task knowledge from text summarization and question answering into a well-designed prefix and applied it. Park and Ko (2022) incorporated a graph attention mecha- nism that calculated the relevance of word nodes to query nodes. Overall, query-document relevance is crucial for query-focusee summarization, and incorporate it into our model.
3 Methodology
In this section, we present our approach to incorpo- rating query-document relevance into the diffusion language model. First, we describe the method of incorporating query-document fragment relevance. Then, introduce the query-documen global rele- vance, which is only used during the training phase, and our model architecture as shown in Figure 1.
Based on the performance of the diffusion lan- guage model GENIE (Lin et al., 2022) in the general text summarization, we choose it as our base model. The GENIE has undergone extensive pretraining and adopts the sequence-to-sequenie framework. Unlike the traditional autoregressive text generation paradigm that generates one token at a time, GENIE parallelly outputs embedding sequences at each denoising step, making it a non- autoregressive model. our parameters is presented Table 2.
2.2 Diffusion Language Models
With the great success of diffusion models in com- puter vision, some researchers have started explor- ing the application of diffusion models in text gen- eration. Hoogeboom al. (2021) introduced poly- nomial diffusion for character-level text genera- tion, applying forward classification noise with a Markov transition matrix. Austin et al. (2021) gen- eralized discrete text diffusion models by introduc- ing an absorbing state IMASK]. However, discrete diffusion models may be influenced by the scale of one-hot row vectors and only generate text sam- ples unconditionally in the discrete space. Li et al. (2022) proposed novel continuous latent repre- sentation language model with different mapping functions that connect text's discrete and continu- ous spaces. Gong et al. (2022) employed an end-to- end, classifier--ree approach to guide the diffusion conditional generation and provided a comparative analysis between the diffusion, autoregressive, and non-autoregressivv models. Lin et al. (2023) pro- posed large-scale pre-trained diffusion language model, which can generate high-quality texts for sequence-to-seqqeence tasks. Compared to previous works, we focus on improving the adaptability of the diffusion language model for the QFS task.
3.1 In recent years, neural models (Yang al., 2019a; Su al., 2019) have shown remarkable achieve- ments in QA tasks. In order to apply QA mod- els to the QFS task, we use HLTC-MRQA (Su et al., 2019) to generate the answer relevance score for each word in context. The reason for choos- ing HLTC-MRQA is twofold: 1) it shows robust generalization and transferring ability on differ- ent datasets, and 2) the model shows great perfor- mance in QA tasks and significantly outperforms the BERT-large baseline by a large margin. The HLTC-MRQA is introduced as follows.
Fragment Relevance
Based on XLNet (Yang et al., 2019b), HLTC- MRQA is fine-tuned on multiple QA datasets.
Given context that contains words, the model
hhttps: thub. com/mi http:///w.nnctetee//////ee/ee/ee master/GENIE

---

Generate summary
Training phase
Inference phase
Desodentawanddrg
Fragment Relevance
Lieer
Nor
global
Ferward
relevance
estimator
Add Nom
Add & Nom
Feod
Muii head
Donotse
Add Nom
Add Nom
DellAddstoon
Stalus step
Fnooiing
Intal Carosonn Nolk
Axcument
Figure 1: The framework of QFS-DLMV For query- query-document fragment relevance, we use QA model extract key fragment from the document and add weight to it in cross- aass-tatention During the training process, calculate the query-document global relevance scores and incorporate them into the model's loss function.
outputs distribution E (0, 1) for each word's probability of being the start word of the answer and probability distribution e E (0, 1) for being the end word of the answer. Su et al. (2021) ap- proach involves adding the scores incorporating this score into the hidden state of the corresponding words in the model's cross-attention. However, this approach can disrupt the continuity information of important fragments and the original task of the QA model is to separately identify the highest scores for and e, considering the middle part between and as the answer (if the position is after e, then considered that there is no fragment to answer the query). To address these two issues, we introduce an additional weighting mechanism specifically for the fragments between and To achieve this, we generate a relevance score sequence F, represented as follows:
layers of Transformers. The decoder stably generates high-quality text iteratively, and cross- attention (Vaswani al., 2017) connections are established between the encoder and decoder. To enable the decoder to better focus on crucial and contiguous information within the key fragments of the document, we increase the weighting of key fragments in the cross cross-attention mechanism. The original formula for cross-attention is follows:
and e and
QK" Vd
Atlention(( K, V) = soflmar
V
where the summary is input into the decoder to obtain the representation Q, where d represents the dimension of Q. In order to enable interaction between the query and document information in the encoder, we concatenate the query and document as input the encoder obtain K and V. Then, in our method, we incorporate the relevance score F' into the cross-atie attention,, as follows:
F [1,
1]
(1)
where represents the position corresponding to the highest score for and represents the posi- tion corresponding to the highest score for e. Next, we incorporate this relevance score into the model. The encoder and decoder of the dif- fusion language model are both composed of
Q(K + F)J
Attention (Q. = soot
V
(3)
Vd
By increasing the weights, the contiguous frag- ments corresponding to the query have a more sig-
nificant impact on the generation the summary

---

3.2 Query-Document Global Relevance
I +++ 1, the latent variable 1 is only determined by the 24 time t, expressed as:
To apply query-document global relevance to the QFS task, we Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) to generate query- document relevance scores. There are two rea- sons for choosing DPR: (I) demonstrates ro- bust generalization and transferability across dif- ferent datasets, and (2) the model performs well in question-answer tasks and is widely used on Hugging Face (with 160k downloads). DPR can effectively retrieve the most relevant documents from large corpus in relation to search query. We the 3PR model calculate the relevance score between the query and the document. This is accomplished through the following equation:
a(*+11 as) N[ vI 1at Bet1)
where 8411 E 1), as t increases, 3y becomes closer to standard Gaussian noise N (2;; 0,I, I).
We then apply the forward diffusion process (equation 7) to obtain the state ž, at step as a function of zo. using the following equation:
g(zr(z0) N (a6 Vasao vI v1-aa T.I)
where 0₁ = 1 St. and 0r = II Og. During the training phase, randomly select step to compute I2 and then utilize denois- ing architecture that predicts noise through cross- attention with the source text representation Hs. The equation below provides the predicted noise:
D.Q ||||| I@@11
DPRst (D,Q)
(4)
3. 2o Lo
(21
where D is the embedding vector of the input doc- ument, and Q is the embedding vector of the input query.
Vou
VI VI- 8
where 29 represents the output of the denoising architecture and represents its parameters.
For direct interaction between the query and document within the bidirectional self- attention, format the source text as "[CT.S] document [SEP] query". The coder computes representations for all tokens and outputs hidden vectors, enabling the source text to be represented as Hlg:
The training objective is to minimize the squared error between the predicted noise v/g and actual noise Pc-1, as well the reconstruction error be- tween To and the gold summary embedding, shown in the equation:
C22s
E ((par)l))
110
11=
(10)
H. = Encoder([CLL]] document [SEP] query) (5)
t=1
+ Emb(y) 11111 lo (2) zo)
where the encoder refers to a 6-layer transformer.
where log Pe mD) represents mapping the contin- uous latent variable æn into the discrete space token of gold summary y-
Training Phase To train the diffusion language model for the QFS task, we first need to transform the gold summary into continuous state zo- We achieve this by applying the embedding function and perturbing it with Gaussian noise. This process be represented by the equation:
Finally, we incorporate query mucrymdo global relevance as shown in the equation:
Finaa ΣB e Ci
(11)
Ele e..
((zoly) (xo: Emb(y), B)
(6)
where B represents the number of samples in a batch, R2 denotes the relevance score the i-th sample between the query and the document, and C) represents the loss of the i-th sample in the diffusion language model.
where Emb(y) represents the embedding of the gold summary, and So represents the scaling of the variance at time step 0.
Given a gold summary distribution øo, gradually adds Gaussian noises to xo according to a variance scheduler B1,..., 3r. corrupting x0 into a standard Gaussian noise
Inference Phase To generate summaries, we start from the last step T. and sample a state Pr from a standard Gaussian distribution. Then, we perform denoising, iteratively generate noise from the previous step, and subtract that noise from
I). At the time step
https: htt.///ggggnggg.eeeegg wtt.ce.....cceeoooeeeeeeoeeeeeee ca/ at//aacboook/dpr-ctte encoder- mul tiettisttabaeee

---

Dehatepedia ROUGE-2
PubMedQA ROUGE-2
Models
ROUGE-1
ROUGE-L
ROUGE-1
ROUGE-L
Original results
ChatGPT one-shot
22.7
6.68
18.8
34.8
12.5
23.7
SD2
41.3
18.8
40.4
32.3
10.5
26.0
OR-BERTSUM-TL.'
58.0
45.2
57.1
MSG
37.2
14.8
30.2
QFS-BART
59.0
44.6
57.4
38.3
16.4
29.1
PreQFASt
59.3
45.6
58.2
QSG BART!
64.9
52.3
63.3
38.4
17.0
29.8
Reproduction on GENIE
QSG BART
53.7
33.9
51.8
34.1
11.6
25.0
QUERYSUM
65.1
51.4
63.0
44.9
18.7
33.4
Prefix-merging
65.4
51.5
63.8
45.3
19.4
33.2
BART
65.9
51.8
64.2
45.5
19.6
33.0
GENIE zero-shot
29.8
9.7
24.9
29.4
7.3
20.1
GENIE finetune
65.4
51.7
63,9
45.2
19.6
33.6
QFS-DLM fragment relevance
66.5
52.4
64.9
45.6
20.1
33.9
QFS- DL.M global relevance
66.2
52.5
64.8
46.3
20.9
34.8
QFS-DLM
67.3
53.1
65.7
46.5
21.0
35.1
Table 3: ROUGE-FI scores for Debatepedia and PubMedQA datasets. QFS-DLM fragment//lobal relevance indicates that only use fragment/global relevance separately, while QFS-DLM indicates that types of relevance simultaneously. denotes results from the original paper.
Rank
Datasets
Summary Sources
fuency
faithfulness
coverage
overall
QFS-BART
3
Debatepedia
GENIE
3
2
QFS-DLM
2
1
QFS-BART
2
3
PubModQA
GENIR
3
2
QFS-DLM
Table 4: The evaluation of summary quality using GPT-4, We feed the query, document, and shuffled summaries inputs GPT-4 to determine which summary performs the best. Subsequently, we rank the number of summaries assessed the best for each specific metric.
Debatepedia valid
PubMedQA
train
valid
Samples
12K
0.7K
21K
Avg Len Query
11.6
11.7
11.3
15.3
15.4
15.3
Avg Len Doc
72.9
71.6
200
Avg Len Sum
9.9
9.8
37.6
37.6
Table 5: Statistics of datasets.
the current state to obtain Tt-1- Upon reaching 0, we apply the clamping trick (Li al., 2022) to replace the value of Zo with its nearest word embedding and then decode discrete tokens from
PubMedQA is biomedical abstractive question- answer dataset that contains 210K samples. We fol- low the standard splits used in the original paper's methods. Table 5 presents the statistical informa- tion of these datasets.
æ0-
4 Experiment
4.2 Training Details
Our setup is largely consistent with GENIE (Lin et al., 2022). Specifically, set the latent vari- able dimension 768 and the embedding variable dimension 128. During training, we utilize the Adam optimizer (Kingma and Ba, 2015) with a learning rate of lc-4 and set the batch size to 64. We train our model using four 24G A5000 GPUs.
4.1 Datasets
We evaluate our proposed model two QFS datasets: Debatepedia (Nema al., 2017) and Pub- MedQA (Jin al., 2019). Dehatepedia contains 12.7K samples crawled from 663 debates 53 diverse categories in an encyclopedia of debates.

---

Input:
4.4 Evaluation Metries
Document: republicans argued in a june 2009 letter to president obama " public insurance J would be a federal government takeover of our healthcare system taking decisions out of the hands of doctors and patients and placing them in the hands of a washington bureaucracy.
We use ROUGE metrics (Lin, 2004), which include Rouge-1, Rouge-2, and Rouge-L, as evaluation metrics.
Additionally, we employ GPT-4 (Ouyang et al., 2022) and humans to assess the quality of the sum- maries based on four aspects: fluency, faithfulness, coverage, and overall performance. Prompt tem- plate examples of GPT-4 evaluation are shown in Table 6, where the order of the summaries gen- erated randomly. When using other metrics, the terms overall can be replaced. For the human eval- uation of summaries quality, in order to ensure ob- jectivity, we shuffled the summaries and asked five text summarization researchers to score the sum- maries. Each perspective was scored on a scale of I to 5 points, with higher score indicating better performance.
Query: is it important to give citizens the choice of public insurance option ?
From overall perspective, which summary is best? If the first one is the best, output "I"; if the second one the best, output "2"; if the third one is the best, output "3":
Summary 1: public insurance would amounts to a federal takeover of health care Summary 2: public insurance takes health care decisions out of the hands of patients.
Summary 3: public insurance could give citizens out of health bureaucracy.
Output:
4.5 Results
2
The experimental results for De- batepedia and PubMedQA are presented in Table 3, with the highest scores highlighted in bold. Our proposed model achieves significant improvements compared to previous works. Although our model outperforms the comparison models in metries on both datasets, is evident that the performance on PubMedQA is better than that on Debatepe- dia. This may be because PubMedQA has much longer summaries (as shown in Table 5), and the autorrrrsrreeivee diffusson language model can alleviate the constraints of gold summary length. We analyzed each method that reproduction on the GENIE separately: (1) QSG BART's repro- duction yielded poor results, as its auxiliary task significantly disrupted the encoder of the diffusion language model.
ROUGE Metrics:
Table 6: An example of GPT-4 evaluation. Summary refers to QFS-BART summary, Summary 2 is QFS- DLM summary, and Summary is GENIE summary. QFS-DLM summary is smoother than QFS-BART and can correct errors made by GENIE.
4.3 Comparative Methods
In this work, we compare QFS-DLM with several other models. These models include: (1) Chat- GPT One-Shot, which utilizes CharGPT to gener- ate summaries based on a provided example. (2) SD2 (Nema et al., 2017), which incorporates a query attention model and diversity-based at- tention model in the aenodding-atteniooddeeeddd paradigm. (3) QR-BERTSUM-IL (Laskar al., 2020), which presents query-relevance technique with the Transformer- BERTSUM model. (4) MSG (Deng et al., 2020b), which utilizes the rel- evance between the query and the sentences and the mutual relationships computed by the atten- tion mechanism. (5) QFS-BART (Su et al., 2021), which incorporates document and answer relevance. (6) PreQFAS (Laskar et al., 2022), which uses the CNN/DM dataset for data augmentation. (7) QSG BART (Park and Ko, 2022), which incorporates graph attention mechanism that calculates the relevance of word nodes query nodes. (8) GE- NIE (Lin et al., 2023), a large-scale pretrained dif- fusion language model.
(2) QUERYSUM (Xu and La- pata, 2020) slightly reduced the effectiveness of the diffusion language model because sentence or- dering single-document datasets is not as crit- ical as in multi-document datasets, and reorder- ing can disrupt continuity between sentences. (3) Prefix-merging (Yuan et al., 2022) had minimal impact on the diffusion language model since it is designed for few-shot scenarios. (4) QFS-BART improved performance by focusing on keywords. Our faagment-relevane method better emphasized contiguous information, resulting in more notice- able performance gains. Overall, our approach exhibits greater adaptability and performs better on the QFS task than other methods.

---

Datasets
summary sources
query-focusing
fluency
faithfulness
coverage
overall
QFS-BART
3.52
3.67
Debatepedia
GENIE
3.56
QFS-DLM
4.07
4.16
4.13
QFS-BART
3.12
4.03
3,27
3.24
3.28
PubMedQA
GENIE
2.95
3.85
3,46
3,47
3.54
QFS-DLM
3.68
4.21
3,94
4.01
3.87
Table Human evaluation Debatepedia and PubMedQA datasets.
Datasets
query-document relevance
high quality
low quality
high relevance
89.3%
10.7%
Debatepedia
low relevance
37%
63%
high relevance
92.3%
7.7%
PubMedQA
low relevance
53.7%
46.3%
Table 8: Correlation between query-document global relevance and sample quality.
Query: Are poor means to protecting endangered species?
advantage over QFS-BART in generating longer summaries. This advantage is attributed to the non-autoregressivve approach, which effectively ad- dresses long-term dependencies and exposure bias issues. Furthermore, the summaries generated by our model outperform GENIE in terms of all met- rics, illustrating that incorporating query-document relevance in the diffusion language model posi- tively impacts QFS.
Document: Visitors to Zoos may raise their awareness of endangered species by being di- rectly exposed to them.
ChatGPT Summary: Zoos can actually be a valu- able means of protecting endangered species. By allowing visitors directly observe endan- gered species, zoos raise awareness and foster a connection between people and these animals, This can lead to increased support for conser- vation efforts and funding for protecting these species in the wild. Overall, zoos can play pos- itive role in educating the public and contribut- ing to the conservation of endangered species. QFS-BART Summary: Zoos provide children with an exposure to endangered species. QFS-DLM Summary: Zoos help securing endan- gered species.
Human evaluation: As shown in Table 7, we can see that our model performs slightly better than the comparison methods on the Debatepedia and Pub- MedQA datasets. The improvement in the query- focusing metrics indicates that approach better cover the information required by the query.
4.6
Case Study
Case studies are conducted to understand the model's performance better. As shown in Table 9, summaries are generated by the proposed method and compared methods for an example chosen from the Debatepedia test set. Although QFS-BART's summary includes the document's content, it can- not answer the query. ChatGPT's summary too long. In contrast, our summary expresses important information from the document and answers the query like the gold summary.
Gold Summary: Zoos raise awareness of endangered species.
Table 9: An example taken from Dehatepedia test set. highlight indicates key phrases that reflect the purpose. The generated summary from QFS-DLM is almost same the gold summary.
GPT-4 Evaluation: We rank the summaries of QFS-BART, GENIE, and QFS-DLM using GPT- 4 on the Debatepedia and PubMedQA datasets. The results are shown in Table 4. On Debatepe- dia, model falls slightly behind QFS- QFS-BART in terms of fluency, which may be attributed to the non-autoregressive approach leading to cer- tain limitations language fluency. However, on PubMcdQA, our model outperforms QFS-BART in all metrics, indicating that our model has am
4.7 Query- Query-Doeument Global Relevance and Sample Quality
Regarding whether the performance of query- document global relevance can reflect the quality of samples, we extracted the top 100 and bottom 100 samples based query-document global rele- vance scores from the training sets of Debatepedia and PubMedQA datasets, and named the subsets as "high-relevance" and "lew-celevance", respectively.

---

Loss Comparison
4.9 Query-Document Global Relevance Analysis
0.0350
00325
Gente
6.0300
We calculate the global relevance score R between the query and the document. Let Rmaz1 denote the highest score and Rmin1 denote the lowest score for Debatepedia, and let Rmor2 denote the highest score and Rmin2 denote the lowest score for Pub- MedQA. We then normalize the scores using the following equation:
0.0250
0.225
0.0000
0.175
0.150
20000
40000
60000
80000
100000 120000 140000
Step
R - min Hmmnny Rmanz)
Figure 2: Convergence speed of QFS- DLM and GENIE.
R
(12)
max Rmmaz Ronaz) (iiss Ramina)
Relevance
After normalization, we utilize the scipy library to compute the kernel density estimation of the nor- malized scores, and the results are illustrated in Fig- ure 3. It be observed that the query-document global relevance scores the PubMedQA dataset are higher. This disparity could be one reason for the greater improvement observed in PubMedQA compared to Debatepedia when the model incorpo- rates query-document global relevance.
Debatepedia PubMed(A
Donay
0.0
0.2
0.4
0.6
0.8
1.0
Conclusion
Relevance Score
In this work, we propose QFS-DLM, a diffusion language model that incorporates query-document relevance. Firstly, extract key fragments from documents based queries and assign higher weights to them, thereby emphasizing crucial and continuous information within the document.
Figure 3: Kernel Density Estimation of Query- Document Relevance Score.
Three text summarization researchers were asked to classify the sample quality "high-quality" or "low-quality", and the results shown in Table which query-document global relevance could in- deed reflect the quality of samples. This finding not only provides a qualitative analysis but also guantitative experimental support.
Then, we calculate global relevance scores between queries and documents, and then integrate them into the model's loss function, enabling the model to prefer high-quality data and distance itself from low-quality data. This approach generates sum- maries that more focused query-related in- formation, improving the performance and adapt- ability of the diffusion language model for the QFS task. Experimental results demonstrate that our model achieves state-of-the-art performance on the Debatepedia and PubMedQA datasets.
4.8 Impact on Convergence Speed
We performed experiments on the Debatepedia dataset, collected the loss iteration data of the GE- NIE and our QFS-DLM models, and plotted their convergence curves. As shown Table 2, our model converges faster.
Both models showed a relatively stable conver- gence process. However, when GENIE was close to complete convergence, the impact of low-quality data on convergence became significant, resulting in considerable oscillations. On the other hand, QFS-DLM reduced the effect of low-quality data and was able to converge continuously without producing considerable oscillations, resulting in a quicker convergence rate.
Limitations
Transformer-baat based summarization models, includ- ing diffusion language models, may encounter chal- lenges in efficiently processing long documents. Long documents often contain substantial amount of information, which makes it difficult to generate concise and accurate summaries while maintaining

---

the focus on the query. Furthermore, the train- ing and testing time of diffusion language models far exceeds that of the BART model, which also restricts its development. Addressing these limita- tions and conducting further research can lead to a more comprchensive and robust solution.
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, and Minlie Huang. 2022. On the learning of non-autoregressive transformers.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-JJCNLP) pages 2567- 2577, Hong Kong, China. Association for Computa- tional Linguistics.
Acknowledgements
The authors would like to thank colleagues Qi Lv and Jun Gao for the discussions. The work de- scribed in this paper was supported by the Na- tional Natural Science Foundation of China (NSFC 621 (2106165).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Fdunnv, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7769-6781, Online. Association for Computational Linguistics.
References
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces. CoRR, abs/2107.03006.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020. Query focused abstractive summariza- tion via incorporating query relevance and transfer learning with transformer models. In Advances in Artificial Intelligence: 33rd Canadian Conference on Artificial Intelligence, Canadian Al 2020, Oitawa, ON, Canada, May 13-15, 2020, Proceedings 33, pages 342-348 Springer.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradi- ent descent is difficult. IEEE transactions on neural networks, 5(2):157-166.
Hoa Trang Dang. 2005. Overview of due 2005 (draft). In Proceedings of Document Understanding Confer-
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain adaptation with pre-trained transformers for query-focused ab- stractive text summarization. Computational Linguis- tics, 48(2):279-320.
ences.
Yang Deng, Wenxuan Zhang, and Wai Lam. 2020a. Multi-hop inference for question-driven summariza- tion. arxiv preprint arXiv:2010.03738.
Yang Deng, Wenxuan Zhang, and Wai Lam. 2020b. Multi-hop inference for question-driven summariza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6734-6744, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising
pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.1344
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to se- quence text generation with diffusion models. arXiv preprint 200.0:221..08933.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022. Diffusion- Im improves controllable text generation. In Ad- vances Neural Information Processing Systems. volume 35, pages 4328 4343. Curran Associates. Inc.
Johan Hasselqvist, Niklas Helmertz, and Mikael Kágebäck. 2017. Query-based abstractive sum- marization using neural networks. arXiv preprint arXiv:/7/2.06/00.
Yanran Li and Sujian Li. 2014. Query- focused multi- document summarization: Combining a topic model with graph-based semi-supervised learning. In Pro- ceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technicul Papers, pages 1197-1207
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, 2021. Argmax flows and multinomial diffusion: Learning categorical dis- tributions.

---

Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances neural information pro- cessing systems, pages 3998-6008.
Jimmy Lin, Nitin Madnani, and Bonnie Dorr. 2010. Putting the user in the loop: interactive maximal marginal relevance for query-focused summarization. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the Association for Computational Linguistics, pages 305-308.
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fine query focused multi-document summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 3632-3645, Online. Association for Computa- tional Linguistics.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Weizhu Chen, and Nan Duan. 2022. Genie: Large scale pre-training for text generation with diffusion model. arXiv preprint arXiv2211.111685.
Wei Yang. Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019 9a. End-to-end open-domain question answering with bertserini. arXiv preprint urXiv:1902.01718.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen, 2023. Text generation with diffusion language mod- els: A pre-training approach with continuous para- graph denoise.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoc V Le. 2019b. Xinet: Generalized autoregressive pretrain- ing for language understanding. arXiv preprint 1.Xiv:1906.0822
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven atten- tion model for query-based abstractive summariza- tion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume Long Papers), pages 1063-1072, Vancouver, Canada, Association for Computational Linguistics.
Ruifeng Yuan, Zili Wang, Ziqiang Cao, and Wenjie Li. 2022. Few-shot query-focused summarization with prefix-merging. Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3704-3714 Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744
Choongwon Park and Youngjoong Ko. 2022. Qsg trans- former: Transformer with query-attentive semantic graph for query-focused summarization. In Proceed- ings of the 45th International ACM SIGIR Confer- ence Research and Development in Information Retrieval, SIGIR '22, page 2589-2594 2594, New York, NY, USA. Association for Computing Machinery.
Chao Shen and Tao Li. 2011. Learning to rank for query-focused multi-document summarization. In 2011 IEEE I Itth International Conference on Data Mining, pages 626-634. IEFE.
Dan Su, Yan Xu, Genta Indra Winata, Peng Xu, Hyeondey Kim, Zihan Liu, Pascale Fung. 2019. Generalizing question answering system with pre- trained language model fine-tuning. In Proceedings of the 2nd Workshop on Machine Reading for Ques- tion Answering, pages 203-211.
Dan Su. Tiezheng Yu, and Pascale Fung. 2021. Im- prove query focused abstractive summarization by incorporating answer relevance. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 3124-3131 3131, Online. Association for Computational Linguistics.


Reviews:
[{"review_id": "GGIomgIkZW", "reviewer": "Reviewer_Feux", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for dedicating time from your busy schedule to assess our paper and offer multiple constructive insights meticulously.\nWe will respond to your comment in the following three aspects:\n\n### **1. incremental work**\nThank you for mentioning this important question.\nIn general, **we adapted the diffusion language model for QFS, not a direct application**, and we have provided a description of this in the Introduction section of our paper.\nSpecifically, the original diffusion language model lacked sensitivity to queries. \nFurthermore, the quality of the QFS data varies, where query-document relevance can reflect sample quality. \nThese two motivations inspired us to modify the original model by incorporating query-document relevance.\nWe calculate the query-document relevance score and assign higher/lower loss weights to high-quality/low-quality data, enhancing the  positive impact of high-quality data and reducing the impairment of low-quality data, which guides the model to prefer samples with high query-document relevance to capture query-required messages.\nExperimental results demonstrate that our approach improves summarization performance. \nAdditionally, over the past few days, we conducted human evaluations, confirming that samples with higher query-document relevance indeed exhibit better quality.\n\n### **2. limited reproducibility**\nWe appreciate your pointing out the re-clarification need for parameter definition and code release. \nConsidering that, we will use a symbol table to describe the meaning of each parameter definition and release our code in the paper's final version.\nFor the issue of loose definitions of parameters, we found that $\\alpha_{t}$ and $\\alpha_{i}$  are somewhat confusing, and we rename $\\alpha_{i}$ parameter as $R_{i}$. \nAfter the revisions, our parameter table is presented below.\nSymbols | Description\n--------|------------\n$t$       | time step\n$\\beta_{t}$     | $\\beta_{t}$ ∈ (0, 1) as different variance scales\n$x_{t}$      | continuous latent representation of the gold summary\n$H_{s}$     | source text representation\n$\\alpha_{t}$    |  $\\alpha_{t} = 1 - \\beta_{t}$\n$R_{i}$      | relevance score of the $i$-th sample between the query and the document\n$Emb(y)$ | embedding of the gold summary\n$u^{t-1}_{\\theta}$ | predicted noise\n$\\hat{\\mu}_{t-1}$ | actual noise\n$z_{\\theta}$   | output of the denoising architecture\n$I$       | covariance matrix of Gaussian distribution\n\n### **3. limited analysis of results**\nThank you for pointing out this drawback. \nWe will revise the experimental results analysis section and add more experiments to demonstrate the effectiveness of our method, and provide a more detailed explanation for the examples in the paper. \n\nIn order to better understand our model results and conduct a more in-depth qualitative analysis, the following content is mainly divided into three aspects: Firstly, we briefly explain the experimental results. \nSecondly, we supplement with quantitative experiments to demonstrate that **query-document relevance can indeed reflect sample quality.** \nFinally, we use human evaluation to illustrate the characteristics of our model summaries, and the results show that **our approach outperforms the comparison methods across all metrics.**\n\n#### **3.1 Explanation of the experimental results** \n\n(1) compared with BART variants, QFS-DLM demonstrated more significant results on the long summary dataset. \nThis advantage can be attributed to our non-autoregressive approach, which effectively addresses issues related to long-term dependencies and exposure bias. \n(2) By leveraging query relevance, our model is equipped with more refined and effective information, leading to the generation of superior summaries compared to GENIE.\n\n#### **3.2 Correlation between query-document relevance and sample quality**\n\nRegarding whether the performance of query-document relevance can reflect the quality of samples, we extracted the top 100 and bottom 100 samples based on query-document relevance scores from the training sets of Debatepedia and PubMedQA, and named the subsets as \"high-relevance\" and \"low-relevance\", respectively. \nThree  text summarization researchers were asked to classify the sample quality as \"high-quality\" or \"low-quality\", and the results showed below that query-document relevance could indeed reflect the quality of samples.\nThis finding not only provides a qualitative analysis but also quantitative experimental support.\n\n#### Debatepedia dataset:\n\n|                | high-quality | low-quality |\n|----------------|--------------|-------------|\n| high-relevance sample | 89.3%        | 10.7%       |\n| low-relevance sample  | 37.0%        | 63.0%       |\n\n#### PubMedQA dataset:\n\n|                | high-quality| low-quality|\n|----------------|-------|-------|\n| high-relevance sample | 92.3% | 7.7%  |\n| low-relevance sample  | 53.7% | 46.3% |\n\n#### **3.3 Human evaluation**\nIn order to obtain statistically significant information, we have conducted a human evaluation in the past few days.\n\nThe results are shown in the table below (maximum score of 5 points). \nFrom the evaluation results, we can see that our model performs better than the comparison methods on the Debatepedia and PubMedQA datasets.  It is because we enhance the positive impact of high query-relevance data and reduce the impairment of low query-relevance data, allowing the model more biased towards learning from high-quality data. \n\n#### Debatepedia dataset (Using 1000 samples from the test set):\n\n| Summary Sources | Query-Focusing | Fluency | Faithfulness | Coverage | Overall |\n|-----------------|----------------|---------|--------------|----------|---------|\n| QFS-BART        | 3.52           | 4.23    | 3.50         | 3.67     | 3.59    |\n| GENIE           | 3.41           | 4.14    | 3.56         | 3.78     | 3.71    |\n| QFS-DLM         | 4.07           | 4.53    | 4.09         | 4.16     | 4.13    |\n\n#### PubMedQA dataset (Randomly selecting 1000 samples):\n\n| Summary Sources | Query-Focusing | Fluency | Faithfulness | Coverage | Overall |\n|-----------------|----------------|---------|--------------|----------|---------|\n| QFS-BART        | 3.12           | 4.03    | 3.27         | 3.24     | 3.28    |\n| GENIE           | 2.95           | 3.85    | 3.46         | 3.47     | 3.54    |\n| QFS-DLM         | 3.68           | 4.21    | 3.94         | 4.01     | 3.87    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* the proposed method allows incorporating query-document relevance into the diffusion language model, which improves its performance\n* the proposed method outperform SOTA approaches", "reasons_to_reject": "* incremental work: the proposed method largely adopts GENIE, a recently proposed diffusion language model\n* limited reproducibility: the code of the proposed method is not released. Coupled with loose definitions of many parameters, this  significantly limits reproducibility\n* limited analysis of results: despite superior performance, authors perform little analysis to identify its reasons. Qualitative analysis is boiled down to anecdotal examples, many of which are confusing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes QFS-DLM, a non-autoregressive diffusion language model for query-focused summarization that incorporates query-document relevance and avoids the issues of long-term dependencies and exposure bias that are inherent to autoregressive LLMs, such as BART.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tMazAIk7CO", "reviewer": "Reviewer_ds5i", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to convey our deepest appreciation to you with wholehearted gratitude.\nThanks to your thoroughly considered and insightful suggestions, we are able to further polish our paper by conducting necessary experiments and clarifying unclear aspects, which significantly improve the credibility, logic, and structure of our paper.\n\nThank you for reminding us of the significance of human evaluation.\nAs a consequence, we have conducted a human evaluation and the results showed that **our summaries were comprehensible to humans.** \nSpecifically, we asked five text summarization researchers to  score the summaries from the perspective of query-focusing (covering the required key information of the query), fluency, faithfulness, coverage, and overall.\nEach perspective was scored on a scale of 1 to 5 points, with a higher score indicating better performance.\n\nAs shown in the table below, the results indicate that our model outperforms the comparison methods across all metrics. \nThis is because our iter-NAR diffusion model can effectively reduce loss decomposition compared to fully-NAR based on BART variants. \nWith enough iterations, the generated summaries become sufficiently stable, and our model is equipped with more refined and effective information, resulting in more fluent summary than GENIE. \n#### Debatepedia dataset (Using 1000 samples from the test set):\n| Summary Sources | Query-Focusing | Fluency | Faithfulness | Coverage | Overall |\n|-----------------|----------------|---------|--------------|----------|---------|\n| QFS-BART        | 3.52           | 4.23    | 3.50         | 3.67     | 3.59    |\n| GENIE           | 3.41           | 4.14    | 3.56         | 3.78     | 3.71    |\n| QFS-DLM         | 4.07           | 4.53    | 4.09         | 4.16     | 4.13    |\n\n#### PubMedQA dataset (Randomly selecting 1000 samples from the test set):\n\n| Summary Sources | Query-Focusing | Fluency | Faithfulness | Coverage | Overall |\n|-----------------|----------------|---------|--------------|----------|---------|\n| QFS-BART        | 3.12           | 4.03    | 3.27         | 3.24     | 3.28    |\n| GENIE           | 2.95           | 3.85    | 3.46         | 3.47     | 3.54    |\n| QFS-DLM         | 3.68           | 4.21    | 3.94         | 4.01     | 3.87    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses the shortcomings in the existing methods in an interesting way by incorporating diffusion models.\nThe paper compares its models with comparative models in different datasets.\nQualitative and quantitative results demonstrate that the method improves upon existing methods.", "reasons_to_reject": "Overall, in terms of novelty, this model exploits diffusion models for text replacing standard auto-regressive models. Still, this is not the main criticism of this paper.\nOne way this paper could be improved is by conducting human experiments. The key question is: do humans understand these summaries?", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors develop a new model to address the existing issues in query-focused summarisation. In IR, query-focused summarisation helps obtain the key summary from the relevant documents that could help answer the information sought in the query. As a result, this not only helps with getting answers to the queries quickly but also helps find relevant documents that answer the query.\n\nThe authors have noted that the BART model and its variants have been popularly used in query-focused summarisation. The authors mention that the issue is that the models that rely on auto-regressive models face issues with long-term dependencies and bias.\n\nThe authors have identified that diffusion models have achieved positive performance that could help address the limitations of the existing methods. To this end, the authors have developed a diffusion language model that incorporates query-document relevance to generate reliable summaries.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8rKvOhR6ab", "reviewer": "Reviewer_SFJ4", "summary": "", "questions": "A. Will integrating query-document relevance into the diffusion model's loss function affect the loss's overall ability to converge? Why or why not?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your conscientious and insightful comments so that we can further refine our paper. We will respond to your comments in the following two aspects:\n### **1. Incremental work**\n\nThank you for mentioning this important question.\nIn general, **we adapted the diffusion language model for QFS, not a direct application**, and we have provided a description of this in the Introduction section of our paper.\nSpecifically, the original diffusion language model lacked sensitivity to queries. \nFurthermore, the quality of the QFS data varies, where query-document relevance can reflect sample quality. \nThese two motivations inspired us to modify the original model by incorporating query-document relevance.\nWe calculate the query-document relevance score and assign higher/lower loss weights to high-quality/low-quality data, enhancing the  positive impact of high-quality data and reducing the impairment of low-quality data, which guides the model to prefer samples with high query-document relevance to capture query-required messages.\nExperimental results demonstrate that our approach improves summarization performance. \nAdditionally, over the past few days, we conducted human evaluations, confirming that samples with higher query-document relevance indeed exhibit better quality, and we will supplement this experiment in the final version of our paper.\n\n### **2. The effects of incorporating query-document relevance into the diffusion model's loss function**\n\nThank you for raising such an important issue to further enhance our model. \nIn order to better understand the characteristics of our model, we conducted two additional experiments:\n* In the first experiment, we conducted human evaluations on summary quality. **The evaluation showed that our model outperforms the comparison methods across all metrics, including query focusing, fluency, faithfulness, coverage, and overall performance.**\nIt is because we enhance the positive impact of high query-relevance data and reduce the impairment of low query-relevance data, allowing the model  to be more biased towards learning from high-quality data.\nWe will supplement this experiment in the final version of the paper.\n\n* In the second experiment, we evaluated the convergence speed of the model and found that **our method converged faster than GENIE**. \nThe experimental results show that GENIE is easily disturbed by noisy data, which delays the convergence.  Meanwhile, with the help of query relevance, our model is equipped with more refined and effective information, resulting in a smaller impairment of noisy data and a faster convergence rate than GENIE.\nWe will provide a detailed description of this experiment below.\n\nWe will detailed description of the experiment regarding the loss's overall ability to converge. We performed experiments on the Debatepedia and PubMedQA datasets, collected the loss iteration data of the GENIE and our QFS-DLM models, and plotted their convergence curves. \nThe curves showed that our model converges faster.\n\nBelow is a brief description of the curves of loss convergence for the Debatepedia dataset:\n* GENIE: The loss value starts at 0.0335 and quickly decreases to 0.0177 within the first 20,000 steps. It continues to decrease slightly to 0.0161 between 20,000-90,000 steps. Between 90,000-100,000 steps, the loss increases slightly to 0.0166 but then slowly decreases again to converge to a final value of 0.0156 between 100,000-140,000 steps.\n* QFS-DLM: The loss value starts at 0.0344 and quickly decreases to 0.0175 within the first 20,000 steps. It then continues to decrease slightly to converge to a final value of 0.0153 between 20,000-100,000 steps.\n\nThe convergence of the model on PubMedQA is similar to Debatepedia.\nBoth models showed a relatively stable convergence process. \nHowever, when GENIE was close to complete convergence, the impact of low-quality data on convergence became significant, resulting in considerable oscillations. \nOn the other hand, QFS-DLM reduced the effect of low-quality data and was able to converge continuously without producing considerable oscillations, resulting in a quicker convergence rate.\nWe will supplement the text content and charts for this experiment in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reasons to accept this paper are the following:\n1. The paper claims to achieve state-of-the-art performance on the very relevant task of Query-Focused Summarisation, which could have direct implications for improved performance in the task of Question-Answering, which is an area of large-scale interest and efforts. The summaries generated by the new model seem to be of an appropriate, not-too-large length, and more accurate and smoother than the outputs of other compared models.\n2. The authors propose a method for the model to intrinsically prefer high-quality data and reject low-quality data during training, by incorporating query-document relevance into the loss function, which could be applied to other Natural Language tasks and reduce the overhead of cleaning curated data before the training processes.\n3. The paper provides appropriate mathematical background and explanations for diffusion models in the NLP domain - which is a relatively new area of widespread research, as well as for their specific QFS model.", "reasons_to_reject": "1. The lack of in-depth analysis and studies to fully understand and explain the effects of having query-document relevance incorporated into the diffusion model's loss function which makes it unclear as to how well the model would generalise and adapt to other domains and datasets (such as facts-based, financial, quantitative, scientific datasets etc.), and to different kinds of queries.\n2. The work mostly seems incremental in nature.", "questions_to_authors": "A. Will integrating query-document relevance into the diffusion model's loss function affect the loss's overall ability to converge? Why or why not?", "paper_topic_and_main_contributions": "The paper addresses the problem of Query-Focused Summarisation (QFS). Mainstream QFS models are autoregressive in nature and, thus, suffer from the issues of long term dependencies and exposure bias. Furthermore, low quality data - with lower relevance between queries and documents - can impair model performance.\nThis paper addresses these two limitations through their main contributions listed below:\n1. They apply a non-autoregressive diffusion language model for the QFS task, thus solving the problem of long term dependencies and exposure bias.\n2. They incorporate query-document relevance into the loss function of the diffusion language model to enable the model to prefer high-quality data and to adapt well to the QFS task.\n3. They conduct experiments on the Debatepedia and PubMedQA datasets, achieving state-of-the-art performance in terms of ROUGE scores and demonstrate promising results in the GPT-4 evaluation.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is about query focused summarization (QFS). It adapts GENIE, a diffusion LM, to query focused summarization. The reviewers highlight that the proposed solution is effective and that it addresses shortcomings of existing autoregressive models via diffusion process that comes with GENIE. Most of the critique of the reviewers centers around the incremental nature of adapting GENIE to QFS, and whether that constitutes a significant contribution. While this is a central point of critique, most reviewers don't rate this as major issue regarding soundness. The next iteration of this paper should, more clearly, elaborate what the novel aspects of this adaptation are and why this is relevant for QFS. Moreover, the reviewers critisize limited analyses, regarding the quality of generated summaries and regarding the reasons for performance improvements. The authors provide additional insights in their response which would benefit the next iteration of this paper. Finally, this paper has been submitted to the efficiency track, though it seems unrelated to that topic. If the authors believe efficiency plays a major role in this submission, they should highlight this more clearly.
Decision: Accept-Findings