Learning to Rank Utterances for Query-Focused Meeting Summarization
Year: 2023

Abstract:
Query-focused meeting summarization(QFMS) aims to generate a specific summary for the given query according to the meeting transcripts. Due to the conflict between long meetings and limited input size, previous works mainly adopt extract-then-summarize methods, which use extractors to simulate binary labels or ROUGE scores to extract utterances related to the query and then generate a summary. However, the previous approach fails to fully use the comparison between utterances. To the extractor, comparison orders are more important than specific scores. In this paper, we propose a Ranker-Generator framework. It learns to rank the utterances by comparing them in pairs and learning from the global orders, then uses top utterances as the generator’s input. We show that learning to rank utterances helps to select utterances related to the query effectively, and the summarizer can benefit from it. Experimental results on QMSum show that the proposed model outperforms all existing multi-stage models with fewer parameters.

heading: 1 Introduction
text: Query-focused meeting summarization(QFMS) aims to summarize the crucial information for the given query into a concise passage according to the meeting transcripts. By responding to the query, QFMS can meet the user’s need to focus on a specific aspect or topic of the meeting (Litvak and Vanetik, 2017; Baumel et al., 2018). Unlike the generic summary, QFMS requires the summary depending on both the given query and meeting transcripts.
Previous works consist of end-to-end and twostage frameworks. The end-to-end models take the whole long meeting as the input. Although some works such as HMNet (Zhu et al., 2020) and HATBART (Rohde et al., 2021) use hierarchical attention mechanism to alleviate the rapid growth
*Yajing Xu is the corresponding author.
in computational complexity, it’s still faced with difficulties in training efficiency. The two-stage models extract utterances related to the query and then pass the concatenation of them to the generator. For QFMS, the key information related to the query scatters in certain parts of the meeting. Therefore, the two-stage framework is considered as a practical approach to balance experimental performance and computational efficiency in the long-input problems.
The two-stage framework mainly includes the Locator-Generator and the Simulator-Generator approaches. As shown in Figure 1, in the first stage, the Locator-Generator (Zhong et al., 2021b) framework considers it as a binary classification task. It predicts a binary label of whether the utterance is relevant to the query and uses cross-entropy loss to update parameters. But the hard binary labels can not reflect the relative quality. Especially when the training data is limited by scarcity, the binary classification will have a large margin between positive and negative samples. So the SimulatorGenerator (Vig et al., 2022) framework considers
8496
it as a ROUGE score regression task. It simulates the ROUGE score and uses MSE loss to update parameters. However, there is a gap between the extractor’s ultimate objective and the objective of minimizing the absolute error between predicted scores and ROUGE scores. In fact, rather than specific scores, we care more about the relative orders of utterances.
To make full use of the comparison information between samples, we propose a Ranker-Generator framework in this paper. To balance experimental effectiveness and computational efficiency, the framework contains three steps. First, the utterances would be divided into samples. We conduct pairwise ranking to get an order for each sample. Second, the top utterances in different samples would be fed into the re-ranker, which would conduct listwise ranking to get a global order. Finally, the top K utterances would be concatenated and passed to the generator.
To summarize, our contributions are as follows: (1) This paper demonstrates that, by enhancing the accuracy of extracting query-relevant utterances, the generator can make the summary more related to the query. (2) We propose a Ranker-Generator framework to extract query-relevant utterances by learning to rank discourse to improve the quality of the generated summaries. (3) Experimental results show that the proposed model outperforms existing multi-stage models with fewer model parameters.

heading: 2 Method
text: The architecture of our method is illustrated in Figure 2. Our model consists of a two-stage ranking step and a generating step. The utterances would be ranked by the Sample Pairwise Ranking module and the Global Listwise Re-ranking module, and top of them can be passed to the generator to produce the final summary.

heading: 2.1 Two-Stage Ranking
text: The utterance ranking orders for a brief meeting can be efficiently obtained using the single-stage ranking paradigm. However, the computing complexity of full-pairwise ranking grows at a square rate as the number of utterances grows. Therefore, we adopt a two-stage ranking framework. In the first stage, we propose sample pairwise ranking to reduce computational complexity. But sample pairwise ranking can only evaluate the relative quality within samples. It performs poorly when applied to utterances from various samples, e.g., the top utterances in sample 1 may be ranked lower in sample 2. To overcome the above problem, we apply global listwise re-ranking and concentrate on the top-k utterances in the second stage. Utterances that are unlikely to appear in the generator are filtered out by the pairwise ranking model, then global listwise ranking is conducted to get better top-k orders.

heading: 2.2 Sample Pairwise Ranking
text: In this paper, the ROUGE (Lin, 2004) scores between utterances U and the gold summary S∗ are considered as the measure of query-relevance. The utterances from one meeting are divided into various samples. In one sample, the utterances would be ordered by the ROUGE scores. The ranker should be encouraged to assign higher relevance scores to these top utterances in the order. By learning to rank in pairwise, the model can distinguish the utterances that are more relevant to the query from the comparison. Following the previous work (Zhong et al., 2020), the loss is as follows:
L = ∑
i
∑
j>i
max(0, f(Uj)− f(Ui) + λij) (1)
λij = (j − i) ∗ λ (2) where Ui and Uj are the i-th and jth utterances in gold ranking orders,
ROUGE(Ui, S∗)>ROUGE(Uj , S∗), ∀i, j, i < j, λ is the base margin. f(Ui) is the predicted query-relevance score given by a cross-encoder model.

heading: 2.3 Global Listwise Re-ranking
text: As shown in Figure 2, the top utterances in different samples are gathered in the re-ranking module. The gold orders would be determined by ranking the utterances according to the ROUGE scores. To obtain a more precise top-ranking order, we would perform a refined global sort on these top utterances from various samples using listwise re-ranking. Inspired by ListNet (Cao et al., 2007), we optimize the permutation probability distribution between predicted scores s and the gold scores s∗. The permutation probability is defined as
Ps(π) = n∏
j=1
ϕ(sπ(j))∑n t=j ϕ(sπ(t))
(3)
π is a permutation on the n objects, and ϕ(.) is an increasing and strictly positive function.
But different with ListNet, we optimize the top-k permutation probability rather than top-1 probability. The top-k permutation probability is as follows:
P ks (π) = k∏
j=1
ϕ(sπ(j))∑n t=j ϕ(sπ(t))
(4)
For example, the top-3 permutation probability of π = ⟨1, 2, 3, 4, 5⟩ is as follows:
P 3s (π) = ϕ(s1)∑5 i=1 ϕ(si) · ϕ(s2)∑5 i=2 ϕ(si) · ϕ(s3)∑5 i=3 ϕ(si)
(5) The predicted top1-to-topk distribution is Ps = (P 1s , P 2 s , · · · , P ks ), the gold top1-to-topk distribution is Ps∗ = (P 1s∗ , P 2 s∗ , · · · , P ks∗) We use KLdivergence to reduce the gap between the above two distributions.
L = KL(Ps∗ ||Ps) (6)
KL(Ps∗ ||Ps) = k∑
i=1
P is∗ · log P is∗
P is (7)

heading: 2.4 Generator
text: As shown in Figure 2, after the two-stage ranking, top-k of the utterances would be concatenated and fed into the generator. In the generation stage, the objective is to minimize the cross-entropy loss:
L = − ∑
i
pgt(Si|S∗<i, U) log p(Si|S∗<i, U) (8)
pgt(Si|S∗<i, U) = { 1 Si = S ∗ i
0 Si ̸= S∗i (9)
U is the generator’s input, S∗ is the gold summary.

heading: 3 Experiments

heading: 3.1 Setup

heading: 3.1.1 Implementation Details
text: Models are implemented using the PyTorch framework. The pre-trained BART* from the Transformers (Wolf et al., 2020) library is used as the base abstractive model. The pre-trained MiniLM† from the sentence-transformers (Reimers and Gurevych, 2019) library is used as the pairwise ranking model and the listwise re-ranking model.
All experiments are conducted on NVIDIA RTX 3090 GPU(24G memory). The generator model is trained for 10 epochs. For one model training, the average running time is around 2 hours. Weight hyperparameter λ is 0.01 in Equation 2. The generator’s max length of the input is 1024, max length of the output is 256. Learning rate is 5e-6.
Models were evaluated using the ROUGE metrics (Lin, 2004) in the SummEval toolkit (Fabbri et al., 2021) and each pair of results was subjected to t-test to confirm the effectiveness of our method.

heading: 3.1.2 Datasets Details
text: QMSum (Zhong et al., 2021b) is a query-focused meeting summarization dataset consisting of 1,808 query-summary pairs over 232 meetings from product design, academic, and political committee meetings. Additionally, QMSum contains manual annotations such as topic segmentation and relevant spans related to the reference summary.

heading: 3.1.3 Baselines Details
text: We compare the proposed method with several baselines. TextRank (Mihalcea and Tarau, 2004) is an extractive summarization method with a graphbased ranking model. PGNet (See et al., 2017) uses pointer mechanism to copy tokens from source texts. BART (Lewis et al., 2020) is a pre-trained encoder-decoder Transformer model with a denoising objective, which achieves advanced performance on several summarization datasets(i.e. CNN/DailyMail (Hermann et al., 2015) and Xsum
*The checkpoint is “facebook/bart-large”, containing around 400M parameters.
†The checkpoint is “cross-encoder/ms-marco-MiniLM-L12-v2”, containing around 134M parameters.
(Narayan et al., 2018)). LEAD+BART uses the beginning utterances as the BART’s input. HMNet (Zhu et al., 2020) uses a hierarchical attention mechanism and cross-domain pre-training for meeting summarization. Longformer (Beltagy et al., 2020) replaces the quadratic self-attention mechanism with a combination of local attention and sparse global attention. DialogLM (Zhong et al., 2021a) is a pre-train model using intrawindow denoising self-reconstruction pre-training task and intra-block inter-block mixing attention. SUMMN (Zhang et al., 2022) is a multi-stage summarization framework for the long-input summarization task. DYLE (Mao et al., 2022) treats the extracted text snippets as the latent variable and jointly trains the extractor and the generator. Point Network+PGNet and Point Network+BART (Zhong et al., 2021b) adopt a twostage approach of locate-then-summarize for long meeting summarization. RELREG-TT (Vig et al., 2022) and RELREG (Vig et al., 2022) considers extracting as a ROUGE regression model using bi-encoder and cross-encoder.

heading: 3.2 Results & Analysis
text: The ROUGE score (Lin, 2004) is adopted as the evaluation metric. The performances of our method and baselines are summarized in Table 1. Experimental results show that our method significantly outperforms the baselines (p < 0.05) on QMSum dataset with fewer parameters.
To have a fair comparison among the three frameworks, we design an experiment to evaluate the performance of these frameworks using the same backbone as the extractor and the same generator. The experimental results show that the proposed model significantly outperforms Locator-Generator and Simulator-Generator, which demonstrates that the ranker can obtain meeting utterances that are more suitable for the generator by learning to rank utterances.
To verify the effectiveness of the two-stage ranking paradigm, we conduct an ablation experiment. Our model significantly outperforms the model without re-ranking module (p < 0.05). Experimental results show that the model without re-ranking module reduces 2.49 ROUGE-1, 2.50 ROUGE-2, 2.13 ROUGE-L scores, which demonstrates the importance of the re-ranking module. By listwise ranking, we can get a more precise top-ranking order.
We have an interesting observation. Unlike the ROUGE score regression model, the ranker is less sensitive to the model size. We believe this is because learning the relative order by comparison is easier than fitting ROUGE scores separately. It reduces the ranker’s reliance on the model size by
making full use of the comparison between samples. As a training task for extractors, learning to rank is a more suitable objective. Since to the extractor, it is the relative order that matters rather than the absolute error in fitting the ROUGE score.

heading: 3.3 Extractor Performance
text: We conduct experiments to evaluate the performance of the extractor, which help to explore the impact of the extractor on the quality of the generated summaries. The lexical overlap metric between the extracted utterances and the gold summary is used to measure the relevance of the meeting utterances to the summary/query. The experimental results show that the ranker significantly outperforms the baselines in extracting relevant utterances. It demonstrates that by learning to rank utterances, the ranker is able to extract the utterances that are more relevant to the summary/query.

heading: 3.4 Human Evaluation
text: We further conduct a manual evaluation to assess the models. We randomly select 50 samples from QMSum and ask 5 professional linguistic evaluators to score the ground truth and summaries generated by 5 models according to 3 metrics: fluency, query relevance and factual consistency. Each metric is rated from 1 (worst) to 5 (best) and the scores for each summary are averaged.
As shown in Table 3, the proposed model significantly outperforms all the baselines on query relevance, which benefits from the extractor’s improvement on selecting the relevant utterances. Besides, the factual consistency score is also improved. We think that by comparing the relevance between utterances and the summary/query, the top utterances are more relevant to each other, which may help to improve factual consistency. In the aspect of fluency, the proposed model has only slight improvement compared to the baselines.

heading: 4 Conclusion
text: This paper proposes a new multi-stage framework for QFMS. It learns to rank the meeting utterances by pairwise and listwise comparison between them. By selecting the utterances with high query-relevance scores as the generator’s input, the generator can produce high-quality summaries that are more relevant to the query. The experiments demonstrate the effectiveness of the RankerGenerator framework.

heading: 5 Acknowledgements
text: This work was supported by MoE-CMCC ”Artifical Intelligence” Project No. MCM20190701 and the National Natural Science Foundation of China (NSFC No.62076031).
We thank the anonymous reviewers for valuable feedback and helpful suggestions.
Limitations
This paper mainly focuses on the Query-focused Meeting Summarization(QFMS) task. Besides, We have explored the performance of the RankerGenerator framework on the long-input summarization task. But the results do not show a significant improvement. Although QMSum dataset is also faced with the long-input challenge, the QFMS task only summarizes specific parts of the original text, so it can take these parts as the input. While the goal of the long-input summarization task is to generate an overall summary, which needs to have a global view on the original text. So we think the extract-then-generate framework is unsuitable for the long-input summarization task. The previous work SUMMN (Zhang et al., 2022) is more suitable for the long-input summarization task.
In addition, the multi-stage approach has a performance disadvantage over the end-to-end approach. However, the computational complexity of the multi-stage approach is much lower than that of the end-to-end approach. The multi-stage approach can balance experimental performance and computational complexity. So it is worthy of exploration as well as the end-to-end approach.
Ethics Statement
In this paper, all experiments are conducted on QMSum (Zhong et al., 2021b), which is open-source and obeys MIT license. The meeting transcripts data doesn’t contain any privacy information(such
as password, phone number and trade secrets) or offensive content.

heading: A Case Study
text: About future trend
Query: Why did the industrial designer disagree with the marketing when talking about some new ideas about future products?
Gold: The marketing thought they should not trust too much the trend because it wouldn't last long - fruits and vegetables were in fashion this year but maybe next year it would be insects. However, the industrial designer believed it was good to follow the flow now and then since electronic devices were not long life products: if people changed, the product appearance should change as well.
Previous: Industrial Designer thought that it would be interesting to see if people would buy this one, and maybe add some features to it after. However, he thought it would not be very productive to go through all the process from the beginning to the end.
Ours: The industrial designer thought it was good to follow the flow and make it now and after, if the people change their mind, you change also the product.
Query: What did Barry Hughes think about the targets regarding performance measure?
Gold: Barry Hughes stated that they did not have targets and only prosecuted with reasonable prospects. As for the police, there were no targets for the police anymore, except for some individuals.
Previous: Barry Hughes thought that there was a shortage of registered intermediaries in Wales, and the Ministry of Justice had taken action to deal with that, and they had had a number of people who were now in a position to act as intermediaries. Now, of course, if they were to decide not to do that anymore, they may have a problem, but, in turn, they would be looking to recruit more people into those positions. So, it had the potential to serve as a barrier, but in practice, he didn't think it would be a barrier.
Ours: Barry Hughes thought that the police were much more sensible these days than they may have been 20 years ago, in terms of trying to get cases charged in order to meet some notional target. In the CPS, they did not have targets for securing convictions or not. Clearly, they prosecuted if they thought that there was a reasonable prospect, a realistic prospect, of conviction and it was in the public interest.
About the performance measure
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The Limitations section.
3 A2. Did you discuss any potential risks of your work? The Ethics Statement section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, the References section.
3 B1. Did you cite the creators of artifacts you used? Yes, the References section.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The Ethics Statement section.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The Limitations section.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The Ethics Statement section.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B.
C 3 Did you run computational experiments? Section 3.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix A.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.4.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.4.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The Ethics Statement section.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: About future trend
Query: Why did the industrial designer disagree with the marketing when talking about some new ideas about future products?
Gold: The marketing thought they should not trust too much the trend because it wouldn't last long - fruits and vegetables were in fashion this year but maybe next year it would be insects. However, the industrial designer believed it was good to follow the flow now and then since electronic devices were not long life products: if people changed, the product appearance should change as well. Previous: Industrial Designer thought that it would be interesting to see if people would buy this one, and maybe add some features to it after. However, he thought it would not be very productive to go through all the process from the beginning to the end. Ours: The industrial designer thought it was good to follow the flow and make it now and after, if the people change their mind, you change also the product. Query: What did Barry Hughes think about the targets regarding performance measure?
Gold: Barry Hughes stated that they did not have targets and only prosecuted with reasonable prospects. As for the police, there were no targets for the police anymore, except for some individuals. Previous: Barry Hughes thought that there was a shortage of registered intermediaries in Wales, and the Ministry of Justice had taken action to deal with that, and they had had a number of people who were now in a position to act as intermediaries. Now, of course, if they were to decide not to do that anymore, they may have a problem, but, in turn, they would be looking to recruit more people into those positions. So, it had the potential to serve as a barrier, but in practice, he didn't think it would be a barrier. Ours: Barry Hughes thought that the police were much more sensible these days than they may have been 20 years ago, in terms of trying to get cases charged in order to meet some notional target. In the CPS, they did not have targets for securing convictions or not. Clearly, they prosecuted if they thought that there was a reasonable prospect, a realistic prospect, of conviction and it was in the public interest. About the performance measure
ACL 2023 Responsible NLP Checklist