Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts
Authors: Haochen Tan
Han Wu
Wei Shao
Xinyun Zhang
Mingjie Zhan
Zhaohui Hou
Ding Liang
Linqi Song
Venue: EMNLP 2023 Main (2023)
Keywords: meeting summarization
essential content extraction
long-text compression

Abstract: Meetings typically involve multiple participants and lengthy conversations, resulting in redundant and trivial content. To overcome these challenges, we propose a two-step framework, Reconstruct before Summarize (RbS), for effective and efficient meeting summarization. RbS first leverages a self-supervised paradigm to annotate essential contents by reconstructing the meeting transcripts. Secondly, we propose a relative positional bucketing (RPB) algorithm to equip (conventional) summarization models to generate the summary. Despite the additional reconstruction process, our proposed RPB significantly compresses the input, leading to faster processing and reduced memory consumption compared to traditional summarization methods. We validate the effectiveness and efficiency of our method through extensive evaluations and analyses. On two meeting summarization datasets, AMI and ICSI, our approach outperforms previous state-of-the-art approaches without relying on large-scale pre-training or expert-grade annotating tools.

Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts
Haochen Tan', Han Wu', Wei Shao', Xinyun Zhang',
Mingjie Zhan², Zhaohui Hou?, Ding Liang,, Linqi Song!
Department of Computer Science, City University of Hong Kong
2 Department of Computer Science and Engineering, The Chinese University of Hong Kong Time Research
haochetan2-cemy cattmy..iiteee city.. edu hk,
cersoonggcityyy edu.
Abstract
learning (Zhu et al., 2020; Rohde al., 2021), tailor the input to an acceptable length through sen- tence compression (Shang et al., 2018a) and coarse- to-fine generation (Zhang et al., 2022). However, these approaches do not specifically target the criti- cal information in meeting transcripts. Feng et al. (202 lb) utilizes the token-wise loss as a criterion to annotate contents with DialoGPT (Zhang et al., 2020b), suffering from labeling unpredictable con- tents as critical information. Besides, the com- monly used pre-processing procedure that extends models' positional embedding by copying, and truncating the lengthy input compromises the posi- tional relationships learned during pre- pre-training, and results in a loss of important information due to the brutal truncation. Consequently, natural question is How can precisely capture the salient con- tents from noisy and lengthy meeting transcripts. and summarize them with conventional language models?
Meetings typically involve multiple partici- pants and lengthy conversations, resulting re- dundant and trivial content. To overcome these challenges, we propose a two-step framework, Reconstruct before Summarize (RbS), for effec- tive and efficient meeting summarization. RbS first leverages a self-supervised paradigm to an- notate essential contents by reconstructing the meeting transcripts. Secondly, we propose a relative positional bucketing (RPB) algorithm equip (conventional) summarization mod- els to generate the summary. Despite the ad- ditional reconstruction process, our proposed RPB significantly compressed the input, lead- ing to faster processing and reduced memory consumption compared traditional summa- rization methods. We validate the effectiveness and efficiency of our method through exten- sive evaluations and analysis. On two meeting summarization datasets, AMI and ICSI, our approach outperforms previous statc-of-the-art approaches without relying on large-scale pre- training or expert-grasd grade annotating tools.
Our observation is that meetings are character- ized by extensive communication and interaction, with specific texts often containing pivotal content that drives these interactions. Based on this un- derstanding, we propose a two-step meeting sum- marization framework, Reconstreut before Sum- marize(RbS), to address the challenge of scattered information in meetings. RbS adopts reconstruc- reconstruct the responses in the meeting, it also synchronically traces out which texts in the meeting drove the responses and marks them as essential contents. Therefore, salient information is captured and annotated as anchor tokens in RbS. To preserve the anchors but compress the lengthy and noisy input, we propose the relative positional bucketing (RPB), dynamic embedding compres- sion algorithm inspired by relative positional en- coding (RPE) (Shaw et al., 2018; Huang al., 2019). Our RPB-integrated summarizer can pre- serve the anchors and compress the less important contents according to their relative position to the
1 Introduction
Although numerous achievements have been made in the well-structured text abstractive summariza- tion (Zhang et al., 2020a; Liu et al., 2018; Lewis et al., 2020), the research on meeting summariza- tion is still stretched in limit. There some out- standing challenges in this field, including 1) much noise brought from automated speech recognition models; 2) lengthy meeting transcripts consisting casual conversations, content redundancy, and diverse topics; 3) scattered salient information in such noisy and lengthy context, posing difficulties for models to effectively capture pertinent details.
To this end, previous works adapt the language model to long inputs through techniques such as long-sequence processing (Beltagy et al., 2020; Tay al., 2020; Zhong et al., 2022) and hierarchical
"Corresponding author.

---

@ Reconstaction
(b) Summmaization
Context:
you supposed original, trendy user friendly. I'm really what you guys have already received in your emails.
0s you con see it's supposed to original, trendy and user friendly.
I'm not really what you guys already received in your emalls.
What did you get?
What did you get? Industriol Designer: got the project announcement what the project is. Designing control...
Response:
Industrial Designer: got the project onmouncement about whot
Context:
I'm not really what you guys have already recelved in your emalls.
What did you get?
Summarizer
Industrial Designer: got the project onnouncement about what
Embedding
Response:
Design a remote control
Amodet
Reconstruct
Retrace
Reconstructor
Annotate anchors
model
Remaining Layers
Figure By computing scaled attention, reconstructor identifies the contribution of each token in the contexts toward recovering the response. Tokens that make significant contributions marked anchors(labeled grey in the figure). Following this, the summarizer embeds the annotated texts. RPB then compresses the embedding from n dmodel x dmodet based on the anchors, where n the length the input(normally 20k), c constant(108) (1024 by default) and >>
This allows the summarizer to generate concise and informative summary of the meeting transcripts.
anchors.
and identifying the context that drives the interac- tion. Meanwhile, before generating the summary, the summarizer compresses the lengthy input and preserves critical content.
Although RbS introduces an additional impor- tance ussessment step, the introduction of RPB greatly compresses the length of the original text, making RbS even faster and memory-efficiee than the traditional one-step approach. The experimen- tal results AMI (Mccowan al., 2005) and ICSI (Janin al., 2003) show that RbS outperforms previous state-of-the-art approaches and surpasses a strong baseline pre-trained with large-scale dia- logue corpus and tasks. Extensive experiments and analyses are conducted to verify the effectiveness and efficiency of each process of our approach.
2.1 Reconstruction and Retracing
To capture the essential information, propose retracing the contexts that drive interactions with reconstructor. We split the meeting transcripts into context-response pairs. By reconstructing the response based on the context and tracking the contributing contexts, we can effectively capture the important content of the transcript.
To sum up, our contributions are as follows: (1) We propose the RbS, efficient and effective framework for long-text meeting transcripts sum- marization; (2) Without external annotating tools large-scale pre-training corpus and tasks, our method can efficiently generate meeting minutes with conventional language models (PLMs); (3) Ex- tensive experiments demonstrate the effectiveness of our framework.
The architecture in Figure 1 il- lustrates the process. To recover each response, use window of size to limit the input his- tory, with w set to in Figure 1. We assume that a meeting transcript contains m sentences and create a sub-dataset consisting of m pairs, i.e., L.e.,[|mmar(0. I= 5:), where E [2 m). To prompt the language model to predict the end of the meeting, we add special token [EOM] at the end of the transcript. Finally, reconstructor cover the response from S,, to [EOM] as closely as possible using the input ([mar(0,)-w)s reconstruction is conducted via the forward pass of language model with teacher forcing (Williams and Zipser, 1989; Lamb et al., 2016).
Reconstruction
2 Methods
The main architecture is shown in Figure/ Our framework comprises two components: the recon- structor and the summarizer. The reconstructor is responsible for reconstructing meeting transcripts
The

---

dgdde!
Calculate Boundary
moddel
Assign Buckets
P TT"
doodet
1
Figure 2: For context-response generation, RbS utilizes the scaled-attention retrace how much each token in context contributes to the recovery of the response.
Bucket-wise Average Pooling
modee
Retracing As shown in Figuree 2, during the re- construction, RhS synchronically retrace the contri- bution of cach token of the context to the recovery the response, from the perspective of attention weights (Bahdanau et al., 2014; Kim et al., 2017; Vaswani et al., 2017) and gradients. Recap the procedure of the attention mechanism, the cross attention (Vaswani et al., 2017) is formulated
Figure An illustration of compressing a dmode! X n sequence embedding with two annotated anchors dmoodt x 10 using RPB, where b is the bucket number. Two orange blocks are anchors.
aging and multi-view voting, Averaging involves taking average of the ratings for each token during the reconstruction process. We then select the top-k tokens with the highest average rating as the salient information, which we refer to as anchor tokens in RbS. This approach leverages the average score to express the overall contribu- tion of each token. Multi-view voting involves se- lecting the top- ... tokens with the highest score in each criterion after the reconstruction is completed. This approach considers multiple perspectives for evaluating contexts, selecting the contexts that con- tribute most prominently under each perspective anchors.
Attention(Q, K, V)= softmarl QKT
VV, (1) Vix
Where Q is the representation of the response be generated in the decoder, K, and V are the memo- ries and values that come from encoded contexts. Inspired by the works that adopt the attention and gradients to retrace which part of the input drives the model to make predictions (Jain et al., 2020; Kindermans et al., 2016 6; Atanasova et al., 2020; Sundararajan et al., 2017), extract the impor- tance scoring with scaled attention (Serrano and Smith, 2019) aa from the last cross-attention layer determine the contribution of each token in contexts (0, (0000-.--ww-ii1 to the restoration of the response Sç. The scaled attention a is denoted as the attention scores scaled by its correspond- ing gradient Va, Bo. prediction.
2.2 Summarization
With the obtained anchors, our hypothesis is that to- kens in meeting texts are more relevant to salient in- formation when they are closer to the anchors, and conversely, tokens that are farther away from the anchors are less relevant the important content. Therefore, we propose relative positional bucket- ing(RPB), which compresses the original input as losslessly as possible by preserving the anchors and dynamically compressing the less-important contexts around the anchors.
Oj
where ý is the model's
Scores Aggregation We utilize context window of size w to reconstruct responses, which leads to reasonable reconstruction. However, this also re- sults in each sentence being treated as context w times to recover responses, which poses a chal- lenge in combining importance-related scores dur- ing tracebacks and scoring. To address this issue, we carefully propose the hypothesis that each to- ken can be considered to be scored according to w different criteria after rated by w different responses. Therefore, two different strategies are investigated this paper, which we refer to aver-
Relative Positional Bucketing RbS employ an conventional language model that accept to- kens input the summarizer, consider quence (4,, ,mm annotated with n an- chors {ag, 0j, ,n), where m > 2 n, and positions of all anchors are {io, io..i..... ..). The summarizer first extract the embeddings

---

{eo, .1,-.. em of the sequence. Then it iden- tified positions {io + 11-10 21 + 12 2"..... 2 + " in the middle of all adjacent anchors as boundaries, Each pair of adjacent boundaries will form sub-sequence containing an anchor point. For each sub-sequence, the summarizer obtains the position code of each token relative to the an- chor. Inspired by the T5 (Raffel et al., 2020) that translates relative position to a bucket number for memory-efficient and weeeeeeeeuueereeeeeeeeeeeeeeeeee tion, we compress the sequence by bucketing the embeddings {eo, e1,- Emm with buckets. We assign larger buckets the embeddings that have a large relative distance to the anchors, and smaller buckets to the embeddings that are close to the anchors. Embeddings that share the same bucket will be compressed by average pool- ing. The calculation of bucket assignment for each token refers to Algorithm 1. Finally, the summarizer processes the compressed embeddings (co, (6, ,'') and generate the summary. Fig- ure demonstrates an example that RPB com- presses dymodel n embeddings containing two anchors to dyoode! x 10 embeddings. Such pro- cess forms dynamic compression based on the importance of the contexts.
of input and target reaches 13,317 and 488.5, re- spectively. while only 59 meeting transcripts are included. Following the preprocessing pipeline pro- posed in Shang et al. (2018a), we split the data into traning/evellpmenneetinngg sets with the list pro- vided in (Shang et al., 2018b): 97/20/20 for AMI and 42/1 1/6 for ICSI. Besides the meeting minutes, decisions, actions (progress in the ICSI), and prob- lems encountered in the meeting are included in the golden summarization. Extra spaces and duplicate punctuation removal are adopted clean the data further.
atten-
Baseline Metric BART (Lewis et al., 2020) is selected as both the baseline and backbone. BART- CNN that finetuned on the CNN-Daily Mail (Her- mann et al., 2015) is also evaluated. Sentence Gated (Goo and Chen, 201 8) utilizes the dialogue acts to generate summaries. PGNet (See et al., 2017) is traditional approach that summarizes the meeting with the pointer network. HMnet (Zhu et al., 2020) adopts cross-domain pre-training be- fore summarizing with a hierarchical attention mechanism. HAT (Rohde et al., 2021) performs a hierarchical attention transformer- based architec- ture. DDAMS (Feng et al., 202 incorporates dis- course information to learn the diverse relationship among utterances. Summ'T NN (Zhang et al., 2022) performs the split-then-summariie in multi-stage for lengthy input. Feng et al. (2021b) employs DialogGPT as the annotator to label the keywords and topics in the meeting transcripts. Besides, Di- alogLM (Zhong et al.. 2022) pre-trained on large- scale dialogue-related corpus and tasks are also compared to show our efficiency. All approaches evaluated with ROUGE (Lin, 2004), namely ROUGE-1, ROUGE-2, and ROUGE-L.
The difference between our proposed summa- rizer and the original BART is only the addition of the RPB module, which is inserted between the embedding layer and the first attention layer. The summarization based on the compressed embed- ding instead of the original one, which greatly saves memory and computation time. Furthermore, is noteworthy that the bucketing operation and batch average-pooling are parallel constant calculation! and seatter-reduce operation", respectively. making the process highly efficient.
3 Experiments
Implementation Details We use the released BART checkpoints in Huggingface's Transform- ers (Wolf et al., 2020) for RbS. Specifically, we ini- tialize RbS with BART-large checkpoints, and the parameters RbS-CNN are initialized by BART- large-CNN. During the response reconstruction, we used eight sentences as contexts. The reconstruc- tor is trained for 2, 300 steps on the split AMI and 1, 500 steps the split ICSI, with learning rate of 5e-5 and total batch size of 256. Once the constructor enabled to recover the response, we perform one forward pass with the teacher-foreing to retrace the contribution of the contexts. During this process, 6.4% of the tokens are annotated as
3.1 Setup
Dataset & Preprocessing RbS is evaluated on two meeting summarization datasets, AMI (Mc- cowan et al., 2005) and ICSI (Janin et al., 2003). AMI is dataset of business project meeting see- narios that contains 137 transcripts. The average length of input and target length is 6,007 and 296, respectively. ICSI aimed at academic discussion scenarios, where professors and other students have discussions with each other. The average length
'T5 implementation "scatter-reduce

---

AMI
ICSI
Model
R-1
R-2
R-L
R-1
R-2
R-L
Backbone
BART-large (Lewis et al., 2020)( 3072)
49.99
16.95
47.79
43.70
9.77
41.34
BART-large-CNN (Lewis al., 2020)(/ 3072)
50.46
17.00
48.28
46.06
10.38
43.86
LSTM and RNN
PGNet (See et al., 2017)
42.60
14.01
22.62*
35.89
6.92
15.67*
Sentence-Gated (Goo and Chen, 2018)
49.29
19.31
24.82*
39.37
9.57
17.17*
Language Model as Annotator
PGN (Feng et al., 2021b)
50.91
17.75
24.59*
Transformers
HMNet (Zhu et al., 2020X( = 8192)
52.36
18.63
24.00*
45.97
10.14
18,54
HAT-BART (Rohde et al., 2021)( = 3072)
52.27
20.15
50.57
43.98
10.83
41.36
DDAMS (Feng et al., 2021a)(( = 13000)
53.15
22.32
25.67*
40.41
11.02
19.18
Summ" N (Zhang et al.. 2022)
53.44
20.30
51.39
45.57
11.49
43.32
Large-Scale Spalofue-Specific Pre-train
Di (Zhong et al., 2022)( 5120)
54.49
20.03
51.92
49.25
12.31
46 .80
Oury
RbS 1024)
54.06
21.02
52.07
50.28
13.24
47.15
RbS-CNN (1 1024)
54.99
20.98
52.40
49.61
12.20
46.97
Table 1: The performance on AMI and ICSL. is the maximum number of input tokens for the corresponding model. denotes the metrics are calculated without sentence split. RbS takes BART-large as the backbone, while the backbone of RbS-CNN is BART-large-CNN.
anchors. The total bucket number is equal to the maximum acceptable input the backbone, which is 1024 for BART. The quantity of buckets for each sub-sequence depends on the length ratio to the total length. For the summarizer, we set the learn- ing rate 3e-5 with total batch size of 64. It is worth noting that RbS trained solely on AMI and ICSI without any external data tools. We do not introduce any pretraining from other domains.
large-scale dialogue-specific pre-training. Even when compared to DialogLM, which is pre-trained on large-scale dialogue-specift corpus and tasks and requires more time-consuming and computa- tion resources, the advantages are still pronounced in ICSI datasets with longer input. The results demonstrate that RbS enhances conventional lan- guage models' ability summarize lengthy meet- ing transcripts by focusing on salient information and compressing irrelevant content. Without ex- tending the models' acceptable input length, RbS enables the conventional language model to sum- marize the meeting transcripts with less memory consumption.
3.2 Main Results
Table 1 presents the ROUGE scores on AMI and ICSI. Our framework outperforms baselines across all metrics. Notably, our model achieves significant improvement on AMI (ROUGE-1 49.99 + 54.06) and more substantial gain on ICSI (ROUGE-1 43.70 + 50.28), compared to the BART-large model. This is due to the longer context in ICSI, which exceeds 10K words on av- erage, demonstrating the remarkable ability of our model to handle extreme-length inputs. When using BART-large-CNN the backbone, ob- serve further improvements in the AMI dataset. Meanwhile, our RRS-CNN outperforms the pre- vious state-of-the-art approach, Summ"N, "N, by ap- proximately 1.5 ROUGE-1 score on AMI, and ROUGE-1 score on ICSI, without requiring the
Analysis
In this section, we conduct further analysis to show the effectiveness the RbS. We aim to investigate the correlation between anchor tokens and salient information in the meeting transcript. Through extensive experiments, we will demonstrate the va- lidity of our approach in capturing anchor tokens, and the significance of anchor tokens in convey- ing important information. Furthermore, we will analyze the impact of different methods for aggre- gating the importance scores. We will also provide justification for our bucketing algorithm, analysis

---

AMI
AMI
Model
Model
R-1
R-2
R-L
R-1
R-2
R-L
RbS-CNN
54.99
20.98
52.40
Scaled Attention
54.99
20.98
52.40
Substitute Random
Attention
52.08
18.96
50.79
25%
52.47
19.12
50.33
Gradient
53.41
20.35
50.80
50%
52.84
19.74
52.14
Token-wise Loss
52.02
19.61
50.
75%
51.03
18.81
48.76
Random
50.84
18.64 68.64
48.88
Substitute High Frequency
25%
53.60
20.32
51.33
Table 3: Ablation studies on different importance scor- ing approaches
50%
53.38
20.15
51.15
75%
52.84
20.70
50.75
is to replace anchors with high high-frequency tokens. For anchor token deletion, there are also two dif- ferent strategies. One is delete the anchor kens randomly, and the other to sort the anchors in descending order importance score and di- vide them into four fractions. Then, we remove one at time to observe the performance change. The results in Table 2 demonstrate that both an- chor token substitution and deletion negatively affect the performance. Specifically, when domly substituting the anchor tokens with the oth- ers, plunge of the ROUGE-1 scores could be observed (51.99 + 52.17). Although the score im- proves slightly after replacing anchors with high- frequency tokens, the performance still falls far short of anchor tokens. This indicates that anchor tokens selected by our framework are informative and play irreplaceable roles. This phenomenon is even more evident in the random removal of the anchor tokens. Results on the performance of different percentages of anchor tokens also show that our framework produces strongly importance- correlated rankings.
Deletion Random
25%
51,42
20.
49.42
50%
51.79
19.26
50.04
75%
49.97
18.78
48.32
Deletion Sorted Fraction
0%-25%
49.10
17.76
47.03
25%-50%
50.97
17.28
48.89
75%
53.48
20.57
51.18
75% 100 %
53.19
21.30
51.20
Table 2: Ablation studies on the substitution and dele- tion of anchor tokens
of the computing complexity is also provided to prove the efficiency of the framework. Addition- ally, the potential for reusing the parameters of the reconstructor is explored in Appendix A.3.
4.1 Importance Scoring
In this section, we examine the impact of impor- tance scoring
framework. To demonstrate the criticality of the anchors selected by our recon- struction and retracing process, we conduct experi- ments in various settings: (1) We delete or substi- tute the selected anchors with different ratios and observe the resulting changes in performance. (2) We test the framework with different indicators of importance, including the attention weights, the gradient of the attention weights, random scoring, and token-wise loss similar Feng al. Ib), which uses r percentage of words with the highest reconstruction loss as keywords. (3) extract and visualize the heatmap of our approach to see if the anchor words are precisely those we need. (4) We investigate the number of anchor tokens required to achieve acceptable performance for different scor- ing algorithms.
A Attention, Gradient, and Token-wise Loss conduct an ablation study on different types of scor- ing indicators, namely the attention weights, gradi- ents of the attention, and token-wise loss. Attention weights and their corresponding gradients are ex- tracted from the last transformer layer of the model. As for the token-wise loss, different from our frame- work that treats the response as a query to rate the importance of the context, this approach scores the response directly according to the generation loss: exp(l,t)
We
I= -= EV exp(f,
(2)
where / is the generated token, is the ground truth, and V the vocabulary size. Similar our setting, all the methods extract 6.4% tokens as anchors. As shown in Table 3, scaled attention achieves the best performance on AMI. The performance
Anchor Deletion and Substitution For substi- tution, we take two measures. One is to replace the anchor tokens with other tokens randomly sam- pled from the meeting transcript, and the other

---

AMI
Model
Aggregation
R-1
R-2
R-L
Vote
54.06
21.01
52.07
RbS
Avg
52.56
20.12
50.34
Vote
54.99
20.98
52,40
RbS-CNN
Avg
53.88
20.61
51.77
Table 4: Ablation study channel aggregation
edge technologies. This excerpt is from meeting discussing the need to include voice recognition in remote controls. The middle of Figure 4 shows that most recovered tokens assign high attention scores to punctuation, indicating that attention weights do not accurately reflect context importance. The bottom part of the figure shows that while gradi- ents can select essential content, they also assign high weights to irrelevant contents, making them an unsatisfactory indicator of importance. The top of Figure 4 shows that scaled attention weights ac- curately detect important content, assigning high scores tokens such "pay more for voice recog- nition" and "cutting-edge technology in remote control," while giving low scores most other content, especially punctuation. This visualization provides an intuitive picture of our framework ss ability to capture key points, further explaining its superior performance.
PPP 4arg popp mm W and bllae and
Figure Visualization of the heatmap. From top to bottom are heatmaps of scaled attention, gradient, and attention weights, respectively.
Number of anchors We conduct the ablation studies on how the number anchors influences the framework's performance, as shown in Fig- ure 5. We gradually increased the number of an- chor points and observed the change in the ROUGE score. Surprisingly, we found that the total num- ber of anchors did not need to be very high; in fact, increasing the number of anchor tokens re- sulted in performance degradation. We attribute this phenomenon to the fact that the total num- ber of buckets for the BART model limited to 1024. The more anchor tokens there are, the fewer buckets the other tokens can share, leading to over- compressed context and performance degradation. We also observed that method achieved strong performance with fewer top-ranked tokens, while the other two methods required more anchor tokens to achieve acceptable performance. This indicates that our approach effectively captures salient infor- mation.
(a) ROUGE-1
(b) ROUGE-L
Figure 5: Trend ROUGE score of different methods with increasing anchor ratio
the gradient is comparable with the scaled attention, while there are sharp decreases when switching the scaled attention attention weights and token- wise loss. These results demonstrate that scaled attention weights are more importance-correlace than the others. This finding is consistent with Chrysostomou and Aletras (2022); Serrano and Smith (2019)
Visualization To validate the importance scoring approaches, visualized the ratings of context in meeting transcripts. Figure 4 displays the heatmap for each scoring method using the response "if that's possible, we might consider getting into it," where "it" refers to voice recognition and cutting-

---

AMI
pressed the length of input from to I(1024 by default), without altering the model structure be- yond its initial form. The time complexity is O x dmoades. Therefore, given the lengthy meeting texts, despite the additional introduction of the reconstructor, the combined complexity O x x dmmodet) + O dmodde) is much lower than that of the regular summary model, which has complexity of On x dmodde). Our proposed approach effectively handles lengthy meeting texts with lower time complexity, making a promising solution for real-world applications.
Truncation
R-1
R-2
R-L
Bucketing
54.99
20.98
52.40
Right
50.46
17.00
48.28
Middle
51.18
19.40
50.78
Left
50.59
18.32
48.34
Random
48.79
18.03
47.13
Hard truncation
51.90
18.00
49.93
Table 5: Ablation study bucketing and truncation
4.2
Scores Aggregation
We conduct an ablation study on two proposed score aggregation methods: averaging and multi- view voting. Results in Table 4 show that multi- view voting outperforms averaging. We attribute this to the fact that averaging disrupts the multi- perspective rating mechanism. This result is con- sistent with motivation that multi-view voting brings multiple horizons to the choice of anchor tokens. Therefore, we conclude that multi-view voting is necessary and beneficial for filtering an- chor tokens.
5 Related Work
Long-sequena processing techniques such as sliding-window attention (Beltagy et al., 2020), sparse sinkhorn attention (Tay et al., 2020), and hierarchical learning (Zhu et al., 2020; Rohde et al., 2021) are well-explored. These approaches tar- get specifically lengthy input but ignore capturing salient information. Sentence compression (Shang et al., 201 8a) and coarse-to-fine generation (Zhang et al., 2022) are developed to tailor the input length. However, the error propagation in intermediate steps severely limits their performance.
4.3 Bucketing and Truncation
Our bucketing strategy can be viewed as "soft- truncation" approach that pools contents dynam- ically instead of truncating the sequence brutally. To justify this compression process, compared bucketing with truncation. For sequence truncation, we truncated the sequence from the left/right/midd- dle side or a random position to fit the input into the summarization model. We also tested anchor-based hard truncation, which keeps only the top-30% anchors as input. Table 5 shows significant per- formance degradation when using hard-truncation, suggesting that it is more sensible compress se- quences dynamically according importance than truncate them brutally. However, cutting se- quences based on anchor points still outperforms direct left/right/middle truncation. These results further demonstrate that anchors are informative tokens.
Meanwhile, language models have gradually equipped with dialogue acts (Goo and Chen, 201 8), discourse relationship (Feng et al., 202 coref- erence resolution (Liu al., 2021), and topic- segmentation (Liu et al., 2019; Li et al., 2019; Feng et al., 2021b). Despite the modest advances, these methods require external annotating tools or expent- grade annotators to accomplish the task. Feng et al. (2021b) employ the DialogGPT (Zhang et al., 2020b) as annotator to capture keywords and top- ics. Despite the performance, adopting the token- wise loss label keywords needs to be considered more deeply.
Additionally, cross-domain (Zhu et al., 2020) and large-scale dialogue-specific pre- training (Zhong et al., 2022) are utilized. However, large-scale datasets, excessive optimization steps, and sufficient computational resources are necessities and luxuries.
4.4 Computational Complexity
The reconstructor divides meetings of length into context-responss pairs, where the average length of each context is c. The values of n, r, and are in the range of 5k-20k, 2-60, and 100-300, re- spectively. The time complexity of the reconstruc- tion process is approximately Or x o² x d.modet). For summarizer, our introduced RPB greatly com-
Conclusion
In this paper, we proposed RbS, a meeting summa- rization framework that accurately captures salient contents from noisy and lengthy transcripts. RbS uses a two-step process to evaluate content impor- tance and dynamically compress the text to gen-

---

erate summaries. We introduce RPB, an anchor- based dynamic compression algorithm that con- denses the original text, making RbS faster and more memory-efficient than one-step approaches. Without resorting to expert-grade annotation tools or large-scale dialogue-related pretraining tasks, ex- perimental results on AMI and ICSI datasets show that RbS outperforms various previous approaches and reaches state-of-the-art performance.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
George Chrysostomou and Nikolaos Aletras. 2022. An empirical study on explanations in out-of-domain settings. In Proceedings of the 60th Annual Meet- ing of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 1920-6938, Dublin, Ireland. Association for Computational Linguistics.
Xiachong Feng, Xiaocheng Feng. Bing Qin, and Xin- wei Geng. 2021a. Dialogue discourse-aware graph model and data augmentation for meeting summariza- tion. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3808-38 14. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Limitations
Our exploration of the summary algorithm focuses on the traditional summary model. However, it is worth noting that in the era of large language models(LLM),, effective compression of the input of LLM is worth being explored. Our future work should investigate how to effectively compress the input of LLM make more efficient.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021b. Language model as an annota- tor: Exploring DialoGPT for dialogue summarization. In Proceedings of the 59th Annual Meeting of the Association for Compudational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1499-1491, Online. Association for Computational Linguistics.
Ethical Considerations
We use publicly released datasets to train/dev/test our models. Generally, these previous works have considered ethical issues when creating the datasets. For the datasets we used in this work, we manually checked some samples and did not find any obvious ethical concerns, such as violent or offensive con- tent. Source code and the models will be released with instructions to support correct use.
Chin-Wen Goo and Yun-Nung (Vivian) Chen. 2018 Abstractive dialogue summarization with sentence- gated modeling optimized by dialogue acts. 2018 IEFE Spoken Language Technology Workshop (SLTT) pages 735-742.
Karl Moritz Hermann, Tomas Kacisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates. Inc.
A Acknowledgements
We would like to thanks the anonymous reviewers for their valuable and constructive comments. This work was supported in part by the Hong Kong Inno- vation and Technology FInd(ITF) PRP/O79/22FX, InnoHK initiative, the Government of the Hong Kong special administrative regions(HKSAR)) of China, Laboratory for Al-Powered Financial Tech- nologies.
Cheng- -hi Anna Huang, Ashish Vaswani, Jakob Uszko- reit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Din- culescu, and Douglas Eck. 2019. Music transformer. In International Conference on Learning Representa- tions.
Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and By- ron C. Wallace. 2020. Learning to faithfully ratio- nalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459 Online. Association for Computational Linguistics.
References
Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classifi- cation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256-3274, Online. Association for Computational Linguistics.
A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelhart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stol- cke, and C. Wooters. 2003. The iesi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)., volume pages I-I.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/ 409.0443.
Yoon Kim, Carl Denton, Luong Hoang, and Alexan- der M. Rush. 2017. Structured attention networks.

---

In International Conference on Learning Representa- tions.
The ami meeting corpus. Int'l, Conf. on Methods and Techniques in Behavioral Research.
Pieter-Jan Kindermans, Kristof Schütt, Roburt Müller, and Sven Dähne. 2016. Investigating the influence of noise and distractors on the interpretation of neural networks.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2 21(140):1-67.
Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL., Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: new algo- rithm for training recurrent networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Tobias Rohde, Xiaoxia Wu, and Yinhan Liu. 2021. Hi- erarchical learning for generation with long source sequences. CoRR, abs/2104.07545.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguisties.
BART: Denoising sequuence-t-sequeeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Sofia Serrano and Noah A. Smith. 2019. Is attention in- terpretable? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 1931-2951, Florence, Italy. Association for Computational Linguistics.
Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. Keep meeting summaries on topic: Abstractive multi-modal meeting summarization. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 2190- 2196. Florence, Italy. Association for Computational I Linguistics.
Guokan Shang, Wensi Ding. Zekun Zhang. Antoine Tix- ier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2018a. Unsupervised abstrac- tive meeting summarization with multi-sentence com- pression and budgeted submodular maximization. Proceedings of the 56th Anmual Meeting of the As- sociation for Computaaionall Linguistics (Volume 1: Long Pupers), pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Peter J. Liu", Mohammad Saleh*. Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In International Conference on Learning Representations.
Guokan Shang. Wensi Ding, Zekun Zhang, Antoine J.P. Tixier, Polykarpos Meladianos, Michalis Vazirgian- nis, and Jean-Pierre Lorre. 2018b. Ac12018 ab- ssumm. /tttps://iibuue ac12018 8 abs
axzg/dascim/
Zhengyuan Liu, Angela Ng. Sheldon Lee, Ai Ti Aw, and Nancy F. Chen. 2019. Topic-aware pointer-generator networks for summarizing spoken conversations. In 2019 IEEE Automatic Speech Recognition and Un- derstanding Workshop (ASRU), pages 814-821.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-668, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. Coreference-aware dialogue summarization. Pro- ceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509-519, Singapore and Online, Association for Computational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Pro- ceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Ma- chine Learning Research, pages 33 3319-3328. PMIRR
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
lain Mccowan, J Carletta, Wessel Kraaij. Simone Ashby, S Bourban, M Flynn, M Guillemot, Thomas Hain, J Kadlec, V Karaiskos, M Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska Masson, Wilfried Post, Dennis Reidsma, and P Wellner. 2005.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da- Cheng Juan. 2020. Sparse sinkhorn attention. In Proceedings of the 37th International Conference on Machine Learning, ICML/20. JorR.org.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Ilaaa Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
A
Appendix
A.1
Training Details
The parameters of the models are initialized from Huggingface Libraries (Wolf et al., 2020) and up- dated by AdamW optimizer (Loshchilov and Hut- ter, 2019). All experiments are conducted on 8 A100 GPUs. The reconstructor and summarizer are both trained on the AMI and ICSI datasets, For the reconstruction, it takes 9 and 11 minutes to fin- ish the 2,300 and 1,500 steps of training on the two datasets, respectively. The batch size is 256 and the learning rate is 5c-5. For the summarizer, it takes and minutes to finish the 100 steps of training. We use batch size of 64 and a learning rate of 3e-5. Average scores of runs are reported.
Ronald Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neu- ral networks. Neural Computation, 1(2):700-20.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
A.2 Bucketing Algorithm
The size of the assigned buckets varies with the dis- tance between the token and the anchor. The closer to the anchor tokens, the smaller the assigned buck- ets are. For the embedding of anchor tokens, the size of the assigned buckets is always 1. Given a maximum distance d, all token embeddings outside the anchors with distance d will be uniformly as- signed to the same bucket. The serial version of the bucketing algorithm is shown in the Algorithm 1.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020a. Pegasus: Pre-training with ex- tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, 1CML'20 20. JMLR.org.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online, Association for Computational Linguistics.
A.3 Re-using the Reconstructor as Summarizer
We conducted an ablation study investigate the effect of initializing the summarizer with the weight of the reconstructor. The results showed that on AMI, changing the backbone led to 0.99 ROUGE-1 score drop for BART-large and 1.11 for BART-Large-CNE CNN. This suggests that with lim- ited samples, directly migrating between response generation and meeting summarization introduces biases.
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022. Summ": multi-stage summarization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics VVolme 1: Long Papers), pages 1592- 1604, Dublin, Ireland. Association for Computational Linguistics.
A.4 Case Study
Ming Zhong, Yang Liu, Yichong Xu. Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summa- rization. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11755-111)
Table 6 compares the summarization generated by our framework and Summ'N. The red and cyan texts are the contents that appear in the gold sum- marization from AMI (Mccowan et al., 2005). In contrast, the magenta texts are the contents that are not in the gold summarization. orange texts in the gold summarization are the contents covered by RbS but ignored by the Summ"N. The violet texts are the common contents obtained by both RbS and Summ 'N. RbS almost highlights all the key-points
Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xue- dong Huang, 2020. A hierarchical network for ab- stractive meeting summarization with cross-domain pretraining. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 194- 203, Online. Association for Computational Linguis- tics.

---

Algorithm 1: Relative Positional Bucket-
ing
Input:
Relative Positions { [R1 ... R
Bucket number
Max distance d
{B1, B2,
B + 0
b < 2
for 4- do
if R; > 0 then
Bi + B; + Rs *
R4 <- RRs
end
b b
for i f-
do
10
R, < v then
11
B - B; + Rs
12
else
t- v + logí (b v)
13
lgg(d/r)
14
4- min ss 1)
15
B₁ + B; +
end
end
Output: Bucket Number B1, .., Br}
the meeting, while Summ"N missed almost all the key points.
A.5 More Visualization Examples
Figure provide more visual samples of the heatmap to show more intuitively the anchor tokens selected by our framework. Keywords or phrases such as "selling price". "wholesale," and "retail" are captured by RbS compared with the other two different methods.

---

Gold
The meeting opens with the group doing introductions by giving their name and role, betty is the project manager, francina is the user interface specialist eileen is the marketing expert and jeanne is the industrial designer. The project manager tells them they will be designing new remote control that should be original trendy and userfriendly. They will be concerned with functional conceptional and detailed design. To try out the whiteboard, each group member draws their favorite animal on the board. They discuss the project budget and then talk about their experiences with remote controls. They scemed to agree that the remote should be compact and have multi- purpos functions. They also agree that it should do something different that current controls cannot do, and that it should be made of different colors, materials, and shapes. They also discuss way of helping people find the remote when it is lost, signal whether it is beep or light. Then they close the meeting with the project manager going over the tasks they are to complete and telling them they will meet again in about thirty minutes. Selling price will be twenty five curo. Company aims to profit fifty million curo. It should be compact, multi-functional difffrrntt in shape, color, material. Have locator to help find the remote when it is lost. The industrial designer will work on the working design and technical function. The interface specialist will do the working design and functional design. The marketing manager will look for user requirement specifications, such friendliness. The group is not sure if they will have the budget to make the gadget multi- functional, but they would like to make one that would control basically all household machines.
RbS
The project manager introduced the upcoming project to the team members and introduced the name and role of each participant in the project. The team then began a training exercise in which they learned how to use the white board, and practiced drawing on the whiteboard. The project manager also introduced the project budget and the projected profit aim of the project which was fifty million euros. The team then discussed their experiences with remote controls and what features they would like to see in the remote they will be producing. They discussed the features they would like to include in the remote control design, such as color options and different shapes. They also discussed the possibility of adding a locator function to help locate the remote when it is lost. They then discussed what features the remote should have and what price point it should be. The industrial designer will work on the working design. The user interface designer, and the marketing expert will work together on the technical design. Whether to have a light on the remote to help find the remote if it misplaced.
Summ"N 'N
The project manager introduced the project to the team members and went over the agenda. The team members discussed the project budget and discussed the features of the remote. The remote will control televisions, computers, and other household appliances. The group decided that the remote should be small, compact, and have a fancy look and feel. The industrial designer and user interface specialist will work on the technical and functional design. The marketing expert will work with the marketing expert to figure out how to sell the product. The project manager closes the meeting and the project manager gives each team member their individual assignments. They will get instructions to work with and if they have any questions, they can ask them. It was decided that it would be a good idea to include a throw signal to help locate the remote when it is lost. It would be possible to make the remote more fashionable by using different colors and materials and using different shapes. The device will be for televisions only, and will not be for teletext. It should be a cuntii-fnnciooal gadget that controls all houschold machines. It could be used for voice recognition as well as voice recognition. It will be made of plastic and rubber and will be shaped like kidney. There will be no LCD screen, and the remote will have buttons for power, volume, mute, channel-changing channel up/down, channel down, and mute. They were not sure how much the remote would cost to produce. They did not know what the profit aim was for the project. They decided to use a white board to draw their favorite animals on the white board. They also decided to include an indicator on the remote so that it will light up when button is pressed. They discussed how to incorporate the company colors and logo into the design.
Table 6: Case study of RbS

---

(a) Attention weights
(b) Gradients value
(c) Scaled attentions
Figure 6: Visualization of the attention map, gradients, and the scaled attention weights


Reviews:
[{"review_id": "uFn3pPtD0U", "reviewer": "Reviewer_hWuH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper on meeting summarization. We appreciate your feedback and have carefully considered all of your concerns.\n***\n**Concern**: Any data augmentation method is performed?\n\n**Response**: Regarding your first concern about data augmentation methods, we did not use any such methods in our framework. While we acknowledge that successful data augmentation approaches could further improve results, we intend to isolate and highlight the effectiveness of our proposed method.\n***\n**Concern**: AMI has 20 test samples and ICSI has 6, this hurts the confidence of the generalization.\n\n**Response**: We understand your concern about the relatively small amount of data tested in our study, with AMI having 20 test samples and ICSI having 6. However, it is worth noting that multiple previous works[1-4] in the field of meeting summarization have validated their results on these datasets. \nAdditionally, these datasets are regarded as the definitive benchmarks in surveys[5, 6] related to meeting summarization due to their extremely realistic meeting scenarios and accurate labeling of decisions, actions, problems, and meeting minutes. Despite the limited amount of training data and the long sequence lengths, these datasets remain significant challenges in the field.\n***\n**Concern**: the compared baselines are not designed for the task of meeting summarization\n\n**Response**: Regarding your concern about the compared baselines not being designed for meeting summarization, we respectfully disagree. In Table 1, with the exception of our backbone BART model, all other methods[1-4, 7] are standard baselines in the meeting summarization field. These methods have been heavily experimented with on AMI, and those capable of dealing with extra long sequences have been tested on ICSI. In surveys[5, 6] related to meeting summarization, these methods are regarded as landmark methods in the field.\n***\n**Concern**: Does this mean that some previous method inputs the long meeting scripts into a model? The lacking of background makes the contributions of this paper ambiguous.\n\n**Response**: We truly appreciate your feedback on the lacking background of our contributions. In meeting summarization, the majority of work chooses to truncate original texts, with the truncated text typically between 3072 and 8192 in length depending on computational resources. Even after sacrificing raw information to save memory overhead and time, the time complexity of truncating a very long sequence to 3072 is still larger than our framework, and the accuracy lags far behind ours(HAT-BART[8]). Meanwhile, expanding the sequence to 5120[4] or 8192[9] consumes more memory. Our plug-and-play RPB module only applies pooling operation on the embeddings, which greatly enhances generalizability compared to frameworks that rely on sparse attention or longformer-based techniques. \n\nIt is also worth noting that during the inference, baselines that use more memory to retain the original lengthy input will have less memory reserved for the output. As in DialogLM's official released AMI and ICSI generation results, more than **50%** of the summaries are incomplete texts. Whereas we are not limited by sequence length, our proposed RbS can efficiently and accurately cover all decisions, actions, and problems in the meeting transcripts.\n***\nThank you for your valuable feedback. We will take your suggestions into account and improve our final version accordingly.\n\n## References\n[1] [Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization](https://aclanthology.org/P19-1210) (Li et al., ACL 2019)\n\n[2] [A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining](https://aclanthology.org/2020.findings-emnlp.19) (Zhu et al., Findings 2020)\n\n[3] [SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://aclanthology.org/2022.acl-long.112) (Zhang et al., ACL 2022)\n\n[4] [Dialoglm: Pre-trained model for long dialogue understanding and summarization](https://ojs.aaai.org/index.php/AAAI/article/view/21432/21181) (Zhong et al., AAAI 2022)\n\n[5] [Meeting Summarization: A Survey of the State of the Art](https://arxiv.org/pdf/2212.08206) (Kumar, L. P., & Kabiri, A. Arxiv 2022)\n\n[6] [Abstractive meeting summarization: A survey](https://doi.org/10.1162/tacl_a_00578) (Rennard et al., Transactions of the Association for Computational Linguistics (2023) 11: 861–884.)\n\n[7] [Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization](https://www.ijcai.org/proceedings/2021/0524) (Feng et al., IJCAI 2021)\n\n[8] [Hierarchical Learning for Generation with Long Source Sequences](https://arxiv.org/abs/2104.07545) (Rohde et al., Arxiv 2021)\n\n[9] [ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining](https://aclanthology.org/2021.acl-long.535) (Fabbri et al., ACL-IJCNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  It is nice to leverage dialogue history as an unsupervised objective in the stage of reconstruction. The tokens in previous utterances that have a greater impact on the next utterance are more likely to contain salient information. This method is reasonable and effective, as shown in Fig 4.\n2. Another highlight of this article is the compression of redundant meeting transcripts. The anchor token selection and Relative Positional Bucketing provide a simple but effective solution in this situation.", "reasons_to_reject": "1.\tThe datasets for evaluation are both in a limited scale. AMI has 20 test samples and ICSI has 6. This hurts the confidence of the generalization. Any data augmentation method is performed? Or the author may consider other evaluations to prove the generalization for scalability.\n2.\tThe related work review provides little information on the task of meeting summarization. Also, in Table 1, the compared baselines are not designed for the task of meeting summarization. In line 516, the complexity of regular summary models is  O(n2 × d_model). Does this mean that some previous method inputs the long meeting scripts into a model?\nThe lacking of background makes the contributions of this paper ambiguous.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes the approach, Reconstruct Before Summarize, for meeting summarization, where the contexts are lengthy and noisy casual conversations.\nRbS consists of a reconstructor and a summarizer. The reconstructor locates the salient information when reconstructing the next utterance. The summarizer compresses the long context based on the distance to the salient information and generates a summary. The performance achieves SOTA on meeting summarization.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tj84COcT0o", "reviewer": "Reviewer_NpwE", "summary": "", "questions": "* For the ablation analysis in 4.1, naively deleting or substituting the anchor tokens would corrupt the syntactic and semantic structure, not to mention that it’s done at  token-level. Have authors accounted for this?\n* For the same analysis, does \"high-frequency tokens\" imply that many of them are function words? Or were those excluded?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback. We appreciate your thoughtful comments and suggestions, which help us to improve the quality of our work.\n***\n**Concern**: After aggregating the embeddings, how to get the final summary?\n\n**Response**: Regarding your concern about how our summarizer generates the final summary after aggregating the embedding, we apologize for any confusion caused by our writing. We have made revisions to our paper to clarify the process. \n\nThe difference between our summarizer and the original BART is only the addition of the relative postitional bucketing(RPB) module, which is inserted between the embedding layer and the first attention layer. RPB compresses the original long embedding to an acceptable length, the rest of the process remains the same. As usual, BART takes the compressed embeddings and processes them with MLP and attention layers, and generates the final summarization. The whole process is not different from the original BART, except that BART gets the compressed embedding instead of the original one.\n***\n\n**Concern**: Run and compare the efficiency between the proposed method and baselines.\n\n**Response**: We also appreciate your suggestion to compare the efficiency of our proposed method with baselines. We will include a comparison of runtimes and memory consumption in the final version of our paper. However, one important issue is that when truncating the meeting transcripts to 3072 - 5120 for the training of BART baselines, the batch size can only be set to less than 4 * 8 = 32 on 8 40G A100, and the batch size is even lower when untying the weights of the embedding and the output layer. In our approach, the total memory usage is only 71.18% with a total batch size of 64, we didn't further increase the batch size because the size of the training set is too small(97 samples for AMI, 42 samples for ICSI). RbS can training with a larger batch size, which not only improves learning efficiency, but also reduces I/O time. We will include a detailed comparison of computation time and I/O time in the final version.\n***\n**Concern**: Statistical testing\n\n**Response**: We have added a comparison of statistical testing relative to DialogLM and Summ^N, as you suggested.\nIt is worth noting that, since we cannot reproduce their results, we chose their results in the original paper as a reference, and we take their ***average results of three runs*** in the original paper and calculate the p-values. The results reported in the DialogLM paper are the average of three runs, while the Summ^N results do not specify whether their results are the average or the maximum. Here we default the Summ^N results to the average of three runs as well. The results of the significance tests are as follows:\n1. On the AMI:\n    - RbS vs. DialogLM: The z-value is 1. The p-value is .15866. The result is **not significant** at **p < 0.10**.\n    - RbS vs. Summ^N: The z-value is 2.33333. The p-value is .00982. The result is **significant** at **p < 0.01**.\n    - RbS-CNN vs. DialogLM: The z-value is 3. The p-value is .00135. The result is **significant** at **p < 0.01**.\n    - RbS-CNN vs. Summ^N: The z-value is 3. The p-value is .00135. The result is **significant** at **p < 0.01**.\n   \n2. On the ICSI:\n    - RbS vs. DialogLM: The z-value is 2.33333. The p-value is .00982. The result is **significant** at **p < 0.01**.\n    - RbS vs. Summ^N: The z-value is 3. The p-value is .00135. The result is **significant** at **p < 0.01**.\n    - RbS-CNN vs. DialogLM: The z-value is 1. The p-value is .15866. The result is **not significant** at **p < 0.10**.\n    - RbS-CNN vs. Summ^N: The z-value is 3. The p-value is .00135. The result is **significant** at **p < 0.01**.\n***\n**Concern**: naively deleting or substituting the anchor tokens would corrupt the syntactic and semantic structure\n\n**Response**: We partially agree with this point of view. However, it is noticed that for meeting transcripts with 5k+ tokens, with 6.4% anchor tokens, removing 25% anchors normally only deletes around 80 tokens. Usually, removing/replacing 80 words in a sequence of 5000 words, from an intuitive point of view, will not have a great impact on the semantics of the original sequence. Unless extremely important keywords are removed. So our experiment proved to a certain extent that the anchor words are mostly some keywords. \n\nBesides, to further prove the effectiveness of the anchors, we did an experiment in the ablation study(See Table 2.), that is, sorting all the anchors according to the importance score and taking 25% as a fraction, and then removing one fraction at a time to see the changes of the performance. It can be seen that removing the top 25% of anchors causes the greatest loss of information in the original text. Through this experiment, we proved that anchors with higher importance scores are more important.\n***\n**Concern**: Does \"high-frequency tokens\" imply that many of them are function words? Or were those excluded?\n\n**Response**: \nWe apologize for overlooking your point about whether \"high-frequency tokens\" imply that many of them are function words or not. For some transcripts, we found that the high-frequency words are mainly filler words and stop words, while for others, high-frequency words are mainly some high-frequency nouns in the meeting content. In our final version, we will consider replacing the anchors with high-frequency words other than filler words and stop words to further verify the effectiveness.\n***\n\nWe are very grateful for your feedback regarding the typos, grammar errors, and readability of the figures. We will make the necessary corrections in the final version.\nThank you again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Proposed mechanisms are reasonable and well-motivated, leading to improvement in ROUGE scores.\n* Extensive ablation study is conducted to support the validity of each component.\n* Outcome is the state-of-the-art method for both AMI and ICSI.", "reasons_to_reject": "* Some key details are missing, making it difficult to understand how exactly the summaries are generated. In particular, the explanation on the summarizer is lacking (after aggregating the embeddings, how to get the final summary).\n* Besides the time-complexity analysis, it would have been nice to run and compare the efficiency between the proposed method and baselines.\n* While the results look better, it’d be better to run statistical testing to support statistical significance.", "questions_to_authors": "* For the ablation analysis in 4.1, naively deleting or substituting the anchor tokens would corrupt the syntactic and semantic structure, not to mention that it’s done at  token-level. Have authors accounted for this?\n* For the same analysis, does \"high-frequency tokens\" imply that many of them are function words? Or were those excluded?", "paper_topic_and_main_contributions": "This paper proposes a new algorithm for summarizing meeting minutes. Motivated by the fact that meeting transcripts tend to be long and that the topics vary within a single meeting, authors propose a two-step approach to obtain meeting summaries. The first stage is responsible for finding the pivotal keywords and expressions according to the \"reconstructor\" model, by utilizing the scaled attention weights by the model. To do this, BART is fine-tuned on the response generation task. After annotating the important words in the meeting, the second stage groups tokens centered around each important keyword and aggregate them according to relative positions. Finally, the aggregated embeddings are used to generate the summary.\n\nAuthors experimented the proposed method on AMI and ICSI and obtained state-of-the-art results in terms of ROUGE scores. Analyses were conducted for each stage to confirm the role and positive impacts of each proposal toward the final results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Add spaces before parentheses in the Figure 1 caption.\n* L277 \"DialogGPT\" instead of DialoGPT\n* L214 remove \"the\" T5\n\n* Figure 5: Too small to read. Use smaller figure size dimensions for better readability"}}, {"review_id": "ypLT6uyOgl", "reviewer": "Reviewer_9DKz", "summary": "", "questions": "None", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our paper. We appreciate the time and effort you put into reviewing our work. We have carefully considered your comments and suggestions and have made the following responses:\n***\n**Concern**: 8 A100 GPUs are luxuries.\n\n**Response**: Indeed, 8* A100 are luxries. However, it is worth noting that our approach is platform-independent. That is, no matter what platform the algorithm runs on, our proposed RbS is more efficient than most baselines(Section 4.4). We use 8 * A100 not because our algorithm relies on the efficiency of the A100 platform, but because we have no other choice but to use A100. \n\nOne important issue is that when truncating the meeting transcripts to 3072 - 5120 for the training of BART baselines, the batch size can only be set to less than 4 * 8 = 32 on 8 40G A100, and the batch size is even lower when untying the weights of the embedding and the output layer. In our approach, the total memory usage is only 71.18% with a total batch size of 64, we didn't further increase the batch size because the size of the training set is too small(97 samples for AMI, 42 samples for ICSI). This indicates that the advantages of our approach in terms of memory usage will be even more pronounced on lower-resource computing platforms.\n\nBesides, It is worth noting that DialogLM is also trained and tested on 8 * A100 platforms. However despite this abundance of computational resources, many important decisions, actions, and problems were not covered in the summaries generated by DialogLM. We found that for all the DialogLM's official released generated summaries in their github repo, **twelve out of twenty AMI summaries** are incomplete texts. We speculate that the reason behind this is that after DialogLM generates a part of the summary, extremely long inputs and outputs consume a lot of memory, making it impossible to generate more tokens due to memory limitations. This phenomenon was also observed in the ICSI dataset, where **three of the six summaries** of DialogLM were extremely incomplete and unreadable.\n\nCompared to DialogLM, we have the same 8*A100 computational platform, but our summaries are complete and are not constrained by the memory pressure imposed by sequence length. Our results cover the decisions, actions, and problems encountered in the meeting transcripts.\n***\n\n**Concern**: There are other directions to be explored, such as a comparison with a linguistic-based (based on the domain knowledge of e.g. speech acts) approach could be a good compliment to the study.\n\n**Response**: We truly appreciate that you pointed this out, we think it will help us improve the overall completeness of the paper. We compared some of the approaches that utilize linguistic knowledge to generate abstracts, such as dialogue acts[1], and discourse relationships[2]. Indeed, we emphasized in the paper that we annotate the key points without the help of specialized linguistic tools. However, we did not analyze them in depth in the paper. I apologize for this, and we will do a more in-depth comparison and analysis in the final version. \n***\n\nThank you again for your thoughtful feedback!\n\n## References\n[1] [Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts](https://ieeexplore.ieee.org/document/8639531)(Goo et al., IEEE SLT 2018)\n\n[2] [Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization](https://www.ijcai.org/proceedings/2021/0524) (Feng et al., IJCAI 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In the era of solving every task with large language models, it is especially noteworthy when some effort is made to enhance the old-fashioned methods based on less costly and energy-consuming models. Moreover, since the proposed method is effective (as shown in the results) and based on a deeper understanding of the data and the task, it is a clear advantage. Some hypotheses were made about the particular text structure resulting from transcribing a multi-speaker meeting, like the distance of salient information from an anchor token, and based on them, a novel variation of existing approaches is introduced. The paper is mainly well-written; the explanations should allow for replicas. Other approaches has been briefly discussed and explained. The visualizations and tables are clear and helpful. Beside the main report on experimental setup and results there is also a deeper analysis offered. ", "reasons_to_reject": "One of the central claims is the model's efficiency - but in the appendix, the reported computation time includes 8 A100 GPUs which is a luxury itself. Reporting the computation time on a more moderate set would be only fair.  \nEspecially, since the conlusion reads \"However, large-scale datasets, excessive optimization steps, and sufficient computational resources are necessities and luxuries\".\nThe limitation section only includes focus on LLMs, whereas there are other directions to be explored, such a comparison with a linguistic-based (based on the domain knowledge of e.g. speech acts) approach could be a good compliment to the study. \n", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The paper proposes a two-step approach to summarizing long meeting transcripts called Reconstruct before Summarize (RbS). The first step includes reconstruction via the forward pass of language model with a teacher forcing and then finding silent words by retracing the anchor tokens. The contribution scores of the anchor tokens candidates are either averaged or counted with multi-view voting). Then the context window surrounding the anchors is compressed and embedded for the summarizer based on BART or BART-CNN. The originally long transcript is first compressed (by means of relative positional bucketing) to its salient essence and then subject to summarization, allowing for significant size reduction while still preserving the most critical information (i.e. in the AMI dataset from the average ca 6000 to 300 words). The authors claim their concept to be effective (outperforming other models in ROUGE-1, ROUGE-2, and ROUGE-L metrics on AMI and ICSI datasets) and particularly efficient since it does not require large language models or additional annotation of salient words. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "Reconstrcut before Summarize(RbS)  line 66"}, "extra_text": {}}, {"review_id": "A3gFHd0zEM", "reviewer": "Reviewer_KpoB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our submission and providing us with your valuable feedback. Your feedback has been immensely helpful in improving our work. We would like to address your concerns as follows:\n\n***\n**Concern**: The description of the methodology is mostly unclear\n\n**Response**: \nRegarding the methodology, we apologize for any confusion caused and fully understand the importance of providing a clear and concise explanation. In the final version, we will provide a more detailed explanation of our methodology, including the bucketing and anchoring algorithms, in a manner that is easy to understand. We appreciate your suggestion and will make every effort to ensure that the methodology is presented in a professional manner.\n***\n\n**Concern**: evaluation issue: automatic-only (furthermore, word overlap-only) metrics are not enough in assessing summarization accuracy\n\n**Response**:  With regard to the evaluation issue, we agree that automatic-only and word overlap-only metrics may not be sufficient in assessing summarization accuracy. As a result, we attempted to organize a human evaluation during the rebuttal period. \nFor the generated summaries released in the DialogLM repo, they didn't indicate the corresponding transcript for each summary. Given one generated summary produced by DialogLM, we had to ask the reviewers to try to figure out its corresponding meeting transcript, and then evaluate the quality. The extraordinarily lengthy transcripts made their work extremely difficult. In the end, we performed a human evaluation of only five samples due to the workload. The results of the remaining samples will be finalized and added to the final version. \n\nThe human evaluation involved two postgraduate MSc. students and an individual engaged in clerical work. They were asked to determine, based on the meeting transcripts, which generated summaries that better covered the important decisions, actions to be taken, and problems encountered in the meeting. They were also required to compare the quality of the final summaries generated. Throughout the process, they were not told which model the results came from. We aggregated the results from all three reviews and tallied the RbS's winning rate as below.\n***\n|               | Win Rate |\n|---------------|----------|\n| Decisions     | 3 / 5    |\n| Actions       | 3 / 5    |\n| Problems      | 4 / 5    |\n| Final Quality | 4 / 5    |\n***\nWe observed that in each data sample, the final summary generated by our framework contained a summary of the meeting procedure, followed by the actions to be taken, the meeting decisions, and the problems encountered in the meeting. In contrast, many important decisions, actions, and problems were not covered in the summaries generated by DialogLM. We then found that for all DialogLM's official released generated summaries in their github repo, **twelve out of twenty AMI summaries** are incomplete results. We speculate that the reason behind this is that after DialogLM generates a part of the summary, extremely long inputs and outputs consume a lot of memory, making it impossible to generate more tokens due to memory limitations. This phenomenon was also observed in the ICSI dataset, where **three of the six results** of DialogLM were extremely incomplete.\n\n***\nWe appreciate your comments and will make every effort to fix the grammatical errors in the final version and to present the methodology in a more understandable manner. Thank you for your help and for the opportunity to address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new method for long input summarization is proposed\n* It attains state-of-the-art results among non-LLM methods in terms of Rouge score\n* anchoring and bucketing also results in a reduced time complexity", "reasons_to_reject": "* description of the methodology is mostly unclear\n* \"Reconstruct Before Summarize\" is not fluent - this could have been in the typos/grammar section as well, but it's in the paper title", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a two-step framework for the abstractive summarization of meetings, Reconstruct Before Summarize (RbS). It works by first, selecting the key points in the input (anchor tokens) in a self-supervised way, and second, summarizing using the Relative Position Bucketing algorithm which compresses the input and makes the method more memory and computationally efficient.\n\nWith BART-large/BART-large-CNN as the backbone, the authors evaluate their approach on AMI and ICSI datasets against a series of baselines and find out that RbS attains the best performance on both datasets and most metrics (Rouge-1/2/L). They further perform ablation study on the contribution of anchor tokens, scoring methods and bucketing to the overall method.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Paper text needs proof-reading/grammar check throughout"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper brings merits of a reconstructor locates the salient information when reconstructing the next utterance, and improves meeting summarization. The paper is expected to added more details and discussions as mentioned in the rebuttal.
Decision: Accept-Main