FREDSum: A Dialogue Summarization Corpus for French Political Debates
Authors: Virgile Rennard
Guokan Shang
Damien Grari
Julie Hunter
Michalis Vazirgiannis
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue
Summarization
Corpus
Dataset
Political Debate

Abstract: Recent advances in deep learning, and especially the invention of encoder-decoder architectures, have significantly improved the performance of abstractive summarization systems. While the majority of research has focused on written documents, we have observed an increasing interest in the summarization of dialogues and multi-party conversations over the past few years. In this paper, we present a dataset of French political debates for the purpose of enhancing resources for multi-lingual dialogue summarization. Our dataset consists of manually transcribed and annotated political debates, covering a range of topics and perspectives. We highlight the importance of high-quality transcription and annotations for training accurate and effective dialogue summarization models, and emphasize the need for multilingual resources to support dialogue summarization in non-English languages. We also provide baseline experiments using state-of-the-art methods, and encourage further research in this area to advance the field of dialogue summarization. Our dataset will be made publicly available for use by the research community, enabling further advances in multilingual dialogue summarization.

FREDSum: A Dialogue Summarization Corpus for French Political
Debates
Virgile Rennard Guokan Shang', Damien Grari Julie Hunter', Michaali Vazirgiannis
Linagora, France École Polytechnique, France 'Grenobee Ecole de Management, France virgile@renmard. org guokan. shang@polyttechniqueee. edu damien. dwm.en.grarienneee. grari@grenoble--m com jhunter@linagora.com mvazirgelix. pol.eeeiieellttttciuuuuue.ee fr
Abstract
Ng. 2019), due in part to the fact that high quality data is readily available online, leading to the cre- ation even multi-lingual datascts for news sum- marization (Hermann et al., 2015; Kamal Eddine et al., 2021; Evdaimon et al., 2023), and scientific articles (Nikolov et al., 2018).
Recent advances in deep learning, and espe- cially the invention of encoder-decoder archi- tectures, has significantly improved the perfor- mance of abstractive summarization systems. The majority of research has focused written documents, however, neglecting the problem multi-party dialogue summarization. In this paper, present a dataset of French political debates' for the purpose of enhancing resources for multi-line multiilinggull dialogue summarization. Our dataset consists of manually transcribed and annotated political debates, covering a range of topics and perspectives. We highlight the importance of high quality transcription and an- notations for training accurate and effective di- alogue summarization models, and emphasize the need for multilingual resources to support dialogue summarization in non-English lan- guages. We also provide baseline experiments using state-of-the-an methods, and encourage further research in this area advance the field of dialogue summarization. Our dataset will be made publicly available for use by the research community.
Such data is much harder to come by for dia- logue, although there are a few corpora of multi- party conversation and chat that have been painstak- ingly created with the aim of furthering research on abstractive meeting and conversation summariza- tion. Some examples include the AMI (Mccowan et al., 2005), ICSI (Janin et al., 2003) and ELITR (Nedoluzhko al., 2022) corpora for meeting sum- marization (Shang, 2021) and the SAMSum cor- pus (Gliwa et al., 2019) for chat summarization (cf. Rennard et al., 2023; Feng et al., 2021, for more dis- cussion.) However, due to the particular nature of dialogical interactions and even different types of dialogue, the relatively small size of conversation data sets compared to those for document summa- rization and other difficulties posed by spontaneous speech (Rennard et al., 2023), dialogue summariza- tion remains an open task.
1 Introduction
Another significant problem for the task of di- alogue summarization is the lack of multi-lingual data. While ELITR (Nedoluzhko et al., 2022) con- tains some meetings Czech, and VCSum (Wu et al., 2023), meetings in Chinese, there are cur- rently no datasets available for multiparty conver- sation in other non-English languages. This im- poses significant limitation on the development of solutions for dialogue summarization in multi- lingual situations. There is thus significant need for multilingual dialogue summarization resources and research.
Summarization aims condensing text into a shorter version that optimally covers the main points of a document or conversation. Extant mod- els adopt one of two approaches: while extractive summarization aims at creating optimal subset of the original source document by simply select- ing the most important sentences without chang- ing them, abstractive summarization aims at re- formulating the most important points with novel sentences and words that may not have been seen before in the text. As such, abstractive summariza- tion is natural language generation task that relies more deeply on the meaning of the source text.
In this paper, present FREDSum, new dataset for FREnch political Debate Summariza- tion and the first large-scale French multi-party summarization resource. FREDSum is designed to serve as a valuable resource for furthering research
A significant research effort has been dedicated to traditional document summarization (Lin and
cnttps://gitaee com/ com./Veeenard//rreSuu
http:///tttuu.cccoooeeeeee

---

Dataset
Lang.
#Transcripts #Words/trans. #Turns/trans. #Speakers/trans. #Words/sum.
dialog
SAMSum
EN
16369
93.8
11.2
2.4
20.3
DialogSum
EN
13460
131.0
23.6
MediaSum
EN
463596
1553.7
30.0
6.5
14.4
AMI
EN
137
6007.7
535.6
4.0
296.6
mecting
ICSI
EN
59
13317.3
819.0
6.3
488.5
MeetingBank
EN
6892
3800.3
146.9
3.2
87.2
VCSum
CN
239
14106.9
73.1
5.6
231.9
ELITR
EN/CS
179
7549.9
884.5
6.5
327.9
debate
FREDSum
FR
142
2595.5
54.2
4.2
238.9
FREDSum preS
FR
1140
71386.8
685.0
53.4
IREEDSumpeeA
FR
4563
28619.8
445.8
56.0
Table Comparison between FREDSUM and other news, dialogue or meeting summarization datasets. # stands the average Essut..REDSuupres and FREDSumpreA represent the pretraining datasets released along FREDSum.
on dialogue summarization in French and multilin- gual dialogue summarization more generally. Each debate in the corpus is manually transcribed and annotated with not only abstractive and extractive summaries but also important topics which can be used as training and evaluation data for dialogue summarization models. The debates were selected with an eye to covering a range of political top- ics such as immigration, healthcare, foreign policy, security and education, and involve speakers with different perspectives, so as to not over-represent party ideology.
ing is conducted, or even the downstream tasks for which dataset is constructed. The data and the models used to treat can indeed be very diverse. Debates are multi-party discussions that are gen- erally long and fairly well-behaved both in the cleanness of the language used and their tight orga- nization. Most debates follow the same structure, an initial topic is given, followed by an assertion from a speaker, refutal by others with alterna- tive, and a further refutal by the original speaker. This systematic organization lends itself well to the study of discourse structure and dialogue dynam- ics.
We highlight the importance of high quality tran- scription and annotations for training accurate and effective dialogue summarization models. We also present baseline experiments using state-of-ther -f-tht-art methods and encourage further research this area to advance the field of dialogue summarization. Our goal is to promote the development of mul- tilingual dialogue summarization techniques that be applied not only political debates but also other domains, enabling a broader audience to engage with important discussions in their native language.
In this section, we aim at explaining the elements of annotations provided in FREDSum as well the instructions followed by the annotators.
2.1 Data Collection, and curation
Data collection and curation are critical compo- nents any data driven project. Specifically, in the case of political debates, dataset needs to cover a wide range of political fields, such as economic policies, social issues and foreign polities, but also needs to be representative of the landscape of the country's 's politics over the years and thus should not fail represent any of the major political par- ties. The debates in FREDSum thus cover diverse range of topics from 1974 to 2023 of political de- bates.
2 FREDSum : FREnch Debate Summarization Dataset
Due to the heterogeneous nature of conversations, different dialogue summarization datasets serve dif- ferent purposes. The uses and challenges inherent to chat data, film meeting transcripts, Twitter threads, and other kinds of daily communication are heterogeneous. Differences may stem from transcript length, the level of disfluencies, the spe- cific speech style of, say, chatrooms and Twitter threads, the goal oriented manner in which a meet-
Due to the length of political debates, which can go for mutiple hours and touch wide vari- ety of specialized topies, each debate by topic. As such, presidential de- bate that touches on foreign affairs, social issues and economic policy with little to no link between these topics can easily be considered as multiple
decided segment

---

standalone debates. The specific way in which televised debate is held, where the topic shift is explicitly conducted by the moderator, whose turn does not impact the debate in any other way, gives the viewer a clear separation of topics that can be used as the segmentation between sub-debates. Each of the debates was transcribed by one an- notator, and corrected by another; subset of the debates for the second tour of the presidential elec- tions had available transcriptions online?. After collecting all of the debates, we asked NLP and po- litical experts to annotate them for multiple tasks. All annotators of the dataset, whether for summary creation or human evaluation, are native french speakers specialized either in computer science or in political science. The following annotations have been conducted all debates:
and humans tend prefer abstractive summariza- tion for conversation (Murray et al., 2010).
Instructions. Considering that debates, unlike meetings and other long, spontaneous discussions, are held between multiple well prepared speakers, some of the classical problems of extractive sum- marization for dialogues are diluted (there are rela- tively few disfluencies and hesitations). However, they are still present. The annotators have been in- structed to (1) Create short summaries, (2) extract the most important pieces of information as well as their context; specifically, extract any specific propositions given by a candidate, (3) extract entire sentences, preceded by the speaker's name, (4) ex- tract rhetorical clauses and conclusions, (5) make summary that is, as much as possible without going against (1), readable as a standalone dialogue.
1. Abstractive Summarization
2.2.2 Abstractive Summaries
2. Extractive Summarization
In contrast to extractive summaries, abstractive summaries are more readable and engaging. On the other hand, they are more complicated to design, as well as generate.
3. Abstractive Community annotation
Each of the debates contains three different ab- stractive summaries, two extractive summaries, and two sets of annotations for abstractive communi- ties, as explained below. The data is released on Github
To construct abstractive summary, the follow- ing steps can be taken: (I) Identify the main points of the text, (2) Write novel sentences that concisely summarize them, (3) Check the text for coherence and readability, as well as completeness.
2.2
Annotations
Debates often follow systematic structure. After the moderator opens the subject, the first candidate who replies provides summary of what the party in place did right/wrong, and then explains what he or she wants to propose - the second one does the same, providing a summary and proposing how best to proceed. After this, the different propositions are discussed in terms of budgeting and feasibility.
Instructions.
In this section, we explain the rules we followed to create each type of annotation.
2.2.1 Extractive summaries
An extractive summary is used to condense a longer text into shorter text that includes only the most important information, used to provide readers with an overview of longer text without requiring them to read the entire document. An effective extractive summary needs to capture the main points of the text while removing unnecessary details to ensure the summary is as condensed as possible, while still giving context to support the main claims.
The question arises to what a summary should contain. Should simply explain the propositions enough to summarize both positions, or should descriptions of rebuttals also be included? Many debates contain heavy ad-hominem subscctions. with personal accusations that are not political - should an accurate summary also mention them?
While extractive summaries lend themselves well to the summarization of traditional documents, the nature of spontaneous speech, especially for long discussions, leads strong preference for abstractive summarization. Indeed, spontaneous speech has many disfluencies, and long conversa- tions tend to have sentences that have repetitions or trail off. The density of information varies greatly,
In FREDSum, the annotators were asked to col- leet the candidates summaries, propositions, and the main counter points given, while removing much as possible any information unnecessary for describing the general idea of the debate.
Annotators started each summary with list the participants, as well as the main theme the
Phttps: //www. vie wiw-publliiieeeee lubbique.ffr

---

debate. Then, the annotators made three summaries in three different styles vary the level of abstrac- tivity. The first type of summary seeks to avoid the problem coreference where possible, by opting for proper names in place of personal pronouns. The second type is an abstractive summary pro- duced based on extractive summary, while the third type involves the conception of summaries as natural documents.
pretraining data is needed. In the scope of French polities, few data are readily available (Koehn, 2005; Ferrara, 2017; Abdine al., 2022), and only a small part of this data comprised of dia- logues. In the hope of further providing data for pre-training, we propose to release a preprocessed corpus of dialogue in French for pretraining models on French political data, containing all of the sen- questions from 1997 as well the Parliament sessions from 2004 onwards.
2.2.3 Abstractive Communities
To gather the necessary data, we scraped the of- ficial websites of the French Senate and Parliament to obtain publicly available datasets. The original data consisted of both HTML pages as well as pdfs that have been processed and formatted to ensure their cleanliness as well as consistency for use in NLP tasks.
Abstractive
abstractive summary
transcription
summary
The aim of gathering this pre-training data is two-fold. Having the data readily available for pre- training will help with a variety of tasks, such as abstractive summarization debates, but can also provide a useful resource for further studies on soci- ological aspects of the evolution of French politics throughout the Fifth Republic, and can therefore be used as foundation not only for developing NLP models, but also for diverse subjects touching on French politics.
Figure 1: The creation process of abstractive sum- mary and abstractive communities. From a source tran- scription, we create an extractive summary, then group parts of it in communities of extractive sentences sup- porting a same topic, summarized by one abstractive sentence
In the scope of the dataset, abstractive commu- nities are groupings of extractive sentences clus- ters that each support a sentence in the abstractive summary; in this way, each of the sentences of the abstractive summary are supported by one or more of the extractive utterances.
Statisties
FREDSum is comprised of 18 debates segmented into 138 sub-debates by topic. Each of the sub- topics have two extractive summaries well three abstractive summaries. Extractive summaries are naturally longer than their abstractive counter- parts; extractive summaries contain an average of 684 tokens while abstractive summaries have an average of 239.
To construct abstractive communities, NLP system first identifies the key phrases and sentences in the transcript that represent the main ideas and arguments. then groups these phrases and sen- tences together based on their semantic similarity, using techniques such as clustering or topic model- ing. The resulting groups of phrases represent the abstractive communities.
Experiments
The instructions for creating ab- stractive communities are fairly straightforward, provided the annotators with both the source de- bate acquaint themselves with the data, well as both the extractive and abstractive summary done by each the annotators. The annotators were then asked to use Alignmeet (Polák et al., 2022) to create the communities.
Instructions
4.1 Abstractive summarization
In this section, we aim to evaluate three differ- ent representative models on our abstractive sum- maries. The three models we choose to use are BARThez (Kamal Eddine et al., 2021), Open As- sistant (Köpf et al., 2023), and ChatGPT (Ouyang et al., 2022). While ChatGPT and Open Assis- tant represent models from the newer paradigm of NLP large language models based on prompt- ing, with Open Assistant relying on LLAMA 30B we also decide to showcase the results previ- ous state of the art french models for generation
2.3 Pretraining corpora
order to have efficient systems for summariza- tion or other generational tasks, a large amount of

---

Debate
exchange and books from Project Gutenberg, as well as from many volunteers. It is based on LLAMA 30B. In this benchmark, we used the OA_ ____F__aam___00 30B__ model, with the preset of k50-Precise, as well as the default parameters. Con- sidering the low amount of tokens it can take into account (1024), we have followed the same rule as with BARThez for the experimentation. Open Assistant would not give summaries depending on the position of the prompt thus, without much prompt engineering, we provided it the debate seg- ment with the prompt "Summarize this debate be- tween X,Y
Alors, l'autre question euh qui ap- parue, qui était très nette pendant cette cam- pagne, c' j'est le pouvoir d' d'achat. Vous en avez fait un des emblèmes de de votre campagne, euh et je viens euh vers vous, peut-être euh Jordan Bardella. [...]
Pl:
JB: Nous le compensons. Le chiffrage, il est public, les français peuvent le voir sur le site de campagne de Marine Le Pen. Un budget de soixante-huit milliards qui est présenté I' T'équilibre.[..]
OV Tout d'abord. pardon, je pense qu'il y a un souci. [...] La totalité des prestations sociales versées aux étrangers chaque année c' c'est neuf milliards d' d'euros. [...] Summary
ChatGPT (Ouyang et al.. 2022). ChatGPT is an LLM made by OpenAT. In this benchmark, we used the online free web interface based on GPT 3.5, with the May 24th version, generate the summaries. ChatGPT has a much higher token limit and only few debates had to be separated into sub segments. When they needed to be, the same rule was followed as with BARThez. ChatGPT was simply given a prompt asking "Summarize this debate between X,Y".
Dans ce débat, Olivier Véran et Jordan Bardella parlent du pouvoir d' achat. Jordan Bardella commence en disant que le budget de son parti est à I' l'équilibre. [...] Il explique que des économies de seize milliards sont faites en supprimant les aides sociales aux étrangers.
We evaluate each of the models with the stan- dard ROUGE metrie (Lin, 2004), reporting the Fi scores for ROUGE-1, ROUGE-2 and ROUGE-L. Considering the known shortcomings of ROUGE (Lin, 2004), and its focus on lexical matching, WC also propose to evaluate the summarics with BERTScore (Zhang et al., 2020). BERTScore takes into account semantic similarity between the generated and reference summary, complementing ROUGE and its lexical matching.
Table 2: An example of a debate annotation, purple text denotes extractive summary.
with BARThez.
The experiments involving ChatGPT and Open Assistant were conducted using the web interface and minimal prompt engineering. The aim was utilize the models as regular users would. The prompts were identical for both models, consisting of the debate input and the request to "Summa- rize this debate between [debate members]." No- tably, Open Assistant required the question for sum- marization to be placed at the end of the prompt, whereas ChatGPT did not have this requirement. BARThez (Kamal Eddine et al., 2021). In the case of BARThez, we want to show the results of tradi- tional encoder-decoder systems to dialogue sum- marization. BARThez is based on BART (Lewis al., 2020), and fine tuned for summarization on OrangeSUM. Being based on BART, it can only handle short inputs. For these experiments, we truncated the transcripts into the minimal amount of segments of 512 tokens, and concatenated the output summary for each of the inputs to create the final summary.
ROUGE (Lin, 2004). The standard metric for both dialogue and general text summarization, scores system produced summaries based purely on sur- face lexicographic matches with a gold summary. We used a publicly available implementation". Rouge-L. is sentence-level.
BERTScore (Zhang et al., 2020). BERTScore is a metric used to evaluate the quality of text generation by comparing them reference texts. BERTScore leverages BERT's contextual embed- dings to measure the similarity between the gener- ated text and the reference text at the token level. It takes into account both the precision and recall of the token matches, providing a comprehensive eval- uation of the text's quality. We used the original implementation! we use bert-base base-mmlilinngulll
Open Assistant (Köpf et al., 2023). Open as- sisstant is an LLM fine-tuned on reddit, stack
wewelleeereeecchttteeeeeeee tree htwpssarcchtreee/msttr////// master/rouge "ht / t/tbbbb...ttttt//// thub. com/Tiiiigerbbertsccoe

---

R-1
R-2
R-L
BERTScore
R-1
R-2
R-L
BERTSScoe
BARThez
39.38
11.77
18.31
69.74
LongestGre
68.95
54.67
33.21
78.80
Open Assistant
39.20
11.31
19.00
71.85
Textrank
63.31
43.89
44.29
80.69
ChatGPT
48.93
19.61
25.60
75.30
CoreRankSub
61.22
41.24
26.96
75.54
BERTExtSum 64.70
45.01
44.96
80.94
Table 3: ROUGE and BERTScore Fl abstractive sum- marization scores.
Table 4: Extractive summarization results with a budged of 684 words (Average length of extractive summaries)
cased, without rescale_ Both for ROUGE and BERTScore, when multi- ple extractive or abstractive reference summaries exist, we score the candidate with each available reference and returning the highest score.
R-1
R-2
R-L
BERTScore
LongestGre
39.74
10.54
16.45
66.60
Textrank
38.20
10.07
16.92
66.86
CoreRankSub
36.97
10.66
15.91
67.82
These results suggest that ChatGPT outperforms both BARThez and Open Assistant in terms of ab- stractive summarization, demonstrating stronger performance across every evaluation metric.
BERTExtSum 40.38
11.16 17.71
67.67
Table 5: Extractive summarization results with budged of 239 words (Average length of abstractive summaries)
4.2 Extractive summarization
used publicly available implementation"
We cast the experiments with our annotated ex- tractive summaries in terms of the budgeted ex- tractive summarization problem, i.c., generating summary with desired number of words. The budget is defined the average length of human extractive summaries, which in our case is 684 words. Following common practice (Tixier al., 2017), we also compare with human abstractive summaries, where the budget is 239 words. Results are shown in Table 4 and 5. As a result of the na- ture of budgeted summarization, utterances may be truncated to meet the hard overall size constraint, which makes evaluating with Precision, Recall and Fl score by matching integral utterances inappro- priate. For this reason, we use the same ROUGE and BERTScore metrics as the previous section.
CoreRank Submodular (Tixier et al., 2017). A fully unsupervised, extractive text summarization system that leverages a budgeted submodular max- imization framework. It features a term scoring on k-core decompo- sition applied on graph-of- words representation of the text. The original implementation is in R code; re- ce-implemented in Python. The hyperparame- ters X, r, W are set following the optimal values found for the AMI meeting corpus (Mecowan al., 2005) the original work.
Aggorihnn-CoeRRan---a
BERTExttSu (Miller, 2019). This approach first embeds input sentences with BERT, and then clus- ters the embeddings with K-Means, the closest sentences the K centroids are selected as extrac- tive summary sentences. To meet our summariza- tion setting budgeted terms of words, we start from setting = and increase its number until the output summary meets our budget constraint (with cutting meaning we cut the summary to meet the expected number of words). We used a publicly available implementation' We use fine- tuned multilingual Sentence-BEiR (Reimers and Gurevych, 2019) model? our underlying model, provided in the CentenceTransformen library", due to its better performance on semantic textual simi- larity tasks.
We experimented with the baselines below rang- ing from heuristic, text graph, and embedding based unsupervised approaches:
Longest Greedy (Riedhammer al., 2008). A recommended baseline algorithm for facilitating cross-study comparison. The longest remaining ut- terance removed at each step from the transcript until the summary size limit is met. We implement this simple baseline.
Textrank (Mihalcea and Tarau, 2004). In this ap- proach, each utterance in the transcript is treated as node within an undirected complete graph. The edges between nodes are assigned weights based on the lexical similarity between the utterances. To generate the summary, we select the top nodes ac- cording to their weighted PageRank scores. We
"nctps::/giithbb.mm///mmerr///// com/ssummannp/textrrnl hhttps://githhe com/dmmiller1122
bert-extractive-sumeee
"arraprrse-mmlliiinguaaa---ii-i "nttpss//wwwsbertt.nt.. www.tt.eett/t///e//

---

The overall low omega index scores in the results indicate the difficulty of this task, which is consis- tent with findings in previous work. Results also These results show that BERTEx(Sum generally outperforms other extractive baselines, for both ex- tractive and abstractive summary budgeting. We can also see that BERTExtSum performs very well when compared to the human annotated extractive summaries on BERTScore and R-L, while Longest- Greedy still outperforms other baselines on R-1 and R-2.
Omegal
TF-IDF
Word2
mBERT
frBERT
k = k 11
11.41
10.64
12.27
20.20
9.6
9.3
10.68
13.26
Table Omega index for detecting communities the crrrctiveeabstractiv linking data hetween extractive and abstractive sum- maries.
100 of k-means algorithm
4.3 Abstractive community detection
Omegal
TF-IDF
Word2
mBERT frBERT
k =
13.25
12.07
10.71
17.14
this section, we report the performance of base- line systems on the task of abstractive community detection, in which utterances in the extractive sum- mary are clustered into groups (abstractive commu- nities), according to whether they can be jointly summarized by a common sentence in the abstrac- tive summary. In our annotated dataset, there are in total 123 available pairs of (Extractive 1, Ab- stractive 1) summaries provided with such linking annotations, and 95 for (Extractive 2, Abstractive 2) summaries. We report their results separately in Table 6 and Table 7.
k
11.92
9.6
12.58
11.77
Table Omega index 100 of k-means algorithm for detecting communities the sttractiveabbstractiv linking data between extractive 2 and abstractive 2 sum- maries.
demonstrate the superior performance of BERT based text embeddings across different settings.
5 Human evaluation
Following previous work (Murray et al., 2012; Shang et al., 2018, 2020), experiment with simple unsupervised baseline by applying k-means algorithm on utterance embeddings, obtained by a variety of text embedding methods, such as TF- IDF, Word2Vec (Fauconnier, 2015), multiligual Sentence-BERG BERT (same as the one use in the previ- ous section), and French BERT (FlauBERT (Le et al., 2020)). To account for randomness, we aver- age results over 10 runs.
While ROUGE is still considered the go-to met- ric for evaluating summaries, it has inherent limi- tations that makes it unreliable when used as the sole measure for the assessment of a summary's 's quality. Indeed, ROUGE focuses on measuring lexical overlaps between reference summaries and generated summaries, lacking the ability to evalu- ate semantic similarity, hallucinations. In this paper, we propose to conduct human analysis be able to evaluate how well both ROUGE and BERTScore scale with human judgment the case of dialogue summarization, as well as to see how well summaries produced by newer LLMs score with respect to human evaluation. We asked the annotators to rate 50 debates from 1-5 in informa- tivity, which requires the human to assess how well the summary captures the essence and context of the original debate; readability, which will further explore the preference of abstractive summaries people tend to have for dialogues; and faithfulness. which aims at evaluating how much of the gener- ated summary is comprised of made up facts and hallucinations.
k-means community assign- ments to the human ground truth using the Omega Index (Collins and Dent, 1988), standard met- ric for this task. The Omega Index evaluates the degree of agreement between two clustering so- lutions based on pairs of objects being clustered. We report results with fixed number of commu- nities 1/12, which corresponds to the average number of ground truth communities per extractive- abstractive linking, and also variable & equal to the number of ground truth communitics (denoted by k=v). We adopt a publicly available metric im- plementation"!
We compare
As language models advance and become more sophisticated, their summary outputs also may not scale proportionally with ROUGE scores. New techniques, such controlled generation and rein- forcement learning methods, allow these models to produce diverse and high-qualtt summaries that cannot be adequately captured by ROUGE alone.
"frWac_non. coc__noon_eemmeeeee nowpooott__ooooeee stt.aaooooohhrae_eee_e_ skip_cut bin 10 "'flaubert_i Baaibbert_lare___ssstt "https: / thw.btt.tttttttteeeeeeeeeee..

---

Readability
Informativity
Faitfulness
tive ones. The higher informativity of extractive summaries likely result of their more extensive content.
Abstractive I
2.97
3.87
4.96 (2)
Abstractive 2
4.32
4.36
4.98 (1)
Abstractive 3
3.79
3.66
4.98 (1)
Meta-evaluation of automatic metrics
Extractive
2.60
4.22
5.00 (0)
Extractive
2.63
4.37
5.00 (0)
As we have mentioned previously, the automatic evaluation metrics used in summarization have lim- itations that prevent them from perfectly assess- ing the quality of given model. In this subsec- tion, we aim determine the correlation between widely used metrics such as ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore, and human judgment. For the sake of comprehensiveness, we employ the grading scheme discussed in section 5, encompass- ing Readability, Informativity, and Faithfulness.
BARThez 1.69
1.30
1.80 (48)
ChatGPT 3.76
3.87
4.50 (19)
OpenAssistant 3.50
2.54
2.61 (40)
Table 8: Human evaluation of Readability, Informativity and Faithfullness out of 5. A high readability score means an easily readable summary, high informativity indicates an exhaustive summary, and high faithfullness means the summary is consistent with the transcript, the number in parentheses is the amount of summaries that contain hallucinations.
Readability Informativity Faithfulness
Rouge-1
0,22
0.54
0.52
Human evaluation provides a complementary as- sessment to identify the strengths and weaknesses of these advanced models and compare their per- formance in terms of faithfulness, readability, and informativity.
Rouge-2
0,24
0.64
0.59
Rouge-L
0.37
0.69
0.64
BERTScore
0.53
0.69
0.64
Table 9: Pearson correlation between human judgement and automatic metrics in Readability, Informativity and Faithfulness
Multiple points are brought out by the human evaluations. While human annotated summaries consistently rank the highest in all metrics, Chat GPT produces remarkably competitive summaries, particularly in terms of informativity and readabil- ity, while maintaining high level of faithfulness. It is worth noting that, in the case of ChatGPT, al- though some of the annotated summaries contained hallucinations, these ballucinations differed signifi- cantly from those generated by ApenAssistar or BARThez and were more similar to what human annotators would produce. These hallucinations typically resulted from either ambiguous speech attempt to capture
The results show that, as expected, ROUGE-1 performs worse of all available metrics across the board. On the other hand, both BERTScore and demonstrate noteworthy Pearson corre- lations, particularly in terms of Informativity and Faithfulness. Notably, BERTScore consistently hibits stronger correlations across all three aspects (Readability, Informativity, and Faithfulness).
Usage of National Assembly and Senate data
in the original transcript the original tone of speaker, such as mentioning irony when it does not exist.
All the data provided in this paper follows the terms and conditions of their respective sources; National Assembly. Senate, following article L. 122-5 of the french intellectual property code, stipulating that there are no copyright for any of the articles, allow- ing people to copy them freely. The caveat being that the usage has to be either personal, associative or for professional uses, forbidding reproduction for commercial and advertisement purposes
Furthermore, although BARThez appeared rel- atively competitive in terms of ROUGE and BERTScore, its summaries were considered signif- icantly worse when compared to human judgment. Lastly, while extractive summaries may rank lower in readability due to the nature of dialogue speech, on average, they outperform abstractive summaries in both informativity and faithfulness. Additionally, it is challenging include halluci- nated facts in extractive summaries, as doing so would require omitting negation clauses entirely. For tasks where faithfulness is crucial, extractive summaries continue to rank higher than abstrac-
8 Conclusion
In this paper, we present the first version of FRED- Sum, a French dataset for abstractive debate sum- marization. The dataset consists of manually tran- scribed political debates, covering a range of topics

---

and perspectives and annotated with topic segmen- lation, extractive and abstractive summaries and conduct extensive analyses on all the annotations and provide base- line results on the dataset by appealing to both automatic and human evaluation.
nuanced arguments and the intricacies of domain-specific language.
abstractive communities.
Ethical Consideration
Considering the nature of any political paper, ethics is concern. We certify that any data provided in this paper follows the terms and conditions of their respective sources; National Assembly, Sen- ate. Moreover all human annotations have been gathered fair fashion, each annotators being fully compensated for the annotations as part their full time job at our organization, as well as be- ing authors or acknowledged members of the paper. Finally, we have ensured that the french political landscape is fairly represented in both the pretrain- ing dataset; all presidential debates have been taken into account and every political proximity that has gotten least 5% of the votes at a presidential elec- tion is present in at least one debate as well as the choices of debates summarized. In addition, it is worth noting that the annotators openly represent diverse political affiliations bias there might be in summaries, however, bias is inevitable in any such task, we hope that having multiple source for summaries helps lowering the average bias.
Limitations
In this section, discuss several limitations on the dataset and baseline used this paper.
Limited amount of data: Although this work represents the first resource of French sum- marized debate data, or dialogue data in gen- eral, the total duration of 50 hours relatively low; this inevitably limits the impact o the dataset for training efficient systems for gen- eralized dialogue summarization. A longer dataset would enhance the robustness and gen- eralizability of the findings
Minimal work on prompt engineering: The nature of the experiments, as well as the need for replicability, made it that both of the prompt based models were not used to their full capabilities, as little prompt engineering has been conducted. This may have hindered the model's ability to adapt and to generate contextually appropriate responses. Further exploration of prompt engineering could lead to improved results comparatively.
to average any
Acknowledgements
This work was supported by both the SUMM-RE (ANR-20-CE23-0017) and the Horizon Europe CORTEX2 We thank Rayan Autones and Jules Peyrat for tributing in the meta-evaluation research of auto- matic metrics, during their internship.
projects.
Lack of linguistic annotations: Many different kinds of systems have been used for dialogue summarization. using a wide variety of anno- tations help with the initial interpretation of transcript, to offset the lengthy input issue. While there is interest in providing abstrac- tive and extractive summaries, as well as ab- stractive communities, some models require dialogue act annotations as well as discourse graphs to function. Providing further annota- tions would allow more traditional systems to be tested on this dataset.
References
Hadi Abdine, Yanzhu Guo, Virgile Remnard, and Michalis Vazirgiannis. 2022. Political communities on twitter: Case study of the 2022 french presidential election. arXiv preprint MXXi:2224.0045:3
I Linda M Collins and Clyde W Dent. 1988. Omega: A general formulation of the rand index of cluster recov- ery suitable for non-disjoint solutions. Multivariate behavioral research, 23(2)2231-242.
Limited fine-tuning: The model used in this study, Barthez, has not undergone extensive fine-tuning specifically for the French de- bate domain, as well for conversations in general. While Barthez is a powerful language model, the lack of fine-tuning tai- lored to debate-related tasks may impact its performance and limit its ability to capture
lakovos Evdaimon, Hadi Abdine, Christos Xypolopou- los, Stamatis Outsios, Michalis Vazirgiannis, and Giorgos Stamou. 2023. Greekbart: The first pre- trained greek sequence-to--eqqeene model, arXiv preprint arXiv:2304.008:
Pean-Philippe Fauconnier. 2015. French word embed- dings.

---

Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar, Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968-988, Online. Association for Computa- tional Linguisties.
Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 7871-7880) Online. Association for Computa- tional I Linguistics.
pre- vre-training
Emilio Ferrara. 2017. Disinformation and social bot op- erations in the run to the 2017 french presidential election. arXiv preprint
Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Bogdan Gliwa, Toona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Sunmarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Hui Lin and Vincent Ng. 2019. Abstractive summariza- tion: A survey of the state of the art. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 9815-9822.
Jain Mccowan, Jean Carletta, Wessel Kraaij, Simone Ashby, Sebastien Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska Masson, Wilfried Post, Dennis Reidsma, and P. Wellner. 2005. The AMI meeting corpus. Inf'l. Conf. on Methods and Techniques in Behavioral Research.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines read and comprehend. Advances in neural information processing systems, 28.
Adam Janin, Don Baron, Jane Fddwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. 2003. The ICSI meeting corpus. In 2003 IEEE In- ternational Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP" 03).. volume pages H. IEEF.
Rada Mihalcea and Paul Tarau. 2004, TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021. BARThez: a skilled pretrained French sequence-to-sequenee model. In Proceedings of the 2021 Conference on Empirical Methods in Nut- ural Language Processing, pages 9369-9390, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Derek Miller. 2019. Leveraging bert for extractive text summarization lectures. arXiv preprint arXiv:7906.0416:
Gabriel Murray, Giuseppe Carenini, and Raymond Ng. 2010. Generating and validating abstracts of meeting conversations: user study. In Proceedings of the 6th International Natural Language Generation Con- ference, Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit Papers, pages 79-86, Phuket, Thailand.
Gabriel Murray, Giuseppe Carenini, and Raymond Ng, 2012. Using the omega index for evaluating ab- stractive community detection. In Proceedings of Workshop on Evaluation Metrics and System Com- parison for Automatic Sunmarization, pages 10-18. Montréal, Canada, Association for Computational Linguistics.
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis, Zhi-Rui Tam. Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stan- ley, Richard Nagyfi, et al. 2023. Openassistant conversafonss-dmmrrattine large language model alignment. arXiv preprint arXiv:2304.07322
Anna Nedoluzhko, Muskaan Singh, Marie Hledíková, Tirthankar Ghosal, and Ondfej Bojar. 2022. ELITR Minuting Corpus: novel dataset for automatic minuting from multi-party meetings in English and Czech. In Proceedings of the 13th International Conference on Language Resources and Evaluation (LRRCC-2022) Marseille, France. European Language Resources Association (ELRA). In print.
Hang Le, Loic Vial, Jibril Frej. Vincent Segonne, Max- imin Coavoux, Benjamin Lecouteux, Alexandre Al- lauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model pre- trainngn for French. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 2479-2499, Marseille, France. European Language Resources Association.
Nikola I Nikolov, Michael Pfeiffer, and Richard HR Hahnloser. 2018 Data-driven summarization of sci- entific articles. arXiv preprint arXiv: 1804::88040888 08875.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Nearal Information Processing Systems, 35:27730-27744,
Han Wu, Mingjie Zhan, Haochen Tan, Zhaohui Hou, Ding Liang, and Linqi Song. 2023. Vesum: A ver- satile chinese meeting summarization dataset. arXiv preprint arXhv:2355.05280.
Shiyue Zhang, David Wan, and Mohit Bansal. 2023, Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 2153 2274, Toronto, Canada. Association for Computational Linguistics.
Peter Polák, Muskaan Singh, Anna Nedoluzhko, and Ondicj Bojar. 2022. ALIGNMEET: A comprehen- sive tool for meeting annotation, alignment, and evaluation. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 1771-1779, Marseille, France. European Language Resources Association.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT, In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
A Additional experiments
In the following appendix, we provide supplemen- tary information that complements the main text of this paper. This includes detailed methodologics, experimental setups, and additional results not pre- sented in the main body. The intention of this appendix is to offer readers deeper understanding of the experiments conducted, as well as to provide transparency regarding our research processes.
Virgile Rennard, Guokan Shang, Julie Hunter, and Michalis Vazirgiannis. 2023. Abstractive meeting summarization: survey. Transactions of the Asso- ciation for Computational Linguistics, 11:861-888
Korbinian Riedhammer, Dan Gillick, Benoit Favre, and Dilek Hakkani-Tür. 2008. Packing the meeting sum- marization knapsack. In Ninth Annual Conference of the International Speech Communication Associa- tion.
Inter-annotator agreement Defining inter- annotator agreement in the context of extractive summarization can be challenging due to the repeti- tion of identical utterances in transcripts, requiring both annotators to select the same portions. Nev- ertheless, we can approximate it by computing the ROUGE and BERTScore scores between extractive summaries.
Guokan Shang- 2021. Spoken Language Understanding for Abstractive Meeting Summarization. Ph.D. thesis, Institut Polytechnique de Paris.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tix- ier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2018. Unsupervised abstrac- tive meeting summarization with mulli-sentenee com- pression and budgeted submodular maximization. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volme I: Long Papers), pages 664-674, Melbourne, Australia. Association for Computational Linguistics.
RI
R2
RL
BERTScore
Ext vs Ext 2
0.72
0.59
0.69
0.95
Table 10: ROUGE and BERTScore comparison of both extractive summaries
Guokan Shang, Antoine Tixier, Michalis Vazirgiannis, and Jean-Pierre Lorré 2020. Energy- -ased self- attentive learning of abstractive communities for spo- ken language understanding. In Proceedings of the Ist Conference of the Asia-Pacific Chapter of the As- sociation for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 313-327, Suzhou, China. Associa- tion for Computational Linguistics.
Human Evaluation criterias Human evalua- tions play a pivotal role in assessing the effective- ness of the methods as well as the quality of the data. While automated metrics give a quantita- tive measure, human judgments provide qualitative insights that are often more aligned with practi- cal use-cases, in this case, we gave the following guidelines for the annotators when rating for faith- fullness, Informativity and Readability.
Antoine Tixier, Polykarpos Meladianos, and Michalis Vazirgiannis. 2017. Combining graph degeneracy and submodularity for unsupervised extractive sum- marization. In Proceedings of the Workshop on New Frontiers in Sunmarization, pages 48-58, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Readability:

---

The given summary is incomprehensible
original transcript; an example "Tonight in emission name 8, Jean Luc Mélenchon interviewed by Laurent ruquier"
2. An overall theme can be gathered from reading the summary, but the sentences heavy and/or repetitive
this
case, the transeript mentions niether the show nor either of the people cited the summary. Barthez is prone to those hallucinations
3. The summary is readable with little repe- tition, the point come across easily
4. The summary reads fluently
Confusion of scale speaker Either attribut- ing decision to the wrong speaker, exager- ating its scale "Jordan Bardella proposes build 35000 prison spots" be wrong in two ways here: the amount of spots, or the fact that was another debater that proposed to build the 35000 places Open Assistant tends to make this kind of mistake.
The summary reads fluently, and is ele- gant
Informativity:
1. The summary captures no information from the original debates
2. The summary captures the general theme of the original debate (The speakers and subject)
Unavailable information correct or not LLMs use information about different char- acters that are not available in the transcript This usually appears when speaker is introduced, such in the example: "Jor- dan Bardella, président du rassemblement na- tional." " These instances of adding information not directly present in the text but in the pre- training data can exhibit varying degrees of inaccuracies. While is arguably already a hallucination when the title is correct, these instances can take two other forms: 1) Inac- curate titles, 2) Once correct titles that longer current. It's worth noting that some speakers might switch parties or titles, partic- ipating in a debate while holding a position of a deputy, yet the summarization process might attribute a different title from a different time period. This tendency towards introduc- ing such hallucinations is more pronounced in ChatGPT, likely stemming from the large pretraining it underwent, although all three models share this characteristic.
3. The summary captures the general theme of the original debate, well as the main points made by the different debaters
4. The summary captures the general theme of the original debate, the main points made by the different speakers, well the responses from the opposite parties 5. The summary perfectly captures the nu- ances of the original debate
Faithfulness:
The summary hallucinates the entirety of the debate.
2. Major hallucinations, such as mistaking the participants of the debates or the main theme of the debate.
3. The summary is faithful enough to be read, but has some minor fabrications, a point made by speaker is however modified enough to introduce confusion.
The summary largely faithful to the original debate, and only includes hallu- cinations that could be made by a human because of ambivalent wordings in the original transcript
Triple coreference and confusing speech Some parts of debates, with many interrup- tion and references to facts can be compli- cated to follow "About that, with the age of the retirement, we will raise it", will often be summarized about raising the age of retire- ment, while in context this is about raising the amount of years of contribution this is often caused, while being correct, by the over- lapping of multiple coreferences. Human hallucinations also fall in this category
The summary entirely faithful
Additionally, we propose to give categorization of hallucinations, from the worse cases that causes complete misunderstanding of the original docu- ment to more ambiguous mistakes
Complete invention: The summary starts and develops a subject completely absent from the
Trony: One orator can ironically agree with a

---

proposition of another while in fact disagree- ing with it ChatGPT can fail to pick up on this kind of nuance.
R-1
R-2
R-L
BERTScore
BARThez
13.28
03.76
08.82
63.06
Open Assistant
29.07
07.45
14.11
68.66
ChatGPT
43.22
16.51
24.19
73.67
We note that while no hallucinations, or rather, mis- understandings, are made in the extractive sum- maries here, it is because we have instructed the annotators to explicitly get rhetorical clauses and context in the extractive summaries (somewhat jus- tifying the lengthy nature of extractive maries)- Systems that create extractive summaries, as well as human annotated extractive summaries that do not gather context can contain hallucina- tions (Zhang et al., 2023)-
Table 13: ROUGE and BERTScore Fl abstractive sum- marization scores against Abstractive No segmenta- tion.
Additional experiments for abstractive summa- rization In the experiments conduct, compare the models with specific truncating strat- egy to optimize their respective outputs and so that all the models get to see pare as fairly as possible the models, we propose to give additional result in which all models only get to see the largest chunk of the original debate they can. Moreover, we propose to share the performance of models against each of the abstrac- tive summaries, to see the impact the style of each summary has on the results.
the debate. To com-
R-1
R-2
R-L
BERTScore
BARThez
14.54
04.87
09.53
64.82
Open Assistant
29.11
07.31
15.21
69.68
ChatGPT
44.63
17.22 24.34
74.54
Table 11: ROUGE and BERTScore Fl abstractive sum- marization scores against Abstractive No segmenta- tion.
R-1
R-2
R-L
BERTScore
BARThez
15.24
04.91
10.22
64.23
Open Assistant
30.06
07.45
16.63
69.91
ChatGPT
42.11
17.10 23.85
73.88
Table ROUGE and BERTScore Fl abstractive sum-
marization scores against Abstractive 2 No segmenta-
tion.


Reviews:
[{"review_id": "RUqiBRAWcW", "reviewer": "Reviewer_P9qY", "summary": "", "questions": "A) Lines 100-105. How does your dataset promote multilingual dialogue summarization techniques? Since the dataset is focused on the French language, it could be to promote language diversity in the research field, but never multilingual promotion.\n\nB) Table 1. Are tokens equal to words? If so, why did you not use \"words\" instead of tokens? If not, how did you tokenize the text to extract the statistics?\n\nC) Table 1. What are FREDSumPreS and FREDSumPreA? Missing on the caption and/or text.\n\nD) Lines 350-354, 361-364. Could it be the fact of splitting the transcriptions for summarizing with BARThez and Open Assistant, that penalized the quality of the final summary for these models? \n\nE) Lines 544-547. How affects the hallucinations in the score? Is it about the severity of the hallucinations, their amount, or a combination of these two?\n\nF) How is Table 9 created? Is the Pearson correlation calculated with all the generated summaries?\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your detailed and insightful review. We respond point-by-point to your comments below.\n\n------------------------------------------\n\n**Reasons to accept :**\n\nThe paper presents a new dataset that is constructed purely with human annotation data. The dataset focuses on French political debates, with a variety of additional annotations for these debates. Also, baselines are provided as the basis for further research.\n\n*We thank the reviewer for the positive assessment of our work. We are glad that you found our dataset is equipped with a variety of additional annotations. We will address\nthe comments and include our response where relevant, especially regarding the annotation procedure, in the next version of the paper.*\n\n------------------------------------------\n\n**Reasons to Reject :** Human annotation procedures are not detailed extensively and provoke the following weakness: It is unclear whether just one annotator selects the relevant information for a given sub-debate or is consensual by more than one annotator.\n\n*Each annotator selected information and made the summaries independently from other annotators. While it is hard to calculate inter annotator agreement for tasks like extractive summarization, we can however estimate the inter-annotator agreement by calculating the ROUGE score between extractive summaries*\n\n|                              | R1   | R2   | RL   | BERTScore |\n|------------------------------|------|------|------|-----------|\n| Extractive 1 vs Extractive 2 | 0.72 | 0.59 | 0.69 | 0.95      |\n\n*As we can observe from this additional experiment, the agreement between annotators for the creation of extractive summaries is high. Those scores will be added to the appendix of the paper*\n\n------------------------------------------\n\n**Reasons to Reject :** It is unclear how many people participate in the annotation and how many coincide in the annotation and evaluation of the summaries, which could lead to a particular bias. Also, how many of them are part of the authoring of the article.\n\n*Five annotators overall worked on the annotations available in the paper, three of them being co-authors. Two co-authors created abstractive summaries. A third coauthor contributed to summary evaluation in collaboration with two independent annotators.* \n\n*Of course, while bias is inevitable in political annotations, the overall agreement calculated shows that the annotators mainly agreed on important points. It is important to note that political bias is prevalent in LLMS [1], and the creation of a dataset that has multiple points of view from different summaries can only try to lessen such bias*\n\n*[1] Feng, Shangbin, et al. \"From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.\" arXiv preprint arXiv:2305.08283 (2023).*\n\n------------------------------------------\n\n**Reasons to Reject :** It is unclear how the Human annotation scale is designed, what makes a given sample obtain a score between 1 and 5 for a given characteristic (readability, informativity, faithfulness)\n\n*Here are the guidelines given to annotators for each of readability, informativity, faithfulness*\n\n*Readability :*\n\n1. The given summary is incomprehensible\n2. An overall theme can be gathered from reading the summary, but the sentences are heavy or repetitive\n3. The summary is readable with little repetition, and the points come across easily\n4. The summary reads fluently\n5. The summary reads fluently, and is elegant\n\n\n*Informativity :* \n1. The summary captures no information of the original debate\n2. The summary captures the general theme of the original debate (Speakers + subject e.g. Ukraine war, retirement age, immigration)\n3. The summary captures the general theme of the original debate, as well as the main points made by the different debaters\n4. The summary captures the general theme of the original debate, the main points made by the different speakers, as well as the responses from the opposite parties\n5. The summary perfectly captures the nuances of the original debate\n\n*Faithfulness :* \n1. The summary hallucinates the entirety of the debate\n2. Major hallucinations, such as mistaking the participants of the debates or the main theme of the debate.\n3. The summary is faithful enough to be read, but has some minor fabrications, a point made by a speaker is however modified enough to introduce confusion.\n4. The summary is largely faithful to the original debate, and only includes hallucinations that could be made by a human because of ambivalent wordings in the original transcript\n5. The summary is entirely faithful\n\n------------------------------------------\n\n**Questions for the Authors :** How does your dataset promote multilingual dialogue summarization techniques? Since the dataset is focused on the French language, it could be to promote language diversity in the research field, but never multilingual promotion.\n\n*Models now thrive in multi-linguality. LLAMA, GPT-3 and others perform well in multi-lingual settings by training themselves majoritarily on English, with very small subsets of data in different languages. We believe that creating data for complicated tasks such as long dialogue summarization in different languages would help improve the quality of the models trained for those tasks in multi lingual settings*\n\n------------------------------------------\n\n**Questions for the Authors :** Table 1. Are tokens equal to words? If so, why did you not use \"words\" instead of tokens? If not, how did you tokenize the text to extract the statistics?\n\n*This is our mistake; we meant words instead of tokens and will correct the problem in the paper*\n\n------------------------------------------\n\n**Questions for the Authors :** Table 1. What are FREDSumPreS and FREDSumPreA? Missing on the caption and/or text.\n\n*PreS represents the statistics of the Senate subset of the pretraining data, while PreA represents the Assembly*\n\n------------------------------------------\n\n**Questions for the Authors :** Lines 350-354, 361-364. Could it be the fact of splitting the transcriptions for summarizing with BARThez and Open Assistant, that penalized the quality of the final summary for these models? \n\n*Both Barthez and OpenAssistant score significantly lower than ChatGPT in debates they can fully treat without segmentation; Indeed, the longer the debate, and the larger the amount of cuts, the lower the score is. It is however impossible for both of those systems to summarize entire meetings without the use of segmentation strategies. We report in the rebuttal of Review #3 the scores of each model when no segmentation strategy is applied, and they are only treating the longest string they can get in one pass.*\n\n------------------------------------------\n\n**Questions for the authors :** Lines 544-547. How affects the hallucinations in the score? Is it about the severity of the hallucinations, their amount, or a combination of these two?\n\n*As shown by the guidelines provided above, the number of hallucinations are taken into account, however, the nature of the hallucinations are also extremely important. We describe the nature of different types of hallucinations in our response to reviewer number two in decreasing order of severity*\n\n------------------------------------------\n\n**Questions for the authors :** How is Table 9 created? Is the Pearson correlation calculated with all the generated summaries? \n\n*All the generated summaries and all their ratings. Each summary has at least two ratings by human annotators as well as their ratings with respect to the metrics. Each score is calculated by the pearson correlation of all the ratings from all the summaries and their respective scores to each metrics . As an example, consider an abstractive summary A with two human ratings, H1 and H2. To get the Pearson correlation for ROUGE 1, we take the Pearson correlation between the ROUGE 1 score for H1 and the Pearson correlation between the ROUGE 1 score and H2 and average these two scores together. We then do this for all transcripts and take the average of those scores.*\n\n------------------------------------------\n\nThank you again for the review, we hope have cleared up the points presented in your reasons to reject", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a new dataset that is constructed purelly with human annotation data. The dataset focuses on French political debates, with a variety of additional annotations for these debates. Also, baselines are provided as the basis for further research.", "reasons_to_reject": "Human annotation procedures are not detailed extensively and provoke the following weakness:\n\n- It is unclear whether just one annotator selects the relevant information for a given sub-debate or is consensual by more than one annotator.\n\n- It is unclear how many people participate in the annotation and how many coincide in the annotation and evaluation of the summaries, which could lead to a particular bias. Also, how many of them are part of the authoring of the article.\n\n- It is unclear how the Human annotation scale is designed, what makes a given sample obtain a score between 1 and 5 for a given characteristic (readability, informativity, faithfulness)", "questions_to_authors": "A) Lines 100-105. How does your dataset promote multilingual dialogue summarization techniques? Since the dataset is focused on the French language, it could be to promote language diversity in the research field, but never multilingual promotion.\n\nB) Table 1. Are tokens equal to words? If so, why did you not use \"words\" instead of tokens? If not, how did you tokenize the text to extract the statistics?\n\nC) Table 1. What are FREDSumPreS and FREDSumPreA? Missing on the caption and/or text.\n\nD) Lines 350-354, 361-364. Could it be the fact of splitting the transcriptions for summarizing with BARThez and Open Assistant, that penalized the quality of the final summary for these models? \n\nE) Lines 544-547. How affects the hallucinations in the score? Is it about the severity of the hallucinations, their amount, or a combination of these two?\n\nF) How is Table 9 created? Is the Pearson correlation calculated with all the generated summaries?\n\n\n", "paper_topic_and_main_contributions": "The article presents a new dataset for multi-party dialogue summarization; FREDSum. The dataset consists of French political debates that were manually transcribed. Then, the debates are segmented by dialogue topic (established by the debate moderator) into sub-debates. Those sub-debates are annotated with additional information such as abstractive and extractive summaries, abstractive community annotations, important topics, and evaluation data.\n\nTo describe the dataset, the authors present essential sample statistics (number of transcriptions, tokens per transcription, turns per transcription, speakers per transcription, and tokens per abstractive summaries). Also, they carry a study about how models with abstractive summarization capabilities (BARThez, Open Assistant, and ChatGPT) and extractive summarization ones (Longest Greedy, CoreRank Submodular, and BERTExtSum) perform in this new dataset, establishing a baseline. Finally, a human evaluation is conducted to measure the quality of the annotated and generated summaries.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 74: Thus, there is a significant\n\n- 110-118: Twitter\n\n- 110-118: Too long sentence; try to split it into parts.\n\n- 142: The debates in FREDSum cover a diverse\n\n- Table 5: It seems to me that there should be 239 words instead of 684 words.\n\n- Table 8: Faithfulness (column head name)"}}, {"review_id": "Gs04nnTVGb", "reviewer": "Reviewer_Xx4E", "summary": "", "questions": "1. Line 469: Explaining what is meant by 'with cutting' could help readers. \n2. Lines 266-278: It is unclear if the abstractive communities are obtained via clustering/topic modeling or human annotations of alignment between extractive and abstractive summary sentences. A clarification can help readers. \n3. Choice of 684 and 249 words for length-constrained summaries can be explained. Why these choices are reasonable to make? The authors provide citations for such choice, but describing in a line or two in the paper itself could help readers. \n4. Line 571-575 It would be interesting to provide examples of hallucinations and ironies the authors describe in the paper. Error analysis could be in general interesting to researchers interested in using this resource. While I do not advocate for English only, providing an English translation of the French examples could help a wider audience who does not speak French and thus enhance the impact of the paper. \n5. Line 592: It is unclear what is the motivation to study the correlation between human annotations and automatic metrics. Such studies have been conducted by the authors of automated metrics, and it is unclear how it is contextualized in this work. More explanation of insights drawn based on this analysis, which is specifically applicable to this paper, could be useful. \n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your detailed and insightful review. We respond point-by-point to your comments below\n\n------------------------------------------\n\n**Reasons To Accept:**\nThe authors provide a valuable resource of human-transcribed French political debates with human-annotated summaries. This resources can not only be useful for summarization task but also to employ computational methods to study French politics.\nThe authors also provide several baseline experiments demonstrating the current capabilities of summarization models.\n\n*We thank the reviewer for confirming that our dataset is a valuable resource for dialogue summarization with multiple types of annotation, and it additionally opens up opportunities for novel research in computational methods to study French politics.*\n\n------------------------------------------\n\n**Reason to reject:** Line 292: The authors scraped the official websites of the French Senate and Parliament to create a publicly available pre-training dataset in French. What were the content use-policies of these websites? Is creation of such dataset and training of models on such data allowed under this policy? If the curated dataset violates any terms of use, then this could be a major concern. While the authors mention briefly in the ethics section that the \"any data provided in this paper follows the terms and conditions of their respective sources,\" it would be important to explicitly state the allowed usage of this dataset (what is allowed and what is not) in the paper itself. For instance, commercial use of any information is not allowed, which should be mentioned in the paper to avoid use of such dataset for training any industrial solutions etc. This is only an example, more thorough understanding of the terms of use is required and need to be discussed in the paper. \n\n*This is an excellent point; an additional section will be added to the paper in order to explain the licensing more thoroughly; Both the National Assembly and the Senate transcripts follow article L.122-5 of the french intellectual property code, stipulating that there are no copyright for any of the articles, allowing people to copy them freely. The caveat being that the usage has to be either personal, associative or for professional uses, forbidding reproduction for commercial and advertisement purposes*\n\n*As such, the creation of a dataset as well as the training of models in the scope of research is allowed*\n\n------------------------------------------\n\n**Reason to reject:** The authors provide baseline experiments for extractive, abstractive, and abstractive community detection. However, no discussion is provided on what are the research gaps addressed by this dataset. Based on the provided experiments, some of the models perform fairly well, which is interesting to note in itself. A discussion of where do current models already do well and where do they lack could be useful for future researchers interested in using this dataset as a resource. \n\n*As evident from the human assessment of summaries, ChatGPT performs well in abstractive summarization, contrary to models such as Open Assistant and Barthez, who display notable shortcomings across all fronts. Although Open Assistant demonstrates relative informativity in managing concise debates, the prevalence of hallucinations and the absence of exhaustive summaries highlight the significant challenges that open-source models still face. Moreover, despite ChatGPT's strong performance, it still has limitations in terms of informativity and faithfulness. This becomes particularly apparent in longer debates and discussions with frequent interruptions or disfluencies. Furthermore, there is a necessity for continued efforts in extractive summarization within the context of dialogues, alongside advancements in abstractive community detection.*\n\n\n------------------------------------------\n\n**Questions for the authors:** Line 469: Explaining what is meant by 'with cutting' could help readers.\n\n*With our described way of using BERTExtSum, the resulting extractive summary may be longer than the budget constraint, “with cutting” here simply means we cut the summary to meet the expected number of words. Clarification will be added to the paper*\n\n------------------------------------------\n\n**Questions for the authors:** Lines 266-278: It is unclear if the abstractive communities are obtained via clustering/topic modeling or human annotations of alignment between extractive and abstractive summary sentences. A clarification can help readers. \n\n*Abstractive communities are obtained via human annotations of alignment between one extractive summary and one abstractive summary. Each of the 138 sub debates thus has two different annotations for abstractive communities*\n\n------------------------------------------\n\n**Questions for the authors:** Choice of 684 and 249 words for length-constrained summaries can be explained. Why these choices are reasonable to make? The authors provide citations for such choice, but describing in a line or two in the paper itself could help readers. \n\n*Using the average length of summaries for budgetting is usual, we will mention this further in the paper*\n\n------------------------------------------\n\n**Questions for the authors:** Line 571-575 It would be interesting to provide examples of hallucinations and ironies the authors describe in the paper. Error analysis could be in general interesting to researchers interested in using this resource. While I do not advocate for English only, providing an English translation of the French examples could help a wider audience who does not speak French and thus enhance the impact of the paper.\n\n*Different level of hallucinations can be observed of varying degrees of importance :*\n\n- Complete invention : The summary starts and develops a subject completely absent from the original transcript; as an example : “Tonight in * emission name *, Jean Luc Mélenchon is interviewed by Laurent ruquier” - In this case, the transcript mentions niether the show nor either of the people cited in the summary. Barthez is prone to those hallucinations\n- Confusion of scale or speaker - Either attributing a decision to the wrong speaker, or exagerating its scale - “Jordan Bardella proposes to build 35000 prison spots” can be wrong in two ways here: the amount of spots, or the fact that it was another debater that proposed to build the 35000 places - Open Assistant tends to make this kind of mistake.\n- Unavailable information - *correct or not* : LLMs use information about different characters that are not available in the transcript - This usually appears when a speaker is introduced, such as in the example: \"Jordan Bardella, président du rassemblement national.\" These instances of adding information not directly present in the text but in the pretraining data can exhibit varying degrees of inaccuracies. While it is arguably already a hallucination when the title is correct, these instances can take two other forms: 1) Inaccurate titles, 2) Once correct titles that are no longer current. It's worth noting that some speakers might switch parties or titles, participating in a debate while holding a position of a deputy, yet the summarization process might attribute a different title from a different time period. This tendency towards introducing such hallucinations is more pronounced in ChatGPT, likely stemming from the large pretraining it underwent, although all three models share this characteristic.\n- Triple coreference and confusing speech - Some parts of debates, with many interruption and references to facts can be complicated to follow : “About that, with the age of the retirement, we will raise it”, will often be summarized about raising the age of retirement, while in context - this is about raising the amount of years of contribution - this is often caused, while being correct, by the overlapping of multiple coreferences. - Human hallucinations also fall in this category\n- Irony - One orator can ironically agree with a proposition of another while in fact disagreeing with it - ChatGPT can fail to pick up on this kind of nuance.\n\n*Finally, it is important to note that while there are no hallucinations made in the extractive summaries here, it is because we have instructed the annotators to explicitly get rhetorical clauses and context in the extractive summaries (somewhat justifying the lengthy nature of our extractive summaries) - Systems that create extractive summaries, as well as human annotated extractive summaries that do not gather context can contain hallucinations [1]*\n\n*[1] Zhang, Shiyue, David Wan and Mohit Bansal. \"Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization.\" arXiv preprint arXiv:2209.03459 (2022).*\n\n\n------------------------------------------\n\n**Questions for the authors :** Line 592: It is unclear what is the motivation to study the correlation between human annotations and automatic metrics. Such studies have been conducted by the authors of automated metrics, and it is unclear how it is contextualized in this work. More explanation of insights drawn based on this analysis, which is specifically applicable to this paper, could be useful. \n\n*In this paper, we can observe from human and automatic evaluation that the ROUGE score is very poor at separating the quality of the summaries from BARTHEZ and Open assistant, where the human evaluations show a large difference between them.*\n\n*Comparative studies [2] [3] show that there is no one-size-fits-all metric that can outperform others on all datasets, suggesting the utility of using different metrics for different datasets to evaluate systems. Our experiments show that both reference based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. For the above reasons, we want to evaluate this by ourself, and results show that BERTScore correlates well with human judgmnt, and it suggests that BERTSCORE can be a reliable metric for our dataset, for future studies.*\n\n*[2] Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating Evaluation in Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguistics.* **Page 6**\n\n*[3] Goyal, Tanya, Junyi Jessy Li, and Greg Durrett. \"News summarization and evaluation in the era of gpt-3.\" arXiv preprint arXiv:2209.12356 (2022).* **Abstract**\n\n------------------------------------------\n\nThank you again for the review, we hope to have cleared up the points presented in your reasons to reject", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide a valuable resource of human-transcribed French political debates with human-annotated summaries. This resources can not only be useful for summarization task but also to employ computational methods to study French politics.   \n2. The authors also provide several baseline experiments demonstrating the current capabilities of summarization models. ", "reasons_to_reject": "1. Line 292: The authors scraped the official websites of the French Senate and Parliament to create a publicly available pre-training dataset in French. What were the content use-policies of these websites? Is creation of such dataset and training of models on such data allowed under this policy? If the curated dataset violates any terms of use, then this could be a major concern. While the authors mention briefly in the ethics section that the \"any data provided in this paper follows the terms and conditions of their respective sources,\" it would be important to explicitly state the allowed usage of this dataset (what is allowed and what is not) in the paper itself. For instance, commercial use of any information is not allowed, which should be mentioned in the paper to avoid use of such dataset for training any industrial solutions etc. This is only an example, more thorough understanding of the terms of use is required and need to be discussed in the paper.\n\n2. The authors provide baseline experiments for extractive, abstractive, and abstractive community detection. However, no discussion is provided on what are the research gaps addressed by this dataset. Based on the provided experiments, some of the models perform fairly well, which is interesting to note in itself. A discussion of where do current models already do well and where do they lack could be useful for future researchers interested in using this dataset as a resource. ", "questions_to_authors": "1. Line 469: Explaining what is meant by 'with cutting' could help readers. \n2. Lines 266-278: It is unclear if the abstractive communities are obtained via clustering/topic modeling or human annotations of alignment between extractive and abstractive summary sentences. A clarification can help readers. \n3. Choice of 684 and 249 words for length-constrained summaries can be explained. Why these choices are reasonable to make? The authors provide citations for such choice, but describing in a line or two in the paper itself could help readers. \n4. Line 571-575 It would be interesting to provide examples of hallucinations and ironies the authors describe in the paper. Error analysis could be in general interesting to researchers interested in using this resource. While I do not advocate for English only, providing an English translation of the French examples could help a wider audience who does not speak French and thus enhance the impact of the paper. \n5. Line 592: It is unclear what is the motivation to study the correlation between human annotations and automatic metrics. Such studies have been conducted by the authors of automated metrics, and it is unclear how it is contextualized in this work. More explanation of insights drawn based on this analysis, which is specifically applicable to this paper, could be useful. \n\n", "paper_topic_and_main_contributions": "This paper provides a resource for dialouge summarization in French. In particular, the authors curate a dataset of human-transcribed political dialogues in French and provide human-annotated summaries (both extractive and abstractive). The authors also provide alignment annotations between extractive and abstractive summary sentences, referred as abstractive communities. The authors further experiment with several existing summarization models and alignment approaches (based on clustering) and demonstrate the performance of current systems on this task. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 428 Rrcall-->Recall\nTable 5 is for 239 words? "}, "extra_text": {"Justification_for_Ethical_Concerns": "The authors scraped the official websites of the French Senate and Parliament to create a publicly available pre-training dataset in French. What were the content use-policies of these websites? Is creation of such dataset and training of models on such data allowed under this policy? If the curated dataset violates any terms of use, then this could be a major concern. While the authors mention briefly in the ethics section that the any data provided in this paper follows the terms and conditions of their respective sources, it would be important to explicitly state the allowed usage of this dataset (what is allowed and what is not) in the paper itself. For instance, commercial use of any information is not allowed, which should be mentioned in the paper to avoid use of such dataset for training any industrial solutions etc. This is only an example, more thorough understanding the policy use is required and need to be discussed in the paper."}}, {"review_id": "IXl2BYc4Fy", "reviewer": "Reviewer_jbix", "summary": "", "questions": "1.\tThe study mention employing “a NLP system” identify key phrases and cluster them into sentence groups by topic (line 263). Could author explain the system and any related quality control measures? Since the accuracy of such automatic grouping may significantly affect the quality of following summary annotation procedures.\n\n2.\tWhat do “preS” and “preA”  subscripts in Table 1 stand for? Also, the # token per transcript is unusually small (2595.5) for a meeting transcript, do the transcripts here are the after-extracted ones rather than full transcripts?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your detailed and insightful review. We respond point-by-point to your comments below\n\n**Reasons To Accept**:\nThe study presents the research community with a novel data in dialogue summarization. Notably, the dataset provides human-written abstractive and extractive summaries with different levels of abstractivity and semantic aspects.\n\n\n*We thank the reviewer for acknowledging that our dataset is a novel contribution as the first French dialogue summarization dataset, and provides multiple types of human annotations. We would like to resolve several misunderstandings.*\n\n---------------------------------------------------\n\n\n**Reasons To Reject:**\nWhile the scope and characteristic of the dataset proposed is specific, the size of the dataset (18 debates consisting of 138 sub-debates), as the author point out in the Limitation section, is relatively small.\n\n*This can be explained by the large time requirement for producing such data. While the size of our dataset is limited (138 sub-debates), it is on the same level as previous similar corpus, such as AMI (137 meetings) and ICSI (75 meetings), of which the small quantity has not prevented them from being widely used in related areas of research. \nAdditionally, we need to recall that for each dialogue transcription, we provide multiple annotations of 3 abstractive summaries, 2 extractive summaries, and 2 abstractive communities, while to the best of our knowledge, no previous dataset proposes more than one annotation for each dialogue.\nIn the end, apart from the annotated set, we also provide an unannotated set of debate transcriptions (FREDSum_preS and FREDSum_preA), consisting of more than 7000 debates, each longer than two hours. We believe these two sets are useful for the continual pre-training and Instruction-tuning of a LLM, to construct a system without requiring a significant amount of human annotations.*\n\n---------------------------------------------------\n\n**Reasons To Reject:** In the experiment, the comparability of the performance of the three models on abstractive summarization is questionable since the models were provided with source transcripts with different truncating strategies (transcripts were truncated by 512 for BARThez, 1024 for LLAMA 30, and mostly untruncated for GPT3.5).\n\n*The logic behind these experiments was to be able to reflect the best performance of each model and using them as would someone who had to summarize the debates with each of those systems. We have however run additional experiments to reflect the results obtained by following an alternative truncating strategy in which all models get as an input the longest string they can compute that does not cut an utterance. Expectedly, the results for Barthez and Open assistant both suffer from this strategy, but this does allow a fairer comparison between the models*\n\n\n\n*All models - No segmentation vs Abstractive 1 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.44 | 0.17 | 0.24 | 0.74      |\n| Open Assistant | 0.29 | 0.07 | 0.15 | 0.69      |\n| Barthez        | 0.14 | 0.04 | 0.09 | 0.64      |\n\n\n*All models - No segmentation vs Abstractive 2 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.42 | 0.17 | 0.23 | 0.73      |\n| Open Assistant | 0.30 | 0.07 | 0.16 | 0.69      |\n| Barthez        | 0.15 | 0.04 | 0.10 | 0.64      |\n\n\n\n*All models - No segmentation vs Abstractive 3 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.43 | 0.16 | 0.24 | 0.73      |\n| Open Assistant | 0.29 | 0.07 | 0.14 | 0.68      |\n| Barthez        | 0.13 | 0.03 | 0.08 | 0.63      |\n\n\n\n\n\n\n\n---------------------------------------------------\n\n**Reason to reject :** Moreover, the dataset contains two extractive and three different abstractive summaries for each sub-debate, and the study “score the candidate with each available reference and returning the highest score”, which is likely comparability-wise unsound.\n\n*When multiple references exist, it is the standard practice to first score the candidate summary with respect to each reference summary, and then keep the highest score as the final one. Please refer to the original ROUGE paper [1] (Section 2.1 Multiple References) and BERTScore paper [2] (page 6), as well as the metric implementations listed in the footnotes of our paper.*\n\n*[1] Lin, C. Y. (2004, July). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out (pp. 74-81).*\n\n*[2] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.*\n\n*Additionally, we propose to report below results obtained by each system against each abstractive and extractive summaries. Results show that the scaling of the scores stay the same with each evaluation strategies on abstractive summaries*\n\n\n*All models vs Abstractive 1 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.45 | 0.18 | 0.24 | 0.74      |\n| Open Assistant | 0.36 | 0.10 | 0.18 | 0.70      |\n| Barthez        | 0.36 | 0.10 | 0.17 | 0.68      |\n\n\n\n*All models vs Abstractive 2 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.43 | 0.17 | 0.23 | 0.74      |\n| Open Assistant | 0.36 | 0.09 | 0.18 | 0.71      |\n| Barthez        | 0.37 | 0.10 | 0.17 | 0.67      |\n\n\n\n*All models vs Abstractive 3 :* \n|                | R1   | R2   | RL   | BERTScore |\n|----------------|------|------|------|-----------|\n| ChatGPT        | 0.47 | 0.16 | 0.23 | 0.73      |\n| Open Assistant | 0.37 | 0.09 | 0.17 | 0.70      |\n| Barthez        | 0.37 | 0.08 | 0.16 | 0.68      |\n\n*All the results above will be provided in an appendix for the final version of the paper*\n\n---------------------------------------------------\n\n**Questions for the authors :** The study mention employing “a NLP system” identify key phrases and cluster them into sentence groups by topic (line 263). Could author explain the system and any related quality control measures? Since the accuracy of such automatic grouping may significantly affect the quality of following summary annotation procedures. \n\n*Abstractive communities are obtained via human annotations of alignment between extractive summary and abstractive summary. We give two pairs of abstractive communities for each of the 138 sub topics*\n\n*Additionally we will refine Section 2.2.3 by briefly introducing the two previous works on the sub-task of abstractive community detection. To find utterance clusters, one [1] uses sentence graph + community detection (CONGA) algorithm, another [2] uses sentence embeddings (obtained with siamese network) + Fuzzy-C means algorithm. The answers to the other questions can be found in Section 4.3, we use Omega Index to evaluate the performance.*\n\n*[1] Murray, G., Carenini, G., & Ng, R. (2012, June). Using the omega index for evaluating abstractive community detection. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization (pp. 10-18).*\n\n*[2] Shang, G., Tixier, A. J. P., Vazirgiannis, M., & Lorré, J. P. (2019). Energy-based self-attentive learning of abstractive communities for spoken language understanding. arXiv preprint arXiv:1904.09491.*\n\n\n---------------------------------------------------\n\n**Questions for the authors :** What do “preS” and “preA” subscripts in Table 1 stand for? Also, the # token per transcript is unusually small (2595.5) for a meeting transcript, do the transcripts here are the after-extracted ones rather than full transcripts?\n\n*PreS represents the statistics of the senate subsate of the pretraining data, while PreA represents the assembly*\n\n*The transcripts here refers to full transcripts, while the #tokens here actually represents the number of words; the number still being fairly low as some sub debates can be very short - Clarification for both of those points will be added to the paper*\n\n---------------------------------------------------\n\n*Thank you again for the review, we hope to have cleared up the points presented in your reasons to reject*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study presents the research community with a novel data in dialogue summarization. Notably, the dataset provides human-written abstractive and extractive summaries with different levels of abstractivity and semantic aspects.", "reasons_to_reject": "While the scope and characteristic of the dataset proposed is specific, the size of the dataset (18 debates consisting of 138 sub-debates), as the author point out in the Limitation section, is relatively small.\n\nIn the experiment, the comparability of the performance of the three models on abstractive summarization is questionable since the models were provided with source transcripts with different truncating strategies (transcripts were truncated by 512 for BARThez, 1024 for LLAMA 30, and mostly untruncated for GPT3.5).\n\nMoreover, the dataset contains two extractive and three different abstractive summaries for each sub-debate, and the study “score the candidate with each available reference and returning the highest score”, which is likely comparability-wise unsound. ", "questions_to_authors": "1.\tThe study mention employing “a NLP system” identify key phrases and cluster them into sentence groups by topic (line 263). Could author explain the system and any related quality control measures? Since the accuracy of such automatic grouping may significantly affect the quality of following summary annotation procedures.\n\n2.\tWhat do “preS” and “preA”  subscripts in Table 1 stand for? Also, the # token per transcript is unusually small (2595.5) for a meeting transcript, do the transcripts here are the after-extracted ones rather than full transcripts?", "paper_topic_and_main_contributions": "The paper presents a novel dialogue summarization dataset in French on political debates, in the context of a lack of non-English data resources of this kind. The paper collects data from open source and annotate transcripts with multi-granularity human-written summaries. The authors also conduct preliminary experiments benchmarking some models on abstractive and extractive summarization tasks against the dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A typo in Table 8, “Faitfullness” should be “Faithfulness”."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new dataset for multi-party dialogue summarization in French. It consists of transcribed political debates, which are segmented into sub-topics and annotated with abstractive and extractives summaries. The paper reports model performance on this dataset for several existing summarization models and alignment approaches (for aligning summaries with original speech).  

The reviewers raise several questions about the annotation process which are not sufficiently explained in the paper, and raise concerns regarding use-policies of the underlying data (the authors address these satisfactorily in their rebuttal).
Decision: Accept-Findings