Abstract: This paper introduces the SAMSum Corpus, a new dataset with abstractive dialogue summaries. We investigate the challenges it poses for automated summarization by testing several models and comparing their results with those obtained on a corpus of news articles. We show that model-generated summaries of dialogues achieve higher ROUGE scores than the model-generated summaries of news -- in contrast with human evaluators' judgement. This suggests that a challenging task of abstractive dialogue summarization requires dedicated models and non-standard quality measures. To our knowledge, our study is the first attempt to introduce a high-quality chat-dialogues corpus, manually annotated with abstractive summarizations, which can be used by the research community for further studies.

Introduction: The goal of the summarization task is condensing a piece of text into a shorter version that covers the main points succinctly. In the abstractive approach important pieces of information are presented using words and phrases not necessarily appearing in the source text. This requires natural language generation techniques with high level of semantic understanding BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6. Major research efforts have focused so far on summarization of single-speaker documents like news (e.g., BIBREF7) or scientific publications (e.g., BIBREF8). One of the reasons is the availability of large, high-quality news datasets with annotated summaries, e.g., CNN/Daily Mail BIBREF9, BIBREF7. Such a comprehensive dataset for dialogues is lacking. The challenges posed by the abstractive dialogue summarization task have been discussed in the literature with regard to AMI meeting corpus BIBREF10, e.g. BIBREF11, BIBREF12, BIBREF13. Since the corpus has a low number of summaries (for 141 dialogues), BIBREF13 proposed to use assigned topic descriptions as gold references. These are short, label-like goals of the meeting, e.g., costing evaluation of project process; components, materials and energy sources; chitchat. Such descriptions, however, are very general, lacking the messenger-like structure and any information about the speakers. To benefit from large news corpora, BIBREF14 built a dialogue summarization model that first converts a conversation into a structured text document and later applies an attention-based pointer network to create an abstractive summary. Their model, trained on structured text documents of CNN/Daily Mail dataset, was evaluated on the Argumentative Dialogue Summary Corpus BIBREF15, which, however, contains only 45 dialogues. In the present paper, we further investigate the problem of abstractive dialogue summarization. With the growing popularity of online conversations via applications like Messenger, WhatsApp and WeChat, summarization of chats between a few participants is a new interesting direction of summarization research. For this purpose we have created the SAMSum Corpus which contains over 16k chat dialogues with manually annotated summaries. The dataset is freely available for the research community. The paper is structured as follows: in Section SECREF2 we present details about the new corpus and describe how it was created, validated and cleaned. Brief description of baselines used in the summarization task can be found in Section SECREF3. In Section SECREF4, we describe our experimental setup and parameters of models. Both evaluations of summarization models, the automatic with ROUGE metric and the linguistic one, are reported in Section SECREF5 and Section SECREF6, respectively. Examples of models' outputs and some errors they make are described in Section SECREF7. Finally, discussion, conclusions and ideas for further research are presented in sections SECREF8 and SECREF9.

Question: What models have been evaluated?
Answer: MIDDLE-n, LONGEST-n, LONGER-THAN-n and MOST-ACTIVE-PERSON are the baselines, and experiments also carried out on Pointer generator networks, Transformers, Fast Abs RL, Fast Abs RL Enhanced, LightConv and DynamicConv