Question: What does the SAMSum Corpus contain?
Evidence: The goal of the summarization task is condensing a piece of text into a shorter version that covers the main points succinctly. In the abstractive approach important pieces of information are presented using words and phrases not necessarily appearing in the source text. This requires natural language generation techniques with high level of semantic understanding BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6.
Major research efforts have focused so far on summarization of single-speaker documents like news (e.g., BIBREF7) or scientific publications (e.g., BIBREF8). One of the reasons is the availability of large, high-quality news datasets with annotated summaries, e.g., CNN/Daily Mail BIBREF9, BIBREF7. Such a comprehensive dataset for dialogues is lacking.
The challenges posed by the abstractive dialogue summarization task have been discussed in the literature with regard to AMI meeting corpus BIBREF10, e.g. BIBREF11, BIBREF12, BIBREF13. Since the corpus has a low number of summaries (for 141 dialogues), BIBREF13 proposed to use assigned topic descriptions as gold references. These are short, label-like goals of the meeting, e.g., costing evaluation of project process; components, materials and energy sources; chitchat. Such descriptions, however, are very general, lacking the messenger-like structure and any information about the speakers.
To benefit from large news corpora, BIBREF14 built a dialogue summarization model that first converts a conversation into a structured text document and later applies an attention-based pointer network to create an abstractive summary. Their model, trained on structured text documents of CNN/Daily Mail dataset, was evaluated on the Argumentative Dialogue Summary Corpus BIBREF15, which, however, contains only 45 dialogues.
In the present paper, we further investigate the problem of abstractive dialogue summarization. With the growing popularity of online conversations via applications like Messenger, WhatsApp and WeChat, summarization of chats between a few participants is a new interesting direction of summarization research. For this purpose we have created the SAMSum Corpus which contains over 16k chat dialogues with manually annotated summaries. The dataset is freely available for the research community.
The paper is structured as follows: in Section SECREF2 we present details about the new corpus and describe how it was created, validated and cleaned. Brief description of baselines used in the summarization task can be found in Section SECREF3. In Section SECREF4, we describe our experimental setup and parameters of models. Both evaluations of summarization models, the automatic with ROUGE metric and the linguistic one, are reported in Section SECREF5 and Section SECREF6, respectively. Examples of models' outputs and some errors they make are described in Section SECREF7. Finally, discussion, conclusions and ideas for further research are presented in sections SECREF8 and SECREF9.
For this purpose we have created the SAMSum Corpus which contains over 16k chat dialogues with manually annotated summaries.
Answer: Over 16k chat dialogues with manually annotated summaries.