# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 1 Introduction  
With the increasing number of scientific publications [1], scientific researchers (including university students with little research experience) are often faced with an enormous number of studies in their domains of interest. It can be very time-consuming and tedious to read these papers. To alleviate this burden, it is important to find ways whereby papers can be surveyed more efficiently. In this paper, we focus on a summary which briefly describes an overview of scientific research. One straightforward approach is to read only the \"abstract\" sections (which are manually written by authors) of papers. While abstracts are intended to attract readers [2], [PERSON] et al. mentioned that abstracts can also be used as summaries of scientific publications [3]. In general, the abstracts of foreground published papers have been evaluated during the peer-review process by third party researchers, so their quality is guaranteed. However, several researchers claimed that abstracts used do not necessarily meet readers' requirements because they are written from the perspective of authors [3, 4]. Hence, instead of author-written abstracts, it is important to stand in the place of the reader who may not have substantive domain-relevant knowledge and construct brief descriptions that help users gain an overview of research papers. In this paper, we considertwo types of summaries: abstracts written by authors and text that briefly describes an entire paper's overview. There are two ways to generate summaries from an original paper. Several research groups in Japan have attempted to manually construct summaries of academic articles to share knowledge concerning computer science research in, for example, the field of Human-Computer Interaction (HCI) [5] and Computer Vision (CV) [6]. As the manual summaries are written from the perspective of readers, it is possible to help other readers understand the research without having to read the target papers. In addition, these manual summaries are written in Japanese, so they are advantageous for these non-native English speakers. However, writing summaries manually is still difficult and tedious since the first readers (who write summaries) must understand the target research papers to a significant extent. In addition, these summaries have not been reviewed by others (i.e., professional reviewers) and the quality of summaries (e.g., adequacy of information) is not guaranteed. To alleviate the issue of manual summary writing, automatic generation of research summaries (written in readers' mother tongues) from the original papers has been demanded. In the field of Natural Language Processing (NLP), automatic summarization [7] and machine translation [8] are active research topics. Although these techniques can generate research summaries without any manual effort, the majority of work on text summarization has focused on news articles, which are markedly different from scientific publications in terms of text length or document structure [4; 9; 10]. Moreover, research on manual evaluation of automatic summarization also utilizes news articles as material [11]. Investigating users' preference and request is beneficial since summaries are intended to convey information to humans, but the majority of automatic summarization research relies on automatic evaluation metrics [12]. To establish a framework to automatically generate summaries of scientific papers, we need to elicit readers' opinions about research article summaries as a first step. In this work, we conduct a case study wherein Japanese university students read summaries written manually (including abstract by authors or summary by readers) and generated automatically (using NLP techniques) to investigate what characteristics readers require in terms of summaries of scientific papers.