# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 2 Related Work  
In the NLP field, an automatic generation of text summaries is an active topic [7] and several researchers have focused on scientific publications [4; 9; 10; 13; 14; 15; 16]. The majority of research on text summarization has focused on news articles [4; 9; 10]. However, summarizing scientific publications raises unique issues compared to summarizing news articles. For example, scientific papers tend to be longer than news articles (often much longer) and scientific articles follow normalized discourse structures including background, experimental settings, and results [17]. Accordingly, research on scientific publication summarization relies on particular characteristics, for example, section-based [9], structure-based [4; 10; 14], and citation-based methods [4; 10; 13]. [PERSON] et al. [18] point out that text summarization results should be evaluated manually based on readability and the appropriateness of the information they contain. In [11], systems to generate summaries are evaluated by human readers using four criteria: relevance, consistency, fluency, and coherence. The authors of [11] also compared human responses with automatic evaluation metrics used in text summarization research [12]. [PERSON] et al. [19] defined eight errors including missing key points or unnecessary repetition and asked users to manually select errors in computer generated summaries to investigate the limitations of prevailing automatic summarization methods. In these studies, the CNN/DM dataset [20], which consists of news articles, is used for manual evaluation of summaries. Compared with news articles, an important characteristic in scientific publication is that target readers are limited to researchers with prior knowledge, as well as text-length and article structure. In this work, inspired by these studies, we conducted a user study of reading summaries of scientific papers. In general, university students working on Bachelor's or Master's theses do not have enough experience of research activities, so we think that summaries are needed to help students understand the main point of research papers. [PERSON] et al. [21] investigated the effect of prior knowledge and reading skill on scientific text comprehension. Their results indicate that comprehension is positively correlated with prior knowledge. [PERSON] and [PERSON] [22] observed that reading strategies of faculty members in mathematics are different from those of undergraduate students. In addition, [PERSON] et al. [23] studied  
Figure 1: Experimental configuration. In experiment 1, subjects compare two types of paper summaries generated with different methods, named “summary A” and “summary B.” In experiment 2, subjects read summaries and judge whether they would like to read the corresponding papers within four weeks. science communications and revealed that specialized terminology is often used, which creates access barriers for beginners. Hence, we focus on university students as subjects, and investigate the characteristics in summaries that they regard as important. Scientific research papers are generally written in English, but it is preferred that readers can read in their mother tongues. This is because non-native English speakers tend to get lower scores on science examinations conducted in English, compared with native English speakers [24]. Previously, [PERSON] and [PERSON] [25] investigated science education for Chinese students in an English-speaking school, specifically an Australian high school. They found that those students prefer to have conversations in Chinese and take notes in Chinese. In this paper, if we conduct a user study with non-native English speakers (Japanese), we need to investigate the influence of language differences. We consider two major directions to obtain information using subjects' mother tongues: manual translation and automatic translation. In terms of the latter, machine translation systems have been well studied and freely accessible online systems like Google translate [26] are available.