# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 3 Problem Formulation
### Reading Material  
As reading material, we randomly selected 70 papers (20 and 50 for the short-term experiment and the long-term experiment, respectively) which were accepted to the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018. This conference is one of the most successful conferences in the field of computer vision (CV) ([[https://cvpr2018.thecvf.com/](https://cvpr2018.thecvf.com/)]([https://cvpr2018.thecvf.com/](https://cvpr2018.thecvf.com/)) last accessed on 9 June 2021). Over 3300 papers were submitted to CVPR 2018 and 979 of them were accepted (i.e., the acceptance rate is about 29.7%), indicating the difficulty of getting accepted to CVPR. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Subject** & **University** & **Degree (Year)** & **English Skill** & **Experiment** \\ \hline ABH01 & A Univ. & Bachelor’s (4) & High a & Experiment 1, 2 \\ ABM02 & A Univ. & Bachelor’s (4) & Medium & Experiment 1 \\ ABM03 & A Univ. & Bachelor’s (4) & Medium & Experiment 1, 2 \\ ABM04 & A Univ. & Bachelor’s (4) & Medium & Experiment 1, 2 \\ AmM05 & A Univ. & Master’s (1) & Medium & Experiment 1, 2 \\ BBL06 & B Univ. & Bachelor’s (4) & Low & Experiment 1, 2 \\ BBL07 & B Univ. & Bachelor’s(4) & Low & Experiment 1, 2 \\ BBL08 & B Univ. & Bachelor’s (4) & Low & Experiment 2 \\ BmM09 & B Univ. & Master’s (1) & Medium & Experiment 1, 2 \\ BmM10 & B Univ. & Master’s (1) & Medium & Experiment 1, 2 \\ BmL11 & B Univ. & Master’s (1) & Low & Experiment 1 \\ BmL12 & B Univ. & Master’s (1) & Low & Experiment 1, 2 \\ BML13 & B Univ. & Master’s (2) & Low & Experiment 1, 2 \\ BML14 & B Univ. & Master’s (2) & Low & Experiment 1 \\ CmM15 & C Univ. & Master’s (1) & Medium & Experiment 1, 2 \\ CmM16 & C Univ. & Master’s (1) & Medium & Experiment 1, 2 \\ CMM17 & C Univ. & Master’s (2) & Medium & Experiment 1, 2 \\ CMM18 & C Univ. & Master’s (2) & Medium & Experiment 1, 2 \\ \hline \hline \end{tabular} a  
\end{table} 
Table 1: Subjects, their characteristics, and experiments in which they participated. Next, we prepared five-different summaries (see Table 2): (1) Manual Japanese (MJ): Japanese summaries written as part of a Japanese research project [6]. To manually construct all of the accepted paper summaries in the CV field, this project took around three months for each CV conference. Note that this project has no requirement for summary length as long as it is no longer than a single page. For our experiments, we chose summaries written by three professional Japanese researchers who received a Ph.D. degree in either the CV field or a related domain within the past five years. The average number of Japanese characters per summary is \(294.9\pm 48.6\). (2) AbE (Abstract English): We use English abstracts written in the original papers by the authors. The average number of words per summary is \(164.3\pm 30.2\). (3) AuE (Automatic English): English summaries generated by an automatic text summarization method. To achieve the assistance of reading comprehension, it is necessary to select an appropriate summarization method without dependency on any additional data, which is applicable to any scientific research paper. We employed an unsupervised algorithm, called TextRank [27], to compute sentence importance based on sentence connectivity and generate summaries that consist of important sentences only. TextRank is still a strong baseline which shows competitive performance to a recent summarization method [19]. Note that we exclude the \"abstract\" and the \"reference\" sections in the original papers and use only the paper bodies as inputs. To compute importance, we use variations of the similarity function [28] implemented in the gensim library ([[https://radimrehurek.com/gensim/](https://radimrehurek.com/gensim/)]([https://radimrehurek.com/gensim/](https://radimrehurek.com/gensim/)) last accessed on 9 June 2021). In addition, we empirically set the maximum number of words in each summary to 250 words.