# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 4 Experiment 1: Paired Comparison
### Experimental Results
#### 4.2.3 Comments from Subjects  
The evaluation results show that abstracts from the original paper (AbE) achieve the best score and manual summaries written in Japanese (MJ) are second (see Table 5). Here, we analyze subjects' comments and reveal the evaluation factors.  
Subjects' comments can be classified into one or more of the following four types; (1) content richness, (2) simplicity, (3) fluency, and (4) format (see Table 6). In terms of content richness, subjects noted whether the main contents of scientific papers (e.g., research purpose and experimental results) were included, and summaries with such information tended to get a higher score. In the case of simplicity, subjects answered whether summaries  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**B**} \\ \cline{3-6}  & & **AbE** & **AuE** & **AbJ** & **AuJ** \\ \hline  & MJ & \(0.06\pm 1.48\) & \(0.76\pm 0.90\) & \(-0.24\pm 1.52\) & \(1.18\pm 1.02\) \\ A & AbE & - & \(\mathbf{1.35\pm 1.06}\) & \(0.47\pm 1.46\) & \(\mathbf{1.47\pm 0.51}\) \\  & AuE & - & - & \(-0.41\pm 1.46\) & \(0.06\pm 1.30\) \\  & AbJ & - & - & - & \(0.65\pm 1.37\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Means and standard deviations of scores in experiment 1 for each pair. Higher mean scores indicate that A is better than B. Emboldening indicates significant difference.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**All**} & \multicolumn{2}{c}{**University**} & \multicolumn{2}{c}{**Degree (Year)**} & \multicolumn{2}{c}{**English Skill**} \\  & & **A Univ.** & **B Univ.** & **C Univ.** & **Bachelor’s (4)** & **Master’s (1)** & **Master’s (2)** & **Low** & **Medium** & **High** \\ \hline MJ & 0.35 & 0.32 & 1.00 & \(-0.12\) & 0.43 & 0.43 & 0.10 & 0.57 & 0.26 & 0.00 \\ AbE & 0.65 & 0.60 & 0.63 & 0.75 & 0.83 & 0.20 & 1.15 & 0.87 & 0.48 & 1.00 \\ AuE & \(-0.49\) & \(-0.36\) & \(-0.96\) & \(-0.64\) & \(-0.37\) & \(-0.49\) & \(-0.70\) & \(-0.53\) & \(-0.56\) & \(-0.60\) \\ AbJ & 0.16 & 0.16 & \(-0.12\) & 0.52 & \(-0.20\) & 0.37 & 0.35 & \(-0.20\) & 0.34 & 0.60 \\ AuJ & \(-0.67\) & \(-0.72\) & \(-0.92\) & \(-0.64\) & \(-0.70\) & \(-0.51\) & \(-0.90\) & \(-0.70\) & \(-0.62\) & \(-1.00\) \\ \hline \#subjects & 17 & 5 & 8 & 4 & 6 & 7 & 4 & 6 & 10 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Summary scores according to subjects’ attribute. The higher the number, the better the summary.  
were simple to read or not. It was considered that simple summaries were preferred; however, they should not be too simple. Indeed, about 20% of the comments on simplicity stated that \"summaries are too simple to grasp the overall meaning of the research.\" Moving on to fluency, this evaluation metric was concerned with whether subjects can read summaries without language difficulties, for example, whether a summary is written in their mother tongue or whether it is grammatically correct. In our experiment, some subjects mentioned that \"summaries written in Japanese are easy to understand\". By contrast, other subjects also answered that \"English is better than incorrect Japanese\" and \"I'm familiar with terminology in English.\" In the case of format, subjects evaluate based on the appearance of texts. For example, if summaries referred to figures or tables, the subjects keep lower scores because the output summaries consisted only of texts. Other comments that cannot be classified into the above four types include \"I cannot understand either summary\" and \"there is no difference between the two summaries.\" None of the comments that fall into the other types mention any characteristics of the summaries.