# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 5 Experiment 2: Reading Thorough
### Experimental Results
#### 5.2.2 Comments from Subjects  
Subjects' comments in experiment 2 can be divided into two categories, as shown in Table 9. The first type is about \"interest.\" A lot of subjects mentioned that they scored based on their interests; for example, the topic was close to their research topic, and they would like to know more detailed experimental results. The second category is based on \"quality or comprehensiveness.\" While some subjects mentioned that they could not understand the summary due to its quality, others mentioned that their interest was stimulated because the summary was high quality. In the case that the comment mentioned both \"interest\" and \"quality or comprehensiveness,\" we counted both. Note that three responses were not considered as one of these two types, in which subjects mentioned that they had already read the paper before. \begin{table}
\begin{tabular}{c c} \hline \hline
**Score** & **Criteria** \\ \hline
5 & I would strongly like to read it. \\
4 & I would like to read. \\
3 & I cannot judge. \\
2 & I would not like to read. \\
1 & I would strongly not like to read. \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Evaluation metric in experiment 2. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**All**} & \multicolumn{3}{c}{**University**} & \multicolumn{3}{c}{**Degree (Year)**} & \multicolumn{3}{c}{**English Skill**} \\  & & **A Univ.** & **B Univ.** & **C Univ.** & **Bachelor’s (4)** & **Master’s (1)** & **Master’s (2)** & **Low** & **Medium** & **High** \\ \hline Overall & 3.30 & 3.08 & 3.26 & 3.54 & 3.31 & 3.18 & 3.60 & 3.33 & 3.34 & 2.84 \\ MJ & 3.93 & 3.91 & 3.85 & 4.19 & 3.98 & 3.97 & 3.40 & 3.57 & 3.96 & 4.18 \\ AbE & 3.59 & 3.72 & 3.34 & 4.00 & 3.75 & 3.19 & 4.38 & 3.53 & 3.56 & 4.00 \\ AuE & 3.02 & 2.92 & 2.99 & 3.19 & 2.92 & 3.04 & 3.25 & 3.02 & 3.15 & 2.20 \\ AbJ & 3.07 & 2.62 & 3.11 & 3.34 & 3.08 & 3.04 & 3.12 & 3.18 & 3.13 & 2.00 \\ AuJ & 2.87 & 2.62 & 2.91 & 3.00 & 2.89 & 2.78 & 3.06 & 2.89 & 2.92 & 2.40 \\ \hline \#subjects & 15 & 4 & 7 & 4 & 6 & 6 & 3 & 5 & 9 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Summary scores according to subjects’ attributes in experiment 2. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**(a) The count of reason of each type.**} \\ \hline  & \multicolumn{5}{c}{**\#answers**} \\ \cline{2-7}  & \multicolumn{1}{c}{**Type of Reason**} & \multicolumn{1}{c}{**MJ**} & \multicolumn{1}{c}{**AbE**} & \multicolumn{1}{c}{**AuE**} & \multicolumn{1}{c}{**AbJ**} & \multicolumn{1}{c}{**Auj**} \\ \hline Interest & Positive & 83 & 70 & 56 & 50 & 48 \\ \cline{2-7}  & Negative & 22 & 36 & 21 & 40 & 30 \\ \hline Quality or comprehension & Positive & 28 & 19 & 7 & 13 & 8 \\ \cline{2-7}  & Negative & 7 & 10 & 50 & 34 & 57 \\ \hline Others & & 1 & 0 & 2 & 0 & 0 \\ \hline Mean score & & 3.93 \(\pm\) 0.84 & 3.59 \(\pm\) 0.93 & 3.02 \(\pm\) 1.18 & 3.07 \(\pm\) 1.09 & 2.87 \(\pm\) 1.20 \\ \hline Total answers & & 129 & 121 & 126 & 122 & 127 \\ \hline \multicolumn{7}{c}{**(b) The examples of the reason.**} \\ \hline  & \multicolumn{5}{c}{**Type of reason**} & \multicolumn{5}{c}{**Example**} \\ \hline Interest & Positive & The topic is relevant to my research. (by BBL(07) & \\ Negative & The topic is not relevant to my research. (by [PERSON]) & \\ \hline Quality or comprehension & Positive & I became interested in the research because the summary was high quality. (by [PERSON]) & \\ Negative & I could not understand the summary. (by [PERSON]) & \\ \hline Others & & 1 have read the paper before. (by [PERSON]) & \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Scoring reasons in experiment 2. We discuss the comments about \"interest\" in more detail. The answers about \"interest\" can be classified according to the components of scientific papers (e.g., topic, method, and experiment), as shown in Table 10. We consider that comments mentioning multiple components are counted for each component, and 14 answers were of this type.