# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 6 Discussion
### Characteristics of Manual and Automatic Summarization
#### 6.2.1 Manual Summarization  
We prepared (a) English abstracts in the original papers (AbE) and (b) manual Japanese summaries written by researchers with a Ph.D. (MJ). In experiment 1 (see Table 5), AbE obtained the highest score of 0.65 and MJ obtained the second highest score 0.35, indicating that manual summarization is highly evaluated. Note that the MJ score was not far behind, since there is not any significant difference with AbE.  
However, in the case of AbE, some subjects answered \"(MJ is better than AbE because MJ) describes meaningful information about the test accuracy with all datasets in the papers' experiments\" (by [PERSON]) and \"(AuE is better than AbE because AuE) describes what is done in this research, while the other one (AbE) does not\" (by [PERSON]). According to these comments, we can see that AbE does not always include important information for readers. This point is consistent with the claim by several researchers that abstract does not always reflect readers' requirements [3,4].  
On the other hand, MJ summaries depend on the Japanese researchers' subjective judgment, but it is also possible to include important information (which is not written in the AbE summary) for increased ease of understanding by considering the readers' perspective. We also observed comments about document simplification in MJ, for example \"(MJ) seems to be re-written by humans who have read the original paper carefully, so I feel (MJ) is easy to understand\" (by [PERSON]) and \"(MJ) is good because it describes the meaning of the terminology\" (by [PERSON]). However, some subjects also mentioned low fluency like \"the Japanese is incorrect in (MJ) and there is typographical error\" (by [PERSON]). There was also a comment that MJ is too simple to understand: \"(MJ) is not bad but too simple\" (by [PERSON]). One of the possible reasons for the above negative comments is that MJ is not checked by third-party readers, while the abstract (AbE) is evaluated during peer review. In addition, MJ is written by researchers who have obtained a Ph.D., but the subjects in our experiments are undergraduate and Master's students. Therefore, the gap in prior knowledge might be a reason for the aforementioned comments that MJ summaries are \"too simple.\"  
In comparison between MJ and AbE in the experiment 1, eight subjects answered \"(MJ) is better\" and nine answered \"(AbE) is better,\" which indicates that opinions are divided in half. Among those who answered \"(MJ) is better,\" five subjects mentioned the reason is that MJ is written in Japanese. Four subjects, including one who mentioned that Japanese is preferred, pointed out terms of simplicity or content richness. According to these comments, we can see that AbE is not necessarily better in simplicity and content richness.  
Whereas we have observed several characteristics of manual summarization through the experiments, it is not easy to draw any definite conclusions regarding the evaluation of manual summarization. This inconclusiveness indicates that more data are necessary while writing a summary of scientific papers is time-consuming. Additionally, the difficulty of concluding is also a sign that evaluation criteria could be more explicit as summaries are evaluated in a subjective manner in this study.