# A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students
## 5 Experiment 2: Reading Thorough
### Experimental Results
#### 5.2.3 Subsequent Question  
After experiment 2, we also asked the subjects to answer whether the four characteristics in experiment 1 (see Table 6) are important or not. The responses from the subjects are summarized in Figure 4.  
For the question about the importance of content richness, about 80% of the subjects answered either \"I think it is important\" or \"I think it is somewhat important.\" On the other hand, two subjects answered \"I do not know\" or \"I think it is somewhat unimportant\". Their reasons were that \"the detailed experimental results are not necessary\" (by AmM05), and \"content richness is not necessary and I would rather know a little bit each about a larger number of papers\" (by CMM17).  
In the case of simplicity, around 80% of subjects agreed that simplicity is important. The reasons include \"a longer summary is difficult to understand\" (by CmM16) and \"I would like to quickly read an easily understandable summary\" (by CmM15). On the other hand, three subjects answered \"I do not know\" because \"I cannot understand the summary with insufficient information even if it is simple\" (by ABH01, BmL12, and BML13). From the answers for content richness and simplicity, it is considered that these aspects need to be well balanced.  
\begin{table}
\begin{tabular}{l l l} \hline \hline  & **\#Comments** & **Example** \\ \hline Topic & 280 & I am interested in face recognition. (by CmM16) \\ Method & 78 & I am interested in the algorithm. (by BBL08) \\ Experiment & 18 & I would like to know the synthesis result. (by ABM04) \\ Other & 84 & I am interested. (by BBL06) \\ \hline \hline \end{tabular}
\end{table}
Table 10: Distribution of interest-related comments regarding the research content.  
Figure 4: Subjects’ opinions about the importance of the four evaluation factors from experiment 1.  
In case of fluency, 53.4% of subjects placed importance on fluency, which is lower than the percentages for the importance of content richness and simplicity. To investigate the importance of fluency further, we also asked subjects to answer \"how often did you prefer Japanese summaries, even if the text was grammatically incorrect, compared to English ones (0%: English is always better, 100%: Japanese is always better)\" to investigate the effect of the mother tongue (see Table 11). All subjects with low-level English skills answered more than \"50%,\" and one subject with medium-level English skill answered \"80%.\" [PERSON], who had lived in the U.S. for 10 years, answered \"0%.\" The reasons are including \"appropriateness is more important than whether the summary is written in the mother tongue or not\" (answer: 20%, by ABM03), \"I usually read scientific papers in English\" (answer: 10%, by CMM17) and \"I can read text in the mother tongue quickly\" (answer: 70%, by CmM16).  
For the importance of format, 73.3% of subjects answered \"I think it is important\" or \"I think it is somewhat important.\" Examples of reasons are \"incorrect formatting bothers me every time\" (by ABM04) and \"I cannot understand the text because of inappropriate formatting\" (by CMM18). On the other hand, those who answered \"I think it is somewhat unimportant\" pointed out that \"I can correct the text in my mind\" (by ABM03) or \"even if the format is incorrect, it does not make hard to read\" (by [PERSON]). While some did not care about incorrect format, most did.