# An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements
## 9 Evaluation and Analysis
### Performance Comparison with Common Literature Search Tools
#### 9.3.1 Results  
Tables 5-9 shows the comparison results.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**\# of Papers**} & \multicolumn{2}{c}{**Access Type**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Accessibility**} & \multirow{2}{*}{**Response Time (s)**} \\ \cline{3-3} \cline{5-9}  & & **\#of Open A.** & & **\#of Closed A.** & & **\#of Papers** & **Rate (\%)** & **\#of Papers** & **Rate (\%)** \\ \hline Publish or Perish [24] & 100 & 57 & 43 & 31 & 31.00\% & 31 & 31.00\% & 1.6 s \\ OpenAIRE [25] & 11 & 6 & 5 & 1 & 9.09\% & 1 & 9.09\% & 3.67 s \\ OpenAlex [26] & 14 & 7 & 7 & 4 & 28.57\% & 4 & 0.00\% & 2.25 s \\ Previous Work [7] & 8 & 8 & 0 & 8 & 100.00\% & 8 & 100.00\% & 40.88 s \\ Google Scholar [27] & 4 & 4 & 0 & 4 & 100.00\% & 4 & 100.00\% & 0.03 s \\
**Proposal** & **100** & **97** & **3** & **99** & **99.00\%** & **99** & **99.00\%** & **41.29 s** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance comparison between proposed system and common literature search tools in _Case 2_.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**\# of Papers**} & \multicolumn{2}{c}{**Access Type**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Accessibility**} & \multirow{2}{*}{**Response Time (s)**} \\ \cline{3-3} \cline{5-9}  & & **\#of Open A.** & & **\#of Closed A.** & & **\#of Papers** & **Rate (\%)** & **\#of Papers** & **Rate (\%)** \\ \hline Publish or Perish [24] & 36 & 23 & 13 & 13 & 36.11\% & 13 & 36.11\% & 1.15 s \\ OpenAIRE [25] & 5 & 4 & 1 & 0 & 0.00\% & 0 & 0.00\% & 3.39 s \\ OpenAIRE [26] & 0 & 0 & 0 & 0 & 0.00\% & 0 & 0.00\% & 2.38 s \\ Previous Work [7] & 7 & 7 & 0 & 5 & 71.43\% & 5 & 71.43\% & 89.02 s \\ Google Scholar [27] & 1 & 0 & 1 & 0 & 0.00\% & 0 & 0.00\% & 0.04 s \\
**Proposal** & **66** & **65** & **1** & **53** & **80.30\%** & **53** & **80.30\%** & **17.7 s** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Performance comparison between proposed system and common literature search tools in _Case 3_.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**\# of Papers**} & \multicolumn{2}{c}{**Access Type**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Accessibility**} & \multirow{2}{*}{**Response Time (s)**} \\ \cline{3-3} \cline{5-9}  & & **\#of Open A.** & & **\#of Closed A.** & & **\#of Papers** & **Rate (\%)** & **\#of Papers** & **Rate (\%)** \\ \hline Publish or Perish [24] & 14 & 7 & 7 & 6 & 42.86\% & 6 & 42.86\% & 4.88 s \\ OpenAIRE [25] & 2 & 2 & 0 & 0 & 0.00\% & 0 & 0.00\% & 4.31 s \\ OpenAlex [26] & 0 & 0 & 0 & 0 & 0.00\% & 0 & 0.00\% & 2.67 s \\ Previous Work [7] & 6 & 6 & 0 & 4 & 66.67\% & 4 & 66.67\% & 27.46 s \\ Google Scholar [27] & 23 & 12 & 11 & 6 & 26.09\% & 7 & 30.43\% & 0.03 s \\
**Proposal** & **13** & **11** & **2** & **13** & **100.00\%** & **13** & **100.00\%** & **33.64 s** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance comparison between proposed system and common literature search tools in _Case 1_.