# An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements
## 9 Evaluation and Analysis
### Discussion on Proper Web APIs  
The proposed system outperforms in providing relevant accessible PDFs using multiple APIs, compared to our previous work and common literature search tools. By understanding the benefit of multiple API utilization, we further discuss the proper external web APIs to highlight the differences in providing reference papers with accessible PDFs and cover various topics in each case.  
Table 10 shows the percentage of each web API in providing accessible PDFs. It suggests that _PubMed_ has the highest contribution at 39%, followed by _Semantic Scholar_ at 32%, and _DOAJ_ has the smallest contribution at 29%.  
_PubMed_ provides 42% in _Case 2_ to the topic \"Deep Learning for Skin Detection\", 74.24% in _Case 3_ to the topic \"Healthcare Chatbot Natural Language Processing\", and 49% in _Case 5_ related to the topic \"Large Language Models\". It suggests that _PubMed_ primarily covers articles in AI applications with a strong emphasis on the medical domain.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**\# of Papers**} & \multicolumn{2}{c}{**Access Type**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Accessibility**} & \multirow{2}{*}{**Response Time (s)**} \\ \cline{3-3} \cline{5-8}  & & **\#of Open A.** & & **\#of Closed A.** & & **\#of Papers** & **Rate (\%)** & **\#of Papers** & **Rate (\%)** \\ \hline Publish or Perish [24] & 100 & 65 & 35 & 27 & 27.00\% & 27 & 27.00\% & 2.45 s \\ OpenAIRE [25] & 3 & 3 & 0 & 0 & 0.00\% & 0 & 0.00\% & 3.64 s \\ OpenAlex [26] & 8 & 5 & 3 & 1 & 12.50\% & 1 & 12.50\% & 2.2 s \\ Previous Work [7] & 5 & 5 & 0 & 5 & 100.00\% & 5 & 100.00\% & 92.05 s \\ Google Scholar [27] & 100 & 77 & 23 & 68 & 68.00\% & 68 & 68.00\% & 0.06 s \\
**Proposal** & **100** & **93** & **7** & **85** & **85.00\%** & **99** & **99.00\%** & **33.8 s** \\ \hline \hline \end{tabular}
\end{table}
Table 8: Performance comparison between proposed system and common literature search tools in _Case 4_.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**\# of Papers**} & \multicolumn{2}{c}{**Access Type**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Accessibility**} & \multirow{2}{*}{**Response Time (s)**} \\ \cline{3-3} \cline{5-8}  & & **\#of Open A.** & & **\#of Closed A.** & & **\#of Papers** & **Rate (\%)** & **\#of Papers** & **Rate (\%)** \\ \hline Publish or Perish [24] & 14 & 11 & 3 & 6 & 42.86\% & 6 & 42.86\% & 1.29 s \\ OpenAIRE [25] & 0 & 0 & 0 & 0 & 0.00\% & 0 & 0.00\% & 2.09 s \\ OpenAIRE [26] & 100 & 81 & 19 & 28 & 28.00\% & 33 & 33.00\% & 1.99 s \\ Previous Work [7] & 2 & 2 & 0 & 2 & 100.00\% & 2 & 100.00\% & 186.09 s \\ Google Scholar [27] & 100 & 86 & 14 & 77 & 77.00\% & 77 & 77.00\% & 0.07 s \\
**Proposal** & **100** & **93** & **7** & **87** & **87.00\%** & **99** & **99.00\%** & **48.74 s** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Performance comparison between proposed system and common literature search tools in _Case 5_.  
_Semantic Scholar_ provides 84.62% in _Case 1_ to the topic \"Web Scraping using Selenium\". It indicates that _Semantic Scholar_ effectively provides articles in fields of computer science, especially with technologies and libraries.  
_DOAJ_ provides 42% in _Case 2_ to the topic \"Deep Learning for Skin Detection\", 54% in _Case 4_ to the topic \"Generative AI for Medical\", and 38% in _Case 5_ to the topic \"Large Language Models\". This is a broad and general topic, which, with _DOAJ_, covers diverse topics across disciplines.  
These findings emphasize the importance of selecting the proper web APIs based on the research focus. However, exploration of other external web APIs such as _Dimensions_, _CrossRef_, _ArXiv_, _BiorXiv_, and _PLOS_[14] have the potential to be integrated into the system to enrich the search results and topic coverage, which will be in our future works.