# An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements
## 9 Evaluation and Analysis
### System Performance Metrics Comparison
#### 9.2.1 Relevant Accessible PDFs  
Access to full-text PDF articles is essential for researchers to conduct literature reviews. We compare the proposed system with our previous work to evaluate its ability to retrieve relevant accessible PDFs. The proposed system integrates the reference paper sources from multiple APIs, including _Semantic Scholar_[44], _DOAJ_[45], and _PubMed_[46], and collects more accessible PDF files than our previous work relying on _Google Scholar_. This evaluation result is illustrated in Figure 13. \begin{table}
\begin{tabular}{p{56.9 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Case(s)** & **Title Search Queries** & **Keyword (s)** \\ \hline Case 1 & Web Scraping using Selenium & Web Scraping, Selenium, Python, Browser Automation, Data Extraction \\ Case 2 & Deep Learning for Skin Detection & Deep learning, Skin Detection, Computer Vision, Convolutional Neural Network, Image Segmentation \\ Case 3 & Healthcare Chatbot Natural Language Processing & Chatbot, Conversational Agent, Natural Language Processing, Health Support, Question Answering System \\ Case 4 & Generative AI for Medical & Generative AI, Synthetic Medical Data, Report Generation, Generative Model, Generative Adversarial Network \\ Case 5 & Large Language Models & Natural Language Processing, Transformer Architecture, Pretrained Language Models, Text Generation, Prompt Engineering \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Five distinct title and keyword search queries. Figure 13 shows that the proposed system retrieved an average of 67 relevant accessible PDFs, while our previous work retrieved only 5 PDFs on average for the five distinct title and keyword search queries. Since the proposed system retrieved 87 articles in _Case 5_, it suggests that publications with the topic \"Large Language Models\" in recent years are well covered in academic literature. The proposed system also demonstrated the ability to retrieve articles with keywords containing disciplines and scientific domains in _Case 2_ and _Case 4_, where 99 articles related to \"Deep Learning for Skin Detection\" and 85 articles related to \"Generative AI for Medical\" were retrieved, respectively. It indicates that applications of AI in the medical field are growing, with an increasing number of articles being _open-access_. Unfortunately, the proposed system has a few relevant accessible PDFs in _Case 1_ and _Case 3_, where the system only obtained 13 articles related to \"Web Scraping using Selenium\" and 53 articles related to \"Healthcare Chatbot Natural Language Processing\", respectively. This suggests that these specific fields may face limited accesses to the academic literature on _web scraping_ that discusses technical aspects using _Selenium_. The applications of _Chattots_ for healthcare may be newer. Furthermore, to measure the effectiveness of the relevant accessible PDFs enhancement, we calculated the percentage increase. This objective is to calculate how large the percentage of enhancements in relevant accessible PDFs between our previous work and the proposed system, following the _Min-Max normalization_, to obtain a scale of 0-100% [55] in Equations (1) and (2):  
\[x_{\text{norm}}=\frac{x-\min(x)}{\max(x)-\min(x)} \tag{1}\]  
\[\Delta_{\text{enhance}}=x_{\text{norm2}}-x_{\text{norm1}} \tag{2}\]  
We assumed that \(x\) represents the number of relevant accessible PDFs for either our previous work or the proposed system, \(min(x)\) does the minimum value as the lowest scale equivalent to 0, and \(max(x)\) does the overall total collected paper. Subsequently, \(\Delta_{\text{enhance}}\) is used to calculate the percentage increase between the normalized values \(x_{\text{norm2}}\) for the proposed system, and \(x_{\text{norm1}}\) is for our previous work. The proposal increases the number of available relevant reference papers with full text PDFs by 64.38% on average, compared to previous work. It indicates a significant performance achievement of the  
Figure 13: Comparison of relevant accessible PDFs.