# An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements
## 2 Related Works in the Literature
### Multi-Threading for Multiple Search Sites in Academic Paper Retrieval  
First, we discuss papers on implementing multi-threading or parallelism techniques for running multiple search sites for academic papers. Multi-threading enables concurrent executions of multiple tasks related to requesting data from multiple sources. It can maximize the throughput for collecting papers and reduce the response time.  
In [28], [PERSON] et al. presented the _CiteSeerX_ milestone as an _open-access_-based scholarly Big Data service. _CiteSeerX_ collected data using an incremental web crawler developed with _Django_. Metadata processing used _Perl_ with a _multithreaded Java wrapper_ that was responsible for retrieving documents from the crawl database and managing parallel tasks [29]. They reported that almost 40-50% of the total PDF documents successfully found per day and identified as scholarly are stored in the database.  
In [30], [PERSON] et al. introduced _CORE_, a service that provides _open-access_ to research publications through bulk data access and downloads them via _RESTful_ APIs. The system ensures up-to-date data with the _FastSync_ mechanism and manages all the running tasks through _The CORE Harvesting System (CHARS)_. However, this paper highlights the limitation of determining the optimal number of workers, as allocation relies on empirical methods.  
In [31], [PERSON] et al. introduced _OpenAlex_, an open bibliographic catalogue covering a wide range of scientific papers and research information. _OpenAlex_ gathers heterogeneous directed graph datasets which collected data from _Microsoft Academic Graph_, _Crossref_, _PubMed_, institutional repositories, and e-prints. _OpenAlex_ used _async_ in _JavaScript_ to handle asynchronous operations and _parallel-transform_ to process data streams in parallel [32].  
In [33], [PERSON] et al. proposed _PaperBot_, an open source web-based software for automating the search process and indexing biomedical literature. It retrieved metadata and PDFs using multiple database APIs. Using _MongoDB_, it achieves scalable cluster operations to enable parallel processing and reducing latency. The results showed the increase in the throughput by five times in paper collections. However, storing PDF files as blobs led to increasing the RAM usage.  
In [34], [PERSON] et al. proposed _OpenAIRE_ workflows for data managements to support open science e-infrastructure, allowing shares and reuses of research products. _OpenAIRE_utilized a graph-based _Information Space Graph (ISG)_ to collect heterogeneous data from semantically linked research products, while the _D-Net Framework_ orchestrated workflows. _ISG_ used _Apache HBase_ for scalable columnar storage and _MapReduce_ for parallel processing of large graph datasets.  
In [35], [PERSON] et al. introduced _Paperity Central_, a global universal catalogue combining gold and green _open-access_ scholarly literature. _Paperity Central_ used an automatic aggregator to accelerate collections of large-scale new publications from _open-access_ sources and repositories while connecting various data items based on semantics. However, this paper has not discussed how the aggregation mechanism distributes tasks to workers while retrieving metadata from various data sources.  
Various academic search engines have been implemented for different types of multi-threading or parallel programming. Notably, it emphasizes the selection of multi-threading libraries and adjusts the number of workers based on the complexity and the system environment to maximize the throughput. Since the proposed system integrates multiple search sites and runs concurrently, implementing multi-threading is highly recommended.