# An Improved Reference Paper Collection System Using Web Scraping with Three Enhancements
## 7 Fast Checking of PDF File Accessibility
### Dynamic URL Redirection  
_Dynamic URL Redirection_ was inspired by the research proposed of [PERSON] et al. (2018), which involves rerouting URL requests to new destinations, to simplify access to online resources. We apply this method to all the external web APIs. Figure 7 shows the flowchart of _Dynamic URL Redirection_ on _Semantic Scholar_. It begins by sending an HTTP GET request to the _Semantic API endpoint_. The response from the API is received in the _JSON_ format, and then, the article metadata is parsed. It is necessary to use a _ThreadPoolExecutor_, and submit the PDF URL of each article and a PDF checking accessibility function to the executor. The function inside retrieves the _open access_ status and check the accessibility of the PDF URL in its variable one. If the PDF URL is accessible, the system will retrieve it and mark it as available. Subsequently, article metadata will be compiled into a paper list. On other hand, if the PDF URL is inaccessible upon accessing the initial URL of _Semantic Scholar_ article, construct an alternative URL. The initial URL [[https://www.semanticscholar.org/paper/](https://www.semanticscholar.org/paper/)]([https://www.semanticscholar.org/paper/](https://www.semanticscholar.org/paper/))[PaperID] is redirected to [[https://www.semanticscholar.org/reader/](https://www.semanticscholar.org/reader/)]([https://www.semanticscholar.org/reader/](https://www.semanticscholar.org/reader/))[PaperID]. The paper ID obtained from the initial URL is assigned to the new URL path. After the redirection to the URL path with the segment \"/reader/\", the system enhances the retrieval of an accessible PDF. If the PDF URL is still inaccessible, mark the PDF URL as Not Available (\"N/A\"). All results are compiled into a paper list and sent to the _future_ to get the result of a task execution. Figure 8 shows the flowchart of _Dynamic URL Redirection_ on _DOAJ_. It also utilizes _ThreadPoolExecutor_ to complete parallel tasks. Figure 7: Checking PDF accessibility using _Dynamic URL Redirection_ on _Semantic Scholar_. The function inside retrieves the _open access_ status of article metadata, whether it contains indicators, including the presence of _BOAI compliance_, _I4 OC citation openness_, and _OA start date_ information. The _DOAI_ API also provides a list of PDF URLs contained in article metadata. If the PDF URL ends with *.pdf, retrieve the URL and mark it as available. Subsequently, article metadata will be compiled into a paper list. Since article URLs in the _DOAI_ database are collected from various journals, adjustments are required using URL pattern replacements when PDF URL is inaccessible. For example, for the _ncbi.nlm.nih.gov_ journal URL path, this method replaces \"/?tool=EBI\" with \"/pdf\". On the other hand, the URL paths from several journals, such as _mdpi.com_, _open-research-europe.ec.europa.eu_, _gl.bmj.com_, and _fnch.bmj.com_, only require adding the suffix \"/pdf\" or \".pdf\" to access PDF files directly. If no URL pattern matches, the system will attempt a fallback check using OJS or DOI-specific pattern. If these also fail, the PDF URL is marked as \"N/A\". Figure 9 shows the flowchart diagram of _Dynamic URL Redirection_ on _PubMed_. As with fast checking of PDF file accessibility in _Semantic Scholar_ and _DOAI_, we also utilize _ThreadPoolExecutor_ to complete tasks in parallel on _PubMed_ metadata. _PubMed_ only provides the limited bibliographic metadata information, while _PubMed Central_ (PMC) provides the complete bibliographic metadata information and the _open-access_ full-text PDF. To access the PDF file, we need to retrieve the PMC ID as a value that can be assigned to the _PubMed Central_ URL path.