# Web Scraping Scientific Repositories for Augmented Relevant Literature Search Using CRISP-DM
## 4 Research Methods
### Web Scraping (Data Understanding & Preparation)  
This section focuses on simplifying the stage of Data Preparation for an automated academic literature content extraction [14; 15; 16]. Earlier literature had explained how readily available web content could be re-cycled as part of various research processes at different stages as well as the top corresponding challenges. However, The process of automated content extraction (widely known as \"Web Scraping\", a.k.a \"Web Crawling\") takes place over three broad steps: (1). Site Access, (2). HTML parsing and contents extraction and (3). Output building [17].  
The current state-of-the-art web scraping solutions and technologies offer a wide variety of simple easy to use tools. These solutions came a very long way from a past that was crowded by complexities impeding content extraction; especially for average skilled users. Many of these methods relied on API and/or code-based methods to extract content from designated web sites. The traditional complexity problems in dealing with these APIs and code-based structures could be explained in the convoluted technicalities behind the three steps within the process. For a given user to scrape or crawl content he/she would usually have to hard code many parameters starting with the web-site URL to be crawled (\"Site Access\"). The problem here lies in the lack of dynamic assignments of URL parameters given the fact that web-site URLs are usually hard coded. This very problem blows out of proportion, especially in the following stage of \"HTML parsing and content extraction\". HTML parsing and content extraction usually takes place by analyzing the anatomy of web-sites and the underlying content placeholders. A traditional example is where a user would have to understand the xPath structure of the web-site (please see Appendix A.2), as well as know its name in order to include/exclude it for content extraction [18]. Nowadays many of the browsers offer built-in development tools that are designed to help web developers to browse through the structure of a given web page. The value of the development tools is definitely very high for web developers. However, when it comes back to the case of extracting web content from scientific repositories the process remains complex due to the hierarchical xPath structures depicted in Figure 2a,b.  
Figure 2: Google Chrome’s Inspect Element panel dissecting a webpage anatomy: (**a**) parent path highlighting a particular literature search hit; (**b**) highlighting a sub-component of the parent search hit.  
Over time, many of the web scraping tools identified in literature [15] had been simplified in order not to only provide free utilities, but also to code free solutions. In our case an extension to the google chrome browser (known as Web Scraper- please see Appendix A.3) had been used to overcome the complexities with HTML parsing and content extraction. The tool provides a set of utilities that enable given users to point-and-extract content of interest by a providing pagination facility to scrape multiple pages as depicted in Figure 3a, visualize the scraping logic through what is known as a selector graph, etc. as depicted in Figure 3b. The output building in this case takes place by exporting the extracted content into csv files which would later be rechanneled into the data preprocessing and modeling.