Measuring observable influence and impact of scientific research beyond academia 📄
By: Chhavi Verma
Tags: Data Science, NLP, Web Scraping, Healthcare, Big Data

METHODOLOGY

EDA:

We conducted EDA using Numpy, Pandas and Excel on Genome BC’s dataset which lists the academic papers funded by them. We explored the downstream documentations using web browser, collecting their citation methods such as by presenting PMID, PMCID, DOI or giving pdf files directly.

Data Collection:

Collection of data is the most challenging part of our project. The downstream documents are variable. By EDA, we find that we can use PMID to represent the papers they cited as most of them are medicine field papers. (If we want to extend our work to the other fields, we can use DOI instead.)

The data is collected from 5 downstream documents entries:

bcm

It has a zip file that contains all the information that the website hosts. So we get all the pdf files by downloading the zip file and unzipping it manually.

bcc, cadth and cpic

We use Scrapy to scrape all the pdf files and extract all the DOI, PMCID and PMID references.

pgkb

pgkb is a javascript-generated dynamic website, so we used Scrapy combined with Selenium to scrape the pdf files and extract all the DOI, PMCID and PMID references.

Web scraping using Scrapy combined with Selenium

Extracting references directly from these documents does not solve the problem as we have to dig in deep by going to each link or reference taken from the previous reference. This task is done up-to a threshold due to the limited computing power and internet availability.

Data Cleaning and Integration:

After collecting the data, for DOI, PMID, and PMCID references, we clean it using python libraries -Numpy and Pandas combined with NCBI APIs which fulfill the null values and get rid of the duplicates.

For the pdf files, at first sight, extracting references from pdf files seems hard. But in fact, as Chenet mentioned [2], reference information is, in fact, structured information. Using regular expression alone can achieve a satisfying result [5].

Still, this problem is not easy, as each website contains many different structures for PDFs and parsing them requires different approaches. We firstly parse pdf files using PDF2XML to get structured data. The lxml library is used to get to reference using XPATH. The implementation of this part becomes more tedious as the type of pdf files increasing.

Then we extract reference paper titles using regular expression combined with NCBI APIs. We set the regular expression rule like this: every title is between two periods, and the number of words in the title should greater than 4. Then we verify this title by querying NCBI API. If we can find a paper the title of whom has an edit distance less than 4 to the title, then we can be sure that the title is indeed a valid paper title. In the meantime, we get the corresponding PMID of this paper.

We use edit distance to determine whether the two papers are the same one based on this fact: each paper’s title is unique. The similarity method, such as Jaccard Similarity, will fail because papers focused on one specific topic will always have similar titles.

Below is our pdf parsing result, we extract 1246 reference paper titles from 4379 pdf files:

Analysis:

We measure the impact of a Genome BC funded paper in real life by counting how many times they are cited in the downstream documentations both directly and indirectly.

We get all the direct and indirect cited papers for every downstream documentation and represent it as a tree.

Accelerate the generation of the reference graph

At first glance, it seems a trivial task — just perform BFS to get the reference tree by query NCBI database. But as 1000 downstream entities will refer to around 40 million papers given a depth constraint of 6, plus that the NCBI API only allows querying 10 times every second, this becomes an impossible mission. We come up with an accelerated solution: as we only concern about the depth but not the actual structure, we can merge nodes at the same depth.

As NCBI API allows a 100 id list query every time, we merge the nodes into a package of 100 nodes and do the query. This method decreases the number of nodes from 40 million to around 400 thousand and reduces the running time to 25 hours given a depth constraint of 6.

After the query, we merge all the nodes of the same depth into one node, which further reduce the running time for later analysis.

Finally, we will go and measure the real-life impact of Genome BC funded papers. It is somehow straightforward: go through the reference map using Breadth-First-Search (BFS) to do the counting for our Genome BC funded papers.

Finally, we find the actual impact of research papers in the real world.

Visualization:

In order to show the relation between downstream documentations and the Genome BC funded papers, we use igraph to construct a map with coordinates associated with every paper and downstream documentation entity. Then, we plot the map using plotly. Plotting this helps both the researchers and the company to move into the right direction. It provides motivation to researchers and at the same time increases revenue for the company. These figures give a better picture to the company and are understandable to a greater audience.