NLP Based Information Retrieval System | by Nikhil Sharma | Towards Data Science

In this post, we will understand how to build a recommendation system with text data. Here text data is comprised of scholarly articles about COVID-19, SARS-COV-2, and related coronaviruses. The dataset is known as CORD-19, prepared by the White House and a coalition of leading research groups. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.
For this post, we will need Python 3.6, spacy, Tensorflow, NLTK, Dask, and we will install them step by step whenever required.
The dataset used here is from Kaggle competition. The dataset contains doc_id, source, title, abstract, and text body. doc_id is document id, the source is the place from where the paper is extracted like biorxiv, abstract is a summary of the paper, and text body is body or content of the research paper.
Our task is to retrieve information based on the following questions.
What is known about transmission, incubation, and environmental stability?
What do we know about natural history, transmission, and diagnostics for the virus? What have we learned about infection prevention and control?
Specifically, we want to know what the literature reports about:
Range of incubation periods for the disease in humans (and how this varies across age and health status) and how long individuals are contagious, even after recovery.
Prevalence of asymptomatic shedding and transmission (e.g., particularly children).
Seasonality of transmission.
Physical science of the coronavirus (e.g., charge distribution, adhesion to hydrophilic/phobic surfaces, environmental survival to inform decontamination efforts for affected areas and provide information about viral shedding).
Persistence and stability on a multitude of substrates and sources (e.g., nasal discharge, sputum, urine, fecal matter, blood).
Persistence of virus on surfaces of different materials (e,g., copper, stainless steel, plastic).
Natural history of the virus and shedding of it from an infected person
Implementation of diagnostics and products to improve clinical processes
Disease models, including animal models for infection, disease and transmission
Tools and studies to monitor phenotypic change and potential adaptation of the virus
Immune response and immunity
Effectiveness of movement control strategies to prevent secondary transmission in health care and community settings
Effectiveness of personal protective equipment (PPE) and its usefulness to reduce risk of transmission in health care and community settings
Role of the environment in transmission
Exploratory Data Analysis (EDA) is the process by which the data analyst becomes acquainted with their data in order to drive intuition. This process typically makes use of descriptive statistics and visualizations. Visually representing the content of a text document is one of the most important tasks in the field of text mining as a Data Scientist or NLP specialist. However, there are some gaps between visualizing unstructured (text) data and structured data.
It is a good practice to understand the data first and try to gather as many insights from it. EDA is all about making sense of data in hand, before getting them dirty with it.
Let’s install some required packages and import them. we will be using the scispacy for analysis. scispaCy is a Python package containing spaCy models for processing biomedical, scientific or clinical text.
import them as usual!!
When you call nlp on a text, spaCy first tokenizes the text to produce a Doc object. The Doc is then processed in several different steps – this is also referred to as the processing pipeline. The pipeline used by the default models consists of a tagger, a parser and an entity recognizer. Each pipeline component returns the processed Doc, which is then passed on to the next component.
tokenizer: Segment text into tokens.
tagger: Assign part-of-speech tags.
parser: Assign dependency labels.
ner: Detect and label named entities.
The processing pipeline always depends on the statistical model and its capabilities. For example, a pipeline can only include an entity recognizer component if the model includes data to make predictions of entity labels.
NOTE: you can create own pipeline components for further processing
now, create a function for cleaning data, this will remove all the stop words, punctuations, extra spaces, URLs, email, and currency. sometimes verbs, adverbs, pronouns are not required and can be removed. this function will return the base form of word which is known as lemmatization.
Text contains lots of abbreviations and full form, this code is used to extract all abbreviations and respective full forms within the text and put them in a dictionary where keys are abbreviations and values are full forms.
Now, we need to create a function to replace all abbreviations with their respective full forms.
similarily, we will create functions for removing email ids, weblink, paper references, extra spaces, unwanted characters, etc.
here, we’re going to use Dask for processing, Dask is a flexible library for parallel computing in python, provides multi-core execution on larger-than-memory datasets.
“Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love” — https://dask.pydata.org/en/latest/
now, call above function to preprocess text
The model is trained and optimized for greater-than-word length text, such as sentences, phrases or short paragraphs. It is trained on a variety of data sources and a variety of tasks with the aim of dynamically accommodating a wide variety of natural language understanding tasks. The input is variable length English text and the output is a 128-dimensional vector. The universal-sentence-encoder model has trained with a deep averaging network (DAN) encoder.
let's install required packages for USE
load model from its URL
now, split text body of research papers by new line into list separate paragraphs, so that we can create embeddings for each paragraph. for example, if a text has 100 paragraphs, then we will get a list of these 100 elements.
Convert the different list of paragraphs for each row of text into a single list
now create embeddings
this will return a tensor object of shape nx128, where n is the number of paragraphs or sentences, as number of sentences are 1052935, therefore the shape of tensor will be 1052935x128
TensorShape([1052935, 128])
Cosine similarity is a measure of similarity that can be used to compare documents or, say, give a ranking of documents with respect to a given vector of query words.
Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space. In this context, the two vectors I am talking about are arrays containing the embeddings of two documents.
Let x and y be two vectors for comparison. Using the cosine measure as a similarity function, we have
where ||x|| is the Euclidean norm of vector x=(x1,x2,...,xn), defined as x1^2+x2^2+⋯+xn^2. Conceptually, it is the length of the vector. Similarly, ||y|| is the Euclidean norm of vector y. The measure computes the cosine of the angle between vectors x and y. A cosine value of 0 means that the two vectors are at 90 degrees to each other (orthogonal) and have no match. The closer the cosine value to 1, the smaller the angle, and the greater the match between vectors.
The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance because of the size, they could still have a smaller angle between them. Smaller the angle, higher the similarity.
Please refer to a link to check more about cosine similarity.
This function extracts information from corpus, based upon query. it calculates cosine similarity of query with all paragraphs or sentences, standardize it to fix its range b/w 0 to 1, and return sentences based upon threshold.
4.8.1 Incubation period
output:
['statements of incubation period and sources of original incubation period data for', ' the incubation period of yellow fever is around 3-6 days. the incubation period of dengue fever is around 8-10 days.', ' the incubation period of lassa fever ranges from 3 to 21 days. symptoms of new world haemorrhagic fever typically develop over 7-14 days.', ]
Output:
 ' determination of the dynamics of disease transmission in geographically widespread areas global spread of foot-and-mouth disease; spread of newcastle disease virus in asia distinction between pathovars and nonpathovars pathogenic and nonpathogenic escherichia coli in petting zoos addressing hospital and institutional infectious disease problems',' respiratory viruses detected in outbreaks in institutions reflect those known to be major causes of acute respiratory disease in the community; prevalence varies based on geographic location, season, and detection methods . free access to such institutions by members of the community , in conjunction with communal close quarters of residents, creates an ideal environment for propagation of viral respiratory outbreaks .', ' incidence of infection within a community social economic status of the community season per capita water use time of day age distribution within the community chronic infections table 2 concentration of enteric viruses in feces.']
4.8.3 Role of environment in Transmission
Output:
[' disease control efforts can also make it challenging to determine effects of climate change on disease. for instance, if climate change causes range shifts of parasites from tropical to temperate countries, this might result in an overall reduction, rather than increase, in human diseases because temperate countries often have superior health infrastructures . regardless of the outcome of climate change on diseases, integrating control efforts into projections should improve predictions of future disease risk for humans and wildlife. furthermore, given that control measures could obscure increases in transmission, incorporating control measures into models could reveal underlying increases in disease risk that might otherwise be missed.', ' environmental variables can affect responses of mice in experimental situations. changes in respiratory epithelial physiology and function from elevated levels of ammonia, effects of temperature and humidity on metabolism, effects of light on eye lesions and retinal function, and effects of noise on neurophysiology are examples of complications that can vary with the form of insult and the strain of mouse employed. ', ' effects of meteorological factors on transmission. climate change can influence the occurrence and spread of infectious diseases through its effect on transmission. for example, climate change can cause air pollution and shortages of food and clean water , leading to changes of high-incidence areas and outbreak patterns, thereby affecting the occurrence and spread of infectious disease.', ' future studies that focused on other components of air pollution and environmental factors are necessary to more fully understand the possible impact on infectious disease transmission . studies are in need to establish the relationships between particle attachment and survival of influenza virus, and also methods should be developed to quantify airborne influenza virus and measure the concentration of influenza virus in ambient air especially for extremely low virus concentrations . we set the number of cases to zero on hot days due to there being no significant effect of pm 2.5 on influenza incidence on hot days .']
Output:
[' seasonal patterns were observed for hmpv, rhinovirus, enterovirus and influenza viruses with peak prevalence in late winter and spring . in addition, a seasonal pattern was detected for tuberculosis . no distinct seasonal patterns were observed for other microbes.', 'not disperse low pathogenic avian influenza virus during autumn migration .', 'the severity of seasonal influenza epidemics is unpredictable and influenced by the predominant circulating virus strains and level of immunity in the population . during peak community influenza activity, hospitals and emergency rooms may be overwhelmed by patients presenting with influenza-like illness and more severe disease , . illness attack rates may be higher among most age groups during pandemics than observed for seasonal influenza due to limited immunity among exposed populations . the re-emergence of highly pathogenic avian influenza a virus among poultry with sporadic transmission to exposed persons and the resulting high mortality has stimulated global influenza pandemic preparedness .',' the two gastrointestinal viruses in the study both also presented with winter seasonality, often with norovirus preceding rotavirus epidemics . norovirus and influenza a virus epidemics usually overlapped, but the norovirus epidemics had broader peaks, and numbers tended to start increasing earlier than influenza a virus diagnoses levels and decreased after the disappearance of influenza a virus. also, rotavirus outbreaks usually occurred before influenza a outbreaks started.','influenza viruses circulate in temperate climates from early autumn to late spring, and in tropical climates throughout the year with unpredictable peaks, usually occurring during the wet season and cooler months. routine surveillance for influenza and influenzalike illness can provide estimates of the relative severity of influenza seasons, and can also provide clinical specimens from which influenza viruses can be detected. this allows comparison of circulating influenza strains with vaccine strains in the current influenza season, and may provide potential vaccine strains for future seasons.]
(e.g., charge distribution, adhesion to hydrophilic/phobic surfaces, environmental survival to inform decontamination efforts for affected areas and provide information about viral shedding).
output:
[' antiadhesive surfaces target microbial adhesion usually by the interaction of antagonist physicochemical properties. easy-clean surfaces that are hydrophobic repel bacteria better than glass-coated controls , while hydrophilic surfaces favour water sheeting and subsequent cleaning. similarly, polyethylene glycol coated surfaces promote a hydrophilic interaction against bacteria, preventing attachment . the use of diamond-like carbon films has already been tried for medical implanted devices such as joint prostheses and stents in order to repel microbial adhesion . despite being non-toxic and appealing, the lack of biocidal properties may turn discouraging a more generalized implementation of such easy-clean technologies.', ' in vitro testing compared the adhesion of representative proteins, prokaryotes, and eukaryotes on nanostructured and flat si3n4 surfaces with lysine-coated glass slides as positive controls. flat si3n4 is moderately hydrophilic and known to vigorously promote cell adhesion and proliferation due to increased adsorption of proteins when compared to more hydrophilic surfaces 33,34. hence, we further increased the hydrophilicity of the si3n4 surface by varying the aspect-ratios of the nanostructures from 0.15 to 0.90 and systematically controlled surface hydrophilicity . once strong hydrophilicity is achieved , a nanostructure-mediated aqueous barrier forms on the surface and limits protein adsorption and cell adhesion to provide an anti-adhesion property 6,35,36.', ' for htcc nanofibers to become an adsorptive membrane material, the fibers must not dissolve in water. crosslinking of polymer structures is a common method to impart membrane stability in water soluble polymers. 21 pva fibers were crosslinked with maleic anhydride to create a filtration membrane. 22 htcc blended with polyvinyl alcohol was photo-crosslinked to impart antimicrobial resistance to the fibers. 10 glutaraldehyde vapor was used to crosslink htcc-pva fibers to create an antimicrobial surface. 7 here, we demonstrate that htcc nanofibers can be crosslinked while retaining their nonwoven structure and their ability to bind to negatively-charged viruses. we have explored the crosslinking conditions that give the greatest water stability. these conditions allowed for high virus removal. our model viruses include the non-enveloped porcine parvovirus , one of the smallest known mammalian viruses and the enveloped virus, sindbis virus. this nanofiber material has the potential to become an inexpensive defense against water-borne diseases.',
output:
[" therefore, china's advancements in research and development, technical and scientific capacity transferring can support african next generation of proactive scientists to develop more sensitive simplified diagnostic tools and reduce the costs of laboratory diagnosis and medical equipment. furthermore, research and development is needed in examining the biological mechanisms of stressors or risk factors exposure and health effects, assessing evidence-based mitigation or adaptation interventions and benefits . innovative solutions and breakthroughs in human-animal-environment fields would not only enable africa to meet its own growing needs, but also support integrating health systems, including strengthening the capacities of laboratory diagnostics and medical care, as well as establishing the china-africa platforms that could generate evidence-based low-cost, available and easy-to-use health packages and solutions for the reduction of public health burden.", ' 1. access to data and samples to enable and accelerate research, which would involve rapid sharing of epidemiological surveillance and clinical data to inform outbreak control strategies; incentives and platforms for open sharing and access to genomic sequencing data; access to specimen samples . 2. appropriate conduct of research, including improved ethical standards for research and development ; previous agreement about experimental protocols, such as trial design, to speed clinical trials when outbreaks strike; 80 access to clinical trial data, such as publication of negative and positive results; clear pathways for approval by stringent regulatory authorities and in countries of use; and building on and investing in research capacities in epidemicaff ected countries. 3. equitable access to the benefi ts of research, including priority, aff ordable access to newly developed health technologies for aff ected populations, including health workers; and ethical guidelines for rationing products with limited availability. an overarching framework is needed to bring coherence and fi ll gaps in the fragmented system of international rules shaping outbreak-related research (including the international health regulations, pandemic',' clinical laboratories are highly regulated, and general laboratory and testing requirements apply to all molecular diagnostic assays reported for patient care90. quality control is paramount, and methods must be developed to ensure analytic accuracy throughout the assay workflow. important quality control steps can include initial sample quality checks, library parameters , sequence data generation , recovery of internal controls and performance of external controls. validation data generated during assay development and implementation should be recorded and made available to laboratory inspectors or submitted to regulatory agencies, such as the fda in the usa or the european medicines agency in europe, for approval.', ' future applications of advanced molecular diagnostics in clinical laboratories will enhance significantly capabilities to diagnose bacterial, parasitic, and viral agents in the early course of disease through enhanced assay sensitivities and specificities and improved turnaround times, theoretically leading to more timely and directed therapeutic intervention. until such time, clinicians must continue to rely on clinical judgment and the diverse battery of traditional culture techniques, direct examination , and immunoassays that are available. cost considerations and the ever-increasing array of infectious agents responsible for infectious gastroenteritis will continue to drive the development of practice guidelines to assist practitioners with reasoned and reasonable approaches to management of diarrheal illnesses. ']
output:
[' this paper reviews animal models currently in use for the study of influenza pathogenesis and immune responses to influenza virus infections and to assess vaccine and antiviral drug efficacy. models that will be reviewed will include: models of benign influenza, typical severe seasonal influenza and pneumonia, influenza-associated sepsis, models using h5n1 viruses, models in which acute respiratory distress syndrome is induced, models of influenzaassociated neurologic disease, models for virus transmission studies, immunocompromised models, and models of host resistance.', ' pregnancy poses an increased risk of severe maternal illness due to a number of acute and chronic viral infections; the common rhinovirus, influenza virus, sars coronavirus and varicella zoster, hepatitis e/b, hiv and cytomegalovirus . viral infections can also lead to pre-term birth and other pregnancy complications . however, few murine models are available to study pathogenesis of viral infections.', ' coronaviruses can cause a variety of diseases in animals and humans . of economical importance are coronaviruses such as porcine transmissible gastroenteritis virus, bovine coronavirus, feline infectious peritonitis virus, and avian infectious bronchitis virus. mhv, a natural mouse pathogen, has been extensively studied in the context of host immune responses and pathogenesis . there are many different and well-characterized strains of mhv which, depending on organ tropism, virulence and host strain, can cause a wide array of diseases ranging from hepatitis, respiratory symptoms and gastroenteritis to cns infection, demyelination, and acute meningitis [2, 17, . therefore, by using appropriate combinations of virus and mouse strains, mhv infections provide suitable models for a number of diseases that are of medical importance, such as encephalitis, immune-mediated demyelination , hepatitis and acute respiratory infections .', ' although in most host-virus interaction studies typically protein-coding genes have been the center of attention, there are few examples of virus and host lncrna interactions in human and mouse models . for instance, hepatitis b virus infection altered lncrna profiles in patients, with about 4% of human lncrnas showing more than 2-fold changes in hbv infected liver tissue . winterling et al. identified a virus inducible lncrna, which is induced by vesicular stomatitis virus and several strains of influenza a virus .']
similarily, we can retrieve information for other queries, you just need to type query and call “recommend_text” function.
Refrences:
[1] Jiawei Han, Jian Pei, Micheline Kamber, Data Mining: Concepts and Techniques provides the concepts and techniques (2012), Elsevier Inc
[2] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, Ray Kurzweil, Universal Sentence Encoder (2018), arXiv.org