Introduction
Coronavirus disease 2019 (COVID-19) is an infectious disease that has affected more than one million individuals all over the world and caused more than 55,000 deaths, as of April 3 in 2020. The science community has been working very actively to understand this new disease and make diagnosis and treatment guidelines based on the findings. One major stream of efforts are focused on discovering the correlation between radiological findings in the lung areas and COVID-19. There have been several works BIBREF0, BIBREF1 publishing such results. However, existing studies are mostly conducted separately by different hospitals and medical institutes. Due to geographic affinity, the populations served by different hospitals have different genetic, social, and ethnic characteristics. As a result, the radiological findings from COVID-19 patient cases in different populations are different. This population bias incurs inconsistent or even conflicting conclusions regarding the correlation between radiological findings and COVID-19. As a result, medical professionals cannot make informed decisions on how to use radiological findings to guide diagnosis and treatment of COVID-19.
We aim to address this issue. Our research goal is to develop natural language processing methods to collectively analyze the study results reported by many hospitals and medical institutes all over the world, reconcile these results, and make a holistic and unbiased conclusion regarding the correlation between radiological findings and COVID-19. Specifically, we take the CORD-19 dataset BIBREF2, which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. We develop sentence classification methods to identify all sentences narrating radiological findings from COVID-19. Then constituent parsing is utilized to identify all noun phrases from these sentences and these noun phrases contain abnormalities, lesions, diseases identified by radiology imaging such as X-ray and computed tomography (CT). We calculate the frequency of these noun phrases and select those with top frequencies for medical professionals to further investigate. Since these clinical entities are aggregated from a number of hospitals all over the world, the population bias is largely mitigated and the conclusions are more objective and universally informative. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19.
The major contributions of this paper include:
We develop natural language processing methods to perform unbiased study of the correlation between radiological findings and COVID-19.
We develop a bootstrapping approach to effectively train a sentence classifier with light-weight manual annotation effort. The sentence classifier is used to extract radiological findings from a vast amount of literature.
We conduct experiments to verify the effectiveness of our method. From the CORD-19 dataset, our method successfully discovers a set of clinical findings that are closely related with COVID-19.
The rest of the paper is organized as follows. In Section 2, we introduce the data. Section 3 presents the method. Section 4 gives experimental results. Section 5 concludes the paper.
Dataset
We used the COVID-19 Open Research Dataset (CORD-19) BIBREF2 for our study. In response to the COVID-19 pandemic, the White House and a coalition of research groups prepared the CORD-19 dataset. It contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. These articles are contributed by hospitals and medical institutes all over the world. Since the outbreak of COVID-19 is after November 2019, we select articles published after November 2019 to study, which include a total of 2081 articles and about 360000 sentences. Many articles report the radiological findings related to COVID-19. Table TABREF4 shows some examples.
Methods
Our goal is to develop natural language processing (NLP) methods to analyze a large collection of COVID-19 literature and discover unbiased and universally informative correlation between radiological findings and COVID-19. To achieve this goal, we need to address two technical challenges. First, in the large collection of COVID-19 literature, only a small part of sentences are about radiological findings. It is time-consuming to manually identify these sentences. Simple methods such as keyword-based retrieval will falsely retrieve sentences that are not about radiological findings and miss sentences that are about radiological findings. How can we develop NLP methods to precisely and comprehensively extract sentences containing radiological findings with minimum human annotation? Second, given the extracted sentences, they are still highly unstructured, which are difficult for medical professionals to digest and index. How can we further process these sentences into structured information that is more concise and easy to use?
To address the first challenge, we develop a sentence classifier to judge whether a sentence contains radiological findings. To minimize manual-labeling overhead, we propose easy ways of constructing positive and negative training examples, develop a bootstrapping approach to mine hard examples, and use hard examples to re-train the classifier for reducing false positives. To address the second challenge, we use constituent parsing to recognize noun phrases which contain critical medical information (e.g., lesions, abnormalities, diseases) and are easy to index and digest. We select noun phrases with top frequencies for medical professionals to further investigate.
Methods ::: Extracting Sentences Containing Radiological Findings
In this section, we develop a sentence-level classifier to determine whether a sentence contains radiological findings. To build such a classifier, we need to create positive and negative training sentences, without labor-intensive annotations. To obtain positive examples, we resort to the MedPix database, which contains radiology reports narrating radiological findings. MedPix is an open-access online database of medical images, teaching cases, and clinical topics. It contains more than 9000 topics, 59000 images from 12000 patient cases. We selected diagnostic reports for CT images and used sentences in the reports as positive samples. To obtain negative sentences, we randomly sample some sentences from the articles and quickly screen them to ensure that they are not about radiological findings. Since most sentences in the literature are not about radiological findings, a random sampling can almost ensure the select sentences are negative. A manual screening is conducted to further ensure this and the screening effort is not heavy.
Given these positive and negative training sentences, we use them to train a sentence classifier which predicts whether a sentence is about the radiological findings of COVID-19. We use the Bidirectional Encoder Representations from Transformers (BERT) BIBREF3 model for sentence classification. BERT is a neural language model that learns contextual representations of words and sentences. BERT pretrains deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. To apply the pretrained BERT to a downstream task such as sentence classification, one can add an additional layer on top of the BERT architecture and train this newly-added layer using the labeled data in the target task. In our case, similar to BIBREF4, we pretrain the BERT model on a vast amount of biomedical literature to obtain semantic representations of words. A linear layer is added to the output of BERT for predicting whether this sentence is positive (containing radiological finding) or negative. The architecture and hyperparameters of the BERT model used in our method are the same as those in BIBREF4. Figure FIGREF7 shows the architecture of the classification model.
When applying this trained sentence classifier to unseen sentences, we found that it yields a lot of false positives: many sentences irrelevant to radiological findings of COVID-19 are predicted as being relevant. To solve this problem, we iteratively perform hard example mining in a bootstrapping way and use these hard examples to retrain the classifier for reducing false positives. At iteration $t$, given the classifier $C_t$, we apply it to make predictions on unseen sentences. Each sentence is associated with a prediction score where a larger score indicates that this sentence is more likely to be positive. We rank these sentences in descending order of their prediction scores. Then for the top-K sentences with the largest prediction scores, we read them and label each of them as either being positive or negative. Then we add the labeled pairs to the training set and re-train the classifier and get $C_{t+1}$. This procedure is repeated again to identify new false positives and update the classifier using the new false positives.
Methods ::: Extracting Noun Phrases
The extracted sentences containing radiological findings of COVID-19 are highly unstructured, which are still difficult to digest for medical professionals. To solve this problem, from these unstructured sentences, we extract structured information that is both clinically important and easy to use. We notice that important information, such as lesions, abnormalities, diseases, is mostly contained in noun phrases. Therefore, we use NLP to extract noun phrases and perform further analysis therefrom. First, we perform part-of-speech (POS) tagging to label each word in a sentence as being a noun, verb, adjective, etc. Then on top of these words and their POS tags, we perform constituent parsing to obtain the syntax tree of the sentence. An example is shown in Figure FIGREF9. From bottom to top of the tree, fine-grained linguistic units such as words are composed into coarse-grained units such as phrases, including noun phrases. We obtain the noun phrases by reading the node labels in the tree.
Given the extracted noun phrases, we remove stop words in them and perform lemmatization to eliminate non-essential linguistic variations. We count the frequency of each noun phrase and rank them in descending frequency. Then we select the noun phrases with top frequencies and present them to medical professionals for further investigation.
Experiment ::: Experimental Settings
For building the initial sentence classifier (before hard-example mining), we collected 2350 positive samples from MedPix and 3000 negative samples from CORD-19. We used 90% sentences for training and the rest 10% sentences for validation. The weights in the sentence classifier are optimized using the Adam algorithm with a learning rate of $2\times 10^{-5}$ and a mini-batch size of 4. In bootstrapping for hard example mining, we added 400 false positives in each iteration for classifier-retraining and we performed 4 iterations of bootstrapping.
Experiment ::: Results of Sentence Classification
Under the final classifier, 998 sentences are predicted as being positive. Among them, 717 are true positives (according to manual check). The classifier achieves a precision of 71.8%. For the initial classifier (before adding mined hard examples using bootstrapping), among the top 100 sentences with the largest prediction scores, 53 are false positives. The initial classifier only achieves a precision of 47%. The precision achieved by classifiers trained after round 1-3 in bootstrapping is 55%, 57%, and 69% respectively, as shown in Table TABREF12. This demonstrates the effectiveness of hard example mining. Table TABREF13 shows some example sentences that are true positives, true negatives, and false positives, under the predictions made by the final classifier.
Experiment ::: Results of Noun Phrase Extraction
Table TABREF15 shows the extracted noun phrases with top frequencies that are relevant to radiology. Medical professionals can look at this table and select noun phrases indicating radiological findings for further investigation, such as consolidation, pleural effusion, ground glass opacity, thickening, etc. We mark such noun phrases with bold font in the table. To further investigate how a noun phrase is relevant to COVID-19, medical professionals can review the sentences mentioning this noun phrase. Table TABREF16,TABREF17,TABREF18 show some examples.
For example, reading the five example sentences containing consolidation, one can judge that consolidation is a typical manifestation of COVID-19. This is in accordance with the conclusion in BIBREF5: “Consolidation becomes the dominant CT findings as the disease progresses." Similarly, the example sentences of pleural effusion, ground glass opacity, thickening, fibrosis, bronchiectasis, lymphadenopathy show that these abnormalities are closely related with COVID-19. This is consistent with the results reported in the literature:
Pleural effusion: “In terms of pleural changes, CT showed that six (9.7%) had pleural effusion." BIBREF6
Ground glass opacity: “The predominant pattern of abnormalities after symptom onset was ground-glass opacity (35/78 [45%] to 49/79 [62%] in different periods." BIBREF7
Thickening: “Furthermore, ground-glass opacity was subcategorized into: (1) pure ground-glass opacity; (2) ground-glass opacity with smooth interlobular septal thickening." BIBREF7
Fibrosis: “In five patients, follow-up CT showed improvement with the appearance of fibrosis and resolution of GGOs.", BIBREF8
Bronchiectasis and lymphadenopathy: “The most common patterns seen on chest CT were ground-glass opacity, in addition to ill-defined margins, smooth or irregular interlobular septal thickening, air bronchogram , crazy-paving pattern, and thickening of the adjacent pleura. Less common CT findings were nodules, cystic changes, bronchiolectasis, pleural effusion , and lymphadenopathy." BIBREF9
Conclusions
In this paper, we develop natural language processing methods to automatically extract unbiased radiological findings of COVID-19. We develop a BERT-based classifier to select sentences that contain COVID-related radiological findings and use bootstrapping to mine hard examples for reducing false positives. Constituent parsing is used to extract noun phrases from the positive sentences and those with top frequencies are selected for medical professionals to further investigate. From the CORD-19 dataset, our method successfully discovers radiological findings that are closely related with COVID-19.

Question: What is the CORD-19 dataset?
Answer: which contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses

contains over 45,000 scholarly articles, including over 33,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses

Question: How large is the collection of COVID-19 literature?
Answer: 45,000 scholarly articles, including over 33,000 with full text