Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a rule-based NLP system for identifying COVID-19 cases from clinical narratives with application to 17 million documents.

Pros
- in scope for the conference
- real-life, large scale deployment
- apparent usefulness
- fairly straightforward system: the paper is accessible to a large audience

Cons
- fairly straightforward system: limited technical innovation
- ad hoc system: no/weak justification for the choice of approaches and technologies; not generalizable
- more technology than science

Overall
The significance of this work essentially comes from the practical utility of the system rather than from its originality or the innovation it brings. This work is reportable nonetheless.

Additional comments
Assuming this is the case, it would be worth emphasizing that 36.1% of confirmed positive cases in a VA surveillance system were ***solely*** identified using this capability. This is not entirely clear from the abstract or the conclusion.


review_title: fairly straightforward, but apparently useful and tested on a large scale

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes the creation of a system from scratch for processing clinical text at the VA to detect positive Covid-19 cases. While lab tests done at the VA will be in structured data, patients who may get some care elsewhere would have positive Covid results from other care recorded only in the free text, and the authors want to be able to capture those cases.

The challenges are that there was no existing data to build training sets for machine learning, there are many mentions of Covid and related terms in contexts that do not represent positive cases (negative test results, family members, describing the reason for a televisit).

The authors of this work created an NLP pipeline built on Spacy, including Context for negation/uncertainty/experiencer, sectionizer, etc. In the end, the decision of positive/negative was made based on finding a target term that was not uncertain, negated, or experienced by someone besides the patient, and that did have a positive modifier, occur in a diagnosis section, or was used to modify another condition ("Covid-19 pneumonia").

Since this is an IR-like problem it is difficult to evaluate recall but straightforward to evaluate precision. For recall what they do is evaluate at the patient level on the subset of patients for whom they do have lab results they can use as a gold standard.

While the NLP described here is fairly straightforward rule-based methods, this is an interesting use case of bootstrapping an NLP system from nothing for a new problem where there is no retrospective data available to build ML systems. The authors tease at the end that now that they've done this bootstrapping work they have essentially a labeled dataset that could be used for an ML system. A little more focus on this aspect would've been interesting to this reviewer, even if there are no ML results to report yet.
review_title: straightforward but interesting case study in building NLP system from scratch for covid-19

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a natural language processing pipeline for identifying positively diagnosed COVID19 patients, which has been deployed as part of the VA national response to COVID19. The objective of the NLP system is to simply detect if a clinical document contains a positive COVID19 case. 

The paper is very USA-specific. One of the motivations of the study is the fact that community-level testing and drive-through testing are often not counted in numbers derived from healthcare systems. Thus, the authors propose an approach that detects these cases from clinical texts within the Department of VA. The motivation could be made stronger by including numbers of people whose COVID19 status are encoded in free text only. What percentage of the patients may potentially be missed by laboratory-based surveillance methods?
It is unclear if the problem of not counting certain positive cases also applies to the VA. Do VA health networks not report laboratory test based COVID19?

Preprocessor: ‘remove problematic template text’ and ‘normalize lexical variants’ are very vague descriptions of what is actually being done. Normalizing lexical variants can be as simple as simply stemming and as complex as mappings multi-word variants to standard IDs (which of course is a very hard problem).
Target matcher: Is exact matching used? What is the rate of misspellings within the clinical notes (i.e., how many misspellings per number of tokens?)
Document Classifier: what are the positive modifiers? How many of them are there? How often can it capture actual expressions of the diagnoses. 
In Document Processing, how many keywords are actually used? How many times are these not spelled correctly?

What are the actual sections (incl. number of sections)?

The number of patients without laboratory evidence but with evidence only in clinical notes is quite large—and it justifies the development of such a pipeline. 

The causes for the false negatives are quite straightforward and can be easily addressed. It would be good to ascertain how precision/recall would change if these simple issues (e..g, new lines) were fixed during the preprocessing phase. 

It appears as though the authors did not consider inexact machine and/or semantic matching (e.g., via word2vec-type vectors). Recent research has shown that lexical similarity and semantic similarity based measures can generate data-centric variations/misspellings of noisy texts (e.g., from EHRs and social media), and such approaches have the potential to significantly improving performance—and they will not require machine learning (which would require large amounts of training data). 

Since the test results for a very large number of patients already available, it would be pretty straightforward to just use those in a supervised learning framework. 
review_title: A Natural Language Processing System for National COVID-19 Surveillance in the US Department of Veterans Affairs

Meta-review: Based on the reviewer feedback, it is clear that the submission reflects a valuable real-world deployment of NLP.

It is a worthwhile contribution to the workshop. Please prepare a 10-minute presentation for Thursday July 9!

Thanks so much for your submission.
Decision: Accept