Using Natural Language Processing (NLP) to 

Lines and Devices in Portable Chest X‐Ray 

Reports

VA H lth ea Services Research & Development C by er Semin

March 29, 2012

Presenters:Mary K. G ld o stein, MD (Pri i l nc pa Investigator)Director, Geriatrics Research Education and Clinical Center (GRECC) at 

Health Care System

Professor of Medicine (Center for Primary Care and Outcomes Researc

University

Dan Y. Wang, PhDResearch Health Science Specialist, VA Palo Alto Health Care System

VP of Technology, Medcisive LLC

Tammy S. Hwang, BAResearch Health Science Specialist, VA Palo Alto Health Care System
Acknowledgments

This study was undertaken as part of the VA Health Services ReDevelopment (VA HSR&D) Consortium for H lth ea care I f n ormat

(CHIR) Translational Use Case Project (TUCP) grant HIR 09‐007 Goldstein), a component of CHIR HIR 08‐374 (PI: Samore).

We also made use of report extraction facilities and a secure s

workspace provided by the VA HSR&D Informatics and Compu

Infrastructure (VINCI).

VA Consortium of Healthcare VA Informatics and Co

Informatics Research Infrastructure

Views expressed are those of the presenters and not necessar

the Department of Veterans Affairs
I ti nves gator Team

Dan Y. Wang, PhD, VA Palo Alto Health Care SystemDaniel L. Rubin, MD, MS, Stanford University

Tammy S. Hwang, BA, VA Palo Alto Health Care System

Dallas A. Chambers, BS, VA Palo Alto Health Care Syst

Justin G. Chambers, BS, VA Palo Alto Health Care Syst

Mdi e c sive LLC

Brett R. South, MS, VA Salt Lake City Health Care SystMary K. Goldstein, Goldstein, MD, MS, VA Palo Alto Health Care 

Stanford University

Additional subject matter expertise from Matthew Sa

MD, PhD of CHIR project and other experts

Statistical Statistical consultants consultants: Shuying Shuying Shen, PhD, and Andre

PhD 
Poll

Please help us understand our audience.S l e ect all that apply: (A) I am primarily interested in the potential applicatio (B) I am primarily primarily interested interested in the underlying underlying technol

technol

technical background of this tool

 (C) I do clinical work as a licensed health professional 

 (D) A substantial part of my work includes informatics (E) Quality assessment/measurement is a substantial p

work

 (F) Research is a substantial part of my work
Goals of the Session Session

At the end of this seminar, the participants 

be able to...

• Explain steps involved in conducting a p 

r

extracting information from free‐text of 

Electronic Health Records

• Describe how to use an annotation tool

(Knowtator) (Knowtator) to create a reference reference standa

standa

• Understand a natural language processin

technique technique that works for information information ext

ext

from chest x‐ray (CXR) reports
Outline

• Background• Methods• Results• Comments• Audience Audience questions questions and discussion discussion
Backgro

und
Structured Structured and Unstructured Unstructured 

• Electronic h l ea th records have extensi

information important for patient care

assessment, quality improvement, an

epidemiologic surveillance

– structured data elements– unstructured data (free text)

• Consortium for Healthcare Informatic

Research (CHIR)

– Translational Use Case Projects
Portable Portable CXR in ICU

• Patients in Intensive Care Units (ICU) often have me

devices inserted

• Such devices can lead to complications such as 

borne infections, increased morbidity, and incre

• Line/device related infections have been correla

length of time of presence and line/device type 

1998; Tokars, 2007)

• Hospitals are often required to report a daily 

patients with specific lines/devices in place

• Inserted devices are radio‐opaque

• usually visible in Portable chest X‐ray (CXR) ima

• usually documented in free‐text CXR reports
• 

Our Aims in Design

• Use Natural Language Processing (NLP)

• avoid manual chart review

• extract detailed information about devices

potentially potentially enable infection infection surveillance surveillance and

epidemiological research

• Focus on the part of the radiology report that desc

the radiologist radiologist sees on the x‐ray (not clinical clinical histor

histor

by the ordering provider)

• Initially focus on accuracy of information extracted 

within a single report as compared to human read

report

• Reserve as future work the comparison of reports for sam

through time
Methods
• 

Overview Overview of Methods Methods

• Specify the report‐types of relevance

We decided decided to focus on portable portable chest x‐ray (CXR) reports reports 

ICU at time the study was done

• In future work expanding to all CXR reports for 

during the same admission admission

• Specify what information should be identified in each 

• Identify a source of documents

• We work de in VA VINCI secure computing environment• Document selection

• Develop a reference standard of annotated reports tha

shared with NLP ith NLP developers

• Develop NLP code to process the text

• Separate set of documents made available to NLP develope

system

• Compare the output from the NLP with the reference 
Staged Project

• Early stage project

– Evaluated with 90 reports (previously rep

• More recent broader evaluation and

improvements

– Evaluated with 500 reports

• Ongoing Ongoing and future work

– Move beyond one CXR report at a time t

reports reports for same patient patient through through time d

acute hospitalization
Identifying Records

• Identify EHR free‐text records to extract

– Use structured data to identify which reports are relevant– Procedure names and CPT codes

• The CPT code for “Chest X‐ray” had 1,749 distinct pro

names

• Many reports had Null procedure code, with procedu

that appeared appeared to be CXR

• We developed list of procedure names for SQL code 

– At present no way to know if we captured every single po

report

» Not essential for purpose of this project

» A major issue if intent were to apply to every releva

– Available Available to other VA staff who wish to use it and improv

improv

» Perhaps libraries of such lists could be made availab

resource
Specifying Specifying Terms to Extra

Extra

• How we identified device/line terms to extract

• Resolved questions about categories of items to include 

with subject subject matter experts experts

• For purpose of this study, included lines and 

the chest, but not hardware in the spine and 

valves

• Device terms obtained from UMLS (Bodenreide

and RadLex (Rubin, J Digital Imaging 2008)

• Note that standardized terminologies do no

the actual language used in many reports

• Reviewed actual CXR reports to identify additio
Developing Developing the Reference Reference Sta

Sta

• Reference Standard: A set of carefully annotated reports against which 

the accuracy of the NLP algorithm

• Annotation training• The annotators refer to a written annotation guideline

– Includes a lines/device appendix with list of terms– Developed in a process of annotating to understand what issues are c

annotators

• The annotation guideline is shared with the NLP Developer

– So everyone is using the same rules

• The Reference Standard set of documents is blinded to the NLP de

– So that the evaluation evaluation will test the NLP on documents documents not used to tra

• Span of text

– every character in the documents is numbered, starting with 1 as the 

the report. the “span” of text is the character character positions positions of a particula

particula

text.

– Example

• report text: “the tube is present present.”

• the word “tube” has span from 5 to 8
‐ ‐

Our Reference Standard

• 500 randomly selected portable CXR exams

obtained in the ICU between 1/ /1 2006 to

1/1/2009 from 46 randomly selected,

geographically geographically dispersed dispersed, Veteran Veteran Affairs Affairs (VA)

medical centers cross the country

excluded sites with small number of ICU

patients

• Reports stratified by medical center and length

of Stay (LOS). LOS groups included short LOS

(1‐3 days), medium LOS (4‐7 days), and long

LOS (8‐10 days) in a 40:40:20 ratio
Evaluation Evaluation of NLP accura

accura

• Compare NLP output with reference standard for each l

and insertion insertion status term

• TP (True Positive): found in both reference standard an

output

• FN(False Negative): found only in reference standard

• FP (False Positive): Positive): found only in NLP output• Compute standard overall measures for information ret

• Precision = TP/(TP+FP)

• Recall = TP/(TP + FN)

• Sebastian
Poll

Select all that apply: (A) I have experience experience using text anno

anno

tools

 ( ) B I have some familiarity wi ht the co

of text annotation

 (C) I am not familiar with text annota

other than what I have heard in this ta
Annotation of Reference Standard:

Team of annotators used Knowtator Tool,

(Ogren, 2006) a plug‐in of open‐source

knowledge acquisition tool Proté ég(protégé.stanford.edu)• Knowtator Knowtator uses an annotation annotation schema•Annotations done by point‐clicking or drag‐anddrop of the mouse over terms or phrases which

are visually highlighted

• 4 classes: Device/Line, Device/Line Status,Laterality and D i ev ce/Line Quantity• Each device term was associated with aninsertion insertion status term and assigned assigned an insertion insertion

status• For comparison with NLP output, Knowtator

provides XML output text file




Annotation Procedure:

• 25 batches of 20 reports each

• Each batch of reports annotated by 3

annotators: annotators: 2 who independently independently annotate annotate it fi

then a 3rd as an adjudicator to decide on possib

differences• Roles rotated for different batches

• If no agreement could be reached by team of 

ref t er o subj t ec matter expert• Annotation guideline and device appendix
Evaluation:

• 3 rounds of testing

• Round 1: NLP vs. original annotations

• Round 2: NLP vs. correct de annot ti a ons

• Round 3: Corrected NLP vs. corrected annotatio
NLP Algorithm: Base GATE pipeline

• Base toolset: GATE (General A hi rc tecture for T

Engineering)

• Particular Particular modules: modules: sentence sentence splitter splitter, En

En

tokenizer, Gazetteer, JAPE (Java Annotati

Patterns Engine), POS (Part of Speech) Tag

• GATE Pipeline Functions:

• identify sentence boundaries• id tif en y custom t ko en types via the

Gazetteer: devices, device fragments, ter

indicating indicating insertion insertion, removal removal, presence presence

• Cunningham et al, App Proc Mtg for Com

2002; 1
NLP Inference Logic:

• After GATE pipeline, custom Java modules to a

statuses with devices

• for each line/device term found, find near

insertion insertion status term within the same sent

sent

subject to refinements.

• Example: "Since previous exam, the endot

tube remained in satisfactory position.“

• "endotracheal tube" found as device term

• "in satisfactory satisfactory position position“ found as an insertion insertion 

term of type “present”• Custom module makes association and inferen
NLP Training Process:

• Incorporation of more complex sentences/text pat

• single sentence with multiple insertion status te• multiple devices with single insertion status ter• mul it ple d i ev ces wi ht mul it ple insertion status 

• Each iteration involves expanding the NLP rules an

terms

• First promising results reported in AMIA 2010

• Using 90 annotated annotated reports, reports, achieved achieved over 90% 

and recall in identifying lines/devices and their 

insertion statuses

• (Rubin et al, AMIA Annu Symp Proc 2
NLP Algorithm Refinements:

Clause B d i oun aries Trump P i i rox m ty

Example: Example: "Nasogastric Nasogastric tube redemonstrated redemonstrated,

the IJ catheter in the main pulmonary artery

removed."

• The status associated with "IJ catheter" is"removed“, si if gn ying removal, and not

"redemonstrated" signifying presence

•"redemonstrated" is separated from "IJ

catheter" by the clause boundary ",".
NLP Algorithm Refinements:

• Precedence Order of Statuses

Example: "Interval placement of the left

subclavian line is seen with its tip projecting projecting ove

the cavoatrial junction.“

• A single device term "subclavian line"

• Two insertion status terms "Interval placemen

" ” seen• Use precedence of recent insertion (andremoval) removal) status terms over presence presence status term

for resolution
Recent NLP Algorithm Refinement

• Device Fragments:

Example: “Interval removal of chest and

nasogastric tubes.“

• The words “chest” and “nasogastric” are

identified as device fragments, but by themselv

do not necessarily indicate a device.• The presence of the plural term “t b ” u es and 

conjunction “and” does form a distinct text

pattern pattern recognized recognized by our NLP Algorithm Algorithm.
Evaluation of NLP accuracy

• Comparison wi ht reference stand d ar for eachdevice and insertion status term

• Overall Overall Measures Measures as noted above:

• Recall = TP/(TP + FN)

• Precision = TP/(TP+FP)
NLP Training Training Tool, Debugger Debugger Platform Platform
NLP Training Training Tool, Debugger Debugger Platform Platform
NLP Training Training Tool, Debugger Debugger Platform Platform
NLP Training Training Tool, Debugger Debugger Platform Platform
NLP Training Training Tool, Debugger Debugger Platform Platform
Results
Round 1 Results Results

Round 1 compared output from the original NLP with 

from the original annotated reference standard

Some errors in the reference reference standard standard found and corr

corr

• 8 devices were mis‐annotated by annotators

• The device was omitted ( g e. . “curvilinear line is 

seen”)

• Inclusion of inappropriate devices (e.g. “aortic v

• 20 devices inappropriately identified by NLP

• The device was omitted (i.e. “tracheostomy app

• Inclusion Inclusion of inappropriate inappropriate devices devices (i.e. “monito

devices”)
Discussion
Discussion Discussion of Main Findings Findings

• The NLP algorithm had excellent recall (95%)and p

(98%) for line/device terms

• also good performance performance on inferencing inferencing about th

found

• insertion (89%) and removal (94%) terms

• It is easy to add additional terms such as abbreviat

are found

• The CXR Debugger tool was very hlfl e p u for improv

in quick rounds of development and testing
Remaining Challenges

• C ld ou possibly improve insertion and removal d t e ectio

further

– we had relatively few CXR reports with this informa– we now plan to obtain sample of all CXR reports, n

portable CXR, during the patient stay

• Expect that this will capture more of the insertio

removals, which often are done outside of ICU s

• Linkages of reports for same patient through time whe

sequence of information is the reports is not as expecte

– will require decision rules about what to impute in the gap

• Challenge Challenge for free‐text analyses analyses of radiology radiology reports reports an

extent all other reports

– Identifying the correct set of reports in the absence of struc
Potential Potential Applications Applications of the 

NLP
• Infection surveillance and epidemiological s

• “central line days”

– Would not require real‐time applicati

• Using NLP to feed into clinical clinical decision decision supp

supp

– Requires real‐time application
References

• Rubin D, Wang D, Chambers DA, Chambers JG, South BR, Goldstein MK. Natural l

processing for lines and devices in portable chest x‐rays. AMIA Annu Symp Proc 2

• Goetz AM, Wagener MM, Miler JM, Muder RR. Risk of infection due to central ve

effect of site of placement and catheter type. Infection Control and Hospital Epid

19(11): 842‐845.

• Cunningham H, Maynard D, Bontcheva K, Tablen V. GATE: an Architecture for dev

robust HLT. Applications Proceedings of the 40th Anniversary Meeting of the Ass

Computational Linguistics; 2002, p.168‐175

• Tokars JI, Klevens RM, Edwards JR, Horan TC. Measurement of the impact of risk 

central line‐days on interpretation of central line‐associated bloodstream infectio

Control and Hospital Epidemiology. 2007; 28(9): 1025‐1029.

• Bodenreider O. The unified medical language s ystem (UMLS): integrating biome

terminology. Nucl. Acids Res. (2004) 32 (suppl 1):D267‐D270

• Rubin DL. Creating and curating a terminology for radiology: ontology modeling 

anal i ys s. Journal of Digital Imaging. 2008 Dec; 21 (4): 355‐62.

• Ogren PV. Knowtator: a protege plug‐in for annotated corpus construction. Proc

2006 conference of the North American Chapter of the Association for Computat

Human Language Language Technology Technology. Association Association for Computational Computational Linguistics; Linguistics; 2006. p
For more information, please contact:

Mary K. Goldstein at mary.goldstein@va.Dan Wang at dan.wang@va wang@va.govTammy Hwang at tammy.hwang@va.gov
Results for 500 reports: preliminary

T t o al T t o al(NLP) TP FN FP Recal

Lines/devices 799 796 766 33 30 0.9

status‐insertion 97 95 76 21 19 0.7status‐removal 66 66 59 7 7 0.8status‐presence 620 635 572 48 63 0.9

Status‐other 16 0 0 16 0 
NLP Algorithm Refinements:

• Assumption of nearest device status: good for

h dli an ng lists.

Example: Example: "The endotracheal endotracheal tube and

nasogastric tube are in satisfactory position."

• "nasogastric tube" has the status of "in

satisfactory position“. No st t a us term for

"endotracheal tube“ is found within its own

clause. However, “endotracheal endotracheal tube” is close 

the device term: “nasogastric tube”, and by ourlogic, assumes the same status.