Consortium for Healthcare- 14 -Department of Veterans Affairs

Informatics Research

**Department of Veterans Affairs**

**Consortium for Healthcare Informatics Research**

**Cyberseminar 03-29-2012**

**Using Language Processing (NLP) to Identify PRIVATE**

**Lines and Devices in Portable Chest X-Ray (CXR) Reports**

**VA Health Services Research & Development Cyber Seminar**

Moderator:  And we are at the top of the hour now, so I would like to introduce our presenters for today.  First, we have Mary K. Goldstein, she is the director at Geriatrics Research Education and Clinical Center also known as GRECC, at VA Palo Alto Health Care System.  She is a professor of medicine in the Center for Primary Care and Outcomes Research at Stanford University.

We also have joining her Dr. Dan Wang, he is a research health science specialist at VA Palo Alto Health Care System and VP of Technology, Medcisive LLC, and finally we have joining them, Tammy Hwang, who is a research health science specialist, also located at VA Palo Alto Health Care System, so at this time I would like to introduce our first speaker, Dr. Goldstein.

Dr. Mary K. Goldstein:  Thank you so much for the introduction and welcome to everyone who is here for our session.  You've already covered the material on the first slide, so I will advance to our acknowledgments.  I did want to let people know that this study was undertaken as part of the VA Health Services Research and Development Consortium for Health Care Informatics Research known as CHIR and the sub project Translational Use Case Project with the grant numbers and PIs shown on the screen.  

We've also made use of VA's report extraction facilities and secure server workspace, provided by the VA HSR&D Information and Computing Infrastructure known as VINCI.  I will also comment that the views expressed today are those of the presenters and not necessarily those of the Department of Veterans Affairs or any other affiliated organization.

I'd like to introduce our investigator team because part of the work of doing a project of this type is assembling a diverse team from different specialties and disciplines.  Dan Wang, who has already been introduced and who will be speaking with us later, is an experienced software developer, who, in addition, to doing natural language processing, has served as a software architect for clinical decision support applications in primary care and in mental health.

Daniel Rubin is an assistant professor of radiology and medicine in biomedical informatics research at Stanford.  His research group focuses on informatics methods to extract information from images and tests to enable computerized decision support.  Among other things, he's the chair of the RadLex steering committee and we'll mention RadLex later--of the Radiological Society of North America and chair of the Informatics Committee on the American College of Radiology Imaging Network.  

Tammy Hwang is a research health scientist specialist here who has an undergraduate degree from the University of California at Berkeley in Public Health and she's previously worked as program coordinator for Robert Wood Johnson Foundation Health and Society Scholars program.

Other members of the team I won't go through in the same degree of detail.  We do want to acknowledge some contributions through Dallas Chambers and Justin Chambers, Brett South at VA Salt Lake City and University of Utah particularly for annotation and additional subject matter expertise on this project, particularly from Matt Samore, who heads up the overall CHIR project and other experts in subject matter and then statistical consultation, we thank Shuying Shen for earlier work and Andrew Redd more recently.

So first for us to get a sense of who our audience is here, please just take a quick look through these and select all that apply so we can get a feel for who is in the audience today.  Molly, I'll let you kind of manage the poll.

Moderator:  Thank you.  Yes, I am launching the poll right now, so everyone should see on their screens.  So if your primary interest in the potential application of this tool, in the underlying technology and technical background of this tool--is it because you do clinical work as a licensed health professional at the VA?  Is it because a substantial part of your work includes informatics?  And, finally, is research and/or quality assessment and measurement a big part of your work?  We do have everyone streaming in their answers right now.  We have about 70 percent of our audience has voted so far and we'll give them just a few more seconds and then I will share the results with everyone.  [Pause]  Okay.  It looks like the answers have stopped streaming in.  We have about 80 percent of people that have voted, so I'm going to go ahead and close the poll now.  I'm going to share the results with everyone and I will take back the screen, so, Mary, you can see the results now.

Dr. Goldstein:  Okay.  Thank you very much, so it looks like we have a pretty substantial group of people who are interested in potential applications of this tool and in understanding the technology and background.  A smaller, but substantial, minority doing clinical work at the VA and groups of people in informatics and more than half in research or quality measurement, so that's great.  As Molly said earlier, we hope you will send in questions as we go.  [Pause]  So I thought about the goals of this session with what we thought would likely be our audience and I think that the poll showed us that it was in keeping with who the audience is.  We're hoping by the end of this seminar that the participants will be able to explain the steps involved in conducting a project to extract information from free-text of the VA Electronic Health Records, will be able to describe how to use an annotation tool--we use Knowtator to create a reference standard and to understand something about a natural language processing technique known as NLP is the abbreviation for natural language processing that works for information extraction from chest x-ray reports.  

To achieve those goals, we're going to follow the following very simple outline of going through the project in background, methods, results, our comments and then audience questions and discussion.

So background for this project: Electronic health records usually have extensive information that is very important for a number of purposes and it's in both structured and unstructured format.  So structured data elements are familiar to people as item  of information like a lab value, a vital sign, a diagnosis,  whereas unstructured data is in free text.  For healthcare systems that had electronic health records for a while, there is a realization that a great deal of the information that's in the system, although it's digital and electronic, it is not easily extracted for analysis purposes.  For patient care, quality assessment, quality improvement and epidemiologic surveillance, you need to have structured data and so the VA established the Consortium for Healthcare Informatics Research with a focus on developing information extraction methods.

One of the projects as a component of CHIR is the Translational Use Case Project, which set out tasks that were solicited from VA sources to be of importance, usually for quality purposes and to do early development of methods to extract the information and the chest x-ray project is one of these Translational Use Case Projects.

So portable chest x-rays are usually done on patients in intensive care units.  The patients often have medical devices inserted and such devices, while important for patient care, can also be associated with complications such as blood borne infections, morbidity and cost.  For example, line and device related infections can be correlated with length of time of presence and the line or device type.  Hospitals are often required to report a daily count of patients with specific lines and devices in place.  

There are many different methods of attempting to get those lists of lines and devices.  The lines and devices that are inserted are usually radio-opaque, so they're usually visible on the portable chest x-ray images that are done routinely for patients in the ICU and then documented by the radiologist reading the film and written into the free-text of chest x-ray reports.  So in this project, we were hoping to see if we could develop ways to pull that information out of the radiology reports to get some structured data about the lines and devices.  In designing this system to do that, we had several things in mind.  We wanted to use Natural Language Processing in order to avoid manual chart review, so we could develop an automated system that could be applied much more broadly at a much lower cost in terms of labor, that the system would extract detailed information about the lines and devices and this could potentially enable infection surveillance, epidemiologic research and eventually clinical decision support.

We intended to focus on the part of the radiology report that describes what the radiologist sees on the x-ray, that is not the clinical history, entered by the ordering provider, but what the radiologist actually sees and describes is there.  This is part of scoping the project and as I discuss here what we did in this project, I hope that people will see parallels for other projects of interest to them, as they set up a project doing information extraction from the electronic records.

We also made the decision to focus initially on accuracy of information, extracted from within a single report as compared to a human reader of that report.  This is another scoping decision that needs to be made, so in a future work, we will look at--given a report, you might then put it into the context of the other reports before it and after it, which may put some additional flavoring on what you read in one particular report.  You might also go beyond radiology reports to do other types of comparisons with other clinical data about the patient, but for an initial project, to scope it out, we started with--"Let's see what we can get from the single report that will compare with what a human reader of that report can pull out."

So the methods we applied to do this as an overview--and again these are things that I think are necessary steps no matter what the topic of study clinically.  So first you have to specify the report types of relevance and we had decided to focus on portable chest x-ray reports for patients in the ICU at the time that the chest x-ray was done.  Future work could expand and we are starting to do this now to look at all the chest x-ray reports for that patient during the same admission because patients often move in and out of different units in the medical center.

Then to specify what information is the information that should be identified in each report.  A next step that takes a great amount of work is to identify the source of the document and to select the document.  In keeping with the tremendous importance of maintaining privacy of all the records, we had the opportunity for which we are very thankful to work within the VA VINCI secure computing environment, so that we do not remove any records from there, everything stays completely in that secure environment.  VINCI staff worked with us to extract a sample of the documents that we would need.  I'll say a little bit more later about that important process of finding the documents.

The next step in the overview is to develop a reference standard, a reference standard is needed so that you have something to test your NLP to see how well it's working.  So we develop a reference standard of annotated reports and this reference standard set of reports is not shared with the NLP developers.

Then the next step is developing the NLP code to process the text.  A separate set of documents, also portable films from patients in ICUs, but not the ones in the reference set, is made available to the NLP developers so that they can train their system.  Then, finally, the evaluation is the comparison of the output from the NLP with the reference standard.

So we did this as a staged project and in the early stage, we evaluated with 90 reports and that's been previously reported in a paper whose citation is listed later in the slide and then more recently we did a broader evaluation with improvements to the system and we'll describe that in some detail today and evaluate it with 500 reports.

We have ongoing and future planned work that involves moving beyond one chest x-ray report at a time to link reports for the same patients through time during an acute hospitalization.

So breaking down these steps, there's a step I mentioned of identifying the records.  So it is not immediately obvious which records to pull to do this evaluation.  We have structured data that's attached to the chest x-ray report and all the radiology reports and we use the structured data to identify which reports are relevant.  This can be quite problematic for many types of data, including radiology data, to say which reports one needs.  The reports are identified by procedure names and in some cases by CPT codes, but these forms of identification are not extremely simple.  There isn't one single standardized set of procedure names to pull.

So, for example, applying the CPT code for chest x-ray, we found that there were 1,749 distinct procedure names and in looking through the other radiology reports that had no CPT code, there were many many thousands of those and some of them had procedure names that appeared to be chest x-rays as well.  So for this project, we developed a list of procedure names that appeared to be the chest x-rays that we were searching for.  At present, we have no way to know if we captured every single portable chest x-ray and we probably did not.  For this project, it was not essential--we needed to just get a good number of them or most of them to have a good sampling of portable films.  For another type of project, this could be a major issue, if, for example, the intent of the project was to find every relevant case in VA, different approaches would be needed to identify all of the relevant reports.

We did spend a lot of time going through procedure names to identify appropriate lists of reports and it was a very, very long list of procedure names and once having done that and created the sequel code for that search we are happy to make that available to other VA staff who want to use it and maybe even improve on it, if they find additional ones.  My thought about this for procedure names would be perhaps libraries of such lists could be made available as a shared resource, perhaps on VINCI.

So then the next step after identifying which records to review is to specify the terms to extract.  We identified line and device terms that we wanted, going through several steps of this and we resolved questions about categories of items that we would include by having discussions with subject matter experts.  Again, there's a lot of scoping that needs to be done and decisions about exactly what to include and what not to include.

For this study, we wanted to include lines and tubes that are in the chest, but not, for example, hardware that was in the spine and we also elected not to include heart valves.  We got lists of device terms from the UMLS and also from RadLex.  RadLex is a listing of a lexicon of procedures and it includes some standard procedure names and is a good starting point for lists of procedures, but note that for both UMLS and RadLex, these use standardized terminology and the actual language used in many reports is quite different from the standardized terminology, so we also reviewed actual chest x-ray reports to identify additional terms and talked with as many people as we could to find out about terms they thought might apply.

We then developed the reference standard and the reference standard is the set of carefully annotated reports against which we can measure the accuracy of the NLP algorithm.  We do annotation training with the people who will be annotating the documents and the annotators are trained both in the process of how to identify things on the document and to use a written annotation guideline that we prepare for them.  The guideline includes an appendix with a list of all the terms that should be identified and in the early rounds of this before moving on to the reference standard, we asked the annotators as they were training and working with the initial documents that were not part of the reference set, to add terms or bring to our attention terms they thought possibly should be added, and to discuss with us any problems they encountered.

In this process of annotation, we understood which issues were confusing for annotators and then revised the annotation guideline to address commonly occurring issues of how things should be classified.  

The annotation guideline is the specification of exactly what should be found and so the guideline is shared with the NLP developer so that everybody is aiming towards the same goal and using the same set of rules.

As I mentioned before, the final reference standard which has been annotated, after we finalize the annotation guidelines, is blinded to the NLP developer, so that the evaluation will test the NLP on documents that were not used to train it.

I also wanted to mention something about the concept of span of text because this comes up in the evaluation.  Every character in the document is numbered starting with 1 as the first character in report and the "span" of text is the character positions of a particular portion of the text.  So, for example, if the report text in it its entirety were, "the tube is present", the word tube has span from 5 to 8, which is its character position.

So in our reference standard, we annotated 500 randomly selected portable chest x-ray exams, obtained in the ICU between January 1, 2006 to January 1, 2009 from 46 geographically dispersed VA medical centers.  We excluded medical centers that had small numbers of ICU patients.

We stratified the reports by medical center and length of stay.  Most patients in ICU have short stays and if we did a simple random selection without stratification on length of stay, we would have a large representation from the small number of patients with long stay because they have so many more x-rays and we wanted to have some from them, but we wanted to have a very good representation of the short length of stay which is more common.

So then the next step in the overall procedure is the evaluation of the accuracy of the NLP, in which you compare the NLP output with the reference standard.  For each line and device and also insertion status, which Dan will explain in his part of the talk--I'm here just introducing the concept of the precision and recall which are based on--what for most people--are familiar concepts, of true positive, false negative and false positive.  With the true positive found in both the reference standard and the NLP output.  The false negative found only in the reference standard and the false positive found only in the NLP output.

We computed the standard overall measures which are [inaudible] standard from information retrieval.  They're described in many sources, including the Sebastiani article referenced below, based on conditional probabilities and can be computed easily as precision.  It's the true positives over the true positive plus false positive and recall as the true positive over the true positive plus false negative.  

So we now have our second poll to give us a sense of how much experience people have with annotation, which is coming up as our next segment of the talk, and I'll turn this back to Molly for the poll.

Moderator:  Great.  I have launched the poll question and people are getting right in there and voting.  We've already had 40 percent of people vote.  [Pause]  So again those options are: I have experience using text annotation tools, I have some familiarities with the concept of text annotation.  I am not familiar with text annotation, other than what I've heard during today's talk.  About 80 percent of our audience has voted, we'll wait just a few more seconds to see if anybody else wants to get in a last minute vote.  Okay.

Dr. Goldstein:  Okay.  Tammy is now taking over, she's our next presenter.  

Tammy S. Hwang:  So it looks like about a little less than half of the audience has no familiarity with text annotation, other than what they've heard today.  So that's good because I'm going to go into a little bit of detail about how you we went about doing the annotation.

Moderator:  Tammy, I have a question for you.

Tammy Hwang:  Yes.  

Moderator:  We do have some audience questions that have come in, do you all want to take this opportunity to address those now or would you like to hold them until the end?

Dr. Goldstein:  We can go ahead and take some questions now, we can probably do it pretty quickly so we can proceed with--but could you go ahead and let us know the question?

Moderator:  Sure.  The first one is: "Can you describe the annotation process?"

Dr. Goldstein:  That's about to happen.  Tammy is going to do that.  Okay.  What's next?

Moderator:  "How is only a part of the CXR reported selected?  What is the process to extract the reports in VINCI?"

Dr. Goldstein:  Okay.  I'm not sure I completely understand the question, but the process of how we identify a portion for the NLP is something Dan will address in his talk, as part of the NLP process.  If you have *a* report and you want to say, "We don't want the clinical section piece," that can be done in the NLP.  The question about how the reports are extracted from VINCI is maybe a little more complicated question that if the person who sent that in could send in more clarifications of what the question is, we could come back and address that at the end.

Moderator:  Excellent.  I'm sure he can write in more information.  Two more quick ones: "How do you calculate sample size for reference standard?"

Dr. Goldstein:  I actually asked Shuying, our statistician, to do that and we would have to have a separate communication about that.  We could provide that information later.  It's standard method of determining what exactly you need to find, what you expect the variability will be et cetera.

Moderator:  Thank you, and the final one that's come in up to this point and you may have addressed this: "Why only 500 CXR records, why not all available records?  Is it due to manual review?"

Dr. Goldstein:  Yes.  This is because 500 were for the reference standard and, of course, it's extremely labor intensive to do the manual annotation to create the reference standard.  The expectation is that if you have good performance, you can then apply it to as many as you like, potentially all of them, but you first need to have a very good reference standard to see what your performance is, so that's the reason for limiting to the 500, to create the reference standard.  Those are great questions and I'm glad we got those.  I think we'll go on now to Tammy's presentation about the annotation.

Moderator:  Great.

Tammy Hwang:  Okay.  In order to actually annotate the text document, we use the Knowtator text annotation tool and Knowtator is integrated with the Protégée knowledge representation system as a plug-in.  It uses Protégé knowledge representation capabilities to allow for development of an annotation schema and the annotation schema that we created comprised four classes: Device/Line, Device/Line Status, Laterality and Device/Line Quantity.  The annotations are done by highlighting the cursor over terms or phrases, which are then highlighted and once annotations were completed, we can use Knowtator to produce an XML output text file, which is ultimately what is used to compare with the NLP output.

So this screen shows the capture of Knowtator and the box on the very left, this pane right here, shows the classes of our annotation schema and each class corresponds to a different color.  So, for instance, the Device/Line class is light blue and the pane in the middle shows the report text and this next slide is a zoom-in on the middle pane, so you can see the report text more easily and this next slide is a zoom-in on the middle pane, so you can see the report text more easily.

  So, as you can see, we highlighted all of the instances of Device Line mentioned in light blue and correspondingly the Device Line Status class corresponds to the magenta color.  So each time the status of a device is mentioned in the report, we highlighted it in magenta and with each status annotation, we're able to link that status to a specific Device Line reference.  So in this case, the status of interval placement highlighted in magenta corresponds to the mention of the nasogastric tube highlighted in light blue and we also annotated information such as laterality, mentions of whether the device was a left-sided or right-sided device and we also annotated Device Line quantity or in other words any mention about the number of particular device and lines mentioned in our report, but we only used the information from the Device Line class and the Device Line Status class to evaluate the performance of the NLP.  

This slide just zooms in on the right-hand pane, where you can link the status of a device to a specific Device Line reference and this is [inaudible], you can see how everything goes together.  I want to talk a little bit about the report, so the 500 reports were divided into 25 batches of 20 reports each and for each batch of reports the annotations were done independently by two annotators and a third annotator then served as an adjudicator of the batch, making a decision on discrepancies between the set of annotations and the role of adjudication rotated among the staff so that for each batch we had three annotators serve in each role.  

So, as Mary mentioned before, we developed an annotation guideline document, which provides specific instructions on how to annotate the report and we also had an appendix of device terms, which the annotators can use as a reference and these documents were consulted prior to and during the annotation task.

As you can imagine, annotation was quite an involved and detailed process and not surprisingly the reports were quite complex and there were many unanticipated cases, which were not detailed by the guideline or the device appendix.  There were cases where the report text was ambiguous, so that a clear decision couldn't be made by the annotation team, so we compiled such cases and resolved them by asking subject matter experts, including radiologists and if even the subject matter expert couldn't make a clear decision on the annotation, the status was declared as cannot be determined.

So our method of evaluating the performance of CSCE against the annotator reference standard was [iterative]--we did multiple rounds of testing and the first round involves running CSCE against the original annotation and after this first round of testing we did an error analysis, using the debugger tool, which Dan will describe a little later in the presentation, and this tool allows us to capture every discrepancy between the NLP and the reference standard.  During this error analysis, we classified every discrepancy we came across and classified them as either an NLP error or an annotation error.  An NLP error indicated that the source of the error was CSCE and not--changes would have to be made to the tool in order to resolve the discrepancy, but on the other hand, an annotation error indicated that there was an error in the reference standard that needed to be corrected.  Although, of course, we intended the reference standard to be error free, there were instances of human error while annotating the report and all of these were noted and the reference standard was updated based on these findings.  After the reference standard was updated, the NLP was re-run against the corrected reference standard and the second round of testing produced a second set of scores and finally for the third round of testing, we made changes to the NLP to improve its performance in light of this discrepancy that resulted from NLP error.  So I'm going to hand it over to Dan, who will talk about the NLP algorithm. 

Dan Y. Wang:  Hi, so I'm going to describe in some amount of detail the Natural Language Processing algorithms that we used to [inaudible] this report.  So we based our algorithms on the GATE pipeline, GATE stands for General Architecture for Text Engineering and there's reference on the bottom of the screen.  The particular modules used in GATE, included the sentence splitter, which allows us to break the report into individual sentences.  English tokenizer, which allows breaking it up into a particular token or set of tokens.  The Gazetteer, which allows you to recognize the particular classes of terms, JAPE, which is the Java Annotation Patterns Engine, which actually allows you to recognize various phrases, which follow a particular text pattern and also the Part of Speech Tagger.

So in terms of using the GATE pipeline, whenever there's a report, we use the GATE pipeline to first identify the sentence boundaries, as I will discuss.  Our major assumption is that when there's a device and also device status, we assume that they are actually in the same sentence.  We're not going across sentence boundaries.  Then we actually customize GATEs, identify custom token types that can be done very conveniently through the Gazetteer, where we can actually manually enter lists of device terms and also terms indicating insertion, removal and presence of the device and these are Device Status terms.  I also mention in this slide about device fragments, which I'll discuss a little bit more in detail in a later slide. 

So after the GATE pipeline runs, we've actually been able to identify particular sentences and in those sentences, the various mentions or terms [inaudible] device and the associated insertion status, at that point, we then have some custom Java logic to associate the status with the device.  So for each line device term found, we find the nearest set insertion status term, which is actually within the same sentence, subject to some refinement.  

So to give you a basic idea, let's say we had a sentence that said, "Since previous exam, the endotracheal tube remained in satisfactory position."  So the Gazetteer identifies endotracheal tube as a device term and also it's able to identify the phrase "in satisfactory position" is actually found as an insertion status term and is also a [type] and this is actually a [type] present.  So the custom module then makes this a very simple instrument that since we can associate "in satisfactory position" with endotracheal tube and, therefore, endotracheal tube is present.  So that's how we actually do the basic inference.

As you might imagine in a chest x-ray report there are more complicated text patterns than that and that's really part of the NLP training process.  So we started out with simple examples, where we have single sentences with one device and one insertion status term, then gradually we incorporated text patterns with multiple insertion status terms and then multiple devices with single insertion status terms and as I will describe, there are various ways that we were handling this and as we have incorporated more complex patterns, our accuracy improved and we were able to handle more reports.  

Each iteration basically involves expanding the NLP rule set and also a lot of times adding terms and I think we have previously said that our first results were reported at the AMIA conference 2010, using 90 annotated reports.  We achieved at that point over 90% precision and recall in identifying lines and also their insertion status [inaudible].

So let me describe a little bit what kind of refinements there are.  So the example: Here is a sentence with multiple device terms and multiple insertion statuses, so we have nasogastric tube re-demonstrated, the IJ catheter in the main pulmonary artery removed.  So the [inaudible] I want to focus on is the device term, IJ catheter, there are actually two insertion status terms found here.  One is the demonstrated, signifying presence.  The other one is removed, signifying removal.  Well, the way that we found through a lot of testing [inaudible] when there is a clause boundary saying--the comma that marks the phrase the clause nasogastric tube re-demonstrated then we don't really want to search for the insertion status term for a given device term across the clause boundary and that's how we know that we actually associated remove with the term IJ catheter.  

Here's an example where you would actually have multiple insertion status terms associated within a given sentence.  So here we have "Interval placement of the left subclavian line is seen with its tip projecting over the cavoatrial junction," and both interval placement and actually "is seen" are actually identified as insertion status terms. 

We have one device for the subclavian line.  We actually have a pre-defined precedent or it means that anything that relates to recent insertion and also recent removal status terms have precedence over the present terms and that basically reflects semantic usage of these terms.  By using this rule, we find that the correct insertion status terms would be associated at interval placements.  

More recently, we've been working on what I call device fragments.  These are really incomplete device fragments that could potentially lead to a lot of false positives.  In this example, we have "Interval removal of chest and nasogastric tubes."  So we really want to identify chest tubes, but really all we have is the fragment "chest".  

So how does that work?  Well, it turns out that we can look for a particular pattern.  First of all, the words "chest" and "nasogastric" are identified separately as potential device fragments and then we have the plural term "tubes" and the conjunction "and", and this particular pattern together allows us to meet our [inaudible] pattern and, therefore, we can say that, "Oh, this also talks about chest tubes.

That gives you a little bit of an idea of what kind of refinements we have in order to be able to handle various text patterns that we've encountered.  Here I think we will continue on talking about the evaluation of the NLP and also the iterations that we did and just to remind you, we have talked previously about this.  We're always comparing it with a reference standard.  For the training, of course we actually have a separate set of training reports.  

What we found that was really, really useful was to be able to do iterations very quickly and we actually developed a [revisual] debugging platform that allows us to be able to do this very quickly.  This is actually an output from that visual debugging platform and on the top screen it really shows a summary of the comparison of whatever number reports that we're looking at, the training reports in this case, against the NLP.  Now I won't go into details too much but the main idea is that you can very quickly do a quick summary to see where you're at.

For example, here, you can see that the status insertion is at .75 [inaudible] of training reports.  We probably want to address that and on the bottom you can actually see the reports in detail and more importantly, if you go down on the right-hand side you can actually do [inaudible] and I'll show you this in a little bit more detail.  So you can actually find the particular reports that [inaudible] devices and also devices with particular status.  So you can look for all reports that include the findings about removal status also reports that have only present status.  You can also segregate them by whether they are the ones that actually have error or the ones that are actually without error because a lot of times you want to look at both of them because you want to confirm that you made a change that the ones that previously didn't have an error continued to be correct.  Of course, for the ones that do have error, you want to see that there is something that we can do either in terms of changing some of the patterns of the rules that we have in NLP.

Then if you are given a report, you can actually drill down and look at the entire text.  We actually have a color coded scheme of looking at the result.  When the standard manually annotated report agrees completely with the NLP, we have a gold color.  When you only get it from the manual applications, it's an orange color and then if you only get it from the NLP, which happens sometimes too, then you get a green color.  Of course the idea is that you'd want to have a lot of gold, but I will mention that we actually in terms of identifying true positives--it is not necessary for us to get a complete exact textual match.  We actually allow for partial matches, so that if the NLP found a tracheostomy tube, and the notator only found a tube that actually was treated as a true positive.

For a deeper analysis, with this tool, you can actually get the actual text positions which is very useful for debugging, particularly in long reports.  Sometimes we found as many as 7 to 8 device terms in a given report, so it was very useful to see what status was associated with which insertion status term, so this allows you besides from the graphical view to look in detail as to what was really going on and to search through them.  The top green graph shows the NLP results and the bottom table shows the manual annotations.

So let me talk about our results.  As Tammy has mentioned, we have done three rounds, so for the first round, we have actually--after finishing up 90 reports, we just compared the output from the NLP with the annotated reference standard and surprisingly we did find that there were actually 8 devices that were mis-annotated in some sense that also gave us some confidence in NLP that we were actually able to find this very quickly.  

An example was that the device was omitted and then we also found out that we actually included some inappropriate devices that I think was probably due to the fact that our annotation guideline was, of course, always being amended and I think that given the length of time that it took to do the annotation that there were certainly some reports that were early annotated that probably didn't quite reflect the most current annotation guideline and then we also found out 20 devices that were inappropriately identified by the NLP which we had to do something about.

We were able to--first of all, let me just show you that we were able to, first of all, correct the reference standard to the best of our ability and here we show a table results.  As I said, the focus is on the device terms and the three insertions that--whether something was recently inserted, something was recently removed or whether something was present.  We also found that there were some other fairly rare statuses, like something was being replaced and as you see, for our reference standard, the results are in the [rough] modes column under the heading "total", so out of 500 reports, we found about 800 device terms, out of which about 94 signified recent insertion, [70] signified removal and 624 actually just mentioned that it was present and there were 12 [inaudible] statuses. 

Then on the rest of the table shows what you find from the NLP results.  It shows the columns totals and also a detailed breakdown in terms of true positives, false negatives, false positives and recall decision.

The message I want to say that even without any further kind of modification after our training and 90 reports, we were able to get excellent recall decision for lines and devices and whenever they were mentioned to be present, and we were getting--relatively good, but not quite as good results for the insertion status recent insertion and recent removal.  

After careful analysis and doing further work on the NLP process, we were able to put in some improvements.  Primarily--and I'll just show you the results very quickly, so this is against the same 500 reports and this is the results that we get after doing some work on the NLP improvement and we now find that again we have excellent recall precision for finding devices and for identifying insertions status at present.  We're around 90 percent for recent insertion and removal.

The good news that we learned is that mostly the improvements were made by actually adding a whole kind of class of terms, which possibly can indicate that our training sets could really still have been improved or really have been expanded some more because clearly as our reference standard grew, we were actually incorporating other VA sites.  Corresponding to that, I think that that really shows that our basic NLP rules are still pretty fairly robust, since we were able to get these very good results.  Now I will turn it over to--

Dr. Goldstein:  We'll have just very quick comments that we have about this, so as Dan mentioned, the NLP algorithm appeared to have excellent recall and precision for lines and device terms and also had good performance on inferencing about the terms that it found identifying as recently inserted or recently removed at the percent shown.  It's very easy to add additional terms, if we find more abbreviations, more terms, they can be very easily added to improve further.

The debugger tool that was developed for this project was very helpful in the process of improving the NLP through quick rounds of development and testing.  There are a lot of future work to be done here too, some remaining challenges, so we can improve insertion and removal detection even further.

Out of the 500 reports, there were relatively few with that information, compared to those where the device was merely present, not specifically recently inserted.  Now that we're working on getting a sample of all the chest x-ray reports to the patients, not just [reportable] film, we expect we'll capture more of the insertions and removals because these often happen outside of the ICU today.

We also have future work for linkages of reports to the same patients through time and that will require its own set of decision rules about what to report to impute gaps where there's not a specific mention on reports in sequence of everything that was there the day before and there are challenges for free-text analyses of radiology reports and to some extent for all other reports.  The challenge I mentioned of identifying the correct set of reports in the absence of a completely standardized set of structured labels for reports.

So potential applications--I think you get the flavor of that from what we've presented so far, but to be applied for infection surveillance and epidemiologic studies, for example, for central line days, this could be done as retrospective analysis of reports, wouldn't require a real-time application.  

Eventually the NLP could feed into clinical decision support potentially for real-time application, perhaps to advise people caring for patients about the number of line days or other information on devices [being] put into NLP.  The references are available on the slides for download and for more information we have our contact info here.  We'd like to thank you all very much for your attention and I turn this back to Mollie.

Moderator:  Thank you very much.  We do have some more questions that have come in and the next one is: "How is your NLP pipeline different than [CTAKS], why do you use GATE?"

Dan Wang:  I think for what we were using CTAKS would probably have worked just as well.  I don't think that there is--for our application a particular advantage of CTAKS or GATE over one another.  I have heard that CTAKS--there have been some sites where CTAKS has been used for large scale surveillance and I don't personally know if there have been other large scale applications of GATE, so I don't--at least from my experience--it seems to me that GATE would also be able to handle that kind of a throughput where you have--maybe looking at thousands of reports at a time.

Moderator:  Thank you for that response, Dan.  The next question: "Could a similar project be performed using automated retrieval consoles known as ARC?"

Dr. Goldstein:  The ARC console developed by Len [DeBoliose] is a great application and we just did not have the opportunity of using it for this particular project because of just the timing of when things were happening and so on.  I don't know enough in detail about ARC to say whether it could do everything that we did here.  There are probably parts of the report selections that could be done with ARC.  This detailed NLP programming would need to be put into it.  

Dan Wang:  Yes.  I would agree.  I mean we would have to somehow incorporate some of our input logic into something like ARC, ARC is more--yes, I think we could conceivably integrate with ARC.

Moderator:  Thank you for that response.  This next question has a few parts to it.  "Can you clarify why partial matches are scored as TP, this introduces mathematical issues.  For example, what's the similarity model, metric versus generalized et cetera?"  

Dr. Goldstein:  We really focused on what was it we were looking for here and we were looking for counts of lines and devices and tubes and lines and it was--for this initial take on things, sufficient to know that there were tubes and lines present.  We feel that you would have to--for any particular project say, "Well, what exactly do you need to define in that project," and define what's allowed as a partial match or not, based on that.  The general idea of allowing partial matches also allows for if something has extra blank lines in it or it has a slight difference in spelling or something else that makes--you very often--even if you want the complete term and multiple component terms, you might still want a partial match to allow for a very slight variation in the spellings or something at the end or beginning of it.  Dan may also have a comment on this.

Dan Wang:  Yes.  Certainly in terms of insertion status, there are multiple ways of--for example, signifying that a device is present and in a given sentence, there can be two or three terms that we identify as being present and it's not really necessary for the understanding to identify a particular one.  It's not really necessary to understand whether a device is present in that context, so actually a partial match is, I think, quite reasonable.

Moderator:  Thank you both for those responses.  The next question we have: "How ready is this algorithm to be used in clinical care?  Can it be used on other note types?"

Dr. Goldstein:  We don't know the answer to that, that would have to be tested.  It was designed for use in chest x-ray reports, which have their own language.  There are likely to be many similarities, but also many differences in other report types once you get past radiology reports.  So the language that radiologists use has a similarity, but also has its own different flavor from the language that's used.  Progress reports are particularly problematic because they tend to be quite idiosyncratic in the abbreviations used.  Admission and discharge notes tend to be somewhat more formally structured and use more standard language, might have more similarities, but that's something that--applying it to other report types is something that would need to be tested, but we'd be interested in doing that.

Moderator:  Thank you for that response.  We have come to our final question: "How can we apply these processes in our VA or non VA institutions?  For example, to use for research--is this proprietary?"

Dr. Goldstein:  No.  This is not proprietary, this is developed as part of VA work and it's our intention that it be available through whatever mechanisms the VA is willing to allow it to be available that from our perspective nothing proprietary or commercial about it.  For VA investigators, it is available on VINCI.  They would, of course, need to go through all the appropriate procedures to have access to any records and, of course, that has its own set of procedures, but it is available for them.  

The question about making it available for outside the VA--I think that's above my pay grade.  That's a question that we would certainly like it to be available and I think the intention of VA is to make these things available, but I think there's some procedure that may be in process of being set up for making VA-developed software available to others and I'm not familiar with all the details.  So if there were a request for that, we would need to explore that question.

Moderator:  Thank you for that response.  That is the final question and we are just a few minutes past the top of the hour, I know that some of our attendees will be exiting the session soon and I just want to ask you when you exit the session, please do complete the short survey that's going to pop up on your screen, it's just six short questions, asking for some feedback and we really do take into account your comments and help that guide our future sessions. I would like to give each of you now the opportunity to give any final statements you'd like to. 

Dr. Goldstein:  Oh, from the speakers?  No.  We just are very happy to have had this opportunity to present the work and we hope that there will be places that the VA can make use of this and apply it and if there are VA staff who are interested in working with us on further development, we would be interested to hear from them.

Moderator:  Excellent.  I very much want to thank each of you for presenting for us today, this was very valuable information.  We had a very attentive audience and we really appreciate you sharing your expertise and as everyone can see on your screen, there is contact information for further followup and also this session was recorded, so you can access it through our Cyber Seminar Archive Page, which can be found through HSR&D Web page.  So once again I want to thank all of our speakers and thank our audience for joining us and this does formally conclude today's HSR&D Cyber Seminar.  Thank you.

Dr. Goldstein:  Thank you, Mollie.  Bye, bye.

Moderator:  Bye, bye.

**[End of Recording]**