**Transcript of Cyberseminar**

**Consortium for Healthcare Informatics Research**

**Voogoo:  A Search Tool For Clinical Data**

**Presenters: Qing Zeng-Treitler, Doug Redd**

**November 27, 2012**

**Voogoo:  A Search Tool For Clinical Data**

**November 27, 2012**

Moderator:  And we are now at the top of the hour, so I would like to introduce our speaker.  Presenting for us today is Dr. Qing Treitler Zeng.  She is one of the CHIR principal investigators. She is also the VINCI lead researcher for developing natural language processing tools for VINCI users.  She is also an associate professor of biomedical informatics at the University of Utah.  At this time I would like to turn it over to her.  Doctor, you will see a pop-up on your screen that says “show my screen.”  Go ahead and click that we’ll get started, perfect.  Thank you.  

Dr. Qing Zeng-Treitler:  Yes.  Are you seeing the full screen?  

Moderator:  Yep.  We should be all set.  Thank you.  

Dr. Qing Zeng-Treitler:  Okay.  So today we will talk about this tool called Voogo.  It’s a search tool for clinical data.  As many of you know, VINCI has data from all of CDW and in fact more than that.  So it’s a huge dataset to use. 

And to facilitate the use of this data we developed the Voogo tool.  The next slide—and also we have an increase in demand from various users who are operational users, researchers who want to access the dataset.  This is although you can call up VINCI help desk to pull down a particular dataset, it is certainly more convenient for researchers and operational users to interrogate the dataset themselves.  

And also our experience is simple queries often result in high false positive and false negative rates. That’s another reason why we developed the tool.  We’re going to give you a demo.  So I wouldn’t go through the details of the features we’re offering.  As a summary we offer both text and structured data search.  You can have multiple views of the data, summary views, patient level views and document views.  And we provide query suggestions.  And also we have a SQL server and within backend.  

At this point the version that allows you to without IRB search the whole of VINCI is still pending NDS approval.  However, we have released a version that you can use to search your own IRB-approved datasets.  And those datasets can be big, can be small.  In fact if you have IRB that covers all of the TIU notes, all the VINCI data, you can search that too, but IRB approval is required for this tool to connect to your dataset.  What I will spend some time talking about is how well how did we evaluate this tool.  So as all search engines, you—everybody knows Google is a prime example for a search tool that everybody is familiar with.  Our tool is a little bit more complicated that you can’t just pack in—you can tap in a strength.  If you can Google you can use our tool. 

On the other hand, clinical data is more complex than Web pages.  And finding information in the clinical data repository has certain requirements on accuracy, well if you’re searching on the Web the more important thing is the top returns need to be relevant.  So there are some differences.  So in the first evaluation we—we come back with several evaluations.  The first one we took 300 notes from PTSD patients randomly selected and 300 notes from diabetes patients.  And we used the twelve topics, for example suicide ideation, diet, exercise, and so on.  And we have a clinician review the 600 notes and mark them as relevant, possibly relevant, or definite or irrelevant. So the reason for this is sometimes just seeing the context one note it is hard to say if a note is relevant to a particular topic.  

We tested using nineteen queries.  So for certain topics we prepared more than one version of the queries because sometimes we prepared more detailed queries like symptoms, PTSD symptoms.  And in one query we just provided a PTSD symptom.  And the other query we actually spelled out certain of the symptoms like insomnia and so on.  We experimented with three query expansion methods.  The baseline results, the baseline method is the sum of the TFIDF.  TFIDF is a common measure of relevance.  It stands for term frequency inverse document frequency.

We used four different performance measures, precision, recall, F-measure, and P10.  The precision is the positive predictive value.  Recall is the sensitivity.  F-measure is the harmonic mean of precision and recall; P10 is the top-ten ranked documents.  The relevance of these is that when you use a search tool especially to search clinical documents, you want to—you wanted the results that came back to be reasonably sensitive and reasonably precise.  And the F-measure gives you a balance of those.  

And in some cases maybe you only care about the top, let’s say top ten, top 200 or top end, and because maybe we’re getting back lots of returns and you only need to find the most relevant hundred people.  That’s why we also measure the top returns’ accuracy, the precision.  So how did it do?  Well here you can see we’re showing you the precision, the recall, and the F-measure.  And as you can see the precision, the baseline scored the highest precision in here, and so it’s 0.62.  And other precisions we’re showing here are lower.  That is actually understandable. Every time where you extend query you include more terms.  So the precision suffers somewhat.  

And how did we do with recall?  Well in the recall the highest number is actually the topic model.  In fact all three expansion methods, using synonyms, using topic model or we used a method called the semantic relations extracted by this program called SemRep, and all the recalls actually are higher, much higher than baseline recalls.  That shows a drop in the false negative.  In fact the topic model doubled the recall baseline, which means you actually cap—we actually captured twice as many relevant cases.  

And in the F-measure, which is a balanced measure of precision recall, you can still see that we achieved a higher F-measure and in all the expansion methods, in some cases almost fifty-three percent higher in the topic model and fifty percent higher than the baseline.  So we think this is actually very useful.  

Moderator:  I do have a clarification question that just came in from the audience.  What is the mean—what is meant by harmonic mean when defining the F-measure?  In slide six what should the harmonic mean of precision and recall—should it be a number between those two values?

Dr. Qing Zeng-Treitler:  Yes.  So it’s hard for me to spell out the formula.  We’ll be happy to send further additional information on to the slide if that’s possible.  We could give the formula.  But if for a common sort of—a similar way of thinking about this is the area under ROC where you look at a combination of precision and recall, which many people are familiar with.  

Is it—sorry, Molly, you were trying to say is it possible for us to send the formula or attach it to the slides after the seminar?

Moderator:  Yes.  That’s not a problem at all.  You can send me a revised version and we’ll upload it with the archive.  

Dr. Qing Zeng-Treitler:  Okay.  So this is treating possibly relevant as relevant.  And we can also treat possibly relevant as irrelevant.  And here we have similar results.  There is less difference in terms of precision.  And there is still quite a big gap in recall, so query expansion greatly improved the recall.  And for the F-measure we still show significant improvement here.  As for the top ten precision you can see here if we treat—the baseline actually has overall the best precision here, which is sixty-one percent.  And the top model actually tied with that if we treat possible, possibly relevant as relevant.  So the gap in precision in the top ten is less obvious in the overall that shows very special, actually improves the ranking.  

So following that we did a second evaluation, same nineteen queries.  We revised the three expansion methods, mostly limited a number of expansion terms we provide.  And we also added ensemble method which where we combined the three methods.   The baseline method is still the same, the sum of the TFIDF.  Actually here we used two measures.  And one is MAP is the mean average precision, one in P10.  The mean average precision is basically a measure combining, calculating the mean precision at different recall points.  So recall is sensitivity.  So we’re basically calculating the positive proof, average, mean average positive predictive value at different sensitivity thresholds. 

So here again you can see that the expansion methods all resulted in better mean average precision.  And the best was achieved by the topic model.  And after we limited the number of terms that we added to the queries we actually achieved a better precision, top-ten precision at will.  Here you can see the topic model achieved a higher precision than the baseline.  So this is the improvement from the previous evaluation. 

More recently we participated in a TREC study where there were 50 queries, 100,000 documents.  We used a number of different methods for this dataset and the queries.  We conducted concept indexing, negation filtering.  We did—we played with various spatial methods.  We created new ensemble methods. 

And in this case we used Lucene, many of you have probably heard about Lucene, as the baseline.  So we here we used the four different precision per in performance measures.  One is inferred average precision.  One is inferred normalized discounted cumulated gain.  One is R-precision.  One is P10.  

So these measures are here because 100,000 documents is too much to provide a full review, a notation by human reference standard.  So there is so a partial reference standard was created.  And to use those partial reference standards we cannot have the usual precision recall, although the P10 still is relevant, the top ten results still relevant.  But roughly on all these measures higher number means better performance.  And also we want to tell you that this is a little bit because this dataset came from Pittsburgh.  And it’s and the queries are also different from the queries we typically use in that it included a certain criteria that we usually use only for the structured data like patient age, patient gender, and certain medications, so on.  

So the performance on this dataset is not exactly what we would expect on the VA dataset.  So we actually did a little bit worse here, but still if you look at—our best run is actually we used ensemble methods for expansion.  We used concept indexing and negation boosting, the concept plus negation still trying to boost the word search.  So when we combined these different techniques we did better than the Lucene baseline.  Like on the first measure we went from 0.15 to 0.20.  So that is about thirty percent increase in performance.  And they also did better on the P10 as well at five percentage points. 

So we also did better than the median.  The median is so many people participated in this challenge.  The median is the median performance of all submitted runs.  So as you can see the median is better than the Lucene baseline, but when we combined all the different techniques we could outperform the median.  We’ll show you the features.  We will show you the demo in just a minute.  I just want to briefly say there are some future steps we’re contemplating, are currently include a number of structured fields, but certainly not all the structured fields.  We’ve been getting some feedback and we welcome your feedback of which structured fields to include for the search.  

And also we want to support more complex text criteria.  We support certain combinations like or and and, but we’re thinking about adding more Boolean function like not.  We’re also working on providing the temporal search support and see our own research we realize this is a very common desire.  Right now we have the capability to index the documents through natural language processing and provide a concept level indexing to supplement the word level indexing, but we haven’t run all the concept indexing on the VINCI data.  We would like to work on that, although that’s going to take some time and computational resources. 

We’re also working on designing more intelligent visual analytics.  We realize that people want to really manipulate these not the query criteria, but also looking at different, looking at the result set in different ways.  So we’re working on providing better visual analytics.  

With that I am going to go to the demo.  So this is the Voogo interface.  It looks actually fairly simple.  To get started all you need to do is to type in a term here.  So we’ll first start with a term, diarrhea.  And recently we had a—actually the director of our group wanted to search for diarrhea in patients with C. diff.  So we first search for diarrhea.  And you can choose different datasets here.  You can use—we have a sample dataset which we’re going to use mostly for this demo.  It has 100,000 documents.  

And on the backend we have two versions.  We have a version that’s linked to the Microsoft SQL server.  We also have a version that’s linked to Lucene.  These are just they have—both of them have word level index.  In some cases the Lucene works faster, but not always.  We can also search all of VINCI.  When people have specific IRB- approved datasets they can go to for example this project, you can see here Pro-Watch is a very big project.  And they have actually almost all of the VINCI data.  

So when people have IRB on that dataset we make a link to it.  People can search on that dataset as well.  So since the search on VINCI because the VINCI—because the network traffic and so on we’re going to show you mostly using the sample dataset on the SQL server.  

So all you need to do is select the dataset, put in the search term and then just hit search.  So you can see we give you an overview.  We found about 3,000 matching patients, also about 3,000, 3,500 matching documents.  And you can look at the patient’s age, they’re deceased, or whether they’re deceased or currently alive.  So some of the patients are already deceased and this is important if you want to do a prospective study.  Sometimes people want to identify cohorts which a patient who are currently under the care of VA and alive.  And we can take a look at the gender distribution.  Since this is VA most patients are male.  Also we provide a list of the top medications these patients are taking, the top diagnoses they have, the top procedures they have received.  Here you can see mostly the top ones are patient visits, but as you go down the list quickly you see other things, lab tests and so on.  

We also give you a sense on the document, the type of document where we found the information.  The top line here is actually nursing admission evaluation note, followed by hospitalization note and discharge summary, long-term care note and so on and so forth.  

And finally you can take a look at the details of the document, assuming this is IRB- approved that the study you have access to.  And here you see aggregated data by patients.  We give you a brief summary on their age and gender.  The blue ones are male.  And if there are female patients they will be shown as pink.  If you click on one of the patients you can see some information about this.  This first patient is ninety-one year old.  He has two relevant notes that we’ll retrieve.  Obviously the person has a lot more notes than that, but we’re only showing you the notes that contain the search term.  We can click on the note and you can see that to provide—to protect the people’s confidentiality we have blocked out all of the text, but the diarrhea term is definitely in there.  And you can see a different note, the different note, but still contains diarrhea.  And usually people can get a very good sense on the patients and whether they truly have diarrhea or not by looking at the notes.  Unfortunately we can’t show you at this moment what’s the context around this term, but if you’re using this not in a demo mode you should be—you would be able to see the actual document text.  

If we go back to the overview for a second you can see that we retrieved 3,400 patients.  And assuming that we are—so we’re thinking, okay, there should be more patients in this dataset that have diarrhea.  So there are other terms related to diarrhea and here we can click on the query recommendation where we will provide a list of terms that based on the methods I described earlier using query expansion methods, some are synonyms, some are because we found their related terms by using a technique called topic modeling to analyzing those documents, or looking at semantic relations that’s documented in the literature. 

Here you can see the first one is diarrhea, but it’s a styling variant.  So let’s assume we’ll check that.  The second one is loose stools.  That’s actually a synonym.  That sounds relevant too. And the user actually can go down and look at various type of terms.  So now we’ll just select a couple terms, well let’s say like also the runny stools here.  And the term above that actually we looked it up.  It’s a southern variation of describing diarrhea.  So we include that.  

And now assuming that we’re satisfied with all the terms here we’ll click okay.  And now we expanded our search and our criteria with the search.  And we got 200 more patients now through this search. 

And if we look at the document detail we can see evidence of this.  This one still most of it is diarrhea.  Oh we see a loose stool here.  And we can go to a different document.  

Here it seems like this document has loose stools and also diarrhea.  And this document actually only has the term loose stools, so by looking at this researcher or user can actually refine the search criteria. For example in our use case the researcher who’s searching for this after reviewing all the reports said it seems most of the diarrhea appearances are negative, but loose stools, runny stools, these types of terms are almost never negative.  So he actually sees—he can actually use those terms to identify the diarrhea better and reduce the number of false positives.  And these are all free text search.  You can easily combine that with ICD code.  In this case we are actually interested in diarrhea in patients with C. diff.  So you can match on that term and do a search.

Again here we see a change in the number of patients returned.  Clearly not all the patients who have diarrhea have C. diff.  So the number of patients actually dropped a little bit from when we were just doing the free text search.  We just also wanted to show you if—you can actually add other criteria.  You can restrict to your document type, age, gender.  In some studies people were focusing on let’s say a women’s health study you clearly would only want women.  And you can also focus on a particular county or a state if you are doing a local study, for example, we’re here in Utah, we may only care about, we may only want to know what happens within a particular state.  And also you can look at—you can also restrict to the medication, the procedure, whether the patient is deceased or not. 

So we’re going to show you so just to give you a variety of the things we can do, another study that we worked with was searching for suicide attempts.  Here was the initial search and there are 270 patients that meet the criteria.  

Let’s say we extended this query and see if we can capture more patients this way.  Here we have suicide attempts.  Since we are doing a string matching we would have already captured that.  And it looks like the attempted suicide is pretty good. Suicide gesture looks like a right term.  And if people want to further expand the query they could also look at maybe and in one case we actually also care about suicide ideation and so on and so forth, but here we’ll just stay with attempted suicide and the suicide gesture. 

As you can see the number of patients now increased to 365.  And that’s actually quite a big increase, over thirty percent increase.  And if we look at the details here we could see patients with suicide attempts, attempted suicide.  We can randomly look at some of the notes.  In this note it seems like only documented as attempted suicide.  So that’s—it’s important sometimes to add additional terms.  And so that’s one patient, and this patient already deceased.  And if we look at his notes we can see here it is a suicide gesture.  

In this particular study I’ve been working with psychiatrists.  We were told particularly interested in suicide attempts or suicide ideation that documented in the mental health notes or psychology notes, because those tend to be more reliable.  So we can actually add that to the query.  So we can better restrict it to mental health.  

And we’ll do a search.  And now the number of matching patients drastically reduced and in the detailed documents we can see all the notes and we’re returning to you have is mental health, typed mental health in the title.  And apparently there are a number of these different types.  And here we have a female patient.  And so we here see the attempted suicide here as well.  

Another interesting aspect of this is when you narrow down by the criteria sometimes you increase the specificity, maybe drop the sensitivity.  So for each study we expect the users to adjust to this criteria based on their individual needs.  And another point we want to make is that the reason why sometimes you will do ICD search, or you do text search alone or in some cases you want to combine them is that the information even though it seems like suicide attempt does have a ICD code, or homelessness has an ICD code.  Some of these are not well documented.  

For example, here we can type in the ICD code for this and so we’ll show you an example to show that, well here this is the code for suicide ideation.  Let’s see.  What’s the overlap between people with suicide attempt?  How many patients have documented suicide ideation and then have a suicide attempt?  So as you can see the number actually changed oh those ICD codes.  

So we have many documents with matching words, but only 359 patients that have both suicide ideation ICD code and suicide, have made a suicide attempt. And if we just look for suicide ideation here with 47 patients, if we add ICD code, yes, this looks like majority of those patients do not have an ICD code we now drop to less than ten matching patients.  

So in this demo for safety reasons we’re not showing any data when we have less than ten matching patients, but you can get the idea that not everybody that had a suicide ideation had ICD code.  That is why we dropped from forty-seven to less than ten patients. That is one benefit of combining the search of ICD codes and the—if you want better recall add the patients, add the free text search to the mix would help. 

We can also do simple Boolean searches here.  And one example we can show you is for example if we simply say diarrhea and fever, because if we put in different text fields we will treat it as or, but if you say diarrhea and fever we can actually capture that as well.  So here you can see we get 994 returns.  Let’s take a look at their notes.  Fever, diarrhea and here the anti-coagulation notes, fever, diarrhea, let’s see this one hematology and oncology note.  We have fever and diarrhea.  Here we have fever, diarrhea also to yet a different type of notes. 

We have diarrhea in this one.  Uh oh, where’s the fever?  Hm, we should have it in here.  So here is the notes.  Oh so we’re showing the patient’s note.  So the patient has a note that has fever and diarrhea.  So not always every single note has fever and diarrhea, but at least one note has to have fever and diarrhea.  So you can restrict your query that way, but here you see fever.  You see diarrhea in these patients.  We rank these—we rank the return results.  You can see next you get more patients as well.  Here you get another list of patients.  

We rank the results by looking at the TFIDF scores.  And then we aggregate them on the patient level.  So patients, if a patient has more notes that are relevant they will be ranked higher.  That is actually very intuitive in clinical setting if you see a patient with three notes all saying the person has fever and diarrhea.  That’s more convincing than just seeing one note with fever and diarrhea.  As you can see as we move down they have fewer notes than in the previous page.  And so here you can see the patient distribution and their age distribution and so on and so forth and in the same way.  

We’re going to show you one last query.  We’re going to do the cavernous malformation, which is a rare disease.  And sometimes the condition is not so common.  That is where you have the advantage of a nationwide dataset.  Also the advantage is you can see where the patients are.  This condition actually was suspected oh we had less than ten matching.  We also have the abbreviation.  We can search for the abbreviation, yeah.  

So in that case you can see if the patients are clustered in one area, which suggests maybe there is something environmental factor in the play.  And here when we used the term CM we actually got a lot of return.  On the other hand this is a little bit suspicious because CM is an ambiguous term.  And to reduce this problem actually you could have further used document type to further narrow down.  But in our case we actually wanted to narrow it down to had CT or had x-ray.  That will then again drop the number of people.  

And I’m going to stop the demo here.  We could show you many more examples, but I want to leave some time for questions.  And so how about we open it up for questions?  

Moderator:  Great.  Thank you very much.  We do have several questions that have come in so we’ll get right to it.  The first question, is it possible to use Voogo if you have operation VINCI approval that is not IRB approval datasets, IRB approved datasets?

Dr. Qing Zeng-Treitler:  I assume it is.  And we—if you have access to the dataset then we can link to it.  That’s the gist of the regulation.  I assume operation people can access, but so far we only provided access to people with IRB approval.  

Moderator:  Okay.  Thank you for that reply.  Can you explain the differences in methodologies baseline, versus synonym, versus topic model versus SemRep?  

Dr. Qing Zeng-Treitler:  Yes, sure.  So when we are using baseline—we showed you two baselines.  One is just using the sum of TFIDF, which actually is fairly common.  Some people use cosigns.  In our experiments in some cases the sum does better.  In some cases the cosign of TFIDF, which is term frequency inverse document frequency.  And actually in both cases those things work.  It’s fairly common.  Lucene has—Lucene actually uses their own version of the TFIDF and cosign ranking.  So it’s a variation of the ranking of the results.  But roughly our baseline is this either we use the sum of TFIDF or in the last study we used the baseline Lucene. 

Lucene is a very popular search engine.  And we use—when we used baseline we didn’t do any query extensions.  So we really tried three different types of query extensions.  One is synonym expansions, which is using UNOS, which is large vocabulary resource.  It contains SnoMed, ICD, about a hundred other vocabularies.  So we look at what those vocabularies declare as synonyms and we add those.  We also add—actually I need to take that back.  We actually don’t add all the vocabulary synonyms.  We focus on a few vocabularies because we utilize for mapping a term to a concept, but then we only add the synonyms from a few selected vocabularies because in our experiment we found many synonyms from certain vocabularies are actually not what most people consider synonyms.  

So just to give you an example like congestive heart failure, the synonym is CHF and a number of other things.  So we can easily add quite a few of those.  So that’s a synonym expansion.  The topic model we use a technique called latent Dirichlet allocation.  It basically discovers semantically related—it groups semantically-related words into a topic.  And when we use the topic model based expansion we look at the weight.  We look at what terms co-occur within the topic.  And we calculate the weighted sum and rank the co-occurring terms in the same topic.  And then we take the top ten ranked terms to do expansion in our experiment.  

And with the SemRep, SemRep is a program.  It’s fundamentally a natural language processing program.  And it’s designed to process and analyze biomedical literature and extract semantically-related predications.  Basically we’re saying a [triz] b, a is a type of b, that kind of relations.  So we take these.  The version we’ve got has millions of these predications that were extracted by SemRep.  We take those those related terms.  Again we aggregate the data.  We rank them by frequency and we take the top ten related terms.  

So those are three different types of expansion methods we have experimented with.  And then we also experimented in several different ways of combining the results because as you can see on here our screen, when we provide a query expansion, let’s say we take a term, CHF.  We don’t want to provide you with three different lists.  

And so we try to combine them into a sensible list.  And also here we initially did automated recommendations, but one thing we learned from our own evaluations and from talking to others who work in the field is the user is ultimately the authority on what terms are relevant.  Automated expansion works only to a certain extent.  So here we took those terms that we used in our experiment to automatically expand the query and provide it as a recommendation.  And the user can select whatever terms they see as relevant.  

Let’s say we can choose congenital heart failure or weak heart, just for the fun of it or cardiac insufficiency, myocardial failure.  And then we can expand the query selectively.  So what the tool uses is the methodology I described in the evaluation, but not necessarily in—we’re not.  We’re definitely not automatically defending the query.  People can choose what they want to expand on.  

Moderator:  Thank you for that reply.  The next question we have, for evaluation three what types of queries were performed?  How were the performance measures determined?  Is Lucene an existing method for text searching?  Where can we find out more about Lucene?

Dr. Qing Zeng-Treitler:  Oh Lucene is—if you just Google Lucene this is a—Lucene is quite a popular methodology. And the only thing is you do need to run the Lucene indexer on whatever text collection you have.  It’s actually a very powerful, very versatile—it’s a very good open-source tool. 

It is not customized for clinical data per se, so when we use Lucene we have to do—we have to work on it a little bit, add certain software, and so and so forth, and link it to the structured data and such.  But Lucene is a very, very—it’s Apache software and it’s a very good, very good search engine.  You can download it open source from multiple places.  Sorry, I forgot the first part of the question. 

Moderator:  The first part was for evaluation number three what types of queries were performed and how were the performance measures determined?

Dr. Qing Zeng-Treitler:  So those queries actually devised by Dr. Bill Hersh’s group in Oregon.  And he based on—his team based it on the AHRQ’s comparative effectiveness study needs.  So essentially they’re kind of cohort definition criteria, so queries such like children with XYZ condition, male over sixty-five admitted to the emergency room and discharged with a particular medication, and so on and so forth.  

So they took these criteria and the reference standard is slightly complicated in because they are only doing partial judgments.  So they focused on the top returns, top 1,000 returns, but they also randomly sampled the things that were not in the top returns to make the measure more sort of closer to if you have somebody looked over the 100,000 datasets.

Having said that, it’s never as good as you can have somebody who will review all the results, the full datasets.  Then you can get a real sense on the precision recall.  In other words we often use in precision recall is people use in the search community and sensitivity and specificity, which the healthcare researchers care a lot about.  So it’s approximation.  It’s not exact of the full human reference standard.  

Moderator:  Thank you for that reply.  Next question, pardon me.  If we have an existing approved VINCI project, what do we need to do to get started?

Dr. Qing Zeng-Treitler:  I think you can get in contact with us.  And assuming that you have—if this data is already saved in the database, have indexed, all it takes is to link it to our system.  If your dataset is in the form of text we could potentially run Lucene indexer on those records and link it to our system.  And, Doug, you wanted to add?

Doug Redd:  Yeah.  We just basically need to know where your data is and what format it’s in, so if it’s already in a SQL server that VINCI put it in then you just basically let us know where your tables are and we can point it to your tables, and then I give you a copy in your project folder that you can run on those tables.  

Moderator:  Thank you. 

Doug Redd:  Or we can do a Lucene or something like that if you have it in a different format.  

Moderator:  Great.  Thank you.  The next question, can you tag the notes that are returned by various search criteria and select notes where there are logical intersects or exclusions?

Dr. Qing Zeng-Treitler:  No. We don’t have that capability right now.  And actually I don’t quite understand the logic that the last part of the question, but we are one function that I didn’t, I forgot to mention is we’re adding in is the relevant feedback.  So we’re working on testing out of the function or saying more like this, so like Amazon or Google.  You can say I want more documents like this.  And also on the demo version we disabled the save function, but if it’s on the data that’ss IRB approved you can save the search results to further analyze.  

Moderator:  Thank you.  Can you explain how additional recommended, how the additional recommended terms list was generated?

Dr. Qing Zeng-Treitler:  So right now we generated it by combining our results from three, is it three or four?  

Doug Redd:  It’s four now.  

Dr. Qing Zeng-Treitler:  Four actually, four different sources actually.  We did—I mentioned earlier we did the synonyms.  And we get a long list of synonyms and sometimes hundreds of those synonyms.  And we also look at again we go to the—UMRS as our knowledge source.  We look at through ontologically, ontological knowledge in there the terms that declared related by those source vocabularies like SnoMed has, not only hierarchical relationships saying a is a b, but also they are also having more in-depth relationships like a symptom may have the location of a body part.  And a medication may be treating a particular disease and so on.  

So that’s another source of we get another list back from there.  And it’s a [inaudible] list based on occurrence frequency, although we do give more weight to the children and the children relationships.  

And we have the topic modeling gets you the related terms based on the context in the notes that we apply the method to.  And the fourth one is the SemRep that gets you to—gets a list from the biomedical literature.  And we take these four weighted lists and we basically combine them by adding—and normalize them, and add them up and look at the top frequency ones and show you the recommendations.  

Moderator:  Thank you for that response.  The next question: can you search medications, for example methotrexate, methotrexate percentage, I’m sorry, methotrex percentage, methotor percentage and PX and other forms of methotrexate?

Dr. Qing Zeng-Treitler:  Well we can search if we know what they are.  So you can definitely go to a drug and then type in—I’m not sure how methotrexate is spelled, but you can—let’s say we type in something I know, statin.  You can just search and it will return, oh there’s less than ten.  So we can just directly search for drug, right, statin.  So you can just search like what I’m showing you by typing in the name of the medication.  

Moderator:  Excellent.  Thank you.  And we’re down to our last two questions.  Does the plus sign in the search indicate the and or the or?  

Dr. Qing Zeng-Treitler:  That is if it’s within the same field, like let’s say you type—it’s interpreted as or, if right now it’s like we put in a plus it’s the different fields of data is interpreted as and.  But if let’s say you put in another drug name that would interpret it as or.  

Moderator:  Thank you.  Our next question, does Voogo support logic expression such as nor or or?

Dr. Qing Zeng-Treitler:  So or is actually we were talking about the plus.  So that’s supporting.  We don’t support nor at this point.  

Doug Redd:  Right now we’ve just defaulted to if it’s two terms for the same field then we regard it as an or, but if they’re different fields then we regard it as an and.  And that’s just a default.  And if you want to have an and within the same field then you can actually type it in and have some.  And well we plan to expand Booleans and give more control for those who want that in the future.  

Moderator:  Thank you.  We do have one more question that came in.  Is it possible to search on a term like SI and not received all words which have SI in them?

Doug Redd:  Yes.  So if you just do SI then it will look for just the word SI, not words that contain an S and an I in it.  So that’s the default behavior right now.  

Moderator:  Thank you.  And we do have another question that has come in.  Can you save, I’m sorry, can you save patient ID/dates for query results into a database?  

Dr. Qing Zeng-Treitler:  We are right now just saving the document and the patient ID I guess, yeah. 

Doug Redd:  The patient ID to a text file right now.  

Dr. Qing Zeng-Treitler:  Yeah.  So we wouldn’t—we don’t—we haven’t developed a very elaborate way of saving the data, although that’s actually not very difficult.  And but right now we have a very simple save.  We are just saving the document and the patient ID.  

Moderator:  Thank you very much, one more question.  What is the unique advantage of this tool as contrasted with other text mining tools?  

Dr. Qing Zeng-Treitler:  Well this is not particularly a text mining tool.  We intend it to be a simple tool.  We actually look at it as a gateway to in depth natural language processing text mining where you would have to get—you would have to have multiple modules to process text or you use tools like you need annotated data to mine the text and so on.  So we intend it really as—we really wanted this to do two things.  

One is to help you do some exploration and because when we work with people who want to work with text data, the first question often is I want to take a look at the data.  And actually most natural language processing tool does not make it easy for you to look at a large corpus of data.  It just is very cumbersome, there intend to process data, but not reveal the data.  

Another part of this is we make it easier to link to the structured data.  So that’s another issue we often encounter when people come to us for text-related questions.  At some point they always say, oh I want patients with this but unplug that, with this high fever, or on statin and so on and so forth.  

And once again with most natural language processing text mining tools this is very intensive work with text and focusing on text.  And somewhere down the line we just lose that easy connection with linking it to the structured data to take a look at what’s the distribution in terms of age?  What’s the population look like?  What medication are they on, what procedures they have received, so that you can do both structured and text data.  

And we actually intend to, as I mentioned earlier, to have this, to have Voogo the ability to search for annotated data, so machine annotated as the NLP process results.  That will—that actually is one thing that we’re kind of missing right now.  

Moderator:  And as a follow-up to that question, the person asks so would you then use this in tandem with a text mining tool?

Dr. Qing Zeng-Treitler:  Yes, yeah.  We would highly recommend people to play with this, select the text corpus, and then feed into text mining tools.  That actually will—in our experience that actually reduces the amount of annotation time you need and make it a lot easier to work on a smaller corpus rather than the whole of the population.  

Moderator:  Thank you.  That is the last of our questions.  Would either of you like to give any final comments?  

Dr. Qing Zeng-Treitler:  Well we just want to acknowledge the funding source.  So this project is funded by both CHIR and VINCI.  Also we’re going to provide our email for you if you have further questions and want to give us additional feedback we would love to hear from you.  

Moderator:  Excellent.  Well thank you very much for sharing your expertise with the field.  And I will remind our attendees as they exit today’s session please wait just a moment and a survey will pop up on your screen.  Please do complete that.  It lets us know what further topics you’d be interested in viewing a cyberseminar on.  

So thank you very much once again to our presenters and audience.  And have a wonderful rest of the day.  

**[End of Recording]**

1 of 12

12 of 12