**Transcript of Cyberseminar**

**Timely Topics of Interest**

**Automatically Extracting Sentences from Medline Citations to Support Clinicians' Information Needs**

**Presenter: Siddhartha Jonnalagadda, PhD**

**1/22/2013**

**Automatically Extracting Sentences from Medline Citations to Support Clinicians' Information Needs**

**1/22/2013**

Moderator:Today’s presenter is Siddhartha Jonnalagadda. I know I am sorry, I probably butchered that pretty badly; he also goes by Sid. He is currently at the Mayo Clinic in Rochester, Minnesota in the department of Health Sciences Research, in the division of Biomedical Statistics and Informatics. And with that, I am going to turn things over to Sid.

Sid:Okay, are you able to see the screen?

Moderator:You need to click on that popup to show my screen.

Sid:Okay, that takes like ten seconds. I click.

Moderator:There, we can see it right now, yes. And if you put it in the slide show mode, perfect, you are good. 

Sid:So hello everyone. My name is Siddhartha and this is a joint work with collaboration from – I will introduce the team later, but this is a joint public collaboration from Utah and UMC and National Library of Medicine too. So to begin with, we have some poll questions to better understand the audience. 

Moderator:And we will give that just a few seconds to let people respond. We are looking for your  primary role in the VA. You can click multiple responses here if you have multiple roles, so feel free to click through as many as you need to there. And there are your results.

Sid:Oh nice. 

Moderator:And are you ready to move onto the next one?

Sid:We can put on the next poll, yes. 

Moderator:Okay, there we go. And here we are looking for which best describes your research experience; we are just looking for one response here. And here are your results.

Sid:So just a couple more questions.

Moderator:Okay, here we go, here is your third question, do you know about information needs at the point of care?

Sid:Nice.

Moderator:And here are your results. And one last poll question here. Now we are wondering if you will be willing to participate or help in recruiting for an international survey on information needs. And if your answer is yes, we would actually, if you could type your name and email address into the Q and A screen, and I can get those collected and sent over to Sid as soon as today’s session gets finished. We would very much appreciate that. 

Sid:Nice. 

Moderator:And there are your results.

Sid:Okay, so as I said, can everybody hear me?

Moderator:We can hear you, I just need you to click on that share my screen, and you are back and live.

Sid:Okay, so this is our team, so it is Hongfang who is at the last and Sid, which is me is at the first, I’m from the Mayo Clinic, Guilherme Del Fiol who is on line with us, he is from Utah. Richard Medlin and Javed Mostafa who is from the second from the end, they are from UNC, and Marcelo Fiszman and Charlene Weir, are from Utah and [inaudible], respectively. The work is about how to help clinicians better handle information needs. So there was a very well connected study by [inaudible] which showed that out every three patients that was seen at the primary care physician appointment, they have two questions. And out of them 70% were unanswered. And subsequent studies also states the same has been consistent and over the last few years especially, the amount of resources and the online sources have increased, and it actually helps in that the answer is somewhere out there. But it also complicates the problem because out of so many millions of potential documents, where to find that precise nugget of knowledge, or what we call it as knowledge. Somebody for that particular situation, is somewhat tough, and if you consider the pinnacle workflow, which is usually busy, the doctors are very caring but they are also very busy. 

So one approach we are exploring is to automatically extract relevant sentences from those multiple documents, summarize them, and present them as a knowledge summary for that particular situation. So we first want to explore what are the possible questions that could arise at the point of care. John Ely, et al from the University of Iowa did a survey a few years back and it was for primary care physicians in the Iowa network, and they found that the treatment kind of questions and diagnosis kind of questions are more prominent.  The treatment – I mean what drugs to give it, what stage of efficacy prevention and diagnosis mean how do we – given this finding what is the condition.

Someone I think has to mute their phone.

Moderator:Yes, it looks like that is Javed.

Sid:That is fine, okay. So right away, Guilherme, Javed, feel free to pitch in if you would like to add something during the presentation. So considering that treatment is the question that is most – that is found to be most needed in this situation. So if we focused on treatment kind of questions, and so before going into, because there was some audience who said they are not familiar with information need and most of the audience, the significant amount of the audience are clinicians and clinical researchers. I want to give a brief background of how this is dealt with in the computer science domain, and even informatics domain, so they initially, am I supposed to see the questions Heidi, during the presentation or can I ignore them.

Moderator: Oh that was just Javed telling us that he muted himself, you can ignore those, I am handling them, nothing to worry about.

Sid:Sure, sounds good. So the most basic kind of question answering, or information need gathering, if you might call, is Google which provides a list of documents for a given answer and somewhat more sophisticated is a Wolfram/Alpha by the same people who produced MatLab. So if you ask some factual queries, like what is the capital of United States, or perhaps or any other question, what is the currency of a particular country, it has structured information and it computes the answer and prints it out. 

Watson is a hybrid of both I would say. It starts with documents, it re-computes structured data from the documents, using its heavy infrastructure and natural language processing methods, and then understands the question and finds the answer from the pre-computed structured database that actually uses documents underneath. So the approach we are going present is going to be something similar to that. And coming to how it is supplied in the medical domain, we see that Watson is taking steps to include that in the medical space to – I am sure you are familiar, most of you are familiar with the Jeopardy competition that brought IBM’s Watson machine into the limelight. So there is a similar competition for clinicians called Doctor’s Dilemma that is connected by American College of Physicians, that they give a very verbose description with symptoms and the doctors are supposed to guess what is herpes or what is some other disease. So that is one of a very popular system these days that is being looked on. And there are also other systems like AskHermes which is online and Medline Plus, again it is more document search, hosted by National Library of Medicine. At Mayo Clinic we have systems for AskMayoExpert with pre-written FAQs so you cannot answer all your questions but it has some FAQs so if your question matches an FAQ it will give you an answer. And the cut in system we are going to describe as a MedKS system. We think it is pretty good too, and there are other systems by Howard and NLM called MiPACQ and Infobot. So that covers the major work done in medical question answering or information need gathering. 

And this is the overall picture those systems are trying to solve. Given a question in English, how do you understand the query, get the relevant abstracts, and do whatever is needed for summarizing it and just give it beta graph. So for our particular research project that was published in JAMIA, we focus only on Medline abstracts. In the future, we would go to the other sources. So getting to the core of the presentation, I will explain this in more detail, I will explain that each individual step in more detail, but in the first step is query processing. Where given a particular disease name or even a particular question name, we first understand what are the MESH terms involved in it, and what are all the [inaudible] terms involved in it. And know there are many librarians on the call so I am sure you are familiar, all the librarians, and even most of us are familiar with using MedLine, and within MedLine there is a particular feature called [inaudible]. That is a web API, web application interface, for querying relevant MedLine abstracts for a particular citation. So with that particular, we create a query and get the relevant PubMed ID’s which correspond to each document. And within each PubMed ID, we look for predications of our treatment type. So we look for all the treatments that treat or act as a remedy for the medical situations in the query, and once we get the predications that get us sentence, and then the sentences are summarized. 

So I will go into a little more detail on how it is done. So the next few slides are going to be a little more technical in terms of how we – in terms of presenting all the natural language processing methods we used. So for query processing first we do tokenization, which means splitting the sentence into individual tokens. And then we do lexical normalization, which means converting all the parts, all the different tenses and different modalities or modes of like singular, plural to all singular. And it also includes converting some of the words to their more frequent used words. So for that, we use a package called Lexical Radiant Group, in short LRG for web national group medicine. And then we use a method for dictionary lookup. The particular method we are using is, it is called a [inaudible] method. It is a pre-based data structure where each of the – which allows us to store the entire UMLS in the RAM, in the memory, so having stored the entire UMLS in memory performs very quick lookups and it gives the UMLS – we call it CUI, concept unique identifier, and from the UMLS CUI, we get the history of the entire concept, like what is its corresponding MESH term, what is its canonical term, and what are its semantic groups like whether it is a treatment or a disorder. Our particular focus is treatment and disorder, so we group them into either, we group all the concepts into treatment or we group all the concepts into disorder. We also get the MESH terms for the next stage. 

So this is just for the programmers and computer scientists, but others I will give you a simplistic picture. So the information retrieval strategy is primarily when you have those MESH terms, which I mentioned here, you use those MESH terms to get the relevant PubMed IDs. So why are we doing this, we are doing this because we cannot look at the entire PubMed. Entire PubMed contains more than 20,000,000 citations and more than half of them have a full abstract, so it is not practical to query the entire PubMed, so we want to first focus on the documents that contain these concepts. So some of you who are informaticians might be familiar with some work done on clinical queries. It is work done by Hanes, et al, from the National Library of Medicine, where they argue that systematic reviews and – so when you go to Medline, you can see the MESH terms for each abstract. In addition to MESH terms for each abstract, you could also see whether it is a systematic review or there are certain clinical filters such as where it is a therapy-related article. If it is therapy-related, does it have a narrow focus, is it really focused or is really broad. 

So we first get all the systematic reviews and all the focused therapies that are related to the query. And that is this line. And if they are not sufficient, then we go and also look for broad therapies, and if they are not sufficient when we look for everything, all the documents that contain the query words. So now we’ve got all the PubMed abstracts. So once we get all the PubMed abstracts, we use a database of semantic predications provided by the National Library of Medicine. So the brief background is, it is called [inaudible], and what [inaudible] is it contains all the [inaudible]. So one example is you can see in this table, so each of these corresponds to a subject. So [inaudible] or vitamin E are all subjects and Alzheimer disease is the object. So those [inaudible] are primarily [inaudible] treats Alzheimer disease, etc. So for this particular example, we found out a [inaudible] treats Alzheimer’s disease 45 times. And similarly other things. So with a natural language processing method so there could be some mistakes and all but we try our best to restrict process of getting the subjects for those particular objects. So the way we do it is – I might remind you that for the question we typed or the query we use, in this case we get all the treatments and disorders. All the disorders would be the objects of those predicates and all the treatments would be the subjects of those predicates. So when someone already mentions a treatment, that means they are looking for abstracts that already mentions – say vitamin E how it is useful for treating Alzheimer’s disease. 

So we do enter all that is required, all the provided treatment information and disorder information, and this is a precise flow chart that you can look at the general publication, which is available to most of you, to get the precise algorithm. But the idea is to get as many predicates as possible that is suitable for that situation. And once we get all the predicates, each – so one way to display those results is just this graph, just this table where list of treatments and number of predicates. But those predicates, that list does not really have any context behind them, so we would also like to provide some sentences. So those sentences are the sentences which contain those predicates. So as a housekeeping exercise, within most sections, we will come back to this, most of the PubMed abstracts, MedLine abstracts, have section names available, 5% to 7% of them do not even have section names. But when we have them, we exclude the object view section and the selection idea, selection method section because they do not contain any historically proven information typically, or even the conclusions. So that way we focus on the important sections like background or study conclusions. So once we guard all the proven sentences, we map all the sentences into a graph where each sentence is a vertex, and all the witnesses are connected. So it is basically a clique if you are familiar with grafting. And the edge weight of each pair of sentences is housed in the middle of those sentences aside. 

So for measuring the similarity we see – we use a common metric – we use a metric that is more popular in the NLP domain called consigned word similarity where we map each sentence into a directory presentation and we compute the cosine between those two vectors. That is edge weight. So using this approach we – this is an approach – TextRank is an approach similar to PageRank and this helps us to not only take into account the similarity between the query and the sentence. But just as PageRank does, it takes into account the overall structure of the graph, and if some sentence is similar to very huge number of sentences, then that sentence would be highlighted more than some sentence which is just similar to the query but it is really isolated. So once we have done that, we are in a position to provide a summary. So as we have these, we have a lot of clinicians and clinical informatics researchers on our team, and the whole goal was not just to create a system, but to create a system that could be useful for clinicians. So we actually tested it on two real conditions, Alzheimer's and depression. And it was just to say that these topics were selected after the system was completely developed. And while developing we used clinical questions from the survey that I mentioned previously by Ely, et al, from Iowa. 

So these are the idea we are using for assessing the sentence. So for the sentences and for the abstracts, we see how much they are relevant to the topic, do they explain how the condition should be treated. And that is the broadest definition of relevance we are taking. And that is actually the scope of the system is to – the scope of the system is find relevance and abstracts. But additionally we wanted to study how by default the sentences are. I mean the nature of the sentences as far as how much conclusive they are. If they are just – maybe if they describe the current state of knowledge or if they describe the study conclusion, that means they are very conclusive. If they are just a hypothesis or a speculation, or even conclusion which is negative, they are not conclusive sentences. And comparative sentences are those which compare and contrast two different treatment approaches and that could be useful when a clinician is faced with a situation where we have to decide one versus other. 

Contextually-constrained is, but it provides a lot of context on what population is being considered with demographics et al and the stage of the disease and so on for the precise context, and they could be useful, focus on specific treatment for a particular clinician. So having said that, is everything okay with the audio?

Moderator:Yes, everything is okay with the audio, yes.

Sid:Okay, just wanted to check. So as I said, so topic-relevant of the main criteria and conclusive sentences, comparative sentences, and contextually constrained sentences of the sub criteria. So once we have done – so these are some examples. So the first sentence, for example is not at all topic relevant because it does not really have a specific use or it does not really mention a particular disease, a particular treatment regime. And similarly we have examples for conclusive and – so the second sentence is an example of conclusive and contextually-constrained. It says, from the conclusion sentence and it is a marginal evidence, and talks about depression in adolescents, which refers to it being constrained. The third sentence is a comparative sentence, comparing a particular drug called Escitalopram with other antidepressant agents. The fourth one is also, it is both conclusive and it is also comparative, conclusive because there is no strong evidence but it is still comparative. There are also the examples – the fifth one is an example of a sentence not being relevant to a topic. 

So two of the clinicians in our team, with an M.D. have studied different – the results for both the cases, they agree for the most part as described in a couple of scores. So as you can see, they feel that most abstracts are topically relevant and the sentences are even so. And but the number of conclusive sentences is only about one-third and the number of comparative sentences decreases further. And there is a discrepancy between depression and Alzheimer’s disease which points perhaps to research being done, perhaps there are more comparative studies in depression as opposed to Alzheimer’s disease. And similarly perhaps there are more contextually constrained situations in depression because it is a disease that could come across multiple age groups, and Alzheimer’s is a particular age group. So those were our ratings. As you can see, all our rates increased with the number of relevant sentences are huge, and it also points to future direction where we have to look more closely into kind of focusing on conclusive sentences and comparative sentences and contextually constrained sentences when the users want us to. 

So this is some of the results we noted in the publication. So most of the non-relevant sentences were actually about diagnosis or prevention. They are still related to that topic but they are not related to the treatment topic. And actually there are very few of them, that is about four to five percent, and we could primarily attribute it to some of the mistakes done by the natural language processing method. And unfortunately, because all of the sentences are relevant, and most of the sentences are relevant, to take strength from all of the sentences seem to have a very high rank or a very similar rank when it comes to the text strength probability. So in that sense, the ranking being provided is a significant discrimination between relevant and non-relevant sentences. And because it is not designed in that way, it was not able to discriminate between conclusive and non-conclusive sentences. So the best indicator we got for conclusive sentences were as anyone could guess, it was the sentences that was close to the end, they tend to be really conclusive. We have a very good p-value for that. 

And also other things that came out of the study, prominently being – the first three I mentioned them already, but sentences with treatment and comparative predications, such as treatment A higher than treatment B may be more likely to be conclusive sentences. So it seems to be a phenomenon that when two treatments are mentioned in the same sentence, and they are referred to a particular medical condition, it is probably, it is also conclusive. And most of the sentences achieved by the system are actually one treatment of a particular condition. Very few of them compared multiple treatment alternatives. And as I mentioned before there were differences in the distribution of contextual constraints for the two diseases we are considering. There are several limitations for our study because it is a prototype. The first question is whether this work could be generalized to other diseases, and we could only do that when the system is tested in multiple disease areas. And we do not have an idea, we have a guess estimate based on the performance of semantic Medline. The NLB machine behind semantic Medline database called SemRep, it is about 70% to 90% accurate when it comes to recall or sensitivity how much information it leaves out versus how much information it gets. That is pretty high, but still there are some issues with recall where we might lose some documents, and the information retrieval stage is stage two, and we just have to decide how many sentences we have to display so depending on the numbers of sentences we display we lose some [inaudible] there. And so we still need a lot of study for estimating the recall. And we also need to focus on the relevancy and give it tighter definition that incorporates how conclusive it is and basically how much clinically actionable the definition is. So those were some of the limitations and future work, but overall we were able to have a prototype system that retrieved a high percentage of topically-relevant sentences. It also points to our future work on what could be done further, but it seems to be a feasible approach to try further away, or what our goal being provide context-specific knowledge summaries at the point of care, help better clinical decision making.       

So those are the main points I wanted to cover in the talk. And I would like to take questions in awhile but I also wanted to mention that we have a couple of grants that we are working that aims at similar study where [inaudible] is a create-element award, which is a similar work we are working, where we are trying to... Given that the next focus is to expand from MedLine to full-text journals because not always information is present in MedLine abstract. And if we actually did a study, we actually did look at Up to Date, which is an evidence-based resource. And most of the evidence they cite comes from full text by most being about 100 differences we saw, about 60% of them or more is the way the information is presented in the full text. So we first have, to go back to the full text, we have the product as which full text to consider. So that is aim one. And the second aim, we need to, as we saw, the number of sentences is huge. So one way to restrict the number of sentences and focus is to find good citations. And once we get good citations, we could then focus on finding good sentences on them. 

So that – I am just saying that we all decided to work on that direction. And we want to acknowledge JAMIA and the reviewers for the encouraging comments and they were the funding sources and the support we got from National Library of Medicine Process and Semantics by Debbie. And I think we are finished there, so I leave the room for questions. I am sure we covered a lot of brick. We covered about information needs, natural language processing and some of the evaluation situations. So we actually have four of the co-authors, me, Guilherme Del Fiol, the second author, and then we also have Rich and Javed from UNC. So you can feel free to ask more questions and then also give us any feedback on how we can proceed further. Thanks a lot for the opportunity to share the research so far. 

Moderator:Great, thank you. We do have a couple pending questions but I did also want to let the audience know to submit questions you can use the Q and A screen that is on the dashboard, on the right hand side of your monitor. If it is has collapsed against the side of your screen, just click on the orange arrow at the upper right hand corner of your screen, that will open and collapse it against the side of your monitor. This is a great opportunity to submit questions, we have some time here, so please send those in. 

The first question that we have here, are the terms searched for as concept equals disorder and/or treatment only being matched to UMSL?

Sid:So let me paraphrase the question and we can see if that is understanding. So the query, this is just an example, for the case study where the query for Alzheimer's disease and depression. But the system could take any question on the planet such as any question in the clinical query database that could be found in National Library of Medicine. So the query processing stage, at that stage we map all – we find all the UMLS terms and for the subsequent stage, because we are focusing on treatments, the subsequent stage, stages two, three, and four are currently customized to only finding treatment related answers and citations. So we only focus on treatment semantic groups and disorder semantic groups. I do not know if that answers the question.

Moderator:The questioner can send in a clarification if they need to, I am not sure, so we will wait and see if we hear back from them.

Sid:Okay.

Moderator:The next question that I have here, did you examine how many of the abstracts used a structured format, presumably identifying conclusive sentences as relatively straightforward in a structured abstract with a section labeled conclusions.

Sid:We have not particularly examined, but there were a couple of studies where one of the co-authors from NLM is part of, [inaudible]. So I do not remember the exact numbers, but as I remember, it was about 90% previously and these days it’s getting to 95%. So it is not going to be a challenge in identifying conclusive sentences in the more recent articles. And that is perhaps the more recent articles are more important. But if we have to get knowledge from some old articles, we still need that section identifier. And the other thing is, in general we do not want to restrict to only MedLine abstracts. MedLine abstract is a beginning. So we are also looking to expand to full text journals as I described in the last but one slide. That for the grant that is funding that work and we are also looking at expanding to other resources. So as part of that, we might need – we also noted in the limitation section that is general, we might need, as opposed to just a heuristic [inaudible] the NLM abstracts or the end of summaries from the documents could be more conclusive. We might also use the Natural Language Processing Base, the linguistic juice and you combine them with some machine learning systems, but they will predict whether sentence is conclusive or not. For those from Natural Language Processing Domain, this is something similar to finding assertions at the drop of the statements made by clinicians. So I would say that is something, some space where a lot of work needs to be done. We just cannot rely on section identifiers, although it is applicable from MedLine abstracts, because there are other kinds of documents we own too.

Guilherme Del Fiol:This is Guilherme Del Fiol, I was just looking up the citation that Sid was mentioning about the – there was a study that looked at the percentage of MedLine citations that have structured abstract. And the prevalence of structured abstract has been increasing over time. In 1992, it was only 2%, but in 2005 it was 20%. But use MedLine overall, and we – in this system, we used the clinical query filter and also systematic reviews. And it is possible that the percentage for those kinds of studies is larger than 20%.

Sid:Okay, thanks for correcting me. 

Moderator:Great, thank you. And that actually is all of the questions that we have received so far. If anyone else does have a question, type that in quick, or Sid, I do not know if you or any of your other co-presenters have any concluding or additional remarks you would like to make?

Sid:Yes, I would certainly like if Guilherme or Javed or Rich would make any remarks. 

Unidentified Male:I just think one quick follow-up on future studies. Sid mentioned that the system in the present study was not able to find or a small percentage of the sentences in the present study were comparative, or compared two or more treatment alternatives. So we did a follow-up study trying to improve that – which could be an additional filter for a knowledge summary, that would focus on comparative studies. So that study has just been submitted for review in the Med Info Conference. 

Sid:Yes. 

Moderator:Great, we have gotten another question in here. Many publishers in progress.html are refusing permission to mine their text. Is this expected to change?

Sid:Frankly, there is more momentum towards lot of authors are trying to make their full text available because that increases accessibility to their research. So in many of the journals the authors could pay an additional fee which allows them to make their particular article open access, even though in general, the articles could be closed access. So JAMIA is an example from the informatics journals. That is one thing, and the second thing is the one year embargo period for funded research. So for the last few years, NIH requires that all the research studies that are funded by NIH, should go into PubMed Central, which is a repository of many full-text journals. So irrespective of whether the journal per se has an open access policy or a closed access policy, if the particular study describes the journal as public is supported by – it is funded by the National Institute of Health, after one year of publication date, they have to be deposit of PubMed Central. And even individual authors could do that through the NIH manuscript submission system. So we anticipate that PubMed Central resolves a lot of that issue. And I would say the publishers are also becoming more aware of it. They are giving options to the authors to let the authors choose if the article can be open access at an additional rate. And there are also some publishers such as Biomed Central which take a fee for all the articles but make them open access. So I definitely think there is a moment towards making the full text available more and more. 

Unidentified Male:Yes, I think, just to add to that comment. I can see, especially for the closed access journals, I can see publishers having some reservations regarding automatic text mining and summarization of articles. But the reality is that these knowledge summaries can be conserved as a carrot to readers who typically would not – would never access a specific article. And we extract sentences in these summaries. I can see many times clinicians having to go to the full text to get a little more context on the study that provided that kind of sentence. So my sense is that you would be silly for publishers not to allow this kind of processing. It could actually attract more readers to their journals than not requiring people to go to their journals. 

Moderator:Okay great, thank you. The next question that we have here. How do you envision this process integrating with that of systematic reviews? Systematic reviews give more attention to the quality of the extracted information, i.e. risk of bias, giving more emphasis to larger studies, randomized trials. Do you envision being able to do this through machine learning? Otherwise you may be extracting a mixture of reliable and less reliable conclusions without allowing user to differentiate between them. It seems largest value may be for rare questions not covered in systematic reviews. 

Sid:That is an excellent question. So I would answer that from two perspectives. One is within the perspective of this particular study focus. So firstly I wanted to mention that there is a recent study which says that – I do not remember the exact number but there are so many – I do not know if I have it here. No, I do not have it here, but there are quite a few – I think about 70 or 60 clinical trials every day, and the amount of systematic reviews being done to capture that knowledge is insufficient. Or it is actually lagging behind. So the system would be good as such if it could just capture most appropriate information that is not captured in systematic reviews. And having said that, one of the approaches we want to use for retrieving the citations in the future, is to get the publication relevance. So the publication relevance over here we say [inaudible] metric same factor, but we would also consider the hierarchy of scientific evidence. So we can hierarchy of scientific, we are already accounting to that in one sense. Because when we look at the – if we look at the study, we are primarily focusing on systematic reviews and therapies. And we are looking at other studies only when they are not systematic reviews instead of filters. 

So the system does have capability to focus more on systematic reviews and highly precise articles. And we are also working towards giving value to other factors that show that the study is more prominent. In the future, we would also like to give more weight to say randomized controlled trials over cohort studies, over cross section studies, over case study studies, or over just clinical case reports. So the future going to it at the present, we are doing it by first focusing on systematic reviews and the clinical filters, therapy or not filters, which were identified by manual experts who annotate them. So that helps us. And for a disease like Alzheimer's disease or depression, all articles or most articles we get are from systematic reviews and the clinical filters. But as a researcher or as the audience member who asked that question pointed out, there could be some studies which are so novel that they will not get captured in systematic reviews. And at present, our system does not ignore them. If they are not in a systematic review we go to them then. 

So that would be one angle from which I would answer the question. In the second angle, from which I would answer the question is, there is also research being done by NLP researchers, by informatics domain which helps to accelerate the process of systematic review, and in reducing the workload in reviewing thousands of abstracts. So initially the abstracts could be screened by a machine, next we can by human. So that is a second angle where similar informatics approaches for the help.   

I do not know if the question was very thoughtful and very nice, so I want to see if that answers the question or if there is a follow-up. 

Moderator:Well I have not received a follow-up.

Sid:Guilherme or Javed, Rich, do you want to add anything to that? 

Unidentified Male:No I think you covered it well Sid. I think the main point is this is not replacing systematic reviews. I think it would be very hard to capture the nuances that a manual review of the literature is able to do. But it is important what Sid said that systematic reviews cover a very small percentage of the literature. And a large bulk of systematic reviews are outdated. So they main contain important information but we should not ignore more recent studies that were not included in a particular review. 

Javed:This is Javed, what I would add is, I do not know if this was implied in the question, but the importance of the source perhaps measured through some other metric like impact factor or some citation frequency could potentially be considered in the importance of the origin of the sentences. And source of the sentence is potentially a factor that could be incorporated in different ways in identifying sentences and ranking them, which we have not really done. 

Rich:Yes and I think – this is Rich Medlin speaking, I think the other part is that the different clinicians are going to want things ranked differently. My clinical training is emergency medicine, I am going to tend to see things from the emergency medicine perspective. A cardiologist may want to see something from cardiology and a surgeon from something different. And so I think that one of the things about this system is neat, it does a great job of retrieving relevant sentences. So there is a lot of – all of the sentences – virtually all of the sentence are topically relevant. And so there is a lot of opportunity to refine and re-rank those results. And this system automatically sort of expands – if there is no systematic reviews, then the next loop goes through and finds things that are not systematic reviews and expands the search. And that is all transparent to the user. 

Sid:Wonderful.

Moderator:Great, thank you. That, once again, is all of the questions that we have. We have just a couple of minutes left, I do not know if any of you have any final remarks you would like to say before we wrap things up?

Javed:Yes, this is Javed, I just wanted to follow back up to the question about publishers’ reluctance to make content available. I think another way to look at this research is independent of what is available or what is not, it is making a contribution toward search and text mining. And techniques like this I am quite sure, will be incorporated into search engine technology in the future. And probably people are working on it already. Whereby you are not going to be just presented with links to content but content directly. And so I think the issue is important how much content is available for mining. But I think the contribution is toward being embedded in the algorithm for identifying content directly. So I just wanted to make that comment. 

Sid:Excellent. 

Moderator:Great, thank you, anyone else have any – any of you have any other final remarks you would like to make? Sounds like that is a no, we will wrap things up. I just want to once again, let our attendees know our presenters today are wondering if you would be willing to participate or help in recruiting for an international survey on information needs. If you are, please send us your email address in the Q and A screen and I will get that over to our presenters as soon as we get finished today. They will follow-up with you. And also, as you are leaving today’s session, you will be prompted with a feedback survey. If you could take a few minutes to fill that out, we would definitely appreciate your feedback on today’s session. We definitely do read though those and we use those in our current and upcoming sessions. So we appreciate the time you put into that. 

Sid and all of other panelists today, we really appreciate the time that you put into putting this together and presenting today, we really want to thank you very much for presenting for today’s timely topics of interest cyber seminar. 

Sid:Thanks a lot. 

Moderator:Thank you, and thank you to our audience for joining us, and we hope to see everyone at a future HSR&D cyber seminar, thank you.  

Page 1 of 13

PAGE  

Page 13 of 13