# INTEGRATED SURVEILLANCE SEMINAR SERIES

CENTERS FOR DISEASE CONTROL

COORDINATING CENTER FOR HEALTH INFORMATION AND SERVICE

NATIONAL CENTER FOR PUBLIC HEALTH INFORMATICS

DIVISION OF INTEGRATED SURVEILLANCE SYSTEMS AND SERVICES

TITLE: Automated surveillance methods for notifiable diseases: theory,
practice, and evaluation

SPEAKER: Matthew Samore, MD (UT, Center of Excellence in Public Health
Informatics)

MODERATOR: Robert Pinner, MD (Director, CDC/CCID/NCPDCID/DEISS)

February 13, 2008

JENNIFER WARD, EPIDEMIOLOGIST, DIVISION OF INTEGRATED SURVEILLANCE
SYSTEMS AND SERVICES, CDC: I'm pleased to begin this session of the
seminar series. We want to welcome everyone that's in the room and those
of us who are joining by webinar. The seminar series is hosted by the
National Center for Public Health Informatics, Division of Integrated
Surveillance Systems and Services. It's part of a larger Integrated
Surveillance Community of Practice which is intended to facilitate
interaction between stakeholders in the field of public health
surveillance. This seminar series strives to establish an interactive
forum for discussing the vision of integrated surveillance, identifying
issues and approaches to its achievement, and providing a mechanism for
discussing best practices in the area. Following the presentation by our
speaker, we will have a question and answer and discussion period. For
the benefit of our participants via webinar and Envision, I'll ask you
to please use the microphones in the center of the room when you're
asking questions. We are recording the webinar, and we want those on the
phone to be able to hear as well. For those who are joining us by phone,
please make sure you put your telephone on mute when you're not
speaking. Also, please avoid placing your telephone on hold as the music
will be heard by everyone. So, for those in the room I ask you to place
your cell phones and pagers on mute or vibrate. So, moderating our
discussion today we have Dr. Robert Pinner. Dr. Pinner is the Director
of the Division of Emerging Infections and Surveillance Services in the
National Center for Preparedness, Detection, and Control of Infectious
Diseases in the Coordinating Center for Infectious Diseases. Please join
me in welcoming Dr. Pinner to the Podium.

ROBERT PINNER, DIRECTOR, DIVISION OF EMERGING INFECTIONS AND
SURVEILLANCE SERVICES, CDC: So, thank you and good morning. I have the
honor and the pleasure of introducing Dr. Matthew Samore. Matthew is a
professor of medicine and Chief of the Division of Clinical Epidemiology
and adjunct professor of biomedical informatics at the University of
Utah, School of Medicine. He's also the Director of the Salt Lake VA,
Decision Evaluation Surveillance and Informatics Center. Matt earned his
undergraduate degree from Macalester College in St. Paul, Minnesota; his
M.D. from the University of Wisconsin in Madison; and then did medical
training, first as a medical intern and resident at Barnes Hospital in
St. Louis, Missouri; and then in Clinical Infectious Diseases Fellowship
at Beth Israel in Brigham and Women's and Dana-Farber Cancer Institute
in Boston, Massachusetts. His bio-sketch lists thirty-some odd
peer-review publications selected from more than one-hundred ten. So,
Matthew's done a lot of work in this field, but also a lot of thinking
and a lot of writing about the subject he's going to present today. He's
going to talk to us -- this title you see here is "Automated
surveillance methods for notifiable diseases: theory, practice, and
evaluation." And, he's also promised to explain to us what situational
awareness means in Salt Lake City. Dr. Samore --

MATTHEW SAMORE, CENTER OF EXCELLENCE IN PUBLIC HEALTH INFORMATICS, UTAH:
The definition of situational awareness in Salt Lake City is knowing how
much snow fell in the last twenty-four hours and what the snow depth is
at each of the eight resorts. So, thank you for the introduction, and
thank you to Scott McNabb, Dr. McNabb, for inviting me here, and thank
you to the CDC. We're very -- I know I'm going to mess something up, but
it's an extreme pleasure to be here. And, I want to thank the CDC for
supporting our research activities in surveillance both through the
Center of Excellence in Public Health Informatics and through the DHTP
funded Epicenter for Prevention of Healthcare Associated Infection. And,
first of all, I want to say that I consider my primary origin to be
epidemiology -- infectious disease epidemiology -- and informatics to be
a passion that I acquired and nurtured and developed over 16 -- 17 years
of post-fellowship training and research experience. I have a distinct
perspective on trying to integrate and combine epi-thinking and
informatics-thinking, and maybe if there's time I'll describe the
anecdote that Bill Yasnof has about that. So, one pleasure for us in
Salt Lake is that having a DHTP-funded center and a NCPHI-funded center
gives us a chance to work with CDC and work with local state departments
and with health care systems -- kind of a variety of different
perspectives. And, so I hope I can communicate that to you today. I'm
used to fielding questions in the middle of my talk. I'm not sure if
that's quite appropriate. I like doing that because it makes me feel
that people are actually paying attention. What else do I want to say
introductory-wise? I guess that's probably about it. Oh, the one thing
-- I apologize -- I never did EIS, and I realize that devalues my worth,
and I apologize up front. Maybe it's not too late. (Indistinct audience
comment) Oh, excellent! Thank you.

Here are the objectives that I sent Jennifer. And, I want to thank
Jennifer for organizing this and putting up with my belatedness for last
minute delivery of various required documents. As Bob said, I am going
to discuss situational awareness. I'm going to discuss it in the context
of public health surveillance, but first \[I'll\] start with
non-epidemiologic examples. And, I'm doing this because it is a term
that is used extensively. Every time I'm in a meeting and ask, "What do
you mean by situational awareness?" the answer is usually something
like, "We want you to tell us." I want to also say that I am going to
discuss notifiable diseases but not only notifiable disease
surveillance. And from my perspective, there's a continuum of different
types of surveillance, and real-time notifiable disease surveillance is
basically a form of biosurveillance in my opinion. So, secondly I will
discuss work that we're doing to automate notifiable disease
surveillance through an inference engine that's supplied through
electronic health records and then a messaging method that transmits
case record data -- not electronic laboratory data only -- but case
record data to the health department. And, this is a project not led by
me, it's led by other investigators in Utah and I hope I do justice. If
they're on this webinar, please forgive me if I made any mistakes --
make any mistakes in my presentation. And those people are Catherine
Staes, Bob Rolfs, and Lisa Wyman, and Deepthi, and Melissa Stevens. So,
I'll discuss that in the second part of my talk. The third part of my
talk, which is the shortest, is about evaluation. And, the reason why
it's the shortest is that we don't have data to report to you yet about
evaluation. So, what I'm going to present is a framework for evaluation.
And then I'll try to sensitize all of this, and I hope it makes some
sense.

So, another way to express the outline is that I'll talk about theory --
and, I have to say that, as an academician, it means that I never have
to actually do anything practical. But, in fact I am very passionate
about trying to do research that brings value to the public health
setting and to the clinical setting -- so, first, theory then practice
than evaluation.

So, paradigms and models -- well, there's a concept or a term called the
panopticon, which some of you may be familiar with. But, it's another
view of surveillance that's distinctly different than what we're used to
in the public health and the healthcare sense. The panopticon is a kind
of malignant tower that watches you and follows you and makes you feel
insecure, like you're living in a Kafka novel. It's the sinister side of
surveillance, and I think we see that also in our current environment
with the controversies about privacy and government surveillance outside
of public health. I mention this just because the relevance, I think, to
public health is that -- the concept of panopticon is that -- you can
monitor everything all the time. And, I think to the extent we try to do
that in public health -- it may be unattainable, it may in fact be
defeating. The other alternative approach to surveillance, so-called
synoptic surveillance, is it takes a more focused approach to identify
who's at risk in the public health context. I mentioned these two terms
just as paradigms for different approaches to surveillance. Then
another, a theory that's very relevant to surveillance -- namely signal
detection theory, comes out of psychology -- addresses the ability to
detect signals. And, one important conclusion of signal detection theory
relates to the intrinsic trade-off between sensitivity or hit rate and
false positivity or false alarm rate. I'm not going to really discuss
this in any detail, but I mention it as an important theoretical
framework for surveillance. And, then thirdly -- situation awareness. I
should say that my guru in situation awareness is a kind of psychologist
that we work closely with named Frank Drews. He's been one of the key
investigators with our Center of Excellence. I have to say that one of
the benefits of our Center is the ability to bring together an
interdisciplinary team, people with expertise -- not just epidemiology
and informatics, but also psychology, geography, and simulation. And,
Frank's comment about situation awareness is not all that favorable, he
doesn't think too highly of it as a theory. But, it is pervasive in the
literature as you know. And, it's classically thought of as having three
levels; namely detection, diagnosis, and prediction. When people talk
about this, I think they're eyes kind of glaze over. And, so I'm going
to try to present situation awareness from a slightly different
perspective. Hopefully, it will make more sense and be more intuitive.

So, the definition I present is the idea that awareness is really the
wrong term, it should be situation comprehension. The key concept is
that it's understanding or comprehension of a dynamic environment. So,
those are the 3 fundamental components of situation awareness:
comprehension, you can think of reading comprehension or text
comprehension as a familiar example of comprehension; dynamic means
something that changes, that evolves -- that can be rapid or slow change
; and then environment is what's around us.

So, I present to you a slide: understanding comprehension. So, the
psychology definition, or the sentence that I've copied here is, "a
phenomenon that emerges from an orchestra of cognitive processes" --
that's kind of hard to understand. But, there's a lot that goes on in
our brains that links to comprehension. There's this perception of the
environment, the surface level appreciation of what's going on, and then
-- similar to what we think of in informatics -- there's a semantic
layer, if you will, that's in our heads, in our brains. This is the
extraction of semantic information from this perceptual input. And then,
there's the term referred to as the situation model, where the
information about the meaning of the perceived data is integrated with
prior knowledge. Now, again I'm referencing a book, a work by the
psychologist Durso here. If you're interested, I can give you the
citation.

Now, let me give an example from outside of public health. So, "on a
beautiful night in October 1978, in the Chesapeake Bay, two vessels
\[see each other\] sighted one another visually and on radar. \[And\] on
one of them, the \[captain\] of the Coast Guard \[sees this\] other ship
up ahead as a small object on the radar, and \[he sees\] two lights,"
and his interpretation of this information was that this vessel is
proceeding in the same direction as his ship. He thought it was possibly
a fishing vessel. My guess -- the idea that it was a fishing vessel has
something to do with the number of lights. I'm not familiar -- I don't
have any expertise in this domain, so I don't know that it's actually
true. Now, he notices that he's actually coming up onto this vessel very
quickly, and so he assumes that -- given that they're traveling the same
direction -- that this must be a very slow boat, and he's about to
overtake him. And, then he is approaching the mouth of the Potomac River
and he has the realization that if he's going to be overtaking this
boat, that he's going to cut it off, because the boat will not be able
to turn into the Potomac River, because he'll be on the left side of the
boat. What happens is that he orders a turn to the port, but in fact, he
was not overtaking this fishing boat, he was bringing his own boat
directly into the path of a freighter that was going toward him. The
freighter hits the Coast Guard boat and, tragically, eleven Coast
Guardsmen died.

So, here's the mental model. The top panel -- the top four squares show
the mental model compared to the real world. So, if we start with the
real world, what you see are two ships that are approaching each other
and the Coast Guard is on the right and the freighter is on the left.
You see them indicated by the boxes with the letters "F" and "C". But,
what happens in the fourth panel is the tragic mistake of the captain,
which is the turn in front of the freighter. But, he was acting
completely in concordance with his mental model of the situation. I
mean, this was his situation comprehension. This was the representation
of his comprehension of what was happening as he was approaching this
other vessel. His representation was that he was on the other side of
the vessel, and that he thought he was doing exactly the right thing by
turning.

Now, I'll give another example of how this can be conceived of outside
of the public health context. You probably know, there's a lot of data
that suggests cell phone is hazardous during driving. In fact, the
psychology group in Utah, including Frank Drew's, has done a substantial
amount of this type of work. And so, one can study the various ways, the
ways that cell phone use distracts driving. One of them is the attention
level, the failure to perceive something, to check your rearview mirror.
Another is a failure -- an extraction of the semantic information, and
that is the -- for instance, it would be the inability to see that -- at
a four-way stop - who actually was the first and who had the appropriate
right of way and should enter the intersection first. The situation
model would be a situation where the driver is observing, say, another
oncoming vehicle, recognizes that that individual, say, is yelling or
turning away from the windshield, does not seem to be looking ahead, but
fails to recognize the implication of that situation with respect to the
actions. The driver observing this does not project the consequence of
this situation.\
So, now what I want to do is bring this back to the public health
context. So, first of all, the health of a population is indeed dynamic.
So, clearly eclipse this notion of comprehension of a dynamic
environment. And, as I mentioned in public health, the time horizons
cover a wide range. So this is something that is different in the
technical systems that I described -- the systems of, say, operating
machinery or flying an airplane. Some of the responses and some of the
situations represent very immediate threats or immediate problems, and
others cover a much longer time frame. Now, that's one distinct aspect
of situation awareness in public health. The other aspect that I think
is worth noting is that the concept appropriately captures the data
hungry nature of epidemiology in public health. Basically, there's no
data that you don't want if you're given the option to have more data.
And now, the way it's usually discussed in public health and
epidemiology in reference to just the current health care capacity --
what's the status of the healthcare systems in your environment, and
what's the status of the population in your environment? And, also it's
usually thought of in the context of early detection. So, one thing I
would maintain to you is that this is too limited of a view of situation
comprehension and also not adequately developed.

So, what I see happen in our literature is that situation awareness is
just described very vaguely as a goal of surveillance without actually
specifying the problems, decisions, or the actions. And, without really
taking the time to develop what the actual situation model is -- what is
being comprehended?

To accomplish this in public health and epidemiology, what's needed is
this deeper probe, not just of the interpretation of what's going on
now, but the projected state -- what might happen in the future. And, I
think the other aspect of this that will be very important and will
reside in a variety of techniques, including scenarios to link theory
and to develop measures of comprehension.

So, this is an approach that we're trying to take in Salt Lake in the
context of various projects around surveillance. And, one of these
projects that is funded in part by the Department of Defense through the
??? and also relates to the COE work, is a project where we're
developing data visualization tools and using multiple, heterogeneous
types of data streams coming into public health. And, one aspect of this
project which involves a collaboration with the Scientific Computing and
Imaging Institute in Utah is that we link together the computer
scientists and the epidemiologists in the local and state health
departments. So, the computer scientist doesn't understand public
health, so he asks the epidemiologist, "Describe what's going on today
-- what's happening today?" And this is what Mary Hill said,
"Significant flu in the valley. Fever and coughs are rising," -- I don't
know what day this was, but we actually have been experiencing
increasing respiratory and viral infection and influenza -- "12 people
hospitalized throughout the valley. 20% are children." She describes
that "most cases are rapid A," she describes the less specific data,
namely "school absenteeism in high in middle and high schools," and "by
the way, the hunting season started." So, what I find fascinating about
this is that it reflects exactly what I think fits the situation
comprehension model. Namely, this is the epidemiologist -- the local
epidemiologist -- their current diagnosis of the state of the population
in Salt Lake Valley. And, I take this to be -- "things are normal" --
sure, there's flu; sure, people are hunting; but, nothing here is
unusual in terms of what we might expect for this given season. If this
was happening in July, it would be unusual, but given this time of the
year this is kind of not a significant departure from normality. That's
my interpretation of this.

Now, what we're trying to do is take these concepts in these terms and
these descriptions of the current state of the population and turn them
into mental maps, and turn those mental maps into data visualization
tools. Now, another way to approach this is through using scenarios. So
another project that's related to our COE is to do interviews with
public health personnel where we ask them about difficult decisions,
about protocols. We also give them an outbreak scenario. We've been
analyzing in a very detailed way the responses to these outbreak
scenarios and find substantial variation across public health personnel.
So, we're trying to understand the problem space and the mental math and
the way the concepts are networked for epidemiologists and they assess
this simulated outbreak

Okay, so that's kind of the theoretical piece. And, when I discuss this
in some context -- I'm told it's not practically useful. So, I'm going
to shift to something more practice-oriented. But, maybe I'll stop and
take some questions now, if that's okay, Scott. Okay.

My first comment -- Bob asked me, "What are you going to say that's
provocative?" So, this is my provocative statement; it's basically: why
RODS was useful during the Winter Olympics and during other special
events, but not particularly useful or maybe almost close to -- well I
shouldn't say this publically -- but, not as useful as other times. I
don't really have enough slides to develop this idea. But, what I would
maintain actually is that it relates back to the idea of situation
comprehension.

And, so what I would say is, if we just think about syndromic
surveillance in terms of: did it detect your cryptosporidium? Did it
detect your neuro-virus? I think that the classic measures of positive
predicted value, sensitivity, etc. are not fully capturing the value of
syndromic surveillance when it is perceived to be useful. As I said,
it's distinctly perceived to be useful during special occasions, during
special events, and was perceived to be useful during the Winter
Olympics in Salt Lake City. So, the -- essentially -- and this is
something I hope we get to during our discussion -- what I would say is
that the basic issue is really that the contacts and the perception of
departure from normality or risk of departure from normality or
suspicion. Essentially, another way to express it is the "current level
of threat." That is where syndromic surveillance, its value for
situation comprehension varies according to that context. But, I welcome
discussion about this.

Okay. So, I said this would be about practice -- public health practice
-- but, actually it's about public health practice-based research or
research center. And, it would be translational research -- research
that's applied.

To do translational research, we need to have a public health research
laboratory. This has to abide by core principles, which I list here.
Namely, the idea that we establish a partnership that -- where public
health leads or co-leads projects, where the synergy achieved through
this close cooperation and collaboration, the projects lead to benefit
public health. There needs to be collaboration at multiple levels of
public health. And, this has been the vision that we've tried to follow
in Utah.

And, fortunately we've had terrific support in Utah from - especially
from -- well, overwhelmingly from the leadership of the Utah Department
of Public Health; the director Dr. David Sundwall; and the state
epidemiologist who many people here know, Dr. Bob Rolfs; and Dr. Wu Xu
whose kind of the mover and the shaker for organizing and integrating
data systems in the Utah Department of Public Health, working with Barry
Nangle. Wu is now the director of the Office of Public Health
Informatics in Utah Department of Health. And, these are some of the
activities of this office. What's very good about it is that it's not
just an internal organization. It reaches out, not just to academia, but
also the clinical world. So, it's coordinating e-health initiatives,
including clinical health information exchange, which of course if very
relevant for public health.

So, our projects support public health, especially in communicable
disease surveillance, from the identification to the investigation and
analysis of data, to the action based on results and the responses to
outbreaks and other advisories. With these boxes -- and this slide, I
should say, was put together by Bob Rolfs -- these boxes are very nicely
tied to the surveillance framework that Scott McNabb - Dr. McNabb has
published. I'll discuss this later in the context of evaluation.

So, I'm going to just discuss -- give a very brief overview of the
projects that relate to surveillance. I put them into two categories;
levering electronic health records is the first category. The first
project, which we called RTCEND, Real-Time Clinical Electronic
Notifiable Disease surveillance is a project I'll spend most of my time
discussing for the second part of my talk here. I also want to mention
that we're doing work in our VA system around text processing. The VA is
actually launching a huge initiative around text processing and
unlocking the value of the text narratives in the very rich electronic
records of the VA system. So, we're very excited about this, I think it
has the potential to lead to some important collaborations with NCPHI. I
mentioned some of the people here that are leads on these projects. I've
already mentioned the names of most of these individuals so I won't go
through these again.

Now, the other category of projects are about surveillance are more in
this situation comprehension, decision support, and data visualization
category. We are doing work with pathogen-specific data that arises from
the program at InterMountain Health Care called GermWatch. I'm not going
to discuss that more, but that is something, I think, is the next
generation after syndromic surveillance -- it provides a much richer,
better approach to tracking the epidemic behavior of various types of
pathogens in syndromic data. And, the data are quite timely. I mentioned
already the visualization work that we're doing. We're using interactive
simulation tools for decision support -- I'm not going to describe that
further, but it may come up during discussion. I think simulation is
very important for understanding and developing tools to measure
situation comprehension. We're also using simulation to help us with
evaluation of the impact of surveillance on resource consumption.
Investigators are listed here and in collaboration with Virginia
BioInformatics Institute and University of Michigan.

So, now the RT-CEND project -- So, this is a project that builds on a
long standing program that's been led by Scott Evans at InterMountain
HealthCare to do automated infection surveillance. Scott has implemented
rules-detection that he's used for nosocomial infections and also for
notifiable diseases, as well as for other types of adverse events in
healthcare settings. So, that system has been in place for a long time,
but what is new for us is to create this automated link from these rules
and alerts to the health departments. So, that's the new part of the
work that is now underway. In Utah, we have a situation where we have a
dominant health care provider, InterMountain HealthCare that provides
for 60% of the population. And, in circles on this map show the various
in-patient facilities that InterMountain HealthCare operates. You can
see where the population is in Utah. They follow Interstate 15. Between
InterMountain HealthCare and the University of Utah, which also has an
automated rule surveillance system developed by Therodock(?), these two
healthcare systems - we have about 70 -- 80% of the population covered.
It's population-based surveillance even though it's a limited number of
health care systems that have these systems in place.

The system that Scott designed involves this construction. It basically
entails the use of data driver to run against data, the electronic
health records that evaluates or triggers the application of various
rules that are in medical logic modules. This engine generates an alert
file.

This alert file then is used to produce a database that prints regularly
for the infection control practitioners and also distributes messages by
other mechanisms such as email. Reportable disease monitor becomes -- is
added back into the data warehouse system for InterMountain HealthCare.

So, what's happened is we have a system where surveillance is
facilitated at InterMountain HealthCare and at the University \[of
Utah\]. What we haven't done yet is actually measure the impact of the
system at the healthcare system level. We have reporting rates in Utah
because of these systems in place which make it much easier for the
infection control practitioner to do reporting. You see here numbers of
cases of various diseases that are detected by the system at
InterMountain HealthCare. For the most part, this is based on laboratory
results. So, it's not yet relying on more fuller \[sic\] set of fields
than the electronic health record.

The problem of the system, and this is true for both InterMountain
HealthCare and the University of Utah and the VA which also has
Therodock, the problem is that it can't get out of the healthcare system
automatically -- it has to be faxed. We have the proverbial single fax
that breaks down. We have a problem with that particular step because it
does not lend itself to dissemination of these reports easily without
redundancy across the different levels of health departments.

So, what we're trying to do is set up a system - we're starting to do --
is implement a system where the data are automatically transmitted into
the health department for distribution across both local state health
departments. Still, we use NETSS with a 'T', so we 're still in the
stage of -- we still don't have NEDSS implemented.

The good news is we're now underway with the NEDSS-implementation, which
I'll talk about. Now, this automated electronic transmission of the case
to the health department is not quite as far along as what you heard
from the ESP program at Harvard. It's got set warnings, I should never
have followed Harvard -- what can I say? I used to be at Harvard. I had
that experience. But, we actually have now modeled the HL7 message
structure and set it up so that implementation is ready to go. When the
health department is ready to receive it, we're actually ready to send
it. This work has been spearheaded by Deepthi Rajeev, one of our public
health informatics students and her supervisor, Catherine Staes. This is
something that is actually quite interesting. I cannot speak in detail
about this; perhaps later in the webinar we can address any questions
about it.

Now, we're planning to also develop an automated HIPAA documentation
system. Also, we'll do a chart review validation of the rules that are
applied at InterMountain HealthCare.

System for data flow between state and local health departments involves
a lot of people, these are people that I've already described. These are
the same people that are spearheading the NEDSS implementation in Utah.

And, we're one of the states that tried to implement NEDSS and had a
failed implementation. The software company, which was also was
contracted in California, went bankrupt and was unable to complete the
project. We have a new vendor now, Collaborative Software Initiative,
CSI. They follow an open source software model. It's agile, rapid
development method is now underway. So, we are trying to stage
implementation over the next year and a half.

Now, what we'd like to do is to go beyond this simple case transmission,
which we expect will improve the timeliness and completeness and
accuracy of notifiable disease surveillance, and we anticipate it will
save time in retrieving these data by other means such as phone calls or
other communication tools that infection control practitioners and other
providers, but we'd also like to go further. We'd like to leverage the
collaboration that we have and leverage the access to electronic health
records to support the investigation. This is a system that is wholly
feasible based on the technology, but requires the setup of a --
establishment of a system for making this happen within the constraints
with respect of privacy and the firewalls that exist between healthcare
systems and health departments. Some of the work that we're doing at the
VA lends itself to this approach where we use the electronic health
records to support investigation because we've developed some text
processing tools to pull out epidemiologic data from the narrative texts
of the VA records.

Then, I want to say something about the intrinsic limitation of any
surveillance system, which is that -- there is never a true gold
standard. The -- I see that there's an extra sentence on the bottom of
this slide -- the problem is that we never actually have a direct gold
standard estimate of disease occurrence in the population. This is
inevitably due to incomplete evaluation, or incomplete healthcare ???
behavior, incomplete testing -- imperfect tests. And, some of the other
ways, specificity and false positivity, is never perfect. I think that
-- one thing this brings up is that electronic health records criteria
which tend to be more objective than some other types of criteria for
case identification may actually be better than subjective criteria,
even if they don't exactly map to the standard confirmatory criteria.
Maybe this is something we can discuss further.

So, I just have a couple more slides -- a few more slides about
surveillance evaluation. What I'm describing now is work that we plan to
do in the context of our implementation of RTCEND and implementation of
NEDSS at the local and state health departments. We try to use the
conceptual framework that Scott has published where he describes six
core activities and, importantly, two types of public health actions,
acute responses and planned responses.

Our approach here is to adopt a method that has been used fairly
extensively in clinical information system implementation. Methods that
we have experience with in implementing clinical decision support and
positional reentry where we use interviews, surveys, and observation to
assess the adoption of new technology and the fit to workflow. Briefly,
I'll plan to adapt these methods for evaluation of implementation, and
for implementation success, adapt these methods in the public health
context, something which I do not believe has been done before.

Our timeline is described here. This is what we hope to accomplish --
begin soon and accomplish over the next year and a half. Now, I have a
few more slides that represent the synthesis. I think I'm going to pass
to use the -- given that it's almost 12 -- I'm going to pass on these,
only bring these up if questions arise that are relevant to these other
slides.

And, my last slide is a picture of the Brighton ski resort Salt Lake
City. The snow is great this year -- speaking of situation awareness in
Salt Lake. Please do not hesitate to visit us and enjoy yourselves.
Thank you.

ROBERT PINNER: So, it's question time. Let's try the telephone lines
first. How do we do this? We just hear voices? Okay. Is anybody there on
the telephone? How about that? So, ya'll on the phone, think it over a
little bit more, we'll come back to you in the meantime. Jerry, do you
have a question?

JERRY TOKARS: So, the people at Harvard are doing their system and
you're doing your system. There's actually a number of other people that
are doing the same thing. And so, it's sometimes good to have multiple,
different competing ones and they sort of evolve, but should we try to
come together with a common approach and what would that be?

MATTHEW SAMORE: That is a fantastic comment. We're definitely eager to
collaborate. Mike Klompas, Ross Azarus, and Richard Platt are very
cooperative as well. We had a couple meetings with them and the
situation was basically that to get something done now we had to do it
our own way. It's just not going to be feasible to transplant their
tools to our context, in part, because we're working with a single
healthcare system that already is generating the data of notifiable
diseases based on the electronic rules. We felt that we needed to
develop our own HL7 model for this. I have to say that Catherine Staes
and Deepthi worked very hard on this and decided that they would do this
locally. We hope that this HL7 model that we have can be used by other
sites. We're very eager to do something that is not a one-off. I think,
moving forward with the COEs, especially the grid computing tools that
we're going to be implementing, there's going to be greater potential
for sharing across COEs.

AUDIENCE MEMBER: I actually am the project officer for the Vaccine
Adverse Event Reporting System or VAERS. We're actually working with the
Harvard group on their HL7 messaging to VAERS. I was interested in
knowing whether or not you'd consider using your system to possibly
extend it and use it as stimulating reports to our VAERS system because
we're always looking for ways enhancing, stimulating, and getting data
to come in, particularly through the electronic messaging system.

MATTHEW SAMORE: Yes. Definitely. Let's talk about it. I think that would
be very interesting. We have a strong interest in vaccine registries and
because of this data warehouse -- enterprise-wide data warehouse at
InterMountain Healthcare and the university, we have actual
opportunities to get additional types surveillance from these electronic
records. I'd like to discuss that further with you.

\[indistinguishable speech\]

The question is from someone obviously more knowledgeable in signs than
I am. \[They\] asked, "What were the color of the lights on the boat?" I
have to say, I don't know. But, it turns out that, on this tragic
accident in Chesapeake Bay, that there was a first mate who actually
correctly -- he had a correct mental model of the situation. So, there
were other problems with communication and - of break down, the type of
error, there was a cascade of events that ultimately led to this
tragedy. But, I don't know the answer to that question. I apologize.

SCOTT DANOS: Hi, Matt. Scott Danos. Great presentation. Thank you very
much. Couple of things -- We're quite interested in working with your
partners in Utah around open source development. We have communicated
with Richard Vursvan(?) and others. The dialogue is open; we want to
expand that. As a matter of fact, we sent not a CD but a DVD of an
exhaustive set of requirements that we had gathered around the work that
we'd done at CDC at their request to maybe accelerate that process. So,
we can collaborate in lots of other ways. I guess my question, and
really acknowledging what you and the folks at Harvard and other places
have faced -- and that is you have a problem which is an issue with
sending electronic message data of case reports and the absence of a
standard -- and you're exactly right to go and approach this as you've
done -- Catherine Staes and your team is collaborating closely with John
Abellera and the CSTE group, so we're really excited about
that...standard to accomplish that. I just want to sort of raise the
idea that we're at the point -- close to the point -- of being able to
pilot test both content as well as structure of messages. And, I know
you guys have settled -- not surprisingly I would say probably wisely on
HL7 model. But, we're receiving input, and it's way too soon to be
asserting this, that we may look at not the HL7 ORU message but HL7
V3CDA message as a complement. And, we'll look to you and others as that
decision is approached. Looking to use the model that you have created
and are looking out to test content as well as structure going forward.

MATTHEW SAMORE: Thank you for your comment Scott. I want to say that
Catherine is really enjoying participating in the calls with you. Ever
since last summer there's been a terrific exchange of ideas, so that's
been very helpful.

ANDREA HEGGEDUS: Hello. Thank you for you presentation. I am Andrea
Heggedus. I work at NCPHI in the Division of Alliance Management and
Consultation, and I am charged with doing an overall evaluation plan.
So, I was delighted to see your emphasis on evaluation and the fact that
you were doing it and see it as important to inform the process. It's
not enough for you to do it alone. We have to get it out there and help
other people do this, too. I'd like to thank you for doing it and for
doing it with the depth and breadth that can actually tell us something.
Hopefully, we'll be in touch and we'll look forward to seeing it over
the next couple of years -- what you're going to be doing and the
results.

MATTHEW SAMORE: Thank you very much. Evaluation is so important --
again, that depends on the collaboration, because health departments are
very much consumed with the operational side of doing implementation.
There really needs to be an extra effort put forward at the beginning to
collect the baseline data to do the surveys, etc. that can inform
implementation in other settings, which is what we're trying to do.
Thank you.

AUDIENCE MEMBER: I have a small question for \[about\] Harvard and also
your data system. One question in my mind is, when you put all the data
to the centralized data warehouse, where are you putting the distributed
computing engine? Especially in the future, region-wide or nationwide
across health plans you must identify the space to the \[sic\] local
healthcare facility. And just before, identify the case to the
centralized data warehouse.

MATTHEW SAMORE: Thank you for your comment. I want to make sure what
you're saying in terms of the distributed agent. Can you say that again?
I want to make sure I understood.

AUDIENCE MEMBER: Okay. My question is this: When you do case
identification in a centralized data warehouse, do you use distributed
computing engines in different healthcare facilities, then report to the
centralized data warehouse?

MATTHEW SAMORE: Okay. So, there's an HL7 interface engine that operates
enterprise-wide for InterMountain Healthcare. The system is set up so as
moved data hits that data interface engine, that triggers the
application of these rules that in real-time then identify patients who
meet criteria for notifiable diseases. Then, when those criteria are
met, additional data are assembled to create a case record which we then
package into an HL7 message and send to the health department. So, it's
a system that leverages the existing decision support engine's rules
engine used by InterMountain Healthcare for all their inferencing that's
enterprise-wide. Does that make sense? Maybe I didn't...

AUDIENCE MEMBER: The rules, simply saying, that kind of computing engine
is allocated to local hospital or to some centralized system?
Distributed or not distributed?

ROBERT PINNER: So, let me once again invite folks on the telephone to
speak up if they have questions or comments.

CECIL LYNCH: Hi, this is Cecil Lynch. Very nice presentation. I think
the questions that I have are similar to the last gentleman there about
distributed engines. We are approaching the same type of a solution and
we talked with Catherine a bit about this in that we also have adopted
an HL7 model, an HL7 version 3 model, we'll also accept the CDA or v2
models as well, but we have approached from the distributed engine
perspective so that you're not particularly bound to any one
institution's rules -- we essentially drop in any rules from any
institution. Have you thought about how your system would scale beyond
your current environment?

MATTHEW SAMORE: We'd like to use OntoReason. Cecil, thank you. I was at
the call, the presentation that you gave on Monday to the EpiCenters
Network -- speaking of synergy between GHTP and NCPHI -- so we've been
very excited about trying to partner with you. I talked to Scott Evans
about this. I know one of your people in Utah, Craig. We'd really like
to follow up with you and Craig about how this might work. I think that
we'd be eager to take the next step. We do want to do local
implementation and evaluation. You could say we're operating on two
different levels here. One is to solve the practical problem for our
local and state health departments and then evaluate the scientific and
practical question of: what's the impact of public health surveillance?
But then, we also want to solve the larger problem that your system
addresses which is how to have the system that can be plug-and-play, and
modular, and implemented anywhere.

CECIL LYNCH: I think the evaluation is the key issue here. That approach
will help to validate all the systems. Thank you for that.

ROBERT PINNER: Other telephone questions? So, while you're thinking, I
have a question. Just as we're getting used to using the term situation
awareness, you decided -- suggested that maybe that's the wrong term and
we ought to be using comprehension instead. I wondered if we need to
start adapting our terminology. But, beyond that in the theoretical part
of your talk, you challenged us to try and work on specifying more
completely what situational awareness is. I wonder if you have at least
some introductory thoughts about what that might look like or even what
the dimensions that need specifying are.

MATTHEW SAMORE: That is a fantastic question. So, first of all, I'm not
sure that it's advisable to change the terms, because I have to say,
again, I'm taking the advise of our co-investigator, Frank Drews, who
has expertise in this area -- I think, in terms of terminology,
"situation awareness" is very much embedded. So, I would not necessarily
abandon it, but I think it is important not to leave it hanging there.
What tends to happen now, is that it's a filler and it never really does
anything, it doesn't really accomplish anything useful in terms of
communication or in terms of moving something that can help us with
understanding, improving, and developing better public health
surveillance systems. So, the examples of how this would work, I have to
say, are something that -- we're just now exploring in our own research.
Some of it -- it's kind of a matter of taking what we already know as
epidemiologists and reframing it. You might say that, to some extent,
it's something similar to the process of creating an ontology in a
knowledgebase. It's organizing the epidemiologic context that we can use
to make an assessment of the situation in a way that helps us understand
how the -- understand measure assessment, if you will. I understand I'm
being incredibly vague, but the examples -- one example I'd give is that
I think it's very important to describe the context and the perceived
level of threat. I believe there's a continuum of assessment, continuum
of belief about the departure from normality. I think that's something
that has to be considered and developed. I think that some of the work
that we traditionally do in outbreak investigation around assessment of
exposure: who, what, when, and where, I think that that can also fit
into a situation model. I think we want to define how people can explore
the difference of diagnosis in an epidemiologic sense. And, so we can
also learn from the analogies of clinical systems and how decisions are
made in clinical settings.

AARON FLEISCHAUER: Hi. Aaron Fleischauer. I guess this follows along
Bob's question. For, whatever the definition of situational awareness
is, I think we realize that not any one system will give us those
answers. It is really the comprehension of all the systems and
relationships within the institution that will provide, potentially, the
answers to that question. So, have you given the thought -- if this is
the use case for notifiable disease, have you given a thought as to what
the architecture might look like for integrating or incorporating the
information from all the systems in relationships within Utah for
example?

MATTHEW SAMORE: Okay. An excellent question, but even harder. Are you
talking about the idea of bringing together all the different data
systems and surveillance systems together, and how they relate to each
other?

AARON FLEISCHAUER: It's not that easy. We talk about the term, situation
awareness, as a big operating picture. What are all the necessary
requirements or pieces of information that would give us that answer?
So, If you have a use case for notifiable disease and you're sort of
mapping out how that information flows, what are the other pieces of
information that are necessary to answer those questions and how do you
sort of map those or integrate those or understand the unique
relationships that each of those systems would bring?

MATTHEW SAMORE: Okay. I think, again, I don't have a real, good, direct
answer to that. Let me say that -- one approach that I've taken to this
is to also start with the perspective of -- is evaluating data and
trying to determine significance, which is an inductive approach, versus
one starting with a hypothesis and evaluating the data to support that
hypothesis. I think that is one way to approach this in a general way.
Classically, the way that this works -- we say in labs -- is we say a
statistical work may be generated and the approach is to assess, for the
human to assess, the evidence that there is something really going on in
other data streams. So, to some extent, what we're trying to do is bring
out the -- pull out from the data some of the useful and important
relationships that are possibly there but not well perceived. I'll give
an example which - probably the more I talk the more I get into trouble.
We have ??? outbreak in Utah over the summer, actually late spring. So,
what they were very interested in is, again, the delay in recognizing
that and understanding also why a system conducting surveillance like it
always does, failed to detect it. One thing that potentially we can do
is to examine attributes of cases and the concepts associated with cases
at an early stage, prior to the overt recognition of that outbreak, to
see whether patterns can emerge from a different way to represent those
attributes. This kind of gets back to this mind map, mental map idea.
This is actually something, again, that our data visualization project
is addressing. The concept is actually using tag clouds to allow the
epidemiologist to navigate these various concepts to find relationships.
To some extent, there's almost too much to try to cover in a short
answer. We should have this discussion further actually.

CECIL LYNCH: Can I just speak? This is Cecil again. Can I make a quick
comment on some things? I chair the homeland security task force on
biosurveillance. And, one of the work products for that is the
aggregation of all of the biosurveillance systems that are currently in
use, and then taking those and breaking them apart attribute by
attribute very much as you've described them, Matthew, against the
concept map essentially of all the things that go into situational
awareness and looking at how each of those addresses a particular area.
The product of this is to really look at gaps that need additional
funding. So, one of the things we'll be looking at is trying to tackle
those individual elements and then contact those who are already working
in those areas and provide additional funding to satisfy those gaps. So,
I can send a paper along that describes the systems.

MATTHEW SAMORE: That sounds very interesting. I mean, I'd love to see
that paper. Thank you.

AUDIENDCE MEMBER: I'm a communications specialist in the education and
outreach team interested in buzzwords and new thoughts, as well as
trying to get some ideas for articles for PHIN news in the PHIN
community. I have a couple questions; they're kind of related. Could you
kindly share some information about how IBUS fits into your work, or
tell us a little bit more about IBIS and maybe your thoughts on open
source?

MATTHEW SAMORE: Wow. These questions are amazing, I have to say. I have
to say that the reason, partly, the reason my answers are a little bit
incomprehensive is that -- incomprehensible -- is that I only had about
two hours of sleep last night working on my talk, trying to get it done.
But, so some of these questions I'm not the best person to answer. I'm
familiar with IBIS. The system as I understand it is used to generate
data about the health of the population. It's a toolset for querying
population health data. People like Wu Xu at the health department have
much better expertise about it. Tell me you question again.

AUDIENCE MEMBER: You gave a broad overview of what's going on in Utah
and I thought that was actually something that was considered a case
study of how to open source. And, so I just thought it was interesting
that you didn't mention it and was curious how it fits in. That doesn't
matter. Really, the other question is just, any thoughts on open source?

MATTHEW SAMORE: My understanding for the open source issue is that it
takes some resources to make that happen. My discussions with Wu Xu
about it are that UT DOH is interested, but looking for some help
perhaps in making that happen. So, I think there's been some discussion
with the CDC about this -- wondering whether CDC can be a partner for
making IBIS open source. I believe this actually came up at a PHIN
meeting, but I don't know the details. Thank you for bringing that up. I
actually -- I'll raise it with Wu Xu when I see him.

ROBERT PINNER: Maybe one final question.

PETER HICKS: Thank you for your presentation. Peter Hicks from NCPHI.
I've got a comment and a question. Your observation about panopticon is
fantastic. If I remember my Foucault properly, it's tower in the middle
of a town, that it's height and dimensions are setup for its scope of
vision to protect the town. Hence, surveillance systems need to be
established and built in a similar manner. To look at situation
awareness, and to situational comprehension, using that as a model to
understand what we are supposed to know -- what can we know? -- in order
to try to document and move forward in that area. I think we're onto
something here. In terms of NLP in Utah, can you talk about processes
pulled what directions, standards?

MATTHEW SAMORE: Okay. Thank you. I'm glad that the panopticon resonated
with you. I don't know the details of the story, but like you say,
Foucault is the person to describe this the way that he did -- that you
mentioned. So, our text processing tools are basically NegX, which was
developed by Monique Chapman. She has a new tool that she calls Context,
which goes beyond simple negations and looks at temporality. The other
tool that we've tested against VA data is Medley, which is developed by
Carol Friedman. What we're planning to do in this partnership within the
VA is compare these tools with other ones such as the mail vocabulary
server system developed by Perry Elkin. We're actually very excited
about that. It would be interesting to potentially tie that into an
OntoReason type project.

ROBERT PINNER: So, Matthew, thank you very much.

JENNIFER WARD: Well, thank you to our presenter and moderator. I will
mention, there's an individual who was on the telephone portion who
mentioned a document containing information about different
biosurveillance systems that are out there. If you have any documents
such as that, we can certainly post them on the discussion forum for the
seminar series. The other participants can view that. I'll put that web
address up here. We'll be pausing our seminar series in March and April.
Our next seminar will be May 14^th^ and we'll focus on electronic
laboratory reporting. We invite your suggestions for future topics.
Please contact us at the email listed, and we look forward to seeing you
at our next seminar. Thank you.