\>\> Good afternoon, welcome to our seminar, this is the emergency
preparedness and biointerest group.

this covers the entire range of emergency and biodefense F. you think of
emergency preparedness dealing with naturally occurring outbreaks and
disease, biodefense focusing more on deliberately produced outbreaks of
disease.

we\'re going across the entire spectrum here.

what our speaker is going to be talking about is how you can notice or
pick up, discover that something is happening out there.

by an \-- in non-traditional ways, not relying on doctors sending in
reports or on these official networks.

but on other ways of noticing that something is starting to happen.

and we\'re talking about various sorts of biological events that
actually goes outside the area of infectious disease as well.

Our speaker Dr. Jim Wilson from Georgetown medical school, he\'s the
head of the Argus project there.

and he\'ll be explaining to us how Argus works.

Jim.

\>\> Thank you.

okay.

everybody hear me okay?

Okay.

all right.

welcome to this talk.

feel free to interrupt me.

feel free to ask me questions in the mid of this if you have any.

this program started about four years ago.

let me give you some history here about who I am and what was my
background and my formative years as a young professional.

about ten odd years ago I worked with the world health organization, I
started off actually at the medical school as an intern and took time
off from the medical school to spend time with (indiscernible) and who
type force project to try to understand the difficulties of trying to
identify the BOLAR reservoir.

while we were there we started taking a look at environment climatic
time series data from satellite instruments, satellite imagery archive
covering the time period and the places over which EBOLA had a period in
the 70s and 90s.

and this sort of began sort of the formulation of thought about how do
you process indications for biological events in general?

How do you look at the pre-event indicators, the perievent and post
event indicators to conditions favorable for outbreak triggering,
conditions that may promote propagation, and then of course resolution.

but during this time this led us to collaborative work with NASA and
NOAH, the world meteorological association and we began looking at how
to incorporate environmental data with the epidata and essentially look
at forecasting models and how to test these models and how to implement
them to mitigate and affect the course of outbreak response.

the problem was, it was a great idea I guess at the time but there was
no place to take this model and test it.

There was no national weather service, if you will, for tactical
detection of biological events, thousands of events a day throughout the
world where I could go to the Cheyenne mountain for infectious disease
and do this kind of work.

so I had to wait.

I hate waiting.

mike will tell you I\'m not patient.

so I spent the next years figuring out how (inaudible) so this began as
a long held dream by many.

so our objective here was to create and implement a global biologic
event detecting and tracking capability that provides early warning
letters.

Every one of those action words has a research backing to it.

We didn\'t know how to create it, we didn\'t know how to implement it or
we didn\'t know what you mean by global really.

There was a lot of questions (inaudible).

I had lots of fun asking people if lots of different Federal agency,
state and local agencies, what do you mean by early?

What do you mean by warning?

What\'s an alert to you?

What\'s actionibility?

That\'s a curse word for many folks, so when talking about this concept
in the beginning, people were terrified.

at the thought of some non-governmental entity the size of the United
States with Cheyenne type capable of infectious disease alerting people
and somebody out there quarantining a plane on the Tarmac as a as a
result.

we want failsafe, checks and balances, right?

So this became a very interesting research topic.

now, again, harking back to my history growing up if you will as a
public health-minded later pediatrician, heavily influence bid the
meteorological community, influence bid the national disaster community
and was heavily influenced by that national time period when CROMED was
created the global response network was create and the Canadian global
health intelligence network was created so in the late 90s all these
exciting IT platforms are being set up to begin this process essentially
what you see described here, this statement of how do you rapidly
identify events of interest and process them in a verification set to
queue people to greater awareness.

let\'s take a closer look.

here is what we don\'t do.

we actually had to specially make this slide because it\'s funny, once
that name Argus circulates inside the belt ways it took on mythical
qualities so we had to be clear about what we don\'t do.

we don\'t do ground troops.

We don\'t send people into the field to look at these events.

We don\'t do diagnostics.

We don\'t codo attribution investigations.

We are simply the folks bird dogging the information, turning the
information around quickly and presenting it to the user community and
the it\'s up to the user community to decide who is going to do what.

the USDA official sees this information they may not take action in
their own environment.

if CDC sees anytime their environment they may do a different thing.

that\'s up to them to decide not us but highlight it is fact and the
demand we do not function independently.

we operate like one instrument in a symphony of instruments for global
biosurveillance.

Hopefully in a complimentary and very redundant global capacity.

the heart of Argus is what we loosely call (inaudible) I won\'t go into
that here.

inundated with disease, (stand by)

\>\> We hook at explicit reports of disease, these are direct markers.

we have an outbreak of cholera, a very explicit clear reporting.

the indicators not so clear and become very important to us when dealing
with situations where folks may not understand what\'s going on even at
the local level, at the indirect indicators.

These are the supply/demand shifts and medical services.

pharmaceutical services or agricultural services.

Veterinary services.

We pay close attention to when we see reporting of lots of people coming
to see the DOCS or people bypassing the doctors to go to alternative
medicine.

traditional healers.

When that supply outstrip\'s excuse me.

when that demand outstrips the supply and now you have stock pile
depletion.

It can be for simple things.

simple things like herbal medicine that will trigger a black market
formation.

trigger people to go ranging out in the surrounding areas to seek
whatever it is that they perceive to be the pristine counter measure for
whatever the issue is.

local perception of threat.

this is an important category for us is we think that our philosophy
operationally is that the locals do have a decent sense of what is
supposed to happen in their local community and what\'s not.

it\'s not a technical expertise.

It\'s just what\'s in their gut feeling as a society.

It\'s kind of difficult to explain but if every year a particular
community see it is monsoon season come around and every now and then
they\'ll see an exacerbation of baseline cholera and now they get a
cholera outbreak epidemic and they\'re use it to it and it\'s been that
way dozens of years, hundreds thousands of years, that community isn\'t
going the get anxious if they see cholera this season in the proper
context F. they see people dying of a hemorrhagic new knownia and don\'t
see it that often and they don\'t know how it got there, that\'s a
different level of anxiety, it tends to provoke anxiety in interesting
groups of people so we pay attention to who is panicking, is it the
public that\'s a different level of interest to us than of course the
local officials.

Different level of interest than saying \-- seeing a veterinarian or a
physician being concerned about the issue.

official acknowledgement of threat.

What are the officials say something if anything at all they\'re
responding to some quirry that\'s of interest to it.

Official action against threat, this is the counter measures.

One of the things we have noticed with emerging pathogens or pathogens
with the assumption is made where you hear hoof you think horses but
actually you\'re dealing with a zebra so they make the assumption
you\'re dealing with a horse until the counter measures fail or until
the disease is inundated the infrastructure to a point where they are
forced to reconsider the original diagnosis.

then you see the oh my gosh moment.

the oh my gosh moment shows up as perception, threat and a lot of other
interesting things happen that we pick up on.

but we actually watch for the flip flopping and the public debate over
diagnosis thatch\'s actually very important indicator for us.

integrity of infrastructure.

we pay close attention to when they say things like hospitals are
getting overrun with patients, you have to push patients in the in the
hallway.

close the doors of a hospital or critical care facility in a city that
could be the primary trauma center for a city where the infrastructure
is basically strained and or overwhelmed.

When the infrastructure collapses where they can\'t take patients an
close the doors and you have exceeded local capability to respond,
that\'s very important to know.

and what this indicator system does for us is it enables us to basically
examine biological event evolution in a different way, a very
complimentary way if you will to a classic public health epidemiological
approaches.

harking back to meteorology and natural disaster we four years ago
implement ad staging system, this is a heuristic model, we had to come
one a different way describing what we were seeing in real time or near
real time rather.

because it was not appropriate in our data environment which at this
point I realize I haven\'t actually told you guys yet is online publicly
available sources, multi-lingual.

we\'ll get into those details in a minute.

if it\'s not perfectly called what we\'re seeing an outbreak.

we don\'t have case count data where we\'re so iting.

the who may not have gotten on the ground or is working with their
partners to figure out what\'s going on.

often times we don\'t have the information.

we have a media report of say children at a day care center with
diarrhea.

that\'s it.

You\'re going to go with that.

okay?

So we decided \-- we elected not to use the public health instead use a
staging system where stage one is simply unifocal biological event,
staging system is meant to be applied at city level or below but if you
have a focal event, example of that would be diarrhea at a day care
center.

a multi-focal would be diarrhea at multiple day care centers, meant to
imply more widespread.

a lot of gush, a lot of judgment calls, arguments, right?

That\'s where we\'re beginning with this.

stage 3 though, this is where we start sitting up more and taking note.

the locals are saying they can\'t handle it.

We have run out of vaccine, antibiotic, this and that.

the healthcare workers are getting infected they won\'t show up for work
because they\'re sick or afraid.

stage 4 social collapse this happens rarely but does happen with certain
pathogen, Ebola a classic example.

bubonic plague being another.

acts of bioterrorism being another one, interesting historical examples
but this is where people behave independent of society.

where they say that\'s it.

I\'m looking out for myself and my family, we\'re going to panic,
evacuate out of here and we do see that.

the interesting thing is we \-- and stage 3, observation of a country
assuming a preparatory posture.

This is interesting in that sometimes we see countries sort of humming
along and then suddenly they\'ll send teams out to farms to go look for
dead birds.

it\'s just a safe precaution, we\'re just checking.

seek and ye shall find.

If they go out looking for deadbirds because they heard a rumor on the
ground they haven\'t expressed clearly in the media and then they find
them that\'s where whole threat of reporting begin, kicks on and we pay
attention so we do watch when people start doing things that appear to
be unusual in the act of investigating.

so at the end of the day here, public health analogy sort of end at the
stage 2 for us.

\[off mic\]

\>\> So the country \--

\[off mic\]

\>\> Yeah.

the question is what if the country is hiding?

Well, what we\'re looking at is publicly available information and what
we have found is if a country is deliberately trying to hide things, if
the issue gets big enough the information finds its way out.

if the Internet has grown and grown and grown and the world is no longer
as walled off as it used to be, people talk about each other\'s
business, anymore when anybody comes to us and says hey, we\'d like to
work with you guys because we have a specific interest in this one
country and this one country\'s diseases, right?

That\'s probably just one person, right?

One linguist to look at that and they say oh, no.

sure.

the one linguist for the one language maybe the main language that puts
out the main volume of media for that country.

but it can be 12 additional languages you need to cover to get the full
scale of what\'s going on.

because everybody talks about each other\'s business.

so if one country is deliberately hiding something there\'s a point at
which it doesn\'t matter.

We\'re going to find out one way or another and we\'ll find out soon
that there are hints that there maybe a problem, not deaf fintive
though.

so we\'ll come back to this.

this information serve as a tip function, a queue function for closer
inspection.

\>\> So what are the resources then you\'re using?

\>\> Okay.

\>\> Information.

\>\> So the resources that we\'re using to get the information is the
question.

the main bulk of the sources that we have are online media sources.

so what the world health organization found pretty quickly and what the
folks in Canada found with the GFIN is online media is a valuable source
of information but not at the CNN Reuters level but you got to get all
the way down into the local grass roots paper level.

they talk about all kinds of things.

This actual site \-- this is a good segue, an example of article hit
counts.

so if you actually reference key wording for bird die-off, this was the
actual signal for the discovery of H 5N 1 in turkey.

note the bimodal appearance to the signature, this is a pattern we have
seen in several countries when H 5N 1 was expanding from southern China
into Russia and westward into east Europe, we were tactically following
that as it happened.

it was a fascinating process.

but you typically saw a smaller blip of interest in reports of mass per
dial.

so people in the global context, everybody is becoming very sensitive to
report of a bird die-off.

just like in this country with west Nile, right?

So we typically see that there\'s a certain level of interest in bird
die-offs right in that community, and then a little blip of interest if
you get a laboratory confirmation, right?

And then they get used to that piece of information, it dies down a bit
and then the first blip that somebody say there\'s a a suspect human
case, there\'s this huge flurry of upswing in reporting and you see that
sharp precipitous spike in the report feed and then it\'s sustained for
a good period afterwards.

we saw that pattern over and again, it\'s an interesting experiencial
observation is people care more about people disease than they do about
animal disease of course so it\'s an interesting signature.

we need to do more research on this, more firm documentation, more
statistical analysis of course.

but that was the actual raw output from the harvesting engine.

those are online media articles only in English but if I had done a
cross section of this in Russian or Chinese, or in some of the other
languages that we cover, it looks very similar.

\[off mic\]

\>\> No, no, no.

I\'ll actually get into that in a minute.

I have a specific slide on that but an excellent question.

this right here the retrospective case study, this highlights the vision
of what we\'re trying to do here.

we\'re trying to create a capability using a harvesting engine that is
customized to our purpose.

that tries to look at society in \-- using an EKG like approach.

so I was thinking with my pediatrician hat on I was to see a 12 EKG on
this city.

and what are the leads?

Okay, so in this case this is the actual tracing of the 1957 pandemic.

in Hong Kong.

and what\'s interesting to us is to look and obviously this is a
retrospective case study and done painstakingly manually, there was no
harvesting engine, it took months to pull this together.

but what we\'re interested in looking at is the different categories of
indicators, the different categories of social disruption and what did
this signature pattern look like and what was the EKG strip looking like
here?

It\'s not meant to show you anything statistically significant but meant
to highlight the concept here.

it was a fascinating retrospective to look at this.

this particular slide actually, these preliminary case study findings
have led us to look at a lot of other times and places where pandemics
in a similar \-- pandemics in a similar matter what we\'re trying to
understand as a team how does local media reporting look like a year
before the pandemic so seasonal influenza like illness versus pandemic.

what are the critical reporting differences?

There are differences, there are far more subtle than I would have
hoped.

I would have hoped for more drama but it is a very difficult problem.

our mantra of court is if you can\'t measure it how do you manage it?

After near real time measurement of basically the pulse of the planet.

so the earliest event detection, high sensitivity tracking, monitoring
response therapy, that\'s where we\'re interested in people saying the
vaccine doesn\'t work T vaccine for whatever disease.

we report that out.

sometimes there\'s something to that.

people might find interesting.

we saw a lot of that in the prior season of influenza.

notice I\'m not saying that that\'s verified information.

this is what is showing up in local public reporting.

actual advisors, there\'s that curse word again.

let\'s explain that.

in our environment actionibility is simply sensitization of the network.

if I tell you all here today that we now have laboratory confirmed Ebola
in Uganda, most folks in this crowd are not going to do much more with
that information.

just an interesting academic FYI and that\'s the end of it one or two
people might go back to their computers and send an email that triggers
a chain of inquiry.

that then ultimate lit gleans some specificity, some more information to
be retrieved.

that\'s what I\'m talking about.

the ebb and flow of sensitivity and specificity in a surveillance
network.

Not quarantining aircraft on a Tarmac or anything drastic like that.

that\'s not what we\'re interested in here.

the idea here is that when you have a a vast network of surveillance,
ideally that network by its very nature will have controls, will be
checks and balances similar to a WIKI like phenomenon.

\[off mic\]

\>\> We\'ll show you that.

so the question is why is the word advisory on there?

We\'ll actually show you and tell you what that is in just a second
here.

so system stats.

We use a harvesting engine, it\'s not a Google.

it is an engine that is specifically designed, very painstakingly to
look at specific sources of information on the Internet that we find
credible.

in about 18 languages, they\'re able to be handled by the computer
system.

we have 34 languages total covered by human analysts.

and about we\'re actually approaching closer on the side of 8,000
sources now.

but these are hand-scripted sources.

they other indexed in the native vernacular, retrievable with multi-year
baselines so you can select by source, by country, by region, how many
times was the word panic used, in the native vernacular?

That\'s important because then you can try to assess the different
levels of use of that word.

some societies, their media reporting is more riddled with hyperbole
than other societies.

Other society is very reliable and our media is dead on target every
time.

That\'s a joke for those of you on the Internet right now.

so we estimate we\'re accessing a million articles a day.

we ingest about a quarter million articles every day into the archive.

what that does is enables us to examine a baseline.

we have 36 full time equivalent analysts fluent in 34 languages these
folks do nothing but analyze information all day long every day and have
now become experts in the data environment they\'re operating in.

we have global coverage.

we have actually now approaching 90 million in archive.

the slide is a little outdated.

On average we have 2,200 active case files on our watch board.

Show you what that is in a moment.

the watch board is basically the repository of the live tracking data
that we have on our so called radar screen.

put that in quote on a daily basis so it can be everything from a bird
die off to cholera or to Ebola or influenza or respiratory disease.

so the disease entities that are tracked sometimes they\'re named
explicitly and sometimes not.

It\'s an dromic.

but when we first started this project we thought that at the end of the
road when we had full operational capability we thought there would be a
thousand active case files on the watch board.

we hit that number two weeks before initial operating capability, 60
days into the contract we hit a peek load of 3,300 this past influenza
season.

we had no idea what we were walking into quite clearly.

yes, sir.

\[off mic\]

\>\> The question is there some mechanism to verify accuracy of the
information.

we are most definitely going to get into those.

I have a slide on that too.

it\'s a very important question.

yes, sir.

\>\> Seems that this approach is one where you\'re kind of sifting
through looking for information that is indicated a certain event.

you said you guys don\'t put people in the fields and stuff like that.

but for certain diseases there would be definite survey reportable
diseases, at least (inaudible) \--

\>\> Sure.

like influenza.

\>\> Right.

how do things do you encorp rate those kind of \-- incorporate those
kinds of data in and how is and approach like this compared to one
that\'s much more targeted and systematic for a specific \--

\>\> Long question.

let me rephrase that.

do we have any studies on how this approach compares to long,
well-established more traditional systems of surveillance, is that a
good way?

I have two slides on that coming up.

excellent question though.

and that question will be asked for years.

we will never stop doing these kind of validation studies of exactly
that ilk.

we have generated over 43,000 reports to date that to our knowledge this
is the highest volume capacity capability with the highest degree of
geographic coverage of any system of its kind in the world.

Our full operation capability at the end of the day will be expandable
to about a half million articles a day.

archivable.

this is off the harvest engine, expandable to 10 million a day that\'s a
theoretical limit.

so to talk about the operations flow, again, we\'re looking at about a
million articles a day in 34 language, resulting in 30 advisories and
then we have to explain what we mean by advisory.

so at the top, that top box, raw media articles that\'s where your
harvest engine is.

the engine has key worded clusters in it so it\'s automatically trolling
and looking for its retrieving information based on key word in
clusters.

There are over 200 clusters it\'s operating against.

they are cross matched to all the indicator sets.

that is one level of filter.

the next level of filter is the junior analyst who then uses a long list
of reporting requirements that define the granularity of a biological
event.

so this is where you would pick up on a report of a bird die-off.

a bird die-off can be due to anything but we report that out because of
possible association to H 5N 1.

that results in a detailed report.

it results in case file.

that case file looks very similar to how it\'s done in PROMED where you
get the initial report and that begins a threat of reporting.

\-- a thread of reporting.

and you can look at all the updates but that report gets passed through
the work flow to a senior analyst.

the senior analyst does quality control on the reports and says met all
the reporting requirements in the quality control on how the document is
written, and then they\'re also assessing that through another, a third
set of reporting requirements to determine if we should issue a warning,
watch or advisory.

I am going to get to that, what that means.

and then that information is passed to the user community so the two
levels of data visualization or data viewing is the high level warning
watch advisory level and at the granular level, if you want to dive all
the way down to some obscure province, some obscure location in the
world you can do it to your heart\'s content, all the way down to the
weeds and look at all the updates.

upper left hand corner is a picture of the computer cluster.

we do not have lasers yet.

I wish we did.

but we just don\'t have them yet.

lower left hand corner is the actual picture of the operations center.

nobody here likes lasers.

I\'m very kiss appointed.

\-- disappointed.

Nobody is laughing.

late in the day.

everybody is slipping on the coffee here.

right hand side is picture of morning report, that happens every
morning.

so we have 36 analysts gathered together.

and they review the critical indicators the last 24 hours or over the
weekend when they were last in the office.

This is a synchronization point.

we\'re interested in group\'s synchrony around certain issues because
these issues get reported in a multi-lingual environment.

our world doesn\'t operate any more in a monolithic linguistic
environment by country, everyone knows each other\'s business.

\[off mic\]

\>\> It\'s actually a proprietary system developed in collaboration with
MINER corporation.

this was a system designed by analysts for analysts.

the beginnings of the platform were originally developed under
(indiscernible) program.

but this engine is something meeting our specific requirements to do
what we do.

so it\'s a very unique system in the world.

\>\> You find that you\'re searching such a large volume \--

\>\> Sorry, the question was asking about the harvesting engine and
particulars of that.

sorry.

\>\> In terms of what you\'re searching with such a large volume is
there a cost benefit ratio that after a while though you\'re searching
so much you\'re not finding anything more?

\>\> Actually that\'s \-- so is there a limit to the number of sources
or the number of \-- or the volume of information that you can pull down
before you just don\'t get much more added value to the information?

Actually the answer to that question, we actually thought we were
hitting until recently where we thought \-- and it was a mistake on my
part but we thought we added a thousand sources and didn\'t see a huge
jump in reporting output as a result that we had expected.

we went back and recalculated.

turns out that wasn\'t the case, that wasn\'t the limit yet.

but we would expect you\'re going to hit some unknown number of sources
where you can just keep adding and adding and you\'re not going to get
closer to adding more value to the information.

and I have to be honest with you, we haven\'t figured out what that
limit is yet.

and I personally am not aware of a standard index or metric for
measuring that.

assessing that.

so it\'s very interesting fun.

I will tell you in qon dong province in China, force years ago when we
did some research studies on that area because of SARS, that was the
event that really kicked off this project, we found 55 sources and we
arbitrarily stopped at 55 and said good lord after 55 sources that ought
to be enough but it\'s arbitrary.

should I go to 100 sources, 150?

We had to figure it out.

The for us, sourcing never stops.

We have an entire team of people who do nothing but script in sources,
the analysts are required to input new sources.

Sources come and go.

source credibility is a moving target.

it\'s credible this year, it slipped the next year.

one year they\'re giving us credible topical coverage of diarrheal
disease of great importance to our analysts, the next year they want to
focus on Anna Nicole Smith.

I\'m sure important to some folks but not to us.

\>\> So you\'re looking at media and other kinds of things?

Are you looking at what\'s coming out of the health department from
these country, ministry of health \--

\>\> We look at official \--

\>\> \-- protection?

\>\> The question is do we only look at media sources or other sources
too?

We certainly look at official websites, official reporting sites.

we do not in contrast though if it\'s public health website it\'s like
putting out a more of an epidemiological output of information like the
CDC\'s influenza page that talks about the percentage of ILI we\'re
seeing this season.

we do not do that.

we will do that internally though just so the analysts are able to
understand, okay, is this appropriate, is the social structure reporting
that we\'re generating here, is that matching up nicely with the epidata
or not, that is an evolving area.

we need to blend that information.

we do pick up on the official public health sites.

We do use PROmed as a source, Google as a source.

we use anybody online as a source.

I want to \-- actually I\'m picking up on a thread here I need to
emphasize not only to y\'all but to folks out in the virtual audience
here.

this talk is not about and it platform.

I really \-- this is a huge point.

is if we think of this surveillance effort as being a Google, that is
absolutely dead on the wrong way of looking at this problem.

this at its fundamental core is the birth of a new professional
discipline.

when I grow up some day to be this.

okay?

There maybe some skepticism at the moment but I can tell you on a daily
basis these analysts are doing things that I have never seen in my ten
years of doing this work, these people are doing something different.

it is an epidemiology, it has flavors of it certainly.

and do they need to become more rigorous and maybe have more vigorous
training in epidemiology?

Some day but we\'re seeing a different animal emerge here.

bottom line, you have to answer the so what.

so if anybody out there comes out with and it platform that claims to do
global surveillance, I\'m going to stand here and I will be honest, if
you\'re listening out there in the virtual audience you\'ll hear me
coming, my question is going to be so what?

Who is using the information and what are you doing with it to change
anything?

If you can\'t answer that question then let us all come back to the
table and let us discuss why we are ear doing this.

that\'s a fundamental question.

on a sort of more granular level is who is looking at the data and what
expertise do they have to look at it?

Okay.

so I will tell you infectious disease expert in our operation center
actually does not function very well in the tactical operations
environment.

they don\'t understand the nuances of run of \-- route in some province
in China.

what that means and what the differential diagnosis for that is.

it\'s akin to it and we welcome that expertise in our center but it\'s
not used on a daily basis, it\'s used as subject matter expert back up
support.

if we get enough information that says we have enough interest and the
locals saying it might be Ebola, hand off time to I guess the \--

\[off mic\]

\>\> You said emerging discipline this, what would you call this?

Besides analysts?

\>\> The question is what do we call this emerging discipline?

Excellent question.

we haven\'t named it yet.

We sort of euphemistically \-- we would call it operational sociology
but it\'s not \-- it is \-- I mean T problem that we have here is we
start off looking at this as a biosurveillance complimentary tool.

it\'s actually incredibly useful for detecting a lot of other things
that are interesting like industrial accident, alleged nuclear accident,
acts of terrorism, anything that generates social disruption.

what do you call that now?

We are on a roll, I was about to call it something related to
biosurveillance when we discovered \-- it went out the window and it
wasn\'t going to be that simple.

This is a snap shot of watch board.

we have over 175 users from 100 organizations.

The bulk are Federal only.

we do have test users at the state and local authority level.

Colorado is a test state.

so is the District of Columbia.

we are very careful about rolling this out prematurely or sort of
involving too many people, the reason why is we are very interested in
understanding very clearly how people are using the information.

I don\'t want to just blow this out into broader national community,
prematurely.

I think it takes time to understand this.

of course we stand on a balancing act of our ethical responsibility to
share this information with people because sometimes it\'s important.

really important.

potentially important.

of course what\'s important to me may not be the same level of
importance to an agriculturalist, a veterinarian, to human physician, to
public health officer to somebody in San Francisco, somebody in New York
City, to somebody at CDC and somebody at NIH.

I can\'t control for that right now.

but this is are where we\'re at.

\[off mic\]

\>\> Yeah.

the analysts are actually they are able to understand, the question is
do the analysts have a mechanism for identifying duplicate reports?

The way the reporting is in our system is we see one nucleus report that
will be copy pasted repackaged, reworded, sometimes frankly plagiarized
sometimes not and we call that simple amplification.

It\'s a trick to sift through because what happen is one random
journalist will have that copy pasted article everyone else is using and
then add a paragraph that adds a critical detail and then boom, we\'re
off and running.

\>\> On the other side of the coin, I don\'t know why people would do
something like this (inaudible) if there\'s a terrorist wanting to
suddenly flood a million reports of some bogus outbreak, can you monitor
that as well?

\>\> The question if a terrorist decide to deliberately flood the
Internet with false report consist we pick up on that?

We can pick up on the blip of reports if they cross matched the
reporting requirements that we\'re looking for.

but as certaining \-- determining the attribution and verification step,
we\'re going to get into that in a minute because that gets at the other
question that we had here about the verification step.

this is screen shot of an advisory page.

now, what\'s an advisory in the watch board?

Which is this screen shot here.

why is something highlighted in red?

This is an actual screen shot from about nine months ago or so.

not quite that long, June 2007.

the reporting requirements, what are the reporting requirements?

That was a fundamental question actually.

so if I give you a system that has 2,200 live tactically evolving
biological events affecting animals and human force the entire planet,
what\'s the priority?

Actually, mike and I had an interesting conversation about this before
we started this seminar.

it is a huge question.

so if I tell you there\'s Ebola today and the city that\'s completely
innone dated with an respiratory disease that\'s not communicated yet to
who, and over here I have cholera, which of the three events is more
important if you could only pick one to pick up the phone and engage a
social network to engage in the verification step which one you\'re
going to pick?

I can tell you that we have an interesting debate, at least two of
those.

maybe not the cholera one but then you never know.

who cares about cholera.

so I guess whey I\'m getting at here is everybody has a different
prioritized list in their minds whether you\'re a researcher or more on
the operations side of public health or a clinician.

everyone has a different emphasis but this particular of the data is
sort of the top level cut of the things most interesting to what we call
the biosurveillance indication warning analysis community.

this is a subset of operational biosurveillance analysts in the Federal
Government who span CDC, USCA, department of Homeland Security,
Department of Defense.

and these are folks that over the last ten years that I have been doing
this, I found to be very team-oriented.

They share information.

they do it for the greater good for the country.

and they\'re very good at working tactically.

and working in the near real time space and we function kind of like
switch board operators.

we\'re not the subject matter experts of Ebola virus.

we\'re not the \-- I\'m not Dr. Peters, not Dr. jaring of that elk or
Dr. Brake but you need people who understand that kind of data
processing environment and are good at it.

Good at sort of pulling on the strings of the web that you need to get
enough pieces of information together so that, okay, people are all
agreeing this is going to be a problem, we need to respond, whether
you\'re CKC, USDA or whomever.

what you\'re seeing is an integrated set of reporting requirements we
drafted when I was at the department of Homeland Security.

it started with an integrated disease list so y\'all have seen the CDC
bioterrorism agent list.

no doubt, this is a biodefense seminar.

but there are other lists.

USDA has a list.

the crop folks have a list.

the animal disease folk versus a list.

the lists look different than the CDC list, look different in the
beginning we tried to integrate the disease list and then tried to
prioritize them.

We got in a room and argued about it like they did with the original CDC
list and tried to figure out one of the folks involved in creation of
that list was involved in this list.

there\'s nothing particularly scientific about it just trying to figure
out what are the top agents the most?

Is Ebola the top agent we\'re going to fuss about today if we see an
event involving that or should it be that city innone dated with the
respiratory disease and we don\'t know what the diagnosis is?

It\'s a big debate, a big question, people have to check in their
subject matter expertise ego at the door because we have to look at this
at the orbittal level.

we have one shot and how are we going to do it?

It\'s tough.

after we prioritized this list, it\'s based on which of these disease
entities interest of interest in the United States could induce a
medical grid black out so to speak, take down an entire hospital grid?

One case of Ebola accidently entering the United States from some
exposed individual coming from DRC or whatever I don\'t think is going
to collapse the entire urban medical grid of the district.

it might be a public relations nightmare for the hospital involved
certainly, okay, I\'ll grant you that, but what could bring down the
whole grid?

Our top interest and concern of course of the respiratory viruses that
have an appropriate transmission efficiency and for which we don\'t have
(indiscernible) more, et cetera, we could go on for hours about this but
what we do is the bio\-- this team of analysts we have gotten to know,
we meet about once a business quarter and go through these indicator
lists line by line and ask if there are revisions needed.

over time we will anticipate that there will be a lot of robust
scientific data and research behind these recommendations eventually.

but you have to start with some place.

That\'s where we\'re at now with this.

so experience is guiding us at this point.

okay.

this is a \-- everyone has got a platform, everyone has a Google earth
platform to show off, this is our version.

It\'s based on open source code NASA whirlwind.

We developed this in collaboration with a company called intel sense.

this is situation awareness ring or construct, the inner ring here the
most recent events, and each of these sectors here is a specific
locality.

At the prevention level in this case.

This is this week, an event involving this week, last week, two weeks
ago, 3, 4, 5 weeks ago.

folks on the Internet, sorry you can\'t see my finger pointing at this
stuff but send me an email and we\'ll try to explain it to you.

but when you scroll this earth around with your mouse it\'s a dynamic
pointing system, just a geographic visualization capability it\'s a
prototype.

in this particular case this is a screen shot of what we saw was
widespread infrastructure strain due to influenza like illness in the
southern cone last season.

as you can see, things were hopping.

the more dark color the pointers are moving from yellow to dark red is
basically that\'s our staging system.

the block in the upper right hand corner there, that\'s an advisory bar
so if you hover your pointer over that it will kick out a text box what
the advisory is for that particular country.

\[off mic\]

\>\> So that advisory in Argentina was posted because of substantial
social structure related to respiratory disease at the time.

we were following closely reports of H 3N 2 generating quite a business
of disease, quite a bit of infrastructure strain.

we understand now the vaccine for the southern hemisphere has been
updated.

we have to look at this phenomenon for many years to come.

we don\'t have a 20 year baseline with.

This this is just the beginning.

but here, this is a six month cut of data off the Argus system at a time
when we had just surpassed the initial operating capability so we were
nowhere near full operational yet.

The Y axis, you\'re not meant to be able to read this, I\'m just giving
you the big picture but the Y axis are 130 disease entities, everything
from pneumonia, to Ebola to cholera, et cetera on this Y axis to so each
line is these.

the X axis are 174 countries who happen to be reporting things at that
period of time.

so the horizontal lines there, the upper one there those are all
possible indicators of cases of avian influenza.

every time we got a case of a bird and it was dead that\'s why they\'re
sensitive.

they use sensitive in quotes because we don\'t have the stats yet on
that.

the line here towards just shy of the midline here is respiratory
disease throughout the world.

That was everything from pneumonia to influenza, RSV or ILI as it was
expressed and actually just as a side note this past influenza season we
issued 3,000 reports in \-- for 127 countries in 28 languages, 300
advisories and 3 red alerts.

The alert again is not alert to spin out the world health organization
but just to gently ping the network \-- encourage people to be aware of
certain situations, encourage people to be tipped to look for closely.

okay.

when you look at this Matrix here, the vertical lines, okay, tell us the
ecology of reporting of that particular country.

what\'s interesting about that is that is an entirely different baseline
than what you see in PROMED or what you see at who\'s data or what you
see at NOIE or FAO\'s data.

this is a very different environment.

is it better or worse?

I would say it\'s complimentary.

definitely a heck of a lot more volume.

no question about it.

but there is a lot more information in here.

we\'re still trying to pick this thing apart but it\'s very encouraging
to see we have a lot of granular access to information here.

now, for some of these vertical lines it\'s very dark.

it\'s dark because they simply don\'t have that disease in the country
to report versus they have a very poor public health infrastructure or
poor infrastructure support society or they\'re closed society that
don\'t like to talk about it at all.

so you\'re able to look at this data in a lot of interesting ways.

Do we expect to see this kind of reporting in this area?

We asked this question every day.

so here.

okay.

so somebody is asking a a question about validation and comparing to
other methodology, more classic systems.

what you\'re seeing here, and I apologize I can\'t go into overdetail
about this but I\'m showing you a very preliminary data but just to give
you a flavor and idea where we are going with our research.

just looking at what the data time series looked like.

This was the past influenza season.

Argentina.

the red line here are \-- is a converted curve for just the sheer bulk
of influenza A positive isolates submitted to the global influenza
surveillance network.

the green line other is the reported assessment of how bad was the
occurrence of influenza in that country.

was it sporadic, widespread, okay?

All right.

the blue line here is the Argus stating system.

so here in this case at this point in time we were just finally getting
\-- we had enough analysts on board to start looking at the southern
cone so we were kind of flat lined in this several weeks before this but
at this point here we jump in the middle of stage 2 reporting.

Multi-focal respiratory disease.

nothing \-- didn\'t look too hot and sexy in terms of social disruption,
just standard influenza like illness being report and we saw a blip of a
unifocal event here at week 5 and then from week 13 culminating out into
weeks 21 to 28 here we had reporting of infrastructure strain due to
influenza like illness.

I look at this and I see possible complimentarity.

it is a different staging system.

it\'s sporadic versus regional widespread.

that\'s sort of assessment scale is more based on how widespread is the
disease geographically if you will.

our system is more based on social disruption and infrastructure
inundation.

they\'re complimentary.

what\'s interesting to us is the next slide.

what happens if you don\'t have who data?

In this case Venezuela, maybe this is influenza, maybe this is not but
we had reporting of respiratory disease from weeks 50 to 29 where there
was infrastructure strain reported.

at various time points during the season but there\'s no who to compare
it to.

is that because the Venezuelan government didn\'t share information at
that time frame?

Is it because it wasn\'t influenza?

Is it because the network was down?

Flu net hadn\'t up loaded the data yet, there\'s lots of questions
obviously you need to run down and cross check and so forth.

we do know this, we covered 46 additional countries that don\'t show up
in flu net.

what is actionable?

Okay.

so these are some questions that we have asked ourselves to help us
prioritize a little bit.

This is clearly from the human health standpoint.

whole different set of questions on the agriculture side.

and yet another set of questions on the veterinary side.

but what agents can compromise and collapse in American ICU?

What can induce complete ambulance diversion status?

What can do that at a urban grid level?

Is this flight connected by U.S. by \-- we have non-stop traffic
database but we don\'t know what the data looks like if you\'re going on
or bits.com and do four or five lay overs coming back from India.

lord only knows.

has complete ground truth been obtained?

We\'re actually sort of pushing right now for all of us to come up with
a standard scale for ground troops.

It is different to have a promed volunteer submit a report who is on the
ground saying yes there\'s mass panic amid rumors of a epidemic.

specifying credibility than \-- picking on Dr. Bray, Dr. Bray on the
ground saying I\'m standing here and I have the PCR, whatever, it\'s
different level credibility in specificity.

he being on the ground is going the hapless frequently than of course a
promed volunteer.

both are important data inputs.

so I\'m not trying to devalue one or the other, it\'s just \-- it sits
on a spectrum of specificity and importance and sort of reliability.

has containment been lost?

What do you mean by lost containment?

We can debate that.

right?

What I mean by that is are the locals saying they have lost control of
the situation?

Are counter measures available should the agent enter the U.S.?

We don\'t know what\'s going on the ground and we\'re starting to see
people talking about possible similar symptomology showing up on the
west coast from Asia, we have a problem.

here is where I told you about the good, the bad, now the ugly of what
we\'re doing.

the ugly is simply this: We are so connected throughout the world now by
non-stop direct air traffic I have little confidence that in cases of
highly efficient respiratory transmitted disease that we can engage in
adequate ground troop fast enough to understand what\'s going on on the
ground in the native country involved before that entity jumps in a
transportation grid and is already here inside the United States before
we realize it.

I think SARS was a very poignant example of that.

I think that will repeat again.

here we are certainly hitting limitation of what Argus can do for
people.

and so this is again coming back to the questions of verification.

Argus is a instrument and a symphony of instruments of different
specificities, different levels of access to ground we must partner with
NGOs, with CDC, with USDA, we must partner with state department, USRAD,
foreign partners the global influenza consortium the Gates and Google
foundation, everybody and anybody who has a global network that is
what\'s needed here.

of course managing that kind of noise, that will emerge from that big of
community of users, that\'s going to be a challenge.

but Argus is in \-- doesn\'t answer all the mail here.

it has to work in concert with other capabilities.

but it does function in the niche, the niche of providing a tipping
function, the function of Argus is to prompt analytic swarming.

so this is a cut of the wild fire capability.

this is again how we engage in verification loops.

wild fire is a sub for at that time inside the watch port.

Only few users have access to it.

This is some examples here.

In this case these are Federal users and red alerts and orange alerts
populate this sub portal.

the point is to engage people in a sort of subset secured environment
HTPS online secure environment to have some confidential conversations
about okay what do you know, what do you know, do we all need to worry
about this as an integrated analytic community or not or do we need to
take a step further, do we need to range outside of our social networks.

So this is become a synchronization point.

what\'s interesting is this works a lot like the GORN.

it\'s unofficial, nothing official about it, there\'s no MOUs or MOAs,
it\'s a gentleman\'s agreement, voluntary, you can participate if you
like or if you don\'t like.

your choice.

if you\'re CDC or USDA or \-- I mean, if this were established for a
state or locality or NGO or what have you, doesn\'t matter what the
group is, you volunteer the information as you see fit.

what\'s interesting is because it\'s not official it functions pretty
doingON well, it serves as a catalyst for season crow recognizing
everybody on the same page with the event.

sharing information quickly so there\'s no miscommunication, and people
dropping off on their own relative \-- respective official environment
it is carry matters further or to end the discussion there, once we
decided it\'s not an issue of concern.

it\'s a test.

it\'s an experiment.

okay.

so for Argus, it is an event detection system, not an outbreak.

tool per se.

so this is a different complimentary approach to identify biological
events.

for us the system has been reduced to practice.

However, the capability most deaf hitly needs to be socialized with the
broader community of public health researchers and society at large.

no doubt about it.

lots of research ahead of us.

years of research ahead of us.

lots of validation studies.

we have a lot of collaborative engagement with the Federal Government.

They have given us some fantastic feedback.

I got to tell you that the Department of Defense and CDC and USDA and
department of Homeland Security have been fabulous giving feedback on
this long process.

as I mentioned before it has broader application potential for other
social disruption topics.

We have a crop disease taxonomy that is ready for operational testing
and shows high degree of promise.

we believe the staging system will be nearly identical for the disease
as it is for human an animal disease.

skipping through a bit.

One question we have right now is this capability only works for
countries outside the United States.

we do not have a domestic capability yet.

We weren\'t funded to do that.

so of course the question of the day is what happens when we start
looking at domestic data too?

We obviously need to do that because disease knows no borders, right?

And we cannot tell you how many times we have chased disease to our
borders only to not know what happened when it may or may not have
crossed the border.

nice to know that.

we have designed training packages we have been deploying for Federal
state and local analysts.

We have established a tract within a Ph.D. program still pending
development of the overall Ph.D. program which I can discuss with you
offline.

community of interest has been established before (indiscernible) and
the BIWAK is a group of folks participating for the greater good of the
country here.

and I have E plained project wild fire.

acknowledgements.

Major kudos to our valued partner in the miner corporation.

they have worked for us for many years.

Very valuable partners and of course the Federal Government.

and private industry and academia.

and thank you, y\'all for listening to my talk.

and if you have any questions actually for those of you on the Internet,
you can \-- I\'ll leave up the title page here that has my email address
and you can shoot me a note.

if y\'all have \-- I know I\'m over time five minutes but I can flip
into the life demonstration of the watch board if you would like.

let\'s go for it.

any time limb facetor the webcast?

\-- limitation for the webcast?

Those on the Internet hope you can see this as well.

Let me click a button here because I\'m sure I have timed out.

occ.

so this is the watch board \-- okay.

this is the watch board portal.

and this is the home page of the watch board.

simple flash map.

each one of these blue points is an actual thread of reporting that is
live in the watch board.

what determines live reporting is fairly arbitrary.

we find that our sources are very good about reporting things on the
uptick of evolution.

They\'re very bad about reporting resolution.

so you\'ll see this precipitous drop off from pouring irbecause Anna
Nicole was doing something that day and they ran off to cover her and
they come back around so we find that arbitrarily we just leave things
on the watch board six weeks.

if we don\'t hear anything for six weeks we pull it off so that was just
an experiencial limit there.

anything flashing on that map, you can see dots flashing are things
updated the last 24 hours.

a lot of dots there.

you see that in Canada here, you have a question about what\'s going on
in Canada, the dots are more widely spaced the dot is simply over the
province so we had to pick an administrative standard \-- geographic
standard to adhere to and we chose preinventoryial for just the
simplicity of this map.

as I showed you before we have other more sophisticated tools to use in
the world wind environment.

but here looks like we have several residence in long-term care
facilities explaining influenza like symptoms.

Looks like the local authorities place the facility anybody here who is
interested in hoof mouth disease?

We have a hoof mouth disease EFFICIANADO here.

we have one on Ebola, I promised mike I would click on this here.

here we here is the \-- you can see we have been issuing advisories on U
Uganda since the 20th of November undiagnosed, mystery disease, this is
often how these threads begin particularly in Africa.

It looks as though we have poor confirmation of Ebola.

CDC official engaged in press statement here looks like.

then as senior analysts note below there, explaining why we care about
this in our analytic group.

it\'s not an assessment.

that\'s a really important distinction.

this is not a finished assessment.

it is simply us telling you against reporting requirements that have
been vetted by our core Federal users, the gist of what\'s going on.

the fact that CDC is involved in this is certainly reassuring and speaks
to verification and validation.

Okay.

let\'s now go back.

so this is where your tip of the iceberg.

as I mentioned before the red and orange alerts get punted to a sub
portal called wild fire which only 11 organizations have access to and a
handful of people.

I can\'t click these threads so forgive me but you can see that for
instance in this case we saw a report of rip valley fever in
(indiscernible) and we didn\'t recall that area had ever reported that
before so that has spark ad discussion.

obviously you can see here for RUGANDA that sparked some discussion as
well as Sudan.

so it\'s a thread of discussion and very limited discussion, very
focused topics of discussion.

here under reports, if the user community has a very granular interest
in a particular country, then you have the ability to drill and drill
and drill down as far as you want to go.

so for the audience, pick a country.

\[off mic\]

\>\> Canada.

we have a lot of interest in Canada.

are you Canadian?

\[off mic\]

\>\> Sorry.

(indiscernible).

okay.

so we have some reports of multi-focal respiratory disease.

stage 2, multi-focal.

not going to wake up anybody out of bed over this.

down the weeds.

okay?

I guess we\'re to me the next exciting step to take research wise is
with rapidly involving healthcare systems we have some hospitals in
America have electronic patient records.

and sophisticated it infrastructures to support the full electronic
medical record.

here is what\'s interesting.

during SARS we didn\'t have much capability sort of install a safety
system if you will for situation \-- so it\'s based on a manual \-- if a
triage clerk sees someone come in threw the emergency department you
have to check a travel history manually.

if a clerk forgets or there\'s no hard coated system in place that you
cannot admit a patient to the hospital if you haven\'t checked that
box,if that box was positive for yes I just got back from Hong Kong or
China, whoa, let\'s just pause a minute and see what\'s going on with
this patient before you admit them all the way to the ER pod to he can
pose whoever as we saw in Toronto some patients made it to the ICU
before somebody asked that question.

where has this patient been.

it\'s a human process, we\'re all human beings.

I\'m a pediatrician and lord knows it\'s busy as heck being a physician.

what I would like to see happen is an experiment where we hook this
thing up automatically to a hospital infrastructure so hospital coating
screens to register a patient has the screen in there that the triage
clerk cannot get through to register the patient if they don\'t have the
travel history question.

if they check that box and the drop down men knew matches up with a
report of an advisory or what have you to be determined of course.

that is the research question is what you show them.

but imagine some day that this kind of information was available
immediately to a physician in any emergency department anywhere in
America, anywhere in the world where they could say hey, I noticed
automatically in your electronic record here there\'s a screen flashing
and saying you just got back from Taiwan.

Taiwan is reporting this horrendous epidemic of whatever.

so that physician\'s differential diagnosis thinks to consider a
possible zebra.

they don\'t normally encounter every day.

so these are the questions we\'re asking now.

this kind of granularity may enable that process to become a reality
some day.

intriguing thought.

anyone want to pick another country?

Let\'s pick an exciting place.

of course, everybody likes to see what is going on in India.

live, right now.

and I pick on India simply because they are very good reporters of
disease and it\'s just they\'re very good about talking about these
issues.

there are a lot of things going on in India.

we can pick a lot of different countries.

This of course, coverage, this raised the question of coverage N. these
countries it is dependent on the population density, you have to have
human beings who live in the geographic area who are connected to a pair
grid, who are connected to online reporting systems.

if they\'re not that will represent a black hole for us.

So so middle of the GOBY desert you don\'t see the same density of
lights you see in India.

so we get a lot of reporting in India.

there is some degree of reporting bias probably no doubt as a result.

its neighbor Bangladesh has a remarkable list of issues as well.

oftentimes similar issues to neighboring provinces in India.

so it\'s just you could just spend your whole lifetime looking at this
data and just looking at trends and asking all kinds of very interesting
questions.

very fun place to work.

and see this kind of information.

so with that I think I will conclude my talk.

any questions?

\>\> I think we have had more questions dug your lecture than we have
ever had \-- (inaudible)

\>\> Really?

\[Laughter\]

\>\> Okay.

thank you all out there on the Internet.

we raise our cups of coffee to you and thanks.

\[Applause\]