1
 Mycin- Stanford

 Doctoral dissertation of Edward Shortliffe designed to 

identify bacterial etiology in patients with sepsis and 

meningitis and to recommend antibiotics

 Had simple inference engine and knowledge base of 600 rules

 Proposed acceptable therapy in 69% of cases which was better 

than most ID experts

 Never actually used in practice largely due to lack of access 

and time for physician entry >30 minutes

 Caduceus – similar inference engine to Mycin and based on 

Harry Pope from U of Pittsburgh’s interviews with Dr. Jack 

Myers with database of up to 1,000 diseases

2
 Internist I and II – Covered 70-80% of possible diagnoses 

in internal medicine, also based on Jack Myers’ expertise

 Worked best on only single disease

 Long training and unwieldy interface took 30 to 90 

minutes to interact with system

 Was succeeded by “Quick Medical Reference” which was 

discontinued ten years ago and evolved into more of a 

reference system than diagnostic system

 Each differential diagnosis includes linkes to origin 

evidence to provide meaningful use of EMR’s and supports 

adoption of evidence based medicine/practice

3
 Dxplain used structured knowledge similar to 

Internetist I, but added hierarchical lexicon of 

findings

 Iliad system developed in ‘90s added probabilistic 

reasoning

 Each disease had associated a priori probability of 

disease in population for which it was assigned

4
 ISABEL uses information retrieval software developed 

by “Autonomy”

 First CONSULT allows search of medical books, 

journals, and guidelines by chief complaints and age 

group

 PEPID DDX is diagnosis generator

5
 Acute cardiac ischemia time insensitive predictive 

instrument uses ECG features and clinical information 

to predict probability of ischemia and is incorporated 

into heart monitor/defibrillator

 CaseWalker system uses four item questionnaire to 

diagnose major depressive disorder

 PKC advisor provides guidance on 98 patient problems 

such as abdominal pain and vomiting

6
 They aren’t integrated into day to day operations and 

workflow of health organizations and patient information 

is scattered in outpatient clinic visits and hospital visits 

and their primary provider and specialists

 Entry of patient data is difficult – requires too much 

manual entry of information

 They aren’t focused enough on recommendations for next 

steps for follow up

 Unable to interact with practitioner for missing 

information to increase confidence and more definitive 

diagnosis

 Have difficulty staying up to date

7
8










19
 Initial research and grant to help educate Watson 

in medical domain

 Could Watson software for Jeopardy! be 

successfully ported into the medical domain?

 Began discussing challenge associated with NEJM 

Clinico-Pathological Conference

 Talked about books and journals and other sources 

that could augment the general knowledge built 

into the Jeopardy! playing software
 E-mails and interviews from all over the world:

 Most were incredibly impressed with potential for 

medicine and opportunities for the future

 Some however:

 SKYNET and end of world as we know it

 Pre-medical students speculating that it really doesn’t make 

sense to attend medical school any more

 Physicians writing blogs predicting that they would be 

replaced by the computer within a short period of time
 Want 3 components similar to medical students 

education

 Book knowledge

 Sim Human Model

 Experiential learning from actual EMR

22
 Textbook, journal, and Internet resource knowledge. 

Quiz materials

 Like medical student this alone not enough don’t want 

to make hypochondriac

23
 Continue to develop medical knowledge database

 Harrison’s

 Merck

 Current Medical Diagnosis and Treatment

 American College of Physicians Medicine

 Stein’s Internal Medicine

 medical Knowledge Self Assessment Program

 NLM’s Clinical Question Repository

24
 Use New England Journal of Medicine 130 CPC cases 

and quiz material

 Additional CPC cases at U of Maryland

 Begin developing interactive capability to develop 

hypotheses and refine them depending on the answer 

to those questions 

 Develop a tool that allows for physician feedback to 

the system for various hypotheses so community can 

interact and teach Watson

25
 SIM Human model of physiology – work done at the 

University of Maryland School of Medicine and UMBC 

by Dr. Bruce Jarrell and colleagues

 Want to have understanding from model of physiology

 Work has been done to create simulations of disease 

processes and then observe how it affects other 

physiology in the body

26
 Consumption of electronic medical record which is largely 

just paper represented digitally, cannot search for “rash” for 

example

 Access to records at U of Maryland and VA but also larger 

repositories from the VA in de-identified manner

27
 Epic system at the University of Maryland

 VA’s VISTA System

 University of Maryland EPIC system

 EMR

 Electronic version of paper records

 Review large number of discharge summaries

 Review progress notes and structured and unstructured 

additional information from EMR

28
 Patient EMR such as VA’s highly publicized and 

praised VISTA revealed numerous challenges

29
 Not possible to search for a term within or among 

patient records such as “rash”

 Majority of data is unstructured and in free text format

 Much of the text in progress notes and other types of 

notes is highly redundant since interns and residents 

and attending physicians typically cut and paste 

information from lab and radiology and other studies 

and other notes

 Information is entered with abbreviations that are not 

consistent and misspellings

Despite the fact that virtually 100% of patient information is 

available in the electronic EMR with records going back 

more than 15 years

30
 Patient problem list has no “sheriff” and each physician 

is free to add “problems” but very few delete them for 

“problems” that are temporary

 The problem lists themselves often have contradictory 

information

31
 5000 questions from American College of Physicians 

Doctor’s Dilemma competition

 E.g.

 The syndrome characterized by joint pain, abdominal 

pain, palpable purpura, and a nephritic sediment

 Henoch-Schonlein Purpura

 Familial adenomatous polyposis is caused by mutations 

of this gene: APC gene

 Syndrome characterized by narrowing of the 

extrahepatic bile duct from mechanical compression by 

a gallstone impacted in the cystic duct: Mirizzi’s

Syndrome

32
 Content

 Organizing domain content for hypothesis and evidence 

generation such as textbooks, dictionaries, clinical guidelines, 

research articles

 Tradeoff between reliability and recency

 Training

 Adding data in the form of sample training questions and 

correct answers from the target domain so system can learn 

appropriate weights for its components when estimating 

answer confidence

 Functional

 Adding new question analysis, candidate generation, and 

hypothesis evidencing analytics specialized for the domain

33
 Text content is converted into XML format used as 

input for indexing

 Text analyzed for medical concepts and semantic types 

using Unified Medical Language System terminology 

to provide for structured query based lookup

 “Corpus expansion technique” used by DeepQA

searches web for similar passages given description of 

symptoms for example and generates pseudo 

documents from web search results

34
 ACP (American College of Physicians) Medicine

 Merck Manual of Diagnosis and Therapy

 PIER (collection of guidelines and evidence 

summaries)

 MKSAP (Medical Knowledge Self Assessment 

Program study guide from ACP)

 Journals and Textbooks
36
 Despite all of the advances in computer technology we 

are arguably still at the paper stage of research as far as 

ability to discover and combine important data

 Research data including those associated with major 

medical journals and clinical trials are typically created for 

a single purpose and beyond a one or two manuscripts, 

remain largely locked up or inaccessible

 Even when the data are made accessible, they are typically 

associated with limited access through a proprietary 

Internet portal or even by requesting data on a hard drive

 Often requires submission of a research plan and data and 

then a considerable wait for permission to use the data 

which is often not granted
 Alzheimer’s Disease Neuroimaging Initiative

 Excellent example of patient data and associated 

images with great sharing model

 However requires access through their own portal 

and requires permission from ADNI Data 

Sharing and Publications Committee
 As an NCI funded Consortium, the Pediatric Brain Tumor Consortium 

(PBTC) is required to make research data available to other 

investigators for use in research projects

 An investigator who wishes to use individual patient data from one or more of the Consortium's completed and published studies must 

submit in writing:

 Description of the research project

 Specific data requested

 List of investigators involved with the project

 Affiliated research institutions

 Copy of the requesting investigator's CV must also be provided.

 The submitted research proposal and CV shall be distributed to the 

PBTC Steering Committee for review

 Once approved, the responsible investigator will be required to complete a Material and Data Transfer Agreement as part of the 

conditions for data release

 Requests for data will only be considered once the primary study analyses have been published



 University hospital databases

 Large medical system e.g. Kaiser Permanente data 

warehouse

 Insurance databases such as WellPoint

 State level databases
 At best, freely sharable databases are accessed using 

their own idiosyncratic web portal

 Currently no index of databases or their content

 No standards exist to describe how databases can 

“advertise” their content and availability (free or 

business model) and their data provenance and 

sources and peer review, etc.

 Would be wonderful project for AMIA or NLM to 

investigate the creation of an XML standard for 

describing the content of databases

 This will be critical to the continuing success of the 

Dr. Watson project in my opinion
 Medical guidelines are increasingly being put into 

machine intelligible form although this is not an easy 

process

 Incorporating these into Watson software could serve 

multiple purposes including health surveillance, could 

factor into diagnostic decision making, and could be 

an early implementation of the Watson technology

 The transition to the 3rd year of medical school begins 

a new phase in education from theoretical to empirical

 Medical students are exposed for the first time to the 

wards and of course, importantly, to one of their major 

jobs for the next few years:

 Maintenance and review of patient charts, nowadays the 

Electronic Medical Record

 Despite the tremendous strides we have made toward 

an electronic medical record, we are really just at the 

1.0 stage and arguably most current EMR systems 

really represent just a digital form of paper

 The Watson development team was really surprised 

when we reviewed the EMR at how primitive it was, 

even in 2011

 Lack of ability to search for terms within a patient’s 

record

 Lack of ability to search across patient records

 Lack of ability to perform basic statistics or have access 

to basic decision support tools in EMR
 The diagnosis of a specific type of pneumonia, for 

example, can be made according to patient signs and 

symptoms using journal articles and textbooks

 But it can also be made more reliably by a system such 

as Watson by also mining the local EMR database as to 

what diagnoses have been made over the past few days, 

weeks, months, etc. locally
 It can then be further refined by not necessarily being 

constrained to tentative diagnoses that have been 

made but the microbiology/pathology proven causes 

of pneumonia

 The EMR provides empirical data about the 

association of these signs and symptoms 

with diagnoses and the means to verify what 

was found by lab tests etc.
 Challenges mining EMR

 Unstructured free text with abbreviations, variable 

terms (e.g. MRI terminology)

 Difficulty in having Watson technology analyze large 

databases such as VA’s EMR due to PHI concerns and 

need to stay within the firewall

 Watson needs to incorporate the concept of changing 

signs and symptoms in a patient over time which creates 

added dimension to diagnosis of a single patient 

presentation

 Challenge is the fragmentation of electronic medical 

records by multiple hospitals, clinics, outpatient 

settings, etc.
 Watson can gain empirical knowledge of vast numbers 

of physicians and patients in a way that would not be 

possible for any single practitioner

 Watson could use EMR to perform research and 

discovery in healthcare such as unanticipated drug 

responses and interactions and factors impacting 

patient response to therapy

 Watson can be impetus to medical community for the 

development of more structured EMR in a more 

friendly machine readable format
 PHR’s will enable Watson to get all 

information in one place when patients 

centralize and take control of their own 

electronic health records

 Patients will be able to control level of 

access to their information