Systems for Helping Veterans 

Comprehend Electronic Health Record 

Notes

Hong Yu1,2, Balaji Polepalli Ramesh1,2, Cynthia Brandt3,4, Sowmya R Rao2,5, Julie Volkman2,5, Donna Zulman6,7, Hua Fang2, Kathleen Mazor2, Thomas

Houston2,5

1VA Central Western Massachusetts

2University of Massachusetts Medical School

3VA Connecticut Health Care System

4Yale University

5Bedford VAMC

6VA Palo Alto Health Care System

7Stanford University
Outline

 Background 

 The NoteAid System

 Evaluation: 3 Pilot Studies

 Self-reported comprehension

 Paraphrase note: “...in your own words” 

 Patient interviews

 Conclusion & Future Work
Background

 Patients reading their EHR notes has the potential to

 Enhance medical understanding 

 Improve healthcare management and outcomes

 Blue Button: The Department of Veterans Affairs (VA) 

Blue Button enables Veterans to view, print, and 

download their EHRs, including clinical notes (e.g., 

progress notes). 

 As of April, 2013, over 750K registered users and over 

3.6 million Blue Button download requests 
The Challenge

 Physicians’ notes are difficult to comprehend 

(Keselman et al) 

 Many Veterans have limited health literacy 

(Schapira et al)
 

“The patient will be scheduled for a repeat EGD in 

one year for surveillance purposes of Barrett's 

esophagus. From a GI standpoint, we recommend to 

proceed with bariatric surgery. However, he will 

need to continue daily PPI administration to 

maximize acid reduction. Otherwise, there are no 

additional recommendations. The patient was treated 

with myocardial infarction.”
The NoteAid System

 A system for helping patients comprehend electronic 

health record notes 

 Automatically 

 Identifies clinically relevant concepts

 Links concepts to their definitions 

 Translates medical jargon into lay language that 

patients understand

 Links notes to other education material
NoteAid Demonstration

 http://www.clinicalnotesaid.org
Poll Question #1

 What is your primary role in VA? 

 student, trainee, or fellow

 clinician

 researcher

 manager or policy-maker

 Other
Poll Question #2

 Which best describes your research experience?

 have not done informatics research

 have some informatics research experience

 have some informatics and some NLP research 

experience

 Have extensive NLP research experience
The NoteAid System

The patient will be scheduled for a repeat EGD in one year for surveillance purposes of 

Barrett esophagus. From a GI standpoint, we recommend to proceed with bariatric 

surgery. However, he will need to continue daily PPI administration to maximize acid 

reduction. Otherwise, there are no additional recommendations.

Knowledge 

Resources

(MedlinePlus, UMLS)

NLP 

Approaches

NoteAid

NLP 

Approaches
The NoteAid System

The patient will be scheduled for a repeat EGD in one year for surveillance purposes of 

Barrett esophagus. From a GI standpoint, we recommend to proceed with bariatric 

surgery. However, he will need to continue daily PPI administration to maximize acid 

reduction. Otherwise, there are no additional recommendations.

EGD 

An endoscopic procedure that visualizes the upper part of the

Gastrointestinal tract up to the duodenum. 

 Why take the test?

How the test is performed?

Knowledge 

Resources

(MedlinePlus, UMLS)

NLP 

Approaches

NoteAid

NLP 

Approaches
NoteAid

 Knowledge Resources

 MedlinePlus

 the National Institutes of Health's Web site for patients and 

their families and friends

 UMLS – Unified Medical Language System

 Files and software integrating biomedical vocabularies

 Consumer Health Vocabulary (Qing et al)

 Definitions 

 Others

 Other NIH resources

 Other consumer health resources 

 NLP approaches
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept 

Mapper 

Concept 

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concepts

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept 

Mapper 

Concept 

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept 

Mapper 

Concept 

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence

Splitter

Concept 

Mapper 

Concept 

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept

Mapper

Concept 

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept 

Mapper 

Concept

Filter 

Definition 

Fetcher 

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
NLP Approach

 Two system components – 

 Concept Identifier

 Process input text and identify clinically relevant concept

 Definition Locator

 Fetch definitions from MedlinePlus, UMLS and Wikipedia 

Input Split

Sentences

Mapped

Phrases

Mapped

Text

Knowledge

Linked Text

Sentence 

Splitter 

Concept 

Mapper 

Concept 

Filter 

Definition 

Fetcher

COMPONENT 1 COMPONENT 2

CONCEPT IDENTIFIER DEFINITION 

LOCATOR
Evaluation

 Three Pilot Evaluations, with the IRB approval from 

UMass

 Lay people self-report comprehension

 Lay people paraphrase notes

 Patient interviews

 No Veterans, pending VA IRB approval
Pilot Evaluation: Self-Reported Comprehension

 Lay people

 De-identified notes (not own notes)

 With or without NoteAid

 Self-reported comprehension

 1 to 5

 1: impossible to understand

 5: understand completely
Four NoteAid Implementations

 NLP+Different Knowledge Resources

 MedlinePlus

 UMLS

 Wikipedia

 50% physicians use Wiki (KevinMD.com, 2009) 

 53% US population use Wiki (NBC News 2011) 

 Hybrid (MedlinePlus+UMLS+Wikipedia)
Readability: Flesch-Kincaid grade level 

 Hypothyroidism 

 “Too little thyroid hormone. Symptoms include weight 

gain, constipation, dry skin, and sensitivity to the cold. 

Also called underactive thyroid” – From NCI

 FK is 9 – corresponding to US 9th grade education

 “deficiency of thyroid gland activity; characterized by 

decreased basal metabolic rate, fatigue and lethargy, 

sensitivity to cold, and menstrual disturbances; 

untreated it progresses to myxedema; in infants severe 

hypothyroidism leads to cretinism.” – From NIH CSP

 FK is 25, much more difficult to understand
Evaluation Data

 20 de-identified progress notes

Type Progress Notes

No of Reports 20 

Total (Avg) # of sentences 473 (23.7)

Total (Avg) # of words 4862 (243)

Avg Flesch-Kincaid Grade 

Level

9.8
Subjects

 The Amazon Mechanical Turk

 Has shown to be reliable for medical annotations and 

evaluations 

 Recruited 25 subjects 

 5 systems (4 implementations + 1 w/o NoteAid)

 5 subjects per system

 21 subjects completed the evaluation
Demographic Information

 Gender 

 Female – 9 (42.9%)

 Male – 12 (57.1%)

 Education 

 High School – 7 (33.3%)

 Associates – 2 (9.5%)

 Bachelors – 6 (28.6%)

 Masters – 6 (28.6%)

57.2%
Evaluation Process

 Subjects randomly assigned to one of the following 

five systems to read 20 progress notes 

 Note only (no NoteAid) 

 NoteAid_MedlinePlus 

 NoteAid_UMLS 

 NoteAid_Wiki 

 NoteAid_Medline+UMLS+Wiki (hybrid)

 Self-reported comprehension

 1 – impossible to understand

 5 – understand completely
Readability and Self-Reported Comprehension

 The FK Grade Level of a note correlated with selfreported comprehension score 

 ρ=-0.77, p<0.01, Spearman rank correlation

 Lower FK Grade level notes associated with higher 

(better) comprehension
Results

 Average self-reported comprehension scores

*p<0.01, Non-parametric Mann-Whitney Wilcoxon signed-rank test

System Note Alone MedlinePlus UMLS Wiki Hybrid

Score 2.95±0.67 4.12±0.33* 3.63±0.57

* 

3.85±0.47

* 

3.92±0.40

* 
Limitations

 Self-reported comprehension score

 Do lay people really understand the notes?

 Lay people--not patients--performed evaluation
Evaluation

 Three Pilot Evaluations, with the IRB approval from 

UMass

 Lay people self-report comprehension

 Lay people paraphrase notes

 Patient interviews

 No Veterans, pending VA IRB approval
Evaluation Process

 Data: 5 de-identified progress notes

 Subjects: 40 subjects recruited from the Amazon 

Mechanical Turk

 Evaluation Process: 

 Each subject was presented with 5 notes, one at a time, 

either with NoteAid (hybrid), or without.

 Each subject paraphrased the main content of the note 

 39 completed the evaluation (20 with NoteAid and 19 

without)
Examples: Without NoteAid







Note : The patient is doing okay today. He has some complaints of discomfort around his 

tracheostomy site but otherwise has no complaints of chest pain, shortness of breath, 

abdominal pain, nausea, or vomiting. OBJECTIVE : Vital Signs : Currently stable. Cardiac : 

Normal S1, S2. No murmurs, rubs, or gallops are present. Lungs : Clear to auscultation 

bilaterally. Abdomen : Soft, nontender, and nondistended. Normoactive bowel sounds are 

present. LABS : There are no new labs from today. ASSESSMENT AND PLAN : 1. Preoperative 

assessment from my prior note. 2. Hyponatremia likely to be SIADH related. Continue to 

monitor for now. 3. Leukocytosis. Likely to be reactive. No clear infectious source. 4. 

Thrombocytopenia. It is probably chronic in nature and appears to be stable. Although , we 

would continue to monitor this for now as well.

Paraphrase note 1: “The patient seems to be doing alright despite some minor discomfort. His 

vital signs are stable and he hasn't had any labs today. The plan going forward is to keep 

monitoring the patient.” 

Paraphrase note 2: “The patient appears to be doing well aside from minor discomfort 

around the site of where the tracheostomy was performed. Other than that, they do not 

appear to have any other discomfort. There are other apparent issues like Hyponatremia 

and Leukocystosis which appear to be a result of the procedure. The patient appears to be 

doing well, but should continue being monitored.” 
Examples: NoteAid





Note : The patient is doing okay today. He has some complaints of discomfort around his 

tracheostomy site but otherwise has no complaints of chest pain, shortness of breath, 

abdominal pain, nausea, or vomiting. OBJECTIVE : Vital Signs : Currently stable. Cardiac : 

Normal S1, S2. No murmurs, rubs, or gallops are present. Lungs : Clear to auscultation 

bilaterally. Abdomen : Soft, nontender, and nondistended. Normoactive bowel sounds are 

present. LABS : There are no new labs from today. ASSESSMENT AND PLAN : 1. Preoperative 

assessment from my prior note. 2. Hyponatremia likely to be SIADH related. Continue to 

monitor for now. 3. Leukocytosis. Likely to be reactive. No clear infectious source. 4. 

Thrombocytopenia. It is probably chronic in nature and appears to be stable. Although , we 

would continue to monitor this for now as well

Paraphrase note: "The patient is doing well although he has some discomfort around 

his tracheostomy area. The patient has stable vitals. Some conditions will continue to 

be monitored, including low salt content in the blood, high white blood cell count, 

and low blood platelet count."
Evaluation of Summary

Terms in 195 Summaries With NoteAid Without NoteAid

Number of medical jargon terms in lay language 268 68

Number of medical jargon terms copied and pasted 70 169
Evaluation

 Three Pilot Evaluations, with the IRB approval from 

UMass

 Lay people self-report comprehension

 Lay people paraphrase notes

 Patient interviews

 No Veterans, pending VA IRB approval
A Pilot Study: Patient Evaluation with NoteAid







Approved by the IRB board at UMMS

Recruit 3 patients, all college graduates

 One Type I diabetes, two Type II diabetes

 Being treated for 5~15 years 

Each patient presented with 1~2 own progress notes of 

his/her most recent clinical visit

 Ask what they don’t understand 

 Present Note+NoteAid

 Think-aloud

 A list of questions
Concepts and Abbreviations

 The 3 patients did not understand 5~10 concepts in 

his/her own progress note

 E.g., FT4, auscultation, post-ablative hypothyroidism
Questions













Do you want to read your EHR notes? Yes (3)

When do you want to read your notes? 

 Before and after a clinical visit (3)

Is there a specific part of notes that you would like to read?

 All notes (2), current plan (1)

Would you be willing to share your notes with your friends or family 

members?

 No (2), spouse if asked (1) 

Is NoteAid helpful? Yes (3)

Do you want to use NoteAid? Yes (3)
How to Improve NoteAid

 More concepts included by NoteAid (3)

 Easy-to-read definitions (3)

 More education material (3)
Evaluation

 Three Pilot Evaluations, with the IRB approval from 

UMass

 Lay people self-report comprehension

 Lay people paraphrase notes

 Patient interviews

 No Veterans, pending VA IRB approval
Limitations

 Main limitations 

 All pilot studies

 Sample size is small

 No Veteran population
Conclusion and Future Work

 NoteAid improved EHR note self-reported 

comprehension 

 NoteAid helped patients comprehend their own 

notes 

 Future Work

 NLP: Improve concept coverage and filtering

 NLP: Improve quality of the definitions

 NLP: Add education material

 Evaluate comprehension

 Evaluate system in which Veterans read their own EHR 

notes 
References



 

Polepalli Ramesh B, Houston T, Brandt C, Fang H, Yu H. 

Improving Patients' Electronic Health Record Comprehension 

with NoteAid, Studies in Health Technology and Informatics 

Vol.192: MEDINFO 2013, pp 714 - 718. DOI:10.3233/978-

1-61499-289-9-714. Best Student Paper Award

 Polepalli Ramesh B and Yu H. Systems for Improving Electronic 

Health Record Note Comprehension, In proceedings of Health 

Search and Discovery 2013 at ACM SIGIR 2013, pp 39-42.
Thanks and Questions

 The NoteAid Prototype Systems:

 http://www.clinicalnotesaid.org

Contact 

 Hong.yu@va.gov or hong.yu@umassmed.edu

 Hong Yu, MS, MA, and PhD

Research Health Scientist,

VA Central Western Massachusetts

 and

 Professor, Department of Quantitative Health Sciences

 Department of Computer Science (Adjunct)

 University of Massachusetts Medical School

 AS6-2071 ASC-QHS, 368 Plantation Street, Worcester, MA 01605 

 508-856-3474/ Fax: 508-856-8993 

