ARC 

Leonard D’Avolio PhD & Thien N

MA
Introduction

Massachusetts Veterans Epidemiology Research an

Information Information Center (MAVERIC) (MAVERIC)

About 140 people focused on: Large scale clinical trials (CSP CC) Epidemiology (CSP ERIC) Biospecimen Repository Informatics

Projects include:

• Million Veteran Program & GenISIS• Point of Care Clinical Trials
Background

Growing need for access to EMR data for secondary  Quality Quality measurement measurement & improvement improvement Comparative effectiveness Evidence Evidence‐based medicine medicine Bio‐surveillance Cohort & registry registry building building Personalized medicine & genomic analyses
The Challenge The Challenge 

 EMRs were designed for 1‐on‐1 interactions As a result: result:

 Few widely implemented standards Questionable Questionable quality quality of these few Large amounts of free text

 Estimated 70% of information in free text

 In response – 40+ years of 1‐off NLP solutions

 113 studies = no systems nor results are gene

 Stanfill et al. 2010. A systematic literature review of automated clinical coding 

systems. JAMIA 17: 646‐651
The Challenge (cont.)

 Of the few implemented systems, most are:

 Applied Applied for specialized specialized applications applications At institutions of origin Heavily Heavily reliant reliant on the systems systems’ developers developers

 Why?

 Complex nature of the challenge Many processes involved Lots of customization Economics of research
The Idea The Idea 

 Change the workflow so that researchers don’t h

service service‐oriented oriented dependency dependency on software software develo

develo

 Move away from rule‐based systems to do inform

retrieval

 Take advantage of 20+ years of empirical evidenc

open source code

 Use existing NLP frameworks to allow for other

developers to extend and modify the tool

 90/90 goal using generalized generalized approach approach

+ +

 Import annotations (eHOST and Knowtator)

 Use existing NLP pipelines (cTAKES) Combine with machine learning (MALLET) Provide a simple workflow taking advantage of p

tools
0.97 0.95 0.

Document Retrieval

Recall Precision F‐

Prostate Cancer Path Reports 0.97 0.95 0.

Colorectal Cancer Path Reports 0.90 0.92 0.

Lung Cancer Imaging Reports 0.76 0.80 0.

PTSD Psychotherapy Notes (seeking PE) 0.72 0.93 0.

PTSD Psychotherapy Notes (seeking CBT) 0.86 0.98 0.

Breast Cancer Path Reports (from 130 hospitals) 0.95 0.95 0.

Breast Cancer Clinic Notes 0.96 0.88 0.

Breast Cancer Operative Reports 0.91 0.85 0.

Pneumonia Imag g in Reports 0.80 0.81 0.
2010 A Medical Problems 0.75 0.93 0.

Concept Retrieval (inexact span matching)

Recall Precision F‐

2010 i2b2/VA Medical Problems 0.75 0.93 0.2010 i2b2/VA Medical Treatments 0.76 0.89 0.

2010 i2b2/VA Medical Tests 0.76 0.90 0.

 2009 cTAKES with minor changes and a couple of additional annotato

sentence) 

 2008 LVG  2008 UMLS  Out-of-the box we were on par with average entrants’ performance 

 With less than 5 minutes of human time per task 
What we learned What we learned 

 Works well on document classification tasks

 “Find more cases like these”

 Use existing structured data to narrow sets

 Known ICD‐9 s’ + RXs then use ARC

 You need to understand how the process works

 Training Training set, test set, performance performance metrics metrics

 Know what is “acceptable” & estimated prevalen

before you start
Demo 

Do-It-Y lf I f Yourself Interface 
 >150 downloads 

 Google group video tutorials dummy data sets e

Google group, video tutorials, dummy data sets, e

 http://maveric.org/mig/arc.html  http://code google com/p/mavericarc/ http://code.google.com/p/mavericarc/  http://groups.google.com/group/ClinicalNLP  Available on VINCI  Installer package via iDASH  iDASH NLP VM also has ARC  Thanks to HSR&D Consortium for Healthcare 

Informatics Research (CHIR)