Good Data Practices 

VIReC Cyberseminar Series

May 2014

Good Data Practices 2014
GDP 2014 Acknowledgements

• Laurel Copeland, San Antonio VA

• Brian C. Sauer, Salt Lake City VA

• Kevin Stroupe, Hines VA

• Linda Williams, Indianapolis VA

• Denise Hynes, VIReC

• Arika Owens, VIReC

• Maria Souden, VIReC

2 
3 

Inter-university Consortium for Political and Social Research (ICPSR). (2009). Guide to Social Science Data Preparation and 

Archiving: Best Practice Throughout the Data Life Cycle (4th ed.). Ann Arbor, MI. ISBN 978-0-89138-800-5 

Research Life Cycle
Good Data Practices Series Overview

May 8

• The Best Laid Plans: Plan Well, Plan Early - 

Jennifer Garvin

May 15

• “The Living Protocol:” Managing Documentatio

While Managing Data - Matt Maciejewski

n 

May 22

• Controlled Chaos: Tracking Decisions During an 

Evolving Analysis - Pete Groeneveld

May 29

• Reduce, Reuse, Recycle: Planning for Data 

Sharing - Linda Kok

4 
Poll #1: About you

• What is your role in research and level of experience? 

▫ Research investigator

New Experienced

▫ Data manager/analyst

New Experienced

▫ Project coordinator

New Experienced

▫ Other – please describe via the Q&A function

5 
Session 1

The Best Laid Plans: Plan Well, Plan Early 

Good Data Practices 2014

Jennifer Garvin, PhD

Salt Lake City VA Healthcare System
Poll #2: Planning for data use

• When do you start planning for data use for your 

research?

▫ During the proposal stage

▫ After I get funding notice

▫ When I prepare the IRB submission

7 
Planning ahead- theory vs. practice

Ideal Reality
Data planning awareness

• Today’s goal: take away one new idea that can help you 

think through data planning earlier in the research 

process

• Develop a strategy for how to do better planning within 

the constraints of your environment and your projects

• Identify documents where you describe the data you 

need, what you want to do with it (including analysis) 

and what the data will be used for during and after the 

study 

9 
Topics to be covered in today’s session

• Benefits of early data planning 

• Planning for data privacy & security

• Feasibility testing

• Planning for data re-use after the research

▫ For all of the above some factors to consider:

▫ data needs and required permissions, organizing data 

for research, where it will be stored, method of 

analysis of the data, documentation, plans for future 

use

10
11

Benefits of early data planning
12

Forces you to think clearly about data 

Helps identify requirements 

Provides a guide 

Helps develop and refine the research 

protocol

Starts data documentation process (data 

provenance)

Serves as a reference for your papers

Simplifies your life in the long run
Informing data planning

Proposal Protocol IRB App Analytic 

Plan

13
14

No “perfect” plan exists 

 – it depends... 
Case Study from VA INSPIRE SDP VA Stroke 

QUERI*

• Developed methods automating VA stroke indicators.

• Chart review QI data compared to automated results.

• Resulting dataset: 3074 ICD-9 identified stroke 

admissions; 2164 chart-validated stroke admissions.

• Individual data elements completed on all subjects—

reason for ineligibility or for failing any indicator can be 

determined. 

* Courtesy Linda S. Williams, MD, Research Coordinator, VA HSR&D 

Stroke QUERI Roudebush VAMC, Professor of Neurology, Indiana 

University, Indianapolis, IN
Considerations for data planning

• Will you need data directly from the subjects?

• If you need existing data, what will you need? 

• What is the time period for needed data and have you 

explored if the data is available for that time period?

• How much data will be generated and is there server 

capacity and software capacity? 

• Do you need to link data from different sources and if 

yes, how will this be done? 

• What software will you need?

• What methods will you use to protect data privacy & 

security? 

• Will you provide your data for re-use?

es 
Case Study* - Lessons Learned

• Standardize Chart Review and Develop Documentation

• Develop standard chart review manual and update with 

local examples as they are noted

• Standardize search features and terms (document what 

was done)

• Organize the process for access requests and designate 

one person from your study to submit and stay in 

communication via the DART process

17

* Courtesy Linda S. Williams, MD, Research Coordinator, VA HSR&D 

Stroke QUERI Roudebush VAMC, Professor of Neurology, Indiana 

University, Indianapolis, IN
18

Factors that influence data needs

Research question

Study design

Available data

Feasibility testing
Good research question - “FINER” 

criteria

19

From: Hulley S, Cummings S, Browner W, et al. Designing clinical research. 3rd ed. Philadelphia (PA): Lippincott 

Williams and Wilkins; 2007. 

Feasible

Interesting

Novel 

Ethical

Relevant 
Good research question – data 

considerations 

20

Adapted from: Hulley S, Cummings S, Browner W, et al. Designing clinical research. 3rd ed. Philadelphia (PA): 

Lippincott Williams and Wilkins; 2007. 

Feasible: Data availability & structure

Interesting: Answers & informs

Novel: New methods, data & tools

Ethical: Protects subjects’ privacy 

Relevant: Research to benefit Veterans and 

others
Study design & data considerations

21

Did investigator 

assign exposure?

Interventional Study

Study Using Existing Data

(Yes) (No)

(No)

(Yes) 

Observational Study

Study Collecting Primary 

Data 

Are all data needed 

available from existing 

sources?
Case study- planned 

data needs...

Research Question: 

Can a pre-developed, off-the-shelf informatics 

tools accurately determine completion status of 

discharge instructions?

Compared results of automated method to both a 

manually developed reference standard and 

External Peer Review Process (EPRP) results.

22

Garvin JH, et al. Automated quality measurement in Department of the Veterans Affairs 

discharge instructions for patients with congestive heart failure. J Healthc Qual. 2013 JulAug;35(4):16-24. PubMed PMID: 23819743 
Versus realized data needs

Document set obtained via a 

data request using ICD-9-CM 

codes for a principal diagnosis 

of CHF at the medical center = 

152 inpatients.

Number of patients with EPRP 

data from the same time 

period with the same 

diagnosis of CHF = 98.

23

Garvin JH, et al. Automated quality measurement in Department of the Veterans Affairs 

discharge instructions for patients with congestive heart failure. J Healthc Qual. 2013 JulAug;35(4):16-24. PubMed PMID: 23819743 
Data planning for research objectives 

varies with unique requirements

24

Primary 

data 

collection 

Secondary 

data sources 

only 

Mixed 

methods

Data from 

multiple 

sources

Partnerbased 

research
Case study- using lessons learned

• Automating Data Acquisition for Heart Failure (ADAHF)

▫ Automate inpatient heart failure quality measure

▫ Cohort identification with (EPRP) to TIU notes from 

each medical center

25
Diagram of Overall Classification and 

Sub-classifications

26
It also depends on...

• Software you will need

• Where you want to use the data

• And more...

27
Getting access to data in VINCI

• Obtain IRB , R&D, and VA NDS approval processes 

• Determine the established VINCI folder for data 

acquisition, analysis and use of VINCI tools

28
Obtaining TIU notes and other data 

• Identify the cohort of patients 

• Determine standardized enterprise name of the 

documents

• Determine where stored in VISTA files

• Determine the frequency of data updating by visiting the 

CDW SharePoint site

• Once data/note are obtained, analyze the content for 

congruency to the data requested

29
And it depends on feasibility evaluation

Proposal addresses sample size estimates—

is there an adequate number of appropriate 

subjects to test the hypothesis?

Is the required data and analytic expertise 

actually available following funding? 

Are the necessary tools available when 

needed (e.g., NLP)?

Is the study still affordable and manageable 

in scope based on current circumstances? 

30
31

Best practices- examples
National Cardiovascular Data Registry

• Developing a data dictionary

• Standardizing the data collection process

• Data quality checks before incorporation into the 

database

• Documentation of changes to the data

• https://www.ncdr.com/webncdr/pinnacle/ 

32
Observational Medical Outcomes Partnership 

(OMOP.org) Common data model*

• Purpose: standardize the format and content of the 

observational data, so standardized applications, tools 

and methods can be applied to them.

• Tried and tested model with good documentation.

▫ http://75.101.131.161/download/loadfile.php?docname=CDM%20Specification%20V4.0

• Used in large comparative effectiveness research 

networks (e.g., SAFTINet) ▫ http://journals.lww.com/lwwmedicalcare/pages/articleviewer.aspx?year=2012&issue=07001&article=00013&type=abstract

▫ http://www.ncbi.nlm.nih.gov/pubmed/22037893 

• Ensures that research methods can be systematically 

applied to produce meaningfully comparable results. 

33

* Courtesy Ramkiran Gouripeddi, M.S., M.B.B.S., ram.gouripeddi@utah.edu
OMOP- based Transparent ReUsable

Software Tools (TRUST)- EpiTools * 

• Veterans Affairs funded initiative.

• Analytic modules designed to be used in workflows that 

transform data, analyze data and generate publishable 

reports. 

* Courtesy of Brian C. Sauer, Ph.D., M.S., Brian.Sauer@hsc.utah.edu

34
• Data Verification 

and Quality

• Verify completeness 

of extracted data

• Correct known 

problems with data 

(e.g., pill splitting or 

improbable values) 

• Analytic workflows

• Transform data to 

operationalize study 

designs

• Analyze data and 

report findings

• Process Monitoring

• Prediction Modeling

* Courtesy of Brian C. Sauer, Ph.D., M.S., Brian.Sauer@hsc.utah.edu

TRUST—EpiTools* Epidemiological Workflow 35
Contact Information

Jennifer H. Garvin, PhD, MBA

• IDEAS 2.0 VA Salt Lake City Healthcare System

• Jennifer.garvin@va.gov

• 801-582-1565 ext. 4420

IDEAS 2.0's mission is to advance scientific discovery, 

implement novel interventions, promote cross-center 

collaboration, increase research capacity, and engage 

operational partners in order to improve the health of 

Veterans.

36
37

Additional data planning for IRB 

submission

(Linda Kok, VIReC) 
Data privacy plan

• How you will protect the subjects’ private health 

information and identity?

• Who will have access?

• How will access be limited? 

• What special handling will be used for real & scrambled 

SSNs or text data? 

• How will you handle data for special subjects?

38
Data security plan

• Where will the study data be stored & used?

• Will you need data from external sources – will that 

require sending real SSNs to the source?

• How will data transfers between study sites be handled?

39
Data sharing at project close

• Will study produce a new dataset that others may want 

to use?

• Will you establish a new VA research data repository or 

deposit your data in an existing repository? 

• Primary data collection & HIPAA authorization language

40
41

Resources
42

VIReC 
43

5/7/2014


More data content resources

▫ Corporate Data Warehouse (CDW) Metadata Report

▫ HSRData-L Listserv

▫ Patient Care Services (PCS)

▫ Pharmacy Benefits Management Services (PBM)

▫ VA Corporate Data Monograph

▫ VHA Data Portal

▫ VIReC HelpDesk

46
Data access resources

• VHA Data Portal data access pages – details of request 

processes

• VIReC Database and Methods series, Research Access to 

Data – cyberseminar updated each year

47
VHA Data Portal

48

5/7/2014
Other resources

• MIT Libraries:

▫ http://libraries.mit.edu/guides/subjects/datamanagement/plans.html

• UCLA Library:

▫ http://guides.library.ucla.edu/content.php?pid=3858

60&sid=3182780&preview=200eac9a6bb823f4503d8

413245daed7

49
50
51

UCLA Library Website

http://guides.library.ucla.edu/content.php?pid=385860&sid=3182780&preview=200eac9a6bb823f4503d8413245daed7
Next Week in GDP...

 

Agenda

• Observation, Objective & Bottom Line

• Living Protocol

• Example: Managing secondary data

• Example: Managing linkage of primary & 

secondary data

• Summary: Value of documentation

52

May 15

• “The Living Protocol:” Managing Documentation 

While Managing Data - Matt Maciejewski
53

Questions?
54

Bonus Slides
55

OMOP tools*

Tool Application Link

Common Data Model Facilitates analysis of disparate data sources http://omop.org/CDM 

Standardized Terminologies Generate meaningful & comparable results http://omop.org/Vocabularies 

Health Outcomes of Interest (HOI) 10 HOI definitions related to drug safety http://omop.org/HOI 

Observational Medical Dataset 

Simulator Generator 2 (OSIM 2)

Creates simulated data sets that conform to the 

OMOP CDM

http://omop.org/OSIM2 

Observational Source 

Characteristics Analysis Report 

(OSCAR) 

SAS program that generates descriptive statistics http://omop.org/OSCAR 

Natural History Analysis 

(NATHAN)

OSCAR’s extension creating a standardized 

summary 

http://omop.org/NATHAN 

Regularized Identification of 

Cohorts (RICO) 

Procedure standardizing patient cohort selection http://omop.fnih.org/Methods

Library 

Generalized Review of OSCAR 

Unified Checking (GROUCH) 

Warns of implausible and suspicious data 

observed in OSCAR summary

http://omop.fnih.org/GROUCH 

Analysis Methods Library of methods and specifications http://omop.fnih.org/Methods

Library 

* Courtesy Ramkiran Gouripeddi, M.S., M.B.B.S. , ram.gouripeddi@utah.edu
56

Standardized Terminologies*

Condition • ICD-9-CM, SNOMED-CT, MedDRA, Read, Oxmis

• RxNorm, NDF-RT, NDC, GPI, Multum, Multilex, VA_Product, FDB Drug ETC, ATC, FDB indications, FDB contra-indications

Procedure • CPT-4, ICD-9-Procedure, HCPCS, SNOMED-CT

Observation • LOINC, SNOMED, UCUM

Demography • CDC Race, HL7 Sex, US Census Geographical Regions

Visit • CMS Place of Service Codes

Person Status • SNOMED

* Courtesy Ramkiran Gouripeddi, M.S., M.B.B.S. , ram.gouripeddi@utah.edu