Good Data Practices 

VIREC Cyber Seminar Series

September 9 – 13, 2013

1 
Acknowledgements

• Initial concept development work by Michael 

Berbaum, PhD, University of Illinois at Chicago

• Review of existing public online course content 

and resources by Margaret Browning, PhD and 

Linda Kok, MA, VA Information Resource Center

• Examples from ongoing and completed research 

supported by the VA, NIH, and PCORI

• Feedback from colleagues, especially Elizabeth 

Tarlov, Kevin Stroupe, Matt Maciejewski, and 

Laurel Copeland

2 
Good Data Practices Series Overview

Mon

9/9/13

1 Early Data Planning for Research - Denise Hynes, PhD, MPH, 

RN 

Managing and Documenting Data Workflow - Denise Hynes, 

PhD, MPH, RN 

Planning for Data Re-use - Linda Kok, MA 

Research Application - Laurel Copeland, PhD

Research Application - Matthew Maciejewski, PhD

3 

Tue

9/10/13

2 

Wed

9/11/13 

3 

Thu

9/12/13

4 

Fri 

9/13/13

5 
Not about...

• How to design a research study

• How to execute each step of the research 

study

• How to structure data

• How to analyze data 

• How to get funding 

4 
5 

Inter-university Consortium for Political and Social Research (ICPSR). (2009). Guide to Social Science Data Preparation and Archiving: Best 

Practice Throughout the Data Life Cycle (4th ed.). Ann Arbor, MI. ISBN 978-0-89138-800-5 
6 

Session 2. 

Analysis

Session 1. 

Develop data 

plan & 

proposal 

Session 2. 

Data cleaning 

& preparation 

The Research Data Life Cycle

In three sessions

Session 3. 

Data Re-use

Session 2. 

Data 

collection

Reality: Research is not a linear process
Session 1: Early Data Planning for 

Research

• Importance of data planning

• Factors that influence data needs

• Additional data planning for IRB submission

• Data planning checklist

7 
Poll Question #1

• What is your primary research role? 

– Investigator 

– Data analyst/programmer or statistician

– Research coordinator or assistant

– Student, trainee, or fellow

8 
Poll Question #2

• What would you say is your level of research 

experience?

1 (Novice)

2 

3 

4 

5 (Expert) 

9 
Session 1: Outline

• Importance of data planning

• Factors that influence data needs

• Additional data planning for IRB submission

• Data planning checklist

10
Importance of data planning

•Forces you to think clearly about work flow

•Helps identify issues before you begin

•Provides a guide for project team

•Helps you write/refine your protocol

•Serves as a reference for future work

•Simplifies your life in the long run

11
Session 1: Outline

• Importance of data planning 

• Factors that influence data needs

• Additional data planning for IRB submission

• Data planning checklist

 

12
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

13
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

14
Example: 

“FINER” criteria for a good research question

15

Feasible • Adequate number of subjects

• Adequate technical expertise

• Affordable in time and money

Interesting • Getting the answer intrigues investigator, peers and community

Novel • Confirms, refutes or extends previous findings

Ethical • Consistent with institutional review board standards

Relevant • To scientific knowledge

• To clinical and health policy

• To future research

From: Hulley S, Cummings S, Browner W, et al. Designing clinical research. 3rd ed. Philadelphia (PA): Lippincott Williams and Wilkins; 2007. 
Example: 

“D-FINER” criteria for a good research question

16

Feasible • Adequate number of subjects

• Adequate technical expertise

• Affordable in time and money

• Practical data plan

Interesting • Getting the answer intrigues investigator, peers and community

• Makes use of data in new ways

Novel • Confirms, refutes or extends previous findings

• Produces new data that may be useful to others

Ethical • Consistent with institutional review board standards

• Data plan protects privacy of subject data

Relevant • To scientific knowledge

• To clinical and health policy

• To future research

• To future data sharing

From: Hulley S, Cummings S, Browner W, et al. Designing clinical research. 3rd ed. Philadelphia (PA): Lippincott Williams and Wilkins; 2007. 
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

17
Factors that influence data needs

18

Did investigator 

assign exposure?

Interventional Study

Retrospective Study

(Yes) (No)

(No)

(Yes) 

Observational Study

Prospective Study?

Are all data needed available 

from existing sources?

Study Design & Data Considerations
19
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

20
Factors that influence data needs

• Data to support your research objectives/aims/hypotheses

– What type of data? PHI or non-PHI?

– How much data?

– Unit of analysis

• One record or multiple records per 

– Event

– Subject

– Encounter, claim, inpatient stay

– Facility, VISN

– County, zip code

– Temporal aspects

– Measurement frequency

– Informatics tools

21
22

Citation: Hynes, DM, et al., Informatics Resources to Support Healthcare Quality Improvement Research in the Veterans Health 

Administration. J of the American Medical Informatics Association. Sep/Oct 2004;11(5):344-350.
23
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

24
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

25
Factors that influence data needs

• Available data

– Conduct review of existing datasets

– Determine whether project will produce a new 

dataset

– Anticipating future sharing, consider special 

archiving challenges, especially informed consent 

and confidentiality

26
Factors that influence data needs

• VA data available for your research 

– Sources of information about VA data

• VHA Data Portal

• VA Information Resource Center (VIReC)

• HSRData-L Listserv

• Health Economics Resource Center (HERC) 

• Corporate Data Warehouse (CDW) Metadata Report

• Patient Care Services (PCS)

• Pharmacy Benefits Management Services (PBM)

• OIA HIG Health Data Quality Program

27
Factors that influence data needs

• Access to VA data

– How do I request VA data?

• VHA Data Portal Data Access Pages

– Research & operations data requests

– DART – NDS managed data

• VIReC – VA/CMS data

• PBM – Pharmacy data

• VIReC Database and Methods series, Research Access to Data 

November 4, 2013

28
VHA Data Portal

9/5/2013 29
30

VIReC 
9/5/2013 31
Factors that influence data needs

• Availability of other secondary data

32 Comparison Cohorts
Factors that influence data needs

• Planning data collection/acquisition 

– Will you need data directly from subjects?

• Do you need any special software to collect the data?

– Do you need site-specific information?

• Is it available in a public data set or will you need to 

interview key informants?

– How much detail do you need about events of 

interest? 

– How will primary data be integrated with 

secondary data?

33
Factors that influence data needs

• Availability of electronic medical record data

– How many different EMR systems will need to be 

accessed?

– What processes do you need to plan for data 

extractions?

– How well do you understand the data structures? 

• What you see in a user interface is not always what you get 

from the operational data store!

– Is there a data warehouse of EMR data available along 

with experts to consult?

– How will you link data with other data sources? 

 

34
Factors that influence data needs

• Primary data collection

• Paper/Scanned forms

• Electronic data entry

• Clinical data management systems

– Research Electronic Data Capture (REDCap) 

» Available soon

» Multi-site, web-based data entry within the VA firewall

» Data export procedures to SAS, SPSS, STATA and R 

» A shared library of data collection instruments and forms 

• Clinical trials management systems

• Electronic data capture /integration with other records

35
Factors that influence data needs

• Other primary data collection issues

– One-time or repeated data capture? 

– Length of each collection period 

– Amount of data generated

– Type of data -- Images? 

– Will data be shared with other sites/collaborators

36
Factors that influence data needs

• Secondary data considerations

– Existing data sources needed

– Identifier 

• Are real SSNs necessary?

– Amount of data to be collected (primary & 

secondary)

37
Factors that influence data needs

• Plan for data integration

– Data from multiple data sources

• Primary + secondary data

• Multiple primary + multiple secondary data from 

different sources

• Structured + unstructured data

– Plan for combining or integrating data 

– Plan for sharing data during and after the study

38
Factors that influence data needs

• Integrating data and documentation 

– Using integrated software 

– Computer-assisted interviewing 

39
Factors that influence data needs

• Research question

• Study design

– Objectives, aims, hypotheses 

– Independent and dependent variables 

– Planned manuscripts 

• Available data

• Feasibility testing

40
Example: 

“D-FINER” criteria for a good research question

41

Feasible • Adequate number of subjects

• Adequate technical expertise

• Affordable in time and money

• Practical data plan

Interesting • Getting the answer intrigues investigator, peers and community

• Makes use of data in new ways

Novel • Confirms, refutes or extends previous findings

• Produces new data that may be useful to others

Ethical • Consistent with institutional review board standards

• Data plan protects privacy of subject data

Relevant • To scientific knowledge

• To clinical and health policy

• To future research

• To future data sharing

Adapted from: Hulley S, Cummings S, Browner W, et al. Designing clinical research. 3rd ed. Philadelphia (PA): Lippincott Williams and Wilkins; 2007. 
Factors that influence data needs

• Feasibility testing

– Adequate technical expertise 

• Data management

• Statistical modeling & analysis

– Affordable in time and money

– Manageable in scope 

– Is there an adequate number of subjects 

characteristics to test the hypothesis?

42
Factors that influence data needs

• Preparatory to research requirements for 

feasibility testing

– VHA 1200.05 - Human Subjects Protection

– Data access is only to develop one specific protocol 

– No IRB or R&D Committee approval needed

– Access ends once the protocol has been submitted to the IRB 

and the Research and Development (R&D) Committee for 

review. 

– Pilot studies are not considered preparatory to research and 

must have separate IRB approval 

43
Session 1: Outline

• Importance of data planning

• Factors that influence data needs

• Additional data planning for IRB submission

– Data planning checklist

 

44
Additional data planning for IRB submission

• Data privacy

• Data security

45
Additional data planning for IRB submission

• Data privacy plan: How you will protect the subjects’ private 

health information and identity?

– Data access restrictions

– Who will have access and how?

– Handling real & scrambled SSNs 

– Data from external sources

– Special subjects

• Deceased Veterans

• Patients with alcohol, drug, HIV & Sickle Cell information

• Non-patient participants

• VHA Handbook 1605.1 Privacy & Release of Information 

• VHA Handbook 1200.05 Requirements for Protecting Human 

Subjects in Research

46
Additional data planning for IRB submission

• Data security plan

– Storage location 

• VA network server

• VINCI Workspace

– Special backup scheduling 

– Plan for data transfers between sites

– VHA requirements 

47
Session 1: Outline

• Importance of data planning

• Factors that influence data needs

• Additional data planning for IRB submission

– Data planning checklist

 

48
Data planning checklist

• Data planning checklist

– What existing data will you need? 

– Will you need data directly from the subjects?

– Will you need both? 

– What is the time-frame for all data needed?

– What are the latest years of those data available?

– How much data will be generated? Have you planned for server 

space? 

– How will you link data from different sources? 

– If the data sources do not have a data element in common, is there a 

third data source available with an element common to each?

– Is the software you need available?

• Source: MIT Library Guides 

49
Data planning checklist

• Formalizing Your Data Management Plan

– Description of the project

– Description of the data to be collected 

– Standards to be applied for formats, metadata, etc.

– Plans for short-term storage and data management: e.g., file 

formats, local storage and back up procedures, and security

– Description of legal and ethical issues: e.g., intellectual property, 

confidentiality of study participants

– Access policies and provisions: i.e., how will you make data 

available to others, any restrictions to data reuse, etc.

– Provisions for long-term archiving and preservation 

– Assigned data management responsibilities: i.e., which persons 

will actually be responsible for ensuring data management; 

compliance monitoring over time

50
UCLA Library Website

51

http://guides.library.ucla.edu/content.php?pid=385860&sid=3182780&preview=200eac9a6bb823f4503d8413245daed7
Additional Resources

• ICPSR Data Management Plan 

– http://www.icpsr.umich.edu/icpsrweb/content/datamanagement/dmp/

• MIT Libraries:

– http://libraries.mit.edu/guides/subjects/data-management/plans.html 

• DMPTool a service of the University of California Curation 

Center, California Digital Library

– https://dmp.cdlib.org/about/dmp_about

• Digital Curation Centre, Scotland: 

– DCC Template for a Data Management Plan 

52
QUESTIONS
Contact Information

Denise Hynes, PhD, MPH, RN 

VA Information Resource Center

A VA Health Services Research & Development Resource Center 

working to improve the quality of VA research that utilizes 

databases and information systems

Hines VA Hospital

VIReC@ va.gov 

708-202-2413

54
Session 1: Early Data Planning for 

Research

• Recap

– Importance of data planning

– Factors that influence data needs

– Additional data planning for IRB submission

– Data planning checklist

55
Session 2: Managing and Documenting Data 

Workflow

• Preview

– Getting started

– Importance of documentation

– Data management workflow

– Analysis workflow

56