Good Data Practices

Session 2 

Managing and Documenting Data Workflow 

Denise Hynes, PhD, MPH, RN 

September 10, 2013

1 
Good Data Practices

• Series Recap

– Session 1: Early Data Planning for Research 

2 
Session 2: Managing and Documenting Data 

Workflow

• Getting started

• Importance of documentation 

• Data management workflow

• Analysis workflow

3 
Poll Question #1

• What would you say is your level of research 

experience?

1 (Novice)

2 

3 

4 

5 (Expert) 

4 
Session 2: Outline

• Getting started

• Importance of documentation 

• Data management workflow

• Analysis workflow

5 
Getting started

• Formalizing your data management plan 

– Description of the project

– Description of the data to be collected

– Standards to be applied for formats, metadata, etc.

– Plans for short-term storage and data management: e.g., file 

formats, local storage and back up procedures, and security

– Description of legal and ethical issues: e.g., intellectual property, 

confidentiality of study participants

– Access policies and provisions: i.e., how will you make data 

available to others, any restrictions to data reuse, etc.

– Provisions for long-term archiving and preservation

– Assigned data management responsibilities: i.e., which persons 

will actually be responsible for ensuring data management; 

compliance monitoring over time

6 
Poll Question #2

• How much experience have you had 

developing a data management plan?

1 (None)

2 

3 

4 

5 (Extensive)

7 
8 http://guides.library.ucla.edu/content.php?pid=385860&sid=3182780&preview=200eac9a6bb823f4503d8413245dae

d7

UCLA Library Website
9 

Example of DMP Plan 

produced by the DMP 

Tool
10
9/9/2013 11
Getting started

• Defining roles and responsibilities 

– Assigning project team responsibilities

• File management (project & data)

• Data management

• Modeling & analysis 

12
Poll Question #3

• What is your primary research role? 

– Investigator 

– Data analyst/programmer or statistician

– Research coordinator or assistant

– Student, trainee, or fellow

13
Session 2: Outline

• Getting started

• Importance of documentation 

• Data management workflow

• Analysis workflow

14
Importance of documentation 

• What should be documented?

– Data management methods

– Data analysis methods

– What you decided to do & why 

– What data were created & how

15
16
Importance of documentation 

• Key components to document

– Sample and sampling procedures

– Weighting

– Date and geographic location of data collection, 

and time period covered

– Data source(s)

– Unit(s) of analysis/observation

17
Importance of documentation 

• What to document about variables 

1. The exact question wording or exact meaning of the datum

2. The text of the question integrated into the variable text 

3. Universe information, i.e., who was actually asked the question

4. Exact meaning of codes

5. Missing data codes

6. Unweighted frequency distribution or summary statistics

7. Imputation and editing information

8. Details on constructed and weight variables 

9. Location in the data file

10. Variable groupings

18
Importance of documentation 

• Summary documents to maintain

– Technical information about data files

– Data collection instruments

– Flowchart of the data collection instrument

– Index or table of contents

– List of abbreviations and other conventions

– Interviewer guide

– Recode logic

– Coding instrument

19
Importance of documentation 

• Where to document?

– Master document 

• “Living protocol”

• Codebook

– Supplemental documents in organized folders

• Program code

• Output & log files

– Many good systems - select one

– Be consistent

20
Importance of documentation 

• When to document?

– While defining your cohort

– During every team meeting

– During data management & analysis

– Inside every program

– In your master document

• Document as you go...

21
Session 2: Outline

• Getting started

• Importance of documentation 

• Data management workflow

• Analysis workflow

22
Data management workflow

• File naming standards and organization

• Documenting cohort derivation

• Primary data collection process

• Accessing secondary data in the VA 

• Linking data from multiple sources

• Cleaning data and documenting data issues

• Documenting the analytic dataset 

• Programing walkthroughs

23
Data management workflow

• File naming standards and organization

– Concepts for naming files

• Easily accessible 

• Accurate & clear

• Meaningful to others

• Easily distinguishable

• Recognizable in different environments 

• Consistent

24
25 Data & Programming Documentation Example
26 Data & Programming Documentation Example
27 Data & Programming Documentation Example
28 Data & Programming Documentation Example
Data management workflow

• Documenting cohort derivation 

– Document the final cohort definition decisions

– Inclusion & exclusion criteria

– All sources used

– Rationale

– Document as you go

– Diagrams are critical and generally required for 

publications

29
30

Citation: Neumayer, L, Giobbie-Hurder, A, Jonasson, O, et al. Open Mesh versus Laparoscopic Mesh Repair of Inguinal Hernia, 

NEJM. 2004; 350 (18): 1819-1827

RCT Cohort 

Flowchart 

Example

Source: Open Mesh 

versus Laparoscopic 

Mesh Repair of 

Inguinal Hernia, VA 

CSP Project # 456; 

PI: L. Neumayer
31

RCT Cohort 

Flowchart 

Example

Citation: Hynes, DM, Stroupe, K, Luo, P, Hurder, AH, Neumayer, L, et al., Cost effectiveness of laparoscopic versus open tension-free hernia repair: 

Results from the Veterans Affairs Cooperative Study. Journal of the American College of Surgeons. 2006; 203: 447-457

Source: Open Mesh 

versus Laparoscopic 

Mesh Repair of 

Inguinal Hernia, VA 

CSP Project # 456; 

PI: L. Neumayer
32

Observational 

Study 

Flowchart 

Example

Source: Quality and Costs 

of Colon Cancer in VA and 

Medicare; VA HSR&D 

Service Project # IIR 03-196

Citation: Hynes, DM, Tarlov, E, Perrin, R, et al., Surgery and Adjuvant Chemotherapy Use Among Veterans with Colon Cancer: Insights Using 

California SEER, Medicare and VA-linked Data. Journal of Clinical Oncology.2010: 28 1-9.
33

Walkthrough 

document for creation 

of final cohort for ESA 

study

Hynes study, 

Clinical Guidelines 

for ESA use in 

Cancer 

Impacts on 

Clinical Practice
Data management workflow

• Primary data collection

– Finalize forms

– Authorization language

– Manage incoming data

– Check the cohort 

34
Data management workflow 

• Finalizing forms 

– Data collection form finalized 

– Final form & language

– Administration method

• Patient HIPAA authorization language 

– Include re-use for further research?

• If needed, changes go to IRB for approval

• Get authorizations signed after revision

 Document as you go

35
Data management workflow 

• Managing incoming data

– How will the data be received? 

– How will data quality be maintained

– What data need to be integrated?

– How will you integrate/link the data? 

• Software tools to capture, link primary & secondary 

data? 

– Access 

– SAS

– SQL Server Management Studio

– REDCap 

• Establish routines to document as you go 

36
Data management workflow

• Accessing secondary data in the VA

– What do you need to prepare?

• Research document packet 

• Data source-specific request forms

• If real SSNs requested, justification showing why they 

are essential

37
Data management workflow

• Factors determining data access process

– Authorization

– Source (what data)

– Level of access (national, VISN, local)

– Identifier (real or scrambled SSN, IENs, etc.)

– Data source location (Mainframe, CDW, PBM, PCS, 

VIReC etc.)

– Data format (SAS, SQL)

• Note: VIReC Database and Methods series, Research Access to 

Data November 4, 2013 38
Data management workflow

• Data management activities documented

– Decisions made in team meetings (minutes, action 

items, etc.)

– Variables pulled from each data source

– Data linkage methods

– Variables derived - definition & process

– Programs developed & revised

– Add to master document as you go

39
Data management workflow

 Data quality 

 Examine the dataset 

 Identify common data quality issues

• Missing data 

• Differences in data type 

• Undocumented and out-of-range code values 

 Identify other/systematic data issues 

 Critical data elements

 Document the process, findings and response as you go...

 

40
Data management workflow

• Checklist for documenting analysis file

preparation

– Data sources 

– Source variables 

– Linkage process

– Derived variables

• Program code – including algorithms

• Algorithms described in standard English 

– A codebook for every dataset

41
9/9/2013 42

Data management workflow
9/9/2013 43
9/9/2013 44
45

Example: Documenting derived variables

Data management workflow
Documentation: Analysis File Preparation Program List
Data management workflow

• Documents for a program walkthrough 

– Prepare 

• Task, definition, instructions, decision

• Program list

• Program code

• Log of program execution

• Output from program

• Processing narrative 

– Purpose 

– Decisions

– Program step by step

– Outcome

47
48

Example: Cohort 

revision 

documentation 
49

Example: Cohort 

revision 

documentation 
50

Example: Cohort 

revision 

documentation 
51

Example: Cohort 

revision 

documentation 
Data management workflow

• Elements of analytic file preparation walkthrough

– Summary of purpose of each program

– Processing decisions and rationale related to program 

– Differences from previous similar programs for this 

project 

– Program name, main function, data source name & 

year

– Step by step description of program

– Variables added, dropped, changed

• Description of how new variables were created

– Sort & index of output data 

– Links to log, list, freqs & stat files

Signature of reviewers

52
53
Session 2: Outline

• Getting started

• Importance of documentation

• Data management workflow

• Analysis workflow

54
Analysis workflow

• Organizing data for analysis

– Master datasets and work files

• Data and document versioning

• Raw data files and statistical analysis files

• File structures 

– Flat

– Hierarchical 

– Relational 

– Longitudinal

– Data back ups

55
• Documenting statistical modelling & analysis

– Narrative

– Programming code with annotation

– Description of model 

– Rationale 

– Statistical package

– Results

– Analysis final walkthrough

56

Analysis workflow
Analysis workflow

57

Program 

description

Cohort 

handling

Example: Analysis program Hynes, Cancer Survival Analysis
Analysis workflow

58

Extended 

Cox Base 

Model

Example: Analysis program Hynes, Cancer Survival Analysis
Analysis workflow

59

Kaplan-Meier 

Survival 

Estimates 

Example: Analysis program Hynes, Cancer Survival Analysis
60

Analysis workflow

Stata: Time-to-Event Survival Analysis Model
61

Analysis workflow

Kaplan-Meier Survival Estimates
62 Data & Programming Documentation Example
63

Citation: Kahn, MG, Weng, C. Clinical research informatics: a conceptual perspective.J Am Med Inform Assoc. 2012;19:e36-e42. 

Example of an Integrated Informatics-Enabled Clinical Research 

Workflow Concept
QUESTIONS
Contact Information

Denise Hynes, PhD, MPH, RN 

VA Information Resource Center

A VA Health Services Research & Development Resource Center 

working to improve the quality of VA research that utilizes 

databases and information systems

Hines VA Hospital

VIReC@ va.gov 

708-202-2413

65
Session 2: Managing and Documenting Data 

Workflow

• Recap

– Getting started

– Importance of documentation

– Data management workflow 

– Analysis workflow 

66
Session 3: Planning for Data Re-use

• Preview

– Data activities at project close

– Why make research-generated data available for 

re-use? 

– Policy on sharing data for re-use 

– Planning for sharing data for re-use

– Documentation required for re-use

67