**Transcript of Cyberseminar**

**VIReC Good Data Practices Miniseries**

**Managing and Documenting Data Workflow**

**Presenter: Denise Hynes, PhD, MPH, RN**

**September 10, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact the VIReC help desk at virec@va.gov.*

*Moderator:*Good morning and good afternoon, everyone, and welcome to day two of VIReC’s Good Data Practices Miniseries.  The Good Data Practices Miniseries is a set of lectures that are focused on good data practices for research.  The series includes five sessions that will take place this week, Monday through Friday, from 1:00 to 2:00 p.m. Eastern.  New researchers and research staff planning new projects may find this series beneficial.  Thank you to CIDER for providing technical and promotional support for this series.  

Today’s lecture in our Good Data Practices series, Managing and Documenting Data Workflow, is presented by Denise Hynes.  Dr. Hynes is Director of the VA Information Resource Center, VIReC, and research career scientist as the HSR&D Center of Excellence at Edward Hines Jr. VA Hospital in Hines, Illinois.  Dr. Hynes holds a joint position at the University of Illinois Chicago as Professor of Public Health and as Director of the Biomedical Informatics Core of the University’s Center for Clinical and Translational Sciences.  Questions will be monitored during the talk and will be presented to Dr. Hynes.  A brief evaluation questionnaire will pop up when we close the session.  If possible, please stay until the very end and take a few moments to complete it.  

I am pleased to welcome today’s speaker, Denise Hynes. 

*Dr. Hynes:*Thank you, Arika.  Thank you, Heidi.  Good morning, good afternoon, everybody.  Hopefully, you’ll have your Adobe Connect all set up.  Mine isn’t working today, so I will be doing it in low tech, and Erica will be supporting me with the slide advancements.  Not to be discouraged.  I want to encourage you to take advantage of the technology and put your questions in as we go along.  We got some feedback from our lecture yesterday and it’s extremely helpful.  We will try today to pause.  I will try to pause and see what questions we have along the way.  Either way, I would encourage you to put your questions in, so that we can address them as we go.  

Audio is good, Heidi and Arika? 

*Moderator:*Yes. 

*Dr. Hynes:*For those of you who are new, yesterday our first lecture addressed, in session one, early data planning for research.  We talked about the importance of data planning, factors that influence data needs and additional data planning for IRB submission.   We also went over a data planning checklist.  On slide three, we’re highlighting the following topics for today focused on managing and documenting data workflow.  We’ll be talking about just initial overview aspects.  I may, because of the volume of slides that we have, try to get to the specifics with some specific examples that are embedded in today’s presentation, and I may skip over some of the lists that are ones that you can refer to later.  We’ll talk about the importance of documentation, data management workflow, and analysis workflow in particular, with some examples. 

Let’s just start out with a poll.  It would help us to know a little bit about you.  Tell us a little bit about your level of research experience.  Our poll question asks if you are on a continuum from one to five, novice being a one, five being an expert.  I’ll let Arika or Heidi tell me how it looks on the poll.  

It’s good to know where our audience is.  For those of you who aren’t aware, we actually have more than—I think I heard a number that was in the hundreds today, so I recognize that we have a diverse audience.  I will also try to be aware of acronyms.  Should I miss the opportunity to define an acronym, let me point you to an important resource on the VIReC website.  This issue comes up a lot, and we’ve actually started to maintain a database of acronyms, but I will try to be diligent today in minimizing the use of acronyms as much as possible.  

How’s our poll question, Erica?   

*Moderator:*It looks like for novice it’s 13 percent; two, 19 percent.  The highest percentage is three; four at 24 percent; and expert, 7 percent. 

*Dr. Hynes:*Okay, good, so a good mix. 

*Moderator:*Yes. 

*Dr. Hynes:*Thank you.  Let’s get started.  One of the things that we talked about yesterday is a lot of the components that we’re addressing in good data practices sounds pretty foundational to doing research.  When you put it all together and we talked with colleagues about which aspects or how robust their own research is in addressing some of these issues, some are better than others.  Our goal is just to lay out some of these foundational issues, especially as health services research has become more inclusive and reaches out to researchers and collaborators in other disciplines.  It’s good to think about how we do data management and data analysis together and think about some of the principles that can bring us together in doing the work well and systematically. 

Let’s just talk about getting started.  When you’re kicking off your project, this is when we talked about—yesterday we talked a bit about, on slide six, you really should be starting your data management plan before your funding notice comes.  This should be an ongoing process.  In particular, you really need to have formulated your data management plan, at least at a high level, before you submit to your institutional review board because it is really required for the research review to understand the logistics of your project.  Formalizing your data management plan, these are some of the aspects that we talked about yesterday.  I won’t go over these details again, but one of the aspects that I left you with yesterday was what experience you have had developing a data management plan.  Some yesterday had said that they had some experience with working with software.  

Our poll question next is, how much experience have you had developing a data management plan?  One-to-five scale, one being none, to extensive experience number five.  I would like to pose a challenge as well.  For those of you with extensive experience and answered yesterday that you use a data management software planning tool, I would encourage you to type into the comments area, what tool do you use?  Tell us a little bit more about that, so that we can share in your knowledge and see if we have some insights into that as well.  

Are the responses for the poll question coming in quickly, or should I go to the next slide? 

*Moderator:*We can address these.  For none, it’s 23 percent.  It looks like the highest percentage is number two at 35 percent, 36 percent.  For three, it’s 27 percent.  Four, it’s nine percent, and extensive experience, about two to three percent.  

*Dr. Hynes:*You may be learning by the end of today’s lecture that you probably have more data management planning experience than you think.  I’m going to move to our next slide, eight.  If anyone types in any other examples, we’ll check in after we talk about this specific example.  I wanted to just finish out an introduction to a tool that I mentioned yesterday called the data management planning tool that we actually discovered and have no experience using.  On the UCLA, University of California Los Angeles, Library website, where they actually have posted something called the DMP tool—this is slide eight—they actually have a nice consolidation, if you will, of several of the requirements of the funding agencies and the requirements that they have for data management plans, which really addresses some of the aspects we’ll talk more about tomorrow with regard to planning for data sharing.  Something to have in your mindset when you’re doing data management planning.  

I thought what I’d show is how this data management planning, the DMP tool, works.  When you see the next slide, on slide nine, I really like this example since I had the honor and privilege to spend a little bit of time in Hawaii this summer.  The example that they have is one that has absolutely nothing to do with health services research, but you can see the similarities across different types of research with regard to data management planning.  This tool helps you to produce structure, again, at a high level of describing your project in terms of its data management.  It looks at issues around types of the data produced where you have to summarize that.  The data and metadata or the documentation standards you’ll be using.  Policies for access and sharing, again, thinking about that long in advance when you’re starting your project, starting to plan for it.  Policies for re-use and distribution.  I chopped up this example, just so you could see the categories and how it lays out. 

On slide ten, you can see that it also talks about some of the other aspects.  It produces, basically, a document that addresses these components of data management planning.  What I found in my review of this particular tool is it provides a systematic summary of the kinds of issues that are important to address.  The degree of granularity is up to the user.  You can make this a longer or shorter document depending upon how much information you can submit to the actual DMP tool.  I introduced it because it is a way to structure your data management plan, and it’s something that’s freely available. 

Let’s check back with Arika on back to slide 11 that has the screen shot of the DMP tool.  Do we have any suggestions or experience that anyone’s written in about data management planning software tools they’ve used? 

*Moderator:*I don’t see any from my view.  What about you, Heidi? 

*Moderator:*We have one person who mentioned they used Microsoft Access.  Another person wrote in, “We do not use a specific data management software for developing plans.” 

*Dr. Hynes:*Okay.  All right.  Thank you.  Well, maybe by tomorrow we’ll have a poll question that asks folks if they’ve tried the DMP tool and see what they think about it.  For those of you who will be on again tomorrow, we look forward to you helping us with evaluating the utility of that tool for your own research, and then you can chime in on our comment bar tomorrow.  

Let’s talk now about getting started.  I’m on slide 12.  Let’s talk a little bit about defining roles and responsibilities.  I’m just going to pull out some specific aspects and try to give you some examples.  Really important to try and talk about, especially in the early stages of your project, maybe now you’re past the IRB and you’re now into the granular level of getting your project actually started.  You’re planning your kickoff meeting, for example.  You need to think about who will be doing what on your team, specifically with the data management.  Who will be responsible for the project and data files?  Do you have a data coordinator?  Who will develop and enforce the naming standards of your variables and your concepts?  How will you do that?  Who specifically needs to access the data that you’re going to be collecting, acquiring, constructing, preparing, normalizing?  Who will be responsible for each of these steps?  Also, will the same person or persons be analyzing the data?  How many people will be analyzing the data?  Do you need to have some sort of shared environment to support multiple users and looks at data for different roles?  How documentation will be included in the daily workflow of the project.  

Really important to think about, as you are defining your roles and responsibilities, getting that solidified and a shared document in your project team that you refer to.  Is your analyst also your data manager?  Is your statistician wearing two hats?  You have multiple statisticians, multiple data managements, data collectors.  At what point does their role begin and end?  

Let’s ask a third poll question, so we can again, know a little bit more about our audience.  We’d like to know what your primary research role is.  We have four categories here:  investigator, data analyst, programmer or statistician, research coordinator or research assistant, and a fourth category is student trainee or fellow.  For those of you who fall into an “other” category, I guess we get to give you a break on answering this poll. 

*Moderator:*It looks like the highest percentage is data analyst; it’s 43 percent.  Next is research coordinator at around 27 to 28 percent, investigator at 18 to 17 percent and student trainee at about 9 to 10 percent.   

*Dr. Hynes:*Okay.  Thank you.  We have a lot of people who have hands on the data and hands on the data management and analysis plan as well.  Appreciate your experience as we go along in some of our questions that we’ll pose later.  We welcome your questions as well.  

**Let’s talk about the importance of documentation, especially since we’re talking a large part of our audience are the analysts and the research assistants.  You probably know more than anyone how important documentation is when your principal investigator comes back to you and says, “Now, how did we define that variable, and can you show me where we made that decision?” and it becomes potentially your job or a colleague’s job to find where you wrote that down someplace in your documents that you’re using for your team.  

On slide 15, importance of documentation, what should be documented?  When we talk about documentation, we don’t want to slow down the project, so we have to find efficient ways to write these things down.  First, what do we need to write down?  I have four categories on here.  Data management methods, how are we going to conduct this project in terms of data management?  Aspects about the role.  How are we going to do this?  With specific software, maybe that’s pretty specific and pretty straightforward.  Data analysis methods, again, the types of decisions that you make and, in particular, documenting as you go is really important.  Also, how did you come to your findings?  How did you get to the point of this particular analysis technique?  It’s good to know what analysis techniques you tried early on and decisions you made about why that one will not continue. 

We know well in research, it’s not a linear process and you learn as you go.  That’s why we call it research.  There’s aspects to learning within the research projects and making sure to learn, you really have to write that down.  Document what decisions you made and how you decided to do it that way, instead of an alternative approach.  Also, what data were created and how?  Technical documentation, sometimes it’s called a codebook.  We’ve given new names to this.  Metadata.  Really, when you think of metadata, you think about granular detail about how specific variables and constructs are defined in a particular dataset.  Another aspect is, how did you get to that particular construct?  It might be a derived variable, so you might need to describe a little bit more of a process, as opposed to this is a very concrete cross-sectional measure.  If it’s something that’s been derived or it has multiple sources that contributed to it, that information is truly important. 

Slide 16.  I just wanted to mention a little bit about, again, introducing you to some documentation standards.  There are several initiatives; I picked out two that are shown here.  Actually, I guess three, really, to highlight.  Only two are shown on the screen.  There’s something called the Dublin Core Metadata Initiative.  Sometimes it’s just known as DC or Dublin Core.  Another, Clinical Data Interchange Standards Consortium, also known as CDISC.  Then there’s DDI, Data Definition Initiative.  I mention these because for those of you whose responsibility it is to provide metadata, to provide the codebook, to provide the technical documentation for the datasets that you’re using in your research, or if you’re producing new data in your research, these standards, if you will, data standards resources, may provide some guidance as to how to structure that documentation as you go and some of the components to it.  I would refer you to these resources to consider how granular your documentation should be.  

One recommendation is, if your project will, at the outset, produce a data resource for others to share, I strongly encourage you to look at these data standards resources.  It really becomes crucial for researchers following you who may want to reuse your data, or if you want to share your own data with your next research team, to have strong standardized data documentation.  

Let’s go to slide 17.  The next couple of slides really get at issues that are recommended by—I mentioned yesterday the Inter- Consortium for Political Science and Research, ICPSR.  These are some examples of key components to documents, slide 17 through slide 19.  I just want to go over these very briefly, and we’ll get into some examples a little bit later.  Slide 17 highlights some of the key components to document sampling, weighting, aspects about your variables and your data sources, units of analysis and observation.  Some guidance on slide 18 about what to document about variables.  Really crucial.  We think a lot about variables, both dependent variables, independent variables and research.  These are not always necessarily concepts that are very black and white as we’re starting out our research project.  They may be derived variables.  They may be derived from a whole battery of questions on a survey.  They may be constructed from multiple data sources.  This construction of your derived variable is really important.  

Another aspect about your variables is if you have some sort of code sets for responses or categorization of values for your variable, you need to document the exact meaning of those codes and especially if there are any codes that represent missing data so that it’s not used as a numeric.  Especially for research that has included some kind of a survey or population component, if there’s any weighting that’s necessary to actually do the analysis.  Obviously, very crucial for your methods and your results to be valid.  These kinds of details are really important to summarize and document.  

Slide 19 talks about, again, source from ICPSR, summary documents that you should maintain.  Again, taking it up from the granular to the summary.  Aspects about your project.  In a clinical trial, an intervention study, oftentimes we refer to standard operating procedure or manual of operations.  Even when you’re not conducting a clinical trial, you need to have a good, solid operations manual, if you will.  Technical information about data files, data collection instruments, the flowchart of your data collection instruments or flowchart of the data sources that you use.  An index or table of contents, it seems obvious, but sometimes *[audio breaks up 24:27].*

*Moderator:*To the audience:  Thank you, everyone, for joining us for today’s HSR&D Cyber Seminar.  Today’s session is the second half of the second and the third session of our VIReC’s Good Data Practices series.  Unfortunately, we did have trouble with our audio bridge yesterday.  The entire system crashed, and we were unable to continue the session. We were just at about the halfway point.  We’ll be finishing the second half of the second session right now, and then at 1:00 we’ll be switching over to the third session, Planning for Data Re-use.  We’ll be transitioning very quickly between the two, so just stick with us.  For those of you who are able to join us early for this second half of the Managing and Documenting Data Workflow, thank you very much.  We appreciate you being able to rework your schedules to join us for this.  Thank you very much.  We’ll be getting started in just about three minutes. 

If anyone is looking for handouts for the Managing and Documenting Data Workflow session, please shoot us an email at cyberseminar@va.gov and we will get that out to you.  I just didn’t have enough space on the screen.  It would have gotten a little cluttered to put that hyperlink up here.  If you are looking for handouts for planning for data re-use, you can click on that “click here for today’s handout” link on your screen, and you will be able to download those handouts.  If you are wanting or needing captioning, we do have that available.  It will be the same length for the full hour and a half.  Just click on that “click here for today’s live captions.”  It will open up in a separate browser window, and you’ll be able to follow along for the full hour and a half there. 

Also, we are recording today’s call.  We will make that available in our catalog archive.  We will be sending that link out as soon as it is available, hopefully tomorrow.  It will be a separate link for each session, and we will get that out to everyone as soon as it is available. I know with yesterday’s splitting up the session, not everyone was able to rejoin us today.  I’m sure there will be a lot of people who are wanting to grab that archive for that.  We will be making that available as quickly as we can.  We’ll be starting in just about one minute. 

Just a reminder for the audience that we do run our calls in lecture mode, meaning everyone’s lines will be muted throughout today’s call, except for the presenter.  We do anticipate that you will have questions that come up during today’s call, and we would ask that you use the Q&A screen in GoToWebinar to submit those questions—I'm sorry, the Q&A screen in Adobe Connect to submit those questions in to us.  The Q&A screen is located right on your screen, in the lower right-hand corner.  Please type those questions in when they come to you.  We’ll hopefully be able to answer some of them during the session.  If not, we will hopefully have time at the end of both parts of the session today to get those questions answered.  Please do send those in to us when they occur to you.  We don’t want you to forget about them.  We want to make sure that we have time to get the questions answered. 

Arika, do I have you back yet? 

*Moderator:*Yes, I’m here.  

*Moderator:*Perfect.  We are just at 12:30, so I am going to turn things over to you. 

*Moderator:*Thank you, Heidi.  Good morning and good afternoon, and welcome to the second half of VIReC’s Good Data Practices, session two.  Today’s lecture in our Good Data Practices series, Managing and Documenting Data Workflow, is presented by Denise Hynes.  New researchers and research staff planning new projects may find this series beneficial.  Questions will be monitored during the talk and will be presented to Dr. Hynes.  A brief evaluation questionnaire will pop up when we close the session.  If possible, please stay until the very end and take a few moments to complete it.  I am pleased to welcome today’s speaker, Denise Hynes. 

*Dr. Hynes:*Thank you, Arika.  Thank you, Heidi.  Just testing the audio to make sure everything’s working okay.  

*Moderator:*Sounds good. 

*Dr. Hynes:*Welcome back, everybody.  Hopefully, we can pick up a little bit where we left off yesterday.  Today’s lecture, I’m going to jump to slide 22 so you can see where we are.  Session two’s outline on data management workflow focuses on these four areas.  Getting started, we got mostly through importance of documentation.  Then we’ll cover data management workflow and analysis workflow.  In the interest of time, I am going to skip to the last two topics, data management workflow and analysis workflow.  We’ll be beginning on slide 22, where I am right now.  For those of you who have handouts offline, I’d ask you to jump to slide 22.  

The reason I want to start here is because this is where we really can show some examples of how to operationalize what we’ve been talking about for our first one and a half lectures.  I would like to just jump into that today.  If there’s any questions about the when and where to document, we can always go back to that.  Then we’ll also try to do some pauses for questions between each of these sections, so that you can anticipate that we will read your Q&A as we go along.  

Let’s just dive in here.  Data management workflow.  Again, I’m going to have some lists here that will mostly be cues for you to think about.  Aspects of data management workflow and documentation is our theme, but I’ll also try to give you some examples.  Also, I want to make sure I mention and acknowledge some of the good help that we have from some of our research team at our Center of Excellence on two of our research projects, Lucy Zhang and Tom Weichle.  Lucy and Tom really helped with some examples for how we actually do some of our documentation and research projects.  You’ll see some of those examples that they provided input on today, in addition to the others I’ve acknowledged earlier. 

In terms of data management workflow, again, we’re thinking now your project has started, context here.  Our first lecture in our series was on just early planning now.  Our research project is really getting going, and we really have to make sure that we understand our processes.  With data management workflow, the first step is to settle on a file organization plan for your data and documentation.  This could also start quite early; it doesn’t have to wait till you get your, quote, funding notice.  We’ll go through some of the issues about documenting cohort derivation, issues to address if you’re collecting primary data and if you’re accessing secondary data in the VA, aspects about linking data from multiple sources, cleaning data, documenting the analytic dataset and program walk-throughs, if you will.  

Different issues around file naming standards and organizations.  Again, these are some examples of what we do at our center.  It might be different with regard to how you might implement it at your center.  It might be different if you’re using a different sort of file share server.  I guess the message I want to really have you take home is you need some kind of standards and organization.  If what we do might be useful to you, by all means, try to adopt it, but you really need to settle on an organizational plan.  You need to come up with some organizational concepts for naming files.  Make sure they’re easily accessible, accurate and clear.  Remember, not everybody on your project is a data manager.  You need to make it accurate and clear for all members of your team who might be accessing these file share folders, if you will, or files.  Make sure it’s meaningful to everybody, they’re easily distinguishable and they can be recognized in different environments. 

Here’s how we’ve set it up for our projects, and this is pretty typical of what we’ve done.  It’s a theme through both our data organization and our programming organization and all of our, if you will, results organization.  I mentioned earlier on that, as you’re working through your early stages of your project, one of the aspects that you should anticipate early on and plan some of your data management around is your aims, your hypotheses and papers, manuscripts.  That has really been how we oriented a lot of our organization.   We found that papers tend to dictate how we will do our results and analyses, how we do our data management, how we do our early data planning.  Ultimately, that’s how we organize a lot of our work.  

We have a file share system.  It’s on our VHA server within the VA firewall.  We have a project documents folder.  We just happen to call it HSRFILES, circled here.  Then we have folders organized by programs.  Each of those within the programs, we have it organized by program folders.  In our situation, we looked at cost, survival and adverse events.  Those happen to also be in line with the manuscripts that we had planned early on.  That also helped us with organizing by our outcome measures as well.  When we got to actually doing our analysis, procedures were actually pretty well lined up.  We organized our program document folders according to each of these manuscripts or outcome measures, and then I’ll talk a little bit about how we have specific activities also documented:  data walk-through, our codebook, administrative aspects and publications and presentations.  

As you can see, the data folders are similarly organized.  We have not only—if you go back here, you see our HSRFILES, our programs, our program documents and then HSRDATA, the datasets that we maintain.  They’re organized, again, by a similar organization, cost, survival and adverse events.  Then we also have what we call HSRXWALK.  This is just what we named our identifier, “crosswalk,” in a secure folder which has very limited access.  In fact, on our team, I think we only have two people, two individuals who have access to this:  our data manager and then our OIMT person who manages our servers.  This gives you an idea of electronically, if you will, on your file servers, how to organize your project digitally.  It’s really important to just have an organizational plan, one that works well for your team and for your computing environment.  

Let’s talk a little bit about some of the work that you do in data management.  Again, I’ll try to highlight some examples.  One aspect is documenting cohort derivations.  Whatever study you’re doing, whether it’s for individual person subjects or whether it’s a unit of analysis is specific events or facilities, whatever the unit of analysis is, you often have to, and pretty routinely, have to document how you’ve derived that cohort or population of interest of study.  You really need to document that pretty clearly with inclusion and exclusion criteria, what sources you used, the rationale.  This is really critically important to document as you go.  As those of you who are involved in research projects know well, you often make decisions as you go along.  You might modify it; you might change your minds about something that’s working well or redefining, adding some exclusion criteria.  You really need to keep track of it.  Diagrams, of course, are really critical because they’re required for publications and there are standards for this.  

I have three examples here.  I’ve taken these from publications that are out in the public domain now.  Whether it’s a clinical trial that you’re involved with, which has very specific standards for what you report in your different arms, keep in mind that, even if it’s a randomized clinical trial, depending upon the outcome measure that you’re presenting or that you’re analyzing, you might need to modify your cohort flowchart because the cohort could change. 

This example that I’m showing on the screen shows that for one analysis, we used a cohort that had 927 and 944 in the two comparison arms; but when we did our cost analysis, it’s 687 versus 708.  You need to keep track of the different analytic cohorts and make sure those are clearly documented.  If you’re doing an observational study, the standards are still expected.  If you’re publishing a manuscript, VA funded and, in particular, NIH funded, these cohorts must be clearly documented.  

How do you keep track of that in your data management workflow?  What we do, and again you may have some other procedures, but what we’ve done in our study might work for you.  We actually maintain what we call a walk-through document for creation of our cohorts.  We actually go through—we have a specific analyst who’s responsible for it.  We date it.  We annotate on our document what the file locations are, where we actually are keeping this cohort dataset.  Where is it on our computing environment?  What’s it called?  Was it modified at all?  

We also are very careful to document if there is any kind of multiple source data.  You can see in this little section here, that I’m not drawing very well, under the SAS dataset name, we have listed here specific datasets that are included in the dataset that is our final cohort.  We also have a description of what that dataset is doing and what we’ve done to it and what it actually contributes to the cohort.  Information about the number of records and the number of variables is also important.  You want this to be recognizable to somebody who walks into your project after this work is already done.  Again, plain and simple, clear and accurate in any environment is really important.  

If you’re doing primary data collections, some aspects to consider with your data management workflow.  Some aspects here.  Definitely, you need to keep in mind aspects about making sure that your data collection forms are final, and that you verify that these are in fact the final ones.  Keep track of what the final forms are.  Often, with IRBs now, you actually have to have that final data collection form stamped and filed.  You need to retain it, but you want to retain with the data forms, information that’s useful for understanding how it’s going to be organized in your data but also information that needs to be included in your forms about any kind of authorization language.  If your subjects, for example, are individuals, whether it be patients or providers or employees, make sure that you have the corresponding consent language and authorization language for any kind of future data linkage.  For example, if you want to link these data with Medicare data, you have to make sure you include that in the consent process, or you’re going to have to go back and ask your subjects later to do that.  Make sure you think about that and anticipate how these data will be coming in.  

In terms of HIPAA language as well, you need to make sure that you consider this and document it as you go, whether you use a walk-through document such as what I described for documenting cohort decisions.  You might come up with a process, for example, to annotate a version of your form that shows where you’re retaining the final forms of these in your file management system and when it was actually finalized or when it was modified. 

In terms of managing incoming data, these are some aspects that you really need to consider.  I’m not going to go through all these questions, in the interest of time today, because I want to make that we get to some examples.  I am going to skip to slide 41, and I’ll be happy to answer questions about any of the ones that I skipped over.  I might be going too fast.  We’re going to jump to Checklist for Documenting Analysis File Preparation.  This slide talks about data sources, where you need to document your data sources.  What datasets, databases, criteria for data extracts that might have been prepared by others.  Source variables.  What was a specific data source or sources that were used to construct any derived variables or if it’s a source variable from a specific dataset, you should actually annotate what the original variable name and code values were in that original source data.  If there was some sort of linkage process involved or this is a link variable, for example, that links multiple datasets, that should really be highlighted, and you should note any conditional linkage criteria that were used.   

Derived variables.  It’s really important to include in your file management system program code, including any kind of subscripts, if you will, in your algorithms, algorithms that also have some annotation described in some sort of standard English, whether it’s annotated within the code or there’s a separate document.  Come up with a routine, so that you can refer to both the explanation of what the code means and what it’s actually doing and the code itself and keeping a codebook for every dataset that you’re using, especially if your analysis file is really critically important. 

Some people actually try to maintain codebooks of the source data.  That can become very cumbersome and also very ambitious.  I have to say that in our research, we haven’t been able to do that, but we absolutely stick to the rule of maintaining a codebook for any dataset that we use regularly and that we create.  

Here’s an example of data management workflow documentation.  We keep track of the input datasets.  Also, in our situation, we try to retain the name of the original dataset.  It helps us keep track of things.  In this example, we used a SAS dataset.  We try to retain some semblance of the name that it came from, but that may not always be—you may not be able to do that, so it’s really important to keep a description that you understand.  We understand in this situation that this is calendar year 2002, the Medicare denominator dataset.  You might need more explanation for your research team.  We also keep track of the records and how many variables.  It just helps us recognize the datasets better. 

Again, here’s an example from one of our research projects.  We also keep track of any data that might have been received.  If part of your workflow requires acquiring data that comes in from other places, whether it’s within VA, external to VA.  In our study, we actually acquired data from several national cancer registries from across the country, so we had to keep track of different datasets.  This is an example of the VA Central Cancer Registry coming to us that provided data for one of our research projects.  This is literally just a Word document that we maintain.  We date it, we update it, and we keep the latest version.  We acknowledge, in plain English, what data we got.  We have it cut off here; I wanted to make sure and fit it on the screen how it was received.  Was it received as an ASCII file?  Was it received as a SQL file?  Did we do anything to it?  What did we do?  That can be really important if there’s any sort of example that could—you come across problems later on that might be due to some sort of data transfer process.  That is also important to note.  Then, where we saved it, and then the kind of variables that were retained and ultimately used.  

Another important aspect.  Again, I hope this is something that you can use.  We also keep track of what we call a sequence number and the date, and the reason.  Sometimes we might modify a specific program or some routine to create a variable, or create a construct, or create a subcohort.  If we modify that particular program, we try to keep track of it by giving it a sequence number, a date and a reason.  This is just an example of how we keep track of that.  We even keep the narrative to describe what that particular job, if you will, is doing and why we did it.  

We also keep track of any new variables that are created.  What the variable name is, what we’ve given it, and then what it is intended to be used for.  In our particular study, here we have an example of an indicator flag if the patient was enrolled in an HMO in each month from each of the calendar years that we have data.  For us, this was important because it meant that there were—it was an indicator to us that there were certain types of data that were going to be missing.  That way, we didn’t spend a whole lot of extra time wondering why individuals with this code didn’t have utilization data.  Again, those are some aspects.  That’s the art of research, if you will, or the science, where you need to develop derived variables and flags that may not be part of your analysis, but is really important for you to manage your data and understand how to construct your analytic files later.  

This is another example of how we keep track of our final programs and what ones we’re really going to use.  There might be a lot of—there could be weeks of developing a code and testing it out on different components of our cohort, but what’s the final program code?  We actually have a walk-through for this.  By a walk-through, a walk-through should include more than one person.  You’re walking with somebody to make sure that what you’ve done works.  There should be a minimum of two people who do the walk-through together.  Walk through the program and make sure that not only is it organized properly and that you’ve maintained it in a well-thought-out process, following the rules we’ve just described.  Annotation within the code is understandable to both people.  It actually works.  Also, to keep track of it in some sort of documentation file, so that once you’ve done your walk-through and you’re happy with it, you say what sequence you like.  You keep track of the dates.  You give it a name and you describe it, again, in plain English, or at least in a format that you and your team will understand.  Then describe whether or not you have some sort of a document that supports it.  This is just an Excel spreadsheet.  You can actually retain a Word document, either within it or behind it, that corresponds to each of these.  We actually keep these for each of our projects.  

I’m giving you some specific examples.  I’m sure, if we had time to poll the audience, you could probably tell us about many examples that you use to keep track of your data management work flow.  Make sure that you come up with a routine, you prepare documents for your walk-through that describe tasks, definition, instructions, decisions you’ve made.  I’ve gone over some of this as examples.  Really important to make sure you don’t lose track of the results of your code, so the output.  Sometimes it’s difficult to catalog all that and keep documentation of it.  In fact, it might not really be necessary as long as you retain the program, the documents about it, the step-by-step and some summary of the outcome.  When you’re working with, if you will, static data, that’s probably accessible, not having to retain all your outputs.  

This is an example of a walk-through form that we’ve used.  This is what we use to revise one of our cohorts.  Not only are we keeping track of the name of the program being updated and a brief description of the program, the analyst making the update, the dates, et cetera, but you’ll also notice that we have also had it signed off so that both people who went through the walk-through, whether it’s your PI, the project manager or the data analyst and the statistician, that you have documentation that you can refer back to a year later of what your decisions were that you made.  We found this really incredibly helpful.  If we want to go back and modify it, it also helps us to understand how we have to modify it. 

Elements of analytic file preparation.  This is just a summary of what we just described, so I’m going to go through this quickly.  I want to make sure also that you understand how important it is, with the resources that are becoming available and changes in the IT environment, for where you keep your data and where you keep your project management files, introducing you to the VINCI, VA Informatics and Computing Infrastructure.  If you haven’t heard about VINCI, I would encourage you to learn about it.  VINCI has a website.  VIReC has been including VINCI information in many of our seminars.  Basically, VINCI provides workspace for research projects in a virtual computing environment that maintains security and privacy of data.  I would encourage you, if you’re embarking on a new project, to seriously consider using the VINCI environment for your research.  Given all the changes that are coming up in IT, information security, consolidation of resources, this would really be a good place.  You can’t probably see all the details, but it does have a lot of the SAS, data, Arc, file management tools.  There’s SPSS available to use on the platform.  Also, there’s SAS and there’s SAS grid for big data.  There are a lot of tools that are available to use.  I would encourage you to become familiar with it because this is the information system platform for the future, centralized resources. 

I want to make sure that we have time to talk about analysis workflow before we move to session three.  Analysis workflow is a little bit different than data management workflow, thinking about how you really organize for data analysis.  It can be organized in parallel to what was done for data management workflow, but it’s a little bit different because there’s different aspects that you probably need to keep track of, such as what are file structures for analysis that might be a little bit different than what your data sources were.  You might need to structure your data files differently for different analytical tools.  These are also important dimensions to keep track of, whether it’s within annotated code or whether you actually have to retain output files.  Again, this is very highly dependent on the type of research that you’re doing.  If your goal is to develop algorithms to do work, that would require a different type of analysis management workflow and documentation than if you’re analyzing the impact of a drug and comparing two different populations. 

Again, very generic here.  You need to keep track of the types of files that you’re using.  If you’re actually working with SQL files, that requires a different type of documentation and organization than if you’re keeping track of flat files or SAS files. 

Please make sure that you’re taking into account also, in your analysis phase of work, backing up your data.  It’s not only for data management.  Make sure that you keep a backup routine for those really important operational analytical code sets and programming code because that’s obviously—think about the time it’s going to take to reproduce that.  Back it up.  Make sure that, in your analysis workflow, you consider documentation for these critical aspects, not only your statistical modeling and annotating your program code.  Don’t forget to keep track of the results they produce.  You don’t want to just produce the results and put them into your published manuscript.  You need to keep track of it in your project management.  Also, make sure that you have an analysis final walk-through.  Make sure that that includes your statistician, and it has to include at least one other person, same important rule as with data management. 

**Here’s an example.  You probably can’t read this very well, but really, I just want to point out this is SAS code.  If you expanded this, you could actually read it pretty clearly.  Make sure that when you’re doing your analysis in whatever programming language you’re doing it in, that you annotate your code.  Give it some kind of description in plain English within your code.  No matter how sophisticated your programmer is, insist that there be some sort of annotation that someone who’s not a programmer can understand what this program is doing.  Also, it should be understandable to the programmer because they’re the ones who are going to have to rerun it or revise it later, so you have to have that kind of annotation.  Make sure that you—this one happens to describe how we built our cohort and what we did with it, so it’s really important to be able to plainly see where that’s happening in your program. 

If you’re doing some particular type of analysis that is really important for you to go back to, make sure you catalog that in some sort of way within your code.  Here, we have an extended Cox base model.  We wanted to make—that was something we actually had to go back to very often, modify.  Knowing where that is and knowing how to find it is really important.  We also did some survival analyses, and these are aspects that we modified.  You call that out, so that you know where it is in your code.  

You want it to work so that—this is some data output.  You want to make sure that whatever annotation that you include in your code, if you’re commenting things out that are only going to show up in your programming code, there’s also labeling that you want to show up in your output.  Make sure that you use good and clear labels that explain clearly what it is that you’ve just produced, so that you can again, refer to it, and when you see your output separate from your program, you can see what was intended.  Then you know what you’ve produced.  

This is an example of just one output, if you will.  I discovered, in preparing for this lecture, that there are some things that we could probably improve upon.  We would welcome some sharing for those of you who do it better.  We don’t often keep track of our output very well because our datasets have been, for the most part, static.  We can always reproduce our output.  It saves us space, if you will, storage.  One dimension that you should carefully consider, if you’re like us and you don’t keep track of this in any kind of documented way, is if you’re working with datasets, source data, that change often, they’re more dynamic, you might want to very seriously consider how you keep track of your output.  Needless to say, if your source data, your analytic dataset is changing, your results will change as well.  If you want to retain those results that you put in that manuscript and you’ll have to go back to that later, you better save that output, and you better have some sort of good workflow for keeping track of that output.  Like I said, for us, our data, we’re not that dynamic.  The only modifications would be ones we would make to it.  Again, something to keep in mind for your research projects.  

We also keep our programming documentation, again, organized in the same way that we kept track of our data management workflow.  It’s completely parallel, and we keep it in our SAS program folders.  You could have the same thing with any output, should that be something that you need.  

I’m going to close with this very complicated slide.  It would sure be nice if all of these data management tools were available in one environment.  Actually, these are dimensions that are often discussed in clinical trials, clinical translation science to Word programs and in some of the conversations that we are, at VIReC, having with VINCI, trying to create an integrated, informatics enabled, if you will, clinical research workflow where you have all the tools that you need to do data management, file server project management, document consent.  There have been some examples described.  Here’s an example that was published in *JAMIA* last year by Kahn and colleagues.  

For the most part, we’ve seen this kind of environment built, I will say, on a project-by-project basis.  By project, I mean really large program projects.  The reality is, in VHA research, this environment doesn’t quite exist yet.  There are definitely critical components that absolutely currently exist on VINCI with regard to data storage, data management, analytical tools, but we don’t have all of these pieces.  Again, this is an ideal.  I think that technology is helping to move towards this.  In the meantime, we are probably going to have to piece together some of these things.  For example, data management tools are one that aren’t integrated very well in any of these environments.  

I’m going to close there to make sure that we have time to get our session three in.  I obviously didn’t stop for questions.  I want to do that now and make sure that we answer any before we transition to the next session. 

*Moderator:*There’s one question.  What does the sequence number pertain to? 

*Dr. Hynes:*Sequence number, we use it in our projects to really just keep track of when we modify a program.  We have SAS programs that, say, construct a derived variable, like we reformat the age categories.  This is just a simple example, but if we modify that code and retain it to condense 12 categories into 5, we’ll give it a new sequence number.  We use it more often in the analytic phase.  If we have a routine that we run a lot, and then we start to modify it in some particular way, we use the sequence number to just indicate specific aspects that we modified. 

*Moderator:*Great.  That’s it for questions.  Thank you to Denise Hynes for presenting today’s lecture.  If you have any other questions about today’s content, please contact the VIReC Help Desk at VIReC@va.gov.

*[End of Audio]*

Page 1 of 17