**Transcript of Cyberseminar**

**VIReC Good Data Practices - Controlled Chaos: Tracking Decisions During an Evolving Analysis**

**Presenter: Linda Kok,**

**May 22, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact virec@va.gov.*

Moderator:Good morning or good afternoon everyone and welcome to the third of four sessions of VIReC “Good Data Practices 2014 Your Guide to Managing Data Through the Research and Lifecycle”. Thank you to CIDER for providing technical and promotional support for this series. A few housekeeping notes before we begin. Questions will be monitored during the talk and will be presented to the speaker at the end of the session. A brief evaluation questionnaire will pop up when we close the session. If possible we ask that you stay until the very end and take a few moments to complete it. 

At this time I would like to introduce Linda Kok. Linda Kok is the Technical and Privacy liaison for VIReC and one of the developers of this series. Ms. Kok will present a brief overview of the series and today’s session and then introduce our speaker. I am pleased to introduce to you now, Linda Kok. 

Linda Kok:Thank you Joanne. Good afternoon and good morning, welcome to VIReC Cyberseminar Series “Good Data Practices”. The purpose of this series is to provide researchers with a discussion of good data practices throughout the research life cycle and provide examples from VA researchers. Before we begin, I want to take a moment to acknowledge those who have contributed to the series. Several of the research examples that you have seen in previous sessions were generously provided by Laurel Copeland of San Antonio VA; Brian Sauer at Salt Lake City; Kevin Stroupe here at the Hines VA and Linda Williams at Indianapolis VA. I also like to acknowledge the work of our team at VIReC. VIReC Director Denise Hynes; our Project Coordinator Arika Owens and VIReC Communications Director Maria Souden. Of course none of this could happen as Joanne said without the great support provided by the Cyberseminar team at CIDER. 

The research lifecycle begins when a researcher sees a need to learn more; formulates a question and develops a plan to answer it. The proposal leads to the protocol and the IRB submission. When funded and approved, data collection begins, then data management and analysis. The project may end when the study is closed, and the data are stored for the scheduled retention period. Or perhaps as we will see next week, the data generated can be shared for reuse and the cycle begins again. 

In the four sessions that make up this years “Good Data Practices Cyberseminar Series” we are following the steps of the research lifecycle. In the first session Jennifer Garvin presented the “Best Laid Plans: Plan Well, Plan Early” which looked at the importance of planning for data in the early phases of research before the IRB submission. Last week, Matt Maciejewski presented the “Living Protocol: Managing Documentation While Managing Data. This focused on adding details of the decisions made and the actions taken to the protocol as the project collects data and creates the analytic data sets. In case you missed them both presentations are available on the HSR&D website Cyberseminar link. 

Today Dr. Peter Groeneveld will describe ways to track research analytic decisions called “Controlled Chaos: Tracking Decisions During an Evolving Analysis”.  Finally on May twenty-ninth I will present “Reduce, Reuse, Recycle: Planning for Data Sharing”. I will look at how we can reuse our research data for additional protocols our own or for others. If you find the Good Data Practices Series helpful and you want to know more about using VA data, be sure to check out VIReC’s database and method cyberseminars hosted by CITER on the first Monday of every month.  Archive database and method seminars address introductory and advanced topics regarding the use of VA databases for research. For more information you can go to the VIReC website. 

Today’s session Controlled Chaos: Tracking Decisions During an Evolving Analysis will illustrate how well organized recordkeeping during analysis can improve our research and the subsequent publications and presentations. Our presenter today is Peter Groeneveld, Dr. Groeneveld is a Center for Health Equity and Research Promotion Core Investigator that is at CHERP; attending physician and Vice Chair of the Research and Development Committee at the Philadelphia VA Medical Center. He is also Associated Professor of Medicine at the University of Pennsylvania Perelman  School of Medicine and Director of the Leonard Davis Institute Health Services Research Data Center. I would like to now introduce Dr. Groeneveld, Pete are you on the line?

Dr. Peter Groeneveld:I certainly am Linda, thank you very much. 

Linda Kok:Thank you. 

Dr. Peter Groeneveld:Great. Good morning and good afternoon to those of you joining the seminar and a big thanks to VIReC for inviting me to speak on this topic. I am painfully aware that some of the audience may have actually worked with me in research so as I present Good Data Practices I am reminded of experiences I am having with my teenage daughter as she is learning how to drive. I tell her what her what are good practices behind the wheel and she gives me a look sometimes and says well you do not always do that. So I undoubtedly am presenting things that I think are good practices and which our group tries very hard to do. Of course like any system there are times when things have to happen quickly and when things need to be cleaned up later. I certainly do not present myself as a perfect practitioner of these guidelines. At the same time I think I have learned a great deal about how things can be done most helpfully so  I consider myself a learner in this endeavor as well and I would be eager to hear of your own experiences at the end of the session. 

Linda introduced me and there are some brief details on that. I have been conducting health services research at the Philadelphia VA for almost eleven years now. I am the PI of a current Merit Review Award through HSR&D as well as NIH-RO1. I have ten plus years of experience using VA data for research and I also Vice Chair the R&D Committee at Philadelphia VA. I would like to hear a little bit about you at this point and this is the awkward experience of lecturing to in cyberspace. You will see on your screen a poll that is popping up and please check the box that makes most sense to you whether you are a new research investigator, an experienced research investigator and those would be faculty level, new data manger analyst, experience data manager analyst, new project coordinator, experienced project coordinator, other or you decline to vote. I will call up the room, the cyber room a little chance to respond. I will give you a few more seconds. 

I would like to just read those results in very brief form. It looks to me that about and see if I have the percentages right about twelve percent of you are new investigators; twelve experienced; twelve percent new data managers; twenty-four percent experienced data managers; twelve percent new project or data; eighteen percent experienced project coordinator and eleven percent others. So an extremely diverse group. So we will end the poll there but thank you very much. 

I am happy we did that because I think what I will have to say will apply to any and all of you if you are involved in the research process and touch documents and that really is everyone in the research process from a research coordinator to the PI. This is going to be my agenda for today. First of all talk about the challenges of documentation in the research process give you some motivation for why you would want to create good documents because there should be some motivation for that. I will give you a schematic for organizing, research documentation, some good practices in the domains of that schematic and last but not least answer fundamental question which again I harken back to my teenage daughter why should I keep my stuff organized. 

So just moving ahead there the challenges of documentation in the research process are as follows. Almost anybody that is involved in research knows that the initial plans one makes in terms of a research protocol or a research project inevitably go through an evolution and that might be changes and refinements in the cohort selection process. Changes and modifications in the analysis. I am sure Matt Maciejiewski talked extensively about those last week. Forming a dataset and designing the analytics or evolving processes and really excellent research needs to evolve in the process of doing a project because  things are learned about the data, things are learned about the analysis and it is essential that the process evolved. It is equally essential that the evolution of the process be documented. The challenges that are thrown into any research project such as unanticipated data problems, absences of key variables, absences of key datasets. If you are working on a collaborative team like I am and almost all of us in health services research are, you may have multiple people treating and modifying study documents virtually simultaneously.  You may have a project manager who is writing up the protocol and you have research coordinators who are adding documents and a programmer or analyst who has discovered something in the data and needs to document that. You have the PI who comes up new ideas because she meets with the co-investigators and figures out a different way of doing things. Of course in the midst of this you have people leaving the study team, because they are moving across the country. You might have a change in PI. You might have a large protocol modification because a new dataset becomes available. Or in fact your research  group may move from one building to another or one side of the city to another and you have to completely reestablish your research documents. All of these things I found are fairly typical in the life of a multi-year investigator team. So all these things create challenges to documentation. 

So why should we do this well? One strongly motivating effect to be good at study documentation is that poor documents are very costly and costly in the following ways. Here may be something and I hope this rings a bell with some of you, but you are two years into a project, you finally collected all the data, you are ready to write up the manuscript and then nobody can quite remember the decisions of how the cohort was selected. So you are writing the method section, you have just no idea how the final cohort was formed because nobody can find the document that explained or the email or wherever else you wrote down. Maybe it was never actually documented maybe it was we all met in the room and made this decision and nobody wrote it down in some way that the manuscript write can access. I have already mentioned those transitions. On every team that I have been a part of has key transitions and if you lose somebody like the PI or somebody even more important like the project manager who keeps things organized and nobody can duplicate his or he memory you have lost sort of the intricate knowledge of the project that is necessary to keep things moving. This is really wasteful because it means that the lessons learned in the project have to either be relearned the hard way, going back to square one and redoing things unnecessarily. Or worse yet that mistakes or other adverse consequences of poor documentation creep in. The methods may not actually describe what was done correctly leading to embarrassing issues in terms of reports and publications that need to be corrected. Again I think there is a lot of cost to doing documentation poorly. 

Really we are talking about performing good science which means in the classic sense your science should be able to be reproducible by another scientific group that has access to your data and your methods. You might actually want to be the reproducer of such research. You might imagine for example what if your dataset becomes corrupted but you have all your statistical code and you have your documents explain what it is that you are going to do. Well that is a much better situation to be than if some how you had created a final dataset on the fly and nobody knows how we actually did it and the SAS code is difficult to read because it was written by somebody who no longer works at your VA center. You want to create reproducible results. This is critical of course for accurate reporting and scientific manuscripts. We actually want what is published in our manuscripts to reflect what we actually did. Also, as Linda mentioned in the introduction, because the research cycle may involve creating data resources that would be used by either your group or others for future research it behooves the original creator of those data resources to document exactly how such data were created. 

It is also essential for good project management and what I mean by that is coordinating the activities of a team of people who need to have their eyes on the same goals, who need to see the milestones passed by. The lack of clarity about such things can bog down the progress of a project, result in misunderstandings, unnecessary work, duplication of tasks. Really wasting the valuable time of an investigator team simply because there is no clear guide to explain well we accomplished this last week and the next step is precisely this and we need to accomplish it by two Friday’s from now. It also can be very difficult to manage from a PI perspective if there is no documentation indicating what was accomplished, what is about to be accomplished, what is the short term target for the next step in the process of research. If a PI cannot actually see that, it is very hard to know if progress is being made and is very hard to manage and monitor that progress. 

These problems can fester, bad documentation can lead to a research project spinning its wheels and not making progress which can lead to further problems such as running to the end of your funding cycle, etcetera. Bad documentation increases the risk of analyzing the wrong data, thus producing the wrong results, using the wrong analytic models. Again not wrong because this was fraudulent research just wrong because this was not what you thought it was as the PI. Wrong because you were just not organized enough to do the things you intended to do with the best of intentions. In the worst case this produces erroneous results that then you cannot reduplicate and again bogs down your process as a career scientist moving your investigative work forward. 

Also I will mention here and this is my Vice Chair of R&D had, this really has become critical for regulatory compliance. We regularly as I am sure all VA Centers are required to do conduct internal audits of our research process. It is essential to have good documentation that demonstrates clearly to auditors that only the data approved by IRB was obtained and used. That the number of the patients research cohorts is within the explicit limits of the research protocol. We can go on and on and on and I think a clean well-organized documentation even on the face of it presents a good picture to auditors that the research team knows what they are doing, they are in control of the research process and therefore, the likelihood of issues that would cause problems are low. If an auditor opens up files and finds a mess, or finds it impossible to discern system of organizing documents, I think that raises a huge number of red flags. Whether that auditor be external or from central office, that things are not well organized here and that there is a high risk of some kind of rules violation. 

As we all know auditors like to see clear evidence that analyses of the data conformed to the IRB approved protocol, that are you not conducting cancer research when your project is about cardiovascular disease and that only authorized project staff are touching the data. Again, this could be made abundantly clear with well-organized and well created documents. 

That is sort of the introductory part of the talk and then I will move to a schematic for document organization. This is my own mental map for the different flavors of documents that are involved in the research process. I think I am mostly an analyst of existing VA data that is why I am such a big friend of VIReC. I have done clinical trials and those things and there may be a slightly different schematic for such science. The basic ideas I think remain the same across the spectrum of scientific endeavors. I will give an example that is primarily and has to do because this is after all a VIReC seminar with analyzing datasets say from the corporate data warehouse. I think the ideas will translate throughout the scientific continuum. 

Here is how it used to be and for those of you I do not want to give an age range, but for those of you for say are like me took our first science classes in the Twentieth Century this is what we collected data in. For those of you who went to college more recently, this is a laboratory notebook and it was a single volume, usually had a nice brown bound cover and lined pages or graph paper. It was a single volume in which experimental ideas, research plans and results were recorded. Primary source document for scientific manuscripts, you would have your lab notebook and you would sit down at your IBM Selectric typewriter and you would write your manuscript. That is how it was done. it was non-sharable, it was irreplaceable, it was made of a highly flammable, soluble material that was easily destroyed or lost. And destruction or loss of the PIs Lab Notebook was a total disaster, sort of a career ending event. This is how we did it way back in the Twentieth Century. Fortunately we no longer have to depend on that kind of fraught with peril irreplaceable paper based data technique. In fact my encouragement to all my staff is that all pieces of paper become scanned and on PDF as soon as possible because it turns out paper is very fragile and easily lost. 

Here are some of the things you would need to document just some of the issues and then I will talk about the various ways in which these are organized. Your search strategy for the initial data poll and again I am giving an example that really is of the variety of data being pulled from say the corporate data warehouse. The data poll might in other data collected by your research coordinator for a survey or from a clinical trial. So we will say what is the strategy? Cleaning of key selection variables, exclusions to make your final cohort, cleaning and recoding analytic variables, merging datasets into the final analytic cohort. I mean these are again, and this is meant to be exhaustive these are the kinds of things we need careful documentation about. 

This is sort of a continuum across the board here where a research team would first figure out an initial data strategy, would move through the process from left to right and might meet on say a weekly or every other week basis to see how things were going. Then to be very specific about who was going to do what.  There are other milestones that need documentation, later on in the project somebody had to produce the summary statistics for the cohort that is going to go into the Table 1 usually in a scientific manuscript. Primary analytic models need to be specific. Deal with missing data, with imputation schemes. Once a model is fit a variety of tests or robustness and assumptions need to be tested statistically. After the initial analysis there may be sub-group and sensitivity analyses. Then some exploratory analyses when thinking about the future. Again a bunch of things that usually happen somewhat sequentially though oftentimes these things are happening in parallels. The sub-group analysis  is happening at the end of the project, but also exploratory analyses are happening at the same time. 

Here is my schema, you can use it, it is not patented and feel free to make your own boxes. Again I just present this to try to put into buckets the types of documents that come with study data. When I say documents I really  mean anything that sits on a computer with a file name associated with that. That includes things such as project datasets so I will put that under the term documents and I will tell you why in a moment. The datasets for the project, let us just imagine that in the top half of this slide, you have datasets and statistical code that are in SAS or SQL or what have you that are sitting on the VINCI system. Do not argue with me about whether it is VINCI or VINCI [pronounced vinchi] I thought it was Da VINCI so those documents are sitting on VINCI. Then you have other documents that are sitting on your local VA research server. This is how we do it at least in Philadelphia, the data and the code are sitting on VINCI, regulatory documentation, cohort selection and analytic plans, analytic results and then the products of research—posters, slides, manuscript files those are on our local server, our research server. These are accessible by the entire research team. 

So the overarching goals of the schema are simply to document all of those things I just mentioned in terms of the scientific process in real time and in particular to make a concerted effort to link the file types across those six buckets. Such that when changes occur in the selection algorithm, you know, and when I say you I mean you the PI, you the project manager, you the analyst know that a change in the cohort selection is related to this change in the statistical code which is related to this version of the dataset. I am going to give some examples of how that works. 

The great thing compared to the research lab notebook is that one can create older versions of a protocol or a dataset or SAS code and save those on one’s shared drive or SharePoint server and have access to those while one is working on the newest version of the protocol. This of course creates  a nice document trail, it could create chaos however, if you do not have a good sense of what is called version control. Meaning when the project team has moved to a new plan, everybody has to be very clear what document indicates the new plan, the new statistical code, the new version of the dataset. Problems creep in when there is trouble understanding what is the most recent version. Again, here the goal would be that an entirely new PI and team should be able to be dropped in, parachuted in to your research center and take over the project with a minimum of hassle and understand what was done and take over the work. I think that is kind of the ultimate goal. 

This does require a little bit of effort. I mean of course most study personnel are going to have shred access to the documents they need. Not everybody for example may need access to the regulatory documentation, not everybody should or will be allowed to have access to the project datasets. Nevertheless there are going to be documents that a lot of people are going to have access to. I am going to lay down some rules here and rules are meant to be broken again says my teenage daughter, but one rule might be that no study document should be stored outside the system. Certainly not on some kind of computer that could die, that is why we depend on our research server and their backup systems such that everything is recoverable. Also so that nobody has to wonder, who has the copy of the updated protocol it should be very clear that all copies of the protocols are put on the shared folder in the protocol selection folder. It should be very clear that nothing exists in the Outlook email of the PI or on the laptop of the project manager, everything is shared and it just is stored in one place. Also everybody needs to follow the same rules of the road for naming, updating, sharing and archiving documents. I will explain one suggestion for the rules of the road. 

Here is an important idea, again I talked about constantly what is going to happen in research is new versions of documents will be created thus creating an older version of the document at the same time, you ‘Save As’ this is the May 22nd version of the protocol. That means that it supersedes the April 29th version of the protocol which has just become old. You are constantly creating old documents when you are doing research and you need to get those out of the way. Old documents are your friend because they demonstrate where your thinking has progressed. They are your enemy if they create confusion about what is the most current version. All six folder types meaning datasets, code, regulatory documents, analytics, output and presentation should have an archive subfolder where old or draft versions of documents are instantly moved as soon as the newer version is created. Whose responsibility is this? It should be the responsibility of the person creating that new document. As soon as the ‘Save As’ new document created old version goes into the archive folder. We do not delete anything we save it in the archive folder and then we know that it is old. 

So now more specifics in regards to these practices. Project datasets, this is the raw data, your SAS datasets at least in my world where the actual analytics will take place. Raw datasets needs to be clearly labeled and stored in their own subfolders. You certainly do not want to have a folder which has all of your raw data and then your process data I think that creates problems. To the extent you can do not mingle the raw data that you did from the initial data pull and datasets that have been processed. and process may mean you derived new variables, you cleaned the data, you linked in additional data. Here is where the naming of the datasets needs to be carefully done. This is just a basic, I began life as a computer programmer at least began my adult life, I do not think I began life, but I began adult life as a computer programmer. One critical thing is to create meaningful names. My suggestion is that you create SAS code that has some kind of name like CohSel_04142014. I read that and I say oh without even opening that file that is the SAS code that does the cohort selection as we determined on April 14, 2014. Ideally that would produce a dataset that has essentially the same name i.e., Cohort_04142014. again the naming of the files has created an organizational schema such that I do not have to dig into those files. In fact I can quickly say okay that code produced that cohort and the date on that is in fact referent to the analytic plan that is being followed. Again I strongly suggest that dataset names and in fact the names of all files be rationally designed such that you can link statistical code to datasets, to analytic plan, to output.

Some words on regulatory documentation. I think all apply as well-meaning all of us I think are moved to the world where our regulatory binder, I still have a paper regulatory binder of course because maybe the computer will go down. But my primary regulatory binder is electronic and the electronic version of the regulatory binder has of course all of the IRB forms, R&D forms, correspondence approvals, dates, file names all on PDF. All of this should be easily organizeable with names that are associated with the rest of the scientific process. If you do a protocol modification that is referring an analytic plan those two documents should share similar file names. Same process of archiving should happen. Every year our R&D Committee we have people submit what was supposed to be their current version of the research protocol and we already have been in receipt of Version 6 and they send us Version 4 because they somehow lost the fact that they had already sent is us version 5 and 6. Again, archiving and moving the defunct or draft versions of regulatory documents is critical. 

Here is an example and I did not want to make this screenshot because there are a lot of regulatory documents and I did not want the screenshot too small but here would be where my IRB initial submission folder would look like. I would have the PDF version of my HIPAA approval, my cover letter addressing stipulations. Note that I put the dates almost always I want to put the date of the document in the file name. I do not want to trust the system and I know there is a system save, this was saved on a date. You can run into trouble with that and most particularly if you need to move these files or copy them or they become corrupted or something like that and you have to regenerate them, that date modified is then going to change. I do not want to ever lose the date on which these documents were created so I almost invariably put the date in the file name  it just makes life easier. Again, I will have subfolders within my IRB submission to make life easier, but the archives folder is critical and things get dumped into archives as soon as they are deemed old. 

So now to the meat of the science which is the cohort selection and analytic plan. All documents should have a date in their file name. Again as I just said, we are not relying on system save dates. We are going to use the date as version control. The first version may be Cohort Selection _04.14.2014 that is going to be a Word Document. Then when we meet again and we change things, my project manager is going to go into that Word Document and use track changes and write in the track changes on top of the April 14th document and it is going to be saved with those track changes baked in as the May 5th version. The date is going to be in the file name, track changes are going to indicate what changed. When we meet again, and we make further changes, all of the changes that occur between the April and the May 5th date are going to be accepted and new changes will be tracked. I do not advice and I know you can do this kind of embedding changes upon changes upon changes so that at nine versions of the cohort selection you have nine different colors in Microsoft Word track changes. I think that is a mess. What you really want to be able to do is go in and see what did we decide between April and May about including patients 18 to 25.  You would go in and you would say that is when we made the change that we were going to exclude patients younger than 25. 

This is how it would look again, I hope you all get the idea I want you to do the archive. Your archive document gets saved, it is moved into the archive folder in your project folder. The original version or excuse me the current version of for example the Revised Plan for Race and Socioeconomic Identification sits on the top of the project folder. If I want to see older versions of that I go into the archive, I have dates in the title of the plans so that I can look back and say that was the June version that was the May version. I can sort things by date so then I can say okay how did our Race and Socioeconomic status identification evolve since maybe we made five different decisions about that over the course of a year. 

I really advice against using some kind of versioning system in the title. It is very tempting to use Cohort_Selection_A and Cohort_Selection_B and then okay the next one is C. But the problem is that somewhere along the way someone will come in and mess you up. So somebody will do an A1, but then somebody else will do a B and so which one, did A1 come after B and there is A2. Then even better someone walks in and says well I fixed the problem in A1 and then you are stuck with looking at saved dates and trying to figure out what the chronological sequence. Do not do that, use a date in the name of your file which indicates when the changes were made.  I advise against the A, B, 1, 2, 3 schema. 

Okay, do these changes as soon as the meeting ends or at least within 24 hours. I mean I know sometimes your meeting at the end of the day, but your analytic team may make some decisions, somebody is scribing those down and it needs to go into an updated version of the cohort selection document. Those should be documented within 24 hours and saved in the shared file and the research team should be then alerted hey everyone take a look at the revised version of our Race Socioeconomic Status Assessment, did this capture what we talked about yesterday. So all participants in the decision are reviewing for clarity and for accuracy. 

Again, you do not have write at home but in these documents this is going to help write the manuscript someday. Think of your poor future self who has to sit down with documents that are two or three years old and decipher them and come up with your method section. Be nice to that version of your future self by making it very clear in your data selection algorithms, in your analytic plans what it is you did. 

Some words on statistical code and again my first professional job was as a coder and there are many things I learned way back then that still apply today. Again, I said please use names for your SAS code or whatever kind of code you write that have a date in it and are linked to the analytic plans. In this case for example the analytic plan is the Microsoft Word document Cohort Selection_04142014. That is linked by the name of the file to the SAS code that executes the cohort selection. That would be CohSel_04142014 those of you who do SAS know that SAS produces .lst and .log files and those will obviously carry the same name. And so quickly I can look at this and say okay all of the stuff that we talked about at the April 14th meeting in terms of cohort selection, that resulted in this SAS file and this output in these datasets, etcetera. It is all linkable by file name. 

Basic computer programming 101 this is not just true for research it is the computer code needs to be well commented. Changes in the code need to be dated. All codes should have a header with the projects author and linking information to the analytic plan. Code should be liberally interspersed with comments. Again they taught me this literally in computer science 101 way back in 1991. Here is how I would sort of recommend doing it. Here is my title and here is where I need a date in here so I already violated my rule. A date should be put in here it is the macro that pulls out resonant to bed ratio follows the analytic plan on February 15, 2008, you can see that this is before I learned valuable lessons. This title should actually be Hcris restobed _02152008 then I have a bunch of these green comments that describe to some future version of myself or the programmer that takes over my position what it is this code does. 

Let us talk about a few risks. and this is sort of touchy stuff when you hand the world’s best analyst things to do and they run SAS code and they hand you results and there is this moment where you either have to just completely trust your programmer that they have never made an error in the life and have completely understood you. You feel like if you question that they will be hurt or they will be angry or they will go take a job at Microsoft or something like that. I want to dispel that culture. I think we have learned in healthcare that even cardiac surgeons make mistakes from time to time and we need to create processes that expect that and search carefully for the occurrence of errors that will not generate syntax errors when you run SAS. SAS does not know if you miscoded a variable or did an inadvertent keyboard drop or misspecified a model because you forgot a key predictor just  by accident because there are 29 predictors and one was inadvertently left out. Therefore, it is critical in the production process to follow the basic QI that we do in healthcare to ensure the quality of complex processes. The idea of assigning a statistical analytic plan to a programmer and having the programmer execute the plan is a complex task. It is fraught with peril and it is almost inevitably likely to create errors from time to time. Therefore, what we need is a walk through where the programmer prints out all the statistical code and output, the SAS file, the list file, the log file and walks through the code with say another programmer or the PI or a second set of eyes. This should be entirely routine in just the same way that before an aircraft takes off you run through a checklist of things probably some of which and I do not fly aircraft for a living, but some of which are probably pretty mundane like are the brakes off. We do these things by routine because this is how we have designed the system to collect and to identify and correct errors. This we do with even the most experienced and meticulous programmers in the very same way they count spongers after cardiac surgery being performed by the best surgeon at the VA because errors do creep in. 

I am going to leave time for questions, we are coming towards the end. Analytic results it should be very clear what code file produced output so again we want good linkages. If you modify your output so this is the same example I have been giving SAS produces a .lst for output. If you then went in with an editor and annotated the output you should add some other information to the title of the document that indicates it was produced on April 14th and annotated on the 19th, etcetera. Then of course you want to present your research to seek fame and fortune. Ideally the same philosophy will guide in the creation of the tables and figures in datasets, tables and figures and output I will say slide presentations, you want those names to be linked to the specific statistical output that produced them. Again, I would suggest doing this via filenames. A second choice would be to somehow embed this information in the file, that is a little harder to do in a Microsoft Word document but not so hard in PowerPoint. Again, the idea would be in this slide on the right, is the table that might appear on a slide or in a manuscript and you wanted to be very clear that this is and here is the name of the table, 2-14-2014-TABLE 1-FROM 1-19-2014_CohSel. Then you can go back and see exactly what SAS output produced the table. 

This is pretty key, right because we want our presentation to reflect the most recent most accurate version of our analytic work. 

That kind of completes my description of a way of organizing your research documents. Now we have to use the system wisely. I would strongly urge against using email attachments to send such documents around. That is again a recipe for disaster. If somebody is editing a version of the analytic protocol they should simply save it as a new version on the shared drive and move the old version to archives and then refer to it in an email, but do not attach it to the email because that creates proliferation of documents and creates version confusion. Do not do that. Refer to the documents in the email by their possibly abbreviated file paths so everybody knows what you are talking about. Be specific about document names, I mean this is a little bit one could say this is silly just tell your programmer to use the most current algorithm. Well, if you change the algorithm five times I think it is better to say use the April 14th algorithm or team please examine the updated data selection algorithm in this document. This kind of careful communication save times. Why organize? It will make writing manuscripts easier, workflow will be clearer, the research team will need less direction from the PI because it should be much more obvious to the team what next steps need to be taken. Audits will not be terrifying but at least less terrifying. Staff transitions will be smoother and ideally your project work may be recyclable rather than an indecipherable mess. 

Here is my last slide, high quality study documents are critical to good science, project management and regulatory compliance. These practices I do not think they are terribly time consuming compared to the waste of time in terms of duplicating and finding documents and trying to figure stuff out. I think this actually saves time. Productivity again from these practices will typically be greater than any time costs. 

I will stop there and I will be happy to take questions from the cyber room and thank you I see still have a hundred and three participants, I am always worried that number drops to zero at the end so I am happy to take your questions. 

Moderator:Thank you Dr. Groeneveld. Well believe it or not maybe other systems are down because we do not have any questions in the Q&A portion just yet [laughter]. Hopefully folks either are totally understood everything you said which I thought was very understandable, that is my side comment or they are just thinking about a lot of questions that they are trying to formulate at this point in time. We will give them another moment or two perhaps to do that. While they are doing that I am just going to advance the slide a little bit, here is your contact information if people want to write to you directly if that is okay.

Dr. Pete Groeneveld: Yes that is fine.

Moderator:Okay and also next week we have a slide here for our last session on “Reduce, Reuse, Recycle: Planning for Data Sharing” and the objective for that particular session. We are getting some questions coming in. 

Dr. Pete Groeneveld: I see them. 

Moderator:Okay I am going to start reading them to you. First off how do we busy coworkers to go along?

Dr. Pete Groeneveld:Well in my world the PI sets the tone and creates the rules. I explain to my team the rationale and I explain to my team the expectation and I try to call people out on when people are not following the plan. My experience has been that people get it, this is how we do things at least on Groeneveld’s team. People are all kinds of organized some are naturally very organized and some naturally are not. It is a point for me for somebody who does not want to follow this I would say they may not be welcome on my team in the future because I do not need somebody who does a bunch of practices that will get me audited or will waste my time as a PI. I think if you are not a PI you might have to manage up a little bit and suggest things that could be helpful, but if you are the PI, I think this is within the purview of team expectations. 

Moderator:Thank you. Next question – who is responsible for setting this process up? PI, project manager, project coordinator?

Dr. Pete Groeneveld:So I could be facetious and just say yes. Everybody is responsible in the same way. Who is responsible for patient safety? Is it the doctor, the nurse, you know what I mean but that is too facetious. I will say typically I have my project manager set up the initial files. Locally I have my programmer analyst set up the file schema on VINCI. I have some strong input in how those are organized. But then all these practices naming of files and things we are always anybody from a research coordinator to the PI can get into a file on the project and has the power to change it, they need to follow the plan. Who sets it up? That might be the project managers role and the program analyst role on a data server side like VINCI but after that it is everybody’s job. 

Moderator:Thank you. The next question is – what documentation of VA data sources such as the CDW do you most wish for?

Dr. Pete Groeneveld:What documentation do I most wish for? Anything that VIReC has not described in excellent detail. I think we are all still learning about VINCI and I think the HSR data list reveals things all the time that I think would be great if we had that kind of description. I am not sure if I am answering the questions that was asked clearly. I think VIReC’s job security is strong because there are always new data sets VA is creating and new mechanisms that require a handbook or a descriptor. I learned how to use VA data using VIReC documents and so anything that is undocumented by VIReC is what I wish for. 

Moderator:Thank you. Next question  - what challenges have you faced implementing your file naming contentions across your team?

Dr. Pete Groeneveld:The grant is due in 48 hours and we have to do a bunch of analyses and we have to get this out and come on, come on, hurry, hurry, hurry. Under the pressure of the clock everybody is naming datasets left and right and the SAS code has been modified five times it is 9:00 PM and we are still here and we have to get the grant in. Obviously there are times in an investigators team life where you are just under the gun and just need to push ahead. Ideally though when the dust settles the grant is in, okay we have to clean up here. In the same way that everybody’s desk gets, well maybe not everybody, I know there are those of you out here who always have clean desks. At some point you say oh my goodness I just have to clean up. Ideally that does not happen every six months, okay the grant is done, let us clean up the files I mean it needs to be that kind of organization time built in. But undoubtedly we all face deadlines where we need to hustle and sometimes people are emailing me with attachments and I do not like it but I say okay okay we just need to do it then we are going to go back and clean this up on Monday once the grant is in. 

Moderator:Thank you. Next question – how do you decide where to put the date? In front or in back?

Dr. Pete Groeneveld:That is a great question. I have taken to putting it in the front because then you can sort it. If you use a date convention where you do 12-01-2014_ well then you can easily sort those files if you are using a Windows based system by their name and that is often useful. Now, there is a counter argument which is you may want to sort by the type of file it is so then you put the date further back in if there were different sub-types of files. I do not claim this is the only way to do it but again I like the date in the file. 

Moderator:Okay, thank you. First we have a comment – thank you for a wonderful presentation. And the question – what is your recommendation for frequency of meetings which produces updates from updates on the analysis plans [Indiscernible]. 

Dr. Pete Groeneveld:It depends on the project. I mean if things are rapidly evolving and new data are coming in and your programmer is getting things done left and right we might have to meet twice a week. I would say more typically in our group it is once every two to three weeks because we could lay out a set of tasks and those get accomplished in say half a months’ time. Maybe if the project is kind of on a low burn and they were waiting for data we do not meet very frequently. I think it is very project dependent. 

Moderator:Thank you. The next question refers to data walk trough’s. We have just one programmer on our team so negotiating using the time of a programmer from another team is tricky. What do you do?

Dr. Pete Groeneveld:I speak SAS still and I typically am the one who looks through the code of programmer and I also like again  I used to do some of this I like sort of saying okay that is how we did that and that is the regression model and things like that. If you really only have one person who can speak SAS and you are depending on that person to always get your SAS coding right and they have nobody checking their work, that is worrisome. I mean I just say I do not know about you, I make computer programming errors all the time and I bet other people do too. Just from a safety perspective and when I mean safety I mean let us not mess up here and not know it. It really behooves you to get a second set of eyes somehow. I understand you cannot borrow somebody’s programmer for two days, but it is the kind of thing which a research shop should say look we all need to do this for each other ideally. 

Moderator:Thank you. We are at just a minute after but do you have time for one more question?

Dr. Pete Groeneveld:Sure why not. 

Moderator:Okay. So what is your favorite scheme or interview question for hiring a project manager analyst that is likely to have incredible organizational skills. 

Dr. Pete Groeneveld:[laughter] I could say what kind of tree would you want to be if you were any kind of tree. Organizational skills are a little bit inherent and a little bit learned. I ask people oftentimes for a programmer how do you organize files. What do you have a scheme to organize your work files. If they say oh well I divide it up by project or they say I put it on my hard drive. So there are ways that people reveal their organizational habits. I think some of this can be trained meaning if people have not been introduced to a scheme like this but they are reasonably well organized people I think they will gravitate to it. Again you can learn a lot by just meeting somebody about how organized do you think they are in terms of all kinds of stuff. You are right that is a critical question, it is always good to have organized people doing science. 

Moderator:Okay, great. Well that is it for today’s questions. Additional questions or information regarding this topic please feel free to email the VIReC Help Desk at VIReC@VA.gov. Thank you to Dr. Groeneveld for taking the time to develop and present today. Please join us next Thursday at 1:00 PM eastern for next week’s fourth and final session in this series entitled “Reduce, Reuse, Recycle: Planning for Data Sharing”. Thank you everyone for attending and have a great day. 

Page **1** of **14**