**Transcript of Cyberseminar**

**VIReC Good Data Practices**

**“The Living Protocol” – Managing Documentation While Managing Data**

**Presented by: Matthew Maciejewski, PhD**

**May 15, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact virec@va.gov.*

*Linda Kok:*Hello.  Good afternoon or good morning.  Welcome to VIReC Cyberseminar Series, Good Data Practices.  The purpose of this series is to provide researchers with a discussion of good data practices throughout the research life cycle, with real, live examples from VA researchers.

Before we begin, I want to take a moment to acknowledge those who have contributed.  Several of the research examples included in the series were generously provided by Laurel Copeland, of San Antonio VA, Brian Sauer at Salt Lake City, Kevin Stroupe, here at Hines and Linda Williams at the Indianapolis VA.

I’d also like to acknowledge the work of our team at VIReC.  VIReC’s director, Denise Hynes, our project coordinator, Arika Owens, and Maria Souden, VIReC’s communication director.  Of course, none of this could happen without the great support provided by the cyberseminar team at CIDER. 

The research life cycle begins when a researcher sees a need to learn more, formulates a question and develops a plan to answer it.  The proposal leads to the protocol and IRB submission.  When funded and approved, data collection begins, then data management and analysis.  It may end when the study is closed and the data are stored for the scheduled retention period, or perhaps the data generated will be shared for reuse and the cycle will begin again for the new data.

In the four sessions that make up this year’s Good Data Practices Cyberseminar Series, we will follow the steps of the research life cycle.  Last week, Dr. Jennifer Garvin presented the Best Laid Plans, Plan Well, Plan Early, and looked at the importance of planning for data in the early phases of research.  In case you missed it, as Heidi mentioned, it is available on the HSR&D website cyberseminar link.

Today, Matt Maciejewski will describe ways to manage data management, called The Living Protocol: Managing Documentation While Managing Data.  Next Thursday, May 22nd, Dr. Peter Groeneveld will present Controlled Chaos: Tracking Decisions During an Evolving Analysis.  Finally, on May 29th, I will present Reduce, Reuse, Recycle: Planning For Data Sharing.

Before we jump into session two, we’d like to know more about today’s participants.  For this question, we’d like to know about your role and also your experience.  Our question is, what is your role in research and level of experience?  

In the polling panel to the right, look for the combination that best describes you.  Are you a new or experienced research investigator?  A new or experienced data manager or analyst or a new or experienced project coordinator?  If your role is not listed, please describe what your role and experience are in the Q&A panel at right.  We’ll give you a moment.  Heidi?

*Moderator:*It looks like we’re still seeing a couple come in here.  We’ll give it a moment to stabilize and then we’ll go through the results.  It looks like we’re fairly stable here.  Linda, if you’d like to read through the results here?

*Linda Kok:*I’ve maximized my screen and I’m not quite sure how to get back.

*Moderator:*That’s okay.  I can read through them here.

*Linda Kok:**[Laughter]* Thank you.

*Moderator:*We’re seeing about 6 and a half percent new research investigators, about 11 percent experienced research investigators, 12 percent new data manager or analysts, 34 percent experienced data manager or analysts, 15 percent new project coordinator, 15 percent experienced project coordinator and about 6 and a half percent giving other.  From the other, we’re seeing data access, IRB coordinator and database manager, administrator, HSR&D Center staff, experienced statistician, quality assurance and Maverick Boston project manager database.

*Linda Kok:*Wow.  We have quite a collection of people and it looks like the experienced coordinators, is that correct, was the highest number?

*Moderator:*Yes, I believe so.

*Linda Kok:*It was 34 percent.  Well, thank you all.  Having these statistics really helps us in planning our series.  We also know as we’re talking today who we will be talking to.  Thank you very much for completing that.

Today’s session, the living protocol or managing documentation while managing data, will provide key insights in how we can capture the reasoning decisions and actions that occur during the data management phase of our research.  

Our presenter today is Matt Maciejewski.  Dr. Maciejewski is a research career scientist and director of the health and economics and policy units in the Center for Health Services Research and Primary Care at the Durham VA Medical Center.  He’s also a professor in a division of general internal medicine of the department of medicine there in Durham.  

Dr. Maciejewski has done VA research on bariatric surgery, community-based outpatient clinics, outpatient care choices by Medicare-eligible veterans and VA copayment policies.  He’s a member of the newly formed panel on statistics and analytics on VHA  data sets.  Welcome, Matt.

*Dr. Maciejewski:*Thank you, very much.  I appreciate the opportunity to present this to the group and I appreciate everyone’s interest in this topic.  To ensure that there’s plenty of time to address any questions people may have, I’ll try to go through the material in the next 30 to 35 minutes.  

Okay, with that, we’ll start.  I just want to, at the outset here, as is typical, since it is the VA there will be a number of acronyms running through the presentation.  They are listed here, which maybe some are familiar and some are not.  BOSS is the bariatric study I’ll be talking about.  COMM is another study I’ll be talking about towards the end.  I just want to put that out there.

The outline is really sort of an observation and objective that sort of sets up this whole presentation and move to what the sort of living protocol really is and then illustrate it through specific parts of a living protocol, I guess we’d call it, around a specific ongoing secondary database study that I’m doing around bariatric surgery.  Then transition to talking about how we document and talk about linkage of primary and secondary data in the COMM study that I mentioned previously here and then end with a conclusion.

To start out, one thing that has struck me and may be true for a lot of you as well is that I had no training in graduate school on how to do the logistics of a trial.  We learn about methods and analysis and data and all sorts of policy issues, but how to actually conduct a study, what’s the day-to-day work, there was really no discussion of.  

There’s no discussion of a host of other topics during graduate school, but this was a big one.  There’s no literature on how best to work with a team when you’re operationalizing a protocol, how to figure out the order of tasks, and lastly, how to document data and decisions made over the course of the study.  

We write grants that are, in some sense, very general overviews of the research projects with some specificity in some areas, more than others.  Even those areas that are more specific about what types of methods you’re going to use, what types of covariates you’re going to use and outcomes.  There may be multiple decisions that actually underly what seem like very specific elements of a grant proposal and documenting those is really critical.

The objective of today’s talk is to share some examples of how I’ve started conducting these tasks in a timely manner, in terms of the documentation and really this living protocol idea really came about through trial and error, through a series of VA studies and non-VA studies, as well.  

Over the course of my career I really basically have been begging, borrowing and stealing best practices from other investigators on how to conduct studies, how to document studies.  This living protocol thing is relatively new for me and it’s proven to be a really helpful mechanism for bringing formality and explicitness to lots of tasks that we all are working through in our various projects.

The bottom line of this really is this documentation, this living protocol, is really important in two ways.  The immediate way is that it provides easy recall for when you’re trying to do manuscript writing and when you’re trying to think, a year ago we made some decisions about some certain things that were very particular in the weeds kinds of questions.  

I don’t know about you all, but my memory is getting worse over time and so I find this document also helps ensure that we have a way of actually recording, having some historical record of why we made the decisions we made and what were those decisions and what were the choices that we chose from.  

Then more importantly, going forward outside of the immediate project and manuscripts is that the documentation may inform future work.  If you’re working in a research stream and the current project you’re working on will inform some future project and you’ll have to replicate, in a sense, some of the decisions and issues that you faced in the current project and some future work, you have some detail about how you did that and why you did that and it may serve you and your colleagues in the future so you don’t have to reinvent the wheel.

The recall piece is right—ultimately, we all have to write papers presenting our results, in terms of the methods pieces, which is really where this living protocol is most helpful.  There’s design decisions, there’s measurement decisions and there’s analytic decisions that we all have to make to turn the grant proposal into some final product.  Those are decisions we go through in project meetings over the course of the study.  

Documentation is the only source for the logic of your choices.  There’s going to be SAS code or RRS data code that the implementation of that, of some final choice, but that final choice that’s executed in code is derived from some process of considering alternatives, potentially, alternative approaches in settling on one.  

If you don’t ever document that, except maybe through minutes, which are going to be potentially more parsimonious than really reflects the key issues you went through, this sort of documentation in the living protocol can be a good way to have that historical record in some detail.  

If you’re documenting as you go along, even though it is some significant amount of work along the way, you end up with a clear historical record.  The alternative, which is frankly, what I was doing for most of my career, was the hard way, which is probably subject to recall by myself and collaborators, which is to go back to the minutes, go through the SAS code, talk to the programmer and any notes that we might have made in various files in case we had to go back and make sense of why did we decide on this inclusion and exclusion criteria or why did we choose to define race this way instead of some other way?  It has it all in one place, hopefully that you’ve accumulated over time.

If you document as you go along, as opposed to doing this recall bias subject process the hard way, you really just have—you pay now or you pay later, in a sense.  It’s important to know, if you haven’t done all the work and for some reason all your staff turns over because people move on to other jobs or something and aren’t there to talk with, how are you going to know exactly how things were implemented?  That can be a problem that everybody runs into at some point in their career and I certainly have.  That’s not a fun place to be.

Then if you’ve got all this detail, as I’d said before, there may be decisions you’ve made or ways of doing things, defining cohorts, measuring outcomes and covariates, that provide a clear road map.  I’ll give a couple examples of that, because I don’t know about you all, but we’re certainly seeming to get busier in our center and we’ve grown as a center in the past five to ten years.  

Finding ways to become more efficient, standardizing things that can be standardized in our center makes everyone’s lives easier.  We don’t have to keep reinventing the wheel.  This may be one way to help in that process of becoming more efficient within a research stream and maybe in the center, in general.

Before going further, here’s the poll question.  How do you all document your projects through the course of these major decisions?  Please, answer now if you would.  Great.  Well, it looks like we’re getting pretty consistent answers now here.  Thank you all for responding.

*Moderator:*Matt, I’m just going to read through the responses here so we have it for the transcript.

*Dr. Maciejewski:*Okay.

*Moderator:*Right now we’re seeing about 77 percent saying that we don’t, about 45 percent saying we use minutes to document major decisions, 11 percent saying we create email summaries that are retained, 20 percent saying we amend protocol for use by team to reflect major decisions and around 16 percent saying other.  The other includes wiki, Excel spreadsheets, hand notes, project log and Excel, different for different projects, create a separate method document for easier recall later.  We document them in data definitions and One Note.  Definitely a lot of different things out there.

*Dr. Maciejewski:*Great.  Thank you.  Well, for those of you who say you were amending your protocol as you reflect major decisions, what I’m going to talk about today is definitely some variant of that.  I’ll be curious to hear from those of you who take this approach and whether there’s things that maybe you’re doing that maybe I’m not doing that we could all learn from.

Then for those of you who reported using minutes, I did the same through all my projects until this most recent one and switched to this most recent—I think minutes are helpful and we continue to do minutes, despite also having this living protocol approach.  The living protocol really is potentially a lot more detailed than minutes might be, depending.  If your minutes are really detailed, then that may be sufficient.

Okay, so the genesis of this, how did this really come about?  Basically after years of, I guess in the recent past, getting a little bit overcommitted and trying to keep all the balls juggling in the air, I had gotten away from what I thought were good project management practices that I had started earlier and I tried to practice early in my career and realized I needed to get back to that.  In reflecting on it, had this, I guess, idea of trying to start something like this.  

Also, I guess, working with colleagues who do randomized trials and seeing that they have this protocol that they have to continually update around the scripts with the interventionists and invitation to patients or providers and all these things that have to be continually updated made me think, well maybe there’s a way to apply some of that trial-based documentation to big claims-based studies.

Really, I should say that really I’m primarily doing large secondary database studies and so the living protocol is sort of built around those kinds of studies.  By having this protocol, as others have noted, it improves the breadth and depth of documentation from the original grant submission.

It serves multiple purposes, really.  You know, it enables me to go back to what we were thinking the alternatives were for any given choice and how did we come to the specific choice that we made in design measurement and analysis?  I, presumably, in this bariatric work am going to be continuing to do hopefully, if I can get funding, additional studies building on what we’re doing now.  There’s real value in the continuity in a lot of the decisions that we have.  

Having record of that will enable us to kind of revisit that in two years or a year when we’re going to be writing new grants to see whether we agree or there’s alternative approaches.  We have made some changes from the prior bariatric study and the reasons for documenting that is really helpful because at the time we’re making any decision, we’re as deep in the weeds as we’re going to get on it.  

Once we make a decision we move on, because there’s so many other decisions to be made.  Trying to make sure that we record all that detail avoids the recall bias.  In making colleagues aware of this, they, themselves, may find this sort of approach useful for themselves and they can probably improve upon it even more and then maybe I can benefit from their improvements on it so that it’s serving the larger research enterprise of the center.

How I work generally, and particularly on the bariatric surgery team, is sort of I try to work fairly collaboratively with interdisciplinary research teams.  This bariatric study I think exemplifies that.  We have two general internists and two bariatric surgeons on the study team and then three methods folks, two biostatisticians, myself, and then really two data analysts, actually, and a project coordinator.  We really welcome feedback from everyone so that the protocol really reflects the entire team input, not just whatever revision I may have or the biostatistician may have.

In planning for this in how we sort of—the living protocol and decisions get made and the living protocol gets revised, I have basically calls every other week, unless we have special calls that are needed to work out specific issues that are particularly thorny.  Then in the call we update on where things stand with data and programming.  

We try to review a recent article periodically.  It’s not been a regular thing now that we’re in analysis, but early on we did to try to stay up on the literature and in particular, find studies that we think we’ll probably have to refer to and give context in relation to that in our work and then get into the weeds on the current issue that we need to make some decision on.  Then that gets baked into the living protocol and then outline where we’re going so everybody knows the big picture of next steps.

Then the living protocol evolves through these bimonthly calls, which is all outlined here, again.  In between these calls, we have the minutes to keep track of the major tasks and who’s responsible for them.  

Then for quite a while I was spending time trying to update the protocol and send it around.  In the early phases of the study, that was happening a lot more.  Now that we’re in analysis, it’s the activity around the updating the protocols been reduced. This master protocol, with its repeated updating, reflects decisions that have just been settled.  

Early on we would share the living protocol with the study team and everybody could edit it in case I wasn’t characterizing things properly and missing things that were important to include.  That feedback from the entire study team was particularly helpful in a couple instances when I wasn’t—there were new statistical methods we were using that I didn’t totally have characterized properly.

A big challenge for me, as I’ll note again at the end, is that making sure all these prior sections are complete, there is some iteration on it.  In the heart of things, in the heart of this project, there’s a lot of moving pieces and sometimes I’ve not always been good about updating prior sections.  As we revised—we have an initial cohort and we’ve changed it in some ways and I need to go back and update that cohort to make sure it’s really reflecting the current and final analytic cohort, not just the initial one.  Then there’s programmer code that goes along with that that’s implementing the decisions from the living protocol.

The major sections of this living protocol are really the design of the study, measurement and analysis.  I started, and this was sort of where I found this document to be so important, was this was the secondary study, a quasi-experiment.  

I hadn’t done this before, but Don Rubin has this advice of thinking about observational studies as broken trials.  I started out the documentation saying, well, we know what our aims are.  We know what our basic study design is.  Well, what would our ideal randomized trial be? If we can articulate that and everyone agrees on that, then we can articulate how our—what we actually have differs from that idealized trial.

This slide here represents really what the study design really is and the nature of the cohort and then the fact that the non-surgical control group are not similar, due to lack of randomization and how we need to handle that non-equivalence in brief, here.  What’s the sequential stratification method, which is sort of analogous to propensity score methods?

The whole protocol, which is now 40-something pages, starts with, well, what is the ideal trial and then what are we really stuck with in our observational—our quasi-experimental study?  Then as a result of the deviation from this ideal trial where we can instantaneously enroll patients for bariatric surgery, randomize them and then immediately upon allocation to treatment or control, take them if they’re in treatment straight to the OR.  

There’s no way we could do that in real life, but we’re just saying in an ideal world with infinite resources and infinite support, what would we do to minimize all the biases?  Since we can’t do that, that has implications for the internal validity threats that make it challenging, to make sure we estimate a treatment effect that’s unconfounded.  

We then outline the cohort derivation criteria, run inclusion and exclusion, which I’ll get into, and then steps for identifying and refining the treatment group.  The first thing, inclusion and exclusion criteria, applies to the treatment and control.  Then we have specific issues about identifying treatment.  Then we develop the consort figure and justification for the exclusions, for both treatment of control and then specific exclusion for treatment and specific exclusions for the control group.

Then we have a major section around the measurement issues.  What are the VA data sets we’re going to use?  What are the major variables that are contained in each?  What are the purposes for them?  These data sets have within them outcomes and how are we going to define it specifically?   Data cleaning decisions, frequency distribution, some general things for the outcomes that are helping us understand if how we’re coding things is correct.  

Then covariates, how exactly are we going to define hypertension, diabetes, race, anything you might imagine?  It’s this part that could particularly be generalized.  We’re hoping to, in Durham, come together as investigators and come up with commonly agreed upon definitions for identifying hypertension, say, by diagnosis only, by medications only or both, across all projects so that we don’t have to keep reinventing the wheel.  It’s just so much time spent doing that project to project.

Then specifying the purpose of covariates, is it inclusion, exclusion criteria?  Is it matching?  Is it adjustment?  Is it outcome?  Could be any of those, potentially.  Then we have a whole section on analytic issues, that is, how are we going to handle missing data and descriptive statistics?  What internal validity threats do we think we have to address through the analytic methods and then the detailed methods for doing each aim?  

Ideally, all of these sections, all this should be outlined before we actually begin analysis.  You know, hiccups may arise and there may be need to change from the original plan.  Imagine, say, five or ten years from now, we all are—suppose that any federally-funded project is required to post sort of a design paper the way trial colleagues publish design papers, outlining their randomized trials and the interventions before they actually do the analysis and see the outcomes.  

Suppose we all had to do that.  That would be a lot of work to figure that out and make sure it was right before you actually get into your analysis.  That would require a whole host of planning and anticipation of what could go wrong.  

In fact, the evaluators of the Oregon Health Insurance Experiment at MIT and Harvard have done exactly that and posted their analysis plan on their website.  They did it before doing any analysis so that it was very transparent that they were not just reporting things that were significant and downplaying other things.  I highly recommend that you all go check out that website, because it’s a pretty remarkable form of transparency on research plans.

Then there’s parallel documentation of the SAS programs in which there’s hopefully really clear documentation in case we have a change in programmers.  That’s where the rubber hits the road.  Each program in our group tends to have a clear statement of purpose and then somewhere relating to what aims of the grant are being addressed or is it setting up a cohort and all that kind of stuff.

Let’s get into a specific example.  Time is flying here pretty quickly.  What I’m going to present is the actual living protocol for this bariatric surgery study listed here with selected parts of it.  Here’s the aims of the study, for those of you who may not know it.  We’re basically looking at veterans who do or do not have bariatric surgery.  Are there differences in weight change and resolution of these conditions, difference in survival, post-surgical complications, which are really only related to surgical patients?  Then are there differences in long-term VA expenditures and utilization between surgical and non-surgical patients?

Okay, so at the end goal, right, the data set that we want to end up with from all the myriad VA data sets we’re polling from are two things: a matching data set on which we’ll take our sampling frame and do matching between surgical and non-surgical patients.  That requires identifying covariates for inclusion and exclusion criteria and then covariates that are actually going to be used to match patients.

Then we need an outcomes data set that’s derived from the matching data set that includes just those people who are going to actually evaluate our outcomes.  We need covariates on which we’re going to be doing adjustments, if further adjustment’s needed.

Data documentation, there’s really two levels of documentation, in a way, around data sets.  The first is at the study level, what specific data sets do we need?  What years and what is the purpose, in general?  Then for each data set, what variables do we need out of them?  What are we going to derive, if anything, if we’re not going to just use what’s there?

Here’s a table verbatim from this bariatric living protocol, which outlines all the different data sets you can see, what aims they’re applied for, which outcomes are going to be drawn from these different data sets so we ensure that we’re not missing something, in particular, which covariates we’re going to be drawing from the different data sets and then what years we have, on the surgical patients and on the non-surgical controls.

You can see in the version of this from two years ago was in process, in process for a whole bunch of stuff.  We have pulled essentially everything now.  Everything’s 2000 to 2011, because that’s what we needed.  This is a high level way of considering all the data sets that we’re considering.  We shared the sense of study team and everyone looked at it and said, “Yeah, that looks about right,” and so it also gives some sense to the programmers what we’re going to be using them for.  It’s something we refer back to periodically.

Then in the document I also, which was actually a figure drawn from the prior study, we articulate visually what the study design is.  That is, we have cohorts of patients who have surgery, which are the small dots above in each of the years 2000, 2011.  

Then potentially matched to them are cohorts of non-surgical controls and they’re contemporaneous since we draw from veterans who are eligible for surgery in each of the years in which we observed cases.  That was different from the prior study in which we only had historical control groups, so only the 2000 big circle of the controls was there.  It’s a simple way of kind of showing the study designs, as it were.  

Pre-post non-equivalent control group cohort study with a non-equivalent control group.  Then we go into, in the living protocol, cohort identification, in terms of inclusion and exclusion.  The way in which we derive these inclusions and exclusions are probably what you all do.  You look in prior trials and observational studies, consider our own prior works and so we’ve done work in this area and look at ongoing trials, since criteria may change in terms of refinement of inclusion, exclusion criteria and trials, in particular, over time.

Then as a group we look at the whole list of criteria and really refine them down and then develop coding rules that are entered into the living protocol.  Once we’ve got that characterized for the overall cohort of cases and controls, surgical patients and non-surgical control, then we go into, well, what do we need for identifying the treatment group?  This is typically the easiest thing to do because we know the treatment we’re interested in.  That’s sort of probably motivation for the study in the first place.

We have to go find these people and we have to figure out how we’re going to identify them.  In the case of bariatric surgery, the options are CPT procedure codes, there’s ICD-9 procedure codes and maybe another project’s medications may be a source of identifying people taking a certain treatment.

Then in particular in this project, which may also apply to medication studies, is the relevant treatments change over time in what’s available and how they’re coded.  That was a big issue in this study because the technology for bariatric surgery has really changed a lot in the past 12 years, 15 years.

We spent a lot of time making sense of what have prior studies done.  We talked to the medical associations to see if they had guidance from their coders and we were able to have some really fruitful conversations with some of those folks, which I guess were external people from the VA and hopefully a gold standard, in some sense.

Through that we were able to characterize all the different bariatric surgical procedures and what we thought were both the ICD-9 and the CPT procedure coding.  Having this down is really important because if this is wrong, our treatment group’s going to be wrong.  We may be having errors of omission or commission and that could be really problematic.  

Something like this could go into an appendix of a paper to help make it transparent, how we identified our treatment group.  Having this sorted out early was really essential.  This is certainly in the living protocol.  We were able to then, from that, figure out how many surgical procedures are done in the VA in these different years, just for your interest.

Then as soon as possible, once we’ve got the treatment group sorted out we do a consort figure.  We all identify our inclusion and exclusion criteria and then populating it is really good to do because if you wait until the end of the study you have to have a program or go back in and make sense of where these exclusions happen, what were the reasons for them, and that can be really painful. I made that mistake early in my career and I just don’t recommend doing that.  It’s just easier to do it as soon as you have the cohort sorted out.

Then we sort of work out—in one of the key issues in this study and maybe lots of others is we’re trying to understand time until some event.  When does time start?  What’s the index date relevant?  

There’s a number of different index dates that can be important to consider.  It’s certainly identifying the date of treatment, in the case of the surgical patients who are not in the controls.  Separates the pre-period and the post-period and then we needed to find pre-surgical covariates for the treatment group and the control group.

Unlike a trial, where you know you have measurements on people at baseline and you can characterize baseline and you can have people come in at routine times, following enrollments, we don’t have that here so we have to figure out how are we going to characterize pre-period measurements for baseline assessments.  Defining the index date’s really critical.

Then we document, in the case of BMI we need a baseline BMI on everybody because we want to know are outcomes different by surgical patients with higher or lower BMI?  We thought we would have a baseline BMI on the day of surgery for everyone and we had them for a lot, but not for everyone.  We had to use CDW data to go find weight data and height data constructed.  

We found that there were real differences in time when we saw the closest date, even for surgical patients, which is a bit of a surprise, but we documented this very clearly here so that we have this and remember the gnarly challenge of defining a baseline BMI.  This is an example where something that we thought would be easy was actually really hard.

Then we go into construction of the cohort, which is challenging because we don’t have patients who are being identified in real time and they’re either not approaching the study staff or actively going out.  We have to go into the claims data and try to find people who are eligible for surgery in this instance.

Before doing that, we have, in principle, non-surgical controls who could be one of many different types in their data patterns and the data is indicated by the X, the measurement of BMI that might make them eligible for surgery. There may be people who are sporadically using the VA or a one-time user, which is the first and second rows here, or there could be people who are routinely coming in the VA and having BMI measurements taken or weight measurements taken in which to construct BMI.

Then there’s people who, in the fourth row, come in the VA for a while, disappear for a year or two, and come back.  All these possible scenarios could be situations who could be potential controls.  Outlining this graphic was just a conceptual way, I guess, a visual way of thinking about the different types of controls and the potential biases we might get, depending on which group you’re in, so just something to think about.  These visual things I find increasingly helpful to make sense of study design and internal validity threats.

Then we get into discussing our covariates.  There’s four purposes, we think, and they’re listed here.  We explicitly talk about that.  Before doing that, though, we have to figure out which covariates theoretically could be relevant to this.  

There’s methods developed by epidemiologists called directed acyclic graphs that are visual ways of showing the pathways by which covariates might impact other covariates and outcomes.  It can get gnarly really quickly, but it’s a nice way of conceptually trying to make sense of what are the relevant covariates, which ones do we measure and then more importantly, which ones don’t we and might be a source of unobserved confounding?

Okay.  Then we get into the outcomes to be documented and defined.  These are the ones that are relevant for this study.  I’m going to speed up a little bit because I think I’m going a little slow.  Here’s a snippet of an actual table out of the living protocol for this study.  You see here a number of covariates that are—we know we’re going to use for some purpose.  

Then as a study team, we went through a very exciting hour meeting or two hour meeting saying, “Okay, let’s take the first one: fiscal year.  Which one, category is this going to be?  Is it inclusion, exclusion, matching or is it an outcome or is it a covariate or what?”  Checks indicate it’s for the relevant purpose, where the outcomes analysis means—checks mean we think these are covariates or confounders in that analysis.

This is a much longer table.  It’s about a four page table, because we just list every covariate. I hadn’t done that until this study and I’m going to start doing it with every study now, because that way the whole study team can chime in and say, “No, you’re missing this variable.  Add that in.  It’s for this purpose,” or, “I don’t agree that age is just for matching. We should also have it as a covariate in these analyses,” or something like that.

Instead of waiting until the end and showing people regression and they say, “Wait, wait, wait, you forgot this,” we talk about it early so that when the regression comes, it’s just following the formula listed here, basically.

Then covariate construction, we define all these different things that you need to think about.  If there’s alternative measures of a construct, how do you do it?  For comorbidity there’s lots of choices and you might make that choice based on clinical interpret ability, predicted power, what’s been used in prior studies and what we did.  We had these debates and had some discussion in the living protocol about that.

What if you have alternative measures of a given construct?  If a construct lacks clarity, like distance to the nearest VA, is it relative distance or is it nearest distance, just from the home to the closest VA?  

There’s situations when a construct is clear, but there’s alternative versions.  For example, if we’re going to identify marital status or race in the VA for veterans who come in a lot in a given year, which one do you use?  Do you take the modal?  Do you take the first?  Do you take the last?  Everybody seems to do things differently in our center, so I imagine each of you do one of these or maybe a different one.  Do you have covariant time-varying or time-invariant?  All these issues you’ve got to work through.

It’s worth documenting this stuff, because you’ll probably have to redefine it for some other purpose or the next study in this research stream, so why not do it here?  This table kind of shows trying to summarize all of this detail.  In cases where we have to define specific conditions out of diagnosis codes, diabetes will deal with these codes, OPC and PTF.  

There’s various algorithms for identifying diabetes in the VA.  From Len Pogach’s group there’s the Medicare Chronic Condition Warehouse.  There’s other algorithms you can find out there.  We document it all in one place and then our programmers went through and said, “Wait a second.  Under hypertension, code 401.1 is also in some other condition,” so there’s duplication and it’s not mutually exclusive, like it should be.  That was really helpful, to go through that exercise.

Managing, linking secondary data.  All that is to say this was all specific examples of specific issues that we went through for the purpose of the bariatric surgery study.  Tables are visuals that we use to try to give detail to the protocol, the original grant proposal that then kind of cemented and clearly documented it.  We don’t have to just go to the SAS code to figure out how we define diabetes.  It’s all here, too.  Then I could give this table of covariates to someone else, and in fact, have done that for a recent project that I’m on.

Now switching tracks, that’s all just a secondary data study.  Now when we’re thinking about linkage of primary to secondary data, how do we do it and how does documentation come in?  In this instance that I’m going to represent of this COMM study that’s now completed, we took a cohort of patients from Durham, got their VA claims to do some claim spaced analyses of medication adherence and utilization, but then we were also interested in surveying a random subset of them to make sense of a couple things.  

We wanted to examine outcomes that are not available in VA claims, like self-reported medication adherence or SF-36 or whatever.  We also saw it as an opportunity to obtain covariates that are not available in VA claims that might enrich our models that would be on outcomes and claims or outcomes in the survey with additional covariance to reduce the extent of unobserved confounding.

We documented the source of the claims and how the survey related and the flow of the overall claims space cohort and the random subset.  Then from that we also had a random subset who were invited for focus groups.  All that documentation made it much easier to report on the papers and how the sampling from the focus group came from the survey, came from the larger claims cohort.

With this survey, one thing if you haven’t done surveys in the VA is if you’re doing a de novo survey you may need to work with the Office of Management and Budget and do so early because it can be a slow process to get approval to survey veterans if you’re going to be surveying more than 10.  In that process you need to justify why your survey is needed and why it’s not overlapping with existing surveys that are already done.  I guess that’s a criteria.  If your surveys overlap too much with existing surveys, you may not be allowed to do it.

We had to clearly identify in this living protocol how we linked patient survey with VA claims and we talked it through with the study team.  We have three identifiers in this context.  This is more logistics of doing the survey.  We have the scrambled SSN that we always use.  We have a unique study ID, which ran through all the claims, the survey and the focus group stuff.  Then we have unique survey ID, so that when we mail out a survey we know that 1000543 refers to somebody with a unique study ID, which links back ultimately to a scrambled SSN.  We can link the survey data back to the claim.  Then there’s two processes for doing this, which is a logistics issue I won’t go into any further.  

Then we talked about how do we link this. Here’s essentially the consort figure, in brief, of how we first derive the analytic cohort for the claims, then from what we derived in the survey subset, then who responded.  Then the arrow means we linked back the survey completed data back to the claims data. 

This documentation helped us understand the way in which we needed to do analyses, that is comparing respondents to non-respondents.  How’s the survey subset?  Is it truly random compared to the group that weren’t approached for a survey?  All that kind of stuff.

Then analyses that are enabled by this I talked about a little bit, which is you can look at association based on survey-based covariates and claims-based outcomes.  If there’s things in certain surveys that aren’t available in claims, can you supplement or enrich?  Alternatively, can you enrich a claims-based set of covariates with survey-based covariates to generate a better prediction of your outcome and to produce the amount of unobserved confounding potentially?  

Then you could look at the association between claims-based covariates and survey-based outcomes or survey-based outcomes survey-based covariates.  Those are the things we thought about and sort of the justification or the proposal included that and we carried that through the living protocol.

That was a lot fast, so I apologize.  The conclusion, here we are.  There’s things I should note that we are not doing that I’m going to start implementing going forward.  We’re going to try to link the documentation of programs, the data cleaning, so that SAS programs are noted in the living protocol, so the living protocol can refer us back.  If everybody on the study team left but me, I could hand the living protocol to a new set of investigators and analysts and say, “Okay, this should have everything we need to reconstruct whatever.”

The other piece is ensuring that we really update the living protocol in a timely manner, because given competing demands and sometimes when the number of issues in the study is getting hairy, I’m not always updating the protocol as quickly as I should be.  That’s something I need and I was good at early on and as we’re moving into analysis, I’ve not been as good at.  Just for full disclosure, I’m not practicing all of what I’m preaching. I’m going to be trying to get better at that in the future.  It’s certainly going to—by doing so it will minimize recall bias when I do get back into updating the protocol.

Really, the documentation needs to be an iterative process.  For those of you who are saying you do that, that’s great.  It is time consuming, but it really is the only historical record, so it’s well worth the time you’re going to put in.  If it’s done well you really will have a comprehensive historical record of your study, which can be useful to new people coming into the project.  It can be useful when you’re going forward.  A new project can be useful for giving a model to other people, for other colleagues.

With that, I’ve reached the end.  If you have questions that aren’t addressed in the question and answer period, you can email me here.  Thanks very much.

*Joanne Stevens:*Thank you very much.  This is Joanne [Stevens].  We’ve got some questions that are coming in, so I’m going to start reading them now.  The first question is what are methods folks?

*Dr. Maciejewski:**[Laughter]*  Good question.  I think of methods folks as anyone who’s methodologically-oriented.  That could be clinicians who develop a passion for methods.  That could be biostatisticians, epidemiologists, economists, quantitative psychologists, quantitative-oriented sociologists or political science folks.

*Joanne Stevens:*Thank you.  Next question, would you recommend going back to create a living protocol for a study that is already in progress?  Do you have any advice on the best way to do this?

*Dr. Maciejewski:*If you’re not too far—if you’re in the analysis stage and the cohort’s already constructed and you’re kind of done, I would say my blanket answer would be yes, it’s going to be harder the farther on the study you are in its life cycle, which is the slide that Linda showed early on.  

Yes, I do think it is really useful, particularly if you see this study as part of a research stream and you’re going to take lessons learned from this current study and apply them going forward or you see applicability to other work you might be doing on other projects.

*Joanne Stevens:*Thank you.  One person has asked, and this is, I guess, out to everyone, as well, does anyone have helpful decision-tracking document templates they could share?

*Dr. Maciejewski:*I mean, I don’t have anything.  There’s no commercial software documents that I used for this.  It was literally taking the original grant proposal, HSR&D grant proposal, and just building out what I thought needed to be built out.  That’s what I did and I think it certainly is a reasonable way to go.  There may be better ways that other folks might think of, but that’s what I did.

*Joanne Stevens:*Great.  Another question is you mentioned that all team members could contribute to the living protocol.  Could you please tell us a bit more about how you kept track of who contributed what and when?

*Dr. Maciejewski:*Okay.  We don’t.  The minutes do reflect what people said on particular issues, but when it came to stating final decisions, we didn’t really keep track of who said what.  It was really the team.  It reads as though the team came to this decision collectively, through consensus, after discussion kind of thing.

*Joanne Stevens:*Okay, great.  Next question, do you know of people who use a wiki for documentation comments linking to other documents?

*Dr. Maciejewski:*Yes, I’ve been involved in a couple projects where people do that.  I use a dumb phone.  I don’t even have a smartphone.  I don’t text, so I’m not the most technologically savvy person.  Being a luddite, I don’t appreciate the full value of those, so I can’t speak to that.

*Joanne Stevens:*Thank you.  Next question, changes to the protocol approved by IRB require advanced approval by IRB.  Could you talk about how you track which revisions to the protocol require prior IRB approval and which do not?  How do you organize the coordination of the IRB protocol with the living protocol?

*Dr. Maciejewski:*That’s a great question.  I guess I think of things that go into the living protocol are of two types, maybe.  The first is major decisions around the design, how we identify patients, how we’re going to define outcomes, the data sets we’re going to pull.  Any revision of those things I think require IRB amendments.

Then there’s things that the protocol is silent about that I don’t think require IRB submission.  I showed a slide earlier where I showed the coding of how we identify diabetes and hypertension.  Our protocol does not—our IRB submission does not include this detail.  This is a level of detail that is not in the IRB submission because we didn’t have it sorted out at the time of the initial submission.  IRB did not ask for this kind of detail and it’s not clear to me why they would.  These sorts of really detailed decisions we did not go back to the IRB with, because we didn’t have that defined in the first place.

*Joanne Stevens:*Thank you.  Next question, does your VA provide access to SAS software?

*Dr. Maciejewski:*Yes.

*Joanne Stevens:*Thank you.  Another question, on slide 21, when the next steps were recorded in the living protocol were the team member—was the team member responsible for each step identified?

*Dr. Maciejewski:*Well, the next steps I list are fairly global.  We are just now finishing the survival analysis and so the next steps are define, for the expenditure outcomes, the major categories of sub-expenditures, subcategories of expenditures.  Another next step is generate descriptive statistics of these expenditure outcomes.  It’s fairly high level and we don’t—and sometimes we do a link, who is responsible for them in the minutes.

*Joanne Stevens:*Okay, thank you.  Those are the last questions that I’ve received so far.  Heidi, perhaps at this time would you like to post the evaluations?

*Moderator:*I can do that.

*Joanne Stevens:*All right.  As some folks are filling that out, if you have any further questions, certainly we can work through that and I’ll tell  you about that in just one second.  Anyway, since we are through today’s questions, and as I just mentioned, for additional questions or information regarding this topic, feel free to email the VIReC Help Desk, VIReC@va.gov.  

I want to thank Dr. Maciejewski and Ms. Kok for taking the time to develop and present today.  We’d like to invite you to join us next Thursday at 1:00 p.m. Eastern time for next week’s session, entitled “Controlled Chaos: Tracking Decisions Through an Evolving Analysis.”  Thank you, everyone, and have a great day.

*[End of Audio]*

Page 1 of 17