**Transcript of Cyberseminar**

**VIReC Good Data Practices**

**Research Application**

**Presenter: Denise Hynes, PhD, RN**

**September 13, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact the VIReC Help desk at**virec@va.gov**.*

Moderator:Good morning and good afternoon, everyone. And, welcome to VIReC’s Good Data Practices mini-series. The Good Data Practices mini-series is a set of lectures that are focused on good data practices for research. The series includes five sessions that will take place this week, Monday through Friday, from 1 to 2 p.m. Eastern. The first two sessions of the series will emphasize the importance of early data planning and continuous documentation, and offer practical information on using VA data for research. The third session will examine data sharing and reuse in the VA, and the last two sessions will be presented by experienced VA researchers, who will describe their own research and how planning and documentation were applied at various stages.

New researchers and research staff planning new projects may find this series beneficial. Thank you to CIDER for providing technical and promotional support for this series.

The first lecture in our Good Data Practices series is early data planning for research presented by Denise Hynes. Dr. Hynes is Director of the VA Information Resource Center, VIReC, and research career scientist at the HSR&D Center of Excellence at Edward Hines, Jr. VA Hospital, in Hines, Illinois. Dr. Hynes holds a joint position at the University of Illinois at Chicago, as professor of public health and as Director of the Biomedical Informatics Core of the University’s Center for Clinical and Translational Sciences.

Questions will be monitored during the talk and will be presented to Dr. Hynes at the end of the session. A brief evaluation questionnaire will pop up when we close the session. If possible, please stay until the very end and take a few moments to complete it.

I am pleased to welcome today’s speaker, Denise Hynes.

Denise:Thank you, Arika, and thanks, Heidi. Just one point, logistics, Arika will you be controlling the slides or do I control them?

Moderator:You can control them, but if you would prefer I control them, I can do that as well. The two arrows at the bottom on the left side are what control them.

Denise:Yeah, I can do that.

Moderator:Okay.

Denise:Okay, thank you. Thanks, everybody, for your patience. We get to be the testers of new technology, too, while we are talking about data and I guess that is good, because data is sort of dependent on technology. So, let us hope this all works well. Heidi and Arika, let me know if there is anything, interrupt me if we need to help those who are having challenges or if the slides are not advancing.

I want to thank everybody who has helped us get to this place. This series has been in development for some time, and I want to make sure and acknowledge colleagues who helped us. In particular, Dr. Michael Berbaum at University of Illinois at Chicago, who helped us with some of the initial concept development. He directs data analysis core biostatisticians here at University of Illinois. We also did a lot of review of existing public online course content and resources, and that was led by Margaret Browning and Linda Kok at VIReC. And, we are also going to be using some examples from ongoing and complete research supported by VA, NIH, and PCORI, and we will cite that as we go. And, in particular, as we were developing this current set of slides and series, we received feedback from colleagues, especially Elizabeth Tarlov, Kevin Stroupe, Matt Maciejewski and Laural Copeland, the latter two, who will be presenting in our research applications at the end of the week. 

So, just to provide you an overview, I wanted to make sure that we have some introductory orientation remarks for the series, because we hope that you will be logging in for the next four sessions as well. This is our schedule for the week. It will go over the course of research with two research applications towards the end of the week, and we will talk a little bit more about this as we go along today. Sessions will all be at noon, Central time, and hopefully you have that login information for subsequent series lectures as well.

I want to make sure that I point out that our series is about data management and data preparation and data documentation in the context of research. What it is not about is we are not going to talk at all about how to design a study. We are not going to talk about how to execute a research project. We also are not going to focus much time on structuring data or analyzing data, but we will talk a little bit about some of the data management tools to facilitate documentation around those. And, of course, we will not be talking at all about how to get research funding.

So, let us dive in to our topic today, which is focused on the early stages of good data practices during the early stages of a research project. You are going to hear many references to resources and we will close each session with our resources that we have tapped, so that you will have a list of resources available to you. One of the things that I should mention, as we were preparing to conduct this series, we interviewed a lot of researchers, junior researchers, senior researchers, project staff. And, one thing that we came across was that, well, of course, everybody knows about the life cycle of a research project, and probably at some point in your graduate training you learned about all these different aspects. But, it is really difficult to implement some of these practices in a real research project. So, we recognize that what we will be presenting are some of the ideal practices and hopefully, we will be able to hear some questions and some responses from you all as we go along, about what aspects are realistic and also what kind of data management practices you have used in your research.

I present here on this slide – you will hear us refer a lot to the Inter-University Consortium for Political and Social Research. I will refer to them as ICPSR. They have really set a lot of the standards for good data practices. You will also hear us refer to a lot of work from Massachusetts Institute of Technology. MIT has a lot of online resources and we will try to cite that as we go along. ICPSR looks at the life cycle of a research project in these seven steps. We will be trying to cover a lot of these materials that range from proposal, planning and writing. We have actually consolidated that a bit with project startup and data management, all the way to anticipating about archiving data from your research, especially with some of the new requirements that are coming from both NIH, Department of Veteran’s Affairs, and other institutes that support our research.

The way we have decided to consolidate is into basically three lectures that will cover from developing your data plan and proposal, data collection, data cleaning and preparation. And, analysis will be the focus of our second lecture. And, then the third lecture will focus on aspects relevant to data re-use. We understand that research is not a linear process and that a lot of times as you are planning and moving along in your research, there are, of course, different resources to support you to do that. Some are formal, as in grants and contracts, some are informal, as in leveraging other research projects and other work that you do. And, we recognize that some of these aspects are ones that you go back to and it informs other steps along the way. Our second lecture will focus on these topics and the third lecture will be on Wednesday. And, lectures four and five will try to take into account the topics we have talked about in these first three lectures and put them in the context of actual research that has been conducted by colleagues.

So, let us just talk a little bit about some of the early data planning for research. These are the topics that I will try to touch on today, and some of these slides I will go over pretty briefly. One thing that you are going to see in our lectures are a lot of lists, and I am going to try not to go over every detail in lists, but we thought some of these lists and checklists were really important. You may see some that are more appropriate for your research, some that are easier to go through in a practical application. So, there will be times that I will skim through some of these just to keep us on track. And, do not be worried, what we do not finish today we can always add on to one of our lectures and slip it over into tomorrow. But, my goal is to try and cover today’s material in one sitting.

So, let us start with the importance of data planning. But, before we get to that it would helpful for us to know, we have a couple of poll questions in here. And, the first question is how early do you usually start planning for data management for your research? During the proposal stage, after you get funding notice, when I prepare the IRB submission. And, this is a nice poll because it comes in in real time, so we can see how many people are actually responding. We actually have a pretty large group today. I do not actually know how many are in groups, but according to my list we over 200 participants. So, we probably will not wait for everybody to sign in, especially if they are in groups. But, it looks like most people try to prepare their data management plans during the proposal state. And, we are glad to hear that, because it fits in with what we are about to talk about. 

If you wait until you get your funding notice, you have probably waited quite a bit too long. These days, IRB requirements are that you submit your plan before you actually submit to your funding agency, and certainly have to have it approved before you receive any funds. So, waiting until you get your funding notice is definitely too late.

Our second poll question, I would really like to know if you use any formal data management planning software. We have come across a couple of tools and we will highlight these, if not in today’s lecture, in our lecture on Tuesday when we talk about data management and analysis. And, we have come across a couple that we will highlight. So, it looks like a very small minority of people have come across some data management planning software. So, with that in mind, we will proceed.

So, some of these seem like it is stating the obvious, but it is the kind of thing we often say three years into our project when we say, “Gee, I wish I would have done that stuff.” And, basically the importance of data planning is that it can simplify your life, definitely in the long-run. It can actually complicate your life in the short-run, because, obviously, there is a lot of work that needs to be put in place for you to think about your workflow, to identify the issues that are important, to think about the roles of your project team as you are developing and refining your protocol. And, when you get down the line at three years, you are going to wish that you documented some of those programs so that you do not have to call that programmer who now moved from the Midwest to the West Coast to figure out how you defined specific variables. Yes, we know there are anecdotes like that.

So, let us talk a little bit about the factors that influence data needs, and this will probably be the bulk of our focus in today’s lecture. I will be trying to cover these aspects, the familiar that comes along with planning your research project around a research question. Your study design, we will try to talk about some of the aspects of your study design that has probably the largest impact on your data needs and planning. And, then availability of data and also feasibility testing. 

So, let us start with the research question. We came across this set of criteria for a good research question called the FINER criteria. This was put forward by Hulley and colleagues, and they come up with basically five categories here. Your research question, is it feasibility? Does it address an adequate number of subjects, does it provide adequate technical expertise, is it affordable in time and money? Is it interesting? That is the ‘I.’ Does it get at the answer that might be intriguing to peers and your community? Is it novel. We are even evaluating this on grant reviewers. Does it confirm or refute or extend previous findings? Also, ethical aspects are critically important in research. Is it consistent with institutional review board standards. Is it relevant to scientific knowledge, clinical and health policy and future research? 

Well, we have added another dimension and that is we felt it was not specifically called out in these criteria. And, that has to do with data and data management. So, if you go back to feasibility, does your research question provide the foundation for a practical data plan? With regard to is it interesting, maybe not essential, but does it, you should consider whether your research question makes use of data in new ways. That is an aspect of being novel, especially with a lot of the opportunities that are available now. There is a lot of exciting work going on with big data, reorganization of data and networks, and you might consider that how you use data and how you plan for data may actually increase the novel dimension of your research. Also, with regard to data, is your plan adequate to protect the privacy and data security of subject data? And, also with regard to relevancy, does your research question suggest aspects about future data sharing? These are important aspects to keep in mind with regard to your research question and how you work on your data management plan.

Let us talk a bit about study design. Study design obviously talks about aspects related to your project and the processes within it. You might think about your study design with regard to how you are organizing your research questions as well as how, as you go along, how your data might be organized. You might think, for example, pardon me, I moved a little bit too fast there. Think about are the data that are needed for addressing your research data, does it support a research study that is retrospective? Does it support a research study that is prospective? If you are conducting a prospective study, you might think about are the data that you require directly from subjects, primary data. Does it, will it also include secondary data? If you are also conducting a study where there is an exposure that is assigned, for example, in an interventional study, you may also have to address issues that are much more formal. For example, with a data monitoring board, so you may need to share your data monitoring plan and your data management plan with a formal data monitoring board. And, this can be a good aspect, especially for intervention studies. It could also be an aspect of an observational study. Either way, you need to think about how will you collect your data and when do you need to review the data. When do you need to have data available for interim analyses, to present to your data monitoring board, if that is the case? These are important aspects to think about in your data management plan. It is not always about getting your data plan ready and analyzing your data at the end. Depending upon your study needs, you may have to have interim steps along the way. Similarly with a retrospective study, you might need to think about are all the data that you require to support your study design existing data, or might there be some retrospective data that you have to collect, even for a retrospective study. Are these data within the VA, or are they external to the VA?

I mentioned ICPSR. This is a really important resource. I would encourage you to take a look at it. We have some web links at the end, and you will see us citing a lot of the guidance that we prepared today and referencing ICPSR.

Let us talk about objectives, aims, hypotheses and your variable decisions with regard to a study design. Obviously, the type of data that you will need to collect will definitely be driven by these aspects. Some things to think about as you consider your research aims, whether you are identifying and describing aspects of your particular research question or on the continuum of complexity examining and comparing the types of data needs that you have, again, sort of following the continuum from existing data to perhaps prospectively collected data and maybe many data sources. You need to think about the type of data that you are collecting. Is it protected health information or is it completely exclusive or protected health information. 

Obviously, we are being pretty general here and we do not want to presume that all studies are addressing patient-specific, identifiable data or even limited data. Obviously, health services research is a broad field and units of analysis can obviously vary. But, you should think about the type of data that you require. How much data? There is a lot of factors that may play in that. We will talk a bit about data storage requirements. Think about the unit of analysis, and how you are going to organize your data. These are really important and can factor into how your data are organized, but also the type of data that you might require. If you need data that gets to a specific event, you need to keep that in mind if that event is actually captured appropriately. If the unit of analysis is at a much higher level, you may not require such granular data. 

Keep in mind temporal aspects. This is a really important factor in considering the feasibility of your research project. You need to think about whether the data that you are anticipating requiring are actually available in the time frame, not only for your research project, but for the time frame that you are interested in studying. If you are planning a study that will take place in 2016 and you are studying retrospective data, when will that data be available for you to use if it is some secondary source? To what extent will you need to get installations of data, because, although the study might be retrospective, the availability of data might become available during the course of your study and you might need to have data uploads, if you will, during the course of your project and refreshes. 

Again, with regard to measurements, you need to think about the frequency of measurements and also we will touch a little bit on informatics tools, software tools. Might you require some software tools to actually collect your data, manage your data, analyze your data. 

Obviously, the structure of your study design, your aims, really is the driving factor for the types of data that you need and how you might acquire those data. I have a table here that we included in one of our publications summarizing some of the research from the Quality Improvement Research program in VA QUERI. And, just to give you an idea, again, the study here highlighted shows a study that has four research objectives.  This is from the ischemic heart disease query, and this is back in 2004, so I am imagining that their study has changed a lot. But, at the time in 2004, they had four research objectives. One was focused on improving health of veterans in the specific VISN. Another was focused on evaluating and refining intervention for lipid measurement. A third objective was developing and evaluating a clinical reminder, and fourth was designing, testing and implementing national data reporting system. Those are pretty ambitious aims, and you can see from the data collection approaches, it ranges from audio taping interviews that need to be transcribed. There is prospective data collection, use of some existing data for vital signs, laboratory data, recruitment data, developing clinical reminders and also retrieving some data from clinical systems. Informatics tools they considered were developments of clinical reminders that was both tool for implementation and also a data collection vehicle.

Some other examples from the mental health query. They had one research objective – I am sorry – three research objectives. To compare QUERI recommended interventions to basic interventions, a comparative effectiveness kind of design. They also sought to prepare a translation package for pilot testing, and they wanted to facilitate a pilot project. So, similarly there, data collection approaches were diverse. They did chart reviews of medical records to determine medication adjustments. They also used informatics tools to facilitate this data collection. They also needed to develop some clinical reminders to implement in electronic health records. So, you can see that the diversity of a research project really dictates the range of the data collection that needs to be considered. Obviously, the types of examples I have given you here are intervention studies. I do not have any examples of retrospective here, but you can imagine that the degree of complexity increases with the complexity of the research objectives and aims.

Let us go back and talk a little bit about some of the planned manuscripts. One aspect to keep in mind with regard to how your manuscript planning should start early as it will affect your data planning needs, is that, obviously, it will help you with thinking through what your primary outcome measures are, what your measures are, how to collect your data, the timeline of your data. It is worth noting that you should think about how many manuscripts you have per aim. Think about whether there is a methods paper than you intend to write, which also might have different implications for the type of data that you are planning. It is worth noting that especially with cooperative studies program coordinating centers and statistical data coordinating centers for clinical trials, often they will make their resources assignments and planning budgets around considering a manuscript development that you have planned for your project. So, for those of you who have worked in cooperative studies planning, you may have some experience with trying to anticipate manuscripts and your data needs organized around your manuscript so that you can help your coordinating center work with you.

Let us talk a bit about data availability. Obviously, there are many good sources for locating information about data available in the VA, but I also want to make sure to mention some of the data availability of data outside VA. Obviously, this has important implications for retrospective studies, to consider using existing datasets. It is really important to think about conducting a review so that you have a good understanding of what resources are available. Highly recommended at the proposal planning stage, to be aware of existing data resources and that you are choosing the right ones and that you have a good understanding of what they have to offer not only in terms of data, but in terms of services. We can talk a little bit about that in a couple of slides. You should think about whether project will also produce a new dataset. This is really important as you think about data sharing in the future, but all integration issues. You should also think about anticipating future sharing, especially if your research project will produce a new dataset that may be useful to others and to your research team, especially since there is often situations where you may be reusing data constructed in one project for a subsequent new project. It also could have implications for informed consent and confidentiality as well. These are all things that are important to think about in the early stages, so that you plan accordingly. Because, in five years when you want to reuse data and perhaps use it for a different purpose, if that is not something that was considered during the consent process, sometimes it is pretty challenging to go backwards with regard to that.

In the VA, of course, there are hosts of resources to point you towards with regard to existing data. I will highlight a couple here. There is something called a VHA Data Portal. This is a new resource. I will show you a screenshot in a moment. VIReC has a lot of resources on our website. The HSRData-L Listserv is a communication tool that has been around for I think about 13, 14 years now, and it has survived the test of time, even with the development if WIKIs and various forum, we still see a lot of clients using the Listserv HSR data. Health Economics Resource Center provides a lot of information about economic datasets. There is also other resources that I will highlight here. I will not point you to, but there are Intranet websites for VA’s corporate data warehouse, which does provide some metadata or data documentation online. Patient care services has a host of data resources that they provide information on their website on the Intranet. Pharmacy benefits management services, PBM, provides listings and information about their data, and the Office of Information and Analytics, Health Information Governance Health Data Quality Program also provided some information about some of the data quality aspects of specific datasets available in the VA.

Accessing VA data is a topic of another whole series. We actually have a Research Access to Data lecture that VIReC sponsors in the Database and Methods series. It is presently scheduled for November 4. In that lecture we try to go over some aspects about how to request VA data, some details about VHA Data Portal and some of the new aspects about it. We also highlight that Data Access Request Tracker, abbreviated DART, for the National Data Systems managed data in VA and we will also highlight the aspects about data that other data stewards in VA manage, in particular, VIReC for Medicare and Medicaid data available through VA, Pharmacy Benefits Management as well. And, we will also try to highlight some of these other data stewards’ data products that they offer and some of the access procedures.

This is a screenshot of the new VHA Data Portal website. This is within VA Intranet. It is a collaborative effort among the four program offices listed here, National Data Systems, VIReC, VINCI, and the VHA Data Quality Program. The VHA Data Portal is intended to provide a knowledge sharing environment that supports the needs of VHA data users. Towards that end, it has information from many programs and it provides links to a lot of educational programs. It provides some links to some of the requirements for different, if you will, stakeholders in VA, both operations and researchers and those supporting veterans’ requests for data. So, it definitely covers a very broad audience of data users. 

For those of you who are researchers and are familiar with VIReC’s Intranet site that will still be there. It definitely has a much narrower focus, both our Intranet site and we also have an Internet site to provide you information about data resources in the VA, along with other topics of relevance for researchers who use data.

This is a screenshot of the Health Economics Resource Center, HERC, our colleagues out in Palo Alto. They also manage some data on cost datasets nationally that researchers in particular use, but also they provide a lot of what we call research user documentation about datasets. And, those are some unique aspects that both VIReC and HERC provide is data documentation about VA datasets that are specifically designed with research end user in mind. That is not to say that other types of data documentation are not available or are not useful, but they just have a broader audience. So, I would encourage you to look at those resources.

So, that is within VA. I have just introduced you to a whole batch of resources that you can review in your spare time about VA data resources and data knowledge. But, of course there is an immense amount of resources and data available outside the VA that can be incredibly useful to VA research as well as non-VA research. And, that is availability of data – I just pulled up a couple of screenshots here, National Cancer Institute. For those of you who have been through, especially training in schools of public health, oftentimes the type of data that you practice on are the publically available datasets from AHRQ, the National Survey Data, NHANS, the Behavioral Risk Factor Surveillance System. There are public datasets, there are also resources that our colleagues in other federal agencies and not-for-profit agencies provide that are non-public. They might be more limited datasets, they might provide more identifiable information, and they may have specific request processes to request those data. So, you definitely should take a look at these resources. These might be resources that you use as part of your primary cohort or your primary investigation. They may be useful for comparison cohorts, and I would strongly encourage you to take a look at these resources. Not all data need to be prospectively collected for your research.

As you are thinking about planning your data collection or your acquisition, depending upon your study design, here are some questions to think about. Will you need a data directly from subjects? If you do so, are you going to require any special software to collect that data? There is a whole host of software available now to support you in that effort. You need to consider that early on in your project, not only to budget for it, but to plan your workflow around it. Do you need study site-specific information, whether that is a VA facility, a geographic location, a region, and is that information available in a public dataset or will you need to interview key informants? If you need information about a VA facility, are these informations that are available in a dataset that is provided, has been collected already, or will you need to collect some of that information yourself? You need to think about how much detail you will need about events of interest, highlighting something I introduced previously. And, you need to think about if you will be collecting primary data, and you will also be using secondary data. You need to think very early on, how those data will be integrated. 

Another important aspect, especially more recently is availability of electronic medical record data if your research design is focused on patients and care issues. Definitely think carefully about, especially if whether it is VA, non-VA, multiple sites, how many different electronic medical records systems will need to be accessed? Are you doing a multi-site study? Even if you are doing a single site study, most clinical care institutions do not have a single EMR. They may have multiple pieces of software. Things you need to think about is that how well do you personally understand the data structures in the electronic medical record. What you see in a user interface like CPRS, is not necessarily what is part of the operational data store, or the structure of the underlying data that supports it. You really need to understand what the data are that you will be tapping. You need to understand what the processes are for you to obtain the data that you need. Are the data, do they need to be extracted, you need to have routines for that. You really need to be working closely with the architects or the CIOs or the information services teams that support that EMR. You should really think carefully, is there a data warehouse that supports that EMR? That may be an easier tool to use than a live care management EMR. Remember EMRs are designed primarily for health care delivery and any additional burdens on that system can sometimes slow it down. But, of course, that depends on your study design, especially if you are doing a study that involves use of the EMR as part of your intervention, such as clinical reminders, a point-of-care kind of care delivery. But, think about whether there is specific experts to consult. Certainly, in the VA, the VA, we have a corporate data warehouse that parallels very well to the electronic medical record data for the, if you will, the operational data store. So, think about whether the data warehouse could support your needs. Also think carefully about how you will link any data from an EMR to the other data sources that you might be using.

Again, with primary data there is still a lot of use of paper and scanned forms and how that data gets integrated into your plan, whether you are using electronic data entry or you are using clinical data management systems or clinical trials management systems. The difference between a CDMS and a CTMS usually tends to be whether you are dealing with drug studies or specific interventions or product or device interventions that really requires the CTMS systems for rigorous FDA scrutiny and some of the regulatory requirements to go along with specific types of studies. A CDMS, a clinical data management system, one example is the research electronic data capture, REDCap. I mention this only because in the VA this will be available soon, is multi-site web-based data entry within the VA firewall. It integrates and supports well export procedures with existing data management systems and statistical analysis packages, and has a shared library about data collection instruments and forms and development tools. This is one, there certainly are others. I am actually not able to talk about, because I am not aware of CDMSs available in VA, but I am imagining that on a site-by-site specific basis, you all may be familiar with specific ones in the VA. REDCap will be deployed centrally and we are in the middle of trying to make it work with collaborating with VINCI.

Other aspects, these are just again some questions. I am going to skim over this to make sure that we have some time, but it give you some more questions to think about with regard to primary data collection issues. 

Secondary data, again, think about what are the existing data sources that you need. Also really important to think about what type of identifier. There has been a lot of pushback for all kinds of reasons, in particular with privacy protections. Do you really need real Social Security numbers for your research. Is this the only way that you can link multiple data sources? Definitely contemplate that question, because if it is not, you should really be thinking about a study ID or some other type of unique identifier that links data across. Definitely in the VA an increasing number of the types of datasets that are available to use can be linked with a unique identifier that is the scrambled SSN, which is better than using a real SSN. Obviously, if you have to go and collect prospective data from patients, you need to think carefully about how you will uniquely identify them and link their prospective data with any existing data.

Plan for data integration, I have rarely come across a research project that uses one data source. Most studies use multiple sources and most studies increasingly are using hybrid designs with both primary and secondary data. So, data integration is critically important to think about how you are going to combine and integrate data and also how you are going to construct your analytic datasets. But, that will be a topic for tomorrow.

Let us go to feasibility testing. And, we will go back to our, as we call it, DFINER criteria for good research question. And, think about as far as testing your feasibility, again, we are talking about the early stages of planning for your research project. And, I do not want to call it a pilot study, but you really to think about ways that you can test your plan. I can tell you that I am planning a study right now and we have a whole bank of data collection tools that we are considering for our research project, and just to time out the battery of survey instruments, we plan to test that out with our staff, just to do some timing. Or, to enter some data into your data forms to test the usability even before you are talking about any kind of research subjects. Timing, practical data entry methods, these are all important aspects that you might consider in terms of your feasibility. Is it feasible with your data management system? If you want to use REDCap, if you want to use some of the SAS data collection tools, if you want to use Access. Is it going to work with the statistical package that you want to use? These are aspects that you really need to think about when you are planning your proposal, not after you get funded. And, obviously, there are some practical considerations in terms of do you have staff who understand it, do you need to bring somebody in to teach you how you to use it before you get your funding notice? And, is it affordable? Some of these packages are open source like REDCap. There is lots of others, and some of them actually cost money if it is part of a vendor provided tool. And, certainly, clinical trials management systems are definitely in the – I am not familiar with any clinical trials management systems that are open source. They tend to be highly supported, vendor dependent, and they definitely require financial resources to support them. And, think about in terms of, obviously your data needs, do you have an adequate number of subjects characteristics to test your hypotheses, and this is where the iterative process of design going back to maybe the beginning and thinking more carefully about how to readjust your study design, which means you have to go back through your data planning as well.

Some things to think about in terms of feasibility, there is this what we know well preparatory to research, which is when we are planning research proposals and we are thinking about the types of data and we actually want to prepare in advance and have a preview of the type of data, but also some sense of the scope. Can we do this research project? And, to try and think about the types of data that we might need to prepare, preparatory to research. This is definitely very useful, but it definitely has some caveats. Some aspects to keep in mind is in terms of protecting human subjects that preparatory to research is really made available for use in a specific research protocol. It does not give you the right to use preparatory to research for anything beyond that. So, it is really in preparation for a specific research protocol. It does not require IRB or R&D committee approval, as in the VA, but I would also suggest that at your university-based research programs, there may be some specific requirements. So, what works in VA may not be exactly the same process outside of VA. Keep in mind that pilot studies are not considered preparatory to research and pilot studies actually have to have separate IRB approval. It is important to note that the HIPAA privacy rule requires researchers to retain use and disclosure requested solely for the preparation of the research protocol, and if it does involve in preparatory to research use of PHI, they may not be removed from the covered entity. And, the PHI for which the use or access is requested is only for the specific research purpose. Generally speaking, counts are okay and you have to comply with privacy limits on the types of restrictions on what types of counts that can actually be provided.

Some additional data planning you need to think about early on for your IRB submission concern data privacy and data security, obviously. You will need to explain in your protocol to your funding agency, to the IRB, what types of data you will be using. You need to explain how it will be used to test your hypotheses. You absolutely need to describe the PHI needs for the study and explain why you need it and how you are going to link it, especially if you get to sensitive identifiers like scrambled Social Security number, real Social Security number or you are linking it with sensitive data like EHRs or EMRs. You absolutely need to explain the number of subjects and the type of data that you will be planning to use and why you need it. There is definitely restrictions on aspects about privacy and how you will be using data. I would definitely refer you to the important and useful VHA handbook 1605.1, which is on privacy and release of information, and 1200.05 for requirements for protecting human subjects in research. There is definitely some special populations that you need to be aware of within the VA and within specific states there are also requirements as well. So, be familiar with the environments in which you are working to conduct your research.

Data security plan, obviously, within the VA you need to have a VA, VHA, within the VHA firewall storage location. Please explore aspects with your OINT services locally and certainly VINCI provides work space available. You need to think about some important aspects about backup and scheduling. Some are routines at local facilities and VINCI, but you should know what those are in planning your study. You might need a more frequent backup schedule if you data that you acquire on a frequent basis. Make sure you plan for data transfers between sites, how you are going to do this, do you have the software to do it. And, what the VHA requirements are.

Let us just go through a checklist here. I want to make sure that we have some time for questions at the end and time for everybody to go through. We borrowed a data planning checklist that we found very useful from the MIT Library Guides. We relied on them heavily for preparing some of the materials in today’s lecture. We have actually found some course curricula available as well, and if you would like to explore developing good data practices course on your own for local resources, I would strongly encourage some of the public online resources. But, as far as data planning checklist, what existing data you will need. Importantly, do you need prospective data from subjects or not? That really is a big determinant in which arm of that tree you need to be on, and do you need to collect both types of data. Think carefully about the timeframe for the data that are needed, whether you need to do interim data analyses, whether you are working with a data monitoring board, whether you need to provide data in preliminary fashion to inform the subsequent years of your study. How much data will be generated and have you planned for storage and server space, which is becoming, you know, there is lots of creative solutions for this now. Will you link data from different sources? If data sources do not have a data element in common is there a third source available with an element common to each. How will you link these data? You need to plan for that. And, also do you have the technology, informatics and software that you need to do your work?

Formalizing your data management plan. What you are going to hear much more in the next two lectures is document, document, document, and formalizing your plan. Write it down, I guess is the best way or type it up, whatever you want to say, and keep it in a place that your entire study team has access to it. Describe the project. Make sure everyone knows your aims. What are the data that you will be collecting to support your aims and your objectives? What standard need to apply to your data? Have a real good data management coordinator for your project. You need to plan for short-term data storage and management. You need to think about legal and ethical issues, because these need to be incorporating in your data workflow. Again, make sure this is documented. You need to have a good standard operating procedures. Make sure you understand what the policies and provisions are. How will you make it available to others, if you have a multi-site study, how will you make the data that you collect available to the others on your research team? How will you even collect data? Do you need a web interface, do you, are you going to be interviewing subjects? So you need to think about the workflow. Think very carefully early on. Again, this is before you actually start your project. What will be the data management responsibilities? Who will be doing what, who do you need to hire, what type of skills do you need to see on your research team, and how will you manage it over time? It is not just in the beginning, it is not just before your project starts.

I want to introduce you to a website that we found very interesting and it is one of the most useful ones we found that has data management for the sciences, very plain and simple. We found it very useful and may borrow heavily from the architects of this site in collaborating with them in the future. They have a nice summary of links to major funding agencies and their guidelines for data management plans, and hopefully, you can easily see the link here. They also have a link to something called a data management plan, and we will show you an example in next lecture. It is kind of a high-level tool, but you might want to take a look at this, even explore it a little bit before tomorrow for those of you who are really seriously inclined for this series. And, we would love to know your thoughts about this tool.

I would like to sort of close our session today with just highlighting some of the resources that I mentioned, with some web links. Please test these out, tell us if the URL is working or not working, and explore some of these, and if you have others that you think are useful to share, we welcome them and we will add them to our additional resources and hope to make this available later.

I am going to pause for questions. I do not know if we have been keeping track of those, Arika or Heidi.

Moderator:There are no questions at this point regarding content, but we do have time for a couple of questions, so if any of the attendees would like to type questions in the Q&A pod on the right, please do so.

Heidi:Arika, we do actually have one question about a couple of acronyms that Denise used earlier.

Moderator:Okay.

Heidi:VANOD, and DSS.

Denise:VANOD, that might have shown up on one of our screenshots. VA Nursing Outcomes Database. And, the other one was DSS, Decision Support System. These are datasets that are, I am imagining that probably the screenshot for HERC has some information about the DSS data. Those are data that are, provide a lot of information about financial aspects, cost aspects in VA, but also provide some information that includes some non-financial data, laboratory results, and those are available on the VIReC website. Any other questions, Heidi or Arika?

I am going to kind of keep moving here while we see if there is any questions that come up, so that we close at the top.

Heidi:We did just have one come in here.

Denise:Okay.

Heidi:Can you answer specific questions about REDCap? If not, who can I contact to find out more? When it will be available, can PHI be stored on it, can non-VA collaborators use it, etc.

Denise:Yes, please send us an email at, I think we have our VIReC@VA.gov shown on this slide. We have staff at VIReC, who are specifically working on the REDCap project. If it is an availability issue, if it is a technical question, we will get some support from VINCI, who is collaborating with us on this. And, if it is for outside VA, we will get you in touch for those outside the VA.

Heidi:Great. Thank you. And, one, hopefully we can sneak this in before we are at the top of the hour. Can you talk a bit more about the pros and cons of paper forms and electronic forms in terms of data collection?

Denise:Well, a little bit about pros and cons. So, paper forms, easy to use, most research assistants of any minimum training can use them. I would say that is right now where the pro ends. The con is then you need a data entry person to get it into an electronic data management system. That requires another person. Depending upon that type of paper form, there is also the scanable forms. Some of the centers are still using Scantron forms. Those can be very useful. That technology has definitely improved over the last ten years, a lot. I have not seen a lot of it used, but there are issues around you have to have good software technology to support the Scantron forms so that the data are entered properly. I will leave it at that.

Heidi:Fantastic. That looks like all the questions we have. Arika, Denise, do either of you have any concluding remarks you want to make before we close out?

Denise:No, just a preview here of tomorrow’s lecture. These are the topics we will be covering. We will get into more data management and analysis workflow.

Moderator:Please join us tomorrow from 1 to 2 p.m. Eastern. Thank you for joining and have a great rest of the afternoon.

Heidi:And, for our audience, I am going to close out the session and bring up a feedback form. If you could take a few moments to fill that out we would appreciate it. Thank you.

[End of audio]

Page 1 of 14