**Transcript of Cyberseminar**

**Timely Topics of Interest Series**

**Selecting a Valid Sample Size for Longitudinal and Multilevel Studies in Oral Behavioral Health**

**Presenters: Henrietta L. Logan, Ph.D.; Aarti Munjal, Ph.D.; Sarah M. Kreidler, DPT, M.S.; Deborah H. Glueck, Ph.D.**

**October 15, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact the VIReC Help Desk at virec@va.gov.*

Moderator:Good morning and good afternoon to everyone, and welcome to VIReC’s special topic seminar entitled, “Selecting a Valid Sample Size for Longitudinal and Multilevel Studies in Oral Behavioral Health.” Thank you to CIDER for providing technical and promotional support for this seminar. Today’s session is presented by Dr. Henrietta Logan, Dr. Aarti Munjal, Dr. Sarah M. Kreidler and Dr. Deborah Glueck. 

Dr. Henrietta Logan is currently a professor and director in behavioral science at the University of Florida College of Dentistry in Gainesville, Florida. Dr. Aarti Munjal is an assistant research professor in the Department of Biostatistics and Informatics at the Colorado School of Public Health at the University of Colorado in Denver, Colorado. Dr. Deborah Glueck, is associate professor of biostatistics at the University of Colorado School of Public Health, and Dr. Sarah Kreidler, is a doctoral student in biostatistics at the University of Colorado, Denver. Questions will be monitored during the talk, and will be presented to the speakers at the end of the session. 

A brief evaluation questionnaire will pop up when we close the session. If possible, please stay until the very end and take a few moments to complete it. Without further ado, I would like to welcome our speakers for today.

Unidentified Woman:Thank you very much. Thank you for those generous introductions and thank you for the opportunity to present our topic to you today. We have co-authors and their names are listed on the screens. I will move quickly through them and contributors. We declare no conflict of interest and we want to acknowledge the National Institute of Dental and Craniofacial Research and NCI for their generous support in the preparation of this material. The poll questions are listed. We would like to know who is in our audience. If you would take a few moments; A) If you are a clinician, B) If you are a data scientist or statistician, C) Student, intern, fellow, D) Other researcher and support staff. If you would mark it please?

Moderator:Thank you very much. It looks like the answers are streaming in. So for our attendees, simply click the circle next to the role that best describes your primary role here at the VA and we should have those results in just a few seconds. In fact, it looks like the results have stopped streaming in, so if you’d like to talk through those real quick, feel free?

Unidentified Woman:Thank you very much. I think we will move on with our learning objectives. We are presenting a conceptual framework for conducting a power analysis. You will understand how to interact with our free web-based power and sample size software, and you will learn to write a sample size analysis at the conclusion of this presentation. Our agenda, we will each be presenting for 10 minutes and as listed on the screen, and we will wrap up with our question and answers. How do we choose a sample size for complex oral health designs? Or any kind of a design? I feel strongly that this is more than just a statistical question, this is an ethical question.

In this era of limited resources, if our sample size is too small, the study will be inconclusive and we have wasted resources and we cannot build on each other’s work. If the sample size is too large, then the study may expose our participants to possible harm due to research. In a prior study of therapeutic interventions to reduce acute pain, we found that an intervention that instructed individuals to focus on their physical sensations, dramatically reduced their experience of pain during root canal treatment. And I believe this driving example is illustrative of any kind of a study in which you have a highly aversive experience, at least from the patient’s perspective.

So we are going to use this to give you an example of how to use this software. In this study, we had identified a group of highly at-risk individuals. It is the aqua box with the number one in it that says, perceived control, low – desired control, high. These people had in many studies shown themselves to be individuals who responded very negatively to many interventions. But in this case, we had randomized them to sensory focus or standard of care and have found that they had done very well. And in a follow up study with my VA colleagues, we had called them and found that their long-term memory of the experience was far less averse than the individuals who had not had the sensory focus intervention. Therefore, we decided we wanted to set up a follow up study, which we believed would provide insights into how to make the therapeutic experience less aversive.

We wanted to randomize individuals to either a sensory focus, which was an audiotaped instruction, asking them to focus on their physical sensations or the standard of care. We wanted to measure pain. And our hypothesis dealt with the fact that we believed that the long-term recall of pain would be different, because different determinates of that experience. We had hypothesized that across time, our sensory focus, which is the yellow line, would show that those people who were at risk would actually recall less pain than those individuals in the aqua line, who had the standard of care. This time by treatment interaction was going to be observed in a one year study. We would measure pain immediately after the root canal, and at six months, and at 12 months.

Now the question becomes, how do we design the study so we find – are most likely to find a difference if it truly exists with this sensory impact – the sensory focus impact intervention. And if it is not there, we don’t want to find it. In other words, we want to increase our probability of finding it if it is there, and not making an error. So we are going to recruit participants who have a high desire for control. And we think we can get about 30 patients per week through our clinic. About 40% of them have consented in prior studies. How do we calculate this? Well we wanted our type-I error rate at 0.01, meaning there is not a very high chance that we will find a difference if it exists by chance. We want to set our power at 0.90, and we know that from prior experience, our lost to follow up is about 25%. Now I will turn it over to the statisticians to tell us how to use the software.

Aarti Munjal:Hello everyone, my name is Aarti Munjal and I am an assistant research professor at the Colorado School of Public Health. In this talk I will first introduce our statistical software called GLIMMPSE, and then describe how to use GLIMMPSE to create study designs that are multi-level, and longitudinal in nature. GLIMMPSE is based on the original POWERLIB software, which was published in the Journal of Statistical Software, and is a SAS/IML module written by Dr. Muller and his colleagues. All of the information can be found on our website listed here. Please know that if you have any trouble accessing the website, it may be just because several of you are trying to access the website at the same time, therefore, we appreciate your patience. 

With GLIMMPSE, we offer scientists and statisticians with a free and open source tool that requires minimal statistical training, allows you to save study designs for future use, is available for download on Smartphones as a mobile app, and is coming soon on an iPad. In this session, we will walk you through a series of snapshots that describe how to use GLIMMPSE to create a study design. To start with, GLIMMPSE provides two study design modes. Guided study design mode is used by scientists. And the Matrix study design mode is appropriate for trained statisticians. You can also use its upload feature to upload a previously saved study design, so you do not have to retype everything all over again. 

To create the study design described by Dr. Logan, let’s get right into the guided study design mode. So, we select the guided study design mode and the first screen that you will see is the solving for screen, which asks you to make a selection whether you are solving for power, or sample size. The panel on the left you may note, lists all of the inputs required to complete the study design. And the check mark next to the input indicates that the input is complete. And the add sign indicates that the input is still incomplete. In this study design, since so far we have specified that we are solving for total sample size, the solving for input appears with a check mark. Since we have specified that we are solving for sample size, GLIMMPSE requires us to enter desired power. As Dr. Logan mentioned, power values are numbers between 0-1 and higher values correspond to a greater likelihood of rejecting the hypothesis.

And in this case, our desired power is 0.9. So, to enter the desired power, we enter the value in this text box, hit add, and the value appears at the bottom. In case you are interested to run the study design with multiple power values, you can follow the same procedure again. Next thing we need to specify in GLIMMPSE is the type-1error rate. We note the type-1 error rate is the probability of a type-1 error occurring, and is often referred to as alpha. Again values range from 0-1 and the most commonly used values are 0.01, 0.05 and 0.1. As Dr. Logan mentioned, in this case we are going to use a type-1 error rate of 0.01. So to enter the value, we enter the value in the text box, hit add, and then the value appears here.

In case you are interested to run the study design for multiple type-1 error rates, you can follow the same procedure again. Next, we specify our predictor variables. In this case it is treatment. So we first type in the predictor variable as treatment here, hit add and the value appears. And then we enter multiple categories for every predictor we want. And we can follow the same procedure if we want to enter multiple predictor variables for the study design. Next, we need to specify outcome or response variables, which is memory of pain in this case. So we type in the variable name and the variable appears. Again if we are following the multi-variable design, we may want to enter multiple outcome variables, so we can follow the same procedure.

Since we have specified that our outcome variables are repeated over three measures, so we need to enter repeated measures information in GLIMMPSE. Since we are measuring the outcome at three time points, we specify the variable and then we specify that it is numeric value. Since we are doing three measurements this time and these are equally spaced at six, 12 and 24 month intervals, so we simply label them as time point one, two and three. In case we want to enter repeated measures at multiple levels, we could just go ahead and add or remove a level as we want. You may recall that we want to know if the memory of pain patterns over time are different between the two treatment groups. Therefore, in this case our hypothesis of interest is time by treatment interaction. To provide this information in GLIMMPSE, GLIMMPSE offers us several choices to choose from and since in this case we are interested for time by treatment interaction out of the possible options, we select interaction and then we specify the factors which are time, versus treatment in this case.

Once we have entered the hypothesis of information, then at this point you have described the study design including predictors, outcome variables and the hypothesis of interest. To complete the sample size calculation, we need to choose means and variances, which you expect to observe in the study. Dr. Sarah Kreidler will now describe how to select these values and how to input the values into the GLIMMPSE software.

Sarah Kreidler:Hi, my name is Dr. Sarah Kreidler. I am a student here in biostatistics at the University of Colorado. And I will be talking about how to choose your means and variances, as well as correlations for your study design. So at this point in the design, we have entered information kind of, of the layout of what our predictor variables are, what our outcome variables are, and also what our primary hypothesis is. And now we have to make some choices about how big of a mean difference we expect to observe, and how much variability we expect to see in our outcomes. And obviously you are not going to have these exact values at this point because if you knew the answer already, you wouldn’t have to run the study. But we can still use the literature and other sources of information to make reasonable choices.

And the choices you make here are really going to drive your final sample size. So the question we get fairly often as statisticians is, you know, how to I pick these numbers for means and variances when I am trying to power a study? Well there are a lot of places that you can look. First there’s obviously if you have run some pilot data previously. Maybe you have done a little study with a few patients and you are looking to expand that further. You can take that data and actually calculate the means and the variances you need, and use that information. So that is a great source for doing a sample size calculation. Another thing, if you don’t have data already, you can look to the literature. And it is unlikely that you are going to find the exact number that you are looking for, but you can usually find a similar study that will be helpful in your power calculation.

And lastly, you can use your clinical experience. So a lot of times, take for example this memory of pain. Maybe if someone’s score goes down by one point, maybe that’s really not clinically important. You may have a sense for how big that difference has to be to be clinically meaningful. And you would – that value can be very useful in your sample size calculation. So before we pick our means and variances for our example, I just want to remind you again of our hypothesis. So it is fairly easy to find, say you know, treatment differences in the literature. But we are actually looking at an interaction hypothesis. So what an interaction is, is a difference of differences. So, that treatment difference that we are observing in each of these time points, we want to see how that differs over time. so, that interaction hypothesis again, it’s that difference of differences. And we will show you sort of a convenient way to find that information from the literature.

So we have a table that we obtained from previous studies that shows us what these memory of pain scores are, our primary outcome, what that looked like over time. We have measurements for our two interventions. We have got baseline six months and 12 months that we sort of laid out in this table. And how we will start to get a sense of our interaction effect is we will first take the differences down the column. So for example, at baseline we see a difference of about -0.9, -1.5 at six months and then at 12 months we have -2.1. So now that we’ve got that information, we can get a sense of our interaction effect. So what we are going to do is we are going to take those individual treatment differences and subtract the 12 months and the baseline from each other and that gives us a value of -1.2.

So again even though this particular research study did not report the interaction explicitly, you can use some information from tables and published studies to get the values that you need. Okay, so now we have got a sense of our mean difference for our specific hypothesis, that time by treatment interaction. Now we need to figure out some information about our variances and correlations. And when I sit down to do a sample size calculation, I like to think of the different sources of variability  and correlation in the design. Again, you may have say a cluster randomized design where people within the cluster are going to be correlated. And in our particular case, we have longitudinal measurements. So when you take measurements on the same person over time, you expect those measurements to be a little more similar than the measurements you would see between two different individuals.

Therefore, our repeated measures within a given participant, we expect to be correlated. And we have to account for that correlation in our power analysis. And the other aspect of this is variability. Again our primary outcome is memory of pain. And if we were to take 20 people and measure this memory of pain score on them, we don’t expect them all to have the same answers. So there is sort of this natural variability in our outcomes. And we need to characterize that as well when we do our power analysis. So, I will show you in this study from Gedney, Logan and Baron from 2003 how we can find some of this information. Again you are not always going to be able to find these correlations, but it is always good to hit the literature and see if you can get reasonable values for these numbers. 

So, in this particular study, they actually reported some correlation for memory of pain over time. and they reported that from one week to 18 months, we had a correlation – there was a correlation of about 0.4, so a fairly moderate correlation there. And that is the only number we have, so we have to make some assumptions here. And what we are going to assume is that, you know, measurements taken 18 months apart, the correlation we observed there will be similar to say, what we will observe between baseline and 12 months. And another common feature with repeated measurements is that measurements taken closer together in time tend to be more strongly correlated than those taken farther apart. So again we are going to take this number from the literature to use for our correlation from baseline to 12 months, but then we are going to assume that measurements closer together from baseline to say, six months, will have a slightly stronger correlation. And we will show you how to type that into the software as we get there.

And again, the other aspect of this is that we have dealt with the correlation, now we need to look at the variability. And so we have another study we found in the literature from Logan, Baron and Kohout, and they actually reported a standard deviation for our particular outcome, this memory of pain. And again standard deviations are actually fairly easy to find in the literature. But in our case, that value is 0.98. Okay, so Dr. Munjal showed you sort of the first few screens that you will encounter in GLIMMPSE. So further down, you will encounter a section where you can encounter these mean differences. And this particular screen that you are looking at, it will show you sort of the means you expect to observe in each treatment group. And you will notice the dropdown list at the bottom tells you what time you are at. Because we have repeated measures, you could either – you know, you can enter the means for each time point using our interface, or you can do that little calculation and just enter it in one spot. So we are using that second method. We are not typing in that entire table of means. So we have selected the third time point, which corresponds to our 12 month follow up. And we are entering that value that we had previously calculated, or an interaction effect of minus 1.2.

And as you can see, you can just type that directly into the text box and move on to the next screen. So once you have entered those means, there is a separate section where you can enter information about variability. And the screen you are looking at here is called the within participant variability screen. And you will notice that it has multiple tabs. And this sort of corresponds to this idea of the different sources of correlations. So if you were to have method repeated measures, let’s say you had time of day within time, you would have another tab here for each of those levels of repeated measures. But for our particular study design, we’ve just got time as our one source of correlation.

And then we’ve got our response variable that potentially has some natural variation in it. So on this first tab where it says responses, you can enter that standard deviation value that you found for the outcome. And then to complete the screen, you will click on the time tab. And what are you looking at here is called a correlation matrix. So you will see that times, one, two and three are listed across the top and also down the side. So for example in the lower left hand corner, you will see a number of 0.4. So in the literature, we had found that value of 0.4 and we were making the assumption that the correlation between baseline – measures taken at baseline and at 12 months, would be about the same. So hence, where you can see that box corresponds to the correlation between time one and time three, that is our baseline to 12 month measurement, so we enter 0.4.

And then in the box above it, you will see a value of 0.5. So that was that other assumption we were making where we thought that measurements taken closer together would have a slightly stronger correlation. And that 0.5 corresponds to the correlation we expect to observe between baseline and six months. Okay, so the next step is, you have to select which statistical tests you plan to use. One of the most important features of doing a power calculation is you want the analysis you do the power for, to match the actual data analysis you are going to do once you have collected the data. Now, there is a lot of different tests for a multi-variant layout or longitudinal study like we have here. And we have a full tutorial on how to select between these tests on our website, which is samplesizeshop.org, which we will show you a little later on.

But as a general rule here, you are fairly safe selecting the Hotelling-Lawley Trace if you are really just not sure. Okay, so once you have entered all of your information, you will know that your design is complete when the calculate button highlights to green. And so you go ahead and click that, and you will see this table. And what you are seeing here is the total sample size that you would need to obtain in order to achieve your target power. Now it is not always going to give you precisely 90% power, because as you can see like on the second row, sample size is 26, gives you 92.5% power. So basically what GLIMMPSE is doing is giving you the smallest sample size you can use that still achieves your minimum power. And another screen that we didn’t show you here, you can – if you are uncertain about those values you typed in, say for the means and variability, you can apply little scale factors to them.

So, what we see in the table here is how that variability might affect our sample size. So for variability we really observed was twice what we anticipated. We would need a much larger sample size of 84. So if we assume that our values we pulled from the literature are appropriate for our study, and we are fairly certain of that, the sample size we would need for 90% power would be 44 people, or 22 in each of our groups. Okay, and now I will turn it over to Dr. Glueck, who will talk about how to write all of this up in your grant proposal.

Debora Glueck:Thank you Dr. Kreidler. Okay so I will be speaking for about 10 minutes about writing the grant, and then we will have some time for question and answer. Here’s the outline of what I’ve been talking about. Like Dr. Kreidler mentioned, We think it is important to align power analysis with data analysis. We will be discussing how to justify the power analysis in your grant application. We will talk about how we might account for uncertainty in the inputs for a power calculation, how we handle missing data, which is inevitable in any randomized controlled trial, or observational study. We will talk about how we can demonstrate enrollment feasibility and we will discuss how to handle planning for multiple aims, which are common in most of the studies that I design. But this is the way we are going to run this part of the talk.

I am going to give you exactly what I would write in a grant, and I am going to go ahead and highlight features of the write up that are important to put in any write up that you have for a grant. So these are things that you would want to have in any grant write up you are going to do. I am going to highlight them in red and we will talk about each one of them in turn. The first thing you need to do is to describe the data analysis that you plan to carry out for your proposed study. Here we are going to do a repeated measures analysis of variants. The next thing you need to do is to tell people what test you are planning to use. So here we are going to use, as Sarah mentioned, the Hotelling-Lawley Trace test, which is a good test for this activity. Next, you have to tell them what hypothesis you are using.

Here we are testing a time by treatment interaction, which is the hypothesis that the pattern of responses the memory of pain will be different between the two treatment groups. You may recall that our type-1 error rate that we planned for the data analysis was alpha=0.01. It is important to use the same proposed type-1 error rate for both the data analysis and the power analysis. Now, many people when they do a power analysis, do something which is wrong. What they do is they use a different hypothesis for the power analysis than they do for the data analysis. In this example, somebody who was doing it wrong, may provide power for a treatment effect when they were planning to do the data analysis for a time by treatment interaction. Instead, one should do power for the time by treatment interaction hypothesis to match the proposed data analysis, which again is a time by treatment interaction.

Here is my summary again, and again I am going to highlight important things that you should include in your sample 5 calculation summary. One of the things I’d like to stress is that you need to give the reviewer all of the information that they might need to exactly replicate your power and sample size analysis. This is to show that you have nothing to hide and also to allow the reviewer to check to see how well you did. Here, we are going to report all of the inputs we had. We report our standard deviation of 0.98, and then we take a fair amount of time and a couple of words to explain the correlation between baseline and six months will be 0.5 and describe that the correlation will decrease slowly over time. So we will have a correlation of 0.4 between the pain recall measures at baseline and 12 months. Now I’d like to point out that we say here that this is based on our clinical experience. If this number which is based on our reading of a previous paper, we would include a citation for that so people would know where these numbers came from.

Again, when you justify your power analysis, please try to give all of the values needed to recreate the power analysis and provide appropriate citations. If it is from unpublished data in your lab, say that. If it is from a previous publication, give the citation. If it is from your clinical belief from years of clinical experience, say so. Here is my sample size calculation summary again, for desired power of 0.9, here I am describing what the power is that is proposed for the study. I also describe the type-1 error rate. The type-1 error rate of 0.01. We estimated that we would need 44 participants to detect a mean difference of 1.2. Here I am describing what inputs I used for the time by treatment hypothesis. Now all of that is very good, but you may have noticed that I got my estimates of variability and correlation and things from a variety of sources. 

Some of them I drew from previous studies, which were published in the literature and some of them I used Dr. Logan’s clinical experience to pick. What if I got the wrong inputs? How will I, and the reviewers, know that I have still done a good power and sample size analysis? In order to answer that question, I like to include a power curve. This is a power curve shown in the figure here. On the vertical axis you see, power, on the horizontal axis, you see mean difference. Here we have a time by treatment interaction, so we have a single mean difference, which encompasses the difference in time between the two treatment groups. On the graph itself, you see three curves. The purple curve shows what the power would be if the variants were half as much as I think I am going to observe. The green line shows the power, if the variance was exactly what I am going to observe. And the yellow line shows what the power would be if the variance was two times what I was going to observe.

Now you may notice, the higher the variance, the lower the power. The lower the variance, the higher the power. Now I am going for a power of 0.9, so I have drawn a horizontal reference line on my figure showing the power of 0.9. I also have drawn a shaded rectangle across my picture. Now what this shaded rectangle represents is the following; you may notice that the – I want to ensure that I have a power of about 0.9, even if I have mis-specified the variants, and even if I have mis-specified the mean difference. The yellow rectangle shows that for a wide range of variances, and for a wide range of mean differences, I end up with power, which is approximately 0.9. Let’s see if I can get my cursor here to show that to you? And my cursor has disappeared, so I won’t be able to show you that. But, imagine in your mind tracing with the cursor...

Unidentified Woman:You can just click anywhere on the slide, and it will move back over.

Debora Glueck: Oh, there we go. Oh, thanks that’s very helpful. I really appreciate that. So you can see here is, this is the point at which I would get power of 0.9 if the variance is half as much as I had observed. Now this is actually the point that I calculated. This is a difference of -1.2 with a variance of exactly what I observed. You can see that anywhere around here the power is about 0.9 for any of these inputs, showing that my power example size analysis is relatively insensitive to my inputs. Okay, so here is the summary that we have written so far. Again, just trying to repeat what we have said, we have indicated the analysis plan, the test, the hypothesis, all of the inputs for the power analysis, what our desired power is, the type-1 error rate, the goal sample size and what our mean difference was for the time by treatment interaction.

But you may remember that Dr. Logan specified that we might have missing data. In fact, Dr. Logan felt that we would have up to 25% lost to follow up based on her previous clinical experience and that of her VA colleagues. The response to this, a conservative approach, is to simply inflate the calculated sample size by 25%. If I take 44 and then inflate it by 25%, I end up with 55. However, you may remember the people are being randomized to one of two treatment groups; a sensory focused interaction, or a standard of care. Because I am trying to maintain equal allocation to each treatment group, I am going to slightly inflate the sample size just a little bit more so I get a number that is evenly divisible by two. Here again to my write up, I explained to reviewers exactly what is going on.

Over 12 months, we expect at 25% lost to follow up. We will inflate the sample size by 25% to account for the attrition for a total involvement goal of 56 participants. I like to spell it out. That is 28 participants per treatment arm. Well here is a question, I want to demonstrate that I can actually accrue that large of a sample size. So I have to answer two questions for the reviewers. Is the target population sufficiently large? And, can the recruitment be accomplished in the proposed time period? Another good question is, what is a good time period for recruitment? Going back to the information that Dr. Logan and her VA colleagues provided, we notice that attending the clinic, we have 30 patients per week who have a high desire for control, but a low felt control coping style.

Among these patients, we expect from her clinical knowledge that we will get about a 40% consent rate. So about 40% of them will consent to enroll in the study. You may recall that we need a sample size of 56. What sample size will be available? If we have a three week enrollment period, we are going to end up with a sample size of only 36. Not big enough. If, however, we shift to a five week enrollment period, the effective sample size will be 60, more than enough to complete the targeted enrollment in a reasonable amount of time. Important to write all of that up. You need to demonstrate enrollment feasibility. Tell them how many patients per week you could accrue that will fit the inclusion criteria.

Tell them what the expected consent rate will be, and then let them know that you will be able to hit your targeted enrollment, and how long it will take. At an effective enrollment of 12 participants per week, we will reach the enrollment goal of 56 participants in five weeks’ time. Now, this study which is a simplified study that we put together so we could give a one hour talk on it, is much simpler than most of the studies I end up writing these days. In fact, most studies I write have multiple aims. If I have multiple aims, how do I do the power and sample size? Well, my approach is this, what I do is I write a section exactly as I have described for each aim. So in this statistical analysis and sample size analysis section, I describe first the data analysis, the plan for each aim, and then I write a corresponding power and sample size analysis for each aim.

At the start of the document, I describe each – the summary of each power analysis. So I tell you what the sample size is for each aim. And then I choose the maximum of the sample sizes calculated for each aim. This ensures that with that enrollment, I will have sufficient power to prove or disprove each hypothesis I plan to test in this study. Now, I want to thank you for taking the time to listen to us today. Our group is happy to present again to the VA cyber seminar. We would like to have some guidance for you as to what topic you would like us to talk on. So if you could go ahead and enter into the interactive poll, please feel free to hit more than one of these if you would like to see more than one of these.

Again, our group is funded by a grant from the National Institute of Dental and Craniofacial Research to provide power and sample size tutorials for behavior and social scientists focusing on oral behavioral health. We are happy to provide more seminars for you, if you enjoyed today’s seminar. Would you like to have us talk about power and sample size for missing data? Power and sample size for mixed models? Power and sample size for binary or Poisson outcomes, or choosing cluster sizes for multi-level studies? Please go ahead and enter that on your screen. I am going to go ahead, because what I’d like to do now while you are doing that is to ask to see if you have any questions or comments on my talk?

Now, we have only been able to talk for about a little less than 45 minutes today. So what I would like to encourage you is, if you have any additional questions, or want to use our free power and sample size software, please visit our website. Go to www.samplesizeshop.org. Feel free to try out our power analysis, read one of our extensive tutorials on power and sample size, or refer to one of the 40 journal articles we have published over many years on power and sample size. If you want to speak to me directly about a power analysis, feel free to drop me a line at Deborah.Glueck@ucdenver.edu. I will leave both of these sites and my email up in front of our happy cow from Crescent Butte while I answer your questions and comments. 

Moderator:Thank you so much Dr. Glueck. Before we get on to moderating the Q&A, I know a lot of our attendees joined us after the top of the hour. So I just want to let you know that to submit a question or a comment for the presenter, please just use that type in box located in the upper right hand corner of your screen, submit a question and then just press, send. And at this point I will turn it over to Arika Owens from VIReC to moderate the Q&A.

Arika Owens:Really quick Molly do you want to go over the poll results for compliance purposes, or just move on?

Moderator:That’s okay. We can move on at this point. It is captured in the recording so people will have a chance to review those if they would like. But thank you for asking.

Arika Owens: Okie doke. First question for the presenter, does the current software handle binary outcomes?

Unidentified Woman:Okay, if you can hold on for a second here. I am going to [informal background conversation] – I am going to stay on the line. If one of my colleagues needs to answer the question, I will pass the phone over to them. But I am going to try to go ahead and answer all of the questions I can handle. The question as, does the current software handle binary outcome data? And so this is going to be a longer answer than you wanted, but I am going to go ahead and give it to you anyway. Okay, so binary outcome data is a kind of data where the answer is yes or no, okay?

And with data like this, it is not usual to use the general linear uni-variant or the general multi-variant models. Because these models typically assume data with a Gaussian or normal distribution, okay? Many investigators suggest that instead, you use the generalized linear model with a link function to handle binary outcome data. However, research by Dr. Munjal and our group suggests that this approach produces two problems. The first problem is, inflation of the type-1 error rate. So if you do that and you think your type-1 error is 0.05, you might increase it to almost double that, or 0.1 depending on the experimental design. The second problem, our convergence problem in the presence of missing data. Sometimes the algorithms of the generalized linear model do not converge. Well what about just taking the binary outcome data and throwing it into the multi-variant model? How does that work?

Again, preliminary results by Dr. Munjal and her group suggest that that approach works great. It has great convergence, even in the presence of up to 10% missing data. And in addition, it controls the type-1 error rate. So the short answer is, yes. You can go ahead and just use our software for binary outcome data. Just go ahead and calculate the mean response. So if the mean response of a yes/no outcome is in fact a proportion, take the proportion, throw it in as an outcome and move ahead. It should work great. We are going to have guidance and tutorials about this subject up within the year, so check back at samplesizeshop.org for updates. Next question?

Arika Owens:Great. Thank you. Do you intend to apply for your software to be listed on the allowed software for VA computers? Right now it is not, as far as I can see, is one comment.

Unidentified Woman:Since – that is a great question and I wish I had written that down. Hopefully I can get a transcript of this. We are happy to have our software listed among the allowed software for VA computers. And as soon as I get off this conference call, I will find out how to do that. And I will get our software listed so it is okay for you to use from VA computers.

[Informal background conversation]

Sarah Kreidler:This is Dr. Kreidler, just another comment. That our software is – except for the mobile versions, it is fully web-based. So, as long as you don’t have any kind of restrictions in what websites you can browse to from a VA computer, you should be able to hit our site. But obviously you’d have to check with your IT people to see if there is any of those types of restrictions in place. I will hand it back to Deb Glueck.

Deborah Glueck:Hi! Deb Glueck back again for your next question?

Arika Owens:Great, thank you. Would you provide an example of selecting efficient size for a power analysis?

Deborah Glueck:Okay, so if I understand the question, what you want me to do is provide an example of selecting an efficient size for power analysis? So, I think what you mean by efficient size, is a size that is just large enough to allow sufficient power, but not too large to avoid exposing people to harm. And so for example, I would refer you to our papers. There are more than 40 papers listed at samplesizeshop.org, each one of which has a detailed description of a power analysis. I’d like to particularly highlight a paper recently published in BMC Medical Research Methodology. This is an open access online journal that you can get, no matter what your library access is, by my colleagues, Dr. Yi Guo and Dr. Henrietta Logan, one of our speakers today, which contains detailed descriptions of a power and sample size analysis, hopefully an efficient one, for a multivaried model. Next question please?

Arika Owens:Great. Thank you for that. Does the software handle inconsistent cluster sizes?

Deborah Glueck:So the question is, does the software handle inconsistent cluster sizes? So right now, the software handles only balanced designs. However, inconsistent cluster sizes are a subject of active research by our group. In fact, Dr. Kreidler’s first paper and her dissertation, which I expect shortly – you hear that Dr. Kreidler? Is – is about cluster sizes which have different sizes. So we are working on that problem, and we will have the results out as soon as we can. Again, watch our site, samplesizeshop.org, and we will get that paper posted. I promised Dr. Kreidler’s mother that she should be graduating by the spring, so I would expect it in the next couple of months. Next question, please?

Arika Owens:Great to hear. I don’t see any more questions at this time, but there are a number of comments?

Deborah Glueck:Okay.

Arika Owens:The first comment, nicely presented, very effective, structured approach, thanks!

Deborah Glueck:Thank you!

Arika Owens:Another comment suggests a topic also, overview of analytical approaches for longitudinal studies and comment problems/mistakes.

Deborah Glueck:Okay, we will – so we are currently funded by the National Institute of Dental and Craniofacial Research to work on power and sample size work. However, I am glad you want to know about how to analyze the data because I find that power analysis is essentially inextricable from data analysis. So when you read our manuscripts, on the site you will see that we often suggest a reasonable data analysis approach in the context of suggesting a power analysis approach. So that might provide some guidance for you. But I will write that down and I am sure I will be writing a grant on that soon, in the future. Thanks for the comment.

Arika Owens:Another comment, excellent, timely presentation. 

Deborah Glueck:Thank you.

Arika Owens:And I see a couple more questions, so let’s get to...

Deborah Glueck:Let’s get the questions. That would be good.

Arika Owens:Can the software calculate sample size for structural equation modeling?

Deborah Glueck:So the question is, can the software construct sample size...

Arika Owens:Calculate.

Deborah Glueck:...calculate sample size for structural equation modeling? 

Arika Owens:Yes. 

Deborah Glueck:Okay, so currently the software cannot calculate sample size for structural equation modeling. However, I am writing down all of your comments and we will see if we can add functionality. One of the major ways that we add functionality is we get requests from users such as yourself, and we write grants and try to get this funded. Dr. Kreidler, would you like to add something?

Sarah Kreidler:So if you go to our GLIMMPSE.samplesizeshop.org, which is where the actual software is running, there is a feedback link where you can submit comments to us on the software, and you can select a feature request. So if there is a type of analysis or, a power that we don’t have available for us, please let us know that because that is the main way that we know how to improve the software. So again, you will find on the top page of the software, this little feedback link at the upper right.

Deborah Glueck:I’d also like to mention in a shameless plea for help, that we are currently resubmitting this grant as an RO1 to the National Institute of Health as a software continuation grant. And one of the ways we can demonstrate to the NIH that we are serving a useful purpose and that there is more work for us to do, are letters from researchers such as yourself. These are worth more to me than gold right now. So if you want to do a good deed for the day, write down what you want, put it in email on your letterhead and email it to me at Deborah.Gleuck@UCDenver.edu. It is up on your screen. Email it to me and I will send that letter in with the grant that I will be submitting if the government is refunded, on November 5. Next question please?

Arika Owens:Great. A couple more questions. Can power be estimated from within level interactions?

Deborah Glueck:Okay so the question is, can power be estimated from within level interaction? Dr. Logan and Dr. Kreidler are smiling proudly at me because yes, we can handle within level interaction without any trouble. And it will be glad to calculate and set up a hypothesis for complex within level interaction. Where in fact, Dr. Kreidler indicates that we can handle up to three levels of repeated measures. And actually, if you are willing or know how to type matrixes in, I think you can specify even more complicated designs using the matrix mode. So if you are – if you understand matrixes and want to go for it, you can construct your own hypothesis for as many as you want, okay? Next question please?

Arika Owens:What remedies perspective studies for when sample size recruitment does not meet sample size projection calculations?

Deborah Glueck:Okay, so what remedy is there for studies when sample size recruitment does not meet projected sample size requirements? This is a problem that we hit all of the time in trials, which is that you think you need 1,000, but you can’t get your 1,000. And I think I’d almost like to hand the phone to Dr. Logan who has much more experience in recruitment than I do, so let me hand it over to Dr. Logan.

Henrietta Logan:Thank you Dr. Glueck. You raise a very serious problem that is current in many, many, many clinical trials. I think that the thing we fail to do is to lay out a very, very clear recruitment plan. There are many of us that are working in communities and have learned a lot about how to recruit and how to retain, and how to build registries. Our experience is based on 10 years of recruiting minority, particularly males in rural North Florida. And what we have found is that you need to have a careful plan. You need to identify the stakeholders. You need to specify all of the kinds of events you are going to go to. You need to specify the efforts to retain individuals that sign in to registries. And to me, the recruitment plan needs to be as detailed as the power and sample size.

Deborah Glueck:Well said. I knew she was the right person to answer that question. I think we might – we have two minutes left so I think we might have time for one more question?

Arika Owens:I don’t see another question, but I see a last comment?

Deborah Glueck:Okay.

Arika Owens:This was immensely helpful, very clearly presented. Wish I had had this info before!

Deborah Glueck:Again, thanks for the comments. Anybody who wants to do a good deed for the day, would write that comment on your letterhead and send it to me at Deborah.Glueck@UCDenver.edu so we can try to get funded again in this competitive funding environment. Than you guys all very much. I have really enjoyed speaking to my invisible audience here, and look forward to doing this again. I notice from the poll that there seem to be roughly equal numbers from each one of our things, so I will negotiate with the VA cyber seminar people to see if we can give four more of these.

Moderator:Thank you to our speakers, Dr. Logan, Dr. Munjal, Dr. Glueck and Dr. Kreidler for developing and presenting this special topic seminar. For remaining questions, please contact Dr. Glueck or VIReC at virec@VA.gov and we will forward your questions to the presenters. Everyone have a great afternoon.

Operator:Thanks for joining us ladies and gentleman. I am going to close the meeting in just a second and please wait while the audience feedback survey pops up on your screen. This is the chance where you get to tell us what sessions you would like to see more of. So thank you so much to our presenters and to our attendees for joining us, and to Arika and VIReC for their sponsorship of this session. Have a wonderful day everyone.

Deborah Glueck:Thank you all.

 [End of audio]

Page 1 of 16