**Transcript of Cyberseminar**

**Research Design**

**HERC Econometrics Course**

**Presenter: Jean Yoon**

**July 11, 2012**

Presenter:Today I will be talking about some basic issues in research design.  So this is the outline for today's lecture.  So first I will be going over some issues regarding causality and study design.  Then I will spend some time talking about several different quasi experimental methods for observational studies.  

I will be talking about sample selection models, differences in-differences, and regression-discontinuity.  Well, these are a little bit different than what was originally advertised that had replaced [inaud].  At first of all I will talk about causality and study design.  

So, why is causality important in help to this research because we often want to be able to understand the impact of implementing a new program or intervention, so it's not enough to say that a program or treatment was associated with better outcome, that we want to know whether that program or treatment lead to the better outcomes.  

So ideally to estimate this causal effect we would want to be able to compare outcome under the counterfactual.  The two main effects would be one we observe the outcome why when the patient gets the treatment, when two is equal to one.  And then we also observed the outcome when the same patient does not get the treatment when two is equal to zero.  

That difference in outcomes would then be the impact of the treatment, but of course the reality we don’t observe the same patients with and without treatment.  So randomized experiments come close to this ideal, basically in the randomized experiment we have two groups, and we both to randomized, and one group gets randomized for treatment which is the top row.  And then we observe outcomes of the two groups.  

And then we compare the outcomes between the treated and untreated groups, in order to get the impact of the treatment, and because the treatment was randomized there are very little possibility if they were systematic differences between these treated and untreated groups, and so any differences that we observed in these outcomes can be attributed the casual effects of the treatment.  

So of course in health based research we don’t see any randomized trials.  Most of the time our studies are observational and usually cross sectional, and it can be difficult to show causality because there is often confounding which is also referred to sometimes a selection or endogeneity, and the reasons for confounding can be because of these reasons like omitted variable bias which is when you have important variables that are not measured.  

These variables are both related to your treatment and to your outcomes.  So when you don’t include those measures you can cause bias in your treatment effect.  It is also a selection which is when because treatment is randomized, patients are selecting in the treatment based on unobservable factors, and something else that can cause confounding as reverse causality.

So you want the treatment -- it's just the treatment that has an impact on outcomes like health status, but it could be that health status actually lead to a patient getting the treatment or not.  And another problem that can cause bias in studies is when there is a measurement error, and this is when there is a consistent, or systematic problems in how variables are measured.

So let's talk a little bit about hypothetical observational study, so let's say that we are doing a study and we want to look at patients SIVDs.  These patients are seen at primary care clinics, and some of them volunteered to participate in a phone-based management programs and some patients don’t.  So in order to understand whether or not this program worked we compare outcomes for every single patient at the end of the program.  

We compare things like there a hemoglobin A1c levels, their cholesterol, their health outcome.  So if we find in our results that patients who participated in the program has better outcomes and those who didn’t, can we really concluded the program caused better outcome?  So as I just explained some of the reasons why that could be confounding another problem.  You can probably think of reasons why this difference in outcomes may be attributed to something else.  

So I would like to hear from you now if you could use your Q&A panel, if you could just maybe give some examples of what other factors could have lead the program participants to have  had better outcomes than the non-participants?.  So once you type this in, Patsy, could you read the responses?

Patsy: I don’t see any of them.  so there is one, so I should bias those more interested volunteers.  Are there other responses to the question?

Moderator: Patsy, we've got several responses in there, I am not sure if you have your Q&A panel expanded all the way. The ones we have in here they are more aware to begin with Hawthorne effects, regularly encourage to pay more attention to their intake in sugars, better compliance.  It could be that they started living a healthier lifestyle because they were in the study. 

Presenter: Okay, great.  Yeah so those are all possible reasons why other than enrolling in the program they might have had better outcomes than the other patients.  And may be there are some of the -- there are some of the examples that you gave so one reason it could be that the characteristics were not balanced.  

It can be through with the patient you know once who were in the program, and those who weren't.  So it could be that those who enrolled in the program had better outcomes to start with.  If we just compare their outcomes at the end, it looks like the program improved their health outcome.  Someone mentioned that the patients may have selected in the treatment, so it could have been that those who are enrolled in the program were just motivated to improve their health overall, and that whether or not they enrolled in the program they would have had better health.  

There could have also been changes overtime that were unrelated to the intervention.  So for example if this program were implemented at a time when the recession began, and people lost their jobs.  And their health status became worse as a result of losing their job.  This can confound I mean so many effect that you see.  It's also could have been that the enrolled patients engaged in other activities that were not part of the program.  

So for example that could have been getting a lot of diabetes information online, and learning how to better care for themselves, so it wasn’t the program itself, it was repeating this information, and being better educated about their healthcare.  So you can see there, it can be many alternative explanations for outcome differences.  And then makes it difficult to [determine] the impact of the program.  

So there are several ways of dealing with these files as I just talked about.  So these observations do not have randomized experiments.  We need to use methods that make studies like an experimental study.  And we do that by identifying a similar control group.  We also try to eliminate any systematic differences in treatment and control groups.  And then we can compare these outcomes or the changing outcomes between two and four groups.  

The first model is I will talk about is sample selection.  So a sample selection occurs when there is bias because the outcome is not observable for some people.  It's just called sample selection.  This is a form of omitted variable bias because the reasons why we might not have data is not observed.  

So an example of this is you have some sort of study where you are collecting data from patients, and so there are patients who enrolled in the study to begin with.  And then you ask the patients to come back to see a nurse, and have their clinical measures collected this nurse visit.  But some patients drop out from the time that they enrolled in the study and the time of those nurse visit, so you don’t have data for everyone.  

So as a result of this, this is impossible to make inferences about determinents of outcomes, and the study population as a whole.  You can only know what was happening in the patients who actually showed up, and had their data collected.  And the bias of the problem is the reasons why patients don't respond are correlated with an observable factors that influence outcomes, and this is an identification problem.  

So an example of this would be like if particular patients were more motivated to stay in the study, and go in for this nurse visit, and have their data collected.  So then you can see that are we are actually measuring health outcomes so that we will cause bias in any treatment effect estimates.  So a solution to this sample selection problem is this Heckman model which is sometimes called Heckman correction probit, or it's also called Heckit. 

And so the basic setup is that there are two equations.  So the first stage use the probit model to the dichotomous outcome so you are predicting whether or not somebody showed up to the nurse visit, whether they participated in the study, start section equation.  And everyone is represented in this equation.  And then from the parameters in this model we calculate the inverse Mills ratio which is a selection hazard for each patient.  

And then we then add this inverse Mills ratio as a variable into our second stage.  So in our second stage we are predicting health outcomes.  And we use the traditional model; we use an OLS model for a linear outcome.  And the people in this model, the observations in this model are the only ones who had their health outcomes reported, so this not it's going to be like your whole sample. 

And the basic assumption that this model makes is that the error terms of the two equations are jointly normally distributed, and they are correlated by some factor row.  So this is just what I just described so you have two equations, so the first equation is a select equation and then the second equation is your outcome equation.

So and the first equation [d] is whether the patent participated or not, and in the second equation [w*] is the health outcome and we only observe this when [d=1] when the patient participated in the nurse visit, so we can run the equation first.  We can estimate patient wanted first, and we use the parameters to the model to get this inverse Mills ratio.  

It's what it is, its just the ratio of the probability distribution functions divided by the cumulative distribution functions.  And then we then take this inverse Mills ratio, and we add it as a parameter into our second equation that predicts health outcome.  So essentially what we are doing is we are estimating our expected count outcome, and we are counting for the fact that not all people had their health outcomes observed, and we are doing that with this parameter lambda.

So we also... the coefficient of lambda is rho, and so we can test whether or not rho=0, and if it's equal to 0 then we can say that there is no sample selection.  If there is no sample selection then you don’t need to use these models we can then go back to your original OLS model.  

We also want instruments that are included in our selection equation that are part of Z that are not related to our health outcomes w.  And the reason why we are doing is because lambda is actually a non-linear function and so we can't be sure that lambda is picking up deflection effects 

without having these instruments in our selection. 

So there are extensions to this Strischter model that just talks about so if the outcome is dichotomous rather than continuous we can use probit model in the outcomes equation and you may have data for everyone, and what I just talked about when you don’t have data for everyone, but if you have data for all patients you can use a treatment effect model.  This is sort of based on the same idea that the ideal why some patients get treated with one treatment versus another is based on unobservable factor.

And there are newer semi parametric models that don’t make this sort of normality functions, so they are less restrictive.  So the strengths of the sample selection model is that we can generalize the population of interest and it address this selection that's based on unobservables.  

Some of the weaknesses of this model is that you know make strong assumptions, and assume strong normality of the error with two equations.  You can still run it if you don’t have good instruments.  This is different from models like [inaud] but then you are relying on distributional functions of the inverse Mills ratio that you can't be sure that lambda is picking up deflection unless you have these instructions, and it can often be difficult to find good instruments that affects selection, but not health outcomes. 

This is just some Stata code, so it runs.  It has been modeled in Stata. And now I just want to talk a little bit about an example of this is the sample selection. 

Moderator:  Jean can you hear me?

Presenter:  Yeah.

Moderator:  We have a question right here.  And the question is for Heckman selection model is there a preference in favor of the two way step OLS versus maximum livelihood estimation?

Presenter:  I think there are sometimes problems with maximum livelihood in that, sometimes the model doesn’t converged and you won't get estimates.  You can use either method basically. 

Moderator:  Okay.

Moderator:  Okay so going back to the selection example so there was a study that was published in the health economics in 2009 and so the authors looked at the effects of rural mutual health care  in outpatient service utilization in tiny villages.

So what they are doing in the study was this tiny villages got it implemented a new health insurance program, so they were offering insurance that covered in outpatient care, and there was also as part of that insurance program a drug policy which was meant to restrict drug prescribing by offering some incentive to doctors, and so they wanted to look at this effect on outpatient medical expenditures.

And the reason why they were using of sample selection models because only some residents had outpatient visits.  There is just a table from their paper, so just look at the first column where it says outpatients visits  per person you can see that the rate of for the number of visits per person is pretty low 0.156 for the insured and 0.098 for the uninsured.  

So there were a lot of people who did not have any outpatient's visit.  So this is after they ran their Heckman selection model these are their results.  You can see in the first, in the second column where there is first stage that looks at probability of any outpatient use, and they had three instruments if you look at towards the bottom which is illness last month, chronic disease existence, so these are variables that they use to predict any outpatient use, or not used in the second stage for the outpatient expenditures. 

So they then estimate the second stage could be here, and it looks like for the outpatient coinsurance rate and the drug policy would had a significant effect on reducing outpatient expenditures.  So it was just example of the sample selection study.  So now, I just wanted to get an idea of whether you understood what I -- the model I just explained and so I just wanted to do a short real poll here. 

So I will give you a minute to respond to that.  

You've been seeing the results we are at above.

Moderator:I can see the results here.  Right now we are at about 45% and started to come in.  I will give it a few more seconds for people to respond and I will close it out here. 

Moderator:Okay so it looks like most of you understood that sample selection model eliminate bias due to unobservable characteristics, so the reason why its not only to observable characteristics is that if you could control, if you already are able to observe the characteristics about why patients get certain treatments you don’t need to use this sample selection model.  You just go back to your regular probit or your regular OLS model. 

Sample selection is meant to deal with unobservable bias.  These are unobservable, and hopefully that clears it up.  Any other further questions, feel free to use the Q&A panel.  

So the next model that I want to talk about is differences in differences.  You can use differences in-differences really only when we had some sort of natural experiments and we also use it when we have [inaud] data or we observe the outcome at different time point for treatment and control groups.

So what this model does is we subtract out differences between treatment and control groups, and differences overtime.  So this is sort of a class of representation of differences and differences, so you can see we have two groups.  The top group is the treatment group, and the bottom group is the control group.  And we observed their outcomes at two points at times.

That's a pre-period and then a post-period, so they can see for the treatment groups their outcomes went from increased from A to B and the control group also.  Their outcomes have also improved from C to D, but the slope of A to B is steeper than C to D so we know that they had a bigger increase, and that difference we attribute to the treatment set.

The difference is just b minus a, minus b, minus c.  So when we have... and to a set a regression equation so let's say we have data on patients, and we have two groups control and treatment groups.  And then we have data on the pre-period and post-periods.  And so we don’t necessarily used to have a data on the same patients.  We just need to have a data at the two points in time for just a group.

We have a program p and it's equal to 0 when patients didn’t get the program and one when they did they were very involved in this program.  We have time c and its 0, and in a pre-period, and one in the post-period, so in our regression equation so y is the outcome, and then data 3 is are the estimate that we care about.  This are the differences, different estimate.  

The data one is just effect of time.  Data two is whether or not you were in the program group or not.  The data three is the interaction piece program in time.  So it's just a change in outcome of the people in the programs minus the change in outcomes for people not in the program.  

The strengths of this model is that we can difference out the time trends, and we can also adjust a omitted-variable bias.  We have unmeasured time and variant factor.  Weaknesses on that, is there are unobservable factors that change over time, some still cause class estimate.  So when we have panel data, we can get rid of omitted-variable bias with these differences, in the differences model.  So, we can include a parameter, so I would hear a sigma i, as all the time in variant characteristics for an individual just both observed and unobserved.  

So, some things that are time  invariant characteristics would be things like gender that’s not going to change overtime.  So we have our -- we could setup out regression equation with the outcome so its YIT, so it's for individual I at time T.  And then we have data 1 times time, data 2 times the program and then sigma i is just the time invariant characteristics.  

So we can setup the model when time is equal to one in the post-period, and then set it up when time is equal to zero within the pre-period.  And then we just difference the model.  This attracts yi1-yi0.  This is change in outcomes. 

When we do this data one is just a time trend.  Data two is a parameter that we care about.  This is our treatment effect.  As we see when we difference the model sigma i drops out of the model so it may have had some impact on y on outcome because it didn't change over time.  

It got dropped in the model so it's no longer going to cause any bias.  So the fixed effects estimate although within estimator is the same.  Its the work differencing two times a year.  So we cannot estimate the effect of time variant factors in the fixed effect model.  

This data, it does have a program, it has xtreg which can run fixed effects.  That is the only the first differencing with two time periods new as fixed effects data.  So the example that I will talk about is the study  that looks out the impact of decreasing copayments for drugs on medication adherence.  

So there were two different health plans, and they both implemented to this management program.  Going to one of these health plans introduced a drug copayment.  So they wanted to look at drug adherence with two groups of patients, and they compared it in the pre and post period were implementation of disease management program, and also the reduction in copays, so there are two sets of lines.  

So the top of the line are also the control groups, and the bottom of the lines are the group prevention.  And so what they are measuring here is drug adherence and they think that there was a seasonal difference in drug adherence, so they did it by quarter.  So we can see in the top two lines that there is not really much difference between the control pre and post period virtually the same.  

You can see for the intervention period.  Intervention group to this bottom two lines that the post period had a higher adherence then the pre-period.  So that difference is then attributed is that to reduce drug cost sharing, so I just had another poll about differences in-differences.   So if you can put up the next poll.

Moderator:Jean there is a question after the poll.

Presenter: Okay sure.  What is it?

Moderator:The first question -- oops they are flying.  How did you deal with the relative treatment effect using panel data?  For example they are only two groups, one will receive treatment one and another one will receive treatment two.  There is no real control group that received no treatment.  How should we estimate the treatment effect relative using difference in difference that we have a panel?

Presenter: Oh sorry, can you read the first part of the question, I didn’t get it?

Moderator:The question is how to deal with a relative treatment effect meaning between two treatment groups rather than a treatment comparison to no treatment.  That’s the question.  How should we estimate the relative treatment effect using differences in differences that we have panel data?

Presenter: Okay so we can't do that with -- it would be difficult to do that with panel data for just when we have multiple observations to the same patients, but you can use it when you have observations for sort of people.  So what you need are two groups of patients.  You need a group of patients who got one treatment and another group who got the second treatment.  And then you need to know what to be able to observe their outcomes before they got the treatment, and then after the treatment.  And then you can then set it up --

Moderator:Right but does it --

Moderator: So then you setup your regression, you have outcomes, and then you have time, the parameter in your model, and then you have treatments, the another parameter on your model for treatment A or treatment B, so you can still difference out in trying to having the differences. 

Moderator:Right so really there is no difference between using the differences in differences model between an intervention, and no intervention, or between doing interventions.

Presenter: Yeah that's correct, yeah.  Right, yeah.

Moderator:And the next question is in the difference model why is the error term epsilon1 minus epsilon 2 it seems that there would also be a non-zero covariant between the two.

Moderator: Right.  Let me go back to the slide then.  So, I just wanted to point out that with the poll so the answer is actually both A and B.  So difference in differences model eliminates confounding due to tine trends and it can also eliminate some of the variant bias of time and variant factors.  So I will go back to my slide so I can answer that question. So the question was is there a non-zero expectation?

Moderator:A non-zero covariant between the two.

Presenter: Okay I'm not quite sure how to answer that.  I mean you have CIT which is the error.

Moderator:Jean we are having trouble hearing you.

Presenter: Okay so you have the error for individual I at time C, so you are just differencing, differencing the error, so I'm not quite sure why there would be a bias to that.

Moderator:So the question is it seems there would be a non-zero covariance between the two rather than the simple 1-0.

Moderator: Okay, I'm not quite sure I understand the question, so I mean they are allowed to be correlated because they are for the same individuals, but you are differencing out the error in those two models, so I will just think about a little bit more, and trying to think of a better explanation.

Presenter:Jean, we can't hear you.  I'm not sure if --

Moderator: Okay.

Presenter:There is something happening with your headset, or are you VOIP?

Moderator: Yeah I'm on the phone.  I will try to speak a little louder.  I'm not quite sure how to answer that question right now.  I will have to think about a better explanation, and try to bring it up at the end of the lecture.  Was there another question ?

Presenter:Yes there are.

Moderator: Okay.

Moderator:Bias due to time varying variables in the differences in-differences model.  Does the time variable, omitted-variable have to be related with as the treatment to bias the estimates?

Presenter: It is to what?  Is it time variant --?

ModeratorDoes time varying or method variable have to be correlated with the treatment to bias estimates?

Presenter: Yes.  I mean if it's not related to your treatment then there is no reason why it would cause a omitted-variable bias.  The omitted-variable bias is when you have unmeasured factors that influence.  So it's the outcome in the treatment that you are looking at.

Moderator:Okay, and the next -- I can hear her fine.  I can hear Jean okay. 

Presenter: Okay so --

Moderator:I'm getting chopping this here.  So, we have a question just to let folks know that it -- I had a couple of request for SAS code instead of Stata code, and just a follow up to any answer that we have just given in, and what I'm doing is sending you either Jean or the her consulting desk information so that you can follow up with us.

We don’t have SAS code.  If the presentation doesn't include the code we can't give it to you right now, but we might be able to give it to you later.  So I'm just asking you to please follow up with us after the session, and we will let you know if we have the code we would be able to share with you.

Presenter: Okay thanks so just in the interest of the time I will just continue, and then if there are still remaining questions, please ask me later.  Okay so next model is the regressions of continuity.

And this is a model that we do when treatment is not randomly assigned, but it's based on some continuous factor called V.  V is sometimes called the force in variable.  And so use this to give us their, this continuity as some cut-off value of V for everyone below V would get a treatment or be enrolled in a program, and everyone up above the cut-off value would not be able to enroll.  

So the assumption of model makes us that individuals cannot manipulate the assignment of V.  And the only jump in outcomes that we have observed was due to this continuity of treatment.  So the cumulative effect its just expected outcome for units or observations just above the cut off minus the expected outcomes for units just below the cut off.  

We assume that they are otherwise identical.  So the strengths of that we can have a direct impact on the outcomes which is unlike models like instrumental variables. and the weaknesses so that you do need to test the functional forms, the effect of the treatment by using things like interactions, and quadratic term, or you can get bias to an effect if the model is mis-specified, so make this model, a little more clear I will talk about this paper, the impact of the medical insurance for the poor in Georgia.

So the study looks that the effect of medical insurance programs for poor residents in Republic of Georgia, and they looked at this impact on healthcare utilization.  And so ultimately so the program was limited to residents who were below the mean test score which they call this SFA. 

So the residents below the SFA were eligible and the residents above this SFA score were not eligible for the program.  So they then compared the outcomes of these two group of patients.  So there were two programs in two parts of the country, so one use this 70,000 threshold of this mean test scores, and the other use this 100,000 threshold, and so what this shows is the share of household enrolled in this insurance program.  

You can see that if you are eligible, if you are below that SFA score, almost a 100% of people were enrolled in the program.  And if you look at the right, the residents were above this threshold and it's almost a 0% were enrolled in this insurance program.  

You can see this short list continuity.  So in this study what they did was like look at the impact of this insurance program, what is just MIP, just impact and health outcomes so the main estimate of interest is data one which is a treatment effect and it's due to the discontinuous change at the cut off value of the main test score.  

Data two measures the independent effects of the mean tests and health outcomes.  And then a good interaction, so they felt that some might be an interacting mean stuff for people who are enrolled in the program.  And this is just to show you what their main results were, if you can look in the far right column, if you look at outpatients or out of  pocket expenditures for healthcare.  And so if you were enrolled in this program, if you were an MIP beneficiary you had 0.5 expenditures compared to those who weren’t enrolled.

So, they had much lower out of pocket expenses than the people who were not enrolled in the insurance program.  So that was a brief overview of regression discontinuity.  Before we do the call, I guess are there any questions so I can answer. 

Presenter:No.  I'm just glad for the questions you have answered [inaud.]. 

Moderator: Okay, so here is the quick poll about a brief regression discontinuity, so I will give a couple of minutes to respond to this. 

Presenter:Is it really that nobody is responding or I just don’t see it.

Moderator:Sorry, Patsi. 

Presenter: I don’t see any response.

Moderator:No, the responses are coming in but they are not -- no, the responses are not dynamic Patsi,  like we have to wait until everyone's responded and then I will show it. 

Presenter: Great.

Moderator:Yeah so 85% of you are right that regression discontinuity can be used when a treatment is designed based on cut off of the continuous score as saw an example of that, mean test score which made you eligible for the insurance program.  

Great, so I have -- I know I did a very broad overview of these three different methods, but mostly what I want to talk about that I did have a final poll for you, so you can put up the poll four.  This is just I asked you about quasi experimental methods in general.  

So yes you are right, the quasi experimental methods can address the bias in the observational study.  

They can reduce systematic differences between groups.  And it can also provide separate designs or to make inferences about causality, so it is all of the above.  So, basically with an overview of several different methods that you can use. There are other lectures in this HERC course series that will talk about even more methods that you can use that basically do the same thing that can we really reduce sources of bias.  

And reduce systematic differences which we make in four group.  But today I just wanted to give you an overview of several different ones, so here are some references.  If you are interested in learning more about them there is a book that looks at experimental and quasi experimental designs research which is student learning more about sample selection model, the original paper was in 1979, a Heckman which has a book on econometric analysis of cost sectional and panel data. There’s a websites that has research methods  knowledge base.  

It has a good description of several different research methods.  And these are the preferences for the studies that I've talked about for example.  So the next two lectures will be July 25th on propensity scores, and August 8th which will be Patsi talking about instrumental variables. 

So that’s all I had to say, so if there are any additional question you can answer them, and you could type them into your Q&A panel and Patsi will read them off.

Alright we don’t have anything.  

Presenter:Good overview is a comment.  And the question from earlier was time varying variables in the differences, indifferences. Does the time varying method variable have to be correlated with the treatment to bias the estimates.

Moderator: You have to be correlated with the treatment, is that the question?

Presenter:It has to do with the daily model, and the question is about the omitted variable.  That's the time varying -- 

Moderator:Yeah so there is some important time in varying variables that you are not measuring that you think it's related to those achievement on to outcome the differencesin-differences model is not going to get rid of that bias, so that will still be a problem.  So only when you have time invariant factor that you think can cause omitted-variable bias, but it's that problem that differences between this model can address.

Presenter: It's about the questioner is saying the question that Jean was to think about how to do with the error term.  The original question was bias due to time varying variables and differences, in-differences model thus the time varying omitted-variable have to be correlated with the treatment to bias the estimate, and I have no and or --

Moderator:Okay well I mean, I think I answered that the best I can.  Feel free to shoot me an email whoever asked that question if its still not clear you.  Are there are any other question?  

Presenter: So not so far.

Moderator:Okay.

Presenter: Thanks very much.

Moderator:Heidi, is there an evaluation that people are asked to fill out?

Presenter: The evaluation actually pops up when people leaves the meetings so as they click to exit, click the X in the upper right hand corner to exit the meeting that should pop up on your screen.  We really appreciate if you guys to take just a few moments to fill that out.  We really do read to your comments, and use them in our current and future planning on cyber seminars.  We want to thank everyone for joining us for today's HERC Econometrics session.  And we hope you are able to join us our for a Todd session on the 25th.  We will be sending registration information out on that to everyone shortly.  Thank you for joining us today.

[End of Audio]

Page 1 of 13