# Presentation: 263346

## Reporting GPRA Indicators  For the GRAA Initiative

**Wayne Harding, Ed.M., Ph.D.**
- Director of Projects
- Social Science Research & Evaluation, Inc.
- 21-C Cambridge Street
- Burlington, Massachusetts 01803
- 781-273-4206
- wharding@ssre.org

- Reporting GPRA Indicators  For the GRAA Initiative

***8******th****** National Technical Assistance Conference – Atlanta, GA – January 9-11, 2007***

## Reporting GPRA Indicators  For the GRAA Initiative

**Wayne Harding, Ed.M., Ph.D.**
- Director of Projects
- Social Science Research & Evaluation, Inc.
- 21-C Cambridge Street
- Burlington, Massachusetts 01803
- 781-273-4206
- wharding@ssre.org

- Reporting GPRA Indicators  For the GRAA Initiative

***8******th****** National Technical Assistance Conference – Atlanta, GA – January 9-11, 2007***

**Notes:**
Now the promised comment about how I expect many of you may respond to this presentation... most of you have seen the USED reporting forms and related information, most of you have been reporting GPRA measures for your project, most of you have attended presentations like this one in the past... Given these conditions,  I would guess many of you will be wondering why I am wasting your time by going over this material again.  The reason as Amalia and Phyllis indicated, is that USED is still not getting the GPRA data it needs to defend the efficacy of the GRAA initiative.  But you ask, why make me sit here when I have been providing the data as I should.  

I wish I had a great answer... There are at least 2 reasons.  One is that some of you who are confident that you have been reporting the GPRA data correctly are mistaken. The second is that we hope that those of you who do report it correctly will mentor those who have not. You likely have some wisdom to offer that may help others and us do a better job... if you do, please speak up and start to share it today.  If you are truly bored you can always entertain yourself by asking me questions you think I can’t answer. And that won’t be hard.

## Topics

- The meaning of the GPRA indicators
- Key features of the indicators
- Examples of acceptable measures
- Definitions of key terms
- A walk through the USED reporting form

**Notes:**
These are the main topics I will be covering. 

As you can see I am going to begin with some background information about the GPRA indicators and end with a tour of the USED reporting form.

## GPRA Indicators for GRAA1.

- 1. **The percentage of grantees** whose target students show a measurable decrease in binge drinking.
- 2. **The percentage of grantees** that show a measurable increase in the percentage of target students who believe that alcohol abuse is harmful to their health.
- 3. **The percentage of grantees** that show a measurable increase in the percentage of target students who disapprove of alcohol abuse.

**Notes:**
These, as you know, are the 3 GPRA indicators  the Office of Safe and Drug Free Schools at USED will use to report on the success of this initiative. That is why the language is about the percentage of grantees that show the change.  

You, therefore, have to make a small translation to apply these same indicators to your grant.

The next slide shows the translation for the first GPRA indicator.

## Translation of the GPRA Indicators

**The percentage of grantees** whose target students show a measurable decrease in binge drinking.
- Becomes. . .
- A decrease in the percentage of targeted students who binge drink.

**Notes:**

The percentage of grantees whose target students show a measurable decrease in binge drinking. 
Becomes. . .
A decrease in the percentage of targeted students who binge drink.

Questions?

We will come back to the translations in a moment.

##  Key Features of the GPRA Indicators 

- They must be reported annually.
- They should be reported promptly.
- The measures used at the grantee level must be comparable.
- The measures have to be taken at baseline and thereafter in order to assess change.

**Notes:**
You are required to report the GRPA measures to USED in each year of the grant.  This is because USED and other federal agencies are required by federal law to report GPRA measures annually to the Office of Management and Budget (OMB).

If grantees do not report the GPRA data in a timely fashion, the the Office of Safe and Drug Free Schools may be asked by OMB to explain why the data from some grantees are missing. Obviously, the Office of Safe and Drug Free schools would rather not do this.  

When you anticipate that you may not be able to provide the data on time, notify your FPO about this. Do not wait to notify them by failing to submit the data on the reporting form. Are there acceptable reasons for failing to report on time?  Yes there are. Hurricane Katrina, for example delayed the reporting of GPRA measures by grantees in Louisiana and Mississippi. A more common reason is that the annual reports are due in the spring of a grant year, but the data were collected later or have not been processed or analyzed, but will be available by September 1 at the close of the grant year. Whatever your situation, consult your Federal Project Officer about any anticipated  delay in providing the GPRA data.  

For new grantees, I strongly advise you to look at the GPRA reporting requirements now, and make sure you will have the data that is required, will have it on time, and will have sufficient time to prepare the report.

##  Key Features of the GPRA Indicators 

- They must be reported annually.
- They should be reported promptly.
- The measures used at the grantee level must be comparable.
- The measures have to be taken at baseline and thereafter in order to assess change.

**Notes:**
In order to assess the extent to which grantees change on any one of the three GPRA indicators, the measures grantees use have to be similar enough to justify combining the results across grantees.  

I want to look at this issue in more detail before moving on to the last feature on the slide.

GO TO NEXT SLIDE

## Comparability of MeasuresExamples of Acceptable Measures

**Binge Drinking:** Think back over the LAST TWO WEEKS. How many times have you had five or more drinks in a row? (There are 7 response options, from *0 occasions*, *1-2 occasions* up to *40 or more*.) 2.
**Perception of Harm (to Self):** How much do you think people risk harming themselves (physically or in other ways) if they take one or two drinks of alcohol nearly every day? (There are 4 response options from *no risk*,* slight risk*, *moderate risk*, to *great risk, *and an additional *can’t say/drug unfamiliar *option.) 2.
**Disapproval of Alcohol Abuse:**  How wrong do you think it is for someone your age to drink beer, wine, or hard liquor...regularly? (the response options are *very wrong*, *wrong,* *a little bit wrong*, *not wrong at all*.) 3.

**Notes:**
This slide provides three examples of measures for the  GPRA indicators.  For example, an acceptable measure for binge drinking is “Think back over the LAST TWO WEEKS. How many times have you had five or more drinks in a row?” (There are 7 response categories, from 0 occasions, 1-2 occasions up to 40 or more.) This item comes from Monitoring the Future.

Your FPO can provide you with a list of suggested measures. They are all drawn from either Monitoring the Future or Hawkins and Catalano’s work on measuring risk and protective factors.

USED recognizes, however, that there will be instances in which the suggested items do not work well.  For example, the pretest and posttest for a prevention program that are provided by its developer may not include these items.  In these cases, you must obtain approval to use an alternative item, or add the suggested item to the instrument you are using.  

One last point, you must hold the items you use constant over time. You can’t change the measures during the grant.  While the need to use the same measure over time is obvious to most of you, some grantees have not done so. 

##  Key Features of the GPRA Indicators 

- They must be reported annually.
- They should be reported promptly.
- The measures used at the grantee level must be comparable.
- The measures have to be taken at baseline and thereafter in order to assess change.

**Notes:**
The last feature deals with the timing of data collection.

The indicators talk about change: “a measurable decrease in binge drinking,” “a measurable increase in perceived harm,” and “a measurable increase in disapproval of alcohol abuse.” To assess change, you need to collect longitudinal data:  to implement a pretest or a baseline measure prior to the intervention you select to change these indicators and a posttest after it ends.  A simple posttest only design will not be enough to meet GPRA reporting requirements.

This means that if you conduct an intervention twice in a year with different groups, you will need to conduct two pretests and two posttests. You will, however, aggregate the data for the two groups when you report it on the ED 524B form.

There is an exception to the pretest/posttest requirement. Suppose  your intervention is designed to produce a population level change and you are assessing it by collecting data in an annual survey of high school students. In this case, you need a baseline survey before the intervention is implemented, but you need only conduct one annual survey thereafter.

##  Key Features of the GPRA Indicators 

- They must be reported annually.
- They should be reported promptly.
- The measures used at the grantee level must be comparable.
- The measures have to be taken at baseline and thereafter in order to assess change.

**Notes:**
Also, in many cases, to show positive change to will need some sort of comparison group. Think about the first measure – decreasing binge drinking.  The usual course of events is that as adolescents age they are more likely rather than less likely to engage in binge drinking, and the best some prevention programs produce is a slower rate of increase among those that receive the intervention than among those who do not.  If you are evaluating such an intervention, you will need to use a design that incorporates a comparison of some sort in order to show a relative decrease in use. 

Next we are going to walk through how to enter data into Section A of the ED 524B reporting form. Again, feel free to interrupt with questions or comments.

##  Section A: Sample Scenario

- 	The Springfield School District ‘s grant targets all 9th-12th graders.
- 	A baseline survey conducted the previous spring showed that 826 of all 3,368 students binge drank during the past two weeks.
- 	One year later the same survey showed that 754 of 3,432 students binge drank.

**Notes:**
The example we will be using is the same one as in your handout.  It focuses on the first GPRA indicator which is about binge drinking.  

The slide summarizes the essential facts.

The Springfield School District ‘s grant targets all 9th-12th graders. 

A baseline survey conducted the previous spring showed that 826 of all 3,368 9th-12th grader respondents binge drank during the past two weeks.

One year later, the same survey showed that 754 of 3432 respondents binge drank. 

## Slide 12

**Notes:**
Here is the the Blank form.  Filling in the descriptive information to for sections 1. 1a and 1b is straightforward.  NEXT SLIDE 

## Slide 13

**Notes:**
In Box 1 you enter the GRPA indicator (slightly reworded) as the Project Objective:  “To reduce the percent of target students who engage in binge drinking.” 

In row 1a you enter the performance measure; “The percentage of students who consumed more than 5 drinks of alcohol in the past 2 weeks – baseline.”  You might go further and note that this is from a baseline survey conducted in the spring of xxx... in the spring of 2004.

In the box labeled “Measure Type”  enter “GPRA.”

In row 1b you enter the same information except that the performance measure has additional information to indicate it is posttest data: ““The percentage of students who consumed  more than 5 drinks of alcohol in the past 2 weeks – year-end.”   Again I would go further an indicate that the data is from a posttest survey administered one year later. 

## 826

**826**

**826/ 3368**

**754**

**754/ 3432**

**Notes:**
*****WAYNE – Need to include narrative here to describe the addition of the numbers

## Section A:  More About Measures

- 	When using scales for GPRA 2 and 3, define how the item was computed in the Performance Measure block.
- 	E.g., “Percentage of students whose perception of harm increased as defined by combining responses to the *slight risk*, *moderate risk*, and *great risk* response categories for the question “How much do you think people risk harming themselves if they take one or two drinks of alcohol nearly every day?” (Responses to *can’t say/drug unfamiliar* are removed from the denominator, i.e. treated as missing values.)

**Notes:**
The procedures for completing the charts for the other two GPRA measures are much the same.  The one change to keep in mind is that you need to carefully specify how percentages are calculated when using scaled items.  

A few minutes ago, I gave an example of a USED approved item for assessing perception of harm. The question was  “How much do you think people risk harming themselves (physically or in other ways) if they take one or two drinks of alcohol nearly every day? There were 4 response categories from no risk, slight risk, moderate risk, to great risk, and can’t say/drug unfamiliar.) As shown on the slide, harm was defined by combining categories to three responses: slight risk, moderate risk, and great risk. 

## Section A:  Explanation of Progress

- Data collection methods.
- Describe progress toward meeting the performance measure. (If progress was not made, explain why.)
- How were the data used to monitor progress and refine project plans as needed.  How were the data analyzed?

**Notes:**
The space provided at the bottom of the Section A form should be used to provide additional detail about the data you post and about the meaning of the data.  The information should allow USED staff and other investigators to assess (at least in a general way) the quality of the data and the extent to which it demonstrates that your grant has made progress.  

USED’s instructions for Section A indicate that you should use it to report on the topics shown here:  data collection, methods, progress toward meeting the performance measure (in this case the GPRA indicators); when appropriate, an explanation of why progress was not made, and how the data may have been used to enhance the operation of the project.  I want to briefly review each of these.

## Section A:  Explanation of Progress–Data Collection Methods 

- What item(s) were used? (From what instrument?)
- When were the data collected?
- How were they collected?
- From what groups(s), population(s) were they collected? (If relevant, include the use of comparison/control groups. If relevant, report data separately for different intervention groups)
- How many cases are missing?
- How were the data analyzed? (If relevant, include  weighting and numbers for each response category.)
- Explain any departures from your approved evaluation plan.

**Notes:**
Again, the GPRA data the USED reports is simply the percentage of grantees that showed the expected changes in the three areas (binge drinking, perception of harm, and  disapproval of use).  The narrative section of the form, however, provides critical information about whether the data entered in the boxes is of sufficient quality to be reported, and whether it needs further interpretation to be reported. For example if there was an increase in binge drinking from baseline to posttests, this negative result may be qualified here by explaining that the increase was less than occurred in a comparison group.  

This list is based on USED instructions. It is mostly self-explanatory, but I would  like to clarify two items. The first concerns the 4th bullet, where it says “If relevant report data for separate groups.” On the form you are supposed to aggregate the data across groups that received the same intervention. So, if you delivered 2 waves of a prevention program to 6th graders during the reporting period, you add the data for them together when you fill-in the raw number and ratio boxes. Here, I am suggesting you disaggregate the data --- that you show the data for each wave.    THE NEXT SLIDE PROVIDES AN EXAMPLE

## Section A:  Explanation of Progress–Data Collection Methods 

- 	If relevant, report data separately for different intervention groups. E.g., report baseline binge drinking data from two cohorts as:

| Group |  Raw Number | Ratio |
| --- | --- | --- |
| Fall Cohort | 420 | 420 : 1768 |
| Spring Cohort | 406 | 406 : 1600 |
| Total Year | 826 | 826 : 3368 |

**Notes:**
The numbers in the last row are the same as those we looked at for binge drinking earlier:  At baseline 826 of 3368 students reported binge drinking during the past two weeks.  Here, we see, however, that this number consists of the aggregated data from the baseline measurement of two cohorts: one that received the intervention in the fall and the other in the spring.  

This sort of information can be very helpful in later cross-site analyses when one cohort does well and the other badly, and the difference is masked when the data are reported together. 

Now jumping back to the list of data collection issues I would also like to clarify one more.

## Section A:  Explanation of Progress–Data Collection Methods 

- What item(s) were used? (From what instrument?)
- When were the data collected?
- How were they collected? (Include sample size)
- From what groups(s), population(s) were they collected? (If relevant, include the use of comparison/control groups. If relevant, report data separately for different intervention groups)
- How many cases are missing?
- How were the data analyzed? (If relevant, include  numbers for each response category.)
- Explain any departures from your approved evaluation plan.

**Notes:**
The next to the last bullet asks that you discuss how the data were analyzed.  In parenthesis it also asks that if you collect the data in response categories that you report the raw data back by response category. 

THE NEXT SLIDE PROVIDES AN EXAMPLE

## Section A:  Explanation of Progress–Data Collection Methods 

- 	If relevant, include weighting and the numbers for each response category. E.g., report baseline binge data from the fall cohort as:

| 0 Occasions | 1348 |
| --- | --- |
| 1-2 Occasions |   350 |
| 3-5 Occasions |     35 |
| 6-9 Occasions |     20 |
| 10-19 Occasions |     10 |
| 20-39 Occasions |      5 |
| 40 or more Occasions |      0 |
| Total | 1768      |

**Notes:**
If you weighted the data in some way, this should be explained when describing the analysis.  The main focus of this slide, though, is on reporting the data by response category. 

We are using a portion of the same binge drinking data.  This is from the fall cohort.  There were 420 students out of 1768 who reported binge drinking.

Obvious, this sort of detail permits more analysis at the USED level across-grants and also provides more insight into the quality of the data.  For example you can examine whether the distribution looks reasonable in light of what we know from national or state level data. 

You are not explicitly required to do this, but it would be helpful if you did. And since you have the data on hand, this poses no significant additional burden... To save space you can simply attach the appropriate frequency distributions to the report.

## Section A:  Explanation of Progress– Describe Progress Toward Meeting The Performance Measure

- Discussion of how grantee has met or is making progress toward the measure.
- Include preliminary findings/data as they become available.
- If expected progress was not made, provide an explanation and plans for resolving the problem/issue.

**Notes:**
USED’s instructions for the “Explanation of Progress” box also ask that you provide the information shown here concerning the extent to which the project has achieved the performance measure – in this case a GPRA measure. 

Essentially, USED wants to know how much progress you have made, and if you have not progressed as expected why not and how you plan to remediate this situation in the future. You are also expected to provide all the preliminary data you have available at the time the report is filed.  For example, if you have baseline data on hand, and are waiting to collect or process  posttest data – report the baseline data now.  

## Section A:  Explanation of Progress– Discuss Use of Data for CQI

- 	Discuss how the data/findings are used to monitor progress and refine project plans when needed (consistent with the grantee’s approved objectives and scope of work).

**Notes:**
The last topic to be addressed in this section is how evaluation information is being used to promote CQI: continuous quality improvement.  This discussion might include how evaluation finding are shared with the project leadership and staff, and examples of how the data have shaped the project. 

## References

- U.S. Department of Education, Office of Safe and Drug-Free Schools. Overview Information Grants to Reduce Alcohol Abuse; Notice Inviting Applications for New Awards for Fiscal Year (FY) 2005.  Information and Application Procedures, FY 2005. Federal Register: February 9, 2005 Volume 70, Number 26: 6843-6846, February 2005. (see: Federal Register Online via GPO, www.wais.access.gpo.gov, accessed 2/2/06.
- Johnston, L.D., O’Malley, L.D., & Bachman, J.G.  (2001).  Monitoring the Future national survey results on drug use, 1975-2001: Volume 1, Secondary School Students 2000 (NIH Publication NO. 01-4924) Bethesda, MD: National Institute on Drug Abuse.
- Arthur, M.W., Hawkins, J.D., Pollard, J.A., Catalano, R.F., & Baglioni, A.J.  (2001). Measuring risk and protective factors for substance use, delinquency, and other adolescent problem behaviors: The Communities That Care Youth Survey.