# Presentation: 678288

## Session #65

- Draft Results of Quality Assurance Program Data from Award Year 2006-07
- David Rhodes and Anne Tuccillo

**Notes:**
Try to check with people to see if they have brought their institutional profile before the session begins. Give them theirs if they forgot.
Welcome audience
Remind attendees that this a session in the QA track – but anybody else is more than welcome 

## Goals

- Share draft results of program-wide analysis of 2006-07 data
- Illustrate additional ways to analyze ISIR data

**Notes:**
Our analysis of the data all QA schools entered into the ISIR Analysis Tool last year.
Additional ways include comparisons across years, looking at verified cases where aid eligibility is unchanged, and exporting data outside of the tool

## Background

- Quality Assurance (QA) Program
- Participants granted regulatory flexibility to decide which ISIR data they verify
- ISIR Analysis Tool

**Notes:**
Provide a very brief history of the QA Program. The Tool was developed to allow QA Participants to monitor the effectiveness of their own tailored verification procedures.

The Tool is now available to all Title IV participants.

## The 2006-07 Analysis

- 146 Quality Assurance Program institutions
- Each school drew a random sample of at least 350 applicants
- Each school verified the ISIR information for the entire random sample of applicants
- We analyzed 68,077 records

**Notes:**
Explain the 2006-07 data – it is sample of ALL initial applicant transactions and examines ALL changes to ISIR fields that occur after the first valid ISIR. 

## Previous data collections	

|  | 2005-06 | 2004-05 |
| --- | --- | --- |
| Population | ISIR Records subject to school verification | Random sample of all applicants |
| Number of schools | 140 | 133 |

**Notes:**
Describe previous data collection effort during the previous two years.  Note that the differences in the number of schools included reflect  new schools joining the program, a few leaving, and data quality checks that exclude a few schools from the program-wide analysis each year.

## See a pattern?	

- Alternating focus
- Random sample of all applicants
- Institutionally verified applicants
- This year (2007-08)
- All QA Program institutions
- ISIRS subject to institutional verification

**Notes:**
Ask the audience if they see the pattern.
Every other year QA schools analyze a random sample of all applicants to look at ALL potential corrections in their aid populations. In the alternative years they analyze the subset of corrections that their current institutional verification procedures are finding. 

## Is analysis over time appropriate?

- For individual schools, yes with care
- For program-wide analyses the answer “should” be no, but.....

**Notes:**
Ask audience for their opinion.

Read bullets and explain that schools can with their own data if they keep some things in mind.  Program-wide shouldn’t but ...

Ask: 
In which population – a sample of all applicants or the records the QA schools choose to verify would you expect to see more changes? 

Answer: school verification – schools are trying to focus verification on applications where it will make a difference to aid awards

## When Looking Across Years Keep in Mind:

- The two different populations
- Changes to institutional verification criteria between years
- Other differences between award years
- Percents and averages not counts

**Notes:**
What schools need to keep in.

Random sample years have ALL potential corrections
Institutionally verified records have all detected corrections

Also need to keep in mind any changes schools make to their verification criteria or other changes (e.g. increase in Pell maximum).  Why would an increase in Pell matter? (answer expands the population at risk for an improper Pell payment)

The number of records in the random samples will generally be considerably smaller than the number of institutionally verified records. You need to make data comparable by using summary statistics (averages, percentages) not raw counts.

## Why we “shouldn’t” 

- QA schools supplying data differ slightly from year to year (146 ≠140 ≠133)
- Unmeasured differences in institutional verification across years

**Notes:**
Our program-wide population changes a few members every year

We don’t keep track of how institutional verification criteria change year to year so comparisons across years are strained.

## Key areas of analysis

- Description of population
- Critical ISIR fields
- How changes affect aid eligibility
- Improper payments in the Pell Grant program

**Notes:**
Read and elaborate on bullets

Simple descriptions of the population are important to understanding the records you are analyzing

As we have in the past we will identify “critical” ISIR fields, that is, the fields most likely to change

We’ll examine the strength of the relationship between a change in each of these critical fields and a change in EFC and Pell

Finally we will look at magnitude of the dollars POTENTIALLY at risk in the Pell Program last year. I say potential because the thought exercise we will do compares Pell awards made on the basis of the initial ISIR transaction and compares that to awards made after all records are verified. All actual awards in the sample were made on the correct verified ISIR information.  

## Characteristics of Aid Applicants at QA Program Schools 2006-07

**Notes:**
Read the chart explaining that in this case we also know the complimentary percentages – that is what percentage of the sample were independent students? (38% - 100-62)

## Dependent and independent record counts over time

**Notes:**
Ask audience: What is wrong with this picture?

Answer: It uses counts and not percentages.  

Ask audience: How would you describe changes in the relative number of dependent and independent students over time?

Answer/Point: It is hard to do. You either end up talking about 1000s of records, trying to eyeball the relative height of the blue and orange in each column, or “astutely” observe that in sample years there are fewer records. 

## Dependent and independent percentages over time

**Notes:**
Percentages make comparisons easy.

Reminder 2006-07 and 2004-05 are sample years – all applicants
2005-06 is a school verification year.

Ask audience: Is there a higher or lower percentage of dependent students in the institutional verified subset of records in 2005-06 compared to a random sample of all applicants in 2006-07?  (Higher)

Ask: What does that mean? – Are dependent student more or less likely to be verified by QA institutions than independent students? (more)
Simply by comparing basic descriptive statistics of your institutionally verified records one year to sample data in the adjacent year you can begin to see what types of students your verification procedures are focusing on. 

## Percentage of dependent applicants with changes to the most commonly changed ISIR Fields: 2006-07

**Notes:**
Here are the five ISIR fields most likely to experience a change after the initial transaction.

Note that roughly one quarter of the records experience a change to each – and that means three quarters do not

## Percentage of independent applicants with changes to the most commonly changed ISIR Fields: 2006-07

**Notes:**
Critical fields for independent students. Note that the incidence of change is even more rare among independent students.

The fact that change to critical fields are less common among independent records is probably a big reason why QA school’s verification focuses disproportionately on dependent students.

## Changes to critical ISIR fields among dependent students over time

**Notes:**
How does this look over time – when we compare all applicants to the school verified records (2005-06)

Similar results for the other fields but just showing two fields for clarity

No real difference 

What does this mean? – Program-wide school verification is NOT selecting dependent records that are more likely to experience a change to a critical field

## Changes to critical fields among independent students over time

**Notes:**
Same comparison for independent students

Ask: Do you see any difference here?  
Answer: The 25% of independent records from 2005-06 (school verified) with a change to AGI is greater than the 18% among all applicants in 2006-07. 

What does this mean? – 
School verification is having some success targeting independent students who are more likely to have a change to AGI – But we can’t say the same for Worksheet B

## Dependent records:  percent of ISIR fields experiencing an EFC change - 2006-07

**Notes:**
Explain that these percentages reflect the proportion of records that experience an EFC change when the indicated ISIR field changes

So what happens most of the time when a critical ISIR field changes? – EFC changes.  
How large is “most?” - Roughly 90 percent

## Independent records: Percent of ISIR fields experiencing an EFC change - 2006-07

**Notes:**
Similar finding for independent students only difference is that “most” is now 70 rather 90 percent.

## Comparison of changes to EFC among dependent students with a change to the indicated ISIR field

**Notes:**
Virtually the same – so school verification is not only not selecting dependent cases that are more likely to experience a change to a critical field but when a change occurs they are just as likely to result in an EFC change

## Comparison of changes to EFC among independent students with a change to the indicated ISIR field

**Notes:**
Same thing for independent students

## Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07

**Notes:**
Now lets look at the proportion of records that experience a change to a critical field and a change to a Pell award.

Note that these percentages are roughly half the EFC percentages 

Ask:
Any one want to explain why?

Answers not all applicants are eligible for Pell and Pell cells allow for the possibility that a minor changes in EFC will not move an applicant out of their initial Pell “cell.” 

## Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07

**Notes:**
Similar results for independent students.

## Comparison of change to a Pell Grant among dependent students with a change to the indicated field

**Notes:**
Double barrel question:
Do these results suggest that QA school verification is effectively targeting changes to Pell grants among dependent student? Why or why not?

Answer
Yes – the percentage of concurrent Pell changes is notably higher in the school verification record year (2005-06) than in either sample year 

## Comparison of change to Pell Grants among independent students with a change to the indicated field

**Notes:**
Same thing for independent students

## Average EFC change among dependent records with a change to the indicated ISIR field: 2006-07

**Notes:**
Here are the average changes to EFC among the dependent records. This average was calculated by summing up the magnitude of all the EFC changes observed among all the records with an indicated change and then dividing by the number of records experiencing that change. – If necessary explain how negative numbers reduce this number. 

Not that the range of EFC changes

## Average EFC change among independent records with a change to the indicated ISIR field: 2006-07

**Notes:**
Here are the average EFC change for dependent students with a correction to the indicated field. Note there is the most change associated with filing status

Note that + and – changes in income from work and student AGI account for the low numbers. 

## Average EFC change among dependent students with a change to the indicated field over time

**Notes:**
Second double barrel question:
Do these results suggest that QA school verification is effectively targeting the larger EFC change among dependent student? Why or why not?

Answer
Yes – the average EFC change is notably higher in the school verification record year (2005-06) than in either sample year 

## Average EFC change among independent students with a change to the indicated field over time

**Notes:**
Same thing for independent records – but note that in both populations the average EFC change is considerably lower for independent than among dependent records. 

## Average Pell change for dependent records with a change to the indicated ISIR field: 2006-07

**Notes:**
Here are the average changes to Pell.  Calculated as Initial award minus paid on value. So negative numbers indicate an “average” potential overpayment
Again this reflects both negative and positive values in the data.

## Average Pell change for independent records with a change to the indicated ISIR field: 2006-07

**Notes:**
Similar results for independent records 

## Average Pell Grant change for dependent records with a change to the indicated field over time

**Notes:**
How should we interpret the above?

Give choices if need be
The verification procedures QA schools were using in 2005-06 were failing to target the ISIRs most at risk for a Pell over-payment
The verification procedures QA schools were using in 2005-06 were effectively targeting the ISIRs most at risk for a Pell over-payment (correct)

## Average Pell Grant change for independent records with a change to the indicated field over time

**Notes:**
Same thing for independent records.

## Potential improper payments in Pell Grants prior to verification at QA Schools: 2006-07

**Notes:**
Explain improper payments can be both positive and negative.

If QA Schools had awarded Pell Grants in 2006-07 based on the information on their students’ initial ISIRs, 9 cents out of every dollar awarded would have been an overaward, 7 cents that should have been awarded would not have been made.

## Improper Payments in Pell Grants before and after school verification

**Notes:**
But QA a schools did not award Pell Grants based on initial transaction data. One way schools prevented improper payments

Explain chart – the third “net” column is the sum of the first two

School verification reduced both potential over-awards and under-awards, cutting the percentage of dollars at risk more than in half

Explain that no improper payments among the records in the sample – but  this is based on a random sample of all applicants.  Presumably a similar pattern is going on among the larger QA school Pell population

## Improper Payments in Pell Grants before and after CPS verification

**Notes:**
This figure represents how effectively CPS verification targets improper payments in Pell. CPS more effectively targets over-awards but less effectively targets under-awards in Pell than school verification

## Lessons Learned

- While verification is crucial in some cases, many ISIR records do not experience a meaningful change from their initial transaction value
- QA school verification procedures target larger EFC and Pell changes
- QA school verification reduces improper payment risks in Pell

**Notes:**
To sum up our program-wide analysis ... 

Bullet three:  Say, QA school verification reduces, but does not eliminate improper payment risks in Pell.  

## What Do the Findings Mean for Schools and FSA?

- Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.

## Two Ways to Focus

- Look for the records that matter
- Look for the records that don’t

**Notes:**
We have tended to focus on the first way of “focusing.” Today we would like to talk about the second. That is identifying records that your school is currently verifying, but may not need to.

## Suggested Additional Analysis

- Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification

**Notes:**
That are two ways to increase the average changes to EFC and Pell Grant among records subjected to school verification. 

The first is to find more of the “big” changes and figure out how to add them to your school’s verification procedures. This is the type of analysis we have tended to emphasize over the years. 

The second way is to look for ways to identify student records that are currently being verified but don’t experience any real change to their aid eligibility. Ideally you want to figure out how to eliminate such records from future verification efforts. 

## Additional Analysis

- ISIR Analysis Tool reports
- Exporting data from the Tool

**Notes:**
You can do this type of analysis within the tool, but you might also want to consider exporting the data from the tool into a spreadsheet or even statistical package for greater flexibility. 

## ISIR Analysis Tool

- Key Filters
- EFC change < 400 & EFC change > -400
- EFC change > 400
- EFC change < -400
- Institutional verification criteria

**Notes:**
In working in the Tool it is essential to use filters to focus on two contrasting groups.

Records that your school is currently verifying that experience “trivial” change to EFC 
Records that experience a magnitude of change sufficient to warrant verification
In the example above I use an absolute value of 400 to distinguish between a trivial and sufficient change, this is ONLY a  suggestion.

It is also probably best to conduct the analysis separately for each of your schools main school verification criteria.

## ISIR Analysis Tool

- Key Reports
- Sample summary
- Field Increment (EFC or AGI)
- Drill down to refine understanding

**Notes:**
Use the filter to run simple descriptive reports. Compare the sample summary report or the field increment report for the population with a trivial change to the same report for records with a substantial change. Look for differences and explore these differences by drilling down on the types of students or value ranges where you notice differences.  

## New Analytic “Recipe”

- See handout

**Notes:**
See more details in our handout –
Check time before going into detail

## Exporting Data

- See handout

**Notes:**
This document explains how to download data from the tool and analyze it in Excel or SAS.

Check time to determine how much detail to cover.

## Institutional Profiles

- Comparison of Program-wide and school specific data

**Notes:**
We created institutional profiles for schools that were part of the analysis group from 06-07.  Several schools were excluded from the analysis group for various reasons, e.g., data was not uploaded in time, data was corrupted, the wrong sample was uploaded, etc.  If your data was included in the analysis group, you should have received an electronic copy of the profile earlier this month.  We also distributed copies of the profiles prior to the start of this session.

The 2006-07 profile looks very similar to the 2005-06 version.  However, there are distinct differences to point out.

## 2006-07 Institutional Profile

**2006-07 Institutional Profile**

- Example of a 2006-2007 Institutional Profile

**Notes:**
Here is an example to the 2006-07 profile.  As you can see it looks pretty much like the profile you received last year.  The top portion of the profile displays dollar values for improper payments for all QA Program schools, as well as the individual school.  The QA Program wide information is represented by the magenta color, and the school specific information is displayed in the periwinkle color.

The bottom portion displays the same information from the top portion, but uses percentages instead of dollar figures.  The bars to the far right display the “net impact” of verification.  For 2006-07 you will probably see the bars for this data point to be pretty equal between the QA Program-wide results and individual school results.  This is because each school was required to conduct 100 percent of verification on records selected for the random sample for 2006-07.  

## Profile Differs from Year to Year

- 2005-06 provided data of improper payments _*PREVENTED*_ by institutional verification
- 2006-07 provides data on estimated levels of _*POTENTIAL*_ improper payments in a schools applicant population

**Notes:**
It should be noted that there are distinct differences b/w the 05-06 and the 06-07 profiles.  Specifically, the 05-06 data represented records that met your schools verification criteria, and therefore the data on the profile represents improper payments that were prevented by institutional verification.  The 2006-07 data represented records that were a random selected of aid applicants at your school.  The records were all verified according to the federal verification worksheet and the profile data represents an estimate of potential improper payments in your applicant population.

As we’ve mentioned earlier in the session, you need to be very careful in comparing data from one year to the next because of the differences in the populations and the type of verification performed.

## Contact Information

- We appreciate your feedback and comments. We can be reached at:
- Email:	david.rhodes@ed.gov 				anne.tuccillo@ed.gov