# Presentation: 711895

## Controlling the Quality of Verification: Verifying a Sample of ISIR Records

- Anne Tuccillo and Michael Cagle
- U.S. Dept. of Education
- David Rhodes and Stacey Merola
- American Institutes for Research

***Session 12***

## Questions

- Are you looking forward to your next program audit?
- Do you relish explaining your institutional verification procedures to students who ask “why me?”
- Are you sure all of your non-verified ISIR data are accurate?

## Questions

- Would you like to maintain a solid empirical basis for your institutional verification practices?
- Would you like to know what types of students your current institutional verification may be missing?
- Would you like to identify efficient means of improving your institutional verification efforts?

## Session 12

- Review of Sampling Principals
- A New Approach to Sampling in the QA Program

## Sampling Principals

- What is a sample?
- A portion of a larger group called a population
- What makes a good sample?
- Representative
- Reflects Target Population
- Random

## Why Sample?

- Efficiency
- Cost
- Focus

## How is sampling relevant in a financial aid context?

- Schools are accountable for the aid they disburse.
- Institutional verification efforts target the groups thought to be most prone to error
- No way to know about the accuracy of non-verified ISIR information
- Confirming the accuracy of data in a sample of ISIRs would allow you to more fully assess your verification procedures

## Possible Sampling Strategies

- Simple Random Sampling
- Stratified Random Sampling
- Systematic Sampling

## Simple Random Sampling Steps

- Sampling frame – list of population members
- Determine population size
- Determine sample size
- Determine sampling probability (#3  #2)
- Generate a random number for every case
- Select cases with random numbers less than or equal to the sampling probability

## Sampling Theory Demonstration

- Sample of 5 blue cards
- population = 45 cards
- cards range from 3 to 7 (nine each)
- Population average = 5
- Sample of 5 yellow cards
- population = 45 cards
- cards range from 1 to 9 (five each)
- Population average = 5

## New Approach To Sampling in the QA Program

- Background
- Why do I have to sample?
- What do I have to do?
- When do I have to do it?

## Background

- Participants in the QA Program have regulatory relief from following the Secretary’s (CPS) verification edits
- These schools develop their own institutional verification criteria based on analysis of their own data

## QA Program Methodologies

- Old methodology
- October “documented” sample
- Readings-based software
- Current methodology
- ISIRs generated during normal processing
- ISIR Analysis Tool (PC-based)

## QA Program Monitoring 

- New methodology
- Biannual verification of a sample of least 350 ISIRs prior to disbursement
- Biannual analysis of ISIRs generated during normal processing
- ISIR Analysis Tool (Web-based)

## Why Do I Have to Sample?

- You cannot know the accuracy of ISIR information that is not verified
- Analysis using data from normal aid processing can identify ways to make verification more efficient
- Analysis using data from normal aid processing cannot identify ways to make verification more accurate

## What Do I Have To Do?New Activities

- Complete Sampling Plan Worksheet online at _[http://](http://ifap.ed.gov/qualityassurance)__[ifap](http://ifap.ed.gov/qualityassurance)__[.ed.](http://ifap.ed.gov/qualityassurance)__[gov](http://ifap.ed.gov/qualityassurance)__[/](http://ifap.ed.gov/qualityassurance)__[qualityassurance](http://ifap.ed.gov/qualityassurance)_
- Randomly select at least 350 non-duplicate ISIRs from your aid applicant population
- Compel each sampled student to complete a federal verification worksheet by placing a hold on their aid disbursement
- Submit ALL changes to ISIR information stemming from the sample through the central processor

## What Do I Have To Do?Continue Current Activities

- Import data and apply institutional verification profiles in the online version of Tool
- Analyze changes in ISIR information detected in sample
- Apply results of analysis to improvements in institutional verification profiles.

## When Do I Need to Do It?

| Activity | Deadline |
| --- | --- |
| Sampling Plan Worksheet | January 15 |
| Draw sample of ISIRs | July 31 |
| Collect verification worksheets | In time to disburse aid on time |
| Submit all ISIR changes to CPS | In time to disburse aid on time  |
| Import data and apply verification profiles | November 30 |
| Analysis of changes to ISIR data | December 15 |
| Apply results of analysis | Start of 2005-06 verification |

## Complete Sampling Plan Worksheet

- Online planning tool on QA website
- Email reminders (nagging)
- FSA will use to monitor collective progress and identify assistance needs
- Timeframe: November 2003 – January 2004
- Deadline: January 15, 2004

## Randomly Sample 350 ISIRs

- At least 350
- Why more?
- attrition
- statistical power
- The online ISIR Analysis Tool will have a limit of 500 pairs
- You can use the Tool to draw a sample, but not until June 27, 2004

## Randomly Sample 350 ISIRs

- Population of interest – aid recipients – is “emerging”
- If practical remove
- applicants who are not admitted
- applicants who decline admission
- applicants who decline financial aid

## Randomly Sample 350 ISIRs

- Rolling admissions
- > 20% apply for aid after fall disbursement
- Spread 350 cases in sample proportionally across the year
- Deadlines apply only to fall cases

## Randomly Sample 350 ISIRsSimple Random Sample Steps

- Sampling Frame – list of SSNs and first two characters of last name for aid applicants with “potential” to receive aid
- Determine Population Size – if need be estimate for whole sampling window
- Determine Sample Size – 350 or more
- Determine sampling probability (#3  #2)
- Generate a random number of every case
- Select cases with random numbers less than or equal to the sampling probability

## Randomly Sample 350 ISIRsFinal Product

- Electronic list(s) of SSN and first two characters of last name of applicants selected into sample
- “Optional” transaction number – important for paid on transaction
- Keep track of initial institutional verification status in a separate file

## Consider the Following in Deciding When to Randomly Sample 350 ISIRs

- When does institutional verification begin?
- What information does institutional verification confirm?
- Is it possible to integrate sample verification with existing verification efforts?
- When will admission and acceptance decisions be known?

## Randomly Sample 350 ISIRs

- Timeframe: January through July
- Online Tool sampling component available June 27, 2004
- Deadline: July 31, 2004

## Complete Federal Verification Worksheets

- Most labor intensive part of sampling
- Pilot schools spent between 2 to 6 FTE weeks completing this process
- Must be done before aid is disbursed
- Make the process as seamless to students as possible

## Complete Federal Verification Worksheets

- Timeframe: January through August
- Deadline: In time to disburse aid on time

## Submit All Changes through CPS

- Usually only submit changes to ISIR information if Pell eligibility is affected
- But data used in analysis needs to reflect all changes
- Only way to get the right ISIRs in the data mart is to submit all corrections
- Keep track of transaction number of corrected ISIRs!

## Submit All Changes through CPS

- Timeframe: January through August
- Deadline: In time to disburse aid on time

## Import Data and Apply Verification Profiles

- FSA Data Mart will make finding the right ISIRs much easier
- Schools only need an SSN file and have the option of specifying a specific transaction number
- Attend Data Mart Session if possible for more detail

## Import Data and Apply Verification Profiles

- Apply verification profiles to data
- Set institutional verification flag
- Write queries that mirror your criteria
- Check off
- Contact regional representative if you have difficulty doing this
- Timeframe: October and November
- Deadline: November 30, 2004

## Analyze Changes to ISIR Data

- Richer, “what did we miss” analysis with sample data
- Enhanced analytic capacity of the online version of the ISIR Analysis Tool
- Predefined reports
- Drill-down capacity
- Set aside staff time for analysis

## Analyze Changes to ISIR Data

- Timeframe: October 17 - December
- Deadline: December 15

## Apply Results to Institutional Verification

- What groups of students (if any) are your institutional verification missing?
- How can you efficiently expand your institutional verification profiles to target these students?
- What groups of students (if any) can you stop verifying?

## Apply Results to Institutional Verification

- Timeframe: November and December
- Deadline: Start of 2005-2006 institutional verification

## Technical Assistance

- We appreciate your feedback and comments.  We can be reached at:
- _[Anne.Tuccillo@ed.gov](mailto:Anne.Tuccillo@ed.gov)_
- _[Michael.Cagle@ed.gov](mailto:Michael.Cagle@ed.gov)_
- _[Drhodes@air.org](mailto:Drhodes@air.org)_
- _[Smerola@air.org](mailto:Smerola@air.org)_