# Presentation: 372353

## <date/time>

**Notes:**
Before we begin, I’d like to point out our “Ask it Basket.”  If you have a question, please feel free to write it down on a post it note and place in in the “Ask it Basket.”  If we can’t answer it during the session, we will try and contact you via email.  So please be sure to include your email address, if you want a reply.

This session last 75 minutes.  We want to allow for time for discussions and questions after the presentation.  So if we could try and hold questions until the Q&A session, that will help us out a lot.

## Session 21

- _Quality Analysis Tool:_
***The Sharpest Tool***
***in the Shed***

- Session 21-

**Notes:**
Good morning/afternoon.  I am Anne Tuccillo, a member of the Performance and Accountability Improvement Staff in SFA.  Together as a team, we work with both the Quality Assurance and Experimental Sites Programs.  I am joined today by Mr. David Rhodes, from the American Institutes for Research, and Sandi Sanders from NCS Pearson.  Together we are going to describe and discuss the Quality Analysis Tool.  The Quality Analysis Tool is a powerful product that can help any school learn more about its applicant and recipient populations as well as fine tune institutional verification procedures.  While this tool was originally developed  for QA Program schools, it is now available for any school to use, and we hope to help you understand the benefits that any school can obtain .  Schools have told us for a number of years the difficulties they experience in trying to obtain analysis information about their student population.  Well we are hear today to tell you that you now have another tool in your toolbox to improve aid delivery and service to students.  If there’s anyone in the audience that is from a QA school, can you please stand up.  If there’s anyone in the audience that currently verifies 100% of their applicants please stand up.  If there’s anyone in the audience that only  verifies the 30% minimum, please stand up.  Look around you, everyone standing here in this room can benefit from this verification tool

## What is the Quality Analysis Tool?

-  Part of the EDE suite of software
-  Available to all schools in ‘02-’03
-  Analyzes FAFSA application/ISIR data
-  Generates reports showing effect of data    element changes on EFC/Pell Grant eligibility

- Session 21-

**Notes:**
Have you ever wondered how effectively verification edits target students in your applicant population?  The Quality Analysis Tool is part of the EDE suite of software and will be available for any school to use beginning January ‘02.  The QA Tool  will assist schools in analyzing FAFSA applicant/ISIR data for their Title IV population and determine if changes in student reported data elements had an impact on EFC and Pell Eligibility.  Users will import ISIR records, both the initial (the one used prior to verification) and then a paid on record (the one used to determine eligibility) into the Tool.  Through the use of queries and specialized reports, users will obtain data about aid applicants at their school.  Schools can use this data to design institutional verification processes that focus in on problematic areas and zeros in on specific EFC ranges, data elements, and populations.  For purposes of this presentation we will be referring to version 1.0 which was the first first released.  We will also talk about some of the enhancements that will be available with version 2.0, that will be available in January ‘02.

## What the Quality Analysis Tool Provides

- Improve/customize verification program
- Data to suggest future changes in federal verification selection criteria (In progress).
- Data to suggest future changes to the FAFSA
- Gives schools control over “local sources” of misreporting

- Session 21-

**Notes:**
The data generated from the reports can be used to help identify information on the FAFSA that is often  misreported and or troublesome for students and parents.  That is powerful information to know and could possibly lead to changes in the FAFSA and federal verification selection criteria..  This tool can also help schools develop locally designed verification  procedures.  To address the needs of their population.

## Evolution of the Tool

- Developed for Quality Assurance Program Schools in 99-00
- QA Program schools pilot tested the product using 1999-00 data.  The result:  A Tool for all Schools!

- Session 21-

**Notes:**
How many of you have heard of the Quality Assurance Program?  How many of you participate?  The Quality Analysis Tool was developed for schools participating in the Quality Assurance Program.  QA Program schools conducted a Pilot test of the QA Tool using 1999-00 ISIR data.  The QA Program schools like what they were able to do with the product and encouraged SFA to provide this Tool to all schools.  QA Program schools have been advocates for streamlining verification that is based on data and determining what particular characteristics are common on individual campuses.  The tool allows just that, it allows for schools to learn about their own applicant population.  It also provides schools with the data that shows the impact on EFC and Pell Grant eligibility due to changes between initially  reported data vs.. paid on data.  How  many of you can easily get your hands on this type of analysis?  And would you see any value in having this information?  Hopefully, by the end of our session, the answer will be “yes.”

## Locally Designed Verification

-  Focuses on local sources of error
-  Identifies changes to FAFSA fields that have an impact on awards
*What are the benefits of school-designed verification procedures?*

- Session 21-

**Notes:**
Let’s stay with the locally designed verification program concept for a little longer.  What do I mean by focusing on local sources of error?

At a given school,, the Quality Analysis Tool enables you  to identify practices that have an impact on aid awards.  For common errors, students and  parents can be advised and educated.
On a more practical level, when you are dealing with the impact that changes on applications have on the Expected Family Contribution, awards can be adjusted so that deserving students who are missed can be helped.  At the same time, knowing the real impact of changes, a school will not have to burden student with unnecessary follow up procedures.  While that may be true specifically for schools participating in the QA Program.  Non QA schools will be able to see if the CPS edits are having an impact on eligibility/awards for students at your school.
SFA wants to help schools be successful, and the Quality Analysis Tool can help a school refine verification procedures to permit schools to redirect their resources from pushing paper to handling activities that make a difference.

## The Power of Data

- The QA Tool allows schools to conduct in-depth analysis of applicant data
- A series of reports can be designed using queries and other filters to extract specific information about a school’s population.
- Provides schools with a profile of who is making changes to data and the impact on Pell eligibility.

- Session 21-

**Notes:**
Later in our presentation, you will learn more about the importance of the data generated by the Tool.  What is important to point out here is that the Quality Analysis Tool can be your office research assistant.  Schools will be able to conduct analysis using a series of reports to determine how well its verification procedures are working.  Using this approach, a QA  institution can stop spinning its wheels and develop verification criteria that fit its particular population.    It can know which application errors occur locally, educate students and parents and work to eliminate errors.  Thus, schools not only can save money, but make it available to students whose mistakes otherwise might cost them eligibility. For those of you who conduct 100% verification, or for those of you who follow the Secretary’s edits, do you feel you are getting the most bang for your buck.  Are your limited resources really  targeting the problematic areas?   Is it important to you to know this information?  What would you do differently?
The Quality Analysis Tool helps schools separate verification practices that make a difference in awards from those that  take time but make little difference.    The analysis capabilities are quite powerful.  Schools will have the opportunity to deal effectively with two kinds of practices - their best and worst.
SFA is excited to provide a tool that helps schools make their work more productive and more service oriented.  Emagine the possibilities.
At this time I am going to turn it over to _____________________from NCS Pearson, who will briefly review the functionality and  proposed enhancements for the Quality Analysis Tool.

## Overview of Software

- Session 21-

## Software Components

- Import
- Tools
- Export
- Process
- Entry
- Print

- Session 21-

## Import

- ISIR Data - ASCII Format
- “Paid On” records
- “Initial” records

- Session 21-

## Import

- 2002-2003 EDExpress Database
- “Paid On” records
- “Initial” records
- Selection Criteria (Select Records)

- Session 21-

## Tools | Custom Formats

- Custom Report
- for Custom Report only
- Selected Fields
- File | Open, Reports
- Field Increments
- Reports
- EFC Ranges
- Reports

- Session 21-

## Tools | Query

- Quality Analysis Tool
- EDExpress
- Field to Field Query Capability

- Session 21-

## Export

- Full Data Export
- Student Data (“Initial” and “Paid On”)
- User Database

- Session 21-

## Process | Data Validation

- After you Import (“Initial” and “Paid On”)
- Deletes “unmatched” records from the database
- If you run process after only “Initial” or “Paid On” Import, *all* records will be deleted.

- Session 21-

## Process | Data Validation

- Calculates and stores data on scheduled Pell awards, which is used in some reports
- When calculating, QA Tool makes these assumptions:
- The student’s enrollment status is full time
- The student is Pell-eligible if the EFC is within a Pell range
- The Pell budget is more than $3750

- Session 21-

## Process | Data Validation

- Must complete successfully before printing student related reports
- Non-student related reports may be printed before running Data Validation

- Session 21-

## Process | Verification 	

- Allow schools to setup verification profiles
- Can be set through Process Menu
- Can be undone through Process Menu
- Sets/Removes Initial School Verification Flag

- Session 21-

## Entry 	

- Transaction Comparison
- Demographic data
- Initial School Verification Flag (the only field that can be “entered”)
- Displays “Initial” values and “Paid On” values side-by-side
- Change?
- Can control which fields are displayed (Selected Fields Code)

- Session 21-

## Print 	

- Reports are the “heart” of the new software. They will be used to analyze student data populations for two major reasons:
- To analyze corrected data in order to develop a better in-house Institutional Methodology
- To identify and target problem populations making the most errors and educate them in the FA process

- Session 21-

## Print

- Report Types
- 9 “analysis” reports
- Custom Report
- Query Management
- 3 Record Layout reports
- System Setup
- User Database

- Session 21-

## Quality Analysis Tool

- ALL analysis is simply structured comparison
- The QA Tool provides a variety of frameworks (reports) to make comparisons
- The QA Tool allows the user a great deal of flexibility in tailoring these comparisons to suit their institution’s needs

- Session 21-

**Notes:**
No matter the how crucial or trivial the topic or how brilliant or dim the analyst, analysis is simply structured comparison.  The QA tool uses reports to present comparisons.  Complete flexibility is offered users in terms of  the ISIR data elements they choose to examine and which subsets of cases they choose to subject to examination. 

## Basic Comparison 

- “Initial” ISIR
- “Paid on” ISIR

- Session 21-

**Notes:**
The Quality Analysis Tool compares the initial ISIR transaction to the paid on transaction.  The pilot version tested last year took advantage of the last QA Program quality control random sample that verified ALL information before “paying.”   WHAT CHANGES AFTER INITIAL APPLICATION

Software tool allows schools to import the correct 
initial = information they first received and
paid on = information they awarded on
NOT NECESSARILY the FIRST and LAST

## Purpose of Comparing

- Descriptions of applicants who experienced _changes_ to EFC but were “_missed_” by verification
- Descriptions of applicants who had _no changes_ to EFC but were “_pestered_” by verification
- Descriptions of applicants who_ self-correct_ information outside the verification process

- Session 21-

**Notes:**
The QA Program provides regulatory relief for participating institutions from following federally proscribed verification edits.  Therefore, the QA Tool was developed to help QA participants monitor and improve their own verification programs.  Verification has twin evils: missing and pestering.  The QA Tool helps schools identify what income bands or other sub-populations are over-represented in the two problematic populations.  The tool is available to all schools and serves the same purpose of monitoring the effectiveness of verification – CPS edits – on any Title IV campus.  By using the QA Tools, schools can identify problematic cases in terms of their verification program, including the Federally proscribed edits. Changes to FAFSA information occurring independently of verification can also be examined.  It might not make sense to expend limited verification resources on cases who are likely to self-correct. 

## Framework of Comparisons:QA Tool Reports (Print)

- Field Range
- Pell Corrected Records in Initial EFC Range
- Transaction Comparison

- Session 21-

**Notes:**
The QA Tool prints 9 analysis reports I want to show you examples of how three of these reports can be used stop by the PC lab if you are interested in seeing other reporting options. 

Field Range – Changes to ISIR fields within ranges of a specified ISIR field, for example income bands.  Allows you to answer question: WHAT RANGES OF STUDENTS ARE MOST LIKELY TO MAKE SUBSTANTIVE CHANGES TO FAFSA INFORMATION?

Pell Corrected Record – displays the level of changes in Pell eligibility among the initially Pell eligible within user specified ranges of any ISIR data element, e.g. income bands.  Allows you to answer the question WHAT TYPES OF STUDENTS ARE LIKELY TO EXPERIENCE OF CHANGE TO PELL ELIGIBILITY?  

Transaction Comparison – Student Level comparison of initial and paid on values for user selected ISIR elements – Allows you to answer questions about how multiple factors combine to influence aid eligibility for individual cases.  

## Content of the Comparison: Query Function

- Selection criteria for reports specified via queries
- Allows user to limit comparisons to specific subsets of applicants
- Uses same query interface as other EDEsuite products

- Session 21-

**Notes:**
Before moving onto the reports though let me stress how central the query function is.  Users have a great deal of flexibility in terms of what cases get “poured into” the report templates through the “selection criteria” they use . Allows users to utilize query expertise developed with other EDEsuite products.

## Field Range Report

- Within what ranges are substantive changes to FAFSA information most problematic?
- Answer helps “target” verification

- Session 21-

**Notes:**
The example I will should is part of analysis conducted on real pilot test data from a QA institution.  This analysis focused on Worksheet A (untaxed income) reporting which was identified as a major problem for many QA institutions in past analysis

## Field Range Example

- Session 21-

**Notes:**
Parental Income by 10,000 for 
Dependent Students with Worksheet A > 0 QUERY
Point out concentration of EFC increases in 30,000 to 50,000 income band
Point out other types of analysis possible with this report. E.g. bands of worksheet A close report

## Pell Corrected Records in Initial EFC Range Report

- How do changes to a given ISIR field contribute to changes in Pell eligibility – within specified Pell Cell(s)?
- Answer helps “fine tune” verification

- Session 21-

## Pell Corrected Example

- Session 21-

**Notes:**
Point out majority of cases 25 of 34 have a change to Worksheet A
BUT only a minority 8 of 25 changes lead to change in Pell

## Pell Corrected Example (Continued)

- Session 21-

**Notes:**
Initial Pell Awards of $66,825 reduced to $48,050
$13,575 was accounted for by the 8 who lost Pell eligibility entirely 
Potential benefit of targeting

## Transaction Comparison Report

- How do the patterns recognized at the aggregate level play out in individual cases?
- Answer helps tie the myriad of “reasons to check” FAFSA information into an integrated verification program

- Session 21-

## Transaction Comparison Example

- Session 21-

**Notes:**
SSN 012-54-3873 moderate income change 
SSN 021-56-7370 low income no change

## Transaction Comparison Example (Continued)

- Session 21-

**Notes:**
SSN 012-54-3873 moderate income change 
SSN 021-56-7370 low income no change

## Who Benefits From Making Comparisons

- QA Program participants – fine tune effective institutional verification programs
- Non-QA Program institutions – evaluate the effectiveness of CPS edits on their campus
- SFA – fine tune CPS verification and evaluation of FAFSA

- Session 21-

**Notes:**
QA Tool designed to monitor effectiveness of any verification system on a given campus

Non-QA schools can use to evaluate their use of CPS edits.  Identify the types of students who are flagged by CPS yet experience no change in award.

SFA can use data to evaluate CPS verification flag present on ISIR data.   

## Policy Applications From Pilot Study

- Improve FAFSA instructions and/or education efforts regarding estimating income and tax information
- Incorporate tax “estimators” into verification profiles
- Use – but don’t overly rely on – income level in verification profiles
- Consider collecting information on cost of verification

- Session 21-

**Notes:**
The three most problematic ISIR elements identified through analysis of QA Tool data were AGI, US Taxes, and Worksheet A.  While the creation of a new Worksheet C on the FASFA since 99-00 may have improved the ability of applicants to report untaxed income, being able to accurately complete all three of these items is very much dependent on having a completed tax form.

Verification profiles at most schools should probably include student and family income level, but in the context of other information.  The CPS verification flags seem to focus disproportionately on dependent students with moderately low family incomes ($20,000 - $39,999). While this group constituted 16.5% of the title IV population, they made up 30.2 % of cases flagged by CPS.    To focus so heavily within this particular income band leads one to think that these students are being “pestered.”  One quarter (24.9%) of students flagged by the CPS for verification that recorded a zero change in EFC were dependent students with incomes between $20,000-$39,999.  Moreover, 23.7 percent of the applications “missed” by CPS verification were among dependent students in the next higher income band $40,000 to $59,999.
Finally, the analysis of verification does not address the cost of verification either to the institution or the student.  Knowing the cost of verification would make for more informed decision making.  Being able to identify situations where the cost of verifying a given type of applicant is greater than the expected change in financial aid would be useful in fine tuning verification profiles.  A good analyst always follows any answers he or she provides by asking another question!

## Further Assistance

- CPS/WAN Technical Support
- 800-330-5947
- cpswan@ncs.com
- QA Program Website
- http://qaprogram.air.org/
- Additional Assistance
- sfatech@inet.gov

- Session 21-

## Questions???

- Session 21-