# Presentation: 623964

## Quality Assurance Program

- Pre-Conference

## Quality Assurance Pre-Conference

**FSA Assessment****Case Study**** **

## “Coming together is a beginning;
	Keeping together is progress;
	Working together is success”

- Henry Ford

**	****Keeping together is progress;**
**	****Working together is success”**
**- Henry Ford**

## Session Objectives

- Experience the Value of an FSA Assessment Activity and the Management Enhancement Form
- Learn the importance of team workto resolve a finding.

## Group Activity

## Instruction

**Work with your table to discuss the Management Enhancement/Action Plan. **
**If other stakeholders are needed to assist in the resolution of your plan, assign a team member to discuss issue(s) with appropriate stakeholders:****Financial Aid Office, Business Office, Student, Senior Management or Computer Center**

## Lessons Learned

- Don’t Assume
- FSA Assessment Activities and the Management Enhancementare valuable tools
- Team Work – is a necessity

## FSA AssessmentRequirements for QA Schools

## Session Objectives

- Learn where to locate the FSA Assessments
- Review FSA Assessment Highlights
- Introduce Policies and Procedures Component
- Learn FSA Assessment requirements for QA schools

## Where to find the FSA Assessments 

**Web address:**** ****We recommend that you save it as a favorite**
- _[http://](http://ifap.ed.gov/qahome/fsaassessment.html)__[ifap.ed.gov/qahome/fsaassessment.html](http://ifap.ed.gov/qahome/fsaassessment.html)_**. **
- Or
**IFAP.ed.gov**

## FSA Assessment Highlights

- This Chart:  Ways to use the assessments
- Policies and Procedures: Ensure you are meeting minimum requirements
- Management Enhancement: How to track your progress

## This Chart:  Ways to use the FSA Assessments

## Policies and Procedures

**Beginner, Intermediate and Advanced Users**
**Minimum Requirements**
**Regulations**** **
**Demo: Go live to the FSA Assessment**

## QA Program Requirements for the FSA Assessments

- Complete two assessments or activities
- Notify QA Regional Representative of the two assessments
- If applicable, complete the Management Enhancement Form

## Timeline for the FSA Assessments

| January - June, 2008 | Continue completion of 2007-2008 assessments or activities. Keep copies of all Management Enhancements in QA file. |
| --- | --- |
| February 1, 2008 | Notify QA Representative which assessments or activities you will complete for 2008-2009 |
| March – June, 2008 | Establish Teams to begin completion of the assessments or activities for the 2008-2009 award year |
| July, 2008 – June, 2009 | Complete Assessments for 2008-2009, and any Management Enhancements |

## Team Building 

## None of us is as smart as all of us.
- Japanese proverb

**- Japanese proverb**

## Session Objectives

- Complete a Team Building Exercise
- Learn the value of a team
- Access Team Building Resources

## Activity

## Team Building Web Site

**http://www.teambuildinginc.com/tps/002.htm**

## Mission Accomplished

- Completed a team building exercise
- Learned the value of a team
- Found Team Building Website

## QA Program Pre-Conference

**The Detail **
**on **
**Data Analysis**
**David Rhodes**
**Anne Tuccillo**

##  “Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital.”

**Aaron Levenstein**

**Notes:**
This quote was waiting for me when I returned to the QA team last fall and still one of my favorites.

## Session Objectives 	

- Share program-wide analyses
- Share institutional profiles
- Go over your analysis requirements for 2008-09 and 2009-10
- Remind schools about verification status codes in COD

**Notes:**
We have a full plate, but we do have nearly 2 hours.

We will take a 10 minute break after passing out the institutional profiles

## Session Objectives (Cont’d)

- Introduce a new “EFC Impact Analysis” standard report
- Preview a new shared report
- Provide tips on applying and sharing results

## Analysis of 2006-07 Sample Data

- 132 schools
- Average enrollment was 20,105
- Disbursed 11% of all Pell Grants in 2006-07
- 62,958 individual ISIR records

**Notes:**
Last year we presented preliminary analysis of 2006-07. Today we would like to present some of the additional analysis we did.  

While 146 schools supplied data, we ONLY used data from 132.  Gently chastise schools about reasons: no school flag set; only school selected records uploaded; no change in data (all initial transaction).

Many QA schools are large so the program wide analysis tends to reflect this characteristic

As we go through the results I will be making some comments on HOW I presented the information as well as discussing the results themselves. There are some things to learn from what we did right and wrong.  

## Schools by Sector

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
Most of the 2006-07 came from four-year public schools but we also had some from private four-year and public two-year

The Good: pie charts are a great way to show your audience the share of different groups:  especially if there is something to see, different size slices and not too many slices.  Note the three shades of the same color. These will still show up in black and white.

The Ugly:  I now think that N,% was a poor design choice on my part, confusing.  Should have gone with just N.

## Verification Status by Sector

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
On average QA schools select more than students for verification than they would have to under CPS

Note that this pattern is reversed for the 8 public two-year schools

Note how easy it is to make the comparison between school and cps for each group. 

## Classifying the Effect of Verification on Eligibility

**No change**—EFC and Pell Grant Award remained the same
**Major decrease**—Pell Grant decreased or EFC increased at least 400
**Major increase**—Pell Grant increased or EFC decreased at least 400

**Notes:**
We came up with a five point classification system for the effect of verification

## Classifying the Effect of Verification on Eligibility (Cont’d)

**Minor decrease**—EFC increased less than 400 and Pell Grant remained the same
**Minor increase**—EFC decreased less than 400 and Pell Grant remained the same

## Change in Eligibility by Sector

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
Stacked bars are a great way of showing and comparing the relative distribution of categories across groups.  Every group sums up to the same 100.  Where they can be misleading is when the groups are unequal size. Note the total and public four-year bars are nearly identical because so a large percentage of schools are public four-year.

Found a “whole lot of nothing going on” – Half of the random sample selected for verification experienced no change to EFC.

Note that this was especially the case for public two years.  We called several public two-year schools that had the highest initial accuracy rates and found that they used outreach efforts or offered in office assistance in completing the initial FAFSA. Thus there were not many errors for verification to detect.

Note it is the two ends we want to verify 

Note that the 2 “minor” categories are well minor.  Less than 5 percent.  In subsequent analysis we began to fold these cases in with the no change records and labeled them “no major change” 

## Analysis of Quality Assurance Program Sample Data: 2006–07

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
Black and white treated “correctly”

It would be nice to loose the blues if we could

Both the School and CPS verification glasses are half full (empty) – just over half of the dependent and independent students are black or white. 

## Analysis of Quality Assurance Program Sample Data: 2006–07

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
We looked at efficiency for a number of different groups in this example EFC value range 

When light blue dominates consider reducing verification

When dark blue dominates consider increasing verification

In general QA school verification over-verifies zero EFC

## Analysis of Quality Assurance Program Sample Data: 2006–07

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
When light blue dominates consider reducing verification

When dark blue dominates consider increasing verification

In general CPS school verification over-verifies zero EFC and under-verifies those above Pell threshold

## IRS and Other ISIR Data

- To what degree are the corrections uncovered by verification limited to the fields addressed by a potential IRS match – adjusted gross income and taxes paid?
- What portion of the changes in aid eligibility  captured by current federal verification  worksheets are due to adjusted gross income and U.S. taxes paid data?

**Notes:**
FSA is still exploring a potential IRS match.  We shared our analysis of your data with the people working out the details of a potential match and decision makers are including our findings in their decision makers.  

(Read slide)

One of the most difficult things for me to learn to accept as a data analysis is that there are often very real reasons why the implications of my analysis cannot be incorporated in policy. By definition analysis looks at a small slice of the picture and at times “externalities” make even valid findings mute. 

## Types of Change Observed

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
Explain graph 
Note that half the records had no change

Only 10 percent of records (9.1 and 9.6) had changes ONLY to IRS fields

## Share of Estimated Change to EFC

*Analysis** **of Quality Assurance Program Sample Data: 2006–07*

**Notes:**
But changes to these fields accounted for half the changes.

Again a half glass findings – analysis is not always (usually) clear cut, but it is better to know the mixed facts than act in ignorance.

## Implications of 2006-07 Analysis

- Neither school nor CPS verification was efficient; this was especially true for the most eligible aid applicants
- CPS may want to consider expanding verification among records initially ineligible for Pell Grants
- An IRS match involving only adjusted gross income and taxes paid will not replace the full “value” of the current verification worksheets

**Notes:**
Read slide

## 2007-08 Data Analysis

**Notes:**
School verification descriptions and ISIR data from your school verification population

## Alternating Focus of QA Analysis 

- Random Sample 2006-07 & 2008-09
- Representative of all potential changes in your aid population
- Question of the year: Who are you missing?

- School Selected 2007-08 & 2009-10
- Changes your verification detected
- Question of the year: Who are you verifying unnecessarily?

**Notes:**
The yin and yang of QA analysis

## 2007-08: School Verified Data

- 141 schools with ISIR data
- 141,484 individual ISIR records
- 134 schools supplied a description of their verification criteria
- 124 of these had 2007-08 ISIR data
- 119 of these had 2006-07 ISIR data

**Notes:**
Explain the data received
ISIR data same format as sample years but different focus

QA descriptions varied from brief emails to formal documents. Encourage more formal documentation of criteria and thoughtful review during the two year analysis cycle

Mild berating of missing data. Not all schools had data loaded, many schools did not have any or all of the school verification flags set; no change – same transaction loaded for both initial and paid on.

## The Number of Verification Criteria at QA Program Schools

| Average | 10 |
| --- | --- |
| Minimum | 1 |
| 25th Percentile | 2 |
| Median | 7 |
| 75th Percentile | 13 |
| Maximum | 59 |

*Analysis** **of Quality Assurance Program Data: 2007–08*

**Notes:**
Explain how Anne and I coded schools descriptions  

First thing we did was count them

Describe what we defined as criteria – basically a query or part of a query separated by an OR.

Range greatly. 1 to 59, but 50 percent of schools had between 3 and 13 criteria.  

Used the 25, 75 cutoffs to define low (bottom 25) medium (middle 50) and high (top 25)

## The Relationship between the Number of Criteria and Verification Outcomes

*Analysis** **of Quality Assurance Program Data: 2007–08 & 2006-07*

**Notes:**
Average of three groups defined by the N of criteria

Explain criteria are from 2007-08 and outcome data from 2006-07.  Explain why?  School verified could not provide the percent of (every potential) just the 100% of what was verified.

Note and explain the counter intuitive finding.  The more criteria a school used the fewer records they selected for verification.  Because schools with few criteria tended to use broad ones, e.g. 100%, Pell Eligible, large EFC or AGI range.  

Note that schools with more criteria reduced the over all level of verification but more importantly percent of zero change records selected for verification. Having more criteria and verifying fewer students did not affect the prevention of Pell over-awards – note these percentage when up.  But did undermine the prevention of Pell under-awards.

Presentation note: 15 years ago I would have presented this in four tables;  7 years ago I would have presented this in four charts; now that computers have better graphics you can place the whole story in a single graph.  But you still have to explain it – verbally or in accompanying text. 

## Top 10 Data Fields Used in QA Schools’ Verification Criteria

| 1. Parents' Adjusted Gross Income | 76% |
| --- | --- |
| 2. Student's Adjusted Gross Income | 63% |
| 3. Parents Filed Tax Return? | 58% |
| 4. EFC | 53% |
| 5. Non-ISIR Field | 50% |

| 6. Student Filed Tax Return? | 46% |
| --- | --- |
| 7. Father's Income from Work | 39% |
| 8. Mother's Income from Work | 39% |
| 9. Student's Income from Work | 39% |
| 10. Parents' Taxes Paid | 37% |
| 10. Parents' Worksheet C | 37% |

*Analysis** **of Quality Assurance Program Data: 2007-08*

**Notes:**
Anne and I coded what ISIR fields were used in the criteria.  We then used EXCEL to determined whether or not each school used each field.  This table identifies the top 10 (11 – there was a tie for 10th)

Note AGI was king

Half used non-ISIR – ask how many schools bring this information into the tool as user defined fields?

## Observed Relationships Between School Using Indicated Data Field and Verification Outcomes

| Data Field | Unnecessary Verification | Prevent Over-Awards | Prevent Under- Awards |
| --- | --- | --- | --- |
| Parents' Adjusted Gross Income |  |  |  |
| Student's Adjusted Gross Income |  |  |  |
| Parents Filed Tax Return? |  |  |  |
| EFC |  |  |  |
| Non-ISIR Field |  |  |  |

*Analysis** **of Quality Assurance Program Data: 2007-08 & 2006-07*

**Notes:**
Comparison of 3 outcomes
Unnecessary verification (selected record with no change to EFC)
Prevented over-award (pell over-awards among school selected / total over-awards in sample)
Prevented over-award (pell over-awards among school selected / total over-awards in sample)

Compared schools that included indicated data field to those schools that did not.  Reviewed the magnitude and the statistical significance of the difference between the two averages.  Small differences that we thought likely to be do to chance were left blank.

The arrows 
Green good
Red bad

Declines in unnecessary verification good (green) increases bad (red)

Increases in preventing over-awards good (green) declines bad (red)  

Increases in preventing under-awards good (green) declines bad (red)  

Make sure everybody understands 

This is an example of summarizing data – the five arrows allow the audience to focus on the 5 out of 15 comparisons (30 means) defined by the table that seemed to suggest a relationship.  

Interpret arrows

Downside of using COLOR no difference in black and white.  This will be changed to light and dark shades of blue in written report.

## Observed Relationship Between School Using Indicated Data Field and Verification Outcomes

| Data Field | Unnecessary Verification | Prevent Over-  Awards | Prevent Under- Awards |
| --- | --- | --- | --- |
| Student Filed Tax Return? |  |  |  |
| Father's Income from Work |  |  |  |
| Mother's Income from Work |  |  |  |
| Student's Income from Work |  |  |  |
| Parents' Taxes Paid |  |  |  |
| Parents' Worksheet C |  |  |  |

*Analysis** **of Quality Assurance Program Data: 2007-08 & 2006-07*

**Notes:**
Interpret arrows – same story  for 3 fields, similar story for fourth.

## Common Strategies for Selecting Students for Verification

| Description of Strategy | % of schools |
| --- | --- |
| Specific value  | 75% |
| Greater than  | 55% |
| Less than | 52% |
| Unlikely combination  | 51% |
| Range | 49% |
| Zero or blank | 49% |

*Analysis** **of Quality Assurance Program Data: 2007-08*

**Notes:**
In addition to coding what fields criteria used we coded HOW a field was used.
V – selects specific value or limited number of values (e.g., estimated filers, zero EFC)
G – selects all records greater than an indicated values (e.g., worksheet B gt 3000)
L – selects all records less than an indicated value, include ranges that begin at zero.  (e.g. agi lt 25,000)
U – unlikely combinations of information (e.g., taxes paid > 25% AGI)
R – selects all records in a range between two non zero values (e.g. agi > 30,000 and agi < 60,000)
Z – selects zero or blank

## Observed Relationship Between School Using Common Strategy and Verification Outcomes

| Data Field | Unnecessary Verification | Prevent Over-  Awards | Prevent Under- Awards |
| --- | --- | --- | --- |
| Specific value  |  |  |  |
| Greater than  |  |  |  |
| Less than |  |  |  |
| Unlikely combination  |  |  |  |
| Range |  |  |  |
| Zero or blank |  |  |  |

*Analysis** **of Quality Assurance Program Data: 2007-08 & 2006-07*

## Less Common Strategies

| Description of Strategy | % of schools |
| --- | --- |
| Student data field applied to dependent records  | 28% |
| Any non-zero or blank value | 13% |
| Excludes (rather than selects) records for verification  | 13% |
| Different from a previous value from this award year  | 4% |
| Different from value in previous award year | 2% |
| Element in a statistical model | 1% |

*Analysis** **of Quality Assurance Program Data: 2007-08*

**Notes:**
Explain why no observed relationship table – too few schools to achieve statistical significance.
S – student field applied to dependent records (leave out if used for independent)
A – any non-zero or blank value (e.g. any Worksheet A)
E – excludes records with the indicated value(s)  
D – different from a previous 2008-09 transaction
Y – different from previous award year
M – an element in a statistical model  

## Apply 2007-08 findings to your analysis

- Fields to consider using in your criteria
- EFC
- Student filed tax return? *****
- Father’s income from work *****
- Mother’s income from work *****
- Student’s income from work

- Strategies to consider using in your criteria
- Specific values *****
- Greater than
- Less than *****
- Range

*** Associated with reduced ability to prevent Pell under-awards**

**Notes:**
Based on our analysis of all QA schools here are fields and strategies to consider in improving the efficiency of your QA schools – note that the entries with asterisks may lead to the undesirable side effect of increasing the number of Pell-under-awards missed by verification. 

## Implications of 2007-08 Analysis

- Generalizations about Quality Assurance School verification are very hard to make
- Roughly half of participating schools use non-ISIR information when selecting students for verification
- School criteria often reduce “unnecessary” verification
- The cost of this efficiency seems to paid solely by the reduced capacity to correct Pell under-awards; the tendency to prevent Pell over-awards was not affected

**Notes:**
Read bullets

Non-ISIR data can be added to ISIR tool as user defined fields.  You can’t analyze these often critical components of your verification efforts unless you bring that data in to the tool. Guidance available on our website.

More involved school verification procedures (more fields, more strategies) do tend to reduce unnecessary verification, but at the expense of missing Pell under-awards.  The tendency to verify a large segment of the Pell population prevents a similar “side-effect” for Pell over-awards.

## Institutional Profiles 2007-08

- Pell improper payments prevented by your school’s verification efforts
- Program-wide averages provided for context

**Notes:**
Now let’s turn to your Profiles

We will describe how the values were calculated using the program-wide findings.

## Analysis of Quality Assurance Program Data: 2007–08

*Analysis** **of Quality Assurance Program Data: 2007–08*

**Notes:**
First we simply summed the information from the 141 schools that supplied ISIR data in 2007-08

329 million in Pell would have been award based on initial data.

45 million dollars in over-payments were prevented (subtracted from this total) by school verification

19 million in under-payments were prevented (added to this total) by school verification

Netting 304 million in Pell grants

## Analysis of Quality Assurance Program Data: 2007–08

*Analysis** **of Quality Assurance Program Data: 2007–08*

**Notes:**
Then we made a line chart from the running totals from these sums

Look familiar?

In the profiles we simply divided this sums by the number of schools (141) to get the averages.

$329 million  subtract the $-45 million in overawards = $284

$284 million subtract the $19 million in underawards = $304 million (rounded up)

Apply the same calculations to the line chart on your own profile.

## Analysis of Quality Assurance Program Data: 2007–08

*Analysis** **of Quality Assurance Program Data: 2007–08*

**Notes:**
The first two columns are simply the improper payment figures in the first graph re-expressed as percent of initial Pell awards.

The third total change column is the sum of the ABSOLUTE value of over-awards and under-wards.  Positive 13.6 + positive 5.9 = 19.5

Remember absolute value = the numeric value of a real number without regard to its sign for example, the absolute value of -15.9% is 15.9.

The fourth net impact is the sum of the real values. Negative 13.6 + positive 5.9 = -7.8 
The figure suggests that collectively, QA schools  prevented almost 8 percent of improper pell grant payments through institutional verification efforts.

See these figures in the bottom chart of your profile.

## Example 2007-08 Profile

*Analysis** **of Quality Assurance Program Data: 2007–08*

**Notes:**
Look at your profiles and explanation sheet

You can compare the effect you are having to  the average QA school.

You can also compare the effect of your school verification to the potential for improper payments found in last year’s sample.  Your percentages for both over- and under- awards in the bottom chart on this years “Effect” profile should be higher then entries on “Potential” profile based on last year’s random sample data.  The reason being is that the records in the 07-08 were specifically targeted because they met criteria that you established to prevent over and underawards.

## Profiles Differs from Year to Year

- 2007-08 provide data of improper payments _*PREVENTED*_ by school verification
- 2006-07 provided data on estimated levels of _*POTENTIAL*_ improper payments in a school’s applicant population

**Notes:**
Here in lies the primary difference between 06-07 and 07-08.  

Questions

Break!!!

## 2008- 2009 Timeline for the ISIR Analysis Tool

| February, 2008 | 2008-2009 Version of IA Tool on the Web is available |
| --- | --- |
| March – September, 2008 | Begin implementing your 2008-2009 verification selection criteria & complete verifying your applicants;
Submit Sampling Worksheet |
| October, 2008 | Prepare flat file, upload 2008-2009 data, set school verification flag  |
| November, 2008 – February, 2009 | Analyze Data; FSA collects school data; Notify QA Rep. of institutional verification criteria for 2009-2010 by February 2009 |

**2008- 2009 Timeline for the ISIR Analysis Tool**

**Notes:**
Hopefully, everyone here is on schedule and has met the requirements outlined in the timeline.  Here, we provide you with a timeline for 08-09 of when you should begin activities that involve the ISIR Analysis Tool. 

By now, you should have verified those students selected in a random sample, prepared and uploaded a flat file and began analyzing your data so that you can revise, update, or validate what your verification program will look like for 09-10.  The random sample should contain not less than 350 (unless you are a small school and verified less than 350 ISIR records). 

## 2009- 2010 Timeline for the ISIR Analysis Tool

| February, 2009 | 2009-2010 Version of IA Tool on the Web is available |
| --- | --- |
| March – September, 2009 | Verify students selected by you  school verification criteria, keep track of initial (selected) and paid on (after) ISIR transaction numbers |
| October, 2009 | Prepare flat file, upload 2009-2010 data, set school verification flag  |
| November, 2008 – February, 2009 | Analyze Data; FSA collects school data; Notify QA Rep. of institutional verification criteria for 2010-2011 by February 2010 |

**2009- 2010 Timeline for the ISIR Analysis Tool**

**Notes:**
Here, we provide you with a timeline of when you should begin activities that involve the ISIR Analysis Tool for 09-10 You will verify those students you have selected for institutional verification. In the fall, you’ll prepare and upload a flat file and begin analyzing your data so that you can revise, update, or validate what your verification program will look like for 09-10. You can upload a maximum of 1,500 records in to the tool to analyze. We recommend not less than 350 (unless you are a small school and verified less than 350 ISIR records). 

## QA Schools Must use these Verification Status Codes

- _V__	__VERIFIED_ by the school.
- _S __	__Selected_ by CPS but NOT verified because the ISIR record did not meet the school verification criteria
- _W__	__Selected_ by the school and the Pell was disbursed without documentation.
- BLANK	Not selected

**Notes:**
When you disburse a Pell grant, you must report through Common Origination and Disbursement (COD) the student’s verification status even if they were 	not selected for verification.

V	You have verified the student. This includes students selected by the CPS (that you verified)  and those your school chose to verify based on its own criteria.
S	The CPS selected the student for verification, but you did not verify him because you already reached the 30% verification threshold or because your school participates in the Quality Assurance Program and the student’s application did not meet your school’s verification criteria. 

W	The student was selected for verification by your school and you chose to pay a first disbursement of Pell without documentation. This code must be updated once verification is complete, or COD will reduce the Pell grant to zero. 

Blank	 Report a blank if you have not performed verification because neither the CPS nor your school selected the student.

If you have W’s – you want to make sure they are updated for the second disbursement or COD will reduce the Pell Grant to 0.

## Tracking your W Codes

- Access the Verification Tracking Report in COD

**Notes:**
Periodically, the Department has the COD system send schools a list of students with a status code of W and a warning that the school must take action to complete verification (if not already completed) and update the student’s verification status in the COD system. The school cannot make any subsequent disbursement for the student until the verification status is updated. However, the QA schools were omitted from this notification. 

It is important that all schools update the verification status of W's by a specified date because COD system reduces the student’s disbursement amount to zero.  We are working on having these reports generated for you. In the meantime, we recommend that you obtain the report from COD for the 05 06 and 06 07 award years.

## Verification Status Report

- To retrieve the report via a Data Request:
- Log into the COD web site, select the Batch option from the top tool bar
- Select Grant Data Requests, select New Request; from the Report Type dropdown box, select Verification Status Report

**Notes:**
If you have problems accessing the report, please call  COD Customer Service Representative (CSR) at 800/4PGRANT  
(800/474-7268)

## Verification Status Report-Cont’d

- Make the appropriate selections for Award Year, Request Type and Data Type, and click the submit buttonThis process will produce a report with current data from COD. *Remember, if there is no student for whom a code is missing, the file will be sent but the results will be blank**.*

**Notes:**

Turn it over to David.

## EFC Impact Analysis Report

- What’s here?
- Graph or table with counts of changes to any and the five most commonly changed critical ISIR fields by initial EFC value range
- How can you use it?
- Determine if changes to ISIR fields are concentrated in an initial EFC range
- Target verification or outreach efforts

**Notes:**

Simply by comparing the height of columns (or magnitude of values in the text version) you can identify which EFC value range is most responsible for changes to the identified ISIR field.

Knowing how initial EFC level relates to corrections to ISIR information may improve your targeting of educational or verification efforts. For example, if you find that relatively few of the corrections to a given error prone ISIR field occur among the zero EFC students selected by your School verification criteria, you may decide to eliminate zero EFC students from your discretionary verification efforts.

## Slide 65

**Notes:**
Explain the example screenshot:

As is the case with all reports, criteria used are listed at the top of the results. Note that you have the option to View Report as Text. This will generate a table containing the same information as the graph. (See the report above viewed as text on the next page). The ISIR Fields being graphed are identified across the top of graphic image. Note that the report is actually six separate graphs. The first displays results for records with any change to ISIR information. The next five graphs display the results for the five most common changes to EFC critical fields – those that factor into the calculation of EFC. The graph displays Zero EFC records in purple; 1 – 4041 in green; and non-Pell eligible in pink.

## New Shared Report: School_Verification_Analysis 

**Notes:**

Based on the 2006-07 efficiency reports presented earlier.

Briefly explain new report

Plug advanced hands on session

## Tips for Applying Analysis

- KISS
- Know
- Integrate
- Summarize
- Sell
- Telling quotations

**Notes:**
Ask if anyone knows what KISS (Keep it simple stupid) means?

This is good advice but HOW do you KISS 

My take on how.

I will introduce each with a quote

##  Last night somebody broke into my apartment and replaced everything with exact duplicates .... When I pointed this out to my roommate he said, “Do I know you?”

**Steven Wright**

**Notes:**
Steven Wright is one of my favorite comedians.  (Read joke) 

The point of the joke is the two humorous meanings of the roommate’s retort. The first, taking the nonsensical premise to the logical conclusion. If everything was replaced by duplicates then the duplicate roommate would not know Steve.  The second and more plausible interpretation would be any a normal person questioning their relationship with a housemate who just made such an off the wall claim.

It is important to keep what you already know in mind when you are interpreting new data – it makes it so much easier. All to often we tend to forget everything else we know when we look at numbers. 

## Know

- What you do
- What has changed
- Remember what you don’t know

**Notes:**
Never forget what your verification criteria are and what information you verify when interpreting your results. 

Remember what has changed – your criteria, the FAFSA, your tuition, state aid – when making sense of your analysis.

Also remember what you don’t know.  When working with the tool you don’t know the accuracy of changes to ISIR information unless that student was verified. If your school doesn’t verify all elements of the federal verification during your normal school verification activities you only don’t know the accuracy of changes to those fields that were not verified.  

Take findings from sample with the right amount of salt.  On average they are representative of populations, but there is always a small chance that you sampled a group of students that is not. The larger the sample the less this chance.

##  In essence, science is a perpetual search for an intelligent and integrated comprehension of the world we live in

**Cornelius Bernardus Van Neil**

**Notes:**
The second word in the acronym is Integrated

Read quote

Analysis is the process of breaking a complex topic into smaller parts to gain a better understanding of it.  That understanding only comes when you “put the results back together” with an integrated picture of your verification efforts.

## Integrate

- Current practices
- Possible practices
- Previous analysis
- Other’s analysis

**Notes:**
What should you be integrating? (Read bullets)

Put plug in for program-wide analysis available on the website.

##  Summarize: 

- “I have made this [letter] longer, because I did have not had the time to make it shorter.”
***Blaise Pascal***

**Notes:**
Pascal quote reminds us that it takes time to summarize things

Read quote

One of the first projects I led at AIR were issue briefs based on larger Department of Education Studies.  These reports were printed on a single piece of paper and therefore had a hard 2 page limit.  The budget for these “briefs” was roughly 10K each.  Which I thought was a very comfortable cushion until I started work on it.

## Summarize

- For your audience
- For you

**Notes:**
The benefits of a good summary are obvious to the audience. People have limited attention spans and time to devote to your analysis. The  shorter you can make your analysis the greater chance you have of getting your point across.  You often then need details to back up your point, but your audience can see forest better on a map you draw than they can from photographs of all the trees.

The process of summarizing (for your audience) often leads you to a fuller understanding of your results. Think about the last time you taught somebody something and I bet at least half of you walked away from the lesson you gave knowing the subject a little better yourself.

##  Between two products equal in price, function and quality, the better looking will out sell the other.

**Raymond Loewy**

**Notes:**
The second “S” is sell

Read quote

Like a lot of people when I finished college I wasn’t sure what I wanted to do with my life. One thing I knew for “sure” was that I didn’t want to work in sales. I though I couldn’t handle the pressure and rejection. If I had known then how much selling was inherently involved in analysis, I may not have pursued it. Any analysis of a business process is looking for a better way to do things. In our case who should we add and who should we subtract from verification. Know matter how “brilliant” your analysis is, you are going to have to sell the improvement to other people to implement change.  As the quote reminds us, looks matter in this process.

## Sell

- Data driven decisions
- Identify how to improve
- Recertify the status quo
- Document the value of your efforts
- Improper payments
- Outreach efforts
- Accurate aid awards

**Notes:**
What are you selling?

Your selling your proposal for moving forward based on a data driven decision.
How to change or why to stay the same

Let’s face we all increasingly have to sell the value of our efforts.  The Sample Summary Report provides a the means of documenting the scale of improper payments prevented by your school verification. Comparing data over time will allow you to demonstrate improved accuracy in initial reports of problematic data fields that were the targets of education outreach efforts. Finally the magnitude of changes to EFC detected by the Tool can help you remind people of the need to verify a selective subset of students.

## Use graphs when you can

| Site | X | Y | Z |
| --- | --- | --- | --- |
| Dallas | 23 | 35 | 54 |
| Las Vegas | 12 | 37 | 46 |

**Notes:**
Just a reminder:
Visual representation of data are much easier (and hence attractive) to audiences.

## Questions

**Questions**

## Contact Us

**Contact Us**

**Regional Representatives:**
**Regions 1, 2, 5** (CT, IN, IL, MA, ME, MI, MN, NH, NJ, NY, OH, RI, VT and WI)Holly Langer-Evans | _[Holly.Langer-Evans@ed.gov](mailto:Holly.Langer-Evans@ed.gov)_
**Regions 3 and 4** (AL, DC, DE, FL, GA, KY, MD, MS, NC, PA, SC, TN, VA and WV)Anne Tuccillo | _[Anne.Tuccillo@ed.gov](mailto:Anne.Tuccillo@ed.gov)_
**Regions 6 and 7** (AR, IA, KS, LA, MO, NE, NM, OK, and TX)Anne Tuccillo | _[Anne.Tuccillo@ed.gov](mailto:Anne.Tuccillo@ed.gov)_
**Regions 8, 9, 10** (AK, AZ, CA, CO, HI, ID, MT, ND, NV, OR, SD, UT, WY and WA)Michael Cagle | _[Michael.Cagle@ed.gov](mailto:Michael.Cagle@ed.gov)_
**Program Director:**Barbara Mroz | _[Barbara.Mroz@ed.gov](mailto:Barbara.Mroz@ed.gov)_
**Program Analysts:**David Rhodes | _[David.Rhodes@ed.gov](mailto:David.Rhodes@ed.gov)_Warren Farr | _[Warren.Farr@ed.gov](mailto:Warren.Farr@ed.gov)_