# Presentation: 173218

## Slide 1

## Sampling and You

- Preliminary results from a pilot study on the return of sampling

- Tsze Chan, The American Institutes for Research
- David Rhodes, Caliber Associates

## Purpose of the Session 

- Share the results from a pilot study of sampling
- Discuss the implications for the possible return of sampling to the Quality Assurance Program

## Problems with sampling in the past

- Too late: after the award
- Cumbersome process: additional data collection, data entry, tracking, award adjustment
- Public relations disaster: disgruntled students and parents

## Reasons for Trying Again

- The need of QA Program participants to monitor the effectiveness of their verification procedures
- Changes within FSA

## Rationale for a Pilot Study

- To detect and address the problems of sampling
- To determine whether results from sampling are actually helpful
- Before! re-imposing it on everyone

## Pilot Institutions

- How were institutions selected?
- Which schools participated?
- What were their obligations?

## Problems Encountered by Pilot Institutions

- Timing: when to draw sample
- Tracking: how to keep track of the additional “documenting” of FAFSA values
- Importing: how to import the right ISIR records into the QA Tool

## Problems with Timing 

- “Emerging” population
- Some applicants will not be eligible
- Some applicants will not enroll
- Some applicants apply “late” in the process
- Competition with other goals
- Efficient aid packaging
- Recruitment

## Solutions for Timing 

- Allow flexibility (before disbursement)
- First valid ISIR
- After student accepted by institution
- After student pays enrollment deposit
- After student accepts aid package
- FSA will provide guidance
- Don’t misapply the old 95 percent response rate rule

## Problems with tracking the documentation process

- Managing the verification of sampled students
- Submitting ALL changes to the ISIR records of sampled students to the central processor

## Solutions for tracking the documents

- Integrate the sampling process into existing institutional verification system(s)
- User database “Sample” field
- Institution verification “S” code
- Treat sampled students who do not provide documentation the same as other verification scofflaws
- Separate stand alone sample “system”
- EDExpress just for doing sample
- Institution created database

## Problems: Importing ISIR’s into Quality Analysis Tool

- Multiple ISIR’s may be generated by students before, during, and after sample documentation
- Keeping the “initial” ISIR records straight
- Keeping the “paid on” ISIR records straight
- Not a problem if using EDExpress to process aid

## Mainframe Solution A – writing SARA files

- Retrieve the ISIR batch from your SAIG mailbox
- Upload the data into your school’s system
- Request an ASCII file of the data from computer support staff
- Import ASCII file into ISIR Analysis Tool
- Correct records as necessary in your school’s system and send the changes to the CPS
- Repeat steps 1 through 5. As you import subsequent ISIR files into your system and the ISIR Analysis Tool

## Mainframe Solution B – use EDExpress

- Retrieve the ISIR batch from your SAIG mailbox
- Before working with file on your system, copy it to a folder so you can import into EDExpress
- Import data into EDExpress (keep all copies of imported files)
- Import data from EDExpress into ISIR Analysis Tool
- Correct records as necessary in your school’s system and send the changes to the CPS
- Repeat steps 1 through 5. As you import subsequent ISIR files into your system and the ISIR Analysis Tool

## Summary of Solutions 

- Consider the “population” at various “sampling windows”
- FSA will provide guidance on pros and cons of different timing
- Integrate the documentation process into the existing system
- Follow guidance that will be in the help text of 2003-2004 ISIR Analysis Tool

## Is it worth the effort? 

- FSA: program wide improvement of targeting the “right” students
- Institution: increase the usefulness of reports from the Quality (ISIR) Analysis Tool

## Results: Size of the Problem 

- Within the samples that were one hundred percent documented, 42 percent of cases experienced a EFC change
- Within the control group, only 4 percent of the EFC changes were self-reported

## Results: Accuracy of Institutional Verification

- Of those who had EFC changes (sample),
- 62% would be identified by institutional verification profiles
- 48% would be identified by CPS edits

## Results: Efficiency of Institutional Verifications 

- 50% of those selectable by institutional verification resulted in EFC changes
- 45% of those selectable by CPS edits resulted in EFC changes

## Results: Magnitude of EFC changes not detected by verification

- 30% of those not selected by institutional verification had EFC changes
- 42% of those not selected by CPS edits had EFC changes

## ISIR Field Changes 

- Dependent Students
- Parent’s Income Tax Paid 	28.3%
- Parent’s AGI			21.9%
- Parent’s Worksheet B		21.8%
- Parent’s Tax Return Status	20.7%
- Student’s Income Tax Paid	16.5%
- Parent’s Worksheet C		13.8%

## ISIR Field Changes 

- Independent Students
- Student’s Income Tax Paid	20.9%
- Student’s AGI			15.3%
- Student Worksheet B		14.5%
- Student Tax Return Status	 9.1%

## ISIR field changes and verification 

- Problematic ISIR fields featured in pilot schools’ verification profiles
- Three basic strategies
- Population at risk
- Unusual values
- Suspect groups
- Fifty percent “efficiency” suggests that profiles could be narrowed

## Population at Risk Strategy

- Estimated tax filers
- (Parents’ Tax Return Filed? = “2”)
- (Student’s Tax Return Filed? = “2”)
- Multiple persons in college
- Dependency Status=“D” and Parents’ number in college > “1”
- Dependency Status=“I” and Student’s number in college > “1”

## Unusual Value Strategy

- Unusually High Taxes
- Dependency Status=“D” and Parents’ U.S. Income Taxes > .3 * tbl_E_Parent.PARAGI
- Dependency Status=“I” and Student’s U.S. Income Taxes > .3 * tbl_E_Student.STUDAGI

## Suspect Group Strategy

- Independent other than by age or graduate student
- (Dependency Status=“I” and Born before 01/01/1979<>“1” and Working on a Master’s or Doctorate <>“1”) and (Have Children you support? =“1” or Have Dependents other than children or spouse=“1” or Is student married=“1” or Orphan or Ward of Court=“1” or Veteran of U.S. Armed forces?=“1”)

## Narrowing Focus

- Run the list verification by code report using queries to detect substantial EFC change (Activity Guide #3)
- Run transaction comparison reports for cases that meet the criteria for each profile and look for differences between cases with and without EFC changes

## Implications

- Sampling will not be required before 2004-2005
- Institutions need lead time to plan integration of sampling into their systems
- FSA needs to provide guidance
- Richer analysis possible

- We appreciate your feedback and comments.  We can be reached:
- 	Email:  _[tchan@air.org](mailto:tchan@air.org)_