# Presentation: 248182

## Meeting Data Collection Challenges of the Future

**Meeting Data Collection Challenges of the Future**

**James Griffith**
**Ted Socha **
**Thomas Weko**
**Postsecondary Studies Division**
**National Center for Education Statistics**

## Purpose of the Presentation

**	**
**To show how our “technological innovations” have responded to challenges facing sample survey and longitudinal study data collections.**

## Overview

**Provide overview of postsecondary sample survey and longitudinal studies as prelude to ...**
**Major challenge – Encourage participation of institutions and students **
**Technological innovations to meet this challenge **
**Future challenges, concluding remarks**

## Postsecondary Longitudinal and Sample  Survey Studies

**National Postsecondary Student Aid Study (NPSAS)**
**Beginning Postsecondary Students Longitudinal Study (BPS)**
**Baccalaureate and Beyond Longitudinal Study (B&B)**
**National Study of Postsecondary Faculty (NSOPF)**

**Notes:**
FOR EACH:
Name
Focus
Design
Periodicity
Sample size

## Primary Purposes

_**NPSAS**_
**NPSAS:08, includes 125,000 undergraduate and 13,000 graduate/first-professional students enrolled in 1,963 institutions**
**Describes how students and their families pay for postsecondary education and the role of federal student aid **
**Describes undergraduate and graduate student populations, including topics such as civic participation, community service, educational aspirations, life goals, disabilities **
**Provides rich database for postsecondary research and policy analysis, data gathered on over 1,100 variables **
**Responds to emerging federal policy interests, such as debt level, use of private loans, training in STEM majors, and state-level financial aid issues **

## Primary Purposes

_**BPS**_
**Includes about 18,500 first-time beginning students identified in NPSAS, followed 2 years and 5 years later**
**Provides data on student “flow” in and out of postsecondary education, such as persistence, transfer behavior, attainment**
**Gathers data on initial work experiences of those who obtain certificates or 2-year degrees**
_**B&B**_
**Includes about 23,500 bachelor degree completers identified in NPSAS, followed 1 year, 5 years, and 10 years later**
**Provides important information on graduate education, work, and career through retrospective information on:**
**	****-- postsecondary experiences **
**	****-- paths taken to the degree award, time to degree completion **
**Focuses on “teacher pipeline,” interest in teaching, preparation, and job search**

## Chronology of PLSSS Studies

- BPS = Beginning Postsecondary Students Longitudinal Study		B&B = Baccalaureate and Beyond Longitudinal Study

## Major Data Collections

**NPSAS occurs in staged sampling periods requiring data collection at several levels:**
**Institution data collection**
**Student interview data **
**Other administrative data**

## Data Collections by Study

|  | NPSAS | BPS | B&B |
| --- | --- | --- | --- |
| Institution-Level Data |  |  |  |
|     Institution characteristics (IPEDS data) | × | × | × |
|     Student records (institutional and state aid) | × |  |  |
|     Transcripts |  | × | × |
| Student Interview Data | × | × | × |
| Other Administrative Data |  |  |  |
|     Central Processing System (e.g., FAFSA)  | × | × | × |
|     NSLDS (Pell and loan files) | × | × | × |
|     National Student Clearinghouse (enrollment) | × | × | × |
|     ACT / SAT / Praxis |  | × | × |

## The Challenge

## Declining participation of institutions and students
Missing data at the case-level and item-level   
Complicated by ...
	-- Diversity in access and use of technology across institutions 	(about 6,700) and  students (about 19 million)
	-- Data privacy/security concerns 
		Government = Misuse of study data
		Institutions = Risks of releasing student information 
		Families = Privacy, identity theft
	-- Tight schedule for data collection, processing, and delivery – 	from lists of students gathered in May to data delivery in 	January
	-- Students as transient population 
GOAL – To maximize the usefulness and quality of the data vis-à-vis the challenge and complicating factors ... 

_**Missing data**_** at the case-level and item-level   **
**Complicated by ...**
**	****-- Diversity in access and use of technology across institutions ****	****(about 6,700) and  students (about 19 million)**
**	****-- Data privacy/security concerns **
**	****	****Government = Misuse of study data**
**	****	****Institutions = Risks of releasing student information **
**	****	****Families = Privacy, identity theft**
**	****-- Tight schedule for data collection, processing, and delivery – ****	****from lists of students gathered in May to data delivery in ****	****January**
**	****-- Students as transient population **
_**GOAL**_** – To maximize the **_**usefulness**_** and **_**quality**_** of the data vis-à-vis the challenge and complicating factors ... **

## Institutional Data Collection

_**Challenge**_**:****  Maximize institution participation by acquiring enrollment lists for sampled institutions.**
_**Innovations:**_
**Contacted early (fall contact for spring collection)**
**Established “Study Coordinator” (IRP as initial commitment)**
**Provided Help Desk (10-12 staff, Mon-Fri work hours)**
**Provided easy access through Website (information, secure login, CADE)**
**Ensured confidentiality / security (FERPA documents, IRBs)**
**Offered multiple options for participation (secure ****	****website, encrypted fax, hard copy)**

**Notes:**
Contact in the fall to collect lists in January.

10 to 12 people staff institution contactor help desk, regular business hours with toll free line

Website provides secure log in to provide student lists and, eventually, student record abstractions.

Multiple options for providing lists – upload to secure website, encrypted email, fax, hard copy

## Institutional Data Collection

_**Challenge**_**: Minimize institution nonresponse.  **
_**Innovations:**_
**Developed real-time monitoring of characteristics of institutions (IMS)**
**Performed bias analysis for < 85% participation overall and/or within strata, NCES standard**
**Made weight adjustment to reduce bias at the institution/unit level**

**Notes:**
Describe 2-stage sampling

Tie to the standards – the standards demand bias analysis to see what bias exists.  Weighting then need to reduce the bias.

## Real-time Monitoring System

**We can see our response statistics in real-time.  **
**We then target “under-responding” institutions for participation in near real-time.**

## Institutional Data Collection

_**Challenge**_**:****  Maximize student record collection.**
_**Innovations:**_
**Provided accessible Web-based instrument (CADE)**
**Provided Help Desk (Mon-Fri during normal work hours) **
**Negotiated reimbursement to “incentivize” participation**** **

## Example – Web CADE Student Record Abstraction

## Student Record Abstraction Method – NPSAS:04

| CADE abstraction method  | Institutions providing CADE |  |  | Total students1  |  |
| --- | --- | --- | --- | --- | --- |
|  | Number | Percent2  |  | Number | Percent2  |
|  Total | 1,300 | 100.0 |  | 103,620  | 100.0 |
| Abstraction method |  |  |  |  |  |
| Web-CADE | 860 | 65.9 |  | 48,860 | 47.2 |
| Data-CADE | 280 | 21.1 |  | 33,210 | 32.0 |
| Field-CADE | 170 | 13.1 |  | 21,550 | 20.8 |

**Student Record Abstraction Method – NPSAS:04**

- 1 The total represents the number of students sampled from institutions that completed computer-assisted data entry (CADE) and may include students who were classified as study nonrespondents.
- 2 Percentage of total number of eligible institutions/students.
- NOTE: Detail may not sum to totals because of rounding.
- SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

**Web CADE preferred choice**

## Institutional Data Collection

_**Challenge**_**:  ****Maximize student transcript collection.**
_**Innovations**_**:**
**Addressed confidentiality/privacy concerns through secure fax, encrypted email, FTP transfer **
**Negotiated reimbursement to “incentivize” participation**** **

## Example – Web Transcript Collection

## Student Interview Data Collection

_**Challenge**_**:**** Maximize student interview collection.**
_**Innovations:**_
**Redefined completed case  **
**Employed multiple tracing vendors, e.g., CPS, NCOA, Telematch, Equifax, Experian,TransUnion**
**Offered response mode options fitting to the population (self-administered via Web, telephone, face-to-face) **
**Provided accessible Web-based instrument**
**“****Incentivized” for early response**
**Provided Help Desk (7 days a week)  **

## Cases Have Multiple Data Sources ...  

_**Primary sources**_
**	****Institution records (CADE)****	****	****95%**
**	****Student interviews ****	****	****	****70%**
**	****Federal aid applications (CPS)****	****60%**
_**Combinations of source**_
**	****All three primary sources****	****	****40%**
**	****Two sources****	****	****	****	****50%**
**	****One source****	****	****	****	****10%**
_**Additional sources**_
**Federal loans & Pell Grants (NSLDS)****	****50%**

## Allowing for Redefining a Case 

**Student case is a “complete” if valid data exist for:**
**	****-- Student type**
**	****-- Birth date or age**
**	****-- Gender,**
**	****-- And at least **_**8 of the 15**_** variables:  **
**Dependent status, marital status, any dependents, income, expected family contribution, degree program, class level, first- time beginner, months enrolled, tuition, received financial aid, received non-federal aid, student budget, race, and parent education.**

## Difficult-to-Locate Students

**Difficult-to-Locate Students**

## Even so, Some Are Locatable 

**Even so, Some Are Locatable **

## Example –B&B:08/09 Self-Administered Web Interview

## Student Mode Choice for Interview

|  | Completed interviews |  |
| --- | --- | --- |
|  | Number | Weighted percent |
|    Total | 62,220 | 100.0 |
| Self-administered  | 28,710 | 46.7 |
| Early response | 17,100 | 27.5 |
| With prompting | 11,610 | 19.2 |
| Interviewer-administered | 33,510 | 53.3 |

- NOTE: Detail may not sum to totals because of rounding.
- SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

**Almost 1⁄2 respond via web.  1/3 receive incentive.  Growing trend.**

## NPSAS:04 Experience with Incentives

## “Incentivized” Respondents

**$20 vs. $30**

**Web rises in preferred choice**

## Student Interview Data Collection

_**Challenge:**_**  Minimize bias from **_**case-level**_** and **_**item-level**_** missing data. **
_**Innovations:**_
**Made weight adjustment at case or student-level **
**Imputed item-level missing data by:   **
**-- Logical imputation or**
**-- Statistical imputation with nearest similar neighbor donor **

## Who Responds and Who Doesn’t

**Who Responds and Who Doesn’t**

## Nonresponse bias was estimated using statistical modeling.   
Dependent variable = Responded or Not responded.
Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.: 
	--  institution type; region; institution enrollment from 	IPEDS file (categorical); student type; FTB status; 	etc.
Weight adjustments = Coefficients for predictor variables used for weighting.

**Dependent variable = Responded or Not responded.**
**Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.: **
**	****--  institution type; region; institution enrollment from ****	****IPEDS file (categorical); student type; FTB status; ****	****etc.**
**Weight adjustments = Coefficients for predictor variables used for weighting.**

**Nonresponse Procedure**

## Example Weight Adjustments

**Example Weight Adjustments**

**Already seen -- hard to locate and under- respond.**
**So, “double-up” responding case.**

## Imputation Procedure

**Logical – use other data sources to determine missing variable values for given case (e.g., substitute FAFSA for institution CADE).**
**Statistical –separate cases into dissimilar groups (using a defined set of variables) such that respective group members are alike.**
**Membership in final group determines donor candidates or “nearest neighbor.”**
**For cases having missing variable values, such values are “borrowed” from the “nearest neighbor.” **

## Item-Level Missing Data

**Item-Level Missing Data**

## Future Challenges to Data Collections

**Increased concerns about security / confidentiality  -- both institutions and students**
**Reduced access to students **
**--****	****Increased liability concerns of institutions to release student-level contact information **
**--****	****Greater cell phone usage**
**Raised respondent expectations– offering $ incentives  **

## Products

_**Data for analysis**_
**	****Data analysis system, **
**	****	****http://nces.ed.gov/dasol**
**	****Restricted data file, ****	****http://****nces.ed.gov/pubsearch/licenses.asp**** **
_**Reports**_
**	****On methods ...  ****	****http://nces.ed.gov/pubsearch/pubsinfo.asp?pu****	****bid=2006180**
**	****On content ...  ****	****http://nces.ed.gov/pubsearch/pubsinfo.asp?pu****	****bid=2006186**

## How to be Informed of Products

_**http://ies.ed.gov/newsflash**_

## Contact Information

**Tom.Weko@ed.gov**
**	****202-502-7643**
**James.Griffith@ed.gov **
**	****202-502-7387**
**Ted.Socha@ed.gov**
**	****202-502-7383**