# 2012 **/1 4**  **B EGINNING**  

# P **OSTSECONDARY** 

# S **TUDENTS**



# L **ONGITUDINAL** 

# S **TUDY** 

# : **(B PS**  **:12 /14**  **) F**  **IELD** 

# T **EST**

**S upporting Statement Part**  **s B**  **and C**

**Request for OMB Review OMB #**  **1850 -**  **0 631**  **v. 7** 

Submitted by National Center for Education Statistics

U.S. Department of Education **October 1**  **1, 201**  **2**III 

## Contents **List of Tables ................................**  **................................ ................................**  **................................ ..............**  **iii List of Exhibits**  **................................ ................................**  **................................ ................................**  **............ iv**  **B. Collection of Information Employing Statistical Methods**  **................................ ................................**  **. 1**  1. Respondent Universe and Sampling ................................ ................................ ................................ ................ 1 a. NPSAS:12 Field Test Institution Universe and Sample ................................ ................................ ............ 1 b. NPSAS:12 Field Test Student Universe and Sample ................................ ................................ ................ 3 c. Identification of FTBs in NPSAS:12 ................................ ................................ ................................ ........ 4 d. BPS:12/14 Field Test Sample ................................ ................................ ................................ ................... 5 2. Procedures for the Collection of Information ................................ ................................ ................................ .. 6 a. Statistical methodology for stratification and sample selection ................................ ................................ 6 b. Estimation procedure ................................ ................................ ................................ ................................ 6 c. Degree of accuracy needed for the purpose described in the justification ................................ ................ 6 d. Unusual problems requiring specialized sample procedures ................................ ................................ .... 7 e. Any use of periodic (less frequent than annual) data collection cycles to reduce the burden ................... 7 3. Methods for Maximizing Response Rates ................................ ................................ ................................ ....... 7 a. Locating ................................ ................................ ................................ ................................ ..................... 7 b. Interviewing Procedures ................................ ................................ ................................ ............................ 8 c. Quality Control ................................ ................................ ................................ ................................ ........ 10 4. Tests of Procedures and Methods ................................ ................................ ................................ .................. 10 a. NPSAS:12 ................................ ................................ ................................ ................................ ............... 10 b. B&B:08/12 ................................ ................................ ................................ ................................ .............. 11 c. ELS:2002/12 ................................ ................................ ................................ ................................ ........... 13 d. BPS:12/14 Field Test ................................ ................................ ................................ .............................. 15 5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study .................. 22 a. Other Contractors’ Staff Responsible for Conducting the Study ................................ ............................ 22 **C. Overview of Analysis Topics and Survey Items**  **................................ ................................**  **................ 22**  **References ................................**  **................................ ................................**  **................................ ...................**  **22** 

## List of Tables Table 1. BPS:12/14 Sample Size by institution characteristics: 2011 ................................ .................... 6 Table 2. Start dates for BPS:12/14 field test interviewing, b y institution sector ................................ .... 9 Table 3. Summary

of Field Test and Full - Scale Data Collection Designs, by Study ........................... 14 Table 4. Distribution of the BPS:12/14 field test sample, by sector and base year response status and mode ................................ ................................ ................................ .................... 16 Table 5. Detectable differences for experimental hy potheses ................................ ............................ 20IV 

## List of **Exhibits** Exhibit 1. Design of the BPS:12/14 field test data collection experiment .................................2 11 

### B. **Collection of Information Employ**  **ing Statistical Methods 1.**  **Respondent Universe**  **and Sampling** The target populatio n for the 2012/ 14 Beginning Postsecondary Students Longitudinal Study ( BPS:12/14 ) field test consist s of all students who began their postsecondary education fo r the first time during the 2010

– 11 academic year at any Title IV - eligible postsecondary institution in the United States. The sample students were the first - time beginners (FTBs) from the 20 1 1 - 1 2 National Postsecondary Student Aid Study (NPSAS:12 ) field test . Because the students in the BPS:12/14 field test sample come from the NPSAS:12 field test sample, this section also describes the NPSAS:12 fi eld test sample design , which was a two - stage sample consisting of a sample of institutions at the first stage, and a sample of students from within sampled institutions at the second stage. The BPS:12/14 field test sample is comprised of students from the NPSAS:12 field test sample who were determined to be FTB s , or were potential FTB s as indicated by the NPSAS institution . **a. NPSA** **S:12 Field Test**  **Institution Universe and Sample** To be eligible for the NPSAS:12 field test, students must have been enrolled in a NPSAS eligible institution in a term or course of instruction at any time during the 2010

– 11 academic year. Institutions must have also met the following requirements:  offer an educational program designed for persons who have completed secondary education;  offer at least one academic, occupational, or vocational program of study lasting at least 3 months or 300 clock hours;  offer courses that were open to more than the emplo ye es or members of the company or group (e.g. union) that administers the institution;  be located in the 50 states or the District of Columbia;  not be a U.S. service academy institution; and  have signed the Title IV participation agreement with the U.S. Department of Education . 1 Institutions providing only avocational, recreational, or remedial courses or only in - house courses for their own employees or members were excluded. U.S. service academies were excluded because of their unique funding/tuition base. The institution samples for the NPSAS field test and full - scale studies were selected simultaneously , prior to the field test study , using stratified random sampling with probabilities proportional to a composite measure of size (Folsom, Potter, and Willi ams 1987). Institution measure of size was determined using annual enrollment data from the most recent IPEDS 12 - Month Enrollment Component and first time beginner (FTB) enrollment data from the most recent IPEDS Fall Enrollment Component. Composite measure of size sampling was used to ensure that target sample sizes were achieved within institution and student sampling strata, while also achieving approximately equal student weights across institutions . The institution sampling frame for NPSAS:12 field tes t was constructed using the 2009 Integrated Postsecondary Education Data System (IPEDS) header, Institution Characteristics

1 A Title IV eligible institution is an institution that has a written agreement (program participation agreement) with the U.S . Secretary of Education that allows the institution to participate in any of the Title I V federal student financial assistance programs other than the State Student I ncentive Grant (SSIG) and the National Early I

ntervention Scholarship and Partnership (NEI SP) programs.2

(I C), Fall and 12 - Month Enrollment, and Completions files. All eligible students from sampled institutions comprised the student sampling frame The field test institution sample for NPSAS:12 was selected using statistical procedures rather than purposively

as had been done in past NPSAS cycles. This provided more control to ensure that the field test and the ful l - scale institution samp les had similar characteristics. It also allowed inferences to be made to the target population, supporting the anal ytic needs of the field test experiments and instrument. This ability

to make analy

tic inferences extends to the BPS:12/14 field test sample . From the stratified frame, a total of 1,970 institutions was selected to participate in either the full scale or field test study . From the 1,970 institutions selected , a subsample of 300 institutions was selected using simple random sampling within in stitution strata to comprise the field test sample. The remaining 1,670 institutions comprise d the sample for the full - scale study . This sampling process eliminated the possibility

that an institution would be burdened with participation in both the field test and full - scale samples , and maintain ed the representativeness of the full - scale sample. All institutions in the NPSAS:12 sample were eligible for either the full - scale or field test studies. The institution strata used for the sampling design were ba sed on institution level, control, and highest level of offering and are the following:

1. public less - than - 2 - ye ar, 2. public 2 - ye ar, 3. public 4 - ye ar non - doctorate - granting, 4. public 4 - ye ar doctorate - granting, 5. private nonprofit less - than - 4 - year, 6. private nonprofit 4 - ye ar non - doctorate - granting, 7. private nonprofit 4 - ye ar doctorate - granting, 8. private for -profit less - than - 2 - year, 9. private for -profit 2 - ye ar, and 10. private for -profit 4 - ye ar. Due to the growth of the for-profit sector, private for-profit 4 - ye ar and private for- p rofit 2 - year institutions were separated into their own strata unlike in previous administrations of NPSAS. Within each institution stratum, additional implicit stratification for the full - scale was accomplished b y sorting the sampling frame within stratum by the following classifications: (1) historically

Black colleges and universities indicator; (2) Hispanic - serving institutions indicator (3) Carnegie classifications of degree - granting postsecondary institutions; (4) 2 - digit Classification of Instructional Programs (CIP) code for less - than - 2 - ye ar institutions; (5) the Office of Business Economics Region from the IPEDS header file (Bureau of Economic Analy sis of the U.S. Department of Commerce Region); (6) state and s ys tem for states with large systems, e.g., the SUNY and CUNY s ys tems in New York, the state and technical colleges in Georgia, and the California S tate University and University

of California systems in California; and (7) the institution measure of size. The objective of this implicit stratification was to approximate proportional representation of institutions on these measures. Approximately

300 institutions were sampled for the NPSAS:12 field test. Overall, almost 100 percent of the sampled institutions met the eligibility requirements; of those, approximately 51 percent (or about 150 institutions) provided enrollment lists.3

**b. NPSA** **S :12**  **Field Test Student**  **Universe and Sample** The students eligible for the BPS:12/14 field test are those eligible to participate in the NPSAS:12 field test who were FTBs at NPSAS sample institutions in the 2010 - 11 academic year. Student s eligible for the NPSAS:12 field te st were those who attended a NPSAS eligible institution during the 2010

– 11 academic year and who were  enrolled in either : (a) an academic program; (b) at least one course for credit that could be applied toward fulfilling the requirements for an academic d egree; (c) exclusively non - credit remedial coursework but determined b y the institution to be eligible for Title IV aid; or (d) an occupational or vocational program that required at least 3 months or 300 clock hours of instruction to receive a degree, certificate, or other formal award;  not currentl y enrolled in high school; and  not solely

enrolled in a General Educational Development (GED) or other high school completion program. The NPSAS:12 field test institution sample included all levels (less - than - 2 - ye ar, 2 - ye ar and 4 - ye ar) and controls (public, private nonprofit and private for - profit) of Title IV eligible postsecondary institutions in the United States. The field test student sample was randomly selected from lists of students enrolled at sampled in stitutions between July, 1 2010 and April 30, 2011. The NPSAS:12 first test study year covers the time period between July 1 , 2010 and June 30 , 2011 , to coincide with the federal financial aid award year. To facilitate timely completion of data collection and data file preparation, institutions were asked to submit enrollment lists for all eligible students enrolled at any t ime between July 1 and April 30 or, for institutions with continuous enrollment, between July

1 and March 31. The March 31 deadline for continuous enrollment institutions was used for the field test due to the compressed data collection schedule and was n ot used in the full - scale. Because previous cycles of NPSAS have shown that the terms beginning in May and June add little to enrollment and aid totals, May- June starters were excluded to allow institutions to provide enrollment lists earlier which, in turn, allowed the student interview process to begin earlier. In the full scale study, post - stratification of survey estimates based on IPE DS records on enrollment and National Student Loan Data Sy stem (NSLDS) records on financial aid distributed was used to adjust for the survey ye ar’s inclusion of any

terms that begin b y April 30 and the consequent exclusion of a small number of students newly

enrolled in May

or June. To create the student sampling frame, each participating institution was asked to submit a list of eligible students. The requests for student enrollment lists specifically indicated how institutions should handle special case s, such as students taking onl y correspondence or distance learning courses, and foreign exchange, continuing education, extension division, and non - matriculated students. The data required for each enrollee were the following:  student’s name;  student ID;  Social Security

number;  date of birth;  date of high school graduation (month and year);4

 degree level during the last term of enrollment (undergraduate, masters, doctoral research/scholarship/other, doctoral -professional practice, or other graduate);  clas s level if undergraduate (first, second, third, fourth, or fifth year or higher);  major;  CIP code;  indicator of whether the institution received an Institutional Student Information Record (ISIR) ( an electronic record summarizing the results of the student’s Free Application fo r Federal Student Aid [FAFSA] processing) from the Central Processing Sy stem (CPS);  FTB status; and  contacting information, such as cell phone number, local telephone number and address, permanent telephone number and address, campus e - mail address, and permanent e - mail address. Requesting contact information for eligible students prior to sampling allowed for student record abstraction and student interviewing to begin shortly after sample selection which helped to ensure the managem ent of the field test schedule for data collection, data processing, and file development. Student sample sizes for the field test were formulated to ensure representation of various ty pes of students. Specifically , the sample included a large number of p otential first - time beginners to provide a sufficient sample size to obtain a sample yi eld of at least 1,000 students for BPS field test. The NPSAS:12 field test sample included 4,530 students of which 4,130 were potential FTBs, 200 were other undergraduat e students, and 200 were graduate students. Students were sampled at fixed rates according to student education level and institution sampling strata. Sample yi eld was monitored and sampling rates were adjusted when necessary , resulting in a statistical s ample of the required sample size for the field test. The same approach was to be used for the full - scale study. Student enrollment lists provided by the institutions were reviewed to make sure that required elements were included, and were also compared f or consist ency with counts from the 2009 IPEDS 12 Month Enrollment Component.

**c. Identification of**  **FTBs in**  **NPSA S**  **:12** To be eligible for the BP S field test, students must have begun their postsecondary education for the first time after completing high school on or after July 1, 2010 , and before July

1, 2011 . Close attention wa s paid to accuratel y identify FTBs in the NPSAS field test to avoid unacceptably high rates of misclassification (e.g., false positives) 2 which can , and have , resulted in (1) excessive cohort loss, (2) excessive cost to “replenish” the sample, and (3) an inefficient sample design (excessive oversampling of “potential” FTBs) to compensate for anticipated misclassification error. To address this concern, particip ating institutions were asked to provide additional information for all eligible students and matching to administrative databases was utilized to further eliminate false positives prior to sample selection. Participating institutions were asked to provide the FTB status and high school graduation date for every listed student. High school graduation date was used to remove students from the frame who were co - enrolled in high school. FTB status , along with class level and student level , were used to exclude misclassified FTB students in their third year or higher and/or those who were not an undergraduate

2 A student identified by the institution on the enrollment list as an FTB who turns out to not be an FTB is a fals e positive.5

student. FTB status , along with date of birth , were used to identify students older than 18 to send for pre sampling matching to administrative databases. If the FTB indicator was not provided for a student on the lists , but the student was 18 ye ars of age or younger and did not appear to be dually enrolled in high school, the student was sampled as an FTB. Otherwise, if the FTB indicator was not provided fo r a student on the list and the student was over the age of 18, then the student was sampled as an “other undergraduate” but would be included in the BPS cohort if identified during the student interview as an FTB. Prior to sampling, students over the age of 18 listed as potential FTBs were matched to NSLDS records to determine if any had a federal financial aid history pre - dating the NPSAS year (earlier than July 1, 2010 for the field test). Since NSLDS maintains current records of all Title IV federal grant and loan funding, any student with disbursements from the prior year or earlier could be reliabl y excluded from the sampling frame of FTBs. Given that about 60 percent of FTBs receive some form of Title IV aid in their first year, this matching process could not exclude all listed FTBs with prior enrollment, but significantl y improved the accuracy of the list prior to sampling, yielding fewer false positives. After undergoing NSLDS matching, students over the age of 18 still listed as potential FTBs were matched to the National Student Clearinghouse (NSC) for further narrowing of potential FTBs based on evidence of earlier enrollment. Matching to NSLDS identified about 19 percent of cases as false positives and matching to NSC identified about 14 percent of cases as false positives. In addition to NSLDS and NSC, a subset of potential FTBs on the student sampling frame was sent to CPS for matching to evaluate the benefit of the CPS match for the full - scale study . Of the 58,690 students sent, CPS identified about 10 percent as false positives. Overall, matching to all sources identified about 32 percent of potential FTB students over the age of 18 as false positives, with many of the false positives identified by CPS also identified by

NSLDS or NSC. The matching appeared most effective among public 2 - ye ar and private for -profit institutions. While public less - than 2 - ye ar institutions have a high percent of false positives, they represent a small percentage of the total sample. Since this pre - sampling matching was new to NPSAS:12, the FTB sample size was set high to ensure that a sufficient number of true FTBs would be interviewed. In addition, FTB selection rates were set taking into account the error rates observed in NPSAS:04 and BPS:04/06 within each sector ( additional information on NPSAS:04 methodology is available in the study methodology report, publication number NCES 2006180 , or BPS:04/06 methodology report, publication number NCES 2008184 . ) . These rates were adjusted to reflect the improvement in the accuracy of the frame from the NSLDS and NSC record matching. Sector- level FTB error rates from the field test will be used to help determine the rates necessary for full - scale student sampling. **d. BPS:12/14 Field Test Sample**  At the conclusion of the NPSAS:1 2 field test, 2,003 students had been interviewed and confirmed to be FTBs , and all will be included in the BPS:12:14 field test. Inaddition, the field test sample will include the 1,493 students who did not respond to the NPSAS:12 field test but were po tential FTBs according to student records or institution lists . The distribution of the BPS:12/14 field test sample is shown in table 1 , by

institution sector.6

**Table 1**  **. BPS:12/14 Sample Size by inst**  **itution characteristics: 2011** Institution characteristi cs Confirmed and Potential FTBs from the NPSAS:12 Field Test Total Confirmed FTBs Potentia l FTBs To tal 3,496 2,003 1,493 Institution type Public Less - than - 2 - year 17 6 11 2 - year 1,526 827 699 4 - year non - doctorate - granting 204 142 62 4 - year doctorate - granting 436 321 115 Private nonprofit Less - than - 4 - year 37 23 14 4 - year non - doctorate - granting 205 155 50 4 - year doctorate - granting 139 107 32 Private for- profit Less - than - 2 - year 139 71 68 2 - year 211 70 141 4 - year 582 281 301 # Rounds to zero. 1 In cludes master's, doctoral - research/scholarship/other, doctoral -professional practice, and other graduate students. NOTE: Detail may not sum to totals because of rounding. FTB = first - time beginner. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2011

– 12 National Postsecondary Student Aid Study (NPSAS:12) Field Tes t **2. Procedures for the Collection of Information** 

**a. Statistical methodology**  **for stratification and sample selection** The BPS:12/14 field test sample will consist of all interviewed and confirmed FTBs from the NPSAS:12 field test and also all potential FTBs who did not respond to the NPSAS:12 field test; no further sampling or subsampling will be needed. The institution and student sampling strata used in the NPSAS:12 field test will be retained for use in analy sis. An experiment to test methods for improving response rates and reducing bias due to nonresponse is being conducted as a part of the field test , and random assi gnment to treatments will be balanced across the NPSAS:12 strata and response status es.

**b. Estimation procedure**  The students in the BPS:12/14 field test were also in the NPSAS:12 field test. Because the NPSAS:12 field test was a probability sample, sampling weights (which are the inverse of the selection probability) are available for each of the students in the NPSAS:12 field test and BPS:12/14 field test. These base weights can be used in anal ys e s of the results of the field test experiment. In addition, response weight adjustments will be applied to these base weights, and the response - adjusted weights will be used in the anal ys is of BPS:12/14 field test interview data.

**c. Degree of accuracy**  **needed for the purpose described in the justification** One purpose of the field test experiments described in section 3 is to improve response rates and reduce nonresponse bias. The sample will include interviewed FTBs from the NPSAS:12 field test and interview nonrespondents who were potential FTBs. Table 5 , at the end of this section, shows that the7

dete ctable differences range from 3 percent to 20 percent for the hypotheses being tested using the field test data. **d. Unusual problems requiring specialized sample procedures**  No special problems requiring specializ ed sample procedures have been identified. **e. An**  **y use of periodic (less frequent than annual) data collection cy** **cles to reduce the burden**  BPS interviews are conducted no more frequentl y than every two ye ars. **3. Methods for Maximizing Response Rates**  **a. Locating**  The response rate for the B PS:12/14 data collection is a function of success in two basic activities: locating the sample members and gaining their cooperation. Several locating methods will be used to find and collect up - to - date contact information for th e BPS:12/14 sample. During BPS : 12/ 1 4 , batch searches of national databases will be conducted prior to the start of data collection. Follow - up locating methods will be employed for those sample members not found after the start of data collection. The meth ods chosen to locate sample members are based on the experience gained from the NPSAS:12 field test and full - scale studies, the B&B:08/12 field test and full - scale studies , and other recent postsecondary education studies. Many factors will affect the abi lity to successfull y locate and survey sample members for BPS:12/14 . Among them are the availability , completeness, and accuracy of the locating data collected in the NPSAS: 12 field test interview and institution records . The locator database includes critical tracing information for nearl y all sample members, including address information for their previous residences, telephone numbers, and e - mail addresses. This database allows telephone interviewers and tracers to have ready access to all the contact in formation available for BPS:12/14 sample members and to new leads developed through locating efforts. To achieve the desired locating and response rates, RTI will use a multistage locating approach that capitalize s on available data for the BPS:12/14 sample . RTI ’ s proposed locating approach includes five basic stages : 1. **Advance Tracing** includes batch database searches, contact information updates , and advance intensive tracing conducted as necessary . 2. **Telephone Locating and Interviewing** includes calling all available telephone numbers and following up on leads provided by parents and other contacts. 3. **Pre -**  **Intensive Batch Tracing** consists of the Premium Phone searches that will be conducted between the telephone locating and interviewing stage and the intensive tracing stage. 4. **Intensive Tracing** consists of tracers checking all telephone numbers and conducting credit bureau database searches after all current telephone numbers have been exhausted . 5. **Other Locating Activities** will take place as needed and may include use of social networking sites and additional tracing resources that are not part of the previous stages.8

The steps described in the tracing plan are designed to locate the maximum number of sample members with the least expense . The most cost - effec tive steps will be taken first so as to minimize the number of cases requiring more costly intensive tracing efforts. BPS:12/14 will also conduct an address update (panel maintenance) procedure t o encourage sample members to update their contact information prior to the start of data collection. All cases will be offered an incentive of $10 to update their contact information via the study website prior to the start of data collection (approximately 2 - 3 weeks before the 2013 field test data collection begins , upon OMB approval ) . W e are also requesting approval to conduct full scale address updates since little time will be available to conduct address update activities once OMB clearance is received in 2014, in advance of the start of data collection. Receiving approval to conduct full scale panel maintenance in conjunction with this field test OMB package will enable the address update activity to take place approximately two months before data collection begins .

**b. Interviewing**  **Procedures Telephone inte**  **rviewer t raining procedures.**  Training for individuals working in survey data collection will include critical quality control elements. Contractor staff with experience in training interviewers will prepare the B PS:12/14 Telephone Interviewer Manual, which will provide detailed coverage of the background and purpose of B PS , sample design, the questionnaire, and procedures for the telephone interview. The manual will also serve as a reference throughout data collection . Training staff will prepare training e xercises, mock interviews (speciall y constructed to highlight the potential of definitional and response problems), and other training aids. **Interviews.** Interviews will be conducted using a single web - based survey instrument for self

administered and telep hone data collection. The data collection activities will be accomplished through the Case Management System (CMS), which is equipped with the following capabilities:  online access to locating information and histories of locating efforts for each case;  qu estionnaire administration module with input validation capabilities (i.e., editing as information is obtained from respondents);  sample management module for tracking case progress and status; and  automated scheduling module, which delivers cases to interviewers and incorporates the following features:

– Automatic delivery

of appointment and call - back cases at specified times.

– Sorting of non - appointment cases according to parameters and priorities set by project staff.

– Restriction on allowable interviewers. Complete records of calls and tracking of all previous outcomes. Flagging of problem cases for supervisor action or supervisor review. Complete reporting capabilities. A sy stem such as the CMS that integrates these c apabilities reduces the number of discrete stages required in data collection and data preparation activities . Overall, the scheduler provides a highl y efficient case assignment and delivery

function , reduc es supervisory and clerical time, improv es execution on the part9

of interviewers and supervisors by automatically

monitoring appointments and callbacks, and reduc es variation in implementing survey priorities and objectives. **CATI Start Dates.** As with NPSAS:12 full - scale study , RTI plan s to ini tiate telephone efforts at varying times in order to maximize participation for sample members from sectors that have historically been challenging. Please see Table 2 below for details. **Table 2**  **. Start dates for BPS:12/14 field test interviewing, by insti**  **tution sector Sector**  **Institution sector Cases begin CATI...**  **Base year respondents 1**  Public less - than - 2 - year 5 days after lead letters are sent **2** Public 2 - year 5 days after lead letters are sent **3** Public 4 - year non - doctorate - granting 3 weeks + 1 day after lead letter mailing **4** Public 4 - year doctorate - granting 3 weeks + 1 day after lead letter mailing **5** Private not - for - profit less - than - 4 - year 5 days after lead letters are sent **6** Private not - for - profit 4 - year non - doctorate granting 3 weeks + 1 day after lead letter mailing **7** Private not - for - profit 4 - year doctorate granting 3 weeks + 1 day after lead letter mailing **8** Private for - profit less - than - 2 - year 5 days after lead letters are sent **9** Private for - profit 2 - year 5 days after lead letters are sent **10** Private for - profit 4 - year 5 days after lead letters are sent **All base year nonrespondents** 5 days after lead letters are sent **Refusal Conversion.** Recognizing and avoiding refusals is important to maximize the response rate. Supervisors will monitor interviewers intensely during the earl y data collection and provide retraining as necessary . In addition, supervisors will review daily interviewer produ ction reports to identify and retrain any

interviewers with unacceptable numbers of refusals or other problems. After encountering a refusal, comments are entered into the CMS record that include all pertinent data regarding the refusal situation, including any unusual circumstances and any reasons given by

the sample member for refusing. Supervisors review these comments to determine what action to take with each refusal ; n o refusal or partial interview will be coded as final without supervisory review and approval. If a follow - up is not appropriate (e.g., there are extenuating circumstances, such as illness or the sample member firml y requested no further contact), the case will be coded as final and no additional contact will be made. If the case appears to be a “soft” refusal, follow - up will be assigned to an interviewer other than the one who received the initial refusal. The case will be assigned to a member of a special refusal conversion team made up of interviewers who have proven especiall y skilled at converting refusals. Refusal conversion efforts will be delayed until at least 1 week after the initial refusal. Attempts at refusal conversion will not be made with individuals who become verball y aggressive or who threaten to take legal or other acti on . Project staff sometimes receive refusals via email or calls to the project10

director’s direct line. These refusals are included in the CATI record of events and coded as final when appropriate.

**c. Quality**  **Control** Interviewer monitoring will be conducted using RTI ’s Quality Evaluation System ( QUEST ) a s a quality control measure throughout the field test and full scale data collections. QUEST is a sy stem developed b y a team of RTIresearchers, methodologists, and operations staff focus ed on develop ing stan dardized monitoring protocols, performance measures, evaluation criteria, reports, and appropriate sy stems security

controls. It is a comprehensive performance quality monitoring s ys tem that includes standard s ys tems and procedures for all phases of quality monitoring, including obtaining respondent consent for recording, procedures for interviewing respondents who refuse consent and for monitoring refusals at the interviewer level; sampling of completed interviews b y interviewer, evaluating interviewer performance; maintaining an online database of interviewer performance data; and addressing potential problems through supplemental training. These s ystems and procedures are based on “best practices” identified by

RTI in the course of conducting thousands of survey research projects. As in previous studies, RTIwill use QUEST to monitor approximately 7 percent of all completed interviews plus an additional 2.5 percent of recorded refusals. In addition, quality supervisors will conduct silent monitoring for 2. 5 - 3 percent of budgeted interviewer hours on the project. This will allow real time evaluation of a variety of call outcomes and interviewer- respondent interactions . Recorded interviews will be reviewed by call center supervisors for key elements such as professionalism and presentation; case management and refusal conversion; and reading, probing, and keying skills. Any problems observed during the interview will be documented on problem reports generated b y QUEST. Feedback will be provided to interviewers and patterns of poor performance (e.g., failure to use conversational interviewing techniques, failure to probe, etc.) will be carefully monitored and noted in the feedback form that will be provided to the interviewers. As needed, interviewers will rec eive supplemental training in areas where deficiencies are noted. In all cases, sample members will be notified that the interview may

be monitored by

supervisory

staff. **4. Tests of**  **Procedures and Methods**  The design of the BPS:12/14 field test experiment exp and s on data collection experiments designed for three NCES studies which preceded it, including the NPSAS:12 field test which served as the base ye ar interview for the current study . Brief summaries of each experiment are provided below , and in table 3 . Although results are not available for the most recent experiments, the design proposed for the BPS:12/14 field test builds on what is known about those experiments and target populations, and how they compare to the BPS:12 cohort. **a. NPSA** **S:12** The re sponse propensity experiment conducted during the NPSAS:12 field test data collection (March 2011 to June 2011) was designed to reduce nonresponse bias through targeted use of incentives. Using data from NPSAS:04, RTI identified variables available prior to data collection which were predictive of response likelihood , then used the variables to estimate a NPSAS:12 field test sample member’s response propensity . Sample members with a low response propensity were sorted at random11

into either a control group, which was offered the usual $30 incentive for participation, or an experimental group, which was offered $45. High response propensity sample members were sorted at random into a control group that was offered $30 or an experimental group that was offere d $15. Following data collection, RTI evaluated the predictive ability of the response propensity model and determined if bias was reduced in the experimental cases . 3 The propensity model successfull y distinguished between high and low propensity cases in terms of response rate. The unweighted low propensity response rate was 57.7% and the unweighted high propensity

respons e rate was 67.7%, a statisticall y significant difference (χ2 = 42.003, p < .0001). However, while the primary goal of the response propensity approach was to reduce bias in key estimates, the weighted estimates in both the low propensity control and treatment groups were virtually identical, suggesting that differential incentives did not have any effect on reducing bias. Although th e NPSAS:12 field test experiment was not designed to increase response rates per se , response rates by incentive amount within propensity groups were tested. Within the low propensity group, no statistically

significant difference between experimental and control groups was noted (χ2 = 2.527, p > .05) while the difference observed between high propensity control and treatment groups was statistically significant, with lower incentives being associated with lower response rates (χ2 = 13.576, p < .001). Given the equivocal results of the response propensity experiment, RTI adopted a responsive design approach to the NPSAS:12 full - scale data collection (February 2012 to present) , dropping pre - data collection modeling of either responses or paradata . All sample members are being offered a $30 incentive . This approach to data collection has been used during the earl y response phase varied by institution sector (e.g., public, 2 - ye ar) as a substitute for response propensity . For example, students in public, 4 - ye ar institutions, with historically higher response rates, were handled with the ty pical data collection plan: three weeks of online - onl y interviewing followed b y outbound calling to nonrespondents. In contrast, students in institutions with hi storically lower response rates and a lower likelihood of responding online, were moved almost immediately to outbound calling, shortening the time to initial contact and, when needed, referral to intensive tracing. As each wave of the NPSAS:12 sample ha s moved from the early response to the production phase of data collection , the approach taken to encourage response continued to depend on institution sector . Other factors about an individual’s experience in data collection were also considered. For ex ample, specialized emails are being prepared based on paradata, such as break offs and expressed preference to complete the online interview , and USPS Priority Mail is being used to contact cases sampled as FTBs. As the data collection period is ending, c ases are being offered the abbreviated interview depending on their time in data collection and expressed reluctance to commit time to the interview. **b. B&B:08/12**  In the B&B:08/12 field - test (July 2011 to October 2011), RTI targeted cases with a low propensity to respond and a high likelihood of contributing to nonresponse bias in order to increase the response rate and yi eld less biased survey estimates. To begin, frame data, paradata, and indicators of previous

3 For the NPSAS:12 field test, the institution sample was selected statistically, rather than purposively, as had been done in past NPSAS cycles, in order to allow inferences to be made to the target population, supporting the analytic needs of the field test experiments.12

response behavior were used to develop a predictive model of a given sample member’s propensity to respond. To build the model, RTI estimated logistic regression coefficients using data from the NPSAS:08 base ye ar to predict response in the first follow - up (B&B:08/09). The resulting model pro duced odds ratios ranging from 0.99 to 2.65 with an r- squared value of 0.19. Before the start of data collection, response propensities for all sample members were calculated based on the developed model, then used to divide the sample into low and high response propensity groups. Approximately

one - third of the cases were in the high propensity group, and two - thirds were in the low propensity

group. The low propensity group was comprised of those less likely to complete and most likely

to introduce nonre sponse bias if they remained nonrespondents. Within each of the propensity levels, cases were randomly assigned into a control group, which received the same incentive offered in the prior field test round ($35 or $55), and an experimental group , whose inc entive amounts varied by response propensity : $20 or $40 was offered to all cases in the high propensity group, and $50 or $70 was offered to cases in the low propensity group, depending on the amount they received in the prior field test, with those rece iving $55 in 2009 receiving $70 in 2011. Like the NPSAS:12 field test, evaluation of the field test results showed that the propensity model was able to accuratel y predict relative likelihood to respond. The proportion of nonrespondents in the low propensi ty

group (39 percent) was more than three times the proportion in the high propensity group (11 percent; χ

2 , (1, N=1,588)=139.0; p <.01). Analyses of response rates for the treatment and control groups indicated that changes in incentives had the stronges t impact on response rates for those individuals in the middle of the propensity

score range. Observed response rates were higher in the incentive treatment group for those individuals with the highest propensity scores within the low propensity classific ation (81.4 and 73.3 percent, t = 2.04, df = 539, p < .05). Those with the lowest propensity scores within the high propensity

classification showed a numerical difference in response rates between treatment and control groups (79 and 89 percent, respectivel y), but the difference was not statistically significant. However, f ield test results did not show a reduction in bias as a result of the additional response. In order to focus on identifying and targeting cases most likely to contribute to nonresponse b ias, a revised approach is being tried in the B&B:08/12 full - scale data collection that uses a responsive design and the Mahalanobis distance measure to identify cases for targeted treatments . For the first three months of data collection, which began in August 2012, all sample members are receiving the same treatment in data collection

– the web, online interview option with “CATI - light,” during which a small number of calls is being made, mainly to prompt sample members to complete the online interview. Incentive offers during the first three months are determined by a case’s propensity score , calculated prior to the start of data collection . Cases with the highest propensity scores are being offered $20, midrange $35, and lowest propensity

$55. The B& B full - scale sample has been split into a treatment and a control group at random and, in Month 3 (November 2012), Mahalanobis values of treatment group nonrespondents will be calculated. C ases above a threshold value (high - distance) will be offered anoth er $15 in addition to their original incentive offer of $20, $35, or $55 ( o nce a case becomes eligible for the additional $15, they will remain eligible for the additional $15 ) . All other treatment group and control group members will continue at their initial incentive level.13

After an additional month of data collection (December 2012), Mahalanobis values will be reevaluated for the remaining treatment group nonrespondents. Those above a new cut point (determined based on the remaining nonrespondents at Month 4) will receive extensive case review in which project staff will review the CMS - CATIevents log and paradata available for a particular case (e.g., availability of e - mail address, parent address, etc.) to identify any specific actions that may encourage a sample member’s participation. Cases eligible for extensive case review will also be prioritized in the CMS. The high - distance nonrespondents in the control group and all low - distance nonrespondents will receive extensive case review, but on the regular schedule (i.e., six weeks later). In Month 6 (February

2013), Mahalanobis values will be evaluated again for remaining nonrespondents, and those above the cut point (determined based on the remaining nonrespondents at month 6) will be offered an abbreviated interview. The high - distance nonrespondents in the control group and all low - distance nonrespondents will receive an abbreviated interview, but on the regular schedule (i.e., 6 weeks later).

**c. ELS:2002/12**  For the ELS third follow - up field test (July 2011 to September 2011), sample members with the lowest response propensities were empiricall y identified, then targeted with interventions in an attempt to encourage participation. A logistic regression model was fitted with the sample member’s ELS:2002 s econd follow - up field test response status as the dependent variable. As independent variables, a range of information known for all respondents and nonrespondents from each prior wave of the longitudinal field test, including information from panel maintenance activities were examined for significance. Predicted probabilities derived from the logistic regression model were used to get an estimate of a case’s response propensity for the field test. Cases were split into two groups of equal size. Field tes t sample members above the median response propensity were classified as high propensity (528 cases), and those below the median as low propensity (527 cases). For the implementation of the experiment, the low propensity

cases were randoml y split into exp erimental and control groups. Low propensity experimental group cases were offered a higher incentive of $45 at the start of data collection (weeks 1 - 9) , increasing to $55 starting at week 10. High propensity and low propensity

control group cases were o ffered $25 until week 10 of data collection, after which the incentive increased to $35. The predictive model developed ahead of the field test data collection effectively predicted the eventual response outcome for sample members. The high propensity group’s response rate (67.4%) was significantl y higher than that of the low propensity control group (45.4%). This difference was statistically

significant (

χ 2 = 34.9; p < .0001). In examining the effect of the higher incentive treatment for low propensity cases, a numerical difference in participation (51.6% for treatment cases v s . 45.4%) was observed, however, the difference was not statistically significant. The small EL S:2002 field test sample size and the brevity of the data collection period may have contributed to the inability to detect a significant difference in the results. In reviewing the mean relative bias, it appeared that including low propensity

cases in the dataset may have help ed reduce bias, if onl y slightly , and the higher incentive fo r the low propensity

experimental cases may

have lowered the bias relative to the low propensity control group.14 **Table 3**  **. Summary of Field Test and Full -**  **Scale Data Collection Designs, by Study**

**Study Field Test**  **Full -**  **Scale*** NPSAS:12  Sample sorted by modeled response propensity into 4 groups:

 High propensity: $15 (E) $30 (C)  Low propensity: $45 (E) $30 (C)  Modeling successfully differentiated groups by propensity  Response rate differences observed only in high propensity group  No effect on bias reduction  $30 for all sample members  Institution sector used as proxy for response propensity  Different data collection strategies applied depending on institution sector B&B:08/12  Sample sorted by modeled response propensity into 4 groups:

 High propensity: $20 /$40 (E) $35/$55 (C)  L ow propensity: $70 (E) $35/$55 (C)  Modeling successfully differentiated groups by propensity  Response rate differences observed only in low propensity group among highest propensity scores  No effect on bias reduction  Calculated response propensities and Mahalanobis distances for all sample members. Sorted sample into experimental and control groups within 3 propensity groups:  Hig h propensity: $20  Medium propensity: $35  Low propensity: $55  At Time 1, Mahalanobis recalculated for experimental group – greatest distance cases offered an additional $15  At Time 2, Mahalanobis recalculated for experimental group – grea test distance cases to receive intensive case review early  At Time 3, Mahalanobis recalculated for experimental group – greatest distance cases offered abbreviated interview early ELS:2002/12  Sample sorted by modeled response propensity into 3 groups:

 High propensity: $25 ($35 in Week 10)  Low propensity: $45 (E; $55 in Week 10) $25 (C; $35 in Week 10)  Modeling successfully differentiated groups by propensity  Response rate differences observed only in high propensity group  Slight reductio n of bias among low propensity group receiving higher incentive  $25 base incentive offered to all sample members  Mahalanobis distance calculated at 3 time points:  Time 1: Highest distance cases offered additional $30  Time 2: New “highest distance” cases offered additional $30; all highest distance cases will receive intensive tracing and limited field interviewing  Time 3: New “highest distance” cases offered additional $30; all highest distance cases will receive $5 prepaid incentive (E) = Experimental Group; (C) = Control Group *Full - scale data collection is underway for all three studies.15

An alternative approach, like that being used for the B&B:08/12 full - scale data collection, is being implemented f

or the ELS:2002/12 full scale study (July 2012 to January

2013) . Mahalanobis distance s are being calculated to identify those nonrespondent cases which are most unlike existing respondents and, therefore most likely to contribute to response bias. Substantive data ( e.g. , enrollment status, parent’s education, high school completion status) and paradata ( e.g. , response status, number of contact attempts in the earl y data collection period) already available from the base year and first a nd second follow - ups are being used to calculate the Mahalanobis distance s . Distance functions are being measured at three points during data collection: 4 weeks and 9 weeks after the start of data collection (Phases 1 and 2, respectively), and at 8 weeks prior to the end of data collection (Phase 3). Most cases are being offered an initial incentive of $25. Cases with the largest calculated distance scores at each time point will be offered an increased incentive of $55 ( once at $55, the incentive offer will not decrease ) . Additional activities will be conducted to locate and interview targeted cases including , at Time 2, performing pre - data collection intensive tracing and pursuing the cases in person with field locator/interviewers and, at Time 3, inc luding a $5 prepaid incentive with the mailing. **d. BPS**  **:12/14 Field Test**  The two BPS:12/14 field test experiment s were developed out of recent experiences with the NPSAS:12 full - scale study and the data collections underway in B&B:08/12 and EL S:2002 third fo llow up. The first experiment emerged from anecdotal reports offered b y NPSAS:12 interviewers that sample members refused to participate upon hearing the estimated time required to complete the interview (average of 25 minutes) ; they simply did not have the amount of time available to complete the interview . For the BPS:12/14 field test, participation rates of base year nonrespondents offered the full length interview (35 minutes) will be compared to the participation rates of nonrespondents offered a shorter, modified interview (20 minutes) to determine if the expected level of effort reported to sample members affects willingness to participate. The BPS:12/14 responsive design experiment builds on the respons iv e design work currentl y being conducted in B&B:08/12 and EL S:2002. The basic design will be the same

– Mahalanobis distances will be calculated at several time points and specific treatments applied to encourage response among sample members likely to contribute the most to bias if they do not participate. There is an important difference in target populations, however, which warrants examination of the responsive design approach for BPS. B&B and EL S follow cohorts of students who are more homogenous th an the BPS cohort will be . That is, while all B&B sample members will have earned a bachelor’s degree during the same academic ye ar and all EL S sample members will be the same age , BP S sample members will not have a common postsecondary experience nor will they

all be of the same age. The distinctl y different populations could result in considerabl y dif ferent Mahalanobis calculations and distributions, and data collection outcomes. 1. Earl y Response Phase (W eeks 1 - 3) . A ll NPSAS:12 base year interview respondents, and all nonrespondents who were potential FTBs, will be included in the BPS:12/14 field test data collection experiment . Data collection will occur in two main phases, an earl y response phase, during the first 3 weeks of data collection, followed by the main,16

production interv iewing phase. The distribution of the NPSAS:12 FTB field test sample is shown in table 4 by the mode in which they

completed the NPSAS:12 field test interview. At the start of the BPS:12/14 field test interview, the entire field test sample (N= 3,496 ) wil l be moved to interviewing . Treatment during this early

phase will depend on a sample member’s base year response status and mode of interview completion as described below . Base Year Respondents Among the 2,003 NPSAS:12 field test base year respondents, there were 725 sample members whose NPSAS institution was a public 4 - ye ar or a private, nonprofit 4 - ye ar institution . Most of the se sample members completed the base ye ar interview online, and will be encouraged to complete the full, 35 - minute BPS:12/14 interview online during the first 3 weeks of data collection . If a s ample member from those institutions contacts the study help desk for any reason , he/she will be offered the option of completing a te lephone interview at that time , but may still complete the interview online if preferred. **Table 4**  **. Distribution of the BPS:12/14 field test sample, by sector and base year response status and mode** Nonrespondents (Potential FTBs) Respondents (Confirmed FT Bs) Web Telephone Partial Type of institution S ample Number Percent Number Percent Number Percent Number Percent

Total 3,496 1,493 42.7 1,535 79.0 409 21.0 59 2.9 Public less - than - 2 - year 17 11 64.7 1 16.7 5 83.3 0 0.0 Public 2 - year 1,526 699 45.8 603 75.1 200 24.9 24 2.9 Public 4 - year non doctorate - granting 204 62 30.4 121 85.8 20 14.2 1 0.7 Public 4 - year doctorate - granting 436 115 26.4 270 87.1 40 12.9 11 3.4 Private nonprofit less than - 4 - year 37 14 37.8 19 82.6 4 17.4 0 0.0 Private 4 - year non doctorate - granting 205 50 24.4 140 90.9 14 9.1 1 0.6 Private 4 - year doctorate - granting 139 32 23.0 100 93.5 7 6.5 0 0.0 Private for - profit less than - 2 - year 139 68 48.9 43 63.2 25 36.8 3 4.2 Private for - profit 2 - year 211 141 66.8 38 58.5 27 41.5 5 7.1 Private for - profit 4 - year 582 301 51.7 200 74.9 67 25.1 14 5.0 The remaining base year respondents

– those whose first institution was a public, less - than 2 - ye ar or 2 - year institution; a private, not - for-profit less - than - 4 - ye ar institution; or any of the 3 levels of for profit institution

– had comparativel y low online completion rates. Those who did complete the NPSAS:12 field test interview online (N=904) will be given time to complete it online again during the first 3 weeks of data collection , or b y telephone if they contact the Help Desk . However, those who completed a telephone interview (N=374) during the base year will be moved to outbound calling within 5 days of the initial data collection mailing, allowing onl y enough time for the initial contact materials to reach them by

regular mail. Calls from interviewers are primarily intended to prompt sample members to complete the online interview, although any willing to complete the interview by telephone will be17

encouraged to do so. During the earl y response phase, b ase ye ar respondents completing the first follow up interview, either online or b y telephone, will receive a check in the amount of $30. Base Year Nonresp o ndents At the completion of the NPSAS:12 field test there were 1,49 3 interview nonrespondents with unknown eligibility

for BPS. For Experiment 1, these nonrespondents will be randoml y assigned into one of two groups. Half of the nonrespondents will be invited to complete the full BPS:12/14 field test interview requiring , on average, approximately 35 minutes of their time (Full Interview group). The other half will be invited to complete a modified BPS:12/14 field test interview , containing just the enrollment, employ ment, and locating sections, of about 20 minutes’ duration (Modified Interview group). All base ye ar nonrespondents will be able to complete the BPS interview online at any time, once the initial notification is received by email. Their cases also will be moved to outbound calling 5 days after the initial data collection mailing, allowing time for initial contact materials to reach them by regular mail. The early response phase will continue for 3 weeks, with all base year nonrespondents receiving a $30 check for a completed interview. Throughout the BPS:1 2/14 field test, the assignment of base ye ar nonrespondents to either the full or modified interview will not change. 2. Production Interviewing Phase (W eeks 4

– end of data collection) . Following the 3 - week early response phase , all sample members who did not respond will be made available to RTI’s Call Center Services (CCS) for outbound calling. While some sample members will have already received calls from CCS during the Earl y Response Phase, the frequency of calls will increase to the level of routine production interviewing. At the end of the first two weeks of the production phase , remaining nonrespondents will be divided into a control and experimental group . E ach group will have an equal number of base year respondents, base ye ar nonrespondents offered the full interview, and base year nonrespondents offered the modified interview from the earl y phase . Time 1 Mahalanobis Calculation . At the end of the first two weeks of production interviewing, a Mahalanobis distance will be calculated across all sample members in the control and experimental groups. Cases with the highest Mahalanobis values will be identified as T ime 1 high distance cases , irrespective of group assignment. Those sample members assigned to the control group will be tracked as either Control - High Distance or Control -Normal Distance but, otherwise, data collection will continue as in the production interviewing phase, with sample members able to complete the interview online or by telephone . Those in the control group who complete the interview will receive a $30 check. In the Experimental group, Experimental -Normal Distance cases will be treated like the c ontrol group in that they will be able to complete the interview online or by telephone, and will receive a $30 check for a completed interview. The Experimental - High Distance cases will , like the other three groups , be able to complete the interview online or b y telephone, and the frequency with which they are contacted will be the same. However, those Experimental - High Distance cases completing the interview will receive a check for $55, instead of $30 . Mailings to the four groups will follow what is considered a standard schedule of contacts, in both conventional mail and electronic mail forms. Production interviewing following the Time 1 Mahalanobis calculation will continue for 3 weeks.18

Time 2 Mahalanobis Calculation . After the 3 weeks of outbound calling, Mahalanobis values for all remaining BPS interview nonrespondents will be recalculated. From th e Control -Normal Distance and Experimental -Normal Distance groups, a subsample of new high - distance nonrespondent cases will be identified. Sample members in the Control - High Distance group will still receive $30 for a completed interview; sample members in the Experimental - High Distance group will have their incentive offer increased to $55. Cases already offered the higher $55 incentive at Time 1 will continue at that incentive level while all other s will continue to be offered $30 for a completed interview whether in the Control High Distance or Normal Distance group. Production interviewing following the Time 2 Mahalanobis calculation will continue for another 3 weeks. Mahalanobis Calculations , Time X 3 - 4 . Mahalanobis values for all remaining interview nonrespondents will be recalculated after two additional 3 - week periods of outbound calling, for a total of 4 calculations covering 12 weeks of data collection . A s in prior weeks, nonrespondents with the highest Mahalanobis distances will be assigned to either the Control - High Distance or the Experimental - High Distance group ($55). The field test design is summarized in exhibit 1 . **Re sponsive Design Re**  **search Questions** With the assumption that increasing the rate of response among high - distance cases will reduce nonresponse bias, the BPS:12/14 responsive design experiment will explore the following research questions: 1. Do response rates differ between high - distance cases in the treatment and control groups? 2. Do estimates of key

variables differ between high - distance and low - distance cases? 3. Does treatment of high - distance cases reduce nonresponse bias ? **Methods and Null Hy potheses**  Research question 1 : do response rates differ between high - distance cases in the treatment and control groups? Because of the assumption that yielding a greater number of high - distance cases will reduce nonresponse bias, response rates for the high - distance control and treatment groups will be compared to determine whether the overall response rate s for the treatment and control group s differ significantly . Specificall y:  H 0 : There will be no difference in response rates between the high - distance treatment and control group s Research question 2 : do estimates on key variables differ between high - distance and low - distance cases? Using administrative record data available for the entire sample, such as indicators of federal aid data from NSLDS, RTI will compare estimates of key variables between high - and low - distance sample members. They will also compare interview outcome measures between all high - distance respondents and all low - distance respondents. Specifically :  H 0 : There will be no difference in estimates for key variables known for all cases (e.g., enrollment beyond year 1, federal financial aid applications, federal loan amount) between the high - and low distance sample members19

 H 0 : There will be no difference in survey estimates between the high - and low - distance survey re spondents  H 0 : There will be no difference in survey estimates between all respondents, excluding the high distance treatment group, and all respondents, including the high - distance control group Research question 3 : does treatment of high - distance cases re duce nonresponse bias? In addition to calculating nonresponse bias statistics for the whole sample, RTI will measure the effect of the responsive design approach on nonresponse bias by comparing estimates between the control and treatment groups. Specific ally :  H 0 : There will be no difference in unit nonresponse bias between the treatment and control groups . The anal ys is will rel y upon the variables listed below , ty pically used in BPS to evaluate bias, and will identify significant bias at the p<.05 level , if any :  Ty pe of base ye ar institution  Region  Central Processing match during base ye ar  Federal aid applicant  Pell Grant recipient  Total Pell Grant amount received  Stafford Loan recipient  Total Stafford Loan amount received  Base ye ar institution undergraduate enrollment  Age at base ye ar  High school graduation year  Dependency

status at base year  In come level at base ye ar  Race/ethnicity  Gender  Marital status at base ye ar  Citizenship status at base year As part of the planning process for developing the experiment design, the differences necessary to detect statistically

significant differences have been estimated. That is, how large of a difference between the control and treatment groups is necessary to determine whether the response rate s are different in hy pothes i s 1 or how large of a difference between the comparison groups is necessary to determine whether the estimates are different in hypothes e s 2 through 5 . Table 5 shows the expected sample sizes and statistically significant detectable difference for the hy potheses to be tested . Several assumptions were made regarding response rates and sample sizes. In general, the closer a rate is to 50 percent (either less than or greater than), the larger the detectable difference. Likewise, smaller sample sizes require larger detectable differences.20

Assumptions: 1. Detectable differences with 95 percent confidence were calculated with a two - tailed test for all hy potheses. 2. Initiall y, the sample will be equall y distributed across experimental cells . 3. All sample members will be included in the analyses of hy potheses 1 and 2 a . 4. All eligible sample members will be included in the anal ys is of hy pothesis 3. 5. Only respondents will be included in the anal ys es of hy potheses 2b and 2c because outcome measure data will only be known for respondents. 6. The top 30 percent will be used for determining the cut point for high and normal distance cases. 7. The response rate for the control group for hypothesis 1 will be 30 percent. 8. Unit nonresponse bias fo r the control group for hypothesis 3 will be ten percent. 4 9. The statistical tests will have 80 percent power with an alpha of 0.05. 10. The statistical tests will use weighted data. 11. A design effect of 2.0 is assumed. **Table 5**  **. Detectable differences for experime**  **nt al**  **hy potheses**  Hypothesis Control Group Ex perimental Group Detectable difference with 95 percent confidence Definition Sample size Definition Sample size 1 High - distance cases that did not

receive treatment 315 High distance cases that ever received treatment 294 11.3 2 a Normal distance cases 1, 692 High distance cases 609 9.0 2b Normal distance respondents 792 High distance respondents 101 20.1 2c All respondents , excluding high distance cases , that ever received treatment, and including high distance cases that never received treatment 2, 490 All respondents, excluding high distance cases , that never received treatment and including high distance cases that never received treatment 2, 505 5.0

3 Eligible cases, excluding high distance cases , that ever received treatment and including high distance cases that never received treatment 2,656 Eligible cases, excluding high distance cases , that never received treatment and including high distance cases that ever received treatment 2,638 2.9

4 Ten percent is generally considered the maximum acceptable value for unit nonresponse bias analysis.**Exhibit 1**  **. Design of the BPS:12/14 field test data collection experiment** 2222

**5. Reviewing**  **Statisticians and Individuals Responsible for Designing and Conducting the Study**  Names of individuals consulted on statistical aspects of study design, along with their affiliation and telephone numbers are:

Name Affiliation Telephone Dr. John Riccobono RTI (919) 541 - 7006 Dr. Jennifer Wi ne RTI (919) 541 - 6870 Dr. James Chromy RTI (919) 541 - 7019 Dr . Natasha Janson RTI (919) 316 - 3394 Mr. Peter Siegel RTI (919) 541 - 6348 Dr. Sara W heeless RTI (919) 541 - 5891 Dr. Alexandria Radford MPR ( 202 ) 478 - 1027 In addition to these statisticians and survey design experts, the following statisticians at NCES have also reviewed and approved the statistical aspects of the study : Dr. Tracy Hunt - White, Ted Socha, Dr. Matt Soldner, Dr. Sean Simone, and Dr. Sarah Crisse y .

**a. Other Contractors’ Staff Responsible for Conducting the Study**  The study is being conducted by

the Postsecondary , Adult, and Career Education (PACE) division of the National Center for Education Statistics (NCES), U.S. Department of Education . NCES’s prime contractor is RTI. RTI is being assisted through subcontracted activities by MPR Associates. Principal professional staff of the contractors, not listed above, who are assigned to the study are provided below: Name Affiliation Telephone Dr. Bryan S hepherd RTI (919) 316 - 3482 Ms. Donna Anderson RTI (919) 990 - 8399 Mr. Jeff Franklin RTI (919) 485 - 2614 M s. Chris Rasmussen RTI (919) 541 - 6775 Mr. Michael Bryan RTI ( 919 ) 541 - 7498 Dr. Jennie Wo o MPR (510) 849 - 4942 Ms. Shirley He Chan MPR (510) 849 - 4942 

### C. **Overview of A**  **nal ysis Topics**  **and Surve**  **y Items**  The BPS:12/14 field test data collection instrument is presented in Appendix H . Many of the data elements to be used in BPS:12/14 appeared in the previously approved NPSAS:12 and BPS:04/09 interviews . New items, as well as items that are to be included in the re - interview and in the modified interview used in the experiment , are identified in Appendix H . 

### References Folsom, R.E., Potter, F.J., & Wi lliams, S.R. (1987). Notes on a Composite Size Measure for SelfWe ighting Samples in Multiple Domains. Pr oceedings of the Section on Survey Resear ch Methods of the American Statistical Association , 792 - 796.