# 201 **5 -**  **1 6**  **N AT**  **IONAL** 

# P **OSTSECOND AR**  **Y** 

# S **TUDENT** 

# A **ID** 

# S **TUD Y**  

# (NPSAS:16 **) S**  **TUDENT** 

# I **NTERVIEW AND** 

# S **TUDENT** 

# R **ECORDS**

**Supporting Statement Part B**  **OMB # 1850 -**  **0666 v.13** 

Submitted by National Center for Education Statistics

U.S. Department of Education

**October 2014** ii 

## Contents **B. Collection of**  **Infor mation Employing Statistical Methods**  **................................ ..............................**  **1** 1. Respondent Universe ................................ ................................ ................................ ................................ ............................ 1 a. Institution Universe ................................ ................................ ................................ ................................ ........................ 1 b. Student Universe ................................ ................................ ................................ ................................ ............................. 1 2. Statist ical Methodology ................................ ................................ ................................ ................................ ......................... 1 a. Institution Sample ................................ ................................ ................................ ................................ .......................... 1 b. Student Sample ................................ ................................ ................................ ................................ ............................... 4 3. Methods for Maximizing Response Rates ................................ ................................ ................................ .......................... 8 a. Collection of Data from Institutional Records ................................ ................................ ................................ ......... 8 b. Student Sur vey: Self

- Administered W

eb and CA

TI ................................ ................................ ................................ .. 9 4. Te sts of Procedures and Methods ................................ ................................ ................................ ................................ .... 14 a. Experiment # 1: Ev

aluation of Burden and Motivation to P ar ticipate ................................ .............................. 14 b. Experiment # 2: Questionnaire Design ................................ ................................ ................................ .................... 15 c. Experimental Design ................................ ................................ ................................ ................................ ................... 16 d. Detectable Differences ................................ ................................ ................................ ................................ ................ 18 5. Reviewing Statisticians and Individuals Responsible for Designing and Conducting the Study ............................ 20 **C. References**  **................................ ................................**  **................................ ................................**  **........ 20**  

## Ta **bles** Table 7. NPSAS:16 field test estimated institution sample sizes and yield ................................ ................................ ......... 3 Table 8. NPSAS:16 preliminar y full

- scale institution sample sizes and yield ................................ ................................ ..... 3 Table 9. Weighted false positive rate obser ved in baccalaureate identification, by sector : NPSAS:08 .......................... 4 Table 10. Expected student sample sizes and yields for the NPSAS:16 field test ................................ ............................... 6 Table 11. Preliminar y student sample sizes and yields , NPSAS:16 full

- scale ................................ ................................ ....... 7 Table 12. Detectable differences for field test expe riment hypotheses ................................ ................................ .............. 191 

### B. **Collection of Information Employ ing Statistical Methods**  This submission requests clearance for the 2015

- 16 National Postsecondary Student Aid Study (NPSAS:16) field test activities. Specific plans are provided be low. Materials for field test institution contacting, enrollment list collection, and list sampling activities were submitted in a separate package and approved in July 2014. **1. Respondent Universe**  **a. Institution Universe**  To be eligible for NPSAS:16, an institut ion will be required, during the 2014

– 15 academic year for the field test and the 2015

- 16 academic year for the full

- scale, to:  Offer an educational program designed for persons who had completed secondary education;  Offer at least one academic, occupational, or vocational program of study lasting at least 3 months or 300 clock hours;  Offer courses that are open to more than the employees or members of the company or group (e.g., union) that administered the institution;  Be located in the 50 state s, the District of Columbia, or Puerto Rico; 1  Be other than a U.S. Service Academy; and  Have a signed Title IV participation agreement with the U.S. Department of Education. Institutions providing only avocational, recreational, or remedial courses or only in

- house courses for their own employees will be excluded. The five U .S . Ser vice Academies are excluded because of their unique funding/tuition base.

**b. Student Universe**  The students eligible for inclusion in the NPSAS:16 sample are those who are enrolled in a NPSAS

eligible institution in any ter m or course of instr uction between J uly 1, 2014 and April 30, 2015 for the field test and between J uly 1, 2015 and April 30, 2016 for the full

- scale who are:  Enrolled in (a) an academic program; (b) at least one course for credit that could be applied toward fulfilling the requirements for an academic degree; (c) exclusively non

- credit remedial coursework but who the institution has determined are eligible for Title IV aid; or (d) an occupational or v ocational program that required at least 3 months or 300 clock hours of instruction to receive a degree, certificate, or other formal award;  Not currently enrolled in high school; and  Not enrolled solely in a GED or other high school completion program. **2. St**  **atistical Methodology a.**  **Institution Sample** The NPSAS:16 field test and full

- scale institution samples will be selected in a different manner than has been done in previous NPSAS studies . The field test institution frame will be constr ucted from the IPEDS:201 2
 13 header, Institutional Characteristics (IC), Completions , and Full

- year Enrollment files . 2 The
1 Institutions in Puer to Rico were not eligible for NPSAS:12. 2 A preliminar y sampling frame has been created using IPEDS:2011

- 12 data, on which frame counts in table 7 are based. The frame will be recreated with the most up

- to - date data prior to both the field test and full

- scale sample selections .2

full

- scale institution frame will be constr ucted a year later from the IPEDS:2013

- 14 header, Institutional Characteristics (IC), Completions , and Full

- year E nrollment files . Creating a separate institution frame for the field test and full

- scale studies carries the advantage of having a more accurate and current full

- scale institution sample since the frame will be constr ucted using the most up

- to - date IPEDS f iles . Also , freshening the institution sample will not be needed since we will be using the most up

- to - date institution frame available. So that we do not burden them with both field test and full

- scale data collections , we will remove from the field test frame any larg e systems (reporters) and individual institutions likely to be selected with certainty (i.e., probability of selection equal to one) for the full

- scale. 3 Also , we will remove field test sample institutions from the full

- scale frame and later adjust the weights for the full

- scale sample institutions so that they represent the full population of eligible institutions . Fo r the small number of institutions on the frames that have missing enrollment infor mation, we will impute the data using the la test IPEDS imputation procedures to guarantee complete data for the frames . Then, a statistical sample of 600 institutions will be selected from the field test frame and about 2,000 institutions will be selected from the full

- scale frame. W e will select in stitutions for both the field test and full

scale studies using stratified random sampling with probabilities proportional to a composite measure of size, 4 which is the same methodolog y that we have used since NPSAS:96. Institution measures of size will be deter mined using full

- year enrollment and baccalaureate completions data. Using composite measure of size sampling will ensure that the full

- scale target sample sizes are achieved within institution and student sampling strata while also achieving approxi mately equal student weights across institutions . W e will purposively subsample 300 of the 600 field test institutions to allow for some flexibility in the sample, such as excluding institutions unlikely to participate based on past experience. The institu tional strata will be the ten sectors that were used for NPSAS:12, which are based on institutional level, control, and highest level of offering:  Public l ess - than

- 2

-Year  Public 2

- year  Public 4

- year non - doctorate

- granting  Public 4

- year doctorate

- granting  P rivate for

- profit less

- than

- 2

- year  Private for

- profit 2

- year  Private for

- profit 4

- year  Private nonprofit less

- than

- 4

- year  Private n onprofit 4

- y ear non - d octorate - granting  Private nonprofit 4

- year doctorate

- granting Further refinement of the ten sectors may be deemed necessar y for the full

- scale in order to targ et specific types of institutions that are not being captured sufficiently with the current ten sectors or to adapt to the changing landscape in postsecondar y education. For example, the private for

- pr ofit 4

- year sector could possibly be split into two strata based on academic offerings .
3 There is a small chance that cer tain institutions may be selected for both the field test and full

- scale studies , such as small systems . 4 Fo lsom, R.E., Potter, F

.J ., and Williams , S .R. (1987). Notes on a Composite Size Measure for Self

-Weighting Samples in Multiple Domains . Pr oceedings ofthe Section on Survey Research Methods ofthe American Statisti cal Association , 792

- 796.3

Fo r the field test and full

- scale, we expect to obtain overall 97 and 99 percent eligibility rates , respectively

, and at least an overall 85 percent institutional parti cipation (response) rate. The eligibility and response rates will likely v ar y by institutional strata. Based on these expected rates , the estimated institution sample sizes and sample yield by the ten institutional strata (described above) for the field te st and full

- scale are presented in tables 7 and 8, respectively . Within each institutional stratum, additional implicit stratification will be accomplished by sorting the sampling frame by the following classifications: (1) historically Black colleg es and universities (HBCU) indicator; (2) Hispanic

- ser ving institutions (HSI) indicator; 5 (3) Carnegie classifications of postsecondar y institutions; 6 (4) the Office of Business Economics (OBE) Region from the IPEDS header file (Bureau of Economic Analysis of the U. S. Department of Commerce Region); 7 (5) state and, for states with larg e systems , e.g., the SUNY and CUNY systems in New Y ork, the state and technical colleges in Georgia, and the California State University and University of California systems in Calif ornia; and (6) the institution measure of size. The objectiv

e of this implicit stratification will be to appro ximate proportional representation of institutions on these measures . **Ta ble**  **7 .**  **NPSA S:16 field test estimated institution sample sizes and yield**  **Institutional sector Frame count**  **1 Number sampled**  **Number eligible List respondents**  To tal 7,278 300 290 247 Public less - than - 2 - year 256 5 5 4 Public 2 - year 1,046 11 11 9 Public 4 - year non - doctorate - granting 348 11 0 106 95 Public 4 - year doctorate - granting 338 0 0 0 Private nonprofit less - than - 4 - year 256 6 6 4 Private nonprofit 4 - year non - doctorate - granting 973 125 122 102 Private nonprofit 4 - year doctorate - granting 609 21 20 17 Private for

- profit less - than - 2 - year 1,637 8 7 5 Private for

- profit 2 - year 1,030 5 5 4 Private for

- profit 4 - year 785 9 9 7 1 Institution counts based on IPEDS:2011 - 12 header files. NOTE: Detail may not sum to totals because of rounding. **Ta ble**  **8 .**  **NPSA S:16**  **preliminary full -**  **scale institution sample sizes and yield Institutional sector**  **Frame count 1**  **Number sampled Number eligible**  **List respondents** To tal 7,278 2,000 1,980 1,683 Public less - than - 2 - year 256 22 22 19 Public 2 - year 1,046 376 375 332 Public 4 - year non - doctorate - granting 348 180 179 162 Public 4 - year doctorate - granting 338 338 337 295 Private nonprofit less - than - 4 - year 256 20 19 15 Private nonprofit 4 - year non - doctorate - granting 973 325 325 277 Private nonprofit 4 - year doctorate - granting 609 268 266 222 Private for

- profit less - than - 2 - year 1,637 70 67 49 Private for

- profit 2 - year 1,030 120 11 7 93 Private for

- profit 4 - year 785 280 273 218 1 Institution counts based on IPEDS:2011 - 12 header files. NOTE: Detail may not sum to totals because of rounding.

5 A Hispanic

- ser ving institutions indicator is no longer available from IPEDS , so we will create an indicator following the logic that was previously used for IPEDS . 6 Wewill decide what, if any , collapsing is needed of the catego ries for the purposes of implicit stratification. 7 Fo r sor ting purposes , Alaska and Hawaii will be combined with Puer to Rico in the Outlying Areas region rather than in the Far W est regi on.4

**b. Student Sample**  **Student Enrollment List Collection** To begin NPSAS data collection, sampled institutions are asked to provide a list of all their NPSAS

eligible underg raduate and g raduate students enrolled in the targ eted academic year, covering J uly 1 through June 30. Since NPSAS:2000, institutions have been asked to limit listed students to only those enrolled through April 30. This tr uncated enrollment period excludes students who first enrolled in May or J une, but it allows lists to be collecte d earlier and, in turn, data collection to be completed in less than 12 months . When evaluated during NPSAS:96, the abbreviated schedule missed only about three percent of the target population, and weighting can account for the minimal lack of coverag e. Given the short time frame for the NPSAS:16 field test, institutions with continuous enrollment will be asked to include students enrolled only through March 31, instead of April 30, to expedite data collection. 8 Howev er, following completion of the field t est, we will again evaluate the impact of this tr uncated enrollment period . We will request that the date first enrolled at the institution be included on the lists and that some field test institutions provide lists with students enrolled through the end of J une. W e will not select student samples from these later lists , but will u se administrative data and frame data from the lists to conduct a bias analysis to deter mine if there are differences between May/J une enrollees and all other students . If this analysis shows that there are differences , we will modify our approach prior to the full

- scale list collection. NPSAS:16 will ser ve as the base year data collection for the 2016/17 Baccalaureate and Beyond Longitudinal Study (B&B:16/17) and will be used to qualify students for cohort membership . T o that end, we will ask institutions that award baccalaureate degrees to identify students who are expected to receive the baccalaureate degree by J une 30 of the NPSAS year (2015 for the field test; 2016 for the full

- scale). Instead of waiting until J une for institutions to positively confir m deg ree award to these students , we will request that enrollment lists include an indicator (B&B flag) of cohort eligibility for students who have received or are expected to receiv

e the baccalaureate degree during the NPSAS year. As shown in table 9, the percentag e of students , initially flag ged as potential baccalaureate recipients , who do not actually receiv e their bachelor’s degree in the NPSAS year (i.e., the false positive rate) is expected to be high. Therefore, the NPSAS sampling rates for potential baccalaureates and other undergraduate students will be adjusted to yield the appropriate sample sizes , after accounting for the expected false positive and false neg ative rates by sector. **Ta ble**  **9 .**  **Weighted false positive rate observed in baccalaureate id entification, by sector: NPSA** **S:08 Institutional sector**  **False positive rate (weighted)** Public 4 - year non - doctorate - granting 34.7 Public 4 - year doctorate - granting 27.2 Private nonprofit 4 - year non - doctorate - granting 22.3 Private nonprofit 4 - year doctorate - granting 20.7 Private for

- profit 4 - year 32.9

8 All institutions will be asked to include students enrolled through April 30 in the full

- scale.5

**Student Stratification** The student sampling strata for the field test will be:  Baccalaureate STEM majors 9  Baccalaureate business majors  Baccalaureate teacher majors  All other baccalaureate students  Other undergraduate students  Masters students  Doctoral STEM majors  Doctoral other majors  Other graduate students Similar to the approach taken in prior NPSAS collections with a B&B spinoff , s everal student subg roups will be intentionally sampled at rates different than their natural occurrence within the population due to specific full

- scale analytic objectives . W e anticipate that the four following groups will be oversampled in the field test: 1. Baccalaureate STEM majors 2. Baccalaureate teacher majors 3. Doctoral STEM majors 4. Undergraduate students at all award levels enrolled in for

- profit institutions In addition, because of their sheer number, we anticipate that baccalaureate business majors will be under

- sampled. Sampling business majors in propor tion to the po pulation would make it difficult to draw inferences about the experiences of baccalaureates more broadly . In the field test, we will investig ate the possibility of identifying federal financial aid applicants or recipients prior to student sampling. If thi s is feasible then, in the full

- scale, we could stratify students by financial aid application status , P ell Grant or Direct Loan receipt, or P ell Grant or Direct Loan amount. This additional stratification for sampling may help the poststratification weigh ting adjustment, which is typically done using P ell Grant and Direct Loan control total s. 10 To deter mine feasibility , NCES will talk with Federal Student Aid (FSA) about obtaining student data from CPS , P ell, and/or Direct loan files prior to sampling. Timing of when the relev

ant data are available from FSA may be an issue. W e will explore how best to combine the financial aid strata with the other strata mentioned above, and we will look at design effects . **Sample Sizes and Student Sampling** Based on past exp erience, NCES expects to obtain, minimally , 95 percent eligibility rates and 70 percent student inter view response rates overall and in each sector. The expected student sample sizes and sample yield are presented in table 10 for the field test. The field test is designed to sample about 4,500 students , which is similar to NPSAS:12. T able 11 does not show sample sizes adjusted for false positives and false neg ativ

es , but a large percentag e of the field test sample will be comprised of potential baccalaureat es in
9 STEM (Science, T

echnolog y,Engineering, and Mathematics) fields include mathematics; physical sciences; biological/life sciences; computer and infor mation sciences; engineering and engineering technologies; and science technologies . 10 In NPSAS:12, poststratification caused an increase in bias and design effects . Accounting for financial aid in the sampling s tratification may help avoid these issues for NPSAS:16 poststratification.6

order to obtain a sufficient sample yield for the B&B field tests . The NPSAS field test sample size of g raduate students is relatively small due to the large baccalaureate sample size. To meet the tr uncated field test schedule, students must be selec ted by mid

- May of 2015 . Like past NPSAS field tests , the 3,000 student respondents to the NPSAS:16 field test will be sufficient to test the data collection instr uments . Howev er, in order to also reach a g ood representation of students across the ten secto rs , the number of participating institutions needs to be at least 150. If more than 150 lists are received by mid

- May

, only 150 will be sampled. Limiting the sampling to 150 institutions will increase the student sample size for each institution, making the Student Records burden closer to what it will be in the full

- scale. Students will be sampled on a flow basis as student lists are received. Stratified systematic sampling procedures will be utilized. Sample yield will be monitored by institutional and student sampling strata, and the sampling rates will be adjusted early , if necessar y, to achiev

e the desired sample yields . The student sampling procedures implemented in the field test will be as comparable as possible to those planned for the full

- scale study to evaluate the feasibility of the process es and procedures required by the full

- scale plan. In particular, following the field test, we will evaluate whether differences in NPSAS and IPEDS definitions of student eligibility result in student counts that either under

- or over- estimate NPSAS

eligible students . The field test sample design will not chang e while we investigate aspects of the sampling plan for the full

- scale study . W e will describe the outcomes of these explorations and any sample design chang es in the full

- scale sampling plan and OMB packag e. **Ta ble**  **10 .**  **Expected student sample sizes and yields for the NPSA S:16 field test** 

**Institutional sector Sample students**  **Eligible students Responding students**  **Responding students**  **per responding**  **institutio n**  **1 Total**  **Bacca**   ** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **Bacca**  

** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **Bacca**  

** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **4,511 1,695**  **2,616 200**  **4,286 1,610**  **2,486 190**  **3,000 1,127**  **1,740 133**  **12** Public less - than 2 - year 123 0 123 0 109 0 109 0 67 0 67 0 16 Public 2 - year 445 0 445 0 407 0 407 0 270 0 270 0 29 Public 4 - year non doctorate - granting 951 501 429 21 910 476 414 20 669 337 317 14 7 Public 4 - year doctorate - granting 0 0 0 0 0 0 0 0 0 0 0 0 0 Private nonprofit less - than - 4 - year 149 0 149 0 142 0 142 0 90 0 90 0 21 Private nonprofit 4 - year non doctorate - granting 969 506 432 31 921 478 414 29 692 345 323 21 7 Private nonprofit 4 - year doctorate granting 948 442 379 127 907 421 366 121 693 304 285 86 42 Private for- profit less - than - 2 - year 249 0 249 0 236 0 236 0 132 0 132 0 25 Private for- profit 2 - year 61 0 61 0 59 0 59 0 38 0 38 0 10 Private for- profit 4 - year 616 246 349 21 594 235 339 20 350 141 219 12 50 1 The number of responding students per participating institution is based on the 247 list respondents shown above in table 7, rather than on the 150 institutions from which students will be selected. NOTE: Detail may not sum to totals because of rounding.7

**Ta ble**  **11 .**  **Preliminary student sample sizes and yields, NPSAS:16 full -**  **scale**

**Institutional sector Sample students**  **Eligible students Responding students**  **Responding students**  **per responding**  **institution Total**  **Bacca**   ** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **Bacca**  

** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **Bacca**  

** laureates** **Other**

**under****** **graduate students**  **Graduate**

**students Total**  **126,316 51,277**  **53,986 21,053**  **120,000 48,713**  **51,287 20,000**  **84,000 34,099**  **35,901 14,000**  **50** Public less - than 2 - year 680 0 680 0 608 0 608 0 382 0 382 0 20 Public 2 - year 21,296 0 21,296 0 19,617 0 19,617 0 13,321 0 13,321 0 40 Public 4 - year non doctorate - granting 12,890 7,141 3,751 1,998 12,342 6,792 3,649 1,901 9,166 4,940 2,858 1,369 57 Public 4 - year doctorate - granting 26,120 13,224 6,346 6,550 24,806 12,487 6,129 6,189 18,892 9,358 4,945 4,589 64 Private nonprofit less - than - 4 - year 870 0 870 0 838 0 838 0 543 0 543 0 37 Private nonprofit 4 - year non doctorate - granting 12,160 6,813 2,601 2,746 11,540 6,434 2,512 2,595 8,682 4,772 2,006 1,904 31 Private nonprofit 4 - year doctorate granting 13,890 7,590 2,271 4,029 13,262 7,219 2,209 3,834 9,920 5,347 1,762 2,811 45 Private for- profit less - than - 2 - year 3,650 0 3,650 0 3,482 0 3,482 0 1,998 0 1,998 0 41 Private for- profit 2 - year 6,890 0 6,890 0 6,737 0 6,737 0 4,450 0 4,450 0 48 Private for- profit 4 - year 27,870 16,509 5,631 5,730 26,768 15,782 5,506 5,481 16,646 9,682 3,636 3,327 76 NOTE: Detail may not sum to totals because of rounding. **Quality Control Checks for Lists and Sampling** The number of enrollees on each institution ’s student list will be checked against the latest IPEDS full

- year enrollment and completions data. The comparisons will be made for each student lev el: baccalaureate, undergraduate, and g raduate. Based on past experience, we expect that only counts within 50 percent of non - imputed IPEDS counts will pass quality control ( QC) and will be moved onto student sampling. We will re

- evaluate these checks after the field test for use in the full

- scale study . Institutions that fail QC will be re

- contacted to resolve the discrepancy and to verify that the institution coordinator who prepared the student list clearly understood our request and provided a list of the appropriate students . When we deter mine that the initial list provided by the institution was not satisfactory, we will request a replacement list. W e will proceed with selecting sample students when we have either confir med tha t the list received is correct or have received a corrected list. QC is ver y important for sampling and all statistical activities , and statistical procedures will underg o thorough quality control checks . W e have technical operating procedures (TOPs) in place for sampling and general prog ramming. These TOPs describe how to properly implement statistical procedures and QC checks . W

e will employ a checklist for all statisticians to use to make sure that all appropriate QC checks are done for student sampling. Some specific sampling QC checks will include, but are not limited to , checking that the:  Institutions and students on the sampling frames all have a known, non

- zero probability of selection;  Distribution of implicit stratification for institutions is r easonable; and  Number of institutions and students selected match the target sample sizes.8

**3. Methods for Maximizing Response Rates**  Response rates in the NPSAS:16 field test and full

- scale study are a function of success in two basic activities: identifying and locating the sample members involved, then contacting them and g aining their cooperation. T

wo classes of respondents are involved: institutions and students who were enrolled in those institutions . Institutions will be asked to provide data from institutional records for sampled students . In this section, we describe our plans for maximizing response to the request for data from institutional records . W e also present our plans for maximizing response to the student sur ve y. The data collection contractor for this effort will be RTI International. RTI has worked with postsecondar y institutions for multiple studies on behalf of the Department with experience in b oth developing a rapport with data providers at postsecondar y institutions and with converting student non respondents via telephone or web inter views .

**a. Collection of Data from Institutional Records**  Our plans for contacting and communicating with institution s , beginning with the process of list acquisition, are designed to ensure the cooperation of as many institutions as possible and to establish rapport with institutional staff . This process will include sending the chief administrator of each institution a package of descriptive materials about the study , follow

- up telephone calls to obtain the chief administrator’s consent and cooperation, and asking the chief administrator to designate an Institutional Coordinator (IC) who will ser ve as our primar y point of contact. All institution coordinators receive infor mation that infor ms them about the purposes of NPSAS , describes their tasks , and assures them of our commitment to maintaining the confidentiality of data. W ritten materials will be provided to coordina tors explaining each phase of the study , as well as their role in each phase . T raining of institution coordinators is g eared toward the method of data collection selected by the institution (see below). The system used for collecting institutional record d ata is accessible only with an ID and password . It provides institution coordinators with instr uctions for all phases of study participation. Copies of all written materials , as well as answers to frequently asked questions , are av ailable on the website. E xperienced NPSAS interview staff carr y out these contacts and are assigned to specific institutions , which remain their responsibility throughout the data collection process . This allows NPSAS staff members to establish rapport with the institution ’s staff and provides those individuals with a consistent point of contact. Staff members are thoroughly trained in basic financial aid concepts and in the purposes and requirements of the study

, which helps them establish credibility with the institution staff . A s an additional means of maximizing institutional participation, we have secured endorsements from 2 4 profes sional associations for NPSAS:16 (see appendix B) . NPSAS staff will offer several options for providing the Student R ecords for sampled stude nts (as in prior NPSAS studies) and will invite the coordinator to select the method that is least burdensome and most convenient for the institution. The optional methods for providing student record data are:  Student Records obtained via a web
- based data entry interface. The web

- based data entry interface displays one student at a time, and the coordinator may enter data in a top to bottom fashion before moving onto the next student.9

 Student Records obtained by completing an Excel workbook. An Excel workbook will be created for each institution and will be preloaded with the sampled students’ ID, name, and SSN (if available). To facilitate simultaneous data entry by different offices within the institution, the workbook contains a separate w orksheet for each topic area. The user will download the Excel worksheet from the secure NPSAS institution website, enter the data, and then upload the data to the website. Validation checks occur both within Excel as data are entered and when the data are uploaded via the website.  Student Records obtained by uploading CSV (comma separated values) files. Institutions with the means to export data from their internal database systems to a flat file may opt for this method of supplying Student Records. Over the last several NPSAS studies, the number of institutions providing data files has increased. Institutions that select this method will be provided with detailed import specifications, and all data uploading will occur through the project’s secure website . Institution coordinators who elect to use the web

- based data entr y interface will receiv e detailed instr uctions for acc essing and using the site . Prior to data collection, student records are matched to the U .S . Department of Education Central Processing System (CPS)

— which contains data on federal financial aid applications

— for locating purposes and to reduce the burden on the institutions for the student record abstractions . The va st majority of the federal aid applicants (about 95 percent) will match suc cessfully to the CPS prior to Student Records data collection. During data collection, institutions will be asked to provide the student’ s last name and Social Security number for the small number of federal aid applicants who did not match to the CPS on the first attempt. After Student R

ecords data collection ends , we will submit the new names and Social Security numbers to CPS for file matching . Any new data obtained for the additional students will be delivered on the Electronic Code Book (ECB) with the data obtained prior to Student Records data collection.

**b. Student Surve**  **y: Self** **- Ad**  **ministered Web and CA TI**  Methods for maximizing response to the study sur vey include: (1) tracing of sample members; (2) thorough training for all staff involved in data collecti on; (3) use of a sophisticated case management system; (4) a carefully designed sur ve y instr ument; and (5) detailed plans for averting and converting refusals . 1. Tr acing of Sample Members To achieve the desired response rate, we propose an integrated tracing approach that consists of up to 12 steps designed to yield the maximum number of locates with the least expense. During the field test, we will evaluate the effectiv eness of these procedures for the full

- scale study effort. The steps of our tracing plan i nclude the following elements .

 Matching student list infor mation with NCO A, T elematch, CPS , and other databases , which will yield locating infor mation for the students sampled for NPSAS:16 .

 Providing a system for moving locator information obtained during collection ofstudent record data quickly into CATI so that this new infor mation can be put to immediate use .

 Lead letter and other mailings as necessar y to sample members. A personalized letter (signed by an NCES official) and study brochure will be mailed to all sample members to initiate data collection. This letter will include a toll

- free number, study website address , and study ID and password, and will request that sample members complete the self

- administered inter view over the Internet. A few days after the lead letter mailing, an email messag e mirroring the l etter will also be sent to sample members .10

 Conducting batch tracing before data collection and before and after the start ofCATI as needed

 Adv ance tracing prior to the start ofCATI ef for ts. Not all schools will be able to giv e complete or up - to - date locating infor mation on each student, and some cases will require more advanced tracing, before mailings can be sent or the cases can be worked in CA TI. NPSAS staff will conduct batch tracing on all cases to obtain updated address infor mation prior to mailing the lead letters . This step will minimize the number of returned letters and maximize the number of early completes . To handle cases for which mailing address , phone number, or other conta ct infor mation is inv alid or unavailable, NPSAS staff plan to conduct advance tracing of the cases prior to lead letter mailout and data collection. This advance tracing will involve searching for address and telephone infor mation . As lead infor mation is found, additional searches will be conducted through interactive databases to expand on leads found. This will be an important step in the tracing components because of the nature of this sample. After locating infor mation is found, more advanced database searches , such as Experian, will be used, to provide more comprehensive infor mation for the individual.

 CATI tracing

 Pre
- intensive tracing including F astData and Accurint. We plan to send cases to both F astData and Accurint to identify a new phone number, to minimize the number of cases requiring more expensive intensiv

e interactive tracing. Through F astData we can tap into 260 million consumer records and over 33 million public records . We are also able to access a national director y assistance database

—updated daily

— of ove r 156 million phone numbers . Fa stData has also recently added a more comprehensive cell phone search (SuperPhones & Phone+Premium) built into existing searches; obtaining reliable cell phone numbers is bec oming an increasingly critical component of locating and inter viewing this population. Accurint is a flexible search vendor capable of providing a variety of contact infor mation for a ver y low cost per case. This vendor provides an indicator that the phone number returned has been verified as accurate and belonging to a subject in the past 24 hours . Accurint uses SSN to search, making it a viable tool for NPSAS:16 and the follow

- up studies due to the high percentag e of SSNs we expect to obtain on the studen t enrollment lists and through tracing sources and student inter views . Also during the field test, NPSAS staff will be evaluating whether matching sample members to the National Student Loan Data System (NSLDS), prior to the start of intensive tracing, imp roves locating rates .

 Conducting intensive in
-house tracing , including proprietar

y database searches. RTI’s tracing specialists conduct intensive interactive searches to locate contact infor mation for sample members . In NPSAS:08, about 60 percent of sample members requiring intensive tracing were located, and about 59 percent of those located responded to the inter view . Intensive interactive tracing differs from batch tracing in that a tracer can assess each case on an individual basis to deter mine which re sources are most appropriate and the order in which they should be used. Intensive interactive tracing is also much more detailed due to the personal review of infor mation. During interactive tracing, tracers utilize all previously obtained contact infor ma tion to make tracing decisions about each case. These intensive interactive searches are completed using a special program that works with RTI’s CMS to provide org anization and efficiency in the intensive tracing process . Sources that may be used, as appro priate, include credit database searches , such as Experian, various public websites , and other integ rated database ser vices .11

 Conducting NPSAS List Completer (NLC) searches. NLC is an RTI software application that compiles all infor mation available for the school and sample members to T racing Ser vices for additional address , phone, and e

- mail searches to be made. The application will send T racing Ser vices the school name, school web address , and total number of students to be worked. If student name, address , and phone number are av ailable, this infor mation will also be sent to the NLC. Tr acing Ser vices will then use the school web pag e directly to conduct searches and update records in the NLC for any new school or student infor mation found. This application was used on the last round of IES’s National Study of Postsecondar y F aculty and more than 4,000 new email addresses were located. Many major universities have student director y infor mation available, and NPSAS staff believe this application could allow fo r additional success on NPSAS:16 .

 University

, college, or personal web pages. 2. Tr aining for Data Collection Staff Te lephone data collection will be conducted at the contractor’s call center. NPSAS staff at the c all c enter will include Quality Control Super visors (QCSs), Help Desk Ag ents (HD As), T elephone Inter viewers (TIs), and Refusal Conversion Specialists . Tr aining prog rams for these staff members are critical to maximizing response rates and collecting accurate and reliable data. Quality control super visors , who are responsible for all super visor y tasks , will attend project

specific training for QCSs , in addition to the content of the HDS and TI training. They will receive an over view of the study , backg round and objectives , and the data collection instr ument through a question

- by

- question review . Super visors will also receive training in the following areas: providing direct super vision during data collection; handling refusals; monitoring inter views and maintaining records of monitoring results; pr oblem resolution; case review; specific project procedures and protocols; reviewing CA TI reports; and monitoring data collection prog ress . Tr aining for HD

As , who assist sample members who call the project

- specific toll

- free line, and Te lephone Inter viewers is designed to help staff become familiar with and practice using the Help Desk application and sur ve y instr ument, as well as to learn project procedures and requirements . Pa rticular attention will be paid to quality control initiatives , including refusal avoidance and methods to ensure that quality data are collected. Both HD As and TIs will receive project

- specific training on telephone inter viewing, and HD As will receiv

e additional training specifically g eared toward solving technical problems and answer ing questions from web participants reg arding the study or related to specific items within the inter view . They will also be able to unlock cases , reissue passwords , and respond to sample member e

- mail messag es , using prepared text approved by the NCES Con tracting Officer’ s Re presentative. At the conclusion of training, all HD As and TIs must meet certification requirements by successfully completing a certification interview . This evaluation consists of a full

- length inter view with project staff obser ving a nd evaluating inter viewers , as well as an oral evaluation of inter viewers’ knowledge of the study’ s Frequently Asked Questions . 3. Case Manag ement System Student inter views will be conducted using a single web

- based sur ve y instr ument for both self

administere d and CA

TI data collection. The data collection activities will be accomplished through the Case Manag ement System (CMS), which is equipped with the following capabilities:

 on - line access to locating infor mation and histories of locating efforts for each c ase;

 state - of

- the

- art questionnaire administration module with full “front

- end cleaning” capabilities (i.e., editing as infor mation is obtained from respondents);12

 sample manag ement module for tracking case progress and status; and

 automated scheduling module which deliv ers cases to inter viewers and incorporates the following features:  Automatic deliver

y ofappointment and call

- back cases at specified times. This reduces the need for tracking appointments and helps ensure the inter viewer is punctual. The sch eduler automatically calculates the deliv er y time of the case in reference to the appropriate time zone.  Sorting ofnon - appointment cases according to parameters and priorities set by project staf f . For instance, priorities may be set to give first prefere nce to cases within certain sub

- samples or g eog raphic areas; cases may be sorted to establish priorities between cases of differing status . Further more, the historic pattern of calling outcomes may be used to set priorities (e.g., cases with more than a ce rtain number of unsuccessful attempts during a given time of day may be passed over until the next time period). These parameters ensure that cases are deliv ered to inter viewers in a consistent manner according to specified project priorities .  Restriction on allowable inter viewers. Groups of cases (or individual cases) may be designated for deliver y to specific inter viewers or g roups of inter viewers . This feature is most commonly used in filtering refusal cases , locating problems , or foreign languag e cases to specific inter viewers with specialized skills .  Complete records ofcalls and tracking ofall previous outcomes. The scheduler tracks all outcomes for each case, labeling each with type, date, and time . These are easily accessed by the inter viewer upon e ntering the individual case, along with inter viewer notes , thereby eliminating the need for a paper record of calls of any kind.  Fl ag ging ofproblem cases for super visor action or super

visor r eview. For example, refusal cases may be routed to super visors f or decisions about whether and when a refusal letter should be mailed, or whether another inter viewer should be assigned.  Complete reporting capabilities. These include default reports on the ag gr egate status of cases and custom report g eneration capabilit ies . The integration of these capabilities reduces the number of discrete stag es required in data collection and data preparation activities and increases capabilities for immediate error reconciliation, which results in better data quality and reduced cost. Overall, the scheduler provides a highly efficient case assignment and deliv er y function by reducing super visor y and clerical time, improving execution on the part of inter viewers and super visors by automatically monitoring appointments and call

- back s, and reducing variation in implementing sur ve y priorities and objectiv

es . 4. Sur ve y Instr ument Design To prepare the student records instr ument, in J anuar y 2014, NCES convened a technical review panel to discuss the challeng es to responding to the NPSAS lis t and student records data collection requests and approaches that might facilitate the process . In J une 2014, NCES receiv ed approval to conduct focus groups with institution staff who have participated in past NPSAS student records data collection. This qualitative evaluation infor m ed refinement of items used in the student records instr ument for which clearance is requested , and system functionality . Student inter view preparation has involved two meetings of the NPSAS technical review panel. The J une 201 4 meeting focused specifically on the design and content of the graduate student13

portion of the sur ve y, while the August 2014 meeting covered the surve y more broadly , across all topics and student levels . Since the August meeting, cognitive testing with gr aduate students has helped to clarify concepts to be used in the student sur ve y, and highlighted differences across the va rious levels and degrees of g raduate students . Fo llowing the field test data collection, additional cognitive and usability is planned with the prog rammed, mobile

- friendly instr ument. The NPSAS:16 instr uments will employ a web

- based instr ument and deployment system, created by RTI, known as Hatteras which has been in use since NPSAS:08 . Hatteras is a flexible system that provides multimo de functionality , whereby the sur ve y instr ument is created one time and can be used for self

- administration, including on mobile devices , CA TI, CAPI, or data entr y. In addition to the functional capabilities of the CMS and web instr ument s described above, our efforts to achieve the desired response rate will include using established procedures proven effective in other larg e - scale studies we hav e completed. These include:

 Providing multiple response modes , including a mobile

- friendly self

- administered and inter viewer

- administered options .

 Offering incentives to encourag e response (see incentive str ucture described below).

 Assigning experienced CA TI data collectors who have proven their ability to contact and obtain cooperation from a high proportion of samp le members .

 Tr aining the inter viewers thoroughly on study objectives , study population characteristics , and approaches that will help g ain cooperation from sample members .

 Maintaining a high level of monitoring and direct super vision so that inter viewers who are experiencing low cooperation rates are identified quickly and corrective action is taken.

 Making ever y reasonable effort to obtain an inter view at the initial contact, but allowing respondent flexibility in scheduling appointments to be inter viewed.

 Thoroughly reviewing all refusal cases and making special conversion efforts whenever feasible (see next section). 5. Refusal A

ve rsion and Conversion Recognizing and avoiding refusals is important to maximize the response rate. W e will emphasize this and oth er topics related to obtaining cooperation during data collector training. Super visors will monitor inter viewers intensely during the early days of data collection and provide retraining as necessar y. In addition, the super visors will review daily inter viewer production reports produced by the CA

TI system to identify and retrain any data collectors who are producing unacceptable numbers of refusals or other problems . After encountering a refusal, the data collector enters comments into the CMS record. These comments include all pertinent data reg arding the refusal situation, including any unusual circumstances and any reasons given by the sample member for refusing. Super visors will review these comments to deter mine what action to take with each refusal. No refusal or partial inter view will be coded as final without super visor y review and approval. In completing the review , the super visor will consider all available infor mation about the case and will initiate appropriate action. If a follow

- up is clearly in appropriate (e.g., there are extenuating circumstances , such as illness or the sample member fir mly requested that no further contact be made), the case will be coded as final and will not be recontacted. If the case appears to be a “soft” refusal, follow

- up will be assigned to an inter viewer other than the one who receiv ed the initial refusal. The case will be14

assigned to a member of a special refusal conversion team made up of inter viewers who hav e proven especially adept at converting refusals . Refusal c onversion efforts will be delayed for at least one week to giv e the respondent some time after the initial refusal. Attempts at refusal conversion will not be made with individuals who become verbally ag gr essiv e or who threaten to take leg al or other actio n. Refusal conversion efforts will not be conducted to a degree that would constitute harassment. W e will respect a sample member’s right to decide not to participate and will not imping e this right by carr ying conversion efforts beyond the bounds of propr iety . **4. Tests of Procedures and Methods**  NCES’s g oal for the full

- scale NPSAS:16 study is to reduce t otal e rror compared to NPSAS:12 so that infor med decisions may be made given the resources provided. This NPSAS:16 field test will address some, although not all of the initiatives that will be implemented in the full

- scale study . Through an analysis of previous NPSAS studies , NPSAS staff is focusing efforts on the following sources of error which are likely to have the larg est impacts on improving estimates . S pecifically:

**Fr ame Er**  **ror** – In order to reduce sampling error due to the sampling frame, we will be using more recent data from IPEDS for the sampling frame to reduce the reliance on post

- stratification adjustment and yield smaller standard errors . **Co vera**  **ge Er**  **ror** – NPSAS staff will re

- test tracing and contacting procedures to reduce non

response due to bad contact infor mation. **Measurement Er ror due to questionnaire design** 

– This field test is testing new questions that were previously tested in cognitive labs . Other questions are being piloted and will be revised using cognitive labs following this data collection. The experiments in the field test are also designed to test if error is being introduced due to how the items are designed. The field test will also allow NPSAS staff to test if supplemental help text is having the desired effect. **Measurement Er ror due to v** **ariability in telephone inter viewer** 

– The field test will test procedures for training telephone inter view staff . We are exploring additional strategies to employ in the full

- scale study to reduce t otal e rror, including responsive/adaptive design to reduce non

- response error, adding loan infor mation/data from NSLDS to the sampling design to reduce frame error, and upd ating poststratification control totals to better match the NPSAS targ et population. The NPSAS:16 field test will include two sets of data collection experiments: the first set of experiments focuses on sur ve y participation, to address nonresponse, and the second set focuses on data accuracy to further investig ate measurement error.

**a. Experiment # 1:**  **Evaluation of Burden and Motivation to Participate** In part because NPSAS:16 will launch a mobile option of the entire sur ve y, a considerable challeng e during dat a collection will likely be to convince sample members to start a 30 minute surve y, raising concern that participation rates in the student sur ve y will decline. Motivated by the “foot in the door” approach (Freedman and Fraser, 1966), where a small request is followed by a larg er request, we propose dividing the sur ve y instr ument in half, into two modules of appro ximately 10 to 15 minutes in duration , to deter mine if breaking the sur ve y into two smaller tasks increases the likelihood that sample members will participate .15

Sample members will be assigned at random into one of three conditions described below . 1. Tr eatment Group 1 : Sample members will be asked to first complete a 10 to 15 minute sur ve y – Module 1

– for an initial incentive offer of $15. At the end of the first module, respondents will be offered the option to continue with the second half of the sur ve y, Module 2, also 10 to 15 minutes in duration, to receive another $15 (for a total of $30 for the sur ve y) . 2. Tr eatment Group 2 : Sample member s will be asked to complete the same 10 to 15 minute Module 1 for an initial incentiv

e offer of $20 , then will be offered the opportunity to continue with Module 2, also 10 to 15 minutes in duration, to receive another $10 (for a total of $30 for the sur ve y) . 3. Control Gr

oup : Sample members will be asked to complete the usual 30 minute surve y, like in prior NPSAS data collections . They will receiv e a $30 incentive for the completed surve y. Fo r the first time, sample members will be offered the option of recei ving their incentives by check, which typically takes 3 to 4 weeks for delivery, or by P ayPal, which can be accomplished immediately upon completion of a module. The modules will include questions designed specifically to assess measurement error due to re call, fatigue, lack of motivation, and so on . Module 1 will include questions to obtain infor mation also available from administrative sources to evaluate accuracy (N16CFED AMT) . Some of those same items will be repeated in Module 2. In addition, Module 2 w ill include questions on fictitious issues (N16ASNO W and N16SPNNO

W) and some questions with reversed wording (N16ACDSA TIS and N16SA

TISACD) . T he three gr oups will be evaluated on participation and response rates , breakoff, and timing.

**b. Experiment #**  **2: Questionnaire Design**  Another potential source of measurement error is the sur ve y instr ument itself . Given the mobile sur ve y option that will be launched for NPSAS:16, we will need to introduce chang es to the sur ve y instr ument design to facilitate completio n on a smaller screen size with mobile phone navigation. Two types of question designs will be restr uctured and evaluated across devices . **Questionnaire Design 1** a) Design Group 1 : Rather than presenting the question on high school coursework as a grid, which asks and obtains responses to several questions in a single screen, a series of three yes/no questions will be asked in succession . Control Gr oup 1 : The alternative design will be the standard presentation in which question s about high school courses are a sked in a single gr id . **Questionnaire Design 2** b) Design Group 2 : Rathe r than asking just one question , parents’ education, w ith several response o ptions , that may not properly display on the screen of a mobile device, we will use a branching design in which a general question with limited response options will be asked firs t , followed by a series of more specific branching question s to obtain the details . Control Gr oup 2 : The alternative design will be the standard presentation in which the question of parents ’ education will be asked as a single question.16

**Questionnaire Design 3** c) Design Group 3 : Among students who report having studied abroad , a follow

- up question will ask respondents to select from a list all countries in which they have studied abroad . Control Gr oup 3 : The alternative design will present respondents with a text box in which to list all countries in which they have studied abroad . **c. Experimental Design**  The “ motivation to participate ” and three questionnaire design experiments described above will test the numerous hypotheses outlined below. The experimental design includes estimation of the minimum difference between the control and treatment groups necessar y to detect statistically significant differences . The control and treatment g roups with the null hypotheses to be tested are defined as follows: **Experiment # 1: Ev**

**aluation of**

**Burden and Moti**

**va tion to P**  **art icipate**   Treatment group 1

– Participants will be asked to complete a 10

- 15 minute survey and be offered $15. At the end of the 15 minute s urvey, participants will be asked if they would like to continue with another 15 minute survey and receive an additional $15.  Treatment group 2

– Participants will be asked to complete a 10

- 15 minute survey and be offered $20. At the end of the 15 minute s urvey, participants will be asked if they would like to continue with another 15 minute survey and receive an additional $10.  Control group

– Participants will be asked to complete a 30 minute survey and receive a $30 incentive. 1) Response rates will not be different for the two treatment groups combined than for the control g roup . 2) Response rates for the first module will not be different for the two treatment groups combined than for the control g roup . 3) Response rates for the second module will not be differe nt for the two treatment groups combined than for the control g roup . 4) Breakoff rates will not be different for the control g roup than for the two treatment groups combined. 5) Pa rticipation rates will not be different for the two treatment groups combined than for the control g roup . 6) Mean time to complete the first module will not be different for the two treatment groups combined than for the control g roup . 7) Mean time to complete will not be higher for the first module for the two treatment groups combined than for the second module for the two treatment groups combined. 8) Item missingness rates for the first module will not be different for the control g roup than for the two treatment groups combined. 9) Item missingness rates will not be higher for the second module of the two treatment groups combined than for the first module of the two treatment groups combined. 10) Pe rcentage differences between estimates and benchmarks will not be different for the control gr oup than for the two treatment groups combined.17

11) Pe rcentage differences between estimates and benchmarks will not be higher for the second module of the two treatment groups combined than for the first module of the two treatment gr oups combined. 12) Pe rcentages of substantive responses to fictitious issues will not b e different for the control gr oup than for the two treatment groups combined. 13) Pe rcentages of substantive responses to fictitious issues will not be higher for the second module of the two treatment groups combined than for the first module of the two treatment gr oups combined. 14) Pa rticipation rates will not be different for treatment group 1 than for treatment g roup 2. 15) Response rates will not be different for treatment group 1 than for treatment g roup 2. **Experiment # 2 :**  **Questionnaire Design** Questionnaire Design 1  Treatment group

– Participants will receive the high school course questions as three yes/no questions.  Control group

– Participants will receive the high school course questions in a grid format. 16) Mean time to complete the inter view will not be higher for the treatment group than for the control g roup . 17) Item missingness rates will not be different for the treatment group than for the control gr oup . 18) Pe rcentage differences between estimates and benchmarks will not be different for the tr eatment group than for the control g roup . Questionnaire Design 2  Treatment group

– Participants will receive the parents’ education questions as a series of branching questions.  Control group

– Participants will receive the parents’ education questions in a checkbox format. 19) Mean time to complete the inter view will not be higher for the treatment group than for the control g roup . 20) Item missingness rates will not be different for the treatment group than for the control gr oup . 21) Pe rcentage differences between es timates and benchmarks will not be different for the treatment group than for the control g roup . Questionnaire Design 3  Treatment group

– Participants will receive the study abroad question in a n open - ended format .  Control group

– Participants will receive the study abroad question in a closed

- ended format. 22) Item missingness rates will not be different for the treatment group than for the control gr oup . 23) The percentag e difference between the mean number of countries provided for the treatment gr oup and the co ntrol g roup will be zero .18

**d. Detectable Differences**  The differences between the control and treatment group(s), the two modules , and the two treatment gr oups necessar y to detect statistically significant differences are shown in table X. The following assumpt ions were made in computing detectable differences: 1. Detectable differences with 95 percent confidence were calculated as follows: a. Hypotheses 7, 9, 11, 13, 16, and 19 assume a one

- tailed test. b. Hypotheses 1, 2, 3, 4, 5, 6, 8, 10, 12, 14, 15, 17, 18, 20, 21 , 22, and 23 assume a two - tailed test. 2. The sample will be equally distributed across the three experimental groups (control and two treatment groups) for hypotheses 1 through 15. 3. The sample will be equally distributed between the two experimental groups (con trol and treatme nt) for hypotheses 16 through 23 . 4. Analysis of participation rates in h ypotheses 5 and 14 will include all sample members , both eligible and ineligible students . 5. Analysis of response rates in h ypotheses 1, 2, 3, and 15 will include all eligible sample members . 6. Analysis of h

ypotheses 4, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21 , 22, and 23 will include respondents . 7. The eligibility rate will be 95 percent. 8. Comparisons between the first and second modules for the two treatment groups co mbined will include only those students who completed both modules . 50 percent of students responding to first module will also respond to second module. 9. The response rate for the control g roup in hypotheses 1, 2, and 3 will be 60 percent. 10. The breakoff rat e for the control group in hypothesis 4 will be 20 percent. 11. The participation rate for the control g roup in hypothesis 5 will be 60 percent. 12. The mean time to complete the first module for the control group for h ypothesis 6 will be 15 minutes . 13. The mean time to complete the second module for the two treatment groups combined for hypothesis 7 will be 15 minutes . 14. The item missingness rate for the first module of the control g roup for hypothesis 8 will be ten percent. 15. The item missingness rate for the second module of the two treatment groups combined for hypothesis 9 will be ten percent. 16. The percentag e difference between estimates and benchmarks for the control group for hypothesis 10 will be 10 percent. 17. The percentag e difference between estimates and benchmarks for the second module of the two treatment groups combined for h ypothesis 11 will be 10 percent 18. The percentag e of substantive responses to fictitious issues for the control g roup for hypothesis 12 will be 30 percent. 19. The percentag e of substantive response s to fictitious issues for the second module of the two treatment groups combined for h ypothesis 13 will be 30 percent. 20. The participation rate for treatment g roup 1 in h ypothesis 14 will be 60 percent. 21. The response rate for treatment g roup 2 in h ypothesis 15 will be 60 percent. 22. The mean time to complete the inter view for the control group for h ypotheses 16 and 19 will be 15 minutes .19

23. The item missingness rate for the control g roup for h ypotheses 17, 20 , and 22 will be ten percent. 24. The percentag e difference b etween estimates and benchmarks for the control group for hypotheses 18 and 21 will be 10 percent. 25. The statistical tests will have 80 percent po wer with an alpha of 0.05. 26. The study design effect will be about 2.0. 27. The intraclass correlation will be about 0 .2 when comparing the control group with the treatment gr oup(s). 28. The intraclass correlation will be about 0.8 when comparing the first module of the treatment gr oups with the second module. **Ta ble**  **12 .**  **Detectable differences for field test experiment hypotheses**  **Hy pothesis**  **Control group Tr** **eatment group Detectable**  **difference with 95 percent confidence**  **Definition Sample size**  **Definition Sample size**  1 30 minute survey 1,429 Two 10 - 15 minute survey modules 2,857 5.7 2 30 minute survey 1,429 Two 10 - 15 minute survey modules 2,857 5.7 3 30 minute survey 1,429 Two 10 - 15 minute survey modules 2,857 5.7 4 30 minute survey 1,000 Two 10 - 15 minute survey modules 2,000 5.7 5 30 minute survey 1,504 Two 10 - 15 minute survey modules 3,007 5.6 6 30 minute survey 1,000 Two 10 - 15 minute survey modules 2,000 3.7 minutes 7 Second of the two 10 – 15

minute survey modules 1,000 First of the two 10 - 15 minute survey modules 1,000 1.9 minutes 8 30 minute survey 1,000 Two 10 - 15 minute survey modules 2,000 4.4 9 Second of the two 10 – 15

minute survey modules 1,000 First of the two 10 - 15 minute survey modules 1,000 2.0 10 30 minute survey 1,000 Two 10 - 15 minute survey modules 2,000 4.4 11 Second of the two 10 – 15

minute survey modules 1,000 First of the two 10 - 15 minute survey modules 1,000 2.0 12 30 minute survey 1,000 Two 10 - 15 minute survey modules 2,000 6.4 13 Second of the two 10 – 15

minute survey modules 1,000 First of the two 10 - 15 minute survey modules 1,000 3.2 14 Two 10 - 15 minute survey

modules for $15 each 1,504 Two 10 - 15 minute survey modules for $20/$10 1,504 6.4 15 Two 10 - 15 minute survey

modules for $15 each 1,429 Two 10 - 15 minute survey modules for $20/$10 1,429 6.6 16 Grid format 1,500 Three new questions 1,500 3.1 minutes 17 Grid format 1,500 Three new questions 1,500 4.2 18 Grid format 1,500 Three new questions 1,500 4.2 19 Checkbox format 1,500 Branching questions 1,500 3.1 minutes 20 Checkbox format 1,500 Branching questions 1,500 4.2 21 Checkbox format 1,500 Branching questions 1,500 4.2 22 Closed - ended format 225 Open - ended format 225 12.2 23 Closed - ended format 225 Open - ended format 225 6.720

**5. Reviewing Statisticians and Individuals Responsible for Designing and**  **Conducting the Study** The study is being conducted for the National Center for Education Statistics (NCES), U .S . Department of Education. NCES’s prime contractor is the RTI International (RTI). Subcontractors include Coffey Consulting; Her mes; HR Directions; Kforce Government S olutions , Inc.; Research Support Services; Shug oll Research; and Strategic Communications , Inc. Consultants are Dr. Sandy Baum, Dr. Stephen Porter, and Ms . Alisa Cunningham. Principal professional RTI staff, not listed above, who are assigned to the study include Mr. J eff Franklin, Ms . Christine Rasmussen, Ms . Kristin Dudley , Ms . Annaliza Nunnery

, and Ms . Tiffany Mattox. The following statisticians at NCES are responsible for the statistical aspects of the study: Dr. T racy Hunt

-White, Dr. Sarah Crissey , Dr. Sean Simone, Mr. T

ed Socha, and Dr. Elise Christopher. The following staff members at RTI are working on the statisti cal aspects of the study design: Dr. J ennifer Wine , Dr. J ames Chromy , Dr. Natasha J anson , Mr. P eter Sieg el , Dr. David Wilson , Dr. Br yan Sh epherd , Dr. Emilia P eytcheva , Dr. Andy P eytchev

, Dr. J ohn Riccobono , Mr. David Radwin , and Dr. J ennie W oo . 

### C. **References** Fo lsom, R.E., Potter, F

.J., and Williams , S .R. (1987). Notes on a Composite Size Measure for Self

We ighting Samples in Multiple Domains . Proceedings of the Section on Sur ve y Research Methods of the American Statistical Association, 792

- 796.