# Appendix I Response to OMB Questions Dated January 25, 2008 {#appendix-i-response-to-omb-questions-dated-january-25-2008 .Appendix-Title}

**Memorandum United States Department of Education**

**Institute of Education Sciences**

National Center for Education Statistics

TO: Edie McArthur February 1, 2008

Assistant to the Commissioner, NCES

FROM: Kristin Perry, Postsecondary Studies Division, NCES

Jennifer Wine, RTI International

SUBJECT: Responses to OMB Questions, dated January 25, 2008 (pertaining
to ED-CO-05-0011, Option 6, BB:08/09)

**1. What is the overall nonresponse bias analysis plan(s) for B&B? 
There are several references to incentives potentially increasing
response and thereby reducing potential bias.  But how will this be
measured?  How will NCES be able to demonstrate that such steps have not
increased bias?**

> The overall nonresponse bias analysis plan for the full-scale
> B&B:08/09 study is to use data that are available for both responding
> and nonresponding sample members to estimate bias in estimates due to
> unit nonresponse. There will be considerable information on sample
> members available from the base year study (NPSAS:08) to facilitate
> the analysis (e.g., NPSAS response status, age, attendance status,
> geographic region, number of telephone numbers obtained, and aid
> status). Any variables found to have significant bias due to
> nonresponse will be included in the weight adjustment model so that
> the bias may be reduced for analyses based on the final statistical
> analysis weights. Additionally, we will consider adding as a predictor
> in the model an indicator of incentive received or time period when
> interview was completed, i.e., early response, production response, or
> nonresponse conversion. Another possible analysis would compare sample
> members who responded at different points in time, i.e., early
> responders, production responders, and nonresponse conversions, to
> determine if bias was introduced by completing interviews at different
> points in time with different incentives. After completing weight
> adjustments, the reduction or removal of significant unit nonresponse
> bias will be validated by comparing the distribution of key variables,
> known for most respondents and nonrespondents using the final weights
> after weight adjustments, with the full sample distribution before
> nonresponse adjustment.
>
> However, the direct effect of incentives on bias cannot be measured
> accurately during the full-scale data collection because, as designed,
> all sample members will be offered at least one incentive. The
> incentives are intended specifically to increase the overall response
> rate, which \-- if effective \-- will reduce overall nonresponse bias.
> Any significant bias remaining after weight adjustments may indicate
> that either the weighting or the incentives did not decrease or
> eliminate nonresponse bias as expected.
>
> In an effort to address the notion of bias due to nonresponse, it may
> be possible to conduct some comparisons of respondent/nonrespondent
> characteristics during the field test, although the sample sizes will
> be quite limited.

**2. Please provide the full report (or a link) from where the 2003
incentives experiments discussion was excerpted.**

> See pages 21-22 of the report:
>
> Wine, J., Cominole, M., Carwile, S., Franklin, J., Carley-Baxter, L.,
> and Wheeless, S. (2004). *1993/03Baccalaureate and Beyond Longitudinal
> Study (B&B:93/03) Field Test Methodology Report* (NCES 2004--02). U.S.
> Department of Education. Washington, DC: National Center for Education
> Statistics.
>
> Web link: <http://www.nces.ed.gov/pubsearch/pubsinfo.asp?pubid=200402>

**3. Does NCES intend to study the number of "good" addresses from the
various locating sources, as well as the unique contributions of each
vendor (versus duplicative contributions)?  The submission currently
refers only to the number of addresses received, some of which may be
erroneous or duplicative of information being received from other
sources. **

> For batch processing, it is possible to determine which locating
> source provided the address at which the sample member was located and
> interviewed and, therefore, it lends itself to analysis. In tracing
> the B&B sample members, for example, RTI will first request address
> updates for the entire sample from three batch tracing databases: (1)
> the Central Processing System (CPS) database, (2) the U. S. Postal
> Service's National Change of Address (NCOA) database, and (3)
> Telematch. These sources are noncompeting, since they each provide
> unique locating data. That is, matches to CPS return the permanent
> addresses of students who have applied for financial aid. Those
> addresses, combined with any other addresses for the entire sample,
> will be sent to NCOA for updating. Updated addresses will be sent to
> Telematch for new telephone numbers. All resulting data will be loaded
> into the case management system to be used for mailouts, prompting,
> and initial calls by interviewers. Since the source of the information
> is retained in the database, RTI can analyze and report on the source
> of the information obtained, although it is possible that multiple
> sources will provide duplicative information. When that occurs, there
> is usually a higher probability that the sample member is at the
> particular location.
>
> Analysis of the quality of locating sources during intensive tracing
> is considerably more complicated and, therefore, more difficult to
> analyze the quality of a particular vendor's information. Once
> telephone interviewers submit a case to RTI's intensive tracing
> process, locating becomes a process of networking. As a result, it is
> difficult to precisely define a "good" address. For example, tracers
> will start with the interview record, which has the last known/best
> addresses. If the sample member is not found at that location, tracers
> will probe to determine if he/she ever lived at the location, if
> anyone there knows the sample member, and if a new telephone number or
> address can be provided. If a new number is obtained, tracers will add
> the number to the record and attempt to reach the sample member. They
> will continue to network in this way until they are able to locate a
> sample member. No one vendor can be identified as providing the
> correct address, but each of several vendors may have provided the
> pieces that led to the sample member. Consequently, simply reporting
> hits and misses by vendor will understate each individual vendor's
> value to the tracing process.
>
> The management team for RTI's tracing unit has over 30 years of
> tracing experience, RTI quality control supervisors have at least 3
> years of experience in tracing, and tracing staff receive 12 hours of
> classroom training and 8 hours of production tracing before they start
> to work on a project such as B&B. Over the years, RTI has made
> extensive use of locating sources such as Accurint, Axiom,
> Choicepoint, Equifax, Experian, FastData, LexisNexis, and TransUnion.
> These sources have been constantly monitored for quality and cost
> effectiveness. When vendors are found to be deficient, they are phased
> out temporarily or permanently depending upon reasons and
> circumstances.
>
> Although there are some exceptions, most credit bureaus (e.g.,
> Equifax, Experian) and proprietary databases (e.g., FastData,
> Accurint) gather most information from identical or very similar
> sources. What makes each of them unique and independently valuable is
> their regional coverage, different logic structures, search criteria,
> and output format. RTI tracers will make recommendations for which
> sources are best for a particular study based on characteristics of
> the sample to be located. For example, while FastData was recommended
> for locating parents participating in NCES' Early Childhood
> Longitudinal Study (ECLS), Accurint was recommended for locating
> student sample members participating in B&B and the Beginning
> Postsecondary Students (BPS) Longitudinal Study.