NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

The Working Paper Series was initiated to promote the sharing of the

valuable work experience and knowledge reflected in these preliminary

reports. These reports are viewed as works in progress, and have not

undergone a rigorous review for consistency with NCES Statistical

Standards prior to inclusion in the Working Paper Series.

U. S. Department of Education

Office of Educational Research and Improvement
NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

Secondary Statistical Modeling With the

National Assessment of Adult Literacy:

Implications for the Design of the

Background Questionnaire

Working Paper No. 2000-05 March 2000

Contact: Sheida White

Assessment Division

E-mail: sheida_white@ed.gov

U. S. Department of Education

Office of Educational Research and Improvement
U.S. Department of Education

Richard W. Riley

Secretary

Office of Educational Research and Improvement

C. Kent McGuire

Assistant Secretary

National Center for Education Statistics

Gary W. Phillips

Acting Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing,

and reporting data related to education in the United States and other nations. It fulfills a congressional

mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in

the United States; conduct and publish reports and specialized analyses of the meaning and significance of

such statistics; assist state and local education agencies in improving their statistical systems; and review

and report on education activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent, reliable,

complete, and accurate indicators of education status and trends; and report timely, useful, and high quality

data to the U.S. Department of Education, the Congress, the states, other education policymakers,

practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is appropriate to a

variety of audiences. You, as our customer, are the best judge of our success in communicating information

effectively. If you have any comments or suggestions about this or any other NCES product or report, we

would like to hear from you. Please direct your comments to:

National Center for Education Statistics

Office of Educational Research and Improvement

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

The NCES World Wide Web Home Page is

http://nces.ed.gov

Suggested Citation

U.S. Department of Education. National Center for Education Statistics. Secondary Statistical Modeling

With the National Assessment of Adult Literacy: Implications for the Design of the Background

Questionnaire, Working Paper No. 2000-05, by David Kaplan. Project Officer, Sheida White.

Washington, DC: 2000.

March 2000
iii

Foreword

In addition to official NCES publications, NCES staff and individuals commissioned by

NCES produce preliminary research reports that include analyses of survey results, and

presentations of technical, methodological, and statistical evaluation issues.

The Working Paper Series was initiated to promote the sharing of the valuable work

experience and knowledge reflected in these preliminary reports. These reports are viewed as

works in progress, and have not undergone a rigorous review for consistency with NCES

Statistical Standards prior to inclusion in the Working Paper Series.

Copies of Working Papers can be downloaded as pdf files from the NCES Electronic

Catalog (http://nces.ed.gov/pubsearch/), or contact Sheilah Jupiter at (202) 502-7444,

e-mail: sheilah_jupiter@ed.gov, or mail: U.S. Department of Education, Office of Educational

Research and Improvement, National Center for Education Statistics, 1990 K Street, NW, Room

9048, Washington, DC 20006.

Marilyn M. McMillen Ralph Lee

Chief Mathematical Statistician Mathematical Statistician

Statistical Standards Program Statistical Standards Program
This page intentionally left blank.
Secondary Statistical Modeling With the

National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Prepared by:

David Kaplan

University of Delaware

Prepared for:

U.S. Department of Education

Office of Educational Research and Improvement

National Center for Education Statistics

March 2000

This project was an activity of the Education Statistics Services Institute.
This page intentionally left blank.
vii

Table of Contents

Foreword...................................................................................................................................................................... iii

Abstract...................................................................................................................................................................... viii

Introduction....................................................................................................................................................................1

1. Issues of Respondent Sampling .......................................................................................................................1

1.1 Issues in Clustered Sampling and Multilevel Modeling...................................................................................2

1.2 Recommendations to NCES ............................................................................................................................3

2. Issues of Selection Bias...................................................................................................................................4

2.1 Methods for Modeling Selection .....................................................................................................................5

2.2 Recommendations to NCES ............................................................................................................................7

3. Issues of Measurement.....................................................................................................................................7

3.1 Recommendations to NCES ............................................................................................................................9

4. Issues Pertaining to Policy Modeling ............................................................................................................10

4.1 Obtaining “Exogenous” Policy-Relevant Variables.......................................................................................11

4.2 Natural Metrics of Observed Variables.........................................................................................................11

4.3 Recommendations to NCES ..........................................................................................................................12

5. Issues Relating to Gauging Cohort Effects....................................................................................................12

5.1 Obtaining Retrospective Background Questions...........................................................................................12

5.2 Linking NAAL With Existing Longitudinal Surveys.....................................................................................13

5.3 Recommendations to NCES ..........................................................................................................................14

References....................................................................................................................................................................15
viii

Abstract

With the advent of simple-to-use advanced statistical software packages, it is becoming increasingly easy to

specify and estimate complex statistical models for addressing substantive questions in adult literacy and other areas

of education. Such problems as the role of literacy in voting behavior or the job market experiences of individuals

with GEDs, can be addressed with relative ease. However, a thoughtful application of statistical models to

educational data leads to the recognition that certain assumptions must be met for the model estimates to be useful

for theoretical explanation and/or policy analysis. The purpose of this paper is to offer recommendations to the

National Center for Education Statistics on the development of the background questionnaire for the National

Assessment of Adult Literacy. The recommendations presented in this paper are from the viewpoint of a researcher

interested in applying sophisticated statistical models to address important issues in adult literacy. This paper will

focus on five issues: (1) sampling, (2) selection bias, (3) measurement, (4) policy analysis, and (5) cohort effects.
1

Introduction

With the advent of simple-to-use advanced statistical software packages, it is becoming increasingly easy to

specify and estimate complex statistical models for addressing substantive questions in adult literacy as well as other

areas of education. However, a thoughtful application of statistical models to educational data leads to the

recognition that certain assumptions must be met for the model estimates to be useful for theoretical explanation

and/or policy analysis. These assumptions concern the sampling of respondents, measurement of constructs, and

selection bias, to name a few.

Pursuant to the Call for Papers on the National Assessment of Adult Literacy (NAAL), this paper provides

technical recommendations to the National Center for Education Statistics (NCES) as it plans for and executes the

NAAL. These recommendations specifically take into account the application of advanced statistical modeling of

secondary public use data.

The technical suggestions provided in this paper are guided by many of the recommendations summarized

in NCES Working Paper No. 98–17 entitled “Developing the National Assessment of Adult Literacy:

Recommendations from Stakeholders” (hereafter referred to as NCES 98–17). Specifically, this paper emphasizes

the utility of the background questionnaire for basic research and policy studies. This report addresses the following

five issues: (1) sampling, (2) selection bias, (3) measurement, (4) policy modeling, and (5) cohort effects.

1. Issues of Respondent Sampling

The issue of sampling discussed in this section concerns the ability to draw accurate inferences about the

population of adults. Fortunately, solutions to this problem are relatively straightforward. Specifically, many national

educational surveys employ multistage clustered sampling with schools sampled first, followed by the sampling of

students and teachers within those schools. Drawing proper inferences from data generated from such sampling

designs not only requires proper statistical modeling, such as multilevel modeling, but also requires the incorporation

of sampling weights.
2

The issue of sampling weights also applies to data drawn from other multistage cluster designs, such as the

household survey sample that will be utilized for NAAL. As noted in NCES 98–17, there is a desire to assess

underrepresented groups based on specific policy needs. These groups include rural adults, adults in welfare-to-work

programs, Native Americans, blacks, and Hispanics, to name a few. The issue of oversampling is subtle, but clearly

important if one wishes to make accurate inferences about underrepresented populations.

In the context of advanced statistical modeling, it is essential that oversampled groups not exert undue

influence on statistical estimates or conclusions of model adequacy. To mitigate this problem it will be essential to

provide the sampling weights for all respondents in the survey. Many statistical software packages allow for the

incorporation of sampling weights in regression-based methodologies.

1.1 Issues in Clustered Sampling and Multilevel Modeling

With the use of multilevel modeling for the analysis of hierarchical data (Bryk and Raudenbush 1992), great

flexibility now exists to study educational issues in the context of the organizational structure of education. Surveys

sponsored by the NCES, such as the NELS:88, sample respondents in ways that reflect the natural structure of

educational organizations, viz., students nested in schools. Indeed, relative to the early discussion, such datasets also

contain appropriate weights at each level and utilize software programs such as hierarchical linear modeling (HLM)

(Bryk 1996).

With regard to the structure of the NAAL background questionnaire, the question is whether the proposed

sampling design yields natural organizational structures that might lend themselves to the application of multilevel

modeling. For example, the National Adult Literacy Survey (NALS) defined a four-stage stratified sampling design

in which primary sampling units (PSUs) consisted of geographic clusters of one or more adjacent counties (first

stage). This was followed by Census blocks within counties (second stage), households within Census blocks (third

stage), and finally, adults within households (fourth stage). To the extent that these PSUs represent groups of

substantive importance to issues of literacy, it may be useful to examine variation in respondent literacy as a function

of demographic characteristics, such as poverty levels, captured by the PSUs.
3

An application of multilevel modeling to the NALS data was recently reported by Sheehan-Holt and Smith

(1999). In their analysis, they used census blocks (segments) as the contextual, or level-two, unit of analysis. They

argued that census blocks, as defined in the NALS sampling design, could serve as a proxy for neighborhood, but

noted that census block level variables were not collected. In their case, they created a “neighborhood average

income” based on the income reported by respondents within the segments.

1.2 Recommendations to NCES

Recommendations for NCES in the construction of the NAAL survey are twofold. First, it is crucial that the

public-use NAAL data contain all appropriate sampling weights in order for accurate inferences based on advanced

secondary statistical analyses to be made. The literature is clear regarding the problem associated with inaccurate

inferences when sampling weights are ignored. Every effort should be made to encourage the use of sampling

weights in secondary analyses.

Second, there is no doubt as to the importance of examining contextual effects on the distribution of literacy

ability in the population. In terms of the design of the next background questionnaire, every attempt should be made

to link the NAAL survey to census-based surveys so that more detailed neighborhood variables can be collected.

Indeed, recent research by Archbald, Kaplan, and Nakib (conducted under OERI grant # R308F60010, 1996-1998)

showed how it was possible to link the National Assessment of Educational Progress (NAEP) to census data and

ultimately to the district data codebook. This allowed for the incorporation of data at the level of the school district to be

merged with data from the NAEP background questionnaire for the purposes of multilevel modeling. We strongly

recommend that the NAAL provide similar links to allow for the development of contextual models of adult literacy.

If interest centers on the variation of respondent literacy as a function of group-level characteristics, such as

GED programs, then it is required that a different form of sampling be employed. For example, some form of

clustered sampling of programs followed by respondents within programs would allow application of multilevel

modeling methods. If such a sub-study is of relevance to the larger goals of NAAL, then the questionnaire should

capture as many features of the group-level variables as possible. With regard to sampling weights, these should be

made available at both the group and respondent levels.
4

One practical recommendation that comes from a consideration of sampling weights is that very clear

guidelines regarding their use should be provided. That is, NCES should make available in their NAAL

documentation clear instructions on how to incorporate sampling weights in secondary analyses of the data.

2. Issues of Selection Bias

The issue of selection bias refers to problems of inference that arise from the nonrandom participation of

respondents and their assignment to groups. A recommendation that appeared in NCES 98–17 was that background

data should provide information regarding participation in adult education programs as well as participation in

citizenship activities. However, in both cases, individuals are not randomly assigned to groups that either participate

or do not participate in these activities. As an example, consider the problem of literacy and voting (Kaplan and

Venezky 1994; Venezky and Kaplan 1998). Before being able to study the impact of literacy on voting behavior, it is

first necessary to recognize that the lack of random assignment into voting and non-voting groups hinder attempts to

make claims about the role of literacy on voting behavior.

The difficulty with asserting claims in this case arises from the fact that those who state that they have voted

are not a random sample of the general population who are eligible to vote. Thus, we need a model for estimating the

probability of observing an affirmative response to the voting question. After accounting for age and citizenship

requirements, it is clear that one cannot vote unless one is registered to vote. However, the probability of registering

to vote is dependent on many factors that are not necessarily related to voting. Thus, without a voter registration

question (and relevant predictors of registration) in the background questionnaire, it would be virtually impossible to

account for nonrandom selection mechanisms within a statistical model of voting behavior. Although the Young

Adult Literacy Survey (Kirsch and Jungeblut 1986) included a question on voter registration, the previous NALS

background questionnaire did not include such an item.

The example of literacy and voting behavior is only one of a number of possible examples where selection

bias can occur. As another example, consider a comparison of those who obtain General Educational Development

(GED) credentials with high school graduates. In this case, while it may be reasonable to assume that high school

graduates are a relatively representative group, it is clear that those who enroll in the GED program and obtain the
5

GED are not. That is, of those who do not obtain their high school diplomas, a nonrandom sample of those

respondents will enter a course leading to the GED and still another nonrandom sub-sample of that group will earn

GED credentials. Thus, a model that predicts the probability of obtaining a GED is needed to account for nonrandom

assignment to these groups.

2.1 Methods for Modeling Selection

For completeness, it is useful to consider the variety of statistical approaches available to secondary users

who wish to account for selection bias. Perhaps the most popular approach to the problem of nonrandom selection

into treatment groups is the analysis-of-covariance (ANCOVA). With ANCOVA, the investigator is required to

choose one or more concomitant variables (covariates) to be used in the analysis. In the GED example given above,

one possible covariate might be a measure of the family socioeconomic status of the respondent—the argument being

that only certain individuals with somewhat higher levels of family SES can afford to return to school to study for the

GED. Scores on, say, document literacy, are adjusted for their relationship to the covariate(s) and the analysis-ofvariance is conducted on the adjusted scores (Kirk 1995).

Although ANCOVA represents a classic approach to the problem of selection bias, it does have certain

limitations. The most important limitation concerns the ANCOVA assumption of homogeneity of regression.

Homogeneity of regression refers to the requirement that the relationship between the outcome and the covariate be

the same for all groups under study. This is difficult from a practical standpoint since, as the number of covariates

increases, it becomes increasingly unlikely that this assumption will be satisfied.

Another approach to selection modeling that has its roots in econometrics is the two-step approach of James

Heckman (1976). The conceptual idea behind two-step modeling is that a substantive equation of interest is

misspecified if it is missing a variable that accounts for the probability of observing the data. In the voting case, one

may be interested only in the relationship between document literacy and educational level for those who voted. A

regression model to study this question would be misspecified because the process of selection creates a disturbance

term with a mean that is no longer zero and which is correlated with the predictor.
6

To correct for the specification error, two regression equations are specified: a selection model equation and

a substantive model equation. In the selection model equation, a probit regression is conducted, regressing a

qualitative group membership variable (voted/did not vote) on a set of variables assumed to predict group

 membership. A new variable is then formed for each person. Referred to as the hazard rate (Heckman 1976)1

, this

variable represents the likelihood that an individual will be excluded from the sample. Next, the hazard rate is added

to the substantive equation of interest and a standard ordinary least squares regression analysis is conducted.

Although Heckman’s approach is a classic methodology for dealing with selection bias, it is known to suffer

from statistical problems arising from the incorporation of the nonlinear based hazard rate into a linear equation. An

approach that derives the conceptual benefit of the Heckman approach without the computational problems, and that

is similar in many respects to ANCOVA, is the propensity score approach (Rosenbaum 1995; Rosenbaum and Rubin

1983).

The propensity score was proposed by Rosenbaum and Rubin as a means for balancing treatment and

control groups with respect to covariates in nonrandomized experimental studies. The propensity score is based on

the conditional probability of assignment to a treatment group given a set of observed covariates.

In a typical application of this approach, each observation is associated with a propensity to be assigned to

the treatment group. The distribution of propensity scores is then usually divided into strata at the quintile points of

the distribution and analyses of treatment group differences are conducted within strata2

. Comparisons of treatment

group differences within and across strata provide evidence for whether or not the bias due to nonrandom selection

into treatment groups has been accounted for by the propensity score adjustment. If, for example, groups are found to

differ by a constant amount across quintiles, this can be taken as evidence that the groups differ beyond what can be

explained by the process that assigned individuals to those groups. If, on the other hand, groups do not differ across

quintiles, this can be taken as evidence that the selection mechanism is accounting for the group differences. Finally,

if the size of the differences between groups varies across strata, this can then be taken as evidence that selection

 

1

 The hazard rate is derived from the expression of the conditional expectation function under truncation (Goldberger

1981). This expectation can be written as E(y|x) = x’β - ωλ(z). In this expression z = (c - x’β)/ ω,where c is a

truncation point, x is a vector of predictors, β is a vector of regression coefficients, and ω is the standard deviation of

the disturbances. The hazard rate λ(z) is the ratio of the probability density of z to the cumulative distribution of z

(Berk and Ray 1982).

2

 For continuous distributions, strata sub-classification at the quintile points has been found to remove at least 90

percent of the bias due to nonrandom selection effects.
7

characteristics are interacting with group differences yielding differential effects on the outcome of interest. This is a

classic example of the selection by treatment interaction.

A review of the substantive literature suggests that the propensity score approach has been used in such

diverse fields as sociology, medicine, psychiatry, and economics. Recently, Hoffer (1994) used the propensity score

approach in an analysis of educational tracking and Kaplan (in press) recently extended the propensity score

approach to latent variable models.

2.2 Recommendations to NCES

Considering the ubiquity of selection bias in social and behavioral science research, it is essential that care

be taken to consider relevant auxiliary variables that would constitute the selection equation. The examples of

participation in adult education or citizenship activities are prototypical, but it would be relatively straightforward to

generate other more subtle examples. The ANCOVA, Heckman two-step, and propensity score approaches for

addressing selection bias have one thing in common—they require the measurement of concomitant variables. Thus,

as consideration is given to the development of background items for NAAL, it is recommended that, for each

question that categorizes respondents into groups, thought be given as to how respondents were allocated to those

groups. Furthermore, auxiliary measures that predict that allocation should be included.

3. Issues of Measurement

It is often the case that behaviors and attitudes of respondents are desired. For example, in the NALS,

questions were posed regarding individuals’ self-perceptions of literacy ability. It may be of interest to determine the

number of factors underlying self-perception of literacy ability. Such underlying factors, if they exist, will be fewer

than the original set of variables used to measure those factors, and could be used in more parsimonious models

linking self-perception to actual literacy skills as measured by the literacy assessments. The development and

assessment of underlying constructs representing behaviors and attitudes constitutes an important part of secondary

statistical analysis.
8

If the goal is to validate a set of underlying behaviors and attitudes, then it is essential that multiple

measures, or observed variables, of those attitudes or opinions be obtained. Generally, the methodology used to

validate the existence of underlying dimensions is factor analysis. From the point of view of utilizing factor analysis,

a number of statistical issues need to be raised to help inform thinking about the development of the NAAL

background questionnaire.

First, it is necessary to consider the issue of the identification of an underlying factor. The concern here is

the extent to which a hypothesized factor could, in principle, be rejected on the basis of the observed data. Factor

analysis requires that, for any given factor, at least three measured variables load on that factor. With respect to

degrees-of-freedom, such a “three-variables-one-factor” model is just-identified—meaning that the factor loadings

can be estimated, but the hypothesis of a single underlying factor cannot be formally tested. For the purpose of

rejecting the hypothesis of a single underlying factor, at least four measured variables are required. In multiple-factor

models, the situation is a bit less constrained, and it is possible to reject, say, a two-factor model as long as there are

at least two measured variables per factor and the factors are allowed to correlate. In general, the principle is that in

the development of a set of items that are hypothesized to measure an underlying set of factors, the more observed,

or measured variables per factor that can be developed, the better.

The second issue concerns the metric of the measures. Typical metrics include five-point or seven-point

Likert scales. The decision regarding the number of scale steps should be made on the basis of substantive

considerations and a knowledge of the sample of respondents3

. From a statistical point of view, a general principle is

that the more scale steps comprising a measure, the better. However, it is no longer entirely necessary that items be

measured on five-point or seven-point scales. Indeed, developments in factor analysis over the past 15 years allow

for factor structures to be tested on measures that are dichotomous, ordered categorical (e.g., Likert scales),

continuous, as well all combinations of these types (Muthén 1978, 1984).

The statistical requirement pertaining to the scale steps is that there be a hypothesized underlying response

propensity for each measure. For example, consider the case where one wishes to assess reading habits. The NAAL

questionnaire could ask respondents whether they read newspapers at least once a week, with the response choice

being “yes” or “no”. To incorporate this and other similar items into factor analysis, it is assumed that underlying the
9

dichotomous response is a propensity to respond “yes.” The actual observation of a “yes” response occurs after a

threshold is exceeded.

When considering the factor analysis of a correlation matrix based on, say, dichotomous response variables,

it is not correct to simply analyze the Pearson product-moment correlation matrix of the data. Indeed, such an

analysis could result in the extraction of so-called “difficulty factors”

4

. Instead, it is necessary to calculate different

types of correlations that account for the scale type under the assumption of the underlying response propensity. For

example, the correlation between two dichotomous variables with an underlying normally distributed response

propensity is referred to as a tetrachoric correlation. Similarly, the correlation between two Likert-scale variables

assuming a continuous response propensity underlying each variable is referred as a polychoric correlation. Under

the assumption of the underlying response propensity, the analysis of these (and other similar) types of correlations

provide accurate assessment of the underlying factor structure of the data.

In the context of confirmatory factor analysis or structural equation modeling, more complicated estimation

procedures are necessary for the analysis of these types of correlation matrices. A discussion of these estimation

procedures is beyond the scope of this paper. It is sufficient to say that, without these specific estimation methods,

which require very large sample sizes, the results may be quite inaccurate.

3.1 Recommendations to NCES

From the standpoint of a literacy researcher using NAAL secondary data, the issue of scale type is less

important than the number of items needed to measure the factor. This is particularly true given the large sample size

proposed for NAAL. Therefore, when building scales to measure underlying attitudes or behaviors, consideration

 3

 For example, in developing opinion or attitude items for young children, dichotomously scored items might be

chosen because they would be easier for the child to understand.

4

 The problem of “difficulty factors” is a classic psychometric problem arising from the factor analysis of

dichotomous variables. The Pearson correlation between two such items is referred as the phi-coefficient. The

problem is that if the dichotomous variables exhibit unequal response frequencies, a factor analysis of such data will

yield a factor that is an artifact of the unequal responses.
10

should be given to the hypothesized number of factors and that at least four variables measuring each factor be

included in the questionnaire.

4. Issues Pertaining to Policy Modeling

In this section, consideration in given to the use of NAAL data for policy studies. It is clear from a perusal

of NCES 98–17 that a major purpose of NAAL is to inform federal and state policy on issues of adult literacy.

Generally, policy-relevant information is conveyed through the use of accurate descriptive statistics and crosstabulations as well as linkages to other relevant data for the purpose of measuring trends in literacy. A discussion of

linking the NAAL to other existing databases was discussed earlier in the context of respondent sampling.

Another approach to providing policy-relevant information is the use of advanced statistical models to

simulate changes in policy-relevant variables and observe the effects of those changes on literacy outcomes of

interest. I refer to such an exercise as policy simulation modeling, which can be defined as a method by which a

statistical model is used for prediction purposes that have policy relevance. The use of advanced statistical models

for policy studies has had a long tradition in economic modeling, but has been lacking in educational research.

Generally speaking, any statistical model can be used for the purpose of policy simulation modeling. However, the

conventional approach to statistical modeling in education has been to develop a model, estimate its parameters, test

the statistical significance of components of the model, and then interpret it. In the case of structural equation

modeling applied to educational data, it is often found that a process of model modification occurs in an effort to

bring models that do not fit in line with the data. Rarely, if ever, is the final form of these models used to simulate

alternative policy or clinical scenarios. Likewise, rarely are models judged on the basis of whether the results gleaned

from alternative policy scenarios make any substantive sense.

Recently, Kaplan and Elliott (1997) argued for such an approach and demonstrated policy simulation

modeling in the context of validating education indicators. In that analysis, Kaplan and Elliott determined a number

of important issues that are relevant to the development of the NAAL questionnaire. These issues concern obtaining

predictor, or exogenous, policy-relevant variables and the metric of policy-relevant variables, both of which will be

discussed, in turn, below.
11

4.1 Obtaining “Exogenous” Policy-Relevant Variables

In consideration of the use of statistical models for policy simulation modeling, it is helpful to distinguish

between the entire set of all possible predictor, or exogenous, variables and a subset of those predictor variables that

may be viewed as policy-relevant. To a certain extent, this discussion is tempered by the fact that NAAL will be a

cross sectional, as opposed to a longitudinal, study. In any case, it is clear that within the set of exogenous variables,

certain variables (e.g., age) cannot, in principle, be manipulated by the investigator, while others (e.g., number of

hours spent in literacy related activities), in principle, can.

A subtle feature of the problem is the extent to which variables that are considered policy-relevant are truly

predictive. To a certain extent, the question of the exogeneity of a variable is an empirical one (Richard 1982). Not

all variables that are termed exogenous truly are, and there are advanced exogeneity tests that allow this assumption

to be assessed. However, in policy simulation modeling it is important that thought be given to the selection of

variables that have policy relevance. This selection should only be guided by theory and the interaction of literacy

policy specialists with survey research specialists and psychometricians.

4.2 Natural Metrics of Observed Variables

Another concern that emerges from a consideration of policy simulation modeling is the metric of the

policy-relevant variables. If secondary statistical modeling is to be used for developing policy-relevant prediction

models, then it is essential that observed variables be measured in their natural metrics. To be specific, many national

surveys code sensitive variables into categories that render them almost useless for policy simulation modeling. For

example, an important variable in an analysis of literacy outcomes is income. Understandably, income is a sensitive

question. However, in collapsing income into large and arbitrary categories we render this variable useless for policy

modeling.

It is understood that such advice may conflict with confidentiality issues. Nevertheless, it is important that

this issue be discussed in designing the background questionnaire and that the statistical ramifications be understood.
12

Two possible approaches for addressing this issue would be to 1) develop more refined codes and/or 2) create

restricted use datasets.

4.3 Recommendations to NCES

The issue discussed in this section concerns using secondary public-use data for basic policy research. If

such an endeavor is deemed valuable, then it is essential that policy analysts and researchers supplement descriptive

statistics with theoretically guided models that have policy relevance. The advanced statistical models are ripe for

this use but require serious consideration of the types of variables and their metrics to be included in the background

questionnaire. Thus, it is advised that, in the development of the background questionnaire, policy analysts in the

field of adult literacy should meet with survey and measurement specialists to consider the development of policyrelevant variables for use in policy simulation modeling.

5. Issues Relating to Gauging Cohort Effects

Finally, it is important to develop measures that assess the extent of cohort effects. As justification for this,

consider that a simple analysis of NALS data shows that there is a precipitous decline in literacy skills after the age

of forty-five. This could very well be due to a differential growth rate in literacy for different age cohorts. Yet, with

cross-sectional data, this differential growth rate hypothesis is hard to test. The section to follow will consider two

approaches to aid in assessing cohort effects: obtaining retrospective background questions and linking NAAL with

existing longitudinal surveys.

5.1 Obtaining Retrospective Background Questions

Considering that NAAL will be based on a cross-sectional design, the first approach which may be the most

feasible would be to include retrospective questions in the background questionnaire. Such questions should be

directed at ascertaining literacy related activities as well as job histories at key points in the lifecycle of respondents.
13

For example, it may be of interest to probe the job histories of respondents and to ascertain the literacy demands

required for each job.

Retrospective questions have a number of serious problems that must be kept in mind (Bijleveld and van

der Kamp 1998). First, it should be noted that the very nature of the response to a retrospective question is

nonrandom. That is, what is being observed, or measured, are the responses of individuals who can reflect back on

their histories. Even if the design had been longitudinal, attrition for any number of reasons would render the

resulting sample of respondents a nonrandom sample.

Additional problems with retrospective designs relate to the unreliability of the responses. Such unreliability

stem from the following sources: (1) memory loss, in which respondents simply forget specific events in their life

histories. To mitigate against this problem, some have advocated the use of checklists to aid in recalling possible

events; (1) telescoping, wherein respondents tend to report events as having happened more recently than they

actually happened; and (3) modification to fit a coherent scheme, wherein respondents tend to interpret events to fit

current perceptions of themselves (Bijleveld and van der Kamp 1998).

These problems render retrospective reports of questionable validity. However, the problems are mitigated

by asking respondents to describe factual events to the extent possible. That is, assessments of job histories and

literacy activities are more reliable than retrospective assessments of feelings and attitudes. So, instead of asking

someone how he felt about reading when he was 10 years old, ask instead, “What was your first job?” “Did the job

require reading?” “What types of material were you required to read on your first job?”, etc. Although such detailed

probing questions will not eliminate these biases, the strategy is known to mitigate them.

5.2 Linking NAAL With Existing Longitudinal Surveys

In the absence of conducting a longitudinal survey, a second approach to gauging cohort effects would be to

consider linking a sub-sample of NAAL respondents to an ongoing longitudinal survey, such as the NCES National

Educational Longitudinal Survey (NELS), the NCES High School and Beyond (HSB), or the U. S. Department of

Labor’s National Longitudinal Survey (NLS). To be certain, the feasibility of such a proposal is unclear. In addition,

the literacy assessment contained in NAAL is not available in other surveys. Yet, with the development of
14

longitudinal statistical methodologies such as growth curve modeling (Muthen and Curran 1997) it is possible to

examine growth in one domain (e.g., reading achievement) as it pertains to the prediction of a distant outcome such

as document literacy. Moreover, linkages to other surveys would not require excessively large sample sizes. The

benefits of seriously considering such linkages is that surveys such as NELS or NLS contain information regarding

reading practices and other schooling measures. In addition, surveys such as NELS contain background parent

questionnaire data that might be useful in examining predictors of adult literacy.

5.3 Recommendations to NCES

To gauge the extent of cohort effects, it is suggested that the NAAL background questionnaire obtain

retrospective data on general and job-specific literacy related activities. It is advisable to construct retrospective

questions that will prompt factual information to the fullest extent possible. Moreover, it is advisable to include

possible checklists to aid in the recall of historical events. In addition, NCES should explore the possibility of linking

a sub-sample of the NAAL respondents to existing surveys sponsored by NCES or the U.S. Department of Labor

(e.g., NELS, HSB, NLS). Finally, NCES should explore the possibility of obtaining a sub-sample of each age cohort

of the NALS survey and include these in the NAAL survey.
15

References

Berk, R. A., and Ray, S. C. (1982). Selection Biases in Sociological Data. Social Science Research, 11: 352–398.

Bijleveld, C. C. J. H., and van der Kamp, L. J. T. (1998). Longitudinal Data Analysis: Designs, Models, and

Methods. Newbury Park, CA: Sage Publications.

Bryk, A. S., and Raudenbush, S. W. (1992). Hierarchical Linear models: Applications and Data Analysis Methods.

Newbury Park, CA.: Sage Publications.

Goldberger, A. S. (1981). Linear Regression After Selection. Journal of Econometrics, 15: 357–366.

Heckman, J. J. (1976). The Common Structure of Statistical Models of Truncation, Sample Selection, and Limited

Dependent Variables and a Simple Estimator for Such Models. The Annals of Economic and Social

Measurement, 5: 475–492.

Hoffer, T. B. (1994). Cumulative Effects of Secondary School Tracking on Student Achievement. Paper presented at

the annual meeting of the American Sociological Association, Los Angeles, CA.

Kaplan, D. (in press). On the Extension of the Propensity Score Adjustment Method for the Analysis of Group

Differences in MIMIC Models. Multivariate Behavioral Research.

Kaplan, D., and Elliott, P. R. (1997). A Model-based Approach to Validating Education Indicators Using Multilevel

Structural Equation Modeling. Journal of Educational and Behavioral Statistics, 22: 323–348.

Kaplan, D., and Ferguson, A. J. (1999). On the Utilization of Sample Weights in Latent Variable Models.

Structural Equation Modeling, 6: 305–321.

Kaplan, D., and Venezky, R. L. (1994). Literacy and Voting Behavior: A Bivariate Probit Model with Sample

Selection. Social Science Research, 23: 350–367.

Kirk, R. E. (1995). Experimental Design: Procedures for the Behavioral Sciences. Pacific Grove, CA: Brooks/Cole

Publishing.

Kirsch, I. S, and Jungeblut, A. (1986). Literacy: Profiles of America’s Young Adults (NAEP Report No. 16–PL–02).

Princeton, NJ: Educational Testing Service.

Muthen, B. (1978). Contributions to Factor Analysis of Dichotomous Variables. Psychometrika, 43: 551–560.
16

Muthen, B. (1984). A General Structural Equation Model with Dichotomous, Ordinal, Categorical, and Continuous

Latent Variable Indicators. Psychometrika, 49: 115–132.

Muthen, B. O., and Curran, P. J. (1997). General Longitudinal Modeling of Individual Differences in Experimental

Designs: A Latent Variable Framework for Analysis and Power Estimation. Psychological Methods, 2:

71–402.

Sherman, R., Condelli, L., and Koloski, J. (1998). Developing the National Assessment of Adult Literacy:

Recommendations from Stakeholders (NCES 98–17). U.S. Department of Education. Washington, DC:

NCES Working Paper.

Richard, J. F. (1982). Exogeneity, Causality, and Structural Invariance in Econometric Modeling. In G. C. Chow

P. Corsi (Eds.), Evaluating the Reliability of Macro-economic Models. New York, NY: John Wiley & Sons.

Rosenbaum, P. R. (1995). Observational Studies. New York, NY: Springer-Verlag.

Rosenbaum, P. R., and Rubin, D. B. (1983). The Central Role of the Propensity Score in Observational Studies for

Causal Effects. Biometrika 70: 41–55.

Sheehan-Holt, J. K., and Smith, M. C. (1999, April). Using Multilevel Modeling to Investigate Predictors of Literacy

in the National Adult Literacy Survey. Paper presented at the annual meeting of the American Educational

Research Association. Montreal, Canada.

Venezky, R. L., and Kaplan, D. (1998). Literacy Habits and Political Participation. In M. C. Smith (Ed.), Literacy

for the 21st Century . Westport, CN: Greenwood Publishing Group.
Listing of NCES Working Papers to Date

Working papers can be downloaded as pdf files from the NCES Electronic Catalog

(http://nces.ed.gov/pubsearch/). You can also contact Sheilah Jupiter at (202) 502-7444

(sheilah_jupiter@ed.gov) if you are interested in any of the following papers.

Listing of NCES Working Papers by Program Area

No. Title NCES contact

Baccalaureate and Beyond (B&B)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Beginning Postsecondary Students (BPS) Longitudinal Study

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Common Core of Data (CCD)

95-12 Rural Education Data User’s Guide Samuel Peng

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Decennial Census School District Project

95-12 Rural Education Data User’s Guide Samuel Peng

96-04 Census Mapping Project/School District Data Book Tai Phan

98-07 Decennial Census School District Project Planning Report Tai Phan

Early Childhood Longitudinal Study (ECLS)

96-08 How Accurate are Teacher Judgments of Students’ Academic Performance? Jerry West

96-18 Assessment of Social Competence, Adaptive Behaviors, and Approaches to Learning with

Young Children

Jerry West

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Education Finance Statistics Center (EDFIN)

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

High School and Beyond (HS&B)

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

HS Transcript Studies

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson
No. Title NCES contact

International Adult Literacy Survey (IALS)

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Integrated Postsecondary Education Data System (IPEDS)

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

National Assessment of Adult Literacy (NAAL)

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

National Assessment of Educational Progress (NAEP)

95-12 Rural Education Data User’s Guide Samuel Peng

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Steven Gorman

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Steven Gorman

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Steven Gorman

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questionnaires)

Steven Gorman

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Steven Gorman

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

National Education Longitudinal Study of 1988 (NELS:88)

95-04 National Education Longitudinal Study of 1988: Second Follow-up Questionnaire Content

Areas and Research Issues

Jeffrey Owings

95-05 National Education Longitudinal Study of 1988: Conducting Trend Analyses of NLS-72,

HS&B, and NELS:88 Seniors

Jeffrey Owings

95-06 National Education Longitudinal Study of 1988: Conducting Cross-Cohort Comparisons

Using HS&B, NAEP, and NELS:88 Academic Transcript Data

Jeffrey Owings

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson
No. Title NCES contact

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

National Household Education Survey (NHES)

95-12 Rural Education Data User’s Guide Samuel Peng

96-13 Estimation of Response Bias in the NHES:95 Adult Education Survey Steven Kaufman

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-21 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School

Readiness, and School Safety and Discipline

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

96-29 Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the

1995 National Household Education Survey (NHES:95)

Kathryn Chandler

96-30 Comparison of Estimates from the 1995 National Household Education Survey

(NHES:95)

Kathryn Chandler

97-02 Telephone Coverage Bias and Recorded Interviews in the 1993 National Household

Education Survey (NHES:93)

Kathryn Chandler

97-03 1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener,

NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education

Kathryn Chandler

97-04 Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in

the 1993 National Household Education Survey (NHES:93)

Kathryn Chandler

97-05 Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National

Household Education Survey (NHES:93)

Kathryn Chandler

97-06 Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National

Household Education Survey (NHES:95)

Kathryn Chandler

97-08 Design, Data Collection, Interview Timing, and Data Editing in the 1995 National

Household Education Survey

Kathryn Chandler

97-19 National Household Education Survey of 1995: Adult Education Course Coding Manual Peter Stowe

97-20 National Household Education Survey of 1995: Adult Education Course Code Merge

Files User’s Guide

Peter Stowe

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

97-28 Comparison of Estimates in the 1996 National Household Education Survey Kathryn Chandler

97-34 Comparison of Estimates from the 1993 National Household Education Survey Kathryn Chandler

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

97-38 Reinterview Results for the Parent and Youth Components of the 1996 National

Household Education Survey

Kathryn Chandler

97-39 Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996

National Household Education Survey

Kathryn Chandler

97-40 Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996

National Household Education Survey

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

National Longitudinal Study of the High School Class of 1972 (NLS-72)

95-12 Rural Education Data User’s Guide Samuel Peng

National Postsecondary Student Aid Study (NPSAS)

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio
No. Title NCES contact

National Study of Postsecondary Faculty (NSOPF)

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Private School Universe Survey (PSS)

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

96-26 Improving the Coverage of Private Elementary-Secondary Schools Steven Kaufman

96-27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 Steven Kaufman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Recent College Graduates (RCG)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Schools and Staffing Survey (SASS)

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American

Statistical Association

Dan Kasprzyk

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) Dan Kasprzyk

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report Dan Kasprzyk

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher

Transcript Study, Schools and Staffing Survey

Dan Kasprzyk

94-06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related

Surveys

Dan Kasprzyk

95-01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American

Statistical Association

Dan Kasprzyk

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing

QED School Estimates with CCD Estimates

Dan Kasprzyk

95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis Dan Kasprzyk

95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates Dan Kasprzyk

95-09 The Results of the 1993 Teacher List Validation Study (TLVS) Dan Kasprzyk

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive

Reconciliation

Dan Kasprzyk

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and

Their Applicability for the Teacher Follow-up Survey

Sharon Bobbitt

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and

Staffing Survey

Dan Kasprzyk

96-01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly

Longitudinal Study

Dan Kasprzyk

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting

of the American Statistical Association

Dan Kasprzyk

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey Dan Kasprzyk

96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to

Inform Broad Education Policy

Dan Kasprzyk

96-07 Should SASS Measure Instructional Processes and Teacher Effectiveness? Dan Kasprzyk

96-09 Making Data Relevant for Policy Discussions: Redesigning the School Administrator

Questionnaire for the 1998-99 SASS

Dan Kasprzyk

96-10 1998-99 Schools and Staffing Survey: Issues Related to Survey Depth Dan Kasprzyk
No. Title NCES contact

96-11 Towards an Organizational Database on America’s Schools: A Proposal for the Future of

SASS, with comments on School Reform, Governance, and Finance

Dan Kasprzyk

96-12 Predictors of Retention, Transfer, and Attrition of Special and General Education

Teachers: Data from the 1989 Teacher Followup Survey

Dan Kasprzyk

96-15 Nested Structures: District-Level Data in the Schools and Staffing Survey Dan Kasprzyk

96-23 Linking Student Data to SASS: Why, When, How Dan Kasprzyk

96-24 National Assessments of Teacher Quality Dan Kasprzyk

96-25 Measures of Inservice Professional Development: Suggested Items for the 1998-1999

Schools and Staffing Survey

Dan Kasprzyk

96-28 Student Learning, Teaching Quality, and Professional Development: Theoretical

Linkages, Current Measurement, and Recommendations for Future Data Collection

Mary Rollefson

97-01 Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the

American Statistical Association

Dan Kasprzyk

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

97-10 Report of Cognitive Research on the Public and Private School Teacher Questionnaires

for the Schools and Staffing Survey 1993-94 School Year

Dan Kasprzyk

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-12 Measuring School Reform: Recommendations for Future SASS Data Collection Mary Rollefson

97-14 Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and

Analysis

Steven Kaufman

97-18 Improving the Mail Return Rates of SASS Surveys: A Review of the Literature Steven Kaufman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

97-23 Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing

Form

Dan Kasprzyk

97-41 Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting

of the American Statistical Association

Steve Kaufman

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

98-05 SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors

Steven Kaufman

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

98-12 A Bootstrap Variance Estimator for Systematic PPS Sampling Steven Kaufman

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

98-14 Variance Estimation of Imputed Survey Data Steven Kaufman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest

Results to Improve Item Construction

Dan Kasprzyk

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

1999-12 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use

Codebook

Kerry Gruber

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk
Listing of NCES Working Papers by Subject

No. Title NCES contact

Adult education

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Adult literacy—see Literacy of adults

American Indian – education

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

Assessment/achievement

95-12 Rural Education Data User’s Guide Samuel Peng

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Larry Ogle

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Larry Ogle

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Larry Ogle

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questions)

Larry Ogle

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Larry Ogle

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Beginning students in postsecondary education

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Civic participation

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Climate of schools

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

Cost of education indices

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Course-taking

95-12 Rural Education Data User’s Guide Samuel Peng
No. Title NCES contact

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

Crime

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Curriculum

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Customer service

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Data quality

97-13 Improving Data Quality in NCES: Database-to-Report Process Susan Ahmed

Data warehouse

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Design effects

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Dropout rates, high school

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

Early childhood education

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Educational attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Educational research

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

Employment

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico
No. Title NCES contact

Faculty – higher education

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Finance – elementary and secondary schools

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Finance – postsecondary

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

Finance – private schools

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

Geography

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

Imputation

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Inflation

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

Institution data

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Instructional resources and practices

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

International comparisons

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-16 International Education Expenditure Comparability Study: Final Report, Volume I Shelley Burns

97-17 International Education Expenditure Comparability Study: Final Report, Volume II,

Quantitative Analysis of Expenditure Comparability

Shelley Burns

Libraries

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers

Presented at Meetings of the American Statistical Association

Carrol Kindel

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Limited English Proficiency

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser
No. Title NCES contact

Literacy of adults

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

Literacy of adults – international

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Mathematics

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

Parental involvement in education

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Participation rates

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

Postsecondary education

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Postsecondary education – persistence and attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Postsecondary education – staff

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Private schools

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman
No. Title NCES contact

Projections of education statistics

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Public school finance

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Public schools

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

Public schools – secondary

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Reform, educational

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

Response rates

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

School districts, public

98-07 Decennial Census School District Project Planning Report Tai Phan

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

School districts, public – demographics of

96-04 Census Mapping Project/School District Data Book Tai Phan

Schools

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Schools – safety and discipline

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Software evaluation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Staff

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Staff – higher education institutions

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

State

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young
No. Title NCES contact

Statistical methodology

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But

Thought You Could Never Understand

Susan Ahmed

Students with disabilities

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

Survey methodology

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Teachers

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

Teachers – instructional practices of

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – opinions regarding safety

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – performance evaluations

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – qualifications of

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – salaries of

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Variance estimation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Violence

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Vocational education

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson