NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

The Working Paper Series was initiated to promote the sharing of the

valuable work experience and knowledge reflected in these preliminary

reports. These reports are viewed as works in progress, and have not

undergone a rigorous review for consistency with NCES Statistical

Standards prior to inclusion in the Working Paper Series.

U. S. Department of Education

Office of Educational Research and Improvement
NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

Evaluation of the 1992 NALS Background

Survey Questionnaire: An Analysis of Uses

with Recommendations for Revisions

Working Paper No. 2000-08 March 2000

Contact: Sheida White

Assessment Division

E-mail: sheida_white@ed.gov

U. S. Department of Education

Office of Educational Research and Improvement
U.S. Department of Education

Richard W. Riley

Secretary

Office of Educational Research and Improvement

C. Kent McGuire

Assistant Secretary

National Center for Education Statistics

Gary W. Phillips

Acting Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing,

and reporting data related to education in the United States and other nations. It fulfills a congressional

mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in

the United States; conduct and publish reports and specialized analyses of the meaning and significance of

such statistics; assist state and local education agencies in improving their statistical systems; and review

and report on education activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent, reliable,

complete, and accurate indicators of education status and trends; and report timely, useful, and high quality

data to the U.S. Department of Education, the Congress, the states, other education policymakers,

practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is appropriate to a

variety of audiences. You, as our customer, are the best judge of our success in communicating information

effectively. If you have any comments or suggestions about this or any other NCES product or report, we

would like to hear from you. Please direct your comments to:

National Center for Education Statistics

Office of Educational Research and Improvement

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

The NCES World Wide Web Home Page is

http://nces.ed.gov

Suggested Citation

U.S. Department of Education. National Center for Education Statistics. Evaluation of the 1992 NALS

Background Survey Questionnaire: An Analysis of Uses with Recommendations for Revisions, Working

Paper No. 2000-08, by M Cecil Smith and Janet K. Sheehan-Holt. Project Officer, Sheida White.

Washington, DC: 2000.

March 2000
iii

Foreword

In addition to official NCES publications, NCES staff and individuals commissioned by

NCES produce preliminary research reports that include analyses of survey results, and

presentations of technical, methodological, and statistical evaluation issues.

The Working Paper Series was initiated to promote the sharing of the valuable work

experience and knowledge reflected in these preliminary reports. These reports are viewed as

works in progress, and have not undergone a rigorous review for consistency with NCES

Statistical Standards prior to inclusion in the Working Paper Series.

Copies of Working Papers can be downloaded as pdf files from the NCES Electronic

Catalog (http://nces.ed.gov/pubsearch/), or contact Sheilah Jupiter at (202) 502-7444,

e-mail: sheilah_jupiter@ed.gov, or mail: U.S. Department of Education, Office of Educational

Research and Improvement, National Center for Education Statistics, 1990 K Street, NW, Room

9048, Washington, DC 20006.

Marilyn M. McMillen Ralph Lee

Chief Mathematical Statistician Mathematical Statistician

Statistical Standards Program Statistical Standards Program
This page intentionally left blank.
Evaluation of the 1992 NALS Background Survey Questionnaire:

An Analysis of Uses with Recommendations for Revisions

Prepared by:

M Cecil Smith

Janet K. Sheehan-Holt

Northern Illinois University

Prepared for:

U.S. Department of Education

Office of Educational Research and Improvement

National Center for Education Statistics

March 2000

This project was an activity of the Education Statistics Services Institute.
This page intentionally left blank.
vii

Table of Contents

Foreword........................................................................................................................................................iii

Abstract........................................................................................................................................................viii

Identification of NALS Data Users ............................................................................................................ 3

Survey............................................................................................................................................................. 5

Published Research Reports...................................................................................................................... 5

Unpublished NALS Reports ....................................................................................................................... 6

Researchers’ Uses of the EBQ Data .......................................................................................................... 6

Survey Results ................................................................................................................................................ 7

Section A: Language Background ............................................................................................................. 7

Section B: Educational Attainment and Experiences................................................................................. 8

Section C: Political and Social Participation........................................................................................... 10

Section D: Labor Force Participation ..................................................................................................... 11

Section E: Literacy Activities and Collaboration with Others................................................................. 12

Section F: Demographic Information...................................................................................................... 14

General Comments................................................................................................................................... 14

Summary .................................................................................................................................................. 15

Content Review............................................................................................................................................. 16

Criteria..................................................................................................................................................... 16

Review of Studies..................................................................................................................................... 16

Smith and Sheehan-Holt Studies.............................................................................................................. 29

Summary of Content Review .................................................................................................................... 32

Conclusions................................................................................................................................................... 33

Technical and Substantive Implications .................................................................................................. 33

Recommended Changes to the EBQ......................................................................................................... 34

Alternative Approaches............................................................................................................................ 39

Assessment Costs...................................................................................................................................... 42

References..................................................................................................................................................... 44

Attachments .................................................................................................................................................. 48

A. Initial Contact E-mail Survey .............................................................................................................. 49

B. NALS Background Survey Data Users Questionnaire ......................................................................... 51

C. Studies Examined for Content Review................................................................................................. 53
viii

Abstract

This report describes how social science researchers and other experts have analyzed data from the

National Adult Literacy Survey (NALS) since the public release of the data in 1993 to serve a variety of

educational, social policy and scientific purposes. The particular focus of our report pertains to researchers’

uses of the English Background Questionnaire (EBQ) portion of the NALS. Our data were gathered by

surveying NALS data users about how they have used the EBQ data, their perspectives on the strengths and

weaknesses of the EBQ, and their recommendations for changes to the EBQ. We also conducted a content

analysis of published research papers that have involved analyses of the EBQ data to determine how the

data were used in these research analyses. Finally, we drew upon the recommendations garnered by a series

of focus groups conducted by the American Institutes for Research in 1998. After integrating these sources

with our own analyses of the NALS data, we recommend several changes to the EBQ for future adult

literacy assessments. These are as follows: (1) expand the questions pertaining to educational experiences

including adult basic education and other nonformal types of educational experiences; (2) expand the

section on social and political participation or move the current items to other areas and delete this section;

(3) add an item on voter registration; (4) acquire more detailed information regarding work history, wages

and parents’ occupations; (5) expand the section pertaining to literacy practices; and (6) gather information

on technological literacy practices. We also recommend the following actions: (1) re-examine the issue of

the unidimensionality of the literacy scales; (2) utilize Reder’s Lifelong Learning Questionnaire for new

items; (3) link the NAAL to the Adult Education Survey of the National Household Education Survey; and

(4) collect data at multiple levels of the survey design.
1

The data for the National Adult Literacy Survey (NALS) were gathered over the first eight months

of 1992. The first report summarizing the results of this largest-ever assessment of adult literacy skills was

issued in September, 1993. The data were subsequently released in the public domain in 1994 by the

National Center for Education Statistics (NCES). The data were therefore available for educational

policymakers and planners; researchers in universities and research organizations; and experts in various

fields such as assessment, labor economics, adult education, sociology, and political science to utilize for

the purposes of secondary analyses. By making the data available to experts for analysis, additional

findings, insights, and implications for policy, program development, and practice can be generated from

the NALS and disseminated far beyond the capabilities of a single agency such as NCES.

Large-scale social science data, of the type obtained in the NALS, are useful for addressing a

number of issues of interest to educators and social science researchers.

For example:

1. The scope and extent of adult literacy;

2. The ways in which literacy contributes to individuals’ well-being, social involvement, and

workplace performance;

3. The associations of various demographic variables such as age, gender, and educational

attainment to literacy proficiencies; and,

4. The nature of observed differences in literacy skills and practices among the different racial,

ethnic, age, and geographic groups throughout the United States.

The purpose of this report is to describe how social science researchers, educators, and policy

experts have used the NALS data since the public release of the data in 1993 to serve a variety of

educational, social policy and scientific purposes. The particular focus of this report pertains to the manner

in which secondary analyses of the data have been conducted, particularly as these analyses have employed

data from the NALS English-language Background Questionnaire (EBQ). Several issues are of interest to

us, and we have attempted to address these in our investigation. We gathered information in two ways: (1)

via a survey of identified NALS data users; and (2) through a content analysis of the published research

papers pertaining to the NALS in which EBQ data were used.
2

We began by conducting a limited survey of several key personnel with expertise in either all or

specific sections of the EBQ. An e-mail survey was sent to identified NALS data users to ascertain their

perspectives on the adequacy of the data provided by the EBQ. The goal was to ascertain individuals’

understandings and uses of these data and their recommendations—for gathering background data—based

on their experiences using the data derived from the NALS background questionnaire.

The key issues resulting from our survey responses are reported here and integrated with

information obtained from our content review of a small number of published studies (i.e., secondary

analyses) of the NALS. The focus of our content review is to identify both the specific background

variables used, and how these variables and analyses appeared to have either contributed to or limited the

researchers’ inferences and/or policy recommendations. Our assumption is that such information is

important to the National Assessment of Adult Literacy (NAAL) survey designers and will ensure that

critical questions are considered, which might then enhance researchers’ and policymakers’ abilities to draw

meaningful conclusions and/or make policy recommendations.

We also weigh the relative advantages and disadvantages of adding items and modifying existing

items on the EBQ. Any changes must, of course, be considered in light of the potential costs—in terms of

real dollars and the sacrifice of useful data. Changing the EBQ may, for example, impede comparisons and

trend analyses for the 1992 and 2002 adult literacy assessments. While it may be possible to create a

parsimonious set of background questionnaire items that provide essential information and allow strong

inferences about adult literacy, this must be accomplished within the parameters of minimal administrative

and data management costs and advantages to data reporting.

The 1992 NALS English-language Background Questionnaire contained six “content areas”

enabling the collection of information on survey participants’: (1) general and language background; (2)

educational attainment and experiences; (3) political and social participation; (4) labor force participation;

(5) literacy activities and collaboration with others, and, (6) demographic information. These data were

deemed by the NALS developers to be important for the purpose of “[gaining] an understanding of the ways

in which personal characteristics are associated with demonstrated performance” on the three NALS literacy

proficiency scales—prose, document, and quantitative literacy (Kirsch, Jungeblut, Jenkins, and Kolstad,

1993, 5). Adult literacy researchers and policy analysts vary in their areas of expertise in relation to these
3

components of the EBQ, and they utilize the data in different ways in order to address questions of interest

to themselves and/or to their organizations, institutions, or sponsors. Thus, we attempted to obtain a diverse

sample of NALS data users and analysts.

Identification of NALS Data Users

NCES has a NALS Web site that contains a link to a list of NALS “data users”

(http://nces.ed.gov/naal/naal92/users.html). This list, dated May 16, 1996, consists of 70 names of

individuals (including the first author of this report) who are academic researchers, policymakers and

analysts for public and private research firms and agencies, journalists, and other experts. The listed

individuals are those who had requested a copy of the data files on diskette from NCES. Because

individuals were able to download the data directly from the Web site as of mid-1996, it was not possible to

maintain a more up-to-date file of data users (A. Kolstad—personal communication, May, 1998). We

attempted to contact as many individuals on this list as possible to gather information regarding their uses

and analyses of the NALS data. Our primary method for contacting individuals was through e-mail

(addresses were available for 58 persons on the list, 83 percent). Several of these addresses were invalid,

however, so we then attempted to obtain up-to-date e-mail addresses and phone numbers.

An additional, updated, list of 11 names of NALS data users was also provided to us through

Andrew Kolstad at NCES. We received replies from two persons among this subgroup, one of whom

reported not having used the data for statistical analyses due to its complexity. A few of the individuals that

we surveyed also referred us to other researchers with whom they had collaborated on their NALS studies.

We were then able to complete one additional telephone interview. We attempted to contact, via e-mail, the

state-level directors of adult education in each of the states that had conducted NALS state-level

assessments. Although we did not expect that this latter group would have conducted analyses of the NALS

data, they could presumably refer us to statisticians and policy analysts on their staffs who had done so. Our

total population of identified “NALS users” equaled 94 persons.

Because the authors of this report have conducted analyses of the NALS data, resulting in one

published research paper, another research report that is now in press, and five paper presentations at

national conferences, we are familiar with several other researchers who have analyzed the NALS data. The
4

first author of this report has also edited a book on the NALS that contains seven chapters by individual

investigators that describe secondary analyses of the NALS data. Also, the first author has reviewed other

research papers based on the NALS. Therefore, the authors of this report are reasonably well informed

about the body of scholarship based on the NALS that has been produced over the past five years. However,

because it would be presumptuous to assume that we are aware of all research pertaining to the NALS, we

conducted an electronic database search of educational and social science literature to determine and locate

all published research papers based on the NALS. The electronic search revealed two additional empirical

studies employing the NALS data—one of which used the EBQ variables.

Finally, we sent a request to the National Literacy Advocacy (NLA) electronic listserv in order to

notify NALS data users whom we may have overlooked. The purpose of the NLA listserv is to provide

information about issues affecting federal legislation, policy, and funding for literacy, basic and secondary

education and English for Speakers of Other Languages. Thus, it is a particularly useful resource for

policymakers and program providers in the adult literacy field. Our message to the NLA listserv read as

follows:

I am preparing, along with my colleague Janet Sheehan-Holt, a report pertaining to the

various uses of the 1992 National Adult Literacy Survey by adult literacy experts—particularly in

regards to the Background Survey (i.e., language background of respondents, educational

background and experiences, labor force and social participation, literacy activities, demographic

information). The purpose of the report is to recommend changes in the Background Survey for the

planned National Assessment of Adult Literacy (NAAL) in 2002. We are particularly interested in

any secondary descriptive or inferential analyses of the NALS as conducted by policy experts at

the federal and state levels, academic researchers, and others.

Although we have identified a number of individuals who have conducted, and

subsequently published such analyses, we want to make sure we haven’t overlooked anyone. If you

have used the NALS Background Survey data for secondary analysis, we would like to include you

in our sample. Please contact us by email (mcsmith@niu.edu) or telephone (815) 753-8448 for

more complete information. We have a very brief survey that can be completed via email.

Unfortunately, we received only one response to this message. We re-posted our request to the NLA listserv

on May 1st, 1999, to further canvass NALS data users, but received no responses. We believe that this lack

of response is largely due to the fact that many NLA subscribers are directors of adult literacy programs

rather than researchers. As such, these individuals are most likely to refer to the NALS results—rather than

to conduct analyses of the data—to educate their respective communities about the need for adult literacy
5

education, and to justify their funding requests in order to raise funds, increase services, and improve

service delivery.

Survey

Our initial e-mail query (see Attachment A) identified the purposes of our project, and asked

individuals to respond briefly to three questions, including their willingness to complete a second brief

survey via e-mail or by telephone. Through this method, we received replies from about one-third of the

identified NALS users (n=30). Sixteen of these persons indicated having used the NALS data for secondary

analyses (in addition, two persons used state-level data and a third used NALS data to replace missing data

for the International Adult Literacy Survey; the two authors of this report have, of course, also used the

NALS for secondary analyses) and nine had not conducted any data analyses. All of those who responded to

our initial survey, and who have analyzed the NALS data, agreed to respond to the second part of our

survey.

The purpose of the second part was to determine (1) the specific sections of the EBQ of greatest

use to these researchers, (2) the specific questions of interest to their research, and (3) their perspectives on

the relative strengths and weaknesses of the EBQ for making strong inferences from the data to populations

and subgroups of interest to them. Upon affirmation that a respondent would complete the second part of

the survey (see Attachment B), it was sent via e-mail to those individuals—with two exceptions. These

persons were either interviewed over the phone or face-to-face.

Published Research Reports

We identified more than a dozen research papers that have been published in the scientific

literature (i.e., journals), or are currently in press, or are contained in the aforementioned edited book

(Smith 1998). All of these reports are based upon secondary analyses of the NALS data (see Attachment C).

Three of these papers involve primarily descriptive analyses examining particular populations (e.g.,

community college students). The remaining reports are largely based on inferential statistical analyses.

Two book chapters (see Finn and Gerber 1998; Howard and Obetz 1998) are essentially summaries of

studies published as journal articles (Gerber and Finn 1998; Howard and Obetz 1996). In these cases, we

examined the journal articles. In another case, a chapter summarized a previously published study (see
6

Smith 1996), but also reported the results of a second analysis that had not been previously reported (see

Smith and Sheehan 1998). Also, Pryor and Schaffer have reported utilizing or referencing the NALS data

throughout their book (Pryor and Schaffer 1999) to support their claims pertaining to associations between

education and literacy, and employment and earnings. The NALS data were not the primary data source for

their book, however. We examined a research report by Pryor and Schaffer (1997) in which they conducted

secondary analyses of the NALS data.

Unpublished NALS Reports

Our database search also revealed that there are an additional fifty-five reports pertaining to the

NALS that are available through the Educational Resources Information Clearinghouse (ERIC) document

retrieval service. These ERIC documents include: the original executive summary of the NALS (Kirsch et

al. 1993); various state-level reports prepared by the Educational Testing Service; state, local and national

estimates of literacy; literacy performance of various populations (e.g., older adults, prisoners); essays on

the state of adult literacy; and, methodology and technical papers pertaining to adult literacy assessment.

These reports were not used in our analyses; however, as they involved descriptive reporting of data directly

from the NALS results and/or the NALS Executive Summary, rather than original analyses of the NALS

and, in particular, the EBQ data.

Researchers’ Uses of the EBQ Data

Because our analysis of uses of the EBQ data is largely a qualitative one, we will refer to

respondents to our survey as “informants” in order to avoid confusion between NALS sample respondents

and those persons who completed our survey. Slightly more than one-third of the informants (11 of 30) from

our initial survey indicated the specific sections of the EBQ that they utilized in their data analyses (e.g.,

general and language background, educational background, demographic information). As might be

expected, there was slightly greater focus on the items pertaining to NALS respondents’ educational

background and experiences. However, five researchers reported having used data from all sections of the

EBQ; others’ analyses have been limited to two, three, or four sections only (e.g., political and social

participation, literacy activities and collaboration), depending upon the particular questions posed in their

research.
7

The second part of our e-mail survey asked informants to identify the research questions of interest

to their studies. Further, we asked that they indicate how the EBQ items strengthened their ability to draw

inferences from the data, how the items may have limited such inferences, and what items or sections should

be revised or dropped for the NAAL. Eleven individuals completed our second survey via e-mail or were

interviewed in person or on the telephone.

Survey Results

A few individuals responded to our initial inquiry and identified themselves as employees of state

divisions or adult education programs; a few others were policy analysts for various social or political

action and lobbying groups (e.g., the American Association for Retired Persons (AARP)). None of these

persons reported having conducted inferential analyses of the NALS data, and so were not included in the

second part of our survey. All of our survey responses were from individuals who are social scientists of

one sort or another (e.g., educational researchers, economists, methodologists), and we therefore limited our

analysis to those researchers who have conducted secondary inferential analyses of the data.

We will identify the issues and concerns identified by these informants regarding their analyses

and interpretations of the EBQ data and summarize their comments as they pertain to each of the six content

areas of the EBQ. Unless otherwise noted, these comments are based upon our survey informants’

responses, rather than our own perspectives and views.

Section A: Language Background

This first part of the EBQ consists of 17 items that obtain information on NALS respondents’

acquisition and usage of English. Only one comment was obtained regarding the characteristics of this

portion of the EBQ. It was suggested that English language proficiency, rather than what people can do in

other languages—such as Spanish—should remain the focus of the NAAL. . We are aware that there has

been some attention devoted to a Spanish-language version of the NALS (Solorzano 1994). Reynaldo

Macias, a member of the NALS literacy definition and technical review committees, was lead author on the

language minority report, which is in draft final form with the American Institutes for Research (R.

Macias—personal communication, February 11, 1999). Two new variables—bilingualism and biliteracy—

were advocated in this report to draw attention to the fact that many individuals—particularly immigrants—
8

may be literate in more than a single language (English), or not literate in English but literate in their native

language (e.g., Spanish) or other languages not assessed by the NALS. It appears likely, however, that

assessment of English language skills will continue to be the focus of the NAAL.

It is our observation that the format of the language background portion of the EBQ may be

confusing for NALS respondents—particularly the contingency questions (i.e., items #A-6, #A-7, #A-8, #A9, #A-10) and the content of the hand cards. We have found, in our analyses, that it would be helpful to

have an item that simply asks, “what is your primary language?” We will have more to say on this, and

other issues, in the Recommendations portion of this report.

Section B: Educational Attainment and Experiences

This part of the EBQ has 14 items that obtain information about NALS respondents’ educational

histories. A few of the informants to our survey suggested that there is a need for additional questions

pertaining to educational and demographic variables. We will summarize their comments in the remainder

of this section. A few informants indicated that more questions about previous educational experiences

would be helpful. The item that indicates “some college” for educational attainment (i.e., items B-1, options

G, H, and I), for example, is too vague; many more people have “some college” experience or credit hours

than have a degree, and more needs to be known about these individuals’ educational histories. Exactly how

much is “some college” experience, in terms of credit hours earned or years of schooling completed? It

seems important to be able to quantify the varying amounts of college experience that respondents indicate;

knowing these amounts would allow for better comparisons to high school graduates who have some

technical school experience or a certificate, college graduates, and others.

While it was possible to find data to describe the prose, document, and quantitative (PDQ) literacy

of community college students, it was not possible to provide sound reasons why these students’ literacy

proficiencies were at given levels. Several persons noted that the data on prior educational experiences, for

example, were limited. Many college students are assigned to remedial courses, which are presently of

much interest to educators. (See Breneman and Harlow 1999.) More specific information about respondents

who are or were in remedial education would be helpful, according to one informant. Focusing on “highest

education achieved” does not allow one to study those who were enrolled in community colleges. Just as the
9

NALS considers literacy as a continuum, educational attainment should also be viewed along a continuum,

thereby avoiding distinctions about education. When the focus is only on the highest degree or educational

status, this disregards the multiple and interacting factors that contribute to educational achievement. One

informant indicated that more detailed information about NALS respondents’ educational background and

experience would be very useful—particularly at the state level. For example, if a high school diploma was

awarded, was it earned in regular or special education? Given the large percentage of individuals scoring at

the two lowest proficiency levels, it is reasonable to expect many of these persons to have been recipients of

special education services.

Other types of credentials earned outside of formal education might be useful to know as well. This

suggestion was made by a few informants—in particular, those who examined relationships between

education, work, income, and literacy. These credentials could include craft apprenticeships and the like,

which include the acquisition of skills that may be valued in the marketplace, but do not result in a degree or

diploma, and may or may not lead to any kind of certification. NALS respondents’ participation in various

forms of adult education would be very useful information for researchers, given the meteoric rise in

participation rates over the past decade.

Another informant commented that there was a need for more data on learning-disabled (LD)

adults within the population. According to this person, some data suggest that as much as 17 percent of the

U.S. population may have one or more learning disabilities. Those individuals at NALS proficiency levels 1

and 2, who have been the subject of most concern to politicians, policymakers, and adult educators, are the

most likely to have a learning disability (and to have been in special education) (Vogel and Reder 1998b).

Therefore, a revised EBQ should have an increased focus on assessing the presence of LD within the U.S.

adult population, according to one informant.

A question of particular interest concerns how these individuals’ learning disabilities were

diagnosed. Diagnoses of LD may be done by psychological (e.g., school psychologist) or medical experts,

and the type of diagnosis will largely determine the LD treatment or intervention. It would be useful to

know when the LD diagnosis occurred (e.g., age and/or grade level), if the respondent received any special

education services while in school, their knowledge of the Americans with Disabilities Act (ADA), and if

they had requested any school or workplace accommodations based upon ADA guidelines. Other questions
10

of interest regarding the LD population include: the impact of LD on employment; how respondents

compensate for their learning disabilities in regards to literacy and work; and how the learning disabilities

impact their personal lives.

Another informant reported that the educational background and experiences data were very

helpful to them in understanding the results—particularly at the state level—with regard to the association

of educational attainment with literacy outcomes. A few others, on the other hand, suggested the NALS

state-level data are very limited, and considered it best to use the national data for analyses and reporting.

We suggest that two questions regarding educational attainment would help to clarify matters

pertaining to educational achievement. One question should ask for the number of years of school

completed. The second question should ask for the highest degree earned. We will address, in a subsequent

section of the paper, what we view as the limitations of the EBQ regarding the items pertaining to NALS

respondents’ participation in basic skills education. Suggestions for increasing the validity of these items

will be provided.

Section C: Political and Social Participation

This portion of the EBQ consists of only four items that pertain to sources for obtaining

information, television viewing, use of libraries, and voting. One informant referred to the lack of a question

related to voter registration. The 1985 Young Adult Literacy Survey (YALS) (Kirsch and Jungeblut 1986)

contained questions regarding both registration and voting, which allowed for modeling of voting behavior

after controlling for the non-random selection of respondents (i.e., respondents can’t indicate “yes” to

voting without being registered to vote). Those who register are not a random sample of the population of

those persons who are eligible to vote on basis of age and citizenship. Also, items pertaining to NALS

respondents’ involvement in different volunteer and civic activities (aside from voting) would be

informative, and might provide additional insights about literacy practices.

Combining the few items from this section of the EBQ with other sections may be worthwhile. For

example, two items (#C-1, “how much information do you get from...” and #C-3, “how often do you use

the services of a library...”) are directly relevant to literacy practices and could be included in the section of

the EBQ that assesses such activities.
11

Section D: Labor Force Participation

The fourth section of the EBQ consisted of 12 items that gathered information about the type and

extent of NALS respondents’ occupational activities and income. An informant noted that there were no

questions asked about the occupations of the parents of the NALS respondents. Because parents’

occupations will have some impact on the educational opportunities available to children, influence the

kinds of literacy practices to which their children may be exposed, and provide models for participation in

the world of work, it is important to determine parents’ primary occupations. Such an item could be

included in the Demographic Information section of the EBQ. Also, it is critical to know respondents’

hourly wage rate. This information can be discerned from responses to items #D-3, #D-4, #D-7, and #D-8,

but a single item which obtains this information would be useful, according to one informant.

Another informant indicated that there is a need for good variables pertaining to income, but the

relevant issues were somewhat different for this individual. According to this informant, it would helpful for

individual states to know, from among those respondents who are receiving welfare benefits, the categories

of welfare benefits received, e.g., food stamps, medical assistance, and/or cash assistance. Some interesting

analyses might emerge from such data—at a basic level, it would be possible to determine the literacy

proficiencies of individuals receiving various levels and types of welfare support. This information may

have some important policy implications, particularly with respect to “welfare-to-work” initiatives. If, for

example, those at the lowest PDQ levels are receiving the highest levels of welfare benefits, this may

indicate that these persons are least likely to be able to enter the workforce without substantial basic

education training.

One informant, an economist, noted that “the data are extremely useful for economic analysis,” but

would have found it more fruitful to have additional questions about the work history of the individual—the

pattern of their previous employment and/or unemployment, how much they earned over each of the past

five years, and what occupations they have held. A similar point was raised by a second economic analyst.

Such information would allow for analyses of the impact of education on job mobility, according to both of

these informants.
12

Section E: Literacy Activities and Collaboration with Others

This portion of the EBQ contained only six items regarding NALS respondents’ uses of different

kinds of text sources, such as newspapers and documents. A number of comments pertaining to this section

of the EBQ were garnered and it appears that the information resulting from these items has been widely

used. We have, of course, used the literacy practices data extensively in our studies (Sheehan-Holt and

Smith in press; Smith and Sheehan 1998; Smith 1996). We therefore have several issues and concerns about

the literacy practices items on the EBQ. While we will describe the issues more fully in a subsequent

section of this report, we will briefly mention a few concerns here. First, the EBQ format pertaining to

newspaper reading did not allow for a straightforward quantification of the amount of reading. Second, the

item pertaining to book reading did not allow for estimates of the number of books read within given

categories.

Technology use has dramatically increased in both the workplace and in the home since the NALS

was carried out in 1992. For example, the advent of the World Wide Web on the Internet, and its

consequent availability to the mass public, has resulted in millions of Americans going “online” for

information, education, commerce and entertainment. How, then, does computer and other technology use

impact adult literacy skills and practices? What technologies are used and how frequently? What impact

does technology use have for adults with learning disabilities? How do other media, e.g., listening to books

on tape or viewing instructional videotapes to improve language or literacy skills, impact literacy? Thus,

there is an expressed need for items that pertain to uses of electronic communications, computers, and other

technologies for literacy uses and practices.

An informant very familiar with the NALS development and the resulting data argued that there is

a need for much greater detail regarding literacy practices. By asking respondents to provide specific

examples of the typical, routine or daily literacy practices in which they engage, for example, this kind of

information can help to validate their self-reports on literacy activities. It is critical for educators,

researchers and policymakers to know what kinds of text materials adults use and the kinds of literacy tasks

in which they typically engage.
13

Perhaps even more relevant, according to this informant, is to determine the literacy tasks that

adults face in specific settings and situations. For example, it is likely that, in most literacy situations,

people must utilize two, three or more kinds of materials. They not only have to use these materials to

accomplish the task(s) at hand, but they have to decide how to use this text or that one and often must gather

information from multiple—and sometimes conflicting—sources. The EBQ does not capture this kind of

real-life complexity of literacy tasks. This informant suggested that efforts be made to determine the nature

and types of challenging literacy tasks that people face by surveying about 10 percent of the NALS sample.

A diary for recording daily reading and writing activities might work well for this purpose (Anderson,

Wilson, and Fielding 1988). This approach would result in a better understanding of how well people can

accomplish more difficult literacy tasks and activities.

Another informant suggested that a revised EBQ should not contain the detailed question (item #E2) pertaining to which sections of the newspaper are read (e.g., main news, financial and editorial pages,

sports). However, because newspaper reading is the most pervasive and common reading activity among

adults (nearly 9 out of 10 adults read a paper on a regular basis; Smith 1996), it is important to know as

much about this particular reading practice as possible. Newspaper reading may be among the most

fundamental behaviors for individuals to obtain information about a wide variety of topics and issues (e.g.,

politics, sports, finance, community development, parenting, entertainment activities) in a relatively simple,

easy and accessible manner.

Others suggest that, rather than eliminating items from the EBQ, it is very important to retain the

literacy practices items. These items are important for determining what individuals with low education

attainment and low literacy proficiencies are doing with the literacy skills they possess. Another informant,

who has worked extensively on the International Adult Literacy Survey (IALS) described several

dimensions of literacy practices that pertain to (1) the incidence and frequency of these activities; (2) the

criticality of the practices; and (3) their complexity or challenge to the reader. At best, the NALS

background questionnaire only captures the incidence and frequency of practices. Research by the first

author (Smith in press) examined the reading practices among a sample of 150 adults and illustrated that it

is essential to obtain the kinds of information called for above in order to more fully understand adults’

motivations for reading, how they respond to demands for literacy, and the nature of these demands (see
14

also, Kirsch and Guthrie 1984). It might be possible, for example, to obtain more extensive reading

practices data among a sub-sample of NALS respondents in future surveys, as suggested above. We will

return to this issue when we discuss our recommendations for the next generation EBQ.

Two informants suggested the need for a cleaner distinction between job literacy and social

literacy practices (i.e., personal reading practices outside of the workplace), as these are likely to be

motivated by different demands, require different text sources and result in very different outcomes and

satisfactions for individuals. A more extensive assessment of literacy activities on the job, in the home, and

in adult education settings would be very useful.

Section F: Demographic Information

 The final part of the EBQ contained 13 items that gathered information on NALS respondents’

marital status, gender, family and personal income, racial/ethnic identity and age. We obtained few

comments from our survey respondents in regards to this section of the EBQ. Generally, these data were

used for descriptive purposes in the various secondary analyses conducted by researchers (e.g., age and

racial group differences in reading practices), and were widely used by nearly all of those who conducted

secondary analyses of the NALS data.

General Comments

Several comments and suggestions were of a more general nature and did not pertain to specific

content categories within the EBQ. We will summarize those comments here.

One informant remarked that, in those cases where specific background items had very few

respondents within a category, then the sample is obviously too small to make any meaningful inferences.

Further, while in some cases, smaller groups can be combined for data analyses to allow inferences, this

approach lacks the level of precision needed for adequate data interpretation. Two informants noted that,

overall, the design of the EBQ is weak (but offered no information about specific weaknesses of the design)

and one of them suggested that a short form of the literacy assessments (either combining PDQ into a single

scale or simply using one scale such as Prose) could be used for the NAAL 2002. Obviously, Reder’s

(1998b) analyses, which indicated that the three literacy scales are highly correlated with one another (rs >

.93), is relevant here. A more extensive EBQ could be developed and used to obtain more detailed
15

background information. However, another informant indicated that it is unnecessary to add items to the

EBQ. If anything, make it shorter and easier to complete in order to save time and money. A larger sample

could then be obtained. Finally, one informant suggested that a much more extensive background

questionnaire based on the Young Adult Literacy Survey (Kirsch and Jungeblut 1986) is needed for the

NAAL.

Summary

We obtained complete surveys via e-mail, on the telephone and in person from eleven individuals

who have employed the literacy proficiencies (PDQ) data from the National Adult Literacy Survey (NALS)

and various data from the NALS English Background Survey in their investigations of adult literacy. The

purpose was to ascertain the following: how these social scientists have used the EBQ data in their analyses;

how the relative strengths and weaknesses of the data have enhanced or hindered their research questions

and policy recommendations; and, how their suggestions for changing the EBQ to create an instrument (and

resulting data) would be more useful to the NAAL 2002.

It is clear that, although the numbers of individuals who have utilized the NALS EBQ data for

inferential analyses is very small (in contrast to other national databases, such as the NELS:88 which has

resulted in hundreds of studies over the years), the NALS data have been extensively used by these

researchers. It is not clear, from our analysis, how extensively the data have been used by those who are not

social science researchers but who nonetheless have a significant stake in adult literacy—local and state

directors of adult literacy programs, for example, or community leaders and politicians, policy analysts, and

leaders of business and industry. Anecdotal evidence, however (Smith and Reder 1998), suggests that uses

of the data by these persons is restricted to descriptive summaries and does not involve detailed, inferential

analyses of any EBQ data. It would be interesting, and useful, to learn more about how persons in these

positions have utilized the NALS data to suit their needs and the needs of their constituents.

The informants to our survey indicated use of the NALS EBQ to address a wide variety of

questions and concerns. They also indicated that, while the EBQ has several virtues, there are nonetheless

significant problems and limitations of the data derived from the EBQ. A number of suggestions regarding

changes to the next generation EBQ were offered with the goal of gathering more useful and informative
16

data that enable stronger inferences. These suggestions pertain to all sections of the EBQ and primarily

called for increasing the number of items to obtain additional and more detailed information.

In the next section of this report, we describe our review of the relevant NALS studies that have

utilized the EBQ data. For the most part, these studies were peer-reviewed prior to publication in a variety

of professional journals. A few reports are book chapters; one is a technical report submitted to the National

Center on Adult Literacy.

Content Review

As indicated above, several studies that involve secondary analyses of the NALS data have

appeared in the social sciences literature. We conducted a content review of the empirical reports that

utilized specific EBQ variables in the analyses. Our approach was to evaluate the researchers’ abilities to

make strong inferences from these data, based on the results of their analyses and interpretations of

findings.

We examined the research questions for each paper, the methodology employed, the relevance of

the NALS data to the research questions ( based either entirely or in part on NALS), the specific sections of

the EBQ and specific items used for the analyses, and the results coupled with researchers’ interpretations

of their findings.

Criteria

The following general criteria were used to review these studies:

1. Is the research method appropriate to address the research questions?

2. To what extent do the EBQ data allow for strong inferences on the investigators’ part?

3. How did the strengths or weaknesses of the EBQ data affect the researchers’ policy

recommendations (if any)?

Review of Studies

Fourteen studies, in which various portions of the EBQ data were employed in the researchers’

analyses of adult literacy, are reviewed. The papers are, for the most part, described in alphabetical order

(by author). Friedman and Davenport (1998) examined gender differences in NALS literacy proficiencies,

by age, educational attainment, and racial/ethnic groups. The research objective posed by the authors was a
17

general one—to explain patterns of gender differences in literacy performance, taking into account age and

ethnicity. The NALS was the only data source for this investigation, and the EBQ data were important to

the study. T-tests were used to compare males and females overall, and then separately for different age and

ethnic groups represented in the sample. The relevant EBQ section pertained to respondents’ demographic

information (age and race/ethnicity).

There were significant gender differences among the 40-54 year old adults and those aged 65 to 80

on Quantitative literacy only. There were a few gender differences on Quantitative literacy for whites only

(for ages 40-54 and 65-80, favoring males). The researchers were able to make reasonably strong inferences

from the data by examining differences among the age and racial/ethnic groups. They did not, however, take

advantage of the large array of control variables in the NALS. In fact, they stated that the purpose of their

study was to focus only on the variables that have been previously examined in the research literature. Their

data analytic approach was exploratory in nature (i.e., they used the first plausible value for each PDQ

variable, rather than the plausible value methodology proposed by Mislevy, Johnson, and Muraki 1992).

Descriptive statistics were used to discuss differences among age cohorts.

Friedman and Davenport concluded that educational attainment accounted for most racial/ethnic

differences in literacy proficiencies, and that, as a nation, we are “doing better” at educating racial and

ethnic minorities. In regards to gender, young women’s performance on the NALS illustrates the strides that

females have made in academic achievement in recent decades. They also claim that the NALS results

illustrate that cultural, rather than genetic, factors account for gender differences. No policy

recommendations were made regarding methods to further close the “gender gap” in literacy.

Gerber and Finn (1998) examined the relationship of literacy practices at both school and work

with document proficiency. They note the discontinuity between work and school literacy and ask: (1) to

what extent is document proficiency related to schooling? and, (2) to what extent are document skills

acquired in the workplace? This study was based entirely upon the NALS data and the EBQ data were

highly relevant. Information from the following EBQ sections was necessary to conduct their analyses: (1)

educational background; (2) labor force participation; (3) literacy activities; (4) language background; and

(5) demographic information. Tests of mean differences and multiple regression analyses were performed,

and descriptive statistics were used to address the research questions. Gerber and Finn found, first, that
18

schooling is associated with better performance on document-related tasks, as adults with more education

outperformed those having less education. Second, those who learned to read documents at work had at

least as high a proficiency in document literacy as those who learned at school and in the community. Third,

they found that reading documents at work is significantly associated with document literacy performance.

Those adults who engaged in high levels of practice with work documents had the highest document literacy

scores. Also, type of occupation is significantly related to document literacy skills. Higher-level occupations

(e.g., professionals, managerial, technical, sales, and clerical) had higher document scores than lower-level

occupations (e.g., service workers, farming, laborers).

Their findings illustrate that the workplace provides a context for adults to acquire document

literacy. Thus, both school and the workplace are important contexts for literacy development. The

workplace may, in fact, be a suitable alternative learning environment for many individuals (e.g., high

school dropouts), although further research is needed to examine this hypothesis. Gerber and Finn were able

to answer their research questions and their use of the EBQ data was very appropriate. They were able to

make strong inferences regarding the relationship between workplace reading practices and document

literacy proficiency. These findings led them to suggest that employers may need to be more proactive in

creating on-the-job educational opportunities—for their workers—which has several implications for

workforce policy development and workplace literacy programs.

Greenberg, Swaim, and Teixeira (1995) analyzed workforce literacy among rural adults in contrast

to urban adults, and also examined differences by age, educational attainment, race/ethnicity, and region of

the country. Comparisons on PDQ scores were made by metropolitan areas (e.g., central city, suburban,

medium, small) and non-metro, rural areas (classified as either urban-adjacent, urban-nonadjacent, or totally

rural). Greenberg et al. were able to distinguish levels of urbanization because they could identify the

county of residence for each NALS respondent. Literacy levels of rural (nonmetro) adults vary widely, but

are quite low, on average (specifically, near the upper end of Level 2).

Greenberg et al. analyzed the literacy skills of rural and urban adults in order to determine the

extent to which rural workers are able to compete economically with urban workers. Because there is a

difference between urban (metro) and rural (non-metro) workers on literacy, this gap suggests that most
19

rural areas have a workforce literacy problem—only 19 percent of rural adults were found to score at levels

4 and 5 on prose literacy.

Greenberg et al. regressed individual literacy scores on either a dummy variable for metro county

or five dummy variables for the most highly urban county types along an urban-rural continuum. The

resulting coefficients measured the extent to which mean literacy is higher for more urban counties than in

the most rural counties (i.e., what they referred to as “the rural literacy gap”). Next, they re-estimated these

models using 33 independent variables measuring factors such as age, gender, marital status, education,

parents’ education, race, ethnicity, native-born speaker of English, and region of country. Thus, data from

the demographic section of the EBQ was particularly vital for these analyses. This modeling approach

allowed the researchers to examine the impact of each of the independent variables on literacy, while

holding other characteristics and urbanization constant. The authors also tested the interaction of the

rural/urban variables with each of the 33 control variables. The resulting model coefficient provided an

estimate of how each characteristic either widened or narrowed the rural gap in average literacy.

Greenberg et al. used similar regression techniques to investigate the effects of literacy on

employment status and earnings. They regressed individual employment status on literacy scores using 24

control variables for labor market experience, gender, marital status, education, race/ethnicity, native-born

speaker of English, region of country, and metro residence, among others. Greenberg et al. wanted to know

what the most important demographic characteristics that depress literacy in rural areas are, and if these

variables are amenable to policy interventions. Also, they wanted to determine whether there are any

advantages to literacy for rural-dwelling adults. Higher age and lower education were found to contribute to

lower literacy levels. The labor market rewards for literacy (e.g., income) are much lower in non-metro

areas than in metro labor markets. Therefore, the demand for workers with good literacy skills is less in

rural than it is in urban areas. Lower economic rewards for literacy in rural areas tend to depress literacy

because individuals have less incentive to develop these skills; those with higher literacy skills go to urban

area jobs, according to Greenberg et al.

The researchers were able to make strong inferences from the data by employing a large number of

background characteristics as control variables. They noted that it was very important that the county of

residence was given, since this information was essential to determining the level of urbanization. The
20

researchers made some suggestions for rural education policy and closing of the wage gap for high-literate

rural workers. The study illustrates the need to prepare the rural workforce to meet the increasing literacy

demands of the workplace in both urban and rural settings. From a methodological standpoint, using a great

variety and number of background variables derived from the EBQ was a particular virtue of the study, and

strengthens the researchers’ inferences.

Howard and Obetz (1996) examined the literacy proficiencies of community college graduates. As

community college educators themselves, they were very interested in examining the relationship between

community college graduates’ literacy skills and literacy practices as these pertained to (1) having

graduated from a community college and (2) graduates’ acquisition of information from a variety of

sources—television viewing, listening to the radio, reading newspapers and magazines, and from family and

friends. Their analyses are straightforward and, for the most part, descriptive. They compared the

percentages of high school, community college and baccalaureate graduates at each prose and document

literacy level. In regards to literacy proficiencies, they found that community college graduates perform “in

the middle,” that is, higher than high school graduates but lower than college graduates. The authors did not

appear to use much of the demographic information to tease out the other factors in the participants’

backgrounds, besides educational attainment, which might have important effects on assessed differences in

literacy.

Community college students’ literacy practices were examined, but were not compared to other

education levels. Howard and Obetz offered several suggestions for community college educators regarding

the improvement of their students’ literacy skills and practices (e.g., “use...students’ background

information...to develop comprehension and writing skills,” and “developing...students’ awareness of

interest in reading for self-improvement or recreation,” 467). The primary EBQ sections relevant to this

study were for demographic information and literacy practices. Because the EBQ item that assesses the

amount of personal use writing one does (#E-7) is largely limited to the types of writing one does at work

(e.g., letters, memos, forms and reports), inferences from these data are not as strong as they would be if the

item measured the types of personal writing individuals’ engage in at home and in other nonwork settings

(e.g., writing recipes, taking phone messages, making grocery lists, journal writing). Further, since this
21

study is largely descriptive, the researchers should be cautious in making inferences that go beyond the

NALS sample.

Pryor and Schaffer (1997) examined the question of why university-educated adults have been

found, increasingly, to take jobs requiring high school level skills. This trend has resulted in rising wages

for these jobs. Their research question is as follows: Who are the university graduates that are ending up in

high school jobs? Although it is somewhat unclear from their description, their analyses appeared to be

based largely on data from the Current Population Survey (CPS) for 1971, 1979, 1987, and 1995. NALS

EBQ data may also have been used to examine the numbers of university-educated adults in high school

jobs. They then used the NALS PDQ scores to link “functional literacy” to labor force status, occupation,

and wages. Literacy proficiency scores and hourly earnings were compared across four levels of educational

attainment (high school dropout, high school diploma only, some university courses, and university degree).

Pryor and Schaffer show that it is primarily those university graduates who lack university-level

literacy skills who are taking high school level jobs. Also, it is mostly those who are university-educated

and in jobs requiring university-level literacy skills who are obtaining the major wage increases. This

analysis resolves the “paradox” of why the apparent surplus of university graduates is associated with the

rising wages of this group.

For the purpose of linking functional literacy to labor force status, occupation, and wages, NALS

proficiency scores were used by taking the average of the three PDQ scales. First, they examined functional

literacy data by education and occupation; they then looked at the relevant wage data. The data show that

the functional literacy of workers with a given level of education increases as the occupational tier

increases. Thus, those with a university education working in occupations in which most workers had less

than a university education showed lower functional literacy than university-educated individuals in

university-level jobs. This suggests that university-educated persons in high school level jobs took these

jobs because they had lower literacy qualifications than other university-educated workers and could not

obtain jobs in occupations commensurate with their education.

Pryor and Schaffer note, and this is also confirmed by Gerber and Finn (1998), that it is possible

that functional literacy can be influenced by on-the-job learning, thus the job itself can play a role in the

relationship between occupation, education, and functional literacy. They concluded that, from 1971–87,
22

university-educated workers of all ages were more frequently taking jobs in which the average educational

level required was much lower (i.e., high school education). Also, university-educated workers who have

experienced downward occupational mobility have, on average, much lower functional literacy than do

other university graduates. Finally, the increase in wages of university-educated adults who are pursuing

higher-level occupations reflects a shortage of university-educated workers with the functional literacy skills

that are commensurate with a university education.

We were unable to confirm from our reading of the report, and Pryor’s responses to our e-mail

survey, whether the background and demographic information used in the Pryor and Schaffer analyses were

derived from the NALS EBQ or the CPS data. The NALS proficiency scores were more crucial to their

arguments than were the background data, in any case. Generally, descriptive analyses were conducted.

Raudenbush and Kasim (1998) examined racial, ethnic and gender gaps in earnings and

employment. Although educational attainment is now similar for American men and women, a large gender

gap remains in earnings. Also, while African-Americans and Hispanic Americans have fewer years of

schooling and fewer degrees than European Americans, large ethnic gaps also exist in access to employment

and earnings, even after educational attainment differences are taken into account. Several explanations for

these gender and ethnic gaps in employment and earnings have been offered, according to Raudenbush and

Kasim. First, quantitative measures of education (e.g., years of schooling and degrees obtained)

inadequately capture the cognitive knowledge and skills that are important in the labor market. Second,

people vary in their preferences for different kinds of work, and therefore select occupations that have

varied job security and earnings. For example, women tend to select jobs that pay less than jobs that men

select. Raudenbush and Kasim tested these explanations by examining between- and within-occupation

inequality in employment and earnings by gender and ethnicity.

They addressed four questions in their investigation: (1) How important is cognitive skill, as

indicated by the NALS literacy assessment, in reducing the risk of unemployment and predicting earnings?

(2) How large are differences in cognitive skills between groups? (3) How important is group inequality in

cognitive skills for understanding social inequality in labor-force outcomes? (4) To the extent that cognitive

skill differences do not account for ethnic and gender differences in economic outcomes, does the remaining

inequality lie between or within occupations, or both?
23

The key variables for their analyses were the following: (1) social origin (age, parent education,

gender); (2) human capital (educational attainment, credentials obtained, labor force experience, and

literacy proficiency); (3) occupation (based on Department of Labor classifications); and, (4) labor force

outcomes (natural log of wages per week and employment status). Therefore, data from several sections of

the EBQ were critical to their analyses. The data analyses included the use of both the ordinary least squares

(OLS) method and two-level hierarchical linear modeling (HLM) procedures (i.e., both between and within

occupations).

Raudenbush and Kasim found that the independent contribution of literacy (cognitive skill) to

earnings is important, but not very large. On the other hand, the independent contribution of education, after

controlling for literacy, is only marginally significant. Also, they determined that most of the effect of

obtaining educational degrees is independent of literacy (cognitive skills). Further, literacy was found to

contribute to reducing unemployment, but the effect was not large.

Gender gaps in literacy are small but vary slightly across ethnic groups (as noted by Friedman and

Davenport 1998). However, the ethnic gaps in literacy are quite large between African-American, Hispanic,

and Asian males as compared to European American males. Ethnic gaps in literacy among females is much

smaller. The OLS model accounted for 57 percent of the variance in literacy, but the HLM analysis

accounted for 95 percent of the variance in literacy between occupations and 41 percent of the variance

within occupations. Thus, literacy appears to be a potentially important variable in understanding gender

gaps in employment and earnings.

The ethnic gap in earnings between African-American and Hispanic males as compared to

European males is quite large, but is substantially reduced by controlling for adult literacy. However, for

females there is no ethnic gap in earnings for these groups. Raudenbush and Kasim found that 40 percent of

the expected difference in log wages between African-American and white males, having the same

education, labor force experience, and parent education, is attributable to the fact that African-American are

less likely than whites to work in favorable occupations. Literacy explains more than half of the betweenoccupation gap, but less than half of the within-occupation gap. Having lower literacy skills, therefore,

denies African-Americans access to favorable occupations and helps to explain wage inequality on the basis

of ethnicity. Adult literacy is an important explanatory variable in regards to SES and ethnic inequalities in
24

earnings, and is an explanatory variable in regards to ethnic inequality in employment and risk of

unemployment.

Gender gaps in earnings ranged from large (e.g., European Americans) to small (e.g., Hispanics).

Controlling for literacy has little or no effect on this gender gap. Further, the majority of the gender gap is

within-occupation, rather than between-occupation. Therefore, neither theories of gender-based

occupational preferences or of discriminatory occupational segregation explain this difference. Controlling

for literacy essentially eliminated the between-occupation component of the ethnic gaps in risk for

unemployment. Ethnic inequality in employment opportunities is a within-occupations phenomenon,

according to Raudenbush and Kasim. However, about two-thirds of the African-American versus European

American male gap in earnings, and essentially all of the gap in unemployment, lies within occupations.

Among Raudenbush and Kasim’s conclusions were that there are important differences in

cognitive (literacy) skills among persons sharing the same educational backgrounds, and these differences

are linked to prospects for employment and earnings. The quality of schooling and of nonschool

environments must, therefore, be improved for Hispanics and blacks. (It is interesting to note that Reder,

1998b, found that inequities in literacy proficiencies may impede any achieved equities in schooling—e.g.,

educational attainment—between minority and nonminority groups. Thus, attention must be devoted to

increasing literacy among minority group members.) However, the situation is different for the gender gap

in earnings, since higher literacy is not a predictor of this earning discrepancy. This is consistent with the

possibility of labor-market gender discrimination.

Raudenbush and Kasim employed very elegant analyses to address each of their research

questions. They were able to make strong inferences from the data because they (a) used many control

variables, and (b) innovatively conducted the analyses to separate within- and between-occupation

differences. They noted in their discussion, however, that although they determined that much of the

earnings gaps are due to within-occupation differences, they could not discern if this result was due to

various noncognitive factors. We assume that this refers to motivational or behavioral differences between

the genders and among the racial/ethnic groups, which may account for some of the earnings gap. This

suggests that assessing some aspects of motivation (as it pertains to literacy) on the next generation EBQ
25

might be worthwhile. Among the limitations of their study, Raudenbush and Kasim suggest that a better

measure of cognitive skills would have perhaps enabled them to account for the wage-earning gap.

Steve Reder has been one of the leading analysts of the NALS data, having published several

studies regarding various aspects of the data (Reder 1995, 1998a, 1998b; Vogel and Reder 1998a, 1998b).

For example, he and Susan Vogel have examined the relationships of educational attainment and literacy

proficiencies among adults having one or more learning disabilities, as self-reported in the EBQ.

The primary purpose of the research reported in Reder (1995) was to disentangle the interactive

effects of learning disabilities, literacy, and education on a number of adult outcomes. Several descriptive

results were also reported. Reder examined both the prevalence of learning disabilities (LD) in the adult

population and its distribution among various age, gender, and ethnic/racial groups. Also, the educational,

social, and economic correlates of LDs were examined. Quantitative models of the joint impact of LDs,

literacy, and education on these social and economic outcomes were tested.

Several regression methods were used to determine how LDs, education, and literacy mutually

affect one another. A reciprocal effects model (e.g., educational attainment affects literacy; literacy

acquisition affects subsequent educational attainment) was also used to assess the direct and indirect effects

of LDs on a number of social and economic outcomes. Structural equation models (SEM) were used to

distinguish the direct effects of LDs on literacy from the indirect effects of other variables on literacy.

Essentially, Reder wanted to know whether or not learning disabilities affect a variety of economic and

social outcomes beyond what would be expected from the lower levels of educational attainment and

literacy proficiencies that are associated with LDs. The reciprocal effects model (e.g., literacy and education

are assumed to mutually influence one another) fit the NALS data extremely well. Thus, Reder was able to

distinguish direct and indirect effects of LDs on literacy, education, and several social and economic

outcomes.

Reder (1998b) also used SEM to test the effects of literacy selection and literacy development on

adults’ literacy skills. As noted above, literacy selection is the influence of literacy skills on educational

attainment. Literacy development, on the other hand, is the effect of educational attainment on literacy. A

third, reciprocal effects, model was also tested. SEM was used because of the simultaneous effects of

education and literacy upon one another. The variables used in the analysis were age, gender, race/ethnicity,
26

educational attainment, parents’ education, learning disability status, log of annual earnings, and PDQ

proficiency scores.

The reciprocal effects model proved to better fit the NALS data than either the literacy

development or literacy selection models by themselves. The reciprocal effects model accounted for 40

percent of the variance in educational attainment and 54 percent of the variance in literacy proficiency,

according to Reder. Next, Reder used the reciprocal effects model to examine the effects of education and

literacy on a single economic outcome (i.e., annual earnings). Again, the reciprocal effects model fit the

NALS data well for predicting annual earnings. Overall, literacy development processes were found to be

more robust than literacy selection processes.

One of the limitations of the analyses, and corresponding results, is the use of the cross-sectional

NALS data to model longitudinal outcomes (i.e., literacy development). Nevertheless, the reciprocal effects

model fit the data well at a single point in time. The study demonstrated the need for longitudinal

examinations of adult literacy acquisition. Clearly, Reder employed very appropriate data analysis

techniques, taking into account the sampling design of the NALS by rescaling the sampling weights

according to the estimated design effect and using plausible values methodology to estimate coefficients and

standard errors (see Mislevy, Johnson, and Muraki 1992). Inferences were strong, mainly because of the use

of a complex model that controlled for many variables that impact literacy and educational attainment.

Policy recommendations were offered to the effect that educational equity must include discussions of both

educational attainment and literacy proficiencies—particularly because minority wage differences disappear

when literacy is taken into account.

Reder, in collaboration with Susan Vogel (Vogel and Reder 1998a), examined the educational

attainment of adults who reported having one or more self-reported learning disabilities (SRLD) in the

NALS. Descriptive statistics were reported (e.g., SRLD prevalence rates, gender ratios) and several

inferential tests were conducted using design-weighted data. This procedure compensates for the complex

sampling design of the NALS.

Among the findings was a significant two-way interaction of group (SRLD vs. nonSRLD) by

gender on years of schooling completed. Having a SRLD had a large effect on educational attainment. The

high school/GED graduation rate was only 48 percent within the SRLD group in comparison to 84 percent
27

in the nonSRLD group. Among the SRLD group, more women (55 percent) than men (42 percent) had

completed at least a high school education. Vogel and Reder provide a number of conclusions to their

findings and offer several implications for educational policy regarding the improvement of educational

outcomes for students with learning disabilities. These include: (1) creating of meaningful school

environments that ensure success for students with learning disabilities; (2) improving the transitions from

junior to senior high and from secondary to postsecondary education, including helping learning-disabled

students to become more aware of the many postsecondary educational opportunities available to them, and

creating personal transition portfolios that help students retain documents pertaining to their learning

disabilities; and, (3) identifying and offering follow-up services to learning-disabled students who drop out

of high school.

This study enabled the researchers to generate educational policy recommendations from the

NALS data. The EBQ data were particularly useful in this regard, as essential data were derived from the

EBQ, including educational attainment (Vogel and Reder could determine not only if, but when respondents

dropped out of school), gender and self-reported learning disability. Their breakdown of the information

from the educational background portion of the EBQ into various types of secondary and postsecondary

educational experiences strengthened their inferences. However, the self-report nature of the learning

disabilities identification item may have limited their inferences because self-identification can be

unreliable. In contrast, in the National Longitudinal Transition Study (NLTS) cited by Vogel and Reder,

learning disability was identified by the respondents’ school rather than self-reported. In comparing rates of

learning disabilities between the NALS and NLTS, Vogel and Reder found large differences.

In a second article, Vogel and Reder (1998b) conducted several inferential analyses to compare

SRLD adults to adults without learning disabilities and, again, examined gender differences between both

groups. Both likelihood-ratio t-tests and analyses of variance were used to compare variable means. The

focus of analyses was specifically on observed differences in literacy proficiency. PDQ scores were

averaged to obtain a single indicator of literacy. The results were parallel to those reported in Vogel and

Reder (1998a), as there were significant differences in literacy between SRLD and nonSRLD adults.

Although overall, men and women do not perform differently on literacy, nonSRLD men slightly

outperform women, while SRLD women perform significantly better than SRLD men. This latter finding is
28

in stark contrast to other studies that have shown that school-age females fare much worse than males on

reading achievement measures. Vogel and Reder attribute this discrepancy to gender bias in school

procedures for special education referrals.

Generally, their data analytic methods were appropriate and the EBQ data allowed for strong

inferences regarding SRLD versus nonSLRD differences in literacy. However, since Vogel and Reder are

making inferences regarding two intact groups, controlling for some additional background variables (e.g.,

family income, parents’ education) could have further strengthened their inferences. An obvious limitation

of the data is the self-reported nature of LD identification. Vogel and Reder made few policy

recommendations in this report. They note, however, that their finding that severe literacy problems persist

well into adulthood for those with SRLDs has important implications for educators. Fortunately, research

has demonstrated some effective methods for teaching reading to students with learning disabilities;

classroom teachers should learn more about how to implement these methods.

Venezky, Kaplan, and Yu (1998) developed a predictive model of adults’ voting behavior using

the NALS data (see also Venezky and Kaplan 1998). Demographic characteristics (e.g., age, gender,

race/ethnicity, educational attainment, household income), literacy proficiencies (e.g., prose literacy) and

literacy practices (e.g., frequency of newspaper reading, frequency of job-related reading, number of

magazines read, and amount of newspaper and book reading) were used to predict voting behavior using

logistic multiple regression analysis.

The researchers’ analyses were appropriate and included very sophisticated multivariate methods.

The results indicated that the likelihood of voting increases with age, literacy proficiency level, educational

attainment, and income. Females were slightly more likely to have voted than males, and whites were

somewhat more likely than other racial/ethnic groups to have voted. However, when age, education,

income, prose literacy proficiency, reading habits, and other variables were statistically controlled, blacks

were found to have a significantly higher probability of voting—indicative of the importance of using

multivariate procedures when analyzing these kinds of data. Education was the single most important

predictor of voting behavior. Other differences in reading practices were reported. Generally, those who

read more text materials (e.g., newspapers, books) were more likely to vote than those adults who read

fewer materials.
29

The researchers’ inferences were limited due to the restrictions of the data. First, the crosssectional nature of the NALS design does not allow for an examination of changes in voting behavior over

time. Second, analyses of voting behavior were restricted because there was no question regarding voter

registration status (e.g., “Are you registered to vote?”) as in the Young Adult Literacy Survey (YALS)

(Kirsch and Jungeblut 1986). Also, the voting question was different from that posed in the YALS (i.e.,

voting in local elections was not included in the NALS—only in national or state elections). Therefore, it

was impossible to compare (1) probability of voting in local elections and (2) results from the two surveys

for the 21-25 year old cohorts (the age range of the YALS).

Due to the wording of the question, inferences regarding educational attainment were also limited.

The NALS EBQ asked for the respondent’s highest level of educational attainment. If, for example, an

individual were to drop out of school in 10th grade, but then earn the GED, that would be coded as

equivalent to an individual who dropped out at grade 12 and subsequently earned the GED. The authors

suggest that these kinds of data limitations restricted their inferences. Yet, they nonetheless put forth some

policy recommendations, and they note that, in future studies, it would be interesting to investigate the

impact of adults’ use of online materials on reading practices. We next turn to a description of three NALS

studies that the authors of this report have conducted.

Smith and Sheehan-Holt Studies

We have conducted several analyses of the NALS using data drawn from the EBQ. These analyses

have largely focused on respondents’ reading practices and literacy proficiencies. Data from Section B

(Educational Background and Experiences), Section E (Literacy Activities and Collaboration) and Section

F (Demographic Information) of the EBQ have been the primary data sources in these analyses, along with

PDQ scores.

Smith (1996) conducted a study examining associations between adults’ PDQ literacy proficiencies

and their reading practices involving the uses of five types of print contents—books, newspapers,

magazines, and personal and work-related documents. Comparisons were made between high- and lowactivity readers within each text category in regards to literacy proficiencies. The cut points for

distinguishing between the high- and low-activity readers corresponded approximately to that point dividing
30

the distribution in half, allowing for use of all of the available data. A secondary focus of the study was on

age differences in reading practices, taking advantage of the cross-sectional design of the NALS. Both

descriptive and inferential results were reported. The primary data analyses employed analysis of variance

to test differences between high- and low-activity readers in regards to the different print contents. Multiple

regression analyses were performed to examine age group differences while controlling for educational

attainment. The findings can be summarized by saying that, in general, those who read more categories of

printed materials—regardless of the nature of these materials—had significantly higher PDQ scores than

those who read fewer categories of texts. Book readers demonstrated the highest PDQ scores. Both

educational attainment and reading practices were found to contribute significantly to PDQ literacy

proficiencies.

Inferences about between-group differences could have been strengthened by taking into account

important demographic characteristics (e.g., parents’ educational attainment, income). The study also would

have benefited from having additional information pertaining to adults’ uses of reading materials and more

detailed information about the materials themselves. For example, respondents were asked to indicate the

number of categories of books read, rather than to estimate the total number of individual books read.

Obviously, some individuals may read dozens of books within a single category, while others may read a

few books from several categories. Thus, this estimate of book reading is a very crude measure of reading

behavior. 42 percent of the adult population read from a book category at least once a week or more. Given

the pervasiveness of book reading, it would be very informative to know more about the quantity and types

of books read. Also, book reading was found to be higher for blacks than for whites when some background

characteristics were controlled. It would, therefore, be interesting to know if this finding holds up for the

total number of books read, or for number of books within certain categories.

Although somewhat more difficult to obtain, data regarding the relative quality of the reading

materials typically used by adults would also be informative. Some book and periodical readers read only

highbrow literature and reputable news dailies and journals (e.g. the New York Times, Atlantic Monthly,

Wall Street Journal); others’ reading tastes tend toward popular fiction (e.g., romance novels) and weekly

tabloids; still others’ reading may lie somewhere between these two extremes. Asking respondents to

indicate specific, prototypical examples of their reading on the EBQ will allow us to obtain highly useful
31

and interesting data and enable much more fine-grained analyses of adult reading practices and how these

practices may contribute to literacy skills (this point was also raised by one of our survey informants—see

above).

The Smith and Sheehan (1998) chapter summarizes the Smith (1996) study and includes a second

set of analyses by the two authors who examined racial/ethnic group differences in reading practices. No

specific research questions were stated. A series of regression analyses were conducted. Age, education,

occupation, and prose literacy, but not income, were controlled in all analyses. Several interesting

racial/ethnic group differences were found with respect to reading practices that require further study.

Recommendations pertained to increasing document literacy instruction in school, particularly for ethnic

minorities, and noted that increased efforts should be directed toward developing the reading skills and

practices of socially and economically disadvantaged adults in ways that lead to higher literacy skills. The

authors could not make strong inferences regarding the literacy practices of Hispanics (those whose native

tongue is Spanish) since the NALS assessed only English literacy skills and practices.

Sheehan-Holt and Smith (in press) also investigated the association between adults’ participation

in basic skills education and their reading practices. The rationale for this study was to ascertain the role

that basic skills programs may play in contributing to adults’ literacy. Obviously, in an ex post facto study

of this type, there are no pretest measures to distinguish basic skills participants from nonparticipants. A

number of control variables were therefore used in the regression analyses, including educational

attainment, English as primary home language, presence of disability or long-term illness, and newspaper

reading practice.

The NALS EBQ contained three items pertaining to basic skills education. The first asked if the

respondent had ever taken part in a program (other than in regular school) to improve their basic skills in

reading, writing, and arithmetic. If the respondent indicated “yes,” s/he was then asked to identify the type

of program in which s/he had participated. Four broad categories of programs were listed, including (1) a

training program or courses given or sponsored by an employer or union, (2) a publicly sponsored

education and training program such as Job Training and Partnership Act (JTPA) or Adult Basic Education

(ABE), (3) a tutoring program sponsored by a library, church, or community organization, and (4) any other
32

program. Finally, they were asked how recently they had taken part in this training: still enrolled; within the

past year; between one and five years ago; more than five years ago.

No evidence was found that, as compared to nonparticipation, participation is related to better

literacy skills. Basic skills participants did have higher reading practices in four of the five reading content

areas. For example, adults who took part in job-related basic skills programs read newspapers and work

documents more frequently than did those who participated in tutoring-type programs. The NALS limited

the investigators’ ability to make inferences regarding possible outcomes when adults participate in basic

skills programs, since there were no data regarding the duration of individuals’ participation in basic skills.

This information would be very useful, as other studies suggest that upwards of hundreds of hours of

instruction may be needed to positively affect literacy skills for those individuals at the lowest literacy

levels (Sticht and Armstrong 1994).

The researchers were able to address their research questions in each of the two studies above.

Appropriate sampling weights and plausible value methodology (Mislevy, Johnson, and Muraki, 1992) were

employed in every case and, in Sheehan-Holt and Smith, the design effect using HLM was taken into

account. The nature of the EBQ affects the inferences made in all three studies. In the case of the SheehanHolt and Smith study of basic skills participation, much more information is needed in regards to the quality

of basic skills programs and instruction, duration of participation in basic skills programs, and so on, to

determine how such programs may impact literacy skills. The use of a large number of control variables

helped to strengthen the inferences made regarding the differences between participants and

nonparticipants, although this approach does not substitute for a longitudinal analysis.

Summary of Content Review

A total of 14 papers reporting analyses of the NALS data, including data that were drawn from the

EBQ, were described in this section of our report. The foci of these studies pertained to the associations

between literacy and education, gender, age, race, ethnicity, and voting behavior, inequalities in economic

opportunities, employment and wages, and learning disabilities, among other topics. We identified the

research questions, the nature of the data analysis procedures, and most significantly, the relative abilities of
33

the investigators to utilize data derived from the NALS EBQ to conduct their studies in a satisfactory

manner. In other words, how adequately did the EBQ data contribute to these studies?

The research questions are obviously constrained by the data available to the analysts. Thus, the

researchers generally did not pose questions that they would be unable to answer with some degree of

confidence. It is true, of course, that most would have preferred to have more variables to draw upon from

the EBQ, and in some cases, entirely different variables to enhance their studies and shed further light on

the explanatory variables regarding adult literacy. Nonetheless, these studies collectively go a long way

toward furthering our understanding of the powerful role that literacy plays in American adults’ lives in the

late twentieth century.

Conclusions

What kinds of information should be obtained in a revised English-language Background

Questionnaire for the NAAL 2002? The recommendations that follow are based on our survey of NALS

EBQ data users, analyses of the published research reports (in the literature) that have used data from the

EBQ, our own extensive experience with these data, and the recommendations of a series of focus groups

conducted by the American Institutes for Research (AIR) in early 1998. We recommend several revisions

and changes to the EBQ for the NAAL 2002. We discuss various technical and substantive implications of

these recommended changes, our views on the impact of these changes in terms of costs and benefits to

adult literacy stakeholders, and how changes to the EBQ might impact the assessment of trends in adult

literacy.

Technical and Substantive Implications

How well do the background data enable researchers and policy analysts to make strong inferences

regarding the effects of various factors on adult literacy? Our evaluation of the published studies that have

utilized the NALS English-language Background Questionnaire data leads us to give the EBQ a “passing

grade” in terms of the utility of the data for enabling strong inferences regarding literacy within the U.S.

adult population. With the advent of sophisticated statistical software and methods in recent years—

particularly hierarchical linear modeling (HLM) and structural equation modeling (SEM)—researchers are

able to test various models to determine the nature of relationships among variables. The NALS data,
34

because of the sampling design employed and large numbers of variables and participants that characterize

the survey, are well suited to these high-level statistical procedures and yield very robust results.

Nonetheless, there are several important limitations and weaknesses of the EBQ data that have

been acknowledged by the developers and pointed out by those who have subsequently utilized the data for

research and policy. The identified problems resulted from the inevitable tradeoffs and compromises

necessary to construct a measure which captures essential data within the practical and logistical constraints

of conducting a large-scale and costly national survey. Clearly, the 1992 NALS should serve as a learning

tool for refinement of the next-generation NAAL and every attempt should be made to correct the

acknowledged flaws and overcome the perceived limitations of the EBQ. At the same time, changes to the

current version of the EBQ must be weighed against the potential loss of opportunities to examine trends in

adult literacy from 1992 to 2002. Such examinations will be significantly hindered if the EBQ is radically

altered. Perhaps a reasonable compromise is to include additional items, as suggested by several of the

researchers whom we interviewed, and to maintain most or all of the present items. This approach, of

course, will impact the manner in which adults’ literacy proficiencies are assessed.

Recommended Changes to the EBQ

The following comprises an extensive list of recommended changes—mostly additions—to the

current form of the EBQ. Virtually every part of the EBQ would be altered by implementing these

recommendations. The result would be more useful information for adult literacy stakeholders. We will

describe the recommended changes section by section.

General and language background. First, as previously discussed, a question should be added

which asks the respondent, “What is your primary spoken language?” Phrasing the question in this manner

allows the respondent to indicate his/her oral language, rather than having the data user try to ascertain the

primary language. That is, currently, different data users can employ different language background items

(e.g., items #A-4, #A-8, #A-15) to determine respondents’ primary language—resulting in a lack of

consistency in target populations across studies. Second, the AIR focus groups recommended that more

information on language use and facility in other languages be obtained. This information was obtained in

the 1992 survey, but was not disseminated.
35

Educational background and experiences. More information is needed about respondents’

educational histories and attainments. This seems particularly critical given the obvious relationship

between education and literacy proficiencies. We recommend that an item that asks for the number of years

of formal schooling completed be added. The addition of this question captures all of the information

presently obtained by item #B-6, except for the “vocational, trade, or business option.” A separate question

could be posed to ascertain type of vocational, trade, or business degree, certification, or other credential

earned, if any

Because the NAAL 2002 will have substantial implications for educational policy as it pertains to

adult education, it seems particularly important to gather more data on adults’ participation in various basic

skills training programs. This recommendation was also made by the AIR focus groups. Specifically, we

recommend that, in addition to the current items, respondents’ estimates of the duration of their

participation in basic skills programs (i.e., number of weeks enrolled) be obtained. Also, was the basic skills

program completed? Because it is typical for people to enter, drop out, and re-enter these kinds of

programs, it is important to ascertain the number of times that the respondent has enrolled in basic skills

programs. For those adults who participated in one or more basic skills programs and completed the

programs, did they subsequently continue their education by enrolling in college or a vocational, trade, or

business school, or pursue any advanced form of occupational training? What other kinds of training and

credentials that prepare individuals for the workplace were earned, aside from educational diplomas and

certificates?

Participants in basic skills education typically, but not exclusively, lack a high school diploma.

People drop out of school at different ages and grades. An individual who drops out at 12th grade, for

example, may be better prepared to obtain the GED than a person who dropped out in 10th grade. This is

another reason for determining highest grade, or number of years of school, completed. In addition, to

capture more complete information about this population, a question could be added that asked whether one

obtained a GED and if so when. Finally, for those who have graduated from high school, how was their

degree earned? In other words, did they receive any special services while in school (e.g., special education)

or participate in talented and academic gifted programs?
36

Political and social participation. It is our recommendation that this section of the EBQ either be

(1) extensively expanded to capture more information about adults’ social participation as it relates to

literacy, or (2) truncated by deleting the four sub-scale items along with other sections of the EBQ. In our

estimation, only two items on this sub-scale pertain to “social participation”—TV viewing and library use.

Thus, the scale extensively taps into neither adults’ involvement in society nor their communities where

they might engage, more or less extensively, in literacy. Questions could be posed that ask more specifically

about the extent of respondents’ community involvement and social networks. Examples include:

“Have you ever:

-written a letter to the editor of your local newspaper?

-written a letter to, contacted by phone, or visited the office of your city mayor, state

representative, Congressional representative, Senator, or other elected or appointed

representative at any level of government?

-volunteered to work in a civic, charitable, political or social organization?

-been a member of a local PTA/PTO or other school organization?

-been a member of a local church or parish?”

 Our view is that items #C-1 (“how do you usually get information...”), #C-2 (“how many hours do

you usually watch television each day?”), and #C-3 (“how often do you use the services of a library...?”)

are literacy practices questions that could be moved to Section E (Literacy Activities and Collaboration).

Only item #C-4 (“have you voted in a national or state election...?) pertains to political participation. As

noted previously, the item could be revised to include voting in local elections. Some people vote in local

elections but not in state or national races; others vote only during national elections, but rarely for local

ones. The primary problem with making this change is that it impedes trend analysis from 1992 to 2002 in

regards to voting behavior, so these changes should be made only after careful consideration. We also

suggest that a second item pertaining to respondents’ voter registration status should be added. Rather than

having a section of the EBQ which contains only two voting status items, however, we recommend shifting

these questions to the Demographic Information section.
37

The AIR focus groups recommended the inclusion of items regarding civic participation, such as

how closely participants follow public affairs, and how they obtain this information. The assessment of

political participation could also be enhanced by asking if respondents have ever (1) held public office, (2)

volunteered to assist in a political campaign, or (3) been involved in any other forms of political activism.

Labor force participation. We recommend several additions to this section of the EBQ in order to

obtain data that will be useful to labor market economists, sociologists, and others. Based upon suggestions

by researchers who have utilized the labor force data extensively, we recommend the addition of a question

which asks respondents to provide their hourly wage rate (presently, this data can be determined by

responses to items #D-3 and #D-7, although data from items #D-4 and #D-5 may also need to be used to

determine hourly wage rate). Also, more questions about respondents’ work histories are important—what

and how many jobs have been held over the past five years? Which jobs were full-time work? Which ones

were part-time work? For part-time workers, is full-time work being sought?

Additionally, researchers interested in labor force participation would have liked to have had more

information on the work and wage histories of unemployed persons in the NALS sample. For unemployed

respondents, it is critical to know something about their most recent job, when they were last employed, the

type of occupation, and hourly wage rate. Several of these recommendations were made by the AIR focus

groups.

Literacy activities and collaboration. We recommend several additions and revisions to this

section of the EBQ. First, given the dramatic increase in technology availability and use for the average

citizen, questions should be included that gather information about the kinds of technology that respondents

are using (e.g., computers), how frequently (e.g., daily, a few times a week), and for what purposes (e.g.,

work, personal, entertainment). The Internet, for example, has become a major information and

entertainment medium in the years since the NALS data were collected and will become even more

significant in everyday life for millions of Americans by 2002. How has Internet use impacted adult

literacy? Who has access and how? These questions could be addressed by including the recommended

items.

The book reading practices item should be changed to capture not only categories of books read

but also the total number of books—either a grand total or within each category. The magazine reading item
38

should be changed to reflect reading on a weekly basis, which is more specific than asking how many are

read on a “regular” basis, which can have different meanings to people. Weekly reading is the same

benchmark for all respondents.

We recommend a cleaner distinction between reading for personal (or social) uses (item #E-5) and

for work (item #E-6). The print content choices (e.g., letters or memos, reports, articles, magazines, or

journals, etc.) are the same for both items. The kinds of reading done for work and for personal/social

reasons appear to be very different in most instances, however (Guthrie, Seifert, and Kirsch 1986; Kirsch

and Guthrie 1984). Allowing respondents to provide examples of their reading for both types of purposes

(i.e., personal/social, work) would be very informative. This is particularly true for the items that pertain to

writing activities (#E-7, #E-8). Only writing for personal use (e.g., writing letters, memos, forms, and

reports) was assessed in the NALS. We recommend expanding this item (#E-7) to include examples of

writing practices which are more typical of everyday life, such as making lists, taking phone messages,

keeping a diary, and summarizing or getting the gist of a lengthy document or article.

In fact, more information could—and needs to—be gathered about adults’ typical literacy

practices, including reading, writing, and arithmetic. This means not only assessing the frequency and type

of practices, but also learning about the challenge, or difficulty, of the literacy tasks that adults face, as well

as the criticality, or significance, of these literacy tasks and activities to their lives and work. Currently,

Statistics Canada is examining these issues, based upon what they have learned from the results of the

International Adult Literacy Survey (IALS) (S. Murray—personal communication, March, 1999). These

data could be gathered on a sub-sample using a diary approach.

Demographic information. Information regarding the occupations of the respondents’ parents is

needed. Parents’ occupational status clearly has a significant impact upon educational, social and job

opportunities and, at least indirectly, has some influence on literacy. This kind of data would have been very

useful to previous NALS-based research on the relationships among occupation, literacy, and wages. We

also suggest that one or more items pertaining to voting and voter registration status be included in this

section if the Political and Social Participation section is not expanded.
39

Alternative Approaches

Increasing the number of items contained within the EBQ will necessarily increase the time needed

to administer the survey. What can be done, then, to reduce the overall administration time for the literacy

assessment and background survey? Some experts have suggested revising the literacy assessment. Reder’s

(1998b) analyses have shown that the PDQ literacy proficiency scales are highly correlated with one

another. In essence, each scale taps into somewhat overlapping dimensions of the same basic ability. These

issues are, by now, well known to the NALS developers so we will not discuss them in detail here. It is

sufficient to note that Reder’s evidence “indicates that the dominant feature of the assessed [italics in

original] construct of literacy is a single general literacy capability” (52). Individual literacy items on the

NALS scales were shown to cut across all three categories; thus, as Reder’s analyses suggest, it should be

possible to assess and interpret literacy performance using a single scale.

There has been some suggestion that, for example, only the document proficiency scale be

employed in future NALS assessments. Individual items contained in the document scale may contain more

or less prose and/or quantitative tasks embedded within them, and these items could then be subjected to

further analyses to ascertain respondents’ prose and quantitative literacy performance, if so desired. Another

approach is to combine all three assessments into a single general literacy scale. This may be a reasonable

approach to reducing the administration time for the literacy assessment and thus allow for increased

administration time of a lengthier EBQ.

Sticht’s (1999) advocacy of a telephone-administered NALS-type literacy assessment also has

implications here. The advantage of a telephone assessment, among other benefits that are described by

Sticht, is the possibility of having multiple contacts with respondents. For example, a lengthier, more

detailed version of the EBQ could potentially be administered in several stages to the same respondents.

Multiple telephone contacts may have the effect of establishing greater rapport between the interviewer and

the respondent and thereby yield a stronger commitment to participation on the respondent’s part and may

result in more reliable data. Sticht has estimated an 80 percent cost savings for a national literacy

assessment through the telephone survey method. Even if multiple telephone contacts with respondents were
40

required to administer a more comprehensive EBQ, and respondents were paid for each interview, the costs

should still be well below that of the 1992 NALS.

We are also informed by the current longitudinal study of adult literacy being conducted by Steve

Reder. The longitudinal study is sponsored by the National Center for the Study of Adult Learning and

Literacy, at Harvard University, and is funded by NCES. The purpose of the longitudinal study is to “better

understand the process and consequences of adult literacy development over time” (S. Reder—personal

communication, March 8, 1998). Information is being gathered about two groups of adults in the Portland

(OR) greater metropolitan area who have low educational attainment—those who are enrolled in adult basic

education programs (e.g., GED) and those who are not. Participants in both groups are going to be followed

over three years and surveyed once each year. Participants in the former group are identified through the

various programs in which they participate (n=500); those in the latter group are identified through a

random-digit dialing telephone screening interview (n=500). In addition to a literacy skills assessment, a

very extensive background survey (“lifelong learning questionnaire”) is individually administered using a

computer-assisted personal interview script. Both the literacy assessment and background interview require

about two hours to complete (S. Reder—personal communication, April, 1999). Participants are paid thirty

dollars. There is some hope that this limited project will be the test ground for a national longitudinal study

of adult literacy.

The Lifelong Learning Questionnaire used by Reder’s research team obtains information on the

following topics:

• Educational background and school history;

• Employment status and work history;

• Contextual influences on learning;

• Language background;

• Learning motivations, contexts and activities;

• Participation in adult education;

• How information is obtained;

• Reasons for dropping out of adult education programs;

• Literacy practices at work, home, and school;
41

• Social networks and civic involvement; and,

• Demographic variables.

It is our recommendation that the NAAL 2002 development team look closely at the Lifelong Learning

Questionnaire and consider incorporating a number of the items contained in this survey consistent with the

above recommendations.

The National Center for Education Statistics conducted, in 1991 and 1995, a National Household

Education Survey (NHES), using random-digit dialing procedures. The 1995 survey consisted of two

components: an adult education component and an early childhood program participation component. The

adult education survey is relevant to our report, as the sampling and data collection methods are quite

similar between the NALS and the NHES:95. Interviews were completed with nearly 20,000 adults ages 16

and older who were not on active military duty for the NHES:95. This survey focused on the participation

of adults in a wide range of educational activities during the year prior to the interview. Respondents were

asked about their participation in seven types of adult education, including basic skills and GED classes,

college coursework and degrees, vocational training, apprenticeships, work-related courses, and other forms

of instruction. Also obtained were data regarding reasons for and barriers to participation in adult education.

A few literacy-relevant variables were also obtained (e.g., public library use).

Our past work with the NALS (see Smith and Sheehan 1996) indicates that variables obtained at

the different levels of sampling might be important to understanding some of the relationships among

literacy and other variables in the NALS. We have found that the intraclass correlations of the three literacy

scales are moderate at the segment level. Because the segment level of the sampling design constitutes

blocks or groups of census blocks, we have termed this segment-level variation “neighborhood effects” (see

Sheehan-Holt and Smith 1998). Our work has determined that the mean income of the neighborhood

accounts for variation in literacy skills among individuals—even beyond what is accounted for by family

income. Therefore, another recommendation which developers of the NAAL 2002 should consider is the

collection of data at higher levels of sampling design—particularly the segment level. For example, such

data might include the crime rate of the neighborhood, the percent of African Americans and Hispanics in

the neighborhood, whether there is either a library or other resources for literacy available in the

neighborhood, and the average family income and average education level of the neighborhood.
42

It may be worthwhile to consider obtaining more extensive literacy-relevant background data (e.g.,

literacy activities and collaboration) in future NCES adult education surveys. Perhaps the NAAL 2002

could be combined or integrated with a future adult education survey or vice versa—a change that was also

called for by the AIR focus groups. A sub-sample of the NAAL 2002 participants could also complete the

NHES adult education survey. Integrating the EBQ and the adult education surveys would result in a

lengthier EBQ which could be administered over the telephone, across multiple contacts, as suggested

above. Combining the two surveys achieves the purposes of both the NAAL and the NHES and can result in

substantial cost savings. Linking information from both surveys would enable researchers to study how

barriers to adult education may, for example, deter the acquisition or development of literacy among some

populations.

The AIR focus groups recommended several changes for the NAAL 2002, as we have previously

noted. Among their recommendations is that a mechanism should be used to “relate the 2002 survey results

to the demographic and dynamic changes in society” (Sherman, Condelli, and Koloski 1998, 6) and to

report the information in a broader context. It may not, however, be possible to implement this valuable

recommendation without the benefit of additional background data. We also note that the consensus of the

focus groups was to expand the EBQ to obtain more descriptive information about the sample, thereby

enabling more in-depth analyses by primary and secondary analysts.

Assessment Costs

Despite the recommendation—by the writers of this report and other adult literacy stakeholders

and experts—for the addition of numerous items to the EBQ, we recognize the need to have a parsimonious

set of background questionnaire items that will provide the most essential information about literacy. The

data must be sufficiently robust and detailed to allow strong inferences about adult literacy. There is a

delicate balance between managing the costs of a survey (which includes administrative and other

personnel, and data management and analysis costs) and obtaining good data. The advantages of adding or

deleting items and/or modifying existing items contained within the EBQ must be weighed against the

disadvantages (i.e., the potential costs—in terms of real dollars and/or the sacrificing of useful data).
43

The addition of items to the EBQ will increase the administration time for the survey and the

overall costs of data collection. Both interviewers and respondents must be paid for their time, among other

costs. If the administration time is increased for the EBQ, then it may be necessary to decrease the time

allocated for the literacy assessment to avoid imposing unreasonable demand upon respondents (which

would make it more difficult to obtain participants—thereby further increasing data collection costs).

Further discussion will be necessary to determine the best course of action in regards to the literacy

assessment strategy.

The primary advantage to changing the EBQ used in the 1992 NALS is that, potentially, more

useful data can be obtained as a result of these revisions. Any content changes to the EBQ may present

complex problems that impede trend analyses. Therefore, the recommendations that involve the addition of

new items are not as dramatic as those that call for some items to be revised. However, the addition of items

can have unintended effects. That is, the antecedent items may suggest a particular frame of reference for

subsequent questions and, therefore, responses to these latter questions may be influenced because the

context has been changed. These changes, if implemented, will present a particular challenge to the EBQ

developers for the NAAL 2002.

It is very important for researchers and policymakers to have an expanded EBQ from which to

draw data. Several researchers have expressed to us that was (and remains) frustrating to have such a

limited EBQ and—at the same time—a very good assessment of literacy proficiencies. Because a great

amount of effort, time, and dollars have been committed to developing a quantitative measure of literacy,

we believe that it will be beneficial to all literacy stakeholders to have an EBQ that will allow researchers to

conduct more thorough investigations of the many factors that are important to adult literacy.
44

References

Anderson, R.C., Wilson, P.T., and Fielding, L.G. (1988). Growth in Reading and How Children Spend

Their Time Outside of School. Reading Research Quarterly, 23: 285–303.

Breneman, D.W., and Harlow, W.N. (1999). Remediation in Higher Education: A Symposium Featuring

“Remedial Education: Costs and Consequences” (Eric Document No. 422770). Washington, DC:

Thomas B. Fordham Foundation.

Finn, J.D., and Gerber, S.B. (1998). Work, School, and Literacy. In M C. Smith (Ed.), Literacy for the

Twenty-First Century: Research, Policy, Practices and the National Adult Literacy Survey (pp.

61–78). Westport, CT: Praeger.

Friedman, L., and Davenport, E. (1998). Literacy Gender Gaps: Evidence from the National Adult Literacy

Survey. In M C. Smith (Ed.), Literacy for the Twenty-First Century: Research, Policy, Practices

and the National Adult Literacy Survey (pp. 95–108). Westport, CT: Praeger.

Gerber, S., and Finn, J.D. (1998). Learning Document Skills at School and at Work. Journal of Adolescent

and Adult Literacy, 42: 32–44.

Greenberg, E.J., Swaim, P.L., and Teixeira, R.A. (1995). Workers with Higher Literacy Skills not as Well

Rewarded in Rural Areas. Rural Development Perspectives, 10: 45–52.

Guthrie, J.T., Seifert, M., and Kirsch, I.S. (1986). Effects of Education, Occupation, and Setting on Reading

Practices. American Educational Research Journal, 23: 151–160.

Howard, J., and Obetz, W.S. (1996). Using the NALS to Characterize the Literacy of Community College

Graduates. Journal of Adolescent and Adult Literacy, 39: 462–467.

Howard, J., and Obetz, W.S. (1998). Community College Literacy: Is the Middle Right? In M C. Smith

(Ed.), Literacy for the Twenty-first Century: Research, Policy, Practices and the National Adult

Literacy Survey (pp.125–138). Westport, CT: Praeger.

Kirsch, I.S., and Guthrie, J.T. (1984). Adult Reading Practices for Work and Leisure. Adult Education

Quarterly, 34: 213–232.

Kirsch, I.S., and Jungeblut, A. (1986). Literacy: Profiles of America’s young Adults (NAEP Report No. 16–

PL–02). Princeton, NJ: Educational Testing Service.
45

Kirsch, I.S., Jungeblut, A., Jenkins, L., and Kolstad, A. (1993). Adult Literacy in America: A First Look at

the Results of the National Adult Literacy Survey (NCES 93–275). U.S. Department of Education.

Washington, DC: U.S. Government Printing Office.

Mislevy, R.J., Johnson, E.G., and Muraki, E. (1992). Scaling Procedures in NAEP. Journal of Educational

Statistics, 17: 131–154.

Pryor, F.L., and Schaffer, D. (1997). Wages and the University Educated: A Paradox Resolved. Monthly

Labor Review, 120(7): 3–15.

Pryor, F.L., and Schaffer, D. (1999). Who’s Working and Why. New York, NY: Cambridge University

Press.

Raudenbush, S.W., and Kasim, R.M. (1998). Cognitive Skill and Economic Inequality: Findings from the

National Adult Literacy Survey. Harvard Education Review, 68, 33–79.

Reder, S. (1995). Literacy, Education, and Learning Disabilities. Portland, OR: Northwest Regional

Educational Laboratory.

Reder, S. (1998). Introduction: Adult Literacy Research and the National Adult Literacy Survey. In M C.

Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy, Practices and the National

Adult Literacy Survey (pp. 3–11). Westport, CT: Praeger.

Reder, S. (1998a). Literacy, Education and Learning Disabilities. Philadelphia, PA: National Center for

Adult Literacy.

Reder, S. (1998b). Literacy Selection and Literacy Development: Structural Equation Models of the

Reciprocal Effects of Education and Literacy. In M C. Smith (Ed.), Literacy for the Twenty-first

Century: Research, Policy, Practices and the National Adult Literacy Survey (pp. 139–158).

Westport, CT: Praeger.

Sheehan-Holt, J.K., and Smith, M C. (in press). Does Basic Skills Education Affect Adults’ Literacy

Proficiencies and Reading Practices? Reading Research Quarterly.

Sherman, R., Condelli, L., and Koloski, J. (1998). Developing the National Assessment of Adult Literacy:

Recommendations from Stakeholders (NCES 98–17). U.S. Department of Education. Washington,

DC: NCES Working Paper.

Smith, M.C. (in press). The Real-World Reading Practices of Adults. Journal of Literacy Research.
46

Smith, M C (Ed.). (1998). Literacy for the Twenty-first century: Research, Policy, Practices and the

National Adult Literacy Survey. Westport, CT: Praeger.

Smith, M C. (1996). Differences in Adults’ Reading Practices and Literacy Proficiencies. Reading Research

Quarterly, 31: 196–219.

Smith, M C., and Smith, M C., and Sheehan, J.K. (1996, April). Racial and Ethnic Group Differences in

Adults’ Reading Practices. Paper presented at the annual meeting of the American Educational

Research Association, New York, NY.

Smith, M C., and Sheehan, J.K. (1998). Adults’ Reading Practices and Their Associations with Literacy

Proficiencies. In M C. Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy,

Practices and the National Adult Literacy Survey (pp. 79–93). Westport, CT: Praeger.

Solozano, R.W. (1994). Conducting a Spanish National Adult Literacy Survey: A Discussion of the Issues

(ED No. 370 451). U.S. Department of Education. Washington, DC: U.S. Government Printing

Office.

Sticht, T.G. (in press). Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-door

Surveys in the Assessment of Adult Literacy. U.S. Department of Education. Washington, DC:

NCES Working Paper.

Sticht, T.G., and Armstrong, W.B. (1994). Adult Literacy in America: A Compendium of Data and

Interpretive Comments. San Diego, CA: Applied Behavioral and Cognitive Sciences, Inc., and San

Diego Community College District.

Venezky, R., Kaplan, D., and Yu, F. (1998). Literacy Practices and Voting Behavior: An Analysis of the

1992 National Adult Literacy Survey. U.S. Department of Education. Washington, DC: U.S.

Government Printing Office.

Venezky, R.L., and Kaplan, D. (1998). Literacy Habits and Political Participation. In M C. Smith (Ed.),

Literacy for the Twenty-first Century: Research, Policy, Practices and the National Adult Literacy

Survey (pp.109–124). Westport, CT: Praeger.

Vogel, S.A., and Reder, S. (1998a). Literacy Proficiency Among Adults with Self-reported Learning

Disabilities. In M C. Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy,

Practices and the National Adult Literacy Survey (pp. 159–174). Westport, CT: Praeger.
47

Vogel, S.A., and Reder, S. (1998b). Educational Attainment of Adults with Learning Disabilities. In S.A.

Vogel and S. Reder (Eds.), Learning Disabilities, Literacy, and Adult Education (pp. 43–68).

Baltimore, MD: Brooks Publishing.
48

Attachments

A. Initial Contact E-mail Survey

B. NALS Background Survey Data Users Questionnaire

C. Studies Examined for Content Review
49

A. Initial Contact E-mail Survey

Dear NALS Data User:

Your name appears on a list of academic researchers, public policy experts, and other professionals who are

users of the 1992 National Adult Literacy Survey (NALS) data

[http://nces.ed.gov/nadlits/naal92/users.html]. We are currently preparing a technical report for the

Education Statistics Services Institute (ESSI), evaluating the uses of background questionnaires in survey

studies such as the NALS, with recommendations for changes in the background questionnaire for the

National Assessment of Adult Literacy planned for 2002. As part of this effort, we would like to ask for

your assistance.

The NALS background survey consisted of six “content areas” that gathered information on respondents’:

(1) language background; (2) educational attainment and experiences; (3) political and social participation

(e.g., voting); (4) labor force participation; (5) literacy activities and collaboration with others; and (6)

demographic information.

We would like to know three things:

First, have you conducted any analyses of the NALS data, or in any way reported the data from each or all

of the six content areas contained in the background survey?

Second, if you answered “yes” to the previous question, which among the six content areas do you consider

to be your area of expertise? (You may indicate one or more areas). For example, if you have analyzed

relationships among educational attainment and literacy proficiencies, indicate “educational attainment” as

your area of expertise.

Third, would you be willing to respond to a brief, follow-up e-mail or telephone survey? Please indicate

your preference for either e-mail or telephone follow-up.

Your responses to this brief survey, and the follow-up survey, are and will remain anonymous.

Thank you for your assistance.
50

M Cecil Smith, Ph.D. Janet K. Sheehan-Holt, Ph.D.

Associate Professor Assistant Professor

Department of Educational Psychology Department of Educational Psychology

Northern Illinois University Northern Illinois University

DeKalb, IL 60115 DeKalb, IL 60115

(815) 753-8448 (815) 753-8523

mcsmith@niu.edu jsheehan@niu.edu
51

B. NALS Background Survey Data Users Questionnaire

You may recall that we recently contacted you and asked you to respond to our brief “pre-survey”

pertaining to your use of data from the National Adult Literacy Survey. At that time, we indicated that we

wished to follow-up with a somewhat more extensive survey regarding your uses of the Background Survey

from the NALS. We are preparing a technical report for the Education Statistics Services Institute in their

preparation for the planned 2002 National Assessment of Adult Literacy. Results from the survey below

will be incorporated into our report and used to make recommendations for the development of the National

Assessment of Adult Literacy. We appreciate your taking time to respond to these questions.

Where relevant, please indicate the specific sections of the NALS Background Questionnaire you are

commenting on. The six sections are:

Section A: General and Language Background

Section B: Educational Background and Experiences

Section C: Political and Social Participation

Section D: Labor Force Participation

Section E: Literacy Activities and Collaboration

Section F: Demographic Information

1.) What were the specific research questions in your study(ies) of the 1992 NALS?

2.) How did the 1992 NALS Background Questionnaire items enhance or strengthen your ability to

draw inferences and/or make policy recommendations?

3.) (a) How did the 1992 NALS Background Questionnaire items limit your ability to draw inferences

and/or make policy recommendations? b.) How were you able to rectify these problems? Please explain as

precisely as possible.

4.) (a) What questions should be added to any or all of the sections of the 2002 National Assessment

of Adult Literacy survey questionnaire, and (b) what existing items from the 1992 NALS Background
52

Questionnaire should be revised and included in the 2002 survey? (c) How would these additions and

revisions allow you or other researchers to make strong inferences from the obtained data?

5.) (a) What topic areas, aside from those indicated by the six Sections in the 1992 NALS Background

Questionnaire, do you think are important to examine for the 2002 National Assessment of Adult Literacy

survey? (b) How would information on these topics be useful to you and other researchers and/or policy

makers?

6.) Which questions and/or sections do you think are least useful and/or informative and should

therefore be dropped from the 2002 National Assessment of Adult Literacy survey?

7.) Please list all papers and research reports you have published that pertain to your

analyses of the 1992 NALS data.

Thank you for your response to this survey. Your responses to the questions are, and will remain,

confidential.

Best Regards,

M Cecil Smith

Janet K. Sheehan-Holt
53

C. Studies Examined for Content Review

Friedman, L., and Davenport, E. (1998). Literacy Gender Gaps: Evidence from the National Adult Literacy

Survey. In M C. Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy, Practices

and the National Adult Literacy Survey (pp. 95–108). Westport, CT: Praeger.

Gerber, S., and Finn, J.D. (1998). Learning Document Skills at School and at Work. Journal of Adolescent

and Adult Literacy, 42: 32–44.

Greenberg, E.J., Swaim, P.L., and Teixeira, R.A. (1995). Workers with Higher Literacy Skills not as Well

Rewarded in Rural Areas. Rural Development Perspectives, 10: 45–52.

Howard, J., and Obetz, W.S. (1996). Using the NALS to Characterize the Literacy of Community College

Graduates. Journal of Adolescent and Adult Literacy, 39: 462–467.

Pryor, F.L., and Schaffer, D. (1997). Wages and the University Educated: A Paradox Resolved. Monthly

Labor Review, 120(7): 3–15.

Raudenbush, S.W., and Kasim, R.M. (1998). Cognitive Skill and Economic Inequality: Findings from the

National Adult Literacy Survey. Harvard Education Review, 68: 33–79.

Reder, S. (1998a). Literacy, Education and Learning Disabilities. Philadelphia, PA: National Center on

Adult Literacy.

Reder, S. (1998b). Literacy Selection and Literacy Development: Structural Equation Models of the

Reciprocal Effects of Education and Literacy. In M C. Smith (Ed.), Literacy for the Twenty-First

Century: Research, Policy, Practices and the National Adult Literacy Survey (pp. 139–158).

Westport, CT: Praeger.

Sheehan-Holt, J.K., and Smith, M C. (in press). Does Basic Skills Education Affect Adults’ Literacy

Proficiencies and Reading Practices? Reading Research Quarterly.

Smith, M C. (1996). Differences in Adults’ Reading Practices and Literacy Proficiencies. Reading Research

Quarterly, 31: 196–219.

Smith, M C., and Sheehan, J.K. (1998). Adults’ Reading Practices and Their Associations with Literacy

Proficiencies. In M C. Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy,

Practices and the National Adult Literacy Survey (pp. 79–93). Westport, CT: Praeger.
54

Venezky, R., Kaplan, D., and Yu, F. (1998). Literacy Practices and Voting Behavior: An Analysis of the

1992 National Adult Literacy Survey.U.S. Department of Education. Washington,

DC: U.S. Government Printing Office.

Vogel, S.A., and Reder, S. (1998a). Literacy Proficiency Among Adults with Self-reported Learning

Disabilities. In M C. Smith (Ed.), Literacy for the Twenty-first Century: Research, Policy,

Practices and the National Adult Literacy Survey (pp. 159–174). Westport, CT: Praeger.

Vogel, S.A., and Reder, S. (1998b). Educational Attainment of Adults with Learning Disabilities. In S.A.

Vogel and S. Reder (Eds.), Learning Disabilities, Literacy, and Adult Education (pp. 43–68.).

Baltimore, MD: Brooks Publishing.
Listing of NCES Working Papers to Date

Working papers can be downloaded as pdf files from the NCES Electronic Catalog

(http://nces.ed.gov/pubsearch/). You can also contact Sheilah Jupiter at (202) 502-7444

(sheilah_jupiter@ed.gov) if you are interested in any of the following papers.

Listing of NCES Working Papers by Program Area

No. Title NCES contact

Baccalaureate and Beyond (B&B)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Beginning Postsecondary Students (BPS) Longitudinal Study

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Common Core of Data (CCD)

95-12 Rural Education Data User’s Guide Samuel Peng

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Decennial Census School District Project

95-12 Rural Education Data User’s Guide Samuel Peng

96-04 Census Mapping Project/School District Data Book Tai Phan

98-07 Decennial Census School District Project Planning Report Tai Phan

Early Childhood Longitudinal Study (ECLS)

96-08 How Accurate are Teacher Judgments of Students’ Academic Performance? Jerry West

96-18 Assessment of Social Competence, Adaptive Behaviors, and Approaches to Learning with

Young Children

Jerry West

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Education Finance Statistics Center (EDFIN)

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

High School and Beyond (HS&B)

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

HS Transcript Studies

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson
No. Title NCES contact

International Adult Literacy Survey (IALS)

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Integrated Postsecondary Education Data System (IPEDS)

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

National Assessment of Adult Literacy (NAAL)

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door

Surveys in the Assessment of Adult Literacy

Sheida White

2000-07 “How Much Literacy is Enough?” Issues in Defining and Reporting Performance

Standards for the National Assessment of Adult Literacy

Sheida White

2000-08 Evaluation of the 1992 NALS Background Survey Questionnaire: An Analysis of Uses

with Recommendations for Revisions

Sheida White

National Assessment of Educational Progress (NAEP)

95-12 Rural Education Data User’s Guide Samuel Peng

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Steven Gorman

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Steven Gorman

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Steven Gorman

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questionnaires)

Steven Gorman

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Steven Gorman

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

National Education Longitudinal Study of 1988 (NELS:88)

95-04 National Education Longitudinal Study of 1988: Second Follow-up Questionnaire Content

Areas and Research Issues

Jeffrey Owings

95-05 National Education Longitudinal Study of 1988: Conducting Trend Analyses of NLS-72,

HS&B, and NELS:88 Seniors

Jeffrey Owings

95-06 National Education Longitudinal Study of 1988: Conducting Cross-Cohort Comparisons

Using HS&B, NAEP, and NELS:88 Academic Transcript Data

Jeffrey Owings

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings
No. Title NCES contact

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

National Household Education Survey (NHES)

95-12 Rural Education Data User’s Guide Samuel Peng

96-13 Estimation of Response Bias in the NHES:95 Adult Education Survey Steven Kaufman

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-21 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School

Readiness, and School Safety and Discipline

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

96-29 Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the

1995 National Household Education Survey (NHES:95)

Kathryn Chandler

96-30 Comparison of Estimates from the 1995 National Household Education Survey

(NHES:95)

Kathryn Chandler

97-02 Telephone Coverage Bias and Recorded Interviews in the 1993 National Household

Education Survey (NHES:93)

Kathryn Chandler

97-03 1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener,

NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education

Kathryn Chandler

97-04 Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in

the 1993 National Household Education Survey (NHES:93)

Kathryn Chandler

97-05 Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National

Household Education Survey (NHES:93)

Kathryn Chandler

97-06 Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National

Household Education Survey (NHES:95)

Kathryn Chandler

97-08 Design, Data Collection, Interview Timing, and Data Editing in the 1995 National

Household Education Survey

Kathryn Chandler

97-19 National Household Education Survey of 1995: Adult Education Course Coding Manual Peter Stowe

97-20 National Household Education Survey of 1995: Adult Education Course Code Merge

Files User’s Guide

Peter Stowe

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

97-28 Comparison of Estimates in the 1996 National Household Education Survey Kathryn Chandler

97-34 Comparison of Estimates from the 1993 National Household Education Survey Kathryn Chandler

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

97-38 Reinterview Results for the Parent and Youth Components of the 1996 National

Household Education Survey

Kathryn Chandler

97-39 Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996

National Household Education Survey

Kathryn Chandler

97-40 Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996

National Household Education Survey

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

National Longitudinal Study of the High School Class of 1972 (NLS-72)

95-12 Rural Education Data User’s Guide Samuel Peng
No. Title NCES contact

National Postsecondary Student Aid Study (NPSAS)

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio

National Study of Postsecondary Faculty (NSOPF)

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Private School Universe Survey (PSS)

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

96-26 Improving the Coverage of Private Elementary-Secondary Schools Steven Kaufman

96-27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 Steven Kaufman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Recent College Graduates (RCG)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Schools and Staffing Survey (SASS)

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American

Statistical Association

Dan Kasprzyk

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) Dan Kasprzyk

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report Dan Kasprzyk

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher

Transcript Study, Schools and Staffing Survey

Dan Kasprzyk

94-06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related

Surveys

Dan Kasprzyk

95-01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American

Statistical Association

Dan Kasprzyk

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing

QED School Estimates with CCD Estimates

Dan Kasprzyk

95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis Dan Kasprzyk

95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates Dan Kasprzyk

95-09 The Results of the 1993 Teacher List Validation Study (TLVS) Dan Kasprzyk

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive

Reconciliation

Dan Kasprzyk

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and

Their Applicability for the Teacher Follow-up Survey

Sharon Bobbitt

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and

Staffing Survey

Dan Kasprzyk

96-01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly

Longitudinal Study

Dan Kasprzyk

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting

of the American Statistical Association

Dan Kasprzyk

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey Dan Kasprzyk

96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to

Inform Broad Education Policy

Dan Kasprzyk
No. Title NCES contact

96-07 Should SASS Measure Instructional Processes and Teacher Effectiveness? Dan Kasprzyk

96-09 Making Data Relevant for Policy Discussions: Redesigning the School Administrator

Questionnaire for the 1998-99 SASS

Dan Kasprzyk

96-10 1998-99 Schools and Staffing Survey: Issues Related to Survey Depth Dan Kasprzyk

96-11 Towards an Organizational Database on America’s Schools: A Proposal for the Future of

SASS, with comments on School Reform, Governance, and Finance

Dan Kasprzyk

96-12 Predictors of Retention, Transfer, and Attrition of Special and General Education

Teachers: Data from the 1989 Teacher Followup Survey

Dan Kasprzyk

96-15 Nested Structures: District-Level Data in the Schools and Staffing Survey Dan Kasprzyk

96-23 Linking Student Data to SASS: Why, When, How Dan Kasprzyk

96-24 National Assessments of Teacher Quality Dan Kasprzyk

96-25 Measures of Inservice Professional Development: Suggested Items for the 1998-1999

Schools and Staffing Survey

Dan Kasprzyk

96-28 Student Learning, Teaching Quality, and Professional Development: Theoretical

Linkages, Current Measurement, and Recommendations for Future Data Collection

Mary Rollefson

97-01 Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the

American Statistical Association

Dan Kasprzyk

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

97-10 Report of Cognitive Research on the Public and Private School Teacher Questionnaires

for the Schools and Staffing Survey 1993-94 School Year

Dan Kasprzyk

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-12 Measuring School Reform: Recommendations for Future SASS Data Collection Mary Rollefson

97-14 Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and

Analysis

Steven Kaufman

97-18 Improving the Mail Return Rates of SASS Surveys: A Review of the Literature Steven Kaufman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

97-23 Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing

Form

Dan Kasprzyk

97-41 Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting

of the American Statistical Association

Steve Kaufman

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

98-05 SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors

Steven Kaufman

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

98-12 A Bootstrap Variance Estimator for Systematic PPS Sampling Steven Kaufman

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

98-14 Variance Estimation of Imputed Survey Data Steven Kaufman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest

Results to Improve Item Construction

Dan Kasprzyk

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

1999-12 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use

Codebook

Kerry Gruber

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley
No. Title NCES contact

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk
Listing of NCES Working Papers by Subject

No. Title NCES contact

Adult education

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Adult literacy—see Literacy of adults

American Indian – education

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

Assessment/achievement

95-12 Rural Education Data User’s Guide Samuel Peng

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Larry Ogle

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Larry Ogle

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Larry Ogle

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questions)

Larry Ogle

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Larry Ogle

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Beginning students in postsecondary education

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Civic participation

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Climate of schools

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

Cost of education indices

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Course-taking

95-12 Rural Education Data User’s Guide Samuel Peng
No. Title NCES contact

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

Crime

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Curriculum

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Customer service

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Data quality

97-13 Improving Data Quality in NCES: Database-to-Report Process Susan Ahmed

Data warehouse

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Design effects

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Dropout rates, high school

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

Early childhood education

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Educational attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Educational research

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

Employment

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico
No. Title NCES contact

Faculty – higher education

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Finance – elementary and secondary schools

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Finance – postsecondary

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

Finance – private schools

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

Geography

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

Imputation

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Inflation

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

Institution data

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Instructional resources and practices

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

International comparisons

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-16 International Education Expenditure Comparability Study: Final Report, Volume I Shelley Burns

97-17 International Education Expenditure Comparability Study: Final Report, Volume II,

Quantitative Analysis of Expenditure Comparability

Shelley Burns

Libraries

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers

Presented at Meetings of the American Statistical Association

Carrol Kindel

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Limited English Proficiency

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser
No. Title NCES contact

Literacy of adults

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door

Surveys in the Assessment of Adult Literacy

Sheida White

2000-07 “How Much Literacy is Enough?” Issues in Defining and Reporting Performance

Standards for the National Assessment of Adult Literacy

Sheida White

2000-08 Evaluation of the 1992 NALS Background Survey Questionnaire: An Analysis of Uses

with Recommendations for Revisions

Sheida White

Literacy of adults – international

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Mathematics

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

Parental involvement in education

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Participation rates

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

Postsecondary education

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Postsecondary education – persistence and attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Postsecondary education – staff

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler
No. Title NCES contact

Private schools

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

Projections of education statistics

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Public school finance

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Public schools

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

Public schools – secondary

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Reform, educational

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

Response rates

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

School districts, public

98-07 Decennial Census School District Project Planning Report Tai Phan

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

School districts, public – demographics of

96-04 Census Mapping Project/School District Data Book Tai Phan

Schools

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Schools – safety and discipline

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Software evaluation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Staff

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk
No. Title NCES contact

Staff – higher education institutions

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

State

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Statistical methodology

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But

Thought You Could Never Understand

Susan Ahmed

Students with disabilities

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

Survey methodology

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Teachers

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

Teachers – instructional practices of

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – opinions regarding safety

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – performance evaluations

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – qualifications of

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – salaries of

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Variance estimation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Violence

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman
No. Title NCES contact

Vocational education

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson