NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

The Working Paper Series was initiated to promote the sharing of the

valuable work experience and knowledge reflected in these preliminary

reports. These reports are viewed as works in progress, and have not

undergone a rigorous review for consistency with NCES Statistical

Standards prior to inclusion in the Working Paper Series.

U. S. Department of Education

Office of Educational Research and Improvement
NATIONAL CENTER FOR EDUCATION STATISTICS

Working Paper Series

Using Telephone and Mail Surveys as a

Supplement or Alternative to Door-to-Door

Surveys in the Assessment of Adult Literacy

Working Paper No. 2000-06 March 2000

Contact: Sheida White

Assessment Division

E-mail: sheida_white@ed.gov

U. S. Department of Education

Office of Educational Research and Improvement
U.S. Department of Education

Richard W. Riley

Secretary

Office of Educational Research and Improvement

C. Kent McGuire

Assistant Secretary

National Center for Education Statistics

Gary W. Phillips

Acting Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing,

and reporting data related to education in the United States and other nations. It fulfills a congressional

mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in

the United States; conduct and publish reports and specialized analyses of the meaning and significance of

such statistics; assist state and local education agencies in improving their statistical systems; and review

and report on education activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent, reliable,

complete, and accurate indicators of education status and trends; and report timely, useful, and high quality

data to the U.S. Department of Education, the Congress, the states, other education policymakers,

practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is appropriate to a

variety of audiences. You, as our customer, are the best judge of our success in communicating information

effectively. If you have any comments or suggestions about this or any other NCES product or report, we

would like to hear from you. Please direct your comments to:

National Center for Education Statistics

Office of Educational Research and Improvement

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

The NCES World Wide Web Home Page is

http://nces.ed.gov

Suggested Citation

U.S. Department of Education. National Center for Education Statistics. Using Telephone and Mail

Surveys as a Supplement or Alternative to Door-to-Door Surveys in the Assessment of Adult Literacy,

Working Paper No. 2000-06, by Thomas G. Sticht. Project Officer, Sheida White. Washington, DC:

2000.

March 2000
iii

Foreword

In addition to official NCES publications, NCES staff and individuals commissioned by

NCES produce preliminary research reports that include analyses of survey results, and

presentations of technical, methodological, and statistical evaluation issues.

The Working Paper Series was initiated to promote the sharing of the valuable work

experience and knowledge reflected in these preliminary reports. These reports are viewed as

works in progress, and have not undergone a rigorous review for consistency with NCES

Statistical Standards prior to inclusion in the Working Paper Series.

Copies of Working Papers can be downloaded as pdf files from the NCES Electronic

Catalog (http://nces.ed.gov/pubsearch/), or contact Sheilah Jupiter at (202) 502-7444,

e-mail: sheilah_jupiter@ed.gov, or mail: U.S. Department of Education, Office of Educational

Research and Improvement, National Center for Education Statistics, 1990 K Street, NW, Room

9048, Washington, DC 20006.

Marilyn M. McMillen Ralph Lee

Chief Mathematical Statistician Mathematical Statistician

Statistical Standards Program Statistical Standards Program
This page intentionally left blank.
Using Telephone and Mail Surveys as a Supplement or Alternative

to Door-to-Door Surveys in the Assessment of Adult Literacy

Prepared by:

Thomas G. Sticht

Applied Behavioral and Cognitive Sciences, Inc.

Prepared for:

U.S. Department of Education

Office of Educational Research and Improvement

National Center for Education Statistics

March 2000

This project was an activity of the Education Statistics Services Institute.
This page intentionally left blank.
vii

Table of Contents

Foreword ........................................................................................................................................iii

Abstract ........................................................................................................................................viii

Introduction .....................................................................................................................................1

The Search for Lower Cost Methods for Representing Adult Literacy .......................................1

Purpose and Plan of the Present Paper......................................................................................2

Part 1. Issues of Validity in Adult Literacy Assessment .................................................................4

Some Issues of Validity in the NALS and IALS ...........................................................................6

Part 2. Review of Related Research ................................................................................................9

Assessing Specific and General -’g’- Cognitive Abilities Using Telephone and Mail Surveys..9

Testing Word Knowledge by Telephone to Estimate General Cognitive Aptitude Using

Computer Adaptive Testing ...................................................................................................10

Assessing Adult Literacy-Based Knowledge and Literacy Practices by Telephone .................11

Part 3. A Cognitive Theory of Literacy.........................................................................................16

The Human Cognitive System....................................................................................................17

A Developmental Model of Literacy..........................................................................................19

Literacy and the Merging of Speech and Graphics Technologies ............................................23

Major Features of Graphics Technology ..................................................................................25

The Graphic Representation of Spoken Language....................................................................26

Part 4. Issues in the Use of Telephone and Mail Surveys in Assessing Adult Literacy...............28

Sampling and Other Issues........................................................................................................33

References .....................................................................................................................................35

Appendix .......................................................................................................................................41
viii

Abstract

This paper discusses the use of telephone surveys as a substitute for, or addition to, the use of door-to-door

surveys in assessing adult literacy ability. Part 1 addresses issues of validity in adult literacy assessment. Part 2 is a

review of related research that has used telephone and mail surveys to assess the cognitive abilities of adults.

Part 3 discusses the theoretical model of literacy that the National Adult Literacy Survey (NALS) and International

Adult Literacy Survey (IALS) have used and how that model can be modified to incorporate a developmental and

componential model of literacy that also includes the oral language aspects of literacy given in the Adult Education

and Family Literacy Act of 1998. Finally, Part 4 addresses several issues about the use of telephone and mail

surveys, including concerns about using these survey methods in such a way as to make continuity in trend data from

previous surveys of adult literacy possible. 
1

Introduction

National assessments of adult literacy have been valued by adult educators because they serve the generally

useful purpose of keeping the issue of adult literacy visible to policymakers, business and industry leaders, labor

unions, educators and adults themselves, who may be stimulated to seek additional literacy development (Fagan

1994).

Recently, however, the assessment of adult literacy in industrial nations has taken on a new significance

(The Organization for Economic Co-operation and Development (OECD), 1995). The OECD notes that, in regard to

concerns for economic competitiveness among member nations, “... one area that is receiving growing attention from

educational policymakers and analysts in a number of OECD countries is the direct measurement of literacy levels in

the labor force of industrialized countries” (Benton and Noyelle 1992, 11). Indeed, results of the first International

Adult Literacy Survey (IALS) involving some ten member nations of the OECD have recently been reported (OECD

1995, 1997).

Despite their usefulness to adult educators and policymakers, the cost involved in conducting adult literacy

surveys is a matter of some concern to these same groups. In the United States, the National Adult Literacy Survey

(NALS) (Kirsch, Jungeblut, Jenkins, and Kolstad 1993), conducted by the National Center for Education Statistics

(NCES) of the U.S. Department of Education cost over $14 million for the national and state samples involved (A.

Kolstad—personal communication, April, 1994). When asked at a planning meeting for the National Assessment of

Adult Literacy (NAAL), state directors of adult literacy education who were present reported that “...it would be

more difficult to find the funds for another round” (U.S. Department of Education 1998, 5).

The Search for Lower Cost Methods for Representing Adult Literacy

The NALS of 1992 assessed adult literacy by personal interviews using door-to-door sampling. Interviewers

asked adults to perform tasks such as filling in a bank check, locating times for travel on a train schedule, calculating

the cost of a meal in a restaurant, and other types of “tasks that simulate the literacy demands that adults encounter in

their daily lives” (Campbell, Kirsch, and Kolstad 1992, 12).
2

To avoid the costs of door-to-door assessments, the U.S. Department of Education has also used statistical

regression techniques, based on data from the National Adult Literacy Survey, to estimate the literacy levels of adults

in states that did not participate in the national survey (Reder 1994(a)). In Canada, too, concerns about the cost of

administering direct assessments of adult literacy led the Department of the Secretary of State in Ottawa to explore

the use of proxy measures of adult functional literacy to predict the direct assessment data (Neice, Adsett, and

Rodney 1992).

Statistical techniques and proxies for literacy assessment are of limited educational use when they include

variables such as ethnicity and gender (Reder 1994(a)) or learning disabilities and mother’s educational level (Neice,

Adsett, and Rodney 1992) in estimating people’s literacy skills. These variables are not amenable to change by

education or training and hence recommendations for action in the form of educational interventions are not feasible.

In a planning meeting for the NAAL, to be conducted in 2002, it was suggested that telephone interviews

might be considered as a cost-reducing method for characterizing aspects of adult literacy (U.S. Department of

Education 1998, 9, 17). Presently, the National Center for Education Statistics conducts biennial telephone surveys

of adults to determine their participation in adult education, the types of education they engage in, and so forth. So

the requisite sampling and telephone survey methodologies are familiar to NCES and it is well recognized that

telephone survey methods are much less costly than door-to-door surveys (Chandler 1994). However, there was

disagreement at the NAAL planning meeting about whether telephone interviews are a viable means for identifying

whether people can “read, write, and comprehend” (17).

Purpose and Plan of the Present Paper

One outcome of the planning meetings for the 2002 NAAL was the suggestion that a paper be

commissioned to discuss the feasibility of using telephone surveys as a substitute for or addition to the use of doorto-door surveys in assessing adult literacy ability (U.S. Department of Education 1998, 9). That is the primary

purpose of the present paper.

The paper includes the consideration of the use of mail surveys along with telephone surveys as methods

that are lower in cost than door-to-door interviewing for assessing adult literacy abilities.
3

Part 1. Issues of Validity in Adult Literacy Assessment. The first part of the paper addresses issues of validity

that have arisen in discussions of national assessments of literacy. This includes a discussion of construct or

inferential validity and the role of convergent and discriminant evidence in establishing the validity of a theory of

literacy. It also includes a discussion of the concept of action or use validity (Messick 1989). This discussion of

validity sets the stage for a later discussion in Part 4 of the paper, which includes a comparison between a telephone

survey approach and the NALS door-to-door approach in producing informational products deemed useful by

policymakers who have supported these large-scale assessments.

Part 2. Review of Related Research. The second segment of the paper provides a review of published studies that

have used telephone and mail surveys to assess the cognitive abilities of children or adults. The research reviewed

includes the assessment of “intelligence” (“g”) in studies of behavioral genetics, verbal “aptitude” in studies of

personnel testing for selection of qualified applicants for military service, and two studies of adult literacy using

telephone and mail surveys.

Part 3. A Cognitive Theory of Literacy. The third section of the paper includes a discussion of the theoretical

model of literacy that the NALS and International Adult Literacy Survey (IALS) have used. It discusses how that

model can be modified to incorporate a developmental and componential model of literacy that also includes the oral

language aspects of literacy given in the Adult Education and Family Literacy Act (AEFLA) of 1998:

The term ‘literacy’ means an individual’s ability to read, write, and speak in English, compute, and

solve problems, at levels of proficiency necessary to function on the job, in the family of the

individual, and in society (page H6636).

To speak in English also implies the ability to listen to and comprehend English. The importance of these

oral language aspects of literacy (i.e., speaking and listening to comprehend the English language) to the learning

and use of the written English language by native and foreign speakers alike is emphasized in the AEFLA, which

stresses the importance of phonemic awareness and phonics in the literacy development of adults in six different

sections of the law. The purpose of phonemic awareness and phonics training is to make it possible for a person to

comprehend the written representation of the English language through reading with the same or greater
4

effectiveness and efficiency as the oral representation of the English language can be comprehended by listening.

The theoretical model of literacy developed in Part 3 incorporates these various oral and written language constructs.

Part 4. Issues in the Use of Telephone and Mail Surveys in Assessing Adult Literacy. The final part of the

paper addresses several issues concerning the use of telephone and mail surveys, including concerns about obtaining

representative samples and about using these survey methods in such a way as to make continuity in trend data from

previous surveys of adult literacy possible. The feasibility of continuity in such NALS and IALS categories of

information as literacy practices, intergenerational transfer, economic indicators, civic participation, and various

demographic information products by telephone/mail surveys is explored.

Part 1. Issues of Validity in Adult Literacy Assessment

How should the literacy abilities of adults be represented? When discussed as a form of human cognitive

ability, as in the NALS and IALS, literacy is a psychological construct and as such “levels” of literacy cannot be

“directly measured” as suggested might be done in the 1992 report from the OECD. In fact, there are no “literacy

levels” to be directly measured in the adult labor forces of nations. Instead, there are various ways of construing the

nature of literacy and different procedures of measurement that can lead to the construction of alternative

representations of adult literacy in society.

In the United States over the last seventy-five years, different representations of adult literacy have been

socially constructed by psychometricians, statisticians, and survey experts in consultation with various stakeholder

groups including adult educators, adult students, literacy researchers, policymakers and others with a declared

interest in adult literacy (Sticht and Armstrong 1994). For instance, the National Assessment of Educational Progress

of 1970–71 assessed adult literacy using the same “academic” tasks that were used with school children, such as

knowledge of word meanings (vocabulary), using visual aids, following written directions, using reference materials,

locating significant facts, getting the main idea from materials, drawing inferences, and critical reading.

The Young Adult Literacy Survey (YALS) of 1985, the NALS of 1992 and the IALS of 1995 are the most

recent representations of adult literacy at national levels. These surveys assessed adult literacy using “tasks that

simulate the literacy demands that adults encounter in their daily lives” (Campbell, Kirsch, and Kolstad 1992, 12).
5

Similar “real world” tasks were used earlier in the “Survival Literacy” survey by Louis Harris Associates in 1970, the

Adult Functional Reading Study of 1973 and the Adult Performance Level Study of 1975 (Sticht and Armstrong

1994).

The fact that assessments of adult literacy over the decades have constructed various representations of

adult literacy (“academic,” “real world”) raises important questions. Just how should the literacy abilities of adults be

represented? Are all representations equally valid? Such questions seem to underlie the concern among participants

at the NAAL planning meeting mentioned earlier on whether telephone interviews are a viable means for identifying

whether people can “read, write, and comprehend.”

Yet, even setting aside the concerns about whether telephone surveys can validly represent the literacy

abilities of adults, the validity of the general practice of governments in conducting large scale assessments of

people’s literacy skills has not gone unchallenged. For instance, Kazemek (1990) stated:

...attempts at defining “ability levels” and “norms” are not only futile but potentially

dangerous as well. How can we possibly arrive at acceptable definitions of literacy when there are

countless life goals, needs, and desires among the adult populations? Realistically we cannot, but

in our attempts to do so we usually produce reductive lists, scales, and criteria...which are then

used to categorize large segments of the population, often in detrimental ways. (56). (See also

papers in Venezky, Wagner, and Ciliberti 1990).

Kazemek’s questioning of the validity of large-scale assessments of adult literacy in terms of both the way

one defines and measures literacy and the way one uses the scores from such assessments raises concerns addressed

by Messick’s discussions of validity (1989, 1988). Messick discusses validity as a “unified concept” with different

facets in which the validity of an assessment refers to the validity of the inferences one makes from the data.

One facet of the validity of test scores is the traditional concept of construct validity. Messick refers to this

as the validity of interpretive inferences and states:

To validate an interpretative inference [italics added] is to ascertain the extent to which multiple

lines of evidence are consonant with the inference, while establishing that alternative inferences are

less well supported. This represents the fundamental principle that both convergent and

discriminant evidence are required in test validation (1989, 5).

Evidence of convergent validity arises when correlations of literacy scores are found with other variables (e.g., years

of education, amount of reading) and when these relationships are predicted by the theory of literacy underlying an
6

assessment. Evidence of discriminant validity arises when variables that the theory of literacy predicts should not be

related are, in fact, unrelated in the data.

Kazemek’s questioning of the definitions and operations used to assess adult literacy is a challenge to the

construct or interpretative validity of the assessments. His questioning of the uses of test scores, including

detrimental uses, is addressed by Messick in the facet of validity that he calls action or use inferences. He states, “To

validate an action inference[italics added] requires validation not only of score meaning but also of value

implications and action outcomes, especially of the relevance and utility of the test scores for particular applied

purposes and of the social consequences of using the scores for applied decision making.” (1989, 5).

In the present paper, evidence for action validity is evaluated in Part 4 through the comparison of telephone

and door-to-door surveys as a means for producing information products about adult literacy valued by the U.S.

Congress, the branch of government that requested the NALS in federal legislation.

Some Issues of Validity in the NALS and IALS

In the United States, despite strongly held doubts by some scholars regarding the feasibility of defining and

assessing adult literacy, as indicated above, the U.S. Congress passed the Adult Education Amendments of 1988 that

required the U.S. Department of Education to submit a report to Congress on the definition of literacy and then to

report on the nature and extent of literacy among adults in the nation (Campbell, Kirsch, and Kolstad 1992).

With the aid of a national Advisory Board, the NCES agreed to define literacy as “Using printed and written

information to function in society, to achieve one’s goals, and to develop one’s knowledge and potential.” (Kirsch,

Jungeblut, Jenkins, and Kolstad 1993, 2-3).

The Advisory Panel for the NALS also agreed on the definitions of three different literacy scales that were

developed to represent the literacy skills of adults. These included: prose literacy—the knowledge and skills needed

to understand and use information from texts; document literacy—the knowledge and skills required to locate and

use information contained in materials; and quantitative literacy—the knowledge and skills required to apply

arithmetic operations embedded in printed materials (Kirsch, Jungeblut, Jenkins, and Kolstad 1993). These same

definitions were used to assess adults’ literacy abilities in the IALS (OECD 1995).
7

It should be noted that in both the NALS and the IALS there was an important conceptual shift in going

from the general definition of literacy as “using printed and written information to function in society, etc.” to the

definition of the three literacy scales that were actually going to be used to characterize adult literacy in the various

countries. None of the definitions of the three scales started with “using printed and written information” as the

defining aspect of literacy. Rather all three definitions referred to the psychological constructs of “the knowledge and

skills” or “the ability” required to perform tasks using prose, document, and quantitative materials. This makes an

important shift in conceptual understanding that, unlike what the general definition of literacy stated, literacy is not

the use of printed materials per se, but rather the knowledge and skills or ability that make possible the use of printed

materials for various purposes.

A recent U.S. General Accounting Office reported that one expert on adult literacy research stated that

functional literacy tests like those of the NALS and IALS may lack validity because they are not derived from

theoretical models of ability but from everyday literacy tasks (U.S. General Accounting Office 1995, 24). Given the

complex nature of such tasks, it is not clear what implications can be drawn from the test performance.

This was clearly illustrated in a manual for item writing produced by the Comprehensive Adult Student

Assessment System (CASAS), an adult literacy assessment system widely used in the United States and in some

other industrialized nations that uses “real world” literacy tasks much like those of the NALS and IALS. The manual

notes that the use of complex, “real world” or “functional” tasks as items “...generally tests the use of two or more

skills [italics added]. Therefore, this context is not appropriate in itself for diagnosing weaknesses in specific skills

since it is difficult to determine which skill was performed incorrectly” (CASAS 1983, 1).

In the absence of a clearly specified theory of “literacy” as a psychological construct, it is not possible to

know how to develop assessments that measure the component knowledge and skills that make-up the ability or

abilities that constitute “literacy.” Without knowing what specific knowledge or skills are being assessed in “real

world” tasks, it is not clear to what extent test performance reflects literacy ability or some other abilities, such as

problem solving, reasoning, language comprehension, vocabulary knowledge, management of test-taking anxiety,

interpersonal skills, or some complex, interactive combination of all these. That is why, in the earlier version of the

NALS and IALS, referred to here as the YALS, performance on the three literacy scales was referred to as “complex

information processing” tasks (Kirsch and Jungeblut 1986).
8

This problem of interpreting what is being measured may not be so important for those who perform well,

and can therefore be assumed to possess whatever knowledge and skill is called for in performing the “complex

information processing” (literacy?) tasks. The problem becomes critical when the focus of concern is on

understanding why it is that those who do not perform well do not perform well. What kinds of services should be

provided to help them improve their ability to perform these kinds of tasks? Should major efforts go into providing

basic phonemic awareness or phonics training, or should vocabulary development and reading comprehension be the

focus of instruction? Should there simply be practice in performing a wide variety of complex, information

processing tasks like those assessed on the literacy surveys? How many of such tasks should be used in training and

how and why would specific task training be expected to generalize to other tasks? Would training in memory

enhancing techniques, logical reasoning, and critical thinking be the thing to do? How much of the problem is a

“literacy” problem in contrast to a “general education” problem?

The lack of a good, workable theory of literacy has led to the development of adult literacy assessments

over the last three decades using a variety of different materials and tasks. Most recently the NALS and IALS have

used three different but highly correlated scales (around r = .90, (Reder 1994(a); Rock 1998)) that produce much

redundant information. If a theory of literacy could help identify the “knowledge and skills” that underlie the

performance of prose, document and quantitative tasks it might be possible to more directly and cost-effectively

assess that knowledge and those skills and to predict the performance of a wide variety of both “academic” and “real

world” literacy tasks that go beyond those used in present assessments.

While it is not possible at the present time to provide a comprehensive theory of literacy that is universally

agreed upon by all concerned, Part 3 of this paper provides aspects of a cognitive theory of literacy that suggests

some of the major components involved in literacy that might be assessed not only in door-to-door surveys, but also

in telephone and mail surveys.
9

Part 2. Review of Related Research

Cognition; Cognitive (adjective): (1) The mental process or faculty by which knowledge is

acquired; (2) that which comes to be known, as through perception, reasoning, or intuition; knowledge.

(The American Heritage Dictionary 1976)

The definition given above indicates that cognition includes (at least) two aspects: (1) the mental processes

by means of which knowledge is acquired and (2) the knowledge that has been acquired. Telephone surveys have

been used to obtain information about both aspects of cognition, the mental processes that people use and the

knowledge that people possess. Three such studies are reviewed here.

Assessing Specific and General -’g’- Cognitive Abilities Using Telephone and Mail Surveys

In studies of behavioral genetics, Plomin and associates (Kent and Plomin 1987; Plomin et al. 1986) used

mail and telephone surveys to assess various cognitive abilities of children, adolescents and adults (ages 9 to 80).

Across the three studies, the tests assessed various verbal, spatial, perceptual speed, and memory abilities. The

general procedure in three different studies (two reported in Plomin, et al. 1986) was to first mail out kits containing

cognitive test batteries to individuals who had agreed to participate in the studies. Then the test administrator would

telephone the subjects and administer the tests over the telephone.

To check for comparability between telephone and face-to-face testing, Kent and Plomin (1987)

administered face-to-face assessments to samples of the two hundred and twelve children and adolescents in their

study. Fourteen were tested face-to-face before the telephone assessment and sixteen after the telephone tests.

Additionally, thirty-two individuals had been tested earlier in telephone sessions and they provided test-retest

information for the telephone assessments. Altogether then, there were sixty-two subjects with test-retest scores.

Data showed correlations among telephone and face-to-face samples of over .92 for a g (general ability)

factor extracted from the factor analysis of scores. Overall, for the total of 62 test-retest scores, correlations were .81

for the Verbal factor, .73 for Spatial, .82 for Perceptual Speed, .57 for Memory, and .79 for g.
10

Kent and Plomin (1987) state that, “These studies indicated that it is possible to assess cognitive functioning

via telephone; importantly, subjects indicated that they preferred telephone testing to face-to-face testing, citing

increased anxiety during in-person testing” (392).

Testing Word Knowledge by Telephone to Estimate General Cognitive Aptitude Using Computer

Adaptive Testing

Research by Legree, Fischl, Gade and Wilson (1998) aimed to determine whether word knowledge

(vocabulary) could be assessed over the telephone to predict scores on the Armed Forces Qualification Tests

(AFQT), a paper and pencil battery of tests used to select applicants for military service.

Earlier research had produced the Computerized Adaptive Screening Test (CAST) to help U.S. Army

recruiters quickly estimate the cognitive aptitude of potential recruits so they could concentrate their time on those

individuals most likely to qualify for service. The CAST is highly correlated (.91) with the AFQT and requires about

10 to 15 minutes for administration. This contrasts with the approximately three hours needed to take the Armed

Services Vocational Aptitude Battery (ASVAB) from which the AFQT is derived using four sub-tests of the battery.

The CAST uses a computerized adaptive testing program to select items from a large bank of items on the

basis of the individual’s current level of performance. Using this program, performance on the AFQT is estimated

with approximately 10 items from the verbal (word knowledge) domain and 5 from the quantitative domain.

To determine whether the CAST verbal items could be given by telephone to yield an estimate of

psychometric g (general ability) and AFQT scores, the CAST verbal items consisting of multiple-choice, word

knowledge questions requiring recognition of synonyms was administered over the telephone to 144 recruits who had

recently enlisted but had not yet entered into the military. The use of new recruits meant that their paper-and-pencil

ASVAB/AFQT scores were available from their official records and could be used to determine correlations among

the telephone and CAST determined vocabulary knowledge scores and the AFQT scores.

Subjects were called by experienced telephone interviewers who explained the purpose of the study and

requested the subject’s participation. The interviewer read each question and response alternatives (e.g., The word

tall means: heavy, large, small, high). Subjects indicated their choice for each item and the interviewer entered the

choice into a computer and the program then selected the next item. The branching algorithm selected 10 or 11 items

for each subject in arriving at an estimate of the AFQT score.
11

Data analyses indicated that the telephone and CAST test scores correlated .66 with the AFQT. Because

range restrictions due to military entrance requirements attenuates all ASVAB correlations, a bivariate correction for

range restriction was used to estimate the correlation between the AFQT and the telephone CAST scores for the

general population and this produced a correlation of .81.

Based on the results of the study, the U.S. Army Research Institute is incorporating the Telephone Test into

a larger research-based telephone survey addressing issues related to youth career selection to link individual

differences in cognitive aptitude to other variables.

The research will also explore the use of the Telephone Test with a broader range of abilities, with

particular interest in lower aptitude groups.

Assessing Adult Literacy-Based Knowledge and Literacy Practices by Telephone

In a study using random-digit dialing telephone sampling techniques, Sticht, Hofstetter and Hofstetter

(1996) assessed various aspects of adults’ literacy-based knowledge and use of printed materials at home and on the

job.

Defining Literacy-Based Knowledge. Several lines of research have converged suggesting that people become

highly literate largely by engaging in numerous literacy practices, such as reading books, magazines, newspapers,

and so forth (Reder 1994(b); Krashen 1993). A review of the major assessments of adult literacy in the United States

since 1937 consistently revealed that as years of education increases there are corresponding increases in both the

number of literacy practices in which adults engage and the amount of skill displayed in the assessments (Sticht and

Armstrong 1994).

In an illuminating program of research, Stanovich and associates explored how the extent of engagement in

literacy practices by children and adults contributed to their development of literacy and, more broadly, “verbal

intelligence” (Stanovich 1993; Stanovich and Cunningham 1993). Using an innovative method for assessing

knowledge with checklists calling for a simple “yes” or “no” judgment on the part of the reader, Stanovich and

associates demonstrated that performance on these checklists correlated significantly with a variety of literacy

activities and cognitive assessments (Allen, Cipielewski, and Stanovich 1992; West, Stanovich, and Mitchell 1993).
12

A major conclusion from this body of research is that the simple checklists assess knowledge indicative of (1) the

extent of use that one makes of printed and written materials, (2) the knowledge one derives from the use of these

materials, and (3) the added potential for engaging in future literacy practices as a consequence of having a broader

base of knowledge prerequisite for wide-ranging reading.

In these studies, Stanovich and associates argue that individuals who read a lot acquire, mostly through

incidental learning, a large declarative knowledge base containing the names of authors, magazines, newspapers,

persons known for their contributions to film, theatre, music and other activities that Hirsch (1987) referred to as

“cultural literacy” and a large vocabulary of words typically not encountered with high frequency in day-to-day oral

communication nor on television or radio. Scores on these checklists are indicators of both the amount of reading in

which individuals engage and of the cognitive outcomes in terms of the individual’s declarative knowledge base (see

comments by Taylor 1994 and reply by Stanovich and West 1994, regarding contrasting perspectives on the use of

the checklist methodology for studying reading behavior).

Using the Stanovich Checklists to Assess Literacy-Based Knowledge by Listening on the Telephone.

The checklist approach to knowledge assessment developed by Stanovich and associates lends itself readily to the

assessment of knowledge by listening since, as suggested by the developmental model, below, research indicates

that, beyond the period of reading in which early decoding skills are developed, listening and reading converge on

the same knowledge base (Sticht et al. 1974; Sticht and James 1984; Stanovich 1991; Sinatra 1990). The theory that

both listening and reading share the same knowledge base in the human cognitive system (see below), which is

made-up of both language (grammar, with lexicon and syntax) and language-based meanings, means that in

individuals, after reading decoding processes are developed, the processes of listening and reading are very largely

the same at the level of the knowledge base (Sticht 1978; Sticht and McDonald 1992). In the individual’s mind,

listening and reading processes are not correlated; rather, they are structurally the same with regard to the knowledge

base that they draw upon (with only slight differences due primarily to the nature of the informational displays

involved in speaking and writing).

Because individual differences exist in the amount of knowledge that individuals possess, listening and

reading will tend to be moderately to highly correlated in the adult population, meaning that low, medium and highly

knowledgeable adult listeners will also be low, medium and highly knowledgeable readers (Sticht, Hooke, and
13

Caylor 1981(a)). The correspondence of listening and reading processes at the level of the knowledge base forms the

mechanism for assessing the knowledge component of literacy by telephone.

Knowledge as an Indicator of Literacy Ability. There is a knowledge component (words, sentences, paragraphs)

in all of the myriad literacy tasks in which people engage in our information age society. This suggests that the

knowledge component can be used as an indicator of literacy ability. The assessment of knowledge as distinct from

the performance of complex tasks that include knowledge along with various types of information processing tasks of

unknown qualities is also warranted when the definition of literacy adopted by the advisory panel of experts for the

NALS is examined. The definition of literacy agreed to was “Using printed and written information to function in

society, to achieve one’s goals, and to develop one’s knowledge [italics added] and potential” (Kirsch et al. 1993, 2-

3).

The importance of knowledge in literacy was also acknowledged by the NALS advisory panel in its

acceptance of the definitions of the three different literacy scales (prose, document, quantitative) that were

developed. In each case, literacy was defined as “the knowledge and skills” needed to perform the three types of

literacy tasks. From these definitions, it is clear that the advisory panel for the NALS understood that the use of

printed and written information to accomplish tasks requires certain knowledge and skills to make such use possible.

In a telephone survey methodology, a respondent’s knowledge may be assessed using Stanovich-type lists

containing discrete items, such as names or single vocabulary words, that require only a yes/no decision for each

item. Such lists are suitable for presentation by telephone for listening and by written checklists for reading because

they do not overload working memory (see the human cognitive system model, below) and introduce irrelevant task

variance (Messick 1989). In contrast, the NALS used literacy tasks that emphasized “search and locate” (Guthrie,

Britten, and Barker 1991) information processing skills requiring subjects to hold instructions in working memory

while performing search and locate actions and problem solving (make inferences; reason) to accomplish unfamiliar

tasks (Kirschet et al. 1993; Mosenthal and Kirsch 1991; Kirsch and Mosenthal 1990).

NALS data for adults above age 54 show a rapid decline of performance (Kirsch et al. 1993). Since it is

well established that working memory becomes increasingly less efficient with advanced age (Bernstein et al.1988;

Meyer, Marsiske, and Willis 1993), these findings strongly suggest that the NALS and IALS tasks derive a great deal

of their difficulty from the load they place upon working memory and this may threaten their validity as literacy
14

measures for the elderly. Additionally, the NALS methodology introduces unknown, and possibly irrelevant test

variance which may lead to serious underestimations of the breadth of materials that older adults can read and

comprehend using their knowledge base and the tasks they can perform in working memory given sufficient time to

study materials, without the pressure for efficiency typical of test-taking situations. The latter are of questionable

ecological validity in the lives of most adults over the age of 25 who are not in school.

Questions for the present study. This study investigated the assessment of the knowledge component of adult

literacy using telephone interviews with five hundred thirty-eight randomly selected adults, all English speakers, as

the mode of presentation, listening as the mode of reception, and four abbreviated versions of the Stanovich

checklists to assess aspects of the declarative knowledge base likely to be developed by reading. The checklists

assessed adults’ knowledge of authors, magazine titles, famous people, and vocabulary. In addition to the telephone

survey, a sub-sample of one hundred forty adults who agreed to complete a mailout version of the various checklists

and a cloze test of reading was surveyed. Several questions that were explored are listed below along with the

relevant findings.

1. Does an oral presentation of checklists by telephone, which requires listening ability, produce

results consistent with a written presentation of the checklists, which requires reading ability?

Yes, data for the telephone and mailed surveys revealed positive relationships between listening and reading (r=.80

for Total literacy scores made-up of the summed percentage correct across the four checklists).

2. Do the knowledge scores obtained using the checklist and telephone survey methodology correlate

with scores on another frequently used method for assessing reading ability?

Yes, both the listening and reading checklist scores were positively and significantly (r=.38, p<.05) related to the

cloze test with its greatly differing task demands and limited content (only one passage was used). These data, along

with the data for the first question, provide evidence for the construct or interpretive validity (Messick 1989) of the

theory of literacy that states that both listening and reading tap a common knowledge base and that the knowledge

base is a component of, not a proxy for, literacy (reading, in this case).
15

3. Do the knowledge checklist scores from the telephone survey show positive relationships with

demographic variables that have been found in previous adult literacy assessments?

Yes, convergent evidence for interpretive validity was found in a positive manifold of statistically reliable correlation

coefficients among the literacy knowledge checklists and other variables that are interpretable in relation to the

development of literacy, such as parent’s education, one’s own years of education, amount of engagement in various

reading practices, occupational status and income.

4. Are the knowledge checklist scores from the telephone survey negatively correlated or uncorrelated

with variables that the construct of literacy encompasses?

Yes, discriminant validity was evidenced by data indicating that checklist scores were negatively correlated with age

and the practices involved in family literacy (e.g., reading to children) which reflects the large proportion of older

adults in the telephone sample whose children are grown and gone, yet whose literacy practices in other areas have

tended to make them more literate (knowledgeable).

This confirmatory evidence of construct validity for the checklist and telephone methodologies as measures

of the knowledge component of literacy is encouraging given that this was the first study of its kind and no attempt

was made to develop checklists or cloze tests that would provide greater differentiation among people.

Domain Specific Knowledge Assessed by Telephone. In a second study of the assessment of adults’ knowledge

by telephone, Hofstetter, Sticht and Hofstetter (1999) used six different checklists of knowledge, five that assessed

adults’ domain specific knowledge of political information, such as famous political figures and various political

policies, as well as a sixth checklist made up of the vocabulary checklist used in the earlier study. A random-digit

dialing telephone survey of six hundred thirty-two English speaking adults provided data to study relationships

among domain specific knowledge, general vocabulary knowledge, amount of reading reported, occupational status,

income and participation in political activity.

As in the first study, a positive matrix of significant correlations was found among literacy practices

(amount of reading in home and work settings), domain and general knowledge, age, and amount of education. Of

interest for demonstrating the importance of knowledge in the “information age,” positive and significant
16

correlations were found among knowledge and occupation, income and political activity even when data were

statistically adjusted to control for age, education, gender, ethnicity, and measures of extent of literacy practice.

The data from the two telephone surveys using the knowledge assessment checklist methodology indicate

that a large amount of information about adults’ literacy-related knowledge can be obtained using telephone surveys.

But can the information obtained by the telephone survey provide the same range of useful information that was

obtained in the door-to-door NALS To answer this question, an action or use analysis (Messick 1989) of the five

informational products that the NALS was commissioned to produce is presented in Part 4 to see how well the

telephone data could be used to provide the same or related types of informational products.

Part 3. A Cognitive Theory of Literacy

Kirsch, Jungeblut, and Mosenthal 1998) trace the evolution of the theoretical framework used to construct

national and international adult literacy surveys from the 1985 YALS, to the 1992 NALS and the 1995 IALS. Across

these surveys, the framework used to develop assessment tasks was based on the types of materials involved in the

assessments, the uses that respondents were asked to make of the materials, i.e., the types of tasks performed or

questions to be answered, and what Kirsch, Jungeblut, and Mosenthal (1998) refer to as “...three important and

distinct areas: prose, document, and quantitative literacy” (106).

In a number of studies, it was found that difficulty of tasks (e.g., percentage getting various items correct)

was a function of the information processing complexity of the task, including things like the numbers of items from

the questions or task instructions that must be held in mind while searching through materials to locate answers, the

amount of materials to be searched or read, or the abstract versus concrete nature of the language in the materials, or

the types of and amounts of calculations that had to be done in the quantitative literacy tasks.

But while there are a great deal of empirical data given to support the usefulness of the various material and

task variables that the researchers identified through their analyses to predict the difficulty of prose, document and

quantitative tasks, there is no explicit discussion of a cognitive theory of the competence (knowledge and skills) that

adults must possess to be able to perform the many tasks. For instance, although all of the tasks required the use of

reading decoding skills and language-based knowledge, and the prose, document and quantitative scales correlated

around +.90, sharing some 80 percent of their variance, there is no theoretical statement about cognition that might

account for this large overlap among the three literacy domains.
17

While theories of literacy acquisition vary, including sociolinguistic and sociocultural theories regarding the

role of social interactions and cultural practices in the acquisition of literacy (e.g., Reder 1994(b)), the NALS and

IALS have been developed based on a cognitive approach (e.g., the “knowledge and skills” needed for Prose,

Document and Quantitative scales). Consistent with this conceptual stance, this section of the paper presents a

cognitive theory of literacy that explains why there is so much overlap among the prose, document and quantitative

scales and how it is possible to focus on different components of literacy to represent different aspects of adults’

performance of various literacy tasks. The theoretical framework includes a conception of (1) a simple model of the

human cognitive system and (2) a conception of how that system develops from a prelanguage, preliterate system at

birth into a language-based, literate system in adulthood.

The Human Cognitive System

The cognitive theory of adult literacy is based on the “modal model of memory” that has been examined in

over thirty years of research (Figure 1) (Atkinson and Shiffrin 1971; Healy and McNamara 1996, 143). Generally

speaking, highly literate individuals possess large bodies of knowledge in long term memory and information

processing capacity and efficiency in working memory to process information in complex graphic documents

(Kyllonen and Christal 1990).

Cognitive development is construed as the change in the human cognitive system that results from changes

in the anatomy and physiology of the human brain as it grows, develops, and deteriorates in later life. There are three

concepts of “context” that are important in understanding cognitive development. Two kinds of context are “inside”

the person. One is the biological, physiological context of the body and brain. The second is the mental context of

the mind in which the person’s psychological life is constructed. The third concept of context refers to the world

“outside” the person. This is the physical and social world in which the person lives, the external environment.
18

Figure 1.—Three components of the human cognitive system.

Sensory/ 

Perceptual 

 System

Working 

Memory

Long Term 

 Memory

Short Term 

Memory

Knowledge 

Base

Human Cognitive System

Realia

Written 

Spoken 

Language

Media

Internal Contexts

Mental Processes, Knowledge, Emotions

Biological Contexts

Psychological Contexts

Physical 

Social 

Cultural 

Contexts

The internal environment or context, which is generally referred to as “mind,” is both the result of and the

cause of cognitive development through experience. That aspect of mind involved in learning and cognitive

development is referred to as the human cognitive system, and includes three major components of the cognitive

system: the sensory-perceptual system, the short term (or working memory) and the long term memory (or

knowledge base). The working memory is where active thinking takes place. Thinking processes lead us to pick up

information from the outside environment and the internal knowledge base and combine these two sources of

information to construct our understanding of the world at a particular moment. To a large extent, it appears to be the

load or strain on working memory that accounts for much of the difficulty that Kirsch, Jungeblut, and Mosenthal

(1998) related to the numbers of features that had to be held in mind and matched in the materials of the NALS or

IALS, along with the types of calculations that had to be performed in working memory (“mental math”) in

quantitative tasks.

As the rough sketch below (Figure 2) indicates, a large amount of the overlap among the prose, document

and quantitative tasks seems also to reflect the role of the knowledge base that is drawn on in comprehending the

interviewer’s task instructions and in reading and comprehending the information in all three types of displays of

information. As the figure illustrates, the vocabulary used to construct prose displays of information may also be

used to construct document and quantitative displays. The extent of the vocabulary knowledge in adults’ knowledge

bases no doubt contributed to their ability to deal with tasks involving “concrete” or “abstract” language and with

larger amounts of information presented in written language. It should be noted that the commonality of language

and working memory processes operate in most cognitive assessments, leading to what some have called “general
19

ability” or “psychometric “g” (Gottfredson 1997). The present cognitive theory goes beyond these concepts by

emphasizing the three-part model of the human cognitive system and the multifaceted components of cognitive

development given in the developmental model of literacy described below.

Figure 2.—Illustration of how language-represented knowledge from the knowledge base of the human cognitive

system model can be used across prose, document and quantitative literacy tasks to produce high correlations among

the three literacy scales.

A Developmental Model of Literacy

A major goal of literate societies (and societies seeking literacy) is that each new generation should develop

cognitive systems with the knowledge and processing skills needed to use (produce and access) the symbolic

knowledge bases stored in the external environment in the form of books, maps, tables, and so forth. For this reason,

a schematic representation of cognitive development in the literacy domain is used to organize information about the

development of the human cognitive system as it transforms itself from preliterate to literate.
20

To (hopefully) facilitate understanding, the major concepts used in modeling the development of literacy in

“the typical case of a child growing up in our literate society” are presented schematically in Figure 3.

Before addressing the details of the model, several orienting comments regarding the figure are in order.

First the figure is meant to portray a developmental sequence when examined from left to right. The sequence begins

with the newborn infant (far left), and goes through stage 4 in which literacy skills are functional. The broad

arrowhead on the far right is meant to imply continued development over the lifespan. The development of literacy,

language, and knowledge is a lifetime activity.

Figure 3.—A Developmental Model of Literacy

Examining the model from top to bottom, the top series of boxes is meant to represent the environment in

which the person exists. This is the environment “outside the head.” This external environment makes available

information displays that the person can explore and transform into internal representations of the external
21

information. These internal representations are developed by the processes in the second series of boxes labeled, on

the far left, Information Processes in Working Memory. These processes go on “inside the head,” and merge

information picked up from the external information displays with information picked up from the third series of

boxes, labeled on the far left as Long Term Memory. Thus, the processes in the working memory are used to pick up

and merge information from outside the body/brain with information in long term memory inside the body/brain to

construct an internal representation of the world as currently experienced, including the construction of a meaning

represented in symbolic information when this is a major domain of information being extracted from the external

world at a given time.

At the top of the model, there are references to four “stages.” In the present case, the term “stage” refers to

what would typically be observed at different times if one studies children growing up in our literate society. For

instance, stage 1 refers to the newborn infant who is considered to be innately endowed with the Basic Adaptive

Processes involved in sensory/perceptual processes such as hearing and seeing, etc., motor movement, and cognition,

including the processes needed to acquire information, mentally manipulate it, store it in memory, form knowledge

structures out of it, retrieve and represent the information in various ways. In stage 1, these processes are assumed to

work more or less automatically without conscious control, hence an observer would note that the infant seems

“captured” by stimuli, rather than selective in observing information in the world.

Stage 2 represents the emergence of conscious control over information pickup and manipulation. This

active process of attending to information distinguishes listening from hearing, and looking from seeing, as

information pick-up processes. Listening and looking build internal representations that may be called images.

Images may also be constructed from data stored in long term memory. These internal imaging processes are

frequently assessed in aptitude tests as “spatial perception” or “mechanical comprehension” in which it is necessary

to mentally visualize and rotate cog-and-gear systems to determine what effects this movement might have on some

other gear.

Stage 2 also introduces the concept of active or working memory, which is defined by the occurrence of

consciously controlled information processing activities. As noted earlier, working memory is a limited memory that

can be easily overloaded (e.g., attending to two or three things at once is difficult—if not impossible). Many of the

information processing activities the person acquires will be techniques to overcome active memory limits (e.g.,

repeating information to oneself keeps the information in active working memory until it can be applied). The

“stage” aspect of cognitive development shows itself by the ability of the infant to attend to information selectively.
22

This is a cognitive capability, which, once developed, is a permanent feature of cognition that distinguishes the stage

2 child from the stage 1 child.

Stage 3 represents the development of language processes out of earlier processes and knowledge stored in

long term memory. In developing oral language, the listening process is used in attending to spoken language to learn

the words and grammar of language. Thus, listening plus languaging, occurs simultaneously. This joint occurrence is

given the special name of auding. On the production side, the joint occurrence of uttering (making sounds through

the mouth) with the production of word forms from the language pool, and stringing the word forms together to make

sentences using the rules of syntax, produces the special process called speaking. Auding and speaking comprise the

oral language information reception and production skills. Speaking is used to represent information that the person

has in his or her mind “outside the head” and in the acoustic medium, while auding is used to pick up and transform

speech information displays into knowledge in the mind of the listener. To an observer, the stage 3 child can respond

to and produce oral language, at least to some minimal degree. Again, as with conscious attending, once oral

language has begun to emerge, the cognitive system is permanently modified (barring physiological trauma of some

sort), and the person is no longer capable of exclusively prelinguistic modes of thought.

In transitioning to stage 4, the information processing skills of looking and marking are used to learn a

representational system which, in many respects, represents the spoken language in a different medium—light—and

in a more or less permanent graphic display: the written language. Looking at written language and transforming the

written language into meaning is called reading. Writing is the special use of marking skills to produce graphic

language (and other symbols and symbol systems).

In the typical case, children develop a fair amount of competence in oral language before they are exposed

to formal instruction in reading in elementary grades (though informal learning of literacy may begin in the home and

community in literate cultures). In literacy instruction, written language skills build upon the earlier developed oral

language skills and add to the person’s knowledge base special knowledge about how to represent and comprehend

information in the graphic medium that was previously represented and comprehended by the person only in the oral

language. Later, learning new vocabulary and conventions of language through reading and writing reflects back on

oral language development and enlarges the person’s oral language abilities. The large arrow at the far right in

Figure 3 is meant to represent the notion that the development of oral and written language ability may continue

indefinitely as the person studies and develops new knowledge domains.
23

A major component of the developmental model of literacy is the person’s long term memory or

“knowledge base.” The long term memory contains all the knowledge developed by the person in interaction with the

environment. Much of the knowledge acquired by the person will not be understood in consciousness (for example,

the rules of grammar). Rather, it will be unconsciously used to accomplish tasks such as developing language

competency and comprehending the events of the world. In addition to the general world knowledge and processes

that are in the mind, though not accessible to conscious understanding without considerable analysis, the memory

also contains the language knowledge (words and grammar) that can be used to represent information that arises

from experience in the world (e.g., bodies of knowledge about machines, parts of the body, houses, neighborhoods—

sometimes called “schema” (Bransford et. al. 1986) or “mental models” (Rumelhart,1980), and from didactic

instruction, as in training programs.

The model holds that the development of the oracy skills of speaking and listening (auding) are built upon

the prior development of prelinguistic knowledge through information processing activities. It is important that it be

understood that this early, prelinguistic cognitive content, or knowledge, will form the foundation for the acquisition

of new knowledge over the person’s lifetime, including that knowledge known as “literacy.”

A final aspect of the model is that it recognizes that, on the one hand, the literacy skills of reading and

writing utilize the same knowledge base that is used in auding and speaking, plus the special decoding and encoding

skills of reading and writing. On the other hand, the very nature of the written language display—characterized by

being more or less permanent, being arrayed in space, and utilizing the features of light (color, contrast)—makes

possible the development of skills and knowledge entirely different from those involved in oral language.

Literacy and the Merging of Speech and Graphics Technologies

In all societies, preliterate people learn to communicate with others in their communities. One way they do

this is by the use of visible movements called gestures. For example, they may make grimaces when food tastes bad

to them, or frown when sad, or point to direct other’s attention to something.

Another way preliterate people learn to communicate with others is through the development of a form of

gesturing that transmits thoughts not by visible means, but by the invisible movement of the molecules of the sea of

air in which people are immersed. That is, they acquire the oral language of their community. This means that their

brains engage in a mental design and execution process that permits them to use their neck and chest muscles, lungs
24

and oral apparatus (i.e., their articulators, tongue, throat, teeth, lips, larynx) and their ears and other auditory

components to produce movements (speech gestures) that result in sounds and sequences of sounds that permit them

to exchange thoughts with others. The brain engages in this mental process without the person being consciously

aware of what the brain is doing or how it is doing it. That is, the actual mechanics of the acquisition of the spoken

language is done automatically by the person’s brain.

A child’s developmental process typically leads to the formulation of a mental (brain) system or component,

called grammar, consisting of a lexicon (a vocabulary of forms, generally words, though sometimes word-parts, e.g.,

-ed, constructed by the mental design process that controls the articulators) and a syntax (rules for arranging the

word forms into sentences) for conveying meaning through sounds in the air. From their parents, siblings and others

in their social group children learn certain formalities or rules about how one talks about certain others, what forms

of address to use, how to formulate and tell a narrative story about some event they experience and many other things

about the use of their native language.

The Person’s Use of the Graphic Medium for Cognition and Communication. In addition to using visual

gestures and those involved in speech for communication, preliterate people also learn to use some forms of marking

of surfaces as a visual, graphic medium of communication. For instance, they may tattoo their bodies with graphic

designs having various symbolic meanings. They may weave such designs into blankets or clothes, carve designs on

wooden utensils or paint them on pottery.

Though the gesture movements that produce both spoken and graphic designs may be used for

communication, the graphic medium has features that differ from speech which permit the user to accomplish

cognitive and communication tasks that would not be possible otherwise due to the limits of human memory. These

features include permanence, spatiality, and light.

The relative permanence of graphic displays makes possible the storage of information “outside the head”

and makes it necessary for others to learn to look at and interpret the graphic “gestures” if they wish to retrieve the

stored information. Similarly, the fact that the graphics medium uses space and light makes possible the design of a

wide variety of displays that permit preliterate people to perform a number of cognitive and communication tasks

that are not possible with the temporally linear, fleeting spoken language.
25

Literacy as the Merging of Speech and Graphics Technologies. In anthropological terms, technology is the

body of knowledge a civilization uses to fashion implements, extract or collect materials, or practice various arts and

skills (Morris 1976). From this point of view, literacy may be thought of as technology for producing and

comprehending graphic displays as tools (implements) for accomplishing various cognitive and communication

activities (Harris 1986; Goody 1977). In writing, a person “extracts” knowledge from the brain and “collects”

(stores) it in graphic displays. Then, through the practice of the skill of reading, the collected knowledge is extracted

by the same or another person from the graphic display and reconstructed in the brain.

Much of what is encountered in teaching and learning reading results from the fundamental characteristics

of graphics displays. These characteristics permit literates to work with the graphics technology in certain ways.

Furthermore, the products of the work of literates create new graphics displays (e.g., history timelines, TV schedules,

troubleshooting flow diagrams, atlases). This imposes additional information processing demands upon new readers

who wish to learn to work with these new displays.

Major Features of Graphics Technology

The major features of graphics technology, examples of the types of products that may be produced by

those in command of the technology, and examples of the types of demands for information processing that the

products of this technology may require include:

(1) Permanence. Graphic information displays, such as this page of print, or forms, tables , graphs, and so forth, are

more or less permanent. Therefore, they can be used to collect or store information, including an extended body of

knowledge. The information can be stored over time and retrieved later on, and it can be transported across space.

(2) Spatiality. Unlike speech, graphic information displays can be arrayed in space. Signs can be placed on doors,

over buildings, alongside highways, and so forth; pages of print with words laid out spatially to permit the recreation

of a temporal flow of speech can be constructed; forms can be developed with “slots” containing labels (“Name;”

“Address;” etc., see Figure 2, above) and myriad other graphic tools to accomplish various information transmission
26

and processing tasks can be developed (Schwartz 1971), such as labels, lists, bus schedules; flow charts; tables;

schematics; transparencies; and so on.

(3) Light. The third major feature of graphics technology that is drawn on in literacy is the use of light. The marks

that are made to produce such graphic symbols as written or printed words, numbers, arrowheads in procedural flow

charts, the white space of the cells in a table (matrix) and so forth are constructed by structuring the light that leaves

the surface of the graphic medium such that the eye can detect the structure in the display.

In addition to aiding in overcoming memory limitations and facilitating learning through various semantic

“chunking” and organizing devices, the properties of light are extensively used to aid attention during information

processing. Bold print may be used to call attention to certain information and color can guide information

processing, as when a red line is used in an electronics diagram to permit a particular circuit to be traced in an array

of circuits printed in black.

As with all technology, the power of graphics technology arises from its use to develop tools for amplifying

and extending human capabilities (Bruner 1968). However, unlike hammers, sewing machines, automobiles, and

other technologies that extend human strength, dexterity, or locomotion abilities, graphic technologies gain their

power from their application to the extension of human cognition and the ability to manipulate information in

symbolic form.

In particular, the merging of graphics technology with spoken language, itself a form of human technology

for communicating with symbols, produces the power behind, and the awe and appreciation of, literacy.

The Graphic Representation of Spoken Language

Perhaps the capstone achievement in graphics technology was the development of the alphabet, a relatively

simple technology by means of which a few graphic marks can represent enough aspects of the oral language that the

marks permit a reader to reconstruct an oral language-based message from the graphic display (McGuinness 1997).

The importance of this is that it permits graphic language to draw on the power of oral language for representing and
27

communicating knowledge, while bringing the power of the three features of graphics technology to bear on the

development of new knowledge and tools for thinking and problem solving.

For those adults with little or no reading ability, instruction generally centers on decoding or reconstructing

a spoken message from the graphic display or “code.” The written display is called a “code” because it is considered

an alternative representation of speech. In a simple substitution code one element in a message, such as the sequence

1, 2, 3 is substituted by another set of symbols, such as a, b, and c. In decoding, then, a is converted to 1, b to 2, and

c to 3.

Though such a simple, one-to-one correspondence of speech and graphic symbols does not hold over the

full range of the English language, enough correspondence is there to make the technology work. In the teaching of

decoding by the principle of the alphabet (Liberman, Shankweiler, and Liberman 1989) learners are taught to

substitute a graphic alphabet symbol for a speech sound (phonemics; phonics). In teaching writing they are taught to

make drawings (alphabet characters) that can, in turn, represent speech sounds.

Discussion. The theoretical model of the human cognitive system given in Figure 1 along with the

illustration of Figure 2 suggest that one reason that the prose, document and quantitative tasks of the YALS, NALS,

and IALS are so highly correlated is because for any given adult, the item tasks in all three domains draw upon the

same long term memory or knowledge base, the same language (vocabulary, syntax) system, the same working

memory system, and the same sensory-perceptual system that the adults have developed over their lives as illustrated

in Figure 3.

As Figure 2 illustrates, it is possible to separate out the language component of the prose, document and

quantitative items from the graphic display components of the tasks. The latter introduce much of the demand for

visual search and locate processes that tend to place an increasing load on working memory and renders tasks more

difficult.

For many people, as characterized in the developmental model of literacy of Figure 3, reading and writing

refers to one’s mastery of the alphabetic principle, that is, the idea that speech sounds can be represented in a graphic

code called the alphabet and reading is the recoding of the graphic code into a spoken language-based meaning. In

fact, the “simple view of reading” (Gough 1996) considers that R=D+C, Reading equals Decoding plus

Comprehension, where the latter is measured by listening tests. Recent research has supported the “simple view of

reading” (Chen and Vellutino 1997).
28

The developmental model, the “simple view of reading, “ and extensive empirical evidence (Sticht et al.

1974; Sticht, Hooke, and Caylor 1981(a); Sticht and James; 1984; Sinatra; 1990) suggest that once adults get beyond

the decoding stage of reading, their comprehension of language can be assessed equally well by listening as by

reading (see the Appendix to this paper). All of the studies of the use of telephone surveys demonstrated that they

could assess vocabulary knowledge by listening, and these assessments correlated well (r’s in the +.80 range) with

performance by reading in face-to-face or mail out assessments.

But given that neither the actual reading of language nor the use of graphics displays occurred in the

telephone studies reviewed above, the question remains as to whether or not the information obtained by telephone

surveys can provide the same range of useful information that was obtained in the door-to-door NALS. This was the

question posed earlier in this paper that addresses the action or use validity (Messick 1989) of the telephone

methodology. The answer to this question is discussed next in the context of the analysis of the cost-effectiveness of

the telephone survey methodology used by Sticht and associates (Sticht, Hofstetter, and Hofstetter 1996; Hofstetter,

Sticht and Hofstetter 1999) in contrast to door-to-door sampling for producing the range of informational products

desired by the U.S. Congress when it commissioned the national survey.

Part 4. Issues in the Use of Telephone and Mail Surveys in

Assessing Adult Literacy

Regarding the cost of telephone surveys, in addition to sponsoring the collection of academic skill

achievement data by direct testing in the United States, the National Center for Education Statistics (NCES) has an

office for conducting telephone surveys. According to Dr. Kathryn Chandler of the NCES, costs per interview by

telephone are a fraction of the costs of conducting door-to-door interviews (K. Chandler—personal communication,

April, 1994). Chandler estimated that a sample of six thousand telephone interviews with adults might easily be

achieved for less than $1 million. This suggests that to interview a national representative sample of 13,600 adults,

with an oversampling of black and Hispanic households, as was done in the 1992 NALS (Kirsch et al. 1993), the cost

would be approximately $2 million or less. This compares to the approximately $10.85 million cost of obtaining the

national sample for the NALS.
29

The concept of cost-effectiveness requires that both the cost and the effectiveness of alternative methods to

achieve the same or very similar outcomes be considered. There is no question that the telephone survey

methodology is a lower cost approach to interviewing than door-to-door canvassing. Many may, however, question

whether the telephone methodology yields inferences that are as valid (useful) for characterizing the literacy of the

adult population as those derived from the actual performance of literacy tasks, as in the 1992 NALS. Fortunately,

the report of the design of the NALS provides a list of outcomes, uses, or “benefits” that the NALS was intended to

provide (Campbell, Kirsch, and Kolstad 1992, pp. 2-3). This makes it possible to conduct an action or use validity

analysis to determine if the same or comparable outcomes as were produced by the NALS were also obtained by the

telephone survey approach of Sticht, Hofstetter, and Hofstetter (1996).

The following discussion first states a given informational product that the NALS was intended to provide

(Campbell, Kirsch, and Kolstad 1992). This is followed by a discussion of what the NALS and the telephone survey

method provided for the informational product. The NCES survey design report stated that the NALS would

accomplish the following:

(1) Describe the levels of literacy demonstrated by the total adult population as well as by adults

comprising various subgroups, including those targeted as “at risk.”

The NALS developed three groups of tasks called prose, document and quantitative literacy, administered

the tasks to samples of adults, and used the tasks to scale both the adults’ literacy proficiencies on each of the three

scales and the difficulty levels of the tasks using Item Response Theory. The difficulty level of each task was defined

as the level of literacy needed to have “... an 80 percent probability of correct response.” (Kirsch, et al. 1993, 71).

Next, subjects were assigned to one of five literacy levels based on their proficiency scores.

How well do these procedures characterize the literacy skills of adults? First, the rationale for the decision

to scale the adults’ literacy proficiency using a probability of .80 of being able to perform a given task, when the

probability could have been set at 70, 60 or any other percent, was not discussed (Kirsch et al. 1993). As Kolstad has

recently noted (Kolstad, 1996; Kolstad, et al. 1998), changes in this percentage clearly change the outcomes of the

analyses. Second, the decision to assign people to five levels of literacy proficiency based on their being able to

perform 80 percent of the average tasks at a given level means that any competence to perform at higher levels was

not “credited” to the adults. For instance, people who scored on average in Level 1 could also perform 50 percent of
30

Level 2 tasks, 25 percent of the Level 3 tasks, 20 percent of the Level 4 and one in six of the Level 5 tasks. Given

these findings, it is difficult to ascertain what the literacy levels of adults “really” are.

The telephone survey methodology also permits people to be assigned to levels based on the knowledge

component of literacy. Levels of literacy knowledge can be established using the same methods as in the NALS, or

using means and standard deviations to establish level ranges. With this approach, the decisions regarding what

probability figures for task performance should be used are eliminated. Further, other available test information

based on normal curve statistics, such as the Armed Services Vocational Aptitude Battery (ASVAB), become

relevant to interpreting the survey data.

(2) Characterize the demonstrated literacy skills in terms of demographic and personal background

information.

Major demographic data collected by the NALS have also been collected in telephone surveys. Whether the

NALS-type of graphic displays or the telephone knowledge assessments are used, representations of literacy

proficiency were positively and consistently related to education, age, ethnicity, income, occupational status, father

and mother’s education, extent of engagement in literacy practices such as newspaper and magazine reading, and

self-ratings of reading and writing abilities. These same relationships have been reported for the last seventy-five

years in every major mass assessment of adult “intelligence,” “aptitude,” or “literacy” (Sticht and Armstrong 1994).

These types of demographic/ knowledge/practice data can be obtained at much less cost by telephone surveys.

(3) Characterize the work force of the country with respect to demonstrated literacy skills and activities

reported by individuals in various occupational categories.

The door-to-door and the telephone surveys both characterize the literacy proficiencies and practices of

adults in major occupational groups. Both types of surveys produce similar results: laborers are not as proficient as

clerical workers who are not as proficient as managers and professionals. These findings have been consistently

found in numerous adult cognitive abilities assessments since the introduction by the U.S. Army of mass testing

during World War I (Sticht and Armstrong 1994). The telephone survey method collects occupational data much less

expensively than the door-to-door survey method.
31

(4) Provide an increased understanding of the skills and knowledge associated with functioning in a

technological society.

Probably the most important question that the NALS researchers were asked to report on was, “Are the

literacy skills of America’s adults adequate ... to ensure individual opportunities for all adults, to increase worker

productivity, or to strengthen America’s competitiveness around the world?” (Kirsch et al. 1993, xviii). The NALS

report answered the question as, “Because it is impossible to say precisely what literacy skills are essential for

individuals to succeed in this or any other society, the results of the National Adult Literacy Survey provide no firm

answers to such questions” (Kirsch et al. 1993, xviii). The authors discussed the relation of being in the lower

literacy levels to one’s social standing as indicated by more limited occupational opportunities, income, and so forth.

From this, readers were invited to make inferences about how lower literacy skills may tend to limit one’s

functioning in society. But all of these same relationships are readily studied by the telephone survey methodology at

a lower cost and with fewer decisions having little or no rationale presented.

It might even be argued that the assessment of knowledge (by checklists and/or other methods) is a more

useful method for characterizing the “skills and knowledge associated with functioning in a technological society.”

Several studies have demonstrated that high levels of “prior” or “background” knowledge in a specific domain can

compensate for several “years” of “general” reading skill (Recht and Leslie 1988; Sticht et al. 1986). The NALS

study itself concluded that high levels of “prior” or “background” knowledge about what one reads is prerequisite for

comprehending at a high level across the wide range of tasks in the battery.

For these reasons, it seems likely that assessments of knowledge might better predict the performance of

“real world” literacy tasks than the NALS-type scales. As illustrated by the research reviewed above, the Armed

Services have spent decades and tens of millions of dollars on the Armed Services Vocational Aptitude Battery

(ASVAB), consisting of ten tests, all of which require some reading and eight of which are tests primarily involving

general and special vocabulary and conceptual knowledge (e.g., knowledge of geometry, electronics, automobiles,

etc.) (Sticht and Armstrong 1994). A composite of four of these knowledge tests forms the Armed Forces

Qualification Test (AFQT) used to select applicants for military service. As noted in Part 2 (above), a telephone

survey methodology using just vocabulary (word knowledge) assessment was able to estimate the AFQT scores of

personnel with just a 10 to 11 item test using an adaptive testing algorithm. Since the AFQT is used to determine

ones qualification for military service, this supports the position that knowledge assessment can serve to identify
32

those who can use printed and written materials to function in society, at least in the high-technology world of the

armed services.

(5) Interpret the findings related to information-processing skills and strategies in a way that can inform

curriculum decisions pertaining to the education and training of adults.

In addition to wanting information about adults’ literacy skills, Congress wanted information regarding

instructional “remedies” that might be taken to improve adults’ literacy skills. The developers of the NALS

suggested that adult basic skills programs should be geared to improving adults’ skills in prose, document and

quantitative (PDQ) literacy (Mosenthal and Kirsch 1994). Indeed, researchers at the Educational Testing Service

worked on an interactive video, computer-based instructional series that would teach document literacy skills. A

small pilot study with a group of some ten to twelve adult basic skills students indicated that, while students made

improvements in document literacy, they made three to four times as much gains on prose and quantitative literacy

tests as on the document literacy tests. This led the instructor who administered the pilot course to observe that, “The

gains were interesting considering the PDQ curriculum did not include instruction in these skills.” (Orr-Holley 1992,

1).

Yet, as noted earlier, the human cognitive system model and the developmental model of literacy suggest

that the three scales are not separate areas of competence. Rather, they draw upon the same underlying cognitive

system with its knowledge base and working memory processes. For this reason, it is to be expected that many

language and processes developed in one domain may be available to other domains because they all draw on the

same cognitive system. The results of the pilot instructional program would seem consistent with this point of view.

The research reviewed above adds support to Stanovich’s (1993) socially relevant work on the origins of

“verbal intelligence.” The theory behind the knowledge-based approach to literacy assessment used in the telephone

and checklist survey is the practice-engagement theory of literacy development (Reder 1994(b); Stanovich 1993;

Krashen 1993). This theory holds that, by engaging in extensive practices involving a wide range of reading, literates

build vast bodies of knowledge (both declarative and procedural) and automaticity of word recognition that in turn

make it possible to engage in and successfully complete a large number of literacy tasks. This suggests one simple

recommendation for curriculum development for adult literacy programs from the telephone and checklist

methodology of Sticht, Hofstetter, and Hofstetter (1996) and Hofstetter, Sticht and Hofstetter, (1999). To help

people develop large bodies of knowledge and hence to become highly literate, literacy programs should arrange
33

conditions that will encourage students to engage in extensive, wide-ranging, substantive listening and reading over

long periods of time. This seems to be the sort of thing that participation in long term education does.

Sampling and Other Issues

The use of sampling techniques raises questions about how well telephone samples represent the adult

population in general, and particular segments in particular. Some may question the validity of the inferences derived

from the telephone methodology for accurately representing the literacy of adults because some adults do not have

telephones, and they are likely to be the least educated for whom literacy is a major concern. But sampling issues

plague all surveys, including the NALS and telephone surveys (Dillman 1978; Frey 1989). Typically, demographic

data from the census can be obtained to permit quite accurate extrapolations of telephone estimates to households

without telephones and for the least educated. In fact, as noted earlier, the U.S. National Center for Education

Statistics already has an office that uses biennial telephone surveys to obtain information about adult education for

national policy purposes. The same sampling methods can be used to acquire a representative sample of adults for

literacy assessment (K. Chandler—personal communication, April, 1994).

Additionally, special populations in correctional facilities or in adult literacy education programs might be

sampled using both face-to-face, paper-and-pencil surveys and telephone surveys to establish correspondences

among telephone knowledge- and self-rating-based estimates of literacy.

Another issue is possible public resistance to the telephone methodology for assessing adult literacy because

there is no actual reading or writing involved. In this regard, it should be noted that Albert Shanker, late President of

the American Federation of Teachers union, devoted his entire Sunday New York Times column of January 14,

1996, to a report on the importance of developing knowledge in the public schools based on the telephone study of

Sticht, Hoffstetter, and Hoffstetter (1996). This suggests a degree of popular acceptance of the telephone and

checklist methodology for assessing literacy.

Additionally, as the research reviewed indicates, mail surveys may be used along with telephone surveys to

make it possible to assess adults’ use of written materials and gain acceptance of the telephone methodology for

assessing literacy. It is not necessary to limit assessments to the use of the checklist methodology. To avoid the types

of criticisms of “cultural imperialism” that faced Hirsch (1987), for instance, a kit of materials might be mailed to a

sub-sample of persons contacted by telephone and then an 800 telephone number could be used for the kit recipient
34

to call in to have a set of directions for performing various tasks with the written materials. Or there could be a callback time for the telephone test administrator to contact the kit recipient. For instance, if the kit included a small

book, then the interviewer might call and ask the respondent to turn to a given page and read a specified sentence

aloud or to summarize the main idea of a paragraph. The book could include stories or information representative of

many different cultural groups and the same methods used by the NALS, IALS and the forthcoming NAAL for

ensuring acceptance by various cultural groups could be used in the design of the book or kit of materials.

If a page of the small book contained a train schedule, then the examiner might ask the respondent the same

sort of working memory loaded tasks involved in search and locate questions used with the document items of the

NALS.

Additionally, reading-related assessments such as phonemic awareness knowledge might be assessed via

telephone using phonemic blending or segmentation tests. With a mail- out kit, the adult might be asked to read

aloud a list of pronounceable nonsense “words.” Such techniques as the “book comprehension” mail-out and

telephone methodology coupled with assessments of decoding skills might be especially useful in generating new

information about the instructional needs of adults in NALS/IALS/NAAL Levels 1 and 2.

Of course, if used as a supplement to a decennial adult literacy door-to-door assessment, then a telephone

and/or mail-out survey methodology and instruments could provide information that has been correlated with NALStype items in addition to other, knowledge based items to track adults’ literacy during years in between the decennial

assessments, perhaps as a part of the NCES regular biennial adult education surveys that are presently conducted.

This would couple the telephone methodology with the NAAL methodology and permit the tracking of change over

time using the NAAL approach every ten years and the telephone/mail approach in between and every ten years.

This paper was prepared under a contract with the Education Statistics Service Institute (ESSI) with funding

from the U.S. Department of Education, National Center for Education Statistics. The opinions and conclusions

stated are those of the author and do not necessarily represent the opinions, policies or positions of the ESSI or the

U.S. Department of Education, National Center for Education Statistics.
35

References

Allen, L., Cipielewski, J., and Stanovich, K. E. (1992). Multiple Indicators of Children’s Reading Habits and

Attitudes: Construct Validity and Cognitive Correlates. Journal of Educational Psychology, 84: 489–503.

Atkinson, R. C., and Shiffrin, R. M. (1971). The Control of Short-term Memory. Scientific American, 225: 82–90.

Benton, L., and Noyelle, T. (Eds.) (1992). Adult Illiteracy and Economic Performance. Paris, France: Organisation

for Economic Co-operation and Development, Centre for Educational Research and Innovation.

Bernstein, D., Roy, E., Srull, T., and Wickens, C. (1988). Psychology. Boston, MA: Houghton Mifflin Company.

Bransford, J., Sherwood, R., Vige, N., and Rieser, J. (1986). Teaching Thinking and Problem Solving. American

Psychologist, 41: 1078–1089.

Bruner, J. (1968). Processes of Cognitive Growth: Infancy. Worcester, MA: Clark University Press.

Campbell, A., Kirsch, I. S., and Kolstad, A. (1992). Assessing Literacy: The Framework for the National Adult

Literacy Survey (NCES 92–113). U.S. Department of Education. Washington, DC: U.S. Government

Printing Office.

CASAS (1983). The CASAS Manual for Item Writing. San Diego, CA: Comprehensive Adult Student Assessment

System.

Chen, R., and Vellutino, F. (1997). Prediction of Reading Ability: A Cross-Validation of the Simple View of

Reading. Journal of Literacy Research, 29: 1–24.

Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method. New York, NY: John Wiley and

Sons.

Fagan, W. T. (1994). Adult Literacy Surveys: A Trans-border Comparison. Journal of Reading, 38: 260-269.

Frey, J. H. (1989). Survey Research by Telephone (2nd ed.). Newbury Park, NY: Sage Publications.

Goody, J. (1977). The Domestication of the Savage Mind. Cambridge, England: Cambridge University Press.

Gough, P. (1996). How children learn to read and why they fail. Annals of Dyslexia, 46: 3–20.

Gottfredson, L. (1997). Why g Matters: the Complexity of Everyday Life. Intelligence, 24: 79–132.

Guthrie, J. T., Britten, T., and Barker, K. G. (1991). Role of Document Structure and Metacognitive Awareness in

the Cognitive Process of Searching for Information. Reading Research Quarterly, 226: 300–324.

Harris, R. (1986). The Origin of Writing. London, England: Duckworth.
36

Healy, A. F., and McNamara, D. S. (1996). Verbal Learning and Memory: Does the Modal Model Still Work? In J.

Spence, J. Darley, and D. Foss (Eds.), Annual Review of Psychology, 47: 143–172. Palo Alto, CA: Annual

Reviews, Inc.

Hirsch, Jr., E. D. (1987). Cultural Literacy: What Every American Needs to Know. Boston, MA: Houghton Mifflin

Company.

Hofstetter, R., Sticht, T., and Hofstetter, C. (1999). Knowledge, Literacy, and Power. Communication Research, 26:

58–80.

Kazemek, F. E. (1990). Adult Literacy Education: Heading into the 1990s. Adult Education Quarterly, 41: 53–62.

Kent, J., and Plomin, R. (1987). Testing Specific Cognitive Abilities by Telephone and Mail. Intelligence, 11: 391–

400.

Kirsch, I., and Jungeblut, A. (1986). Literacy: Profiles of America’s Young Adults. Princeton, NJ: Educational

Testing Service, National Assessment of Educational Progress.

Kirsch, I. S., and Mosenthal, P. B. (1990). Exploring Document Literacy: Variables Underlying the Performance of

Young Adults. Reading Research Quarterly, 25: 5–30.

Kirsch, I., Jungeblut, A., and Mosenthal, P. (1998). Adult Literacy in OECD Countries: Technical Report on the

First International Adult Literacy Survey (NCES 98–053). U.S. Department of Education. Washington,

DC: U.S. Government Printing Office.

Kirsch, I., Jungeblut, A., Jenkins, L., and Kolstad, A. (1993, September). Adult Literacy in America: A First Look at

the Results of the National Adult Literacy Survey (NCES 93–275). U.S. Department of Education.

Washington, DC: U.S. Government Printing Office.

Kolstad, A. (1996). The Response Probability Convention Embedded in Reporting Prose Literacy Levels from the

1992 National Adult Literacy Survey. U.S. Department of Education. Washington, DC: National Center for

Education Statistics.

Kolstad, A., Cohen, J., Baldi, S., Chan, T., DeFur, E., and Angeles. (1998). The Response Probability Convention

Used in Reporting Data from IRT Assessment Scales: Should NCES Adopt a Standard? U.S. Department of

Education. Washington, DC: National Center for Education Statistics.

Krashen, S. (1993). The Power of Reading: Insights from the Research. Englewood, CO: Libraries Unlimited.

Kyllonen, P. C., and Christal, R. E. (1990). Reasoning Ability is (little more than) Working-memory Capacity?!.

Intelligence, 14: 389–433.
37

Legree, P., Fischl, M., Gade, P., and Wilson, M. (1998). Testing Word Knowledge by Telephone to Estimate

General Cognitive Aptitude Using an Adaptive Test. Intelligence, 26: 91–98.

Liberman, I., Shankweiler, D., and Liberman, A. (1989). The Alphabetic Principle and Learning to Read. In D.

Shankweiler and I. Liberman (Eds.). Phonology and Reading Disability: Solving the Reading Puzzle. Ann

Arbor, MI: The University of Michigan Press.

McGuinness, D. (1997). Why Our Children Can’t Read and What We Can Do About It. New York, NY: The Free

Press.

Messick, S. (1989). Meaning and Values in Test Validation: The Science and Ethics of Assessment. Educational

Researcher, 18: 5–11.

Messick, S. (1988). The Once and Future Issues of Validity: Assessing the Meaning and Consequences of

Measurement. In H. Wainer and H. I. Braun (Eds.), Test Validity (pp. 33–45). Hillsdale, NJ: Erlbaum.

Meyer, B. J. F., Marsiske, M, and Willis, S. L. (1993). Text Processing Variables Predict the Readability of

Everyday Documents Read by Older Adults. Reading Research Quarterly, 28: 235–248.

Morris, W. (Ed.). (1976). The American Heritage Dictionary of the English Language. Boston, MA: Houghton

Mifflin Company.

Mosenthal, P. B., and Kirsch, I. S. (1991). Toward an Exploratory Model of Document Literacy. Discourse

Processes, 14: 147–180.

Mosenthal, P. B., and Kirsch, I. S. (1994, December). Defining the Proficiency Standards of Adult Literacy in the

U.S.: A Profile Approach. Paper presented at the annual meeting of the National Reading Conference.

Syracuse, NY: Syracuse University.

Neice, D., Adsett, M., and Rodney, W. (1992). Direct Versus Proxy Measures of Adult Functional Literacy: A

Preliminary Re-examination. In L. Benton and T. Noyelle (Eds.), Adult Illiteracy and Economic

Performance (pp. 69–87). Paris, France: OECD, Centre for Educational Research and Innovation.

Orr-Holley, L. (1992). Operation PDQ: An ETS Pilot Program. Jacksonville, FL: Florida Community College at

Jacksonville, Urban Resource Center.

Organization for Economic Co-operation and Development. (1995). Literacy, Economy, and Society. Paris, France:

Organisation for Economic Co-operation and Development, Centre for Educational Research and

Innovation.
38

Organization for Economic Co-operation and Development. (1997). Literacy Skills for the Knowledge Society. Paris.

France: Organisation for Economic Co-operation and Development.

Plomin, R., McClearn, G., Pederson, N., and Nesselroade, J. (1986, June). Adopted and Non-adopted Twin

Resemblance for Cognitive Abilities Assessed by Telephone Testing in the Swedish Adoption/Twin Study of

Aging. Paper presented at the annual meeting of the Behavior Genentics Association, Honolulu, HI.

Recht, D. R., and Leslie, L. (1988). Effect of Prior Knowledge on Good and Poor Reader’s Memory of Text. Journal

of Educational Psychology, 80: 16–20.

Reder, S. (1994a). Synthetic Estimates of National Adult Literacy Survey Literacy Proficiencies from 1990 Census

Microdata. U.S. Office of Education, Office of Vocational and Adult Education, Division of Adult

Education and Literacy. Washington, DC: U.S. Government Printing Office.

Reder, S. (1994b). Practice-engagement Theory: A Sociocultural Approach to Literacy Across Languages and

Cultures. In B. M. Ferdman, R-M. Weber, and A. G. Ramirez (Eds.), Literacy Across Languages and

Cultures (pp. 33–74). New York, NY: State University of New York Press.

Rock, D. (1998, January). Validity Generalization of the Assessment Across Countries. In Murray, T., Kirsch, I. and

Jenkins, L. (Eds.), Adult literacy in OECD countries: Technical Report on the First International Adult

Literacy Survey. Washington, DC: National Center for Education Statistics.

Rumelhart, D. (1980). Schemata. In B. Bruce and W. Brewer (Eds.), Theoretical issues in Reading Comprehension.

Hillsdale, NJ: Lawrence Earlbaum.

Schwartz, S. (1971). Modes of Representation and Problem Solving: Well Evolved is Half Solved. Journal of

Experimental Psychology, 2: 347–350.

Sherman, R., Condelli, L., and Koloski, J. (1998). Developing the National Assessment of Adult Literacy:

Recommendations from Stakeholders (NCES 98-17). U.S. Department of Education. Washington, DC:

NCES Working Paper.

Sinatra, G. M. (1990). Convergence of Listening and Reading Processing. Reading Research Quarterly, 25: 115–

130.

Stanovich, K. E. (1991). Discrepancy Definitions of Reading Disability: Has Intelligence Led Us Astray? Reading

Research Quarterly, 26: 7–29.

Stanovich, K. E. (1993). Does Reading Make you Smarter? Literacy and the Development of Verbal Intelligence. In

H. Reese (Ed.), Advances in Child Development and Behavior (vol. 24). New York, NY: Academic Press.
39

Stanovich, K. E., and West, R. F. (1994). Reply to Taylor. Reading Research Quarterly, 29: 290–291.

Stanovich, K. E., and Cunningham, A. E. (1993). Where Does Knowledge Come From? Specific Associations

between Print Exposure and Information Acquisition. Journal of Educational Psychology, 85: 211–229.

Sticht, T. G. (1978). The Acquisition of Literacy by Children and Adults. In F. Murray and J. Pikulsky (Eds.), The

Acquisition of Reading (pp. 131–162). Baltimore, MD: University Park Press.

Sticht, T. G., and Armstrong, W. B. (1994). Adult Literacy in the United States: A Compendium of Quantitative

Data and Interpretive Comments. Washington, DC: National Institute for Literacy.

Sticht, T. G., and James, J. H. (1984). Listening and Reading. In P. Pearson, R. Barr, M. Kamil, and P. Mosenthal

(Eds.), Handbook of reading research (pp. 293–317). New York, NY: Longmans.

Sticht, T. G., and McDonald, B. A. (1992). Teaching Adults to Read. In S. J. Samuels and A. E. Farstrup (Eds.),

What Research Has to Say about Reading Instruction (pp. 314–334). Newark, DE: International Reading

Association.

Sticht, T. G., Hooke, L. R., and Caylor, J. S. (1981). Literacy, Oracy, and Vocational Aptitude as Predictors of

Attrition and Promotion in the Armed Services ( HumRRO FR-ETSD-81-11). Alexandria, VA: Human

Resources Research Organization.

Sticht, T. G., Hooke, L. R., and Caylor, J. S. (1981). Manual for the Administration and Interpretation of the

Literacy Assessment Battery (LAB) (HumRRO FR-ETSD-81-10). Alexandria, VA: Human Resources

Research Organization.

Sticht, T. G., Beck, L., Hauke, R., Kleiman, G., and James, J. (1974). Auding and Reading: A Developmental Model.

Alexandria, VA: Human Resources Research Organization.

Sticht, T. G., Armijo, L. A., Koffman, N., Roberson, K., Weitzman, R., Chang, F., and Moracco, J. (1986).

Teachers, Books, Computers, and Peers: Integrated Communications Technologies for Adult Literacy

Development. Monterey, CA: U.S. Naval Postgraduate School.

Sticht, T. G., Hofstetter, C. R., and Hofstetter, H. H. (1996). Assessing Adult Literacy by Telephone. Journal of

Literacy Research, 28: 525–559.

Taylor, D. (1994). The Trivial Pursuit of Reading Psychology in the “Real World”: A Response to West, Stanovich,

and Mitchell. Reading Research Quarterly, 29: 277–288.

United States General Accounting Office (1995). Adult Education: Measuring Program Results Has Been

Challenging (GAO/HEHS-95-153). Washington, DC: U.S. Government Printing Office.
40

Venezky, R. L., Wagner, D. A., and Ciliberti, B. S. (Eds.) (1990). Toward Defining Literacy. Newark, NJ:

International Reading Association.

West, R. F., Stanovich, K. E., and Mitchell, H. R. (1993). Reading in the Real World and its Correlates. Reading

Research Quarterly: 28, 35–50.
41

Appendix

This appendix summarizes an empirical study of the proposition stated on page 20 of the paper that “once adults get

beyond the decoding stage of reading, their comprehension of language can be assessed equally well by listening as

by reading.”

The manual for the Literacy Assessment Battery (LAB) (Sticht, Hooke, and Caylor 1981(b)) includes data showing

the results of testing five hundred ninety-three applicants for military service using both the Gates-MacGinitie

Reading Test (Level D) and the Listening and Reading Paragraphs subtests of the LAB. The latter was developed for

the U.S. Air Force to measure the ability of adults to comprehend language by listening and reading. The LAB

includes other components, too, but they are not relevant to the present discussion.

With test scores on the LAB Listening and LAB Reading subtests as well as the G-M reading test, made up of

combined vocabulary and paragraph reading subtests, it was possible to develop regression formulas to estimate GM reading scores from LAB Listening and Reading subtest scores. This permits one approach to determining the

extent to which the above proposition is accurate. If it is assumed that the widely used G-M is a valid measure of

language comprehension by reading, then the question is, does the LAB Listening subtest estimate G-M reading in a

similar manner as the LAB Reading subtest?

A maximum score of 24 points was possible on either the Listening or Reading LAB subtests. The figure shows

estimated G-M reading grade levels as a function of raw scores on the LAB paragraph tests for Listening and

Reading.

The correlation of LAB

Listening with Lab Reading was

.749 (56% shared variance).

LAB Reading with G-M

Reading was .804 (65% shared

variance), LAB Listening with

G-M Reading was .744 (55%

shared variance).

By way of comparison, the

G-M correlated with the Nelson

Denny Reading test .696 (48%

shared variance) for a sample of

1,020 applicants for military

service. Thus the LAB

Listening subtest was more

highly correlated with the G-M

reading test than was the Nelson

Denny reading test.

The data from the LAB, G-M,

and Nelson Denny testing are

consistent with the proposition

that “once adults get beyond the

decoding stage of reading, their

comprehension of language can

be assessed equally well by

listening as by reading.”
This page intentionally left blank.
Listing of NCES Working Papers to Date

Working papers can be downloaded as pdf files from the NCES Electronic Catalog

(http://nces.ed.gov/pubsearch/). You can also contact Sheilah Jupiter at (202) 502-7444

(sheilah_jupiter@ed.gov) if you are interested in any of the following papers.

Listing of NCES Working Papers by Program Area

No. Title NCES contact

Baccalaureate and Beyond (B&B)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Beginning Postsecondary Students (BPS) Longitudinal Study

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Common Core of Data (CCD)

95-12 Rural Education Data User’s Guide Samuel Peng

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Decennial Census School District Project

95-12 Rural Education Data User’s Guide Samuel Peng

96-04 Census Mapping Project/School District Data Book Tai Phan

98-07 Decennial Census School District Project Planning Report Tai Phan

Early Childhood Longitudinal Study (ECLS)

96-08 How Accurate are Teacher Judgments of Students’ Academic Performance? Jerry West

96-18 Assessment of Social Competence, Adaptive Behaviors, and Approaches to Learning with

Young Children

Jerry West

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Education Finance Statistics Center (EDFIN)

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

High School and Beyond (HS&B)

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

HS Transcript Studies

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson
No. Title NCES contact

International Adult Literacy Survey (IALS)

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Integrated Postsecondary Education Data System (IPEDS)

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

National Assessment of Adult Literacy (NAAL)

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door

Surveys in the Assessment of Adult Literacy

Sheida White

National Assessment of Educational Progress (NAEP)

95-12 Rural Education Data User’s Guide Samuel Peng

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Steven Gorman

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Steven Gorman

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Steven Gorman

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questionnaires)

Steven Gorman

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Steven Gorman

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

National Education Longitudinal Study of 1988 (NELS:88)

95-04 National Education Longitudinal Study of 1988: Second Follow-up Questionnaire Content

Areas and Research Issues

Jeffrey Owings

95-05 National Education Longitudinal Study of 1988: Conducting Trend Analyses of NLS-72,

HS&B, and NELS:88 Seniors

Jeffrey Owings

95-06 National Education Longitudinal Study of 1988: Conducting Cross-Cohort Comparisons

Using HS&B, NAEP, and NELS:88 Academic Transcript Data

Jeffrey Owings

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings
No. Title NCES contact

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

National Household Education Survey (NHES)

95-12 Rural Education Data User’s Guide Samuel Peng

96-13 Estimation of Response Bias in the NHES:95 Adult Education Survey Steven Kaufman

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-21 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School

Readiness, and School Safety and Discipline

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

96-29 Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the

1995 National Household Education Survey (NHES:95)

Kathryn Chandler

96-30 Comparison of Estimates from the 1995 National Household Education Survey

(NHES:95)

Kathryn Chandler

97-02 Telephone Coverage Bias and Recorded Interviews in the 1993 National Household

Education Survey (NHES:93)

Kathryn Chandler

97-03 1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener,

NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education

Kathryn Chandler

97-04 Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in

the 1993 National Household Education Survey (NHES:93)

Kathryn Chandler

97-05 Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National

Household Education Survey (NHES:93)

Kathryn Chandler

97-06 Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National

Household Education Survey (NHES:95)

Kathryn Chandler

97-08 Design, Data Collection, Interview Timing, and Data Editing in the 1995 National

Household Education Survey

Kathryn Chandler

97-19 National Household Education Survey of 1995: Adult Education Course Coding Manual Peter Stowe

97-20 National Household Education Survey of 1995: Adult Education Course Code Merge

Files User’s Guide

Peter Stowe

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

97-28 Comparison of Estimates in the 1996 National Household Education Survey Kathryn Chandler

97-34 Comparison of Estimates from the 1993 National Household Education Survey Kathryn Chandler

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

97-38 Reinterview Results for the Parent and Youth Components of the 1996 National

Household Education Survey

Kathryn Chandler

97-39 Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996

National Household Education Survey

Kathryn Chandler

97-40 Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996

National Household Education Survey

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

National Longitudinal Study of the High School Class of 1972 (NLS-72)

95-12 Rural Education Data User’s Guide Samuel Peng

National Postsecondary Student Aid Study (NPSAS)

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio
No. Title NCES contact

National Study of Postsecondary Faculty (NSOPF)

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Private School Universe Survey (PSS)

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

96-26 Improving the Coverage of Private Elementary-Secondary Schools Steven Kaufman

96-27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 Steven Kaufman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Recent College Graduates (RCG)

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

Schools and Staffing Survey (SASS)

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American

Statistical Association

Dan Kasprzyk

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) Dan Kasprzyk

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report Dan Kasprzyk

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher

Transcript Study, Schools and Staffing Survey

Dan Kasprzyk

94-06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related

Surveys

Dan Kasprzyk

95-01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American

Statistical Association

Dan Kasprzyk

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing

QED School Estimates with CCD Estimates

Dan Kasprzyk

95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis Dan Kasprzyk

95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates Dan Kasprzyk

95-09 The Results of the 1993 Teacher List Validation Study (TLVS) Dan Kasprzyk

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive

Reconciliation

Dan Kasprzyk

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

95-12 Rural Education Data User’s Guide Samuel Peng

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and

Their Applicability for the Teacher Follow-up Survey

Sharon Bobbitt

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman

95-18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and

Staffing Survey

Dan Kasprzyk

96-01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly

Longitudinal Study

Dan Kasprzyk

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting

of the American Statistical Association

Dan Kasprzyk

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey Dan Kasprzyk

96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to

Inform Broad Education Policy

Dan Kasprzyk

96-07 Should SASS Measure Instructional Processes and Teacher Effectiveness? Dan Kasprzyk

96-09 Making Data Relevant for Policy Discussions: Redesigning the School Administrator

Questionnaire for the 1998-99 SASS

Dan Kasprzyk

96-10 1998-99 Schools and Staffing Survey: Issues Related to Survey Depth Dan Kasprzyk
No. Title NCES contact

96-11 Towards an Organizational Database on America’s Schools: A Proposal for the Future of

SASS, with comments on School Reform, Governance, and Finance

Dan Kasprzyk

96-12 Predictors of Retention, Transfer, and Attrition of Special and General Education

Teachers: Data from the 1989 Teacher Followup Survey

Dan Kasprzyk

96-15 Nested Structures: District-Level Data in the Schools and Staffing Survey Dan Kasprzyk

96-23 Linking Student Data to SASS: Why, When, How Dan Kasprzyk

96-24 National Assessments of Teacher Quality Dan Kasprzyk

96-25 Measures of Inservice Professional Development: Suggested Items for the 1998-1999

Schools and Staffing Survey

Dan Kasprzyk

96-28 Student Learning, Teaching Quality, and Professional Development: Theoretical

Linkages, Current Measurement, and Recommendations for Future Data Collection

Mary Rollefson

97-01 Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the

American Statistical Association

Dan Kasprzyk

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

97-10 Report of Cognitive Research on the Public and Private School Teacher Questionnaires

for the Schools and Staffing Survey 1993-94 School Year

Dan Kasprzyk

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-12 Measuring School Reform: Recommendations for Future SASS Data Collection Mary Rollefson

97-14 Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and

Analysis

Steven Kaufman

97-18 Improving the Mail Return Rates of SASS Surveys: A Review of the Literature Steven Kaufman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

97-23 Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing

Form

Dan Kasprzyk

97-41 Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting

of the American Statistical Association

Steve Kaufman

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

98-05 SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors

Steven Kaufman

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

98-12 A Bootstrap Variance Estimator for Systematic PPS Sampling Steven Kaufman

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

98-14 Variance Estimation of Imputed Survey Data Steven Kaufman

98-15 Development of a Prototype System for Accessing Linked NCES Data Steven Kaufman

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest

Results to Improve Item Construction

Dan Kasprzyk

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

1999-12 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use

Codebook

Kerry Gruber

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk
Listing of NCES Working Papers by Subject

No. Title NCES contact

Adult education

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult

Education Component

Steven Kaufman

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education

Survey

Peter Stowe

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Adult literacy—see Literacy of adults

American Indian – education

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of

Indian Affairs (BIA) Restricted-Use Codebook

Kerry Gruber

Assessment/achievement

95-12 Rural Education Data User’s Guide Samuel Peng

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? Larry Ogle

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable

Assessment Results

Larry Ogle

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational

Progress

Larry Ogle

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background

Questions)

Larry Ogle

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items Larry Ogle

97-44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using

State Assessments and State NAEP, Feasibility Study

Michael Ross

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Beginning students in postsecondary education

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Civic participation

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Climate of schools

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used

in NCES Surveys

Samuel Peng

Cost of education indices

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Course-taking

95-12 Rural Education Data User’s Guide Samuel Peng
No. Title NCES contact

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson

Crime

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Curriculum

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Customer service

1999-10 What Users Say About Schools and Staffing Survey Publications Dan Kasprzyk

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Data quality

97-13 Improving Data Quality in NCES: Database-to-Report Process Susan Ahmed

Data warehouse

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Design effects

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Dropout rates, high school

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and

NELS:88 Sophomore Cohort Dropouts

Jeffrey Owings

Early childhood education

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early

Childhood Education, and Adult Education

Kathryn Chandler

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early

Childhood Program Participation, and Adult Education

Kathryn Chandler

97-24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood

Programs: A Review and Recommendations for Future Research

Jerry West

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Educational attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

Educational research

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

Employment

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico
No. Title NCES contact

Faculty – higher education

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Finance – elementary and secondary schools

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

96-19 Assessment and Analysis of School-Level Expenditures William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Finance – postsecondary

97-27 Pilot Test of IPEDS Finance Survey Peter Stowe

Finance – private schools

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

Geography

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

Imputation

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Inflation

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

Institution data

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Instructional resources and practices

95-11 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of

Recent Work

Sharon Bobbitt &

John Ralph

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

International comparisons

97-11 International Comparisons of Inservice Professional Development Dan Kasprzyk

97-16 International Education Expenditure Comparability Study: Final Report, Volume I Shelley Burns

97-17 International Education Expenditure Comparability Study: Final Report, Volume II,

Quantitative Analysis of Expenditure Comparability

Shelley Burns

Libraries

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers

Presented at Meetings of the American Statistical Association

Carrol Kindel

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

Limited English Proficiency

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser
No. Title NCES contact

Literacy of adults

98-17 Developing the National Assessment of Adult Literacy: Recommendations from

Stakeholders

Sheida White

1999-09a 1992 National Adult Literacy Survey: An Overview Alex Sedlacek

1999-09b 1992 National Adult Literacy Survey: Sample Design Alex Sedlacek

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates Alex Sedlacek

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments Alex Sedlacek

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates Alex Sedlacek

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy

Levels

Alex Sedlacek

1999-09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability

Convention

Alex Sedlacek

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

2000-05 Secondary Statistical Modeling With the National Assessment of Adult Literacy:

Implications for the Design of the Background Questionnaire

Sheida White

2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door

Surveys in the Assessment of Adult Literacy

Sheida White

Literacy of adults – international

97-33 Adult Literacy: An International Perspective Marilyn Binkley

Mathematics

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test

Results to Improve Item Construction

Dan Kasprzyk

Parental involvement in education

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

97-25 1996 National Household Education Survey (NHES:96) Questionnaires:

Screener/Household and Library, Parent and Family Involvement in Education and

Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement

Kathryn Chandler

1999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West

Participation rates

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks

and Empirical Studies

Peter Stowe

Postsecondary education

1999-11 Data Sources on Lifelong Learning Available from the National Center for Education

Statistics

Lisa Hudson

Postsecondary education – persistence and attainment

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Postsecondary education – staff

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

Private schools

96-16 Strategies for Collecting Finance Data from Private Schools Stephen Broughman

97-07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary

Schools: An Exploratory Analysis

Stephen Broughman

97-22 Collection of Private School Finance Data: Development of a Questionnaire Stephen Broughman
No. Title NCES contact

Projections of education statistics

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates Aurora D’Amico

Public school finance

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model

Approach

William J. Fowler, Jr.

Public schools

97-43 Measuring Inflation in Public School Costs William J. Fowler, Jr.

98-01 Collection of Public School Expenditure Data: Development of a Questionnaire Stephen Broughman

98-04 Geographic Variations in Public Schools’ Costs William J. Fowler, Jr.

1999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results Dan Kasprzyk

Public schools – secondary

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in

Mathematics for High School Graduates—An Examination of Data from the National

Education Longitudinal Study of 1988

Jeffrey Owings

Reform, educational

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and

Issues

Jeffrey Owings

Response rates

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report Steven Kaufman

School districts, public

98-07 Decennial Census School District Project Planning Report Tai Phan

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

School districts, public – demographics of

96-04 Census Mapping Project/School District Data Book Tai Phan

Schools

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young

Schools – safety and discipline

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Software evaluation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

Staff

97-42 Improving the Measurement of Staffing Resources at the School Level: The Development

of Recommendations for NCES for the Schools and Staffing Survey (SASS)

Mary Rollefson

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Staff – higher education institutions

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler

State

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection,

Processing, and Editing Cycle

Beth Young
No. Title NCES contact

Statistical methodology

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But

Thought You Could Never Understand

Susan Ahmed

Students with disabilities

95-13 Assessing Students with Disabilities and Limited English Proficiency James Houser

Survey methodology

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report Andrew G. Malizio

97-15 Customer Service Survey: Common Core of Data Coordinators Lee Hoffman

97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996

National Household Education Survey

Kathryn Chandler

98-06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second

Follow-Up: Final Methodology Report

Ralph Lee

98-11 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field

Test Report

Aurora D’Amico

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey Stephen Broughman

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey Stephen Broughman

1999-17 Secondary Use of the Schools and Staffing Survey Data Susan Wiley

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps Valena Plisko

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Teachers

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey Steven Kaufman

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook Kerry Gruber

Teachers – instructional practices of

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – opinions regarding safety

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper Dan Kasprzyk

Teachers – performance evaluations

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – qualifications of

1999-04 Measuring Teacher Qualifications Dan Kasprzyk

Teachers – salaries of

94-05 Cost-of-Education Differentials Across the States William J. Fowler, Jr.

Variance estimation

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing

Variances from NCES Data Sets

Ralph Lee

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and

1999 AAPOR Meetings

Dan Kasprzyk

Violence

97-09 Status of Data on Crime and Violence in Schools: Final Report Lee Hoffman

Vocational education

95-12 Rural Education Data User’s Guide Samuel Peng

1999-05 Procedures Guide for Transcript Studies Dawn Nelson

1999-06 1998 Revision of the Secondary School Taxonomy Dawn Nelson