INTRODUCTION

In 1995, approximately 65 million Americans participated in elementary, secondary, or postsecondary education in the United States (U.S. Department of

Education 1995b). Young people spent from one-fourth to one-half of their waking hours in school and school-related activities, while Americans of all ages continued to pursue some form of active learning that added to their repertoire of

knowledge and skills. To serve these students, the nation’s schools, colleges, and

universities directly employed some 11.2 million people, even more than in the

health industry. In addition, these direct services supported a substantial number

of additional jobs in companies serving education through the production of

everything from buses and computers to textbooks and software. All told, the

nation spent more than $500 billion1 on formal education, or approximately 7.4

percent of Gross Domestic Product (U.S. Department of Commerce 1995).

Probably no other single activity has occupied so prominent a place in family,

community, and working life.

The primary purpose of the

National Center for Education

Statistics (NCES) is to describe

this education enterprise and

inform the nation about it.

Congress charges NCES with collecting and reporting “statistics

and information showing the condition and progress of education

in the United States and other

nations in order to promote and

accelerate the improvement of

American education.”2 In doing

so, the Center conducts a range of

ongoing national surveys examining early childhood education,

elementary and secondary education, postsecondary education,

adult literacy, and the nation’s

libraries. Further, in cooperation with many other countries, it supports international surveys that aid in comparing educational progress and processes across nations.

NCES carries out numerous analyses of these data and annually prepares more than

From Data to Information 1

PERCENTAGE OF THE POPULATION ENROLLED IN

SCHOOL, BY AGE, OCTOBER 1993

100

80

60

40

20

0

3 5 7 10 13 16 19 22 25 28 31 34

Percent

Age

SOURCE: U.S. Department of Education, National Center for Education

Statistics, The Condition of Education 1995, Washington, D.C., 26–27.
100 reports targeted toward policymakers, educators, researchers, and the American people.

Doing this job well would be necessary no matter

what the focus, but when the subject assumes the

magnitude and importance of education, this

responsibility takes on special significance.

Therefore, NCES must ensure that it continues to

describe education fully and accurately and that it

performs this function efficiently and thoroughly—

in other words, that it remains well informed about

important education issues, key advances in methods, and new developments in the technology of collecting, managing, analyzing,

and reporting large amounts of information.

To this end, NCES undertook an in-depth examination of how best to direct its

responsibilities for collecting and reporting information on education over the next

decade. Three principal questions guided this effort:

1) What are the major issues and trends in education that NCES should aim

to address through the first decade of the next century?

2) What are the most important advances in methods for collecting and analyzing information that should guide how NCES surveys are designed

and used? 

3) What opportunities do technological advances in data management and 

communications present for improving data collection and analysis and

for disseminating findings and information effectively? 

To help answer these questions, NCES conducted four activities: 1) a survey of leading educators and researchers, asking them to answer one or more of the three

questions listed above; 2) commissioned papers addressing key topics suggested by

the survey results; 3) a conference where the authors of the commissioned papers

presented their work, with subsequent discussion by NCES staff and external

reviewers; and 4) a published volume, From Data to Information: New Directions for the

National Center for Education Statistics, of the commissioned papers and discussants’

comments. This paper summarizes and synthesizes the results of this work and

consists of four major sections. This first introductory section provides a synopsis of

the major themes and conclusions emerging from the papers and the conference.

The second section describes the current foundation of NCES, delineating its core

functions, operating principles, and program of work. The third summarizes some

new directions that NCES could pursue to provide information for policy, research,

2 From Data to Information

ongress charges NCES

with collecting and

reporting “statistics and

information showing the

condition and progress

of education in the

United States and other

nations in order to promote and accelerate the

improvement of

American education.”

C
and practice in American education, and also addresses some important methodological and technological opportunities. The paper ends with a brief conclusion.

Two dominant themes emerged from this collaborative effort. First, NCES

must place greater emphasis on transforming raw data into information useful

to policymakers, educators, researchers, and the general public than it does

today. Accomplishing this goal will require that the relationships between

NCES and data providers and between NCES and data users change significantly. During the next 5 to 10 years, the distinctions among these three parties—NCES, data providers, and data users—will become increasingly blurred,

and their communications will probably become much more interactive, continuous, and two-way, with all three parties actively and simultaneously

engaged in survey design, data collection, analysis, interpretation, presentation, and dissemination. Although technology will help pave the way for this

transition, considerable conceptual thinking will also be required to take full

advantage of the technological opportunities.

Second, in order to be more responsive to the

demands for information about education, NCES

will need to broaden its conception of what constitutes “data” and strategies for their collection.

Traditionally, NCES has concentrated on designing

and conducting surveys asking “closed-ended”

questions that lend themselves to rapid, welldefined quantification. Although such surveys are

likely to remain the hallmark of NCES’s data collection activities for some time, the agency will need to

pay more attention to how to supplement these data

with various forms of “prequantified” material and

observations. Technological developments will permit inexpensive collection of increasing amounts of textual, visual, and auditory

data as integrated supplements to surveys. This capacity should make it easier for

researchers to ask questions and explore subjects that they did not foresee when

designing the survey, thus enriching analytic power and reducing the expense of

designing and conducting new surveys to examine unanticipated concerns.

In addition to these two major themes, this effort led to five important conclusions about future directions for NCES. First, NCES should strive to produce

information that addresses more immediate and specific policy concerns. While

the agency’s role in monitoring and describing major long-term trends in education must not be compromised, this role will assume even greater importance if

the agency can also contribute in a timely way to more focused policy debates.

The widespread emphasis on education reform during the past 10 years has

From Data to Information 3

wo dominant themes

emerged . . . First, NCES

must place greater

emphasis on transforming

raw data into information

useful to policymakers,

educators, researchers,

and the general public . . .

Second, NCES will need to 

broaden its conception of

what constitutes “data”

and strategies for their

collection.

T
spawned a large number of different strategies for improving education. As a

result, policymakers at all levels—national, state, and local—want to know

more about what has and has not been accomplished.

Second, surveys yielding better information that

bears directly on the practices of teaching and

learning would significantly enhance the contribution of NCES to both policy debate and research.

Since current surveys produce scant data on the

specific content of curriculum, the nature and frequency of discrete classroom activities, the practices

of teachers, or the kinds of tasks students perform in

order to learn, the classroom remains largely a

“black box” that defies clear understanding and

precise strategies for improvement. Without a clearer

understanding of what constitutes effective classroom practices, it will be difficult to do more than

simply describe what kinds of education reforms

have been implemented. Whether they have, in fact,

improved teaching and increased learning will

remain unknown.

Third, survey designers should consider more

carefully strategies that will permit integrated

analysis of the interrelationships among education inputs, processes, and outcomes. Although existing surveys do an excellent job of providing nationally representative descriptive data on many important aspects of education, they do not,

however, lend themselves very well to reliable causal analyses that might

increase knowledge about what works and why. The descriptive power of national data must be preserved, but there are promising new designs emerging, which,

if selectively incorporated into national surveys, might generate more robust conclusions about the relative effectiveness of various educational practices. 

Fourth, NCES should make better use of data already collected and maintained

by others. Doing so will help NCES simultaneously accomplish three aims: 1)

expand the amount and type of data it collects; 2) adopt a wider range of data collection and analytic methods; and 3) function within the tight resource constraints

that are certain to affect almost all federal agencies. Previously, NCES has pursued such a strategy with some success—for example, through the Common Core

of Data (CCD) for elementary and secondary education and the Integrated

Postsecondary Education Data System (IPEDS); however, high standards for data

quality, especially comparability and reliability, have frequently forced the

agency to collect new data that were already available in a somewhat different

4 From Data to Information

ive Important Conclusions:

1. Produce information

that addresses more

immediate and specific

policy concerns.

2. Obtain better information that bears directly on

the practices of teaching

and learning.

3. Integrate analysis of

the interrelationships

among education inputs,

processes, and outcomes.

4. Make better use of

data already collected

and maintained by others.

5. Place greater emphasis

on dissemination.

F
form or from a different time period. Without doubt, NCES must maintain its

data quality standards, but increasing cooperation with states and localities, combined with rapidly improving data management technology and communications, should create opportunities for the Center to do a better job of streamlining

and coordinating data collection.

Fifth, NCES will need to place increasingly greater emphasis on dissemination.

Data and information are only as valuable as the breadth, quality, and timeliness

of the uses made of them. Electronic storage media (data tapes and compact

disks) and printed publications will surely remain the cornerstone of the agency’s

strategy for distributing data, tabulations, and the results of analysis. However,

NCES should pay more attention to clearinghouse and brokerage functions, as

well as effective use of electronic networks.

These themes and conclusions do not represent radical departures from the major

path NCES has been pursuing in recent years. Indeed, as the following section

illustrates, they are well suited to building on the foundation of core functions,

operating principles, and programs of work that support the current agency.

Nevertheless, serious attention to these ideas will almost certainly produce

important differences in what the agency now does and how it does it.

BUILDING THE FUTURE ON THE CURRENT FOUNDATION

In 1986, the Panel to Evaluate the National Center for Education Statistics, a

group created under the auspices of the National Academy of Sciences (NAS),

reported on the results of its 2-year assessment of the mission and effectiveness of

NCES (Levine, ed. 1986). The Panel was created to address the widespread perception that the existing agency had not yet developed “the image and the reality of a competent and objective major statistical organization serving the wide

need for statistics about education in the United States” (Levine, ed. 1986, p. 13).

To address such problems as quality of data, timeliness, conceptual obsolescence,

and insufficient funding and staff, the Panel made many important recommendations, including the following:

■ Clearly establish and define the Center’s role in ensuring the availability

of data needed to describe the condition of education in the United States;

■ Improve the compilation of education program, staff, and financial data

from the states, including developing closer collaboration with the states

to ensure that the Center’s program of work meets both NCES and state

requirements for usefulness, relevance, quality, and reliability;

From Data to Information 5
■ Strengthen the Center’s methodological and technical capacity through

more systematic use of outside expertise in the Advisory Council of

Education Statistics, as well as ad hoc advisory groups;

■ Develop, publish, disseminate, and implement standards to guide all

phases of the Center’s work, including establishing an office of statistical

standards headed by a chief statistician;

■ In collaboration with the states, assess and improve the quality, consistency, and reliability of data obtained from state and local agencies, from

institutions of higher education, and from other sources; and

■ Institute a publications policy that clearly distinguishes between different

types of reports—for example, statistical summaries and digests, analytic

reports, descriptive reports, and reports on methodology—and develop a

schedule of fixed release dates for selected key education statistics.

Ten years later, with the direction provided by the

Panel, strong leadership at NCES, and support from

Congress and the larger education community,

NCES is much stronger and has become a widely

respected statistical agency. The agency has significantly strengthened its core functions; operates

under well-defined guiding principles and high

standards for data collection, analysis, and reporting; and has established a clear program of work

for reporting on the major aspects of education in the United States and other

nations. Whereas 10 years ago, the future of NCES depended on rectifying fundamental weaknesses, today the agency’s future can build on a strong foundation.

The Core Functions of NCES

The National Academy of Sciences’ Committee on National Statistics defines the

principal purpose of a federal statistical agency as “the compilation and analysis

of data and the dissemination of information for statistical purposes” (Martin and

Straf, eds. 1992). NCES adheres to this primary purpose by organizing its work

around three core functions:

1) Survey Design and Data Collection

2) Information Production—data analysis, translation, and interpretation

3) Dissemination

hereas 10 years ago, the

future of NCES depended

on rectifying fundamental

weaknesses, today the

agency’s future can build

on a strong foundation.

6 From Data to Information

W
Since national surveys are the primary means for

NCES to collect data on education, during the

past 10 years, the Center has devoted much

effort to improving survey design and data collection. For instance, following the recommendations of the 1986 NAS Evaluation Panel, it has

developed and implemented various strategies

to improve data quality, to detect and reduce

error, and to expedite data collection. The Center

has also significantly improved the sophistication and efficiency of its sampling methods,

increased its use of computer-assisted telephone interviewing, and systematically assessed the quality of data generated in its national surveys.

Moreover, NCES has strengthened and substantially expanded its capacity to

analyze data. Rigorous statistical standards now govern all aspects of its analytic

function, from simple tabulations to the most sophisticated multivariate analyses.3 The Center routinely applies procedures for quality control to all of its surveys, which include analyzing data quality, eliminating unacceptable error, and

producing methodological and descriptive summary reports before releasing survey data for public use.

Finally, NCES has greatly expanded and improved its dissemination function.

Toward this end, the agency has developed and implemented publication standards that now guide the production of NCES reports and the release of public

use data files. A central new feature of the agency’s dissemination function has

been developing strict policies for protecting the privacy of participants in NCES

surveys. The Center not only applies safeguards to the data released to the public

but also requires that its analytic contractors follow strict requirements for limiting

access to prerelease data files and for maintaining the confidentiality of survey

respondents. Failure to adhere to these requirements carries stiff fines, as well as

the possibility of imprisonment.

Operating Principles

In carrying out these core functions, NCES adheres to three operating principles:4

1) Produce information that is policy relevant, while maintaining strict

impartiality, institutional independence, and neutrality with respect to

programmatic effectiveness;

2) Maintain credibility with users of its data, analysis, and publications; and

3) Maintain trust among those who provide data, including individuals,

institutions, and public and private agencies.

From Data to Information 7

CES organizes its work

around three core 

functions:

1. Survey Design and

Data Collection

2. Information Production

3. Dissemination

N
NCES’s program of work must be guided by the issues and requirements of

public policy and federal programs, while scrupulously avoiding specific policy

recommendations or identification with particular policy agendas or ideological perspectives. This principle is perhaps easiest to achieve when the Center

performs its responsibilities for providing data to others for analysis or when it

produces tabulations and descriptive reports.

When NCES engages in analysis or interpretation, however, it must exercise greater care to

remain policy neutral while still contributing

relevant information to policy debates.

Attention to this principle has important implications for charting future directions for NCES.

The suggestions that the Center address more

immediate, specific policy concerns and that it

develop survey designs more strongly suited

for evaluation of what works could lead it

beyond the boundaries of policy relevance into

policy statements and evaluation. This, in turn, could jeopardize its position of

impartiality. Deriving greater policy benefit from data and information produced

by NCES, therefore, must proceed with great care.

Attention to this first principle also contributes to realizing the second, credibility

with users of NCES data and information. However, credibility depends on more

than policy relevance and impartiality. It also derives from confidence in the rigor

of survey design, the quality of the data, the strength of analysis, and the accessibility and usability of its products, publications, and services. Here again, as NCES

considers making greater use of data collected and maintained by others, it will

need to guard against undermining its credibility with users who now depend on

the Center’s increasing emphasis on methodological rigor and data quality.

Finally, the success of NCES as an information agency rests on the trust it

engenders among those who supply it with data. Protecting the privacy of survey participants is a key aspect of maintaining this trust, and integrating new

types of data into national surveys will pose challenges for assurances of confidentiality. Use of video and audio data—for example, taping teachers in the

classroom—will require close scrutiny of this issue. Confidentiality, however, is

not the only condition for securing trust among data providers. Suppliers of

data also need to be confident that the information being requested is truly

needed, that it will be tabulated and analyzed accurately, and that providers

will be given opportunities to correct errors or clarify ambiguities. Pressures for

greater timeliness or more direct electronic access to decentralized, raw data

files may undermine the confidence of data providers in the absence of explicit

attention to new strategies and safeguards.

8 From Data to Information

CES adheres to three operating principles:

1. Produce information

that is policy relevant.

2. Maintain credibility with

users of its data.

3. Maintain trust among

those who provide data.

N
Program of Work

NCES has organized its current program of work around seven major topics:5

■ Elementary and Secondary Education

■ Postsecondary Education

■ Educational Assessment

■ National Longitudinal Studies

■ International Comparative Studies

■ Vocational Education

■ Libraries

Information on each of these topics is produced from a variety of surveys and

studies, several of which supply data to more than one topical area. Some of the

surveys, such as the CCD (on elementary and secondary schools and school districts), are designed as a census of the universe of respondents, which then serves

as a sampling frame for more in-depth cross-sectional or longitudinal surveys on

smaller samples of the population. The Schools and Staffing Survey (SASS), for

example, collects detailed information on teachers and administrators in a sample of schools drawn from the CCD.6 In other instances, a large comprehensive

survey provides the basis for a more intensive study of a subset of respondents.

In this vein, the National Postsecondary Student Aid Study (NPSAS)—a nationwide survey of students enrolled in postsecondary institutions—provides the

basis for more targeted longitudinal studies of students who are starting postsecondary education, the Beginning Postsecondary Students (BPS) Longitudinal

Study, and of students who have completed a baccalaureate degree or higher, the

Baccalaureate and Beyond (B&B) Longitudinal Study.

Central to the NCES program of work are various surveys and studies

designed to assess the knowledge, skills, and performance of American students.

For instance, the National Assessment of Educational Progress (NAEP), which

conducts assessments of reading, mathematics, writing, science, history, and

geography for samples of students enrolled in elementary and secondary education,7 is probably the best known of these efforts. In addition to NAEP, NCES also

provides other data on student performance through transcript studies (at both

the secondary and postsecondary levels); through the National Adult Literacy

Survey, which examines adults’ ability to use prose, documents, and mathematics in a variety of commonplace daily activities; and through international assessments that provide comparative information about student performance in the

United States relative to that of other countries.

From Data to Information 9
Finally, the Center conducts several long-term longitudinal studies designed to

track students’ paths through school and into subsequent stages of working and

family life. These have included such studies as the 1980 High School and Beyond

(HS&B Study), the National Education Longitudinal Study of 1988 (NELS:88),

and the Early Childhood Longitudinal Study (ECLS), which is still in the planning and testing stage and is expected to begin with a kindergarten class in school

year 1998–99.

These surveys now contribute to approximately 100 publications that NCES produces each year, including descriptive reports, analysis reports, methodological

reports, issue briefs, and a variety of other documents. Three of these documents—the Digest of Education Statistics, Projections of Education Statistics, and The

Condition of Education—annually provide a broad national overview of education

at all levels in the United States.

In summary, during the past 10 years, NCES has been engaged in a process of

steady development and improvement. In 1996, NCES is a viable and credible statistical agency, applying high standards to the provision of information on the

condition of education in the United States and the nation’s progress toward

improving mastery of knowledge and skills among all its citizens. With this

strong foundation, the agency is now well positioned to pursue some new directions that will enhance its ability to produce important information for policy,

research, and practice in American education.

NEW DIRECTIONS IN INFORMATION FOR POLICY,

RESEARCH, AND PRACTICE

In considering how NCES can best chart a course over the next decade that will

capitalize on the foundation of work already in place, it is useful to consider its

contribution to three domains of education: policy, research, and practice. These

domains are by no means mutually exclusive; in fact, they overlap and interact in

important ways. There are, however, information needs that are either unique or

more dominant in each, and it is therefore instructive to consider the following

questions individually:

■ How can NCES best contribute information to discussions of education

policy at the national, state, and local levels?

■ How can NCES contribute information that will support significant

research on education effectiveness and improvement?

10 From Data to Information
■ How can NCES contribute information that supports practice—i.e., the

“front-line” activities that develop knowledge and skill in the nation’s

students?

A fourth question constantly weaves through these first three: what advances in

methodology and technology can assist NCES in providing useful information to

each of these domains? This section addresses each of these four questions in turn.

Information for Policy

The agenda of NCES is, in the first instance, greatly influenced by public policy

issues and the requirements of federal, state, and local programs affecting education.

Information contributing to policy debates can assume at least three major forms:

1) System indicators that describe the functioning of the education enterprise,

broadly and over the long term;

2) Implementation indicators that describe the breadth and depth of the execution of policies and practices; and

3) Effectiveness indicators that describe the results achieved by students and

educational institutions and programs.

Although system indicators have been a long-standing focus of NCES, there are

potentially important new developments for the agency to consider. NCES surveys have included information on the implementation of some generic policies

and practices, but specific federal and state policy initiatives have not been examined. Surveys have also included measures of student outcomes—the NAEP is

the best known example; however, these measures typically cannot be directly

linked to particular policies or educational practices to permit rigorous assessments of effectiveness. What are some possible new directions for NCES to consider with respect to each of these three types of indicators?

System Indicators

Data that portray the major aspects of the American education enterprise, both

cross-sectionally and over time, form the core of the mission and functions of

NCES. Reporting basic descriptive information on students, faculty and other

staff, institutions and governing districts, and education finances must continue

to be the primary focus of NCES and should not be compromised by new initiatives. The authors contributing to this examination of new directions for NCES

are unanimous on this point: the primary purpose of NCES is and should remain

representatively describing and documenting the condition of education in

America and other nations.

From Data to Information 11
This basic description of the education system can, of course, be

improved, and several of the

papers included in From Data to

Information: New Directions for the

National Center for Education

Statistics, offered suggestions.8

Among the kinds of system information the authors would like to

see developed are the following:

■ Detail on curriculum content, including rigor and

substance;

■ Detail on the nature and frequency of particular teaching

practices, especially those

which research indicates are

effective;

■ Attention to the nature and frequency of student behavior that reflects

engagement in learning;

■ Resource allocation at the institutional and classroom level;

■ Measures of teacher quality and the ways in which teachers apply their

knowledge and skills in the classroom;

■ More contextual information on postsecondary institutions, especially

their objectives in awarding student financial aid and improved coverage

of proprietary institutions;

■ More attention to the interaction between education and work; and

■ More attention to governance issues, particularly new organizational and

oversight arrangements.

In many respects, these recommendations represent requests for “finer grain” in

the descriptive data presently collected by NCES. In some instances, this aim can

be accomplished by asking for more detailed information; in other instances, collecting and reporting existing data at lower levels of aggregation (the classroom,

for example, rather than the school or school district) will be necessary.

12 From Data to Information

PERCENTAGE OF HIGH SCHOOL GRADUATES WHO EARNED THE

RECOMMENDED UNITS IN CORE COURSES, BY RACE/ETHNICITY: 

1982, 1987, 1990, AND 1992

Percent

0

10

20

30

40

50

60

1982 1987 1990 1992 Year

White Black Hispanic Asian American Indian

SOURCE: U.S. Department of Education, National Center for Education

Statistics, The Condition of Education 1995, Washington, D.C., 78–79.
Implementation Indicators

The widespread attention on education reform during the past 10 to 12 years

has spawned a number of new policy initiatives at the national, state, and local

levels. Congress periodically revises such mainstay education legislation as the

Elementary and Secondary Education Act, the Higher Education Act, or the

Individuals with Disabilities Education Act. Additionally, it undertakes new

education policy initiatives such as GOALS 2000 or the School-to-Work

Opportunities Act. States have also initiated many new policies to strengthen

elementary, secondary, and postsecondary education. These have included

changes in the requirements for high school graduation, new teacher certification regulations, modifications to postsecondary admissions standards, and

new policies on college tuition and student financial aid. 

Traditionally, NCES has not monitored the implementation of specific federal or

state legislation. At the national level, Congress has typically provided for independent assessments or evaluations of education legislation, such as the National

Assessments of Vocational Education, which have been conducted approximately every 5 years. While these national assessments make extensive use of NCES

data, they also conduct independent surveys that focus more particularly on key

features of the legislation being examined.

Several of the authors involved in this project have urged NCES to monitor some

of the key policies and practices that have emerged from federal, state, and local

legislation during the 1980s and 1990s.9 It should be emphasized that they are not

recommending that NCES assume responsibility for evaluating particular legislation, because they believe this function should continue to rest elsewhere.

Rather, they are urging NCES to examine policies and practices that became more

generic as they have been adopted and implemented through various federal,

state, and local initiatives and are, therefore, no longer associated with any single

piece of legislation. Some specific examples include the following:

■ Curriculum content standards and measures of student or institutional

performance;

■ Length of the school day or year;

■ Requirements that students complete particular courses (for example, in

math, science, or foreign language) or accumulate a minimum number of

credits for graduation;

■ Participation in a variety of “work-based” learning opportunities, including apprenticeship, cooperative education, tech-prep programs, or schoolbased enterprise;

■ Operation of charter schools;

From Data to Information 13
■ Prevalence and nature of home schooling;

■ Availability and use of school choice;

■ Participation in reform networks, such as the Coalition of Essential

Schools, Accelerated Schools, or High Schools That Work;

■ Changes in affirmative action policies;

■ Changes in postsecondary admission requirements;

■ Prevalence of state takeovers of local school districts or other forms of

state intervention in financially troubled localities; and

■ Changes in state policies affecting postsecondary tuition or student

financial aid.

More attention to such issues by NCES would help ensure that its data are policy

relevant, while still leaving responsibility for policy evaluation to independent

studies and other agencies in the Department of Education.

Effectiveness Indicators

In addition to information on how to implement

policies and practices, policymakers would also

like better information on their effectiveness. Even

though it is useful to know how widespread the

adoption of a particular strategy for improving

education has been, it is even more useful to know

how well it has worked, and why or why not.

This, of course, is a primary aim of most policy

evaluation, as well as many research projects.

Much of the credibility of NCES rests on its clear

separation from policy evaluation and research

on education impacts and outcomes. Although

NCES contributes essential data and information

to these efforts, it remains well removed from the

conduct of any of these activities. This separation of functions contributes to the

neutrality and objectivity that NCES must maintain as the nation’s primary statistical agency for education. The impartial character of NCES must be preserved.

Consequently, any initiative to make NCES surveys more conducive to assessments of policy effectiveness must proceed with great care.

Why consider such a course at all? First, there is potentially a substantial payoff

from better integrating the nationally representative features of NCES surveys

n the current environment, policy analysts

often face a frustrating

choice: asking the right

question with weak

methodology and data

that were not collected

specifically for that purpose, or asking a much

less important question

with sound methodology

and specially tailored

information.

14 From Data to Information

I
with the more rigorous but also more narrowly

circumscribed designs of policy impact studies.

In the current environment, policy analysts

often face a frustrating choice: asking the right

question with weak methodology and data that

were not collected specifically for that purpose,

or asking a much less important question with

sound methodology and specially tailored information.10 Clearly, answering important questions with sound methods and precise information is more likely to improve education and the

policies that support it. Combining the representative power of national surveys with the

methodological rigor of experimental design

would help realize this objective.

Second, there may be significant cost savings from integrating some impact evaluations with national surveys. Both kinds of efforts are quite costly. It is not

unusual for a national survey to cost in excess of $10 million, and the more rigorous policy evaluations adopting experimental design frequently cost as much or

more. Both efforts often collect similar kinds of data at approximately the same

points in time, sometimes even from the same respondents. Eliminating this

duplication would not only reduce costs but also alleviate some of the burden on

respondents participating in national surveys and evaluations.

Cost savings aside, the primary benefit of integrating methodologically rigorous effectiveness assessments with nationally representative surveys lies in

increasing the usefulness of these two activities beyond the results obtained

when they are conducted independently. National survey data would more

directly and authoritatively address questions about policy effects; impact

studies would be conducted in a nationally representative context that would

increase the likelihood that study results could be generalized.

To achieve this result, NCES should carefully consider piloting the inclusion of

an experimental study in one of its national surveys. Any of the longitudinal

surveys now under way are potential candidates, including SASS, ECLS, or the

longitudinal spin-offs of NPSAS.

What should be the focus of experimental studies imbedded or linked to

national surveys? Clearly, the choice must be considered carefully, with ample

input from interested policymakers, researchers, and educators. Given the mission of NCES, focusing on a particular type of educational practice would

probably be more appropriate than on an assessment of a specific legislative

program. One possibility, for example, would be to conduct a careful study of

the consequences of homogeneous versus heterogeneous grouping of students

by academic ability.11

From Data to Information 15

he primary benefit of

integrating methodologically rigorous effectiveness assessments with

nationally representative

surveys lies in increasing

the usefulness of these

two activities beyond the

results obtained when

they are conducted 

independently.

T
Information for Research

Researchers are heavy users of information produced by NCES, and while the

boundaries between policy and research are fuzzy, the interests of the research

community deserve some separate attention. In the papers produced for this project, three themes emerged as priorities for focusing NCES’s contribution to

research over the next decade:

1) Teaching and Learning—illuminating more clearly what actually happens

in the classroom;

2) Education Production—clarifying the processes of transforming education resources into student, program, and institutional outcomes; and

3) Education Outside the Classroom—depicting what and how learning

occurs beyond the walls of the traditional classroom in homes, workplaces, and the community at large.

Though not exhaustive, this list provides some important directions for NCES to

consider. In the next section, each topic will be briefly discussed.

Teaching and Learning

Much of the business of education occurs in the nation’s classrooms—elementary,

secondary, and postsecondary—yet national surveys presently tell us relatively

little about what actually takes place at the classroom level. Currently, good information is available about the different types of courses taught (at both the secondary and postsecondary levels), but there is little or no nationally representative detail on the content of the curriculum or how it varies among classrooms,

institutions, or states. Similarly, not much data are available on teaching practices, either the range of strategies adopted by faculty or the frequency of their

use. Finally, most surveys do not offer much description of what students do to

facilitate or impede learning in the classroom.12

While richer information on these three aspects—curriculum, pedagogy, and student behavior—would be useful in and of itself, the greatest benefit to research is

likely to be achieved when information on all three is simultaneously available at

the individual classroom level. That is, ideally researchers would want to examine how these three aspects of classroom activity interact and to understand how

they relate to various types of education outcomes. In this way, more can be

learned about what works and why in the daily business of education.

To realize this objective, one implication for future NCES surveys is clear: survey

designs need to pay more attention to using the classroom as a unit of analysis.

Additionally, the designs should strive to produce an integrated package of infor16 From Data to Information
mation on curriculum content, teaching practices, student behaviors, and student learning

outcomes. It is not sufficient, for example, to

simply expand transcript studies to include

more information on course content; rather,

expanded information on course content must

be linked to other data on teaching practices,

student behavior, and student achievement.

In addition to the question of what kinds of

information on classroom activity can best

advance future research, there is also the issue

of how best to collect it. Traditionally, to obtain

data on classroom activities, NCES has asked

respondents questions through paper questionnaires or telephone interviews. Thus, to the

extent that current surveys yield information on

teaching practices or student behavior, they rely mainly on self-reports.

An alternative to collecting information through respondent self-report is direct

observation by trained researchers. Until recently, direct observation has been a

very expensive alternative, indeed prohibitively so for large-scale surveys involving thousands of respondents. However, recent technological and methodological advances are making direct observation, as well as the collection of source

materials, more feasible.13 Video is one of the most promising strategies for linking direct observation to more traditional survey techniques, and NCES is using

this technique for the first time in designing the Third International Math and

Science Survey (TIMSS).14

Video, of course, is not an especially new technology. What is new, however, is its rapidly

growing capacity to store large amounts of video

information inexpensively in digital form that

enables fast retrieval and analysis. Additionally,

researchers are making steady progress in developing analytic techniques that simplify and

accelerate transforming video information into

coded data suited for analysis using quantitative

methods. Video, therefore, can add significantly

to the richness and analytic potential of a survey,

since it reduces the need to anticipate all of the

questions the survey must ask of respondents.

As researchers observe video records, they can formulate completely new variables that may not have been considered in the design phase of the survey. In the

past, such new formulations usually required asking respondents follow-up

questions or designing a new survey, if such avenues were pursued at all.

From Data to Information 17

ne implication for future

NCES surveys is clear:

survey designs need to

pay more attention to

using the classroom as 

a unit of analysis. Additionally, the designs

should strive to produce

an integrated package of

information on curriculum content, teaching

practices, student

behaviors, and student

learning outcomes. 

O

ideo is one of the most

promising strategies for

linking direct observation

to more traditional survey

techniques, and NCES is

using this technique for

the first time in designing

the Third International

Math and Science Survey

(TIMSS).

V
Closely related to this kind of use of video technology is the increasing capacity to collect, store,

and analyze large amounts of textual information.

For example, if researchers need better information on the content of textbooks or other printed

materials used in classrooms, it is now possible to

optically scan samples of these classroom documents for subsequent coding and analysis. As

with video images, electronically storing and

retrieving large amounts of textual information is

relatively inexpensive.

These advances in storing and analyzing large amounts of what is essentially

“prequantified” data promise to integrate survey research with case study

methods, and represent research strategies that until now have been pursued

independently of one another, each with its own strengths and weaknesses.

This integration has the potential to link the representative statistical power of

survey design with the richness and variety of case study information, simultaneously obtaining the best of both worlds.

Expanding surveys to include systematic collection of prequantified visual, textual, and even auditory information could produce significant new contributions

to research on teaching and learning. Consequently, NCES should carefully consider how best to capitalize on its initial experience with this strategy in TIMSS,

with special attention to adapting the approach to other surveys such as SASS,

ECLS, or the longitudinal spin-offs of NPSAS. Additionally, the use of video in

national surveys might prove especially beneficial if it were initially combined

with efforts to imbed experiments in national surveys. The combination of these

two methods targeted on analyzing the effectiveness of particular teaching interventions, for example, could yield very useful and robust results.

Education Production

Better understanding the interactions among curriculum, pedagogy, and student

behavior in the classroom is an important piece of a larger set of research questions—how dollars are allocated (to localities, institutions, and classrooms), transformed into various resources, organized into programmatic and teaching strategies, and used to produce increases in students’ knowledge and skills. In short,

NCES data could play a much more significant role in expanding knowledge

about how to better use education resources to improve student performance,

thereby improving the overall process of education production.15

Achieving this goal will require some changes in the way NCES currently collects

data on the financing of elementary, secondary, and postsecondary education. At

present, there are two main surveys collecting financial data, CCD at the elemen18 From Data to Information

xpanding surveys to

include systematic collection of prequantified

visual, textual, and even

auditory information

could produce significant new contributions

to research on teaching

and learning.

E
tary and secondary level and IPEDS at the postsecondary level.16 With respect to financial information, both of these surveys focus primarily on

providing detail on revenues and expenditures,

for local school districts in the case of CCD and

for individual institutions in the case of IPEDS.

Both surveys are designed to collect financial

data primarily from an accounting perspective

and are not now well suited for cost-benefit

analysis of educational programs or cost-effectiveness analysis of particular teaching strategies.

Neither provides information on the allocation of

resources at the classroom level.

Providing data that better inform understanding of the production process of

education would be aided by NCES expanding its present focus on finance to

embrace a broader concentration on the economics of education. This larger conception would aim to integrate data on finance with other data on education

processes and practices, as well as student outcomes. Additionally, new kinds of

economic data would be required. Rather than needing more detail on expenditures for such general functions as administration, instruction, maintenance, or

capital outlay, researchers would want to obtain data on the costs of specific

types of staff, different kinds of school improvement strategies, alternative

teaching strategies, and so on. They would also want to know more about the

costs of different kinds of course content, equipment, instructional products, and

assessment. In short, rendering the process of education production more intelligible depends on moving

beyond traditional concerns

about the distribution and

expenditure of dollars to a more

careful examination of how to

transform dollars into effective

teaching and learning in the

classroom.

Three strategies for improving

NCES data on education finances would help accomplish this

goal. First, what constitutes useful financial data needs to be

reconsidered, with special attention to better information on

unit costs and transforming dollars into education processes

and practices. Second, data will

be needed at the classroom

From Data to Information 19

roviding data that better

inform understanding of

the production process

of education would be

aided by NCES expanding its present focus on

finance to embrace a

broader concentration

on the economics of

education.

P

PUBLIC EDUCATION REVENUE PER STUDENT, 1930–93

SOURCE: U.S. Department of Education, National Center for Education

Statistics, The Condition of Education 1995, Washington, D.C., 148–49.

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

'30 '40 '50 '60 '66 '70 '75 '80 '85 '90 '93

Year

Revenue

Higher education

Elementary/secondary
level; information on districts or institutions is not likely to contribute much to

this kind of research. Third, it must be possible to link these financial data to

other data on teacher characteristics, classroom practices, student demographics and behavior, and learning outcomes. Without this kind of integrated data

about how education occurs, understanding more precisely how to efficiently

allocate resources for education will continue to elude researchers and policymakers.

Education Outside the Classroom

Although elementary, secondary, and postsecondary classrooms are the centers

of formal education in America, it is widely understood that much learning also

takes place outside the classroom in the home, the workplace, and the community at large. However, we know relatively little about what or how learning occurs

in these settings, nor do we know much about how learning in these places interacts with learning in the classroom. Moreover, given that most Americans spend

only 12 to 16 years in formal schooling but another 50 years or so learning in these informal environs, a thorough description of the condition of

education in America would require closer attention to the learning that transpires beyond classroom walls.

NCES surveys have already paid some attention to

nonschool settings. At the present time, probably

the largest of such efforts is the Early Childhood

Longitudinal Study (ECLS), which will begin by

focusing on the preschool lives of a cohort of children who will be followed over their early years of

development. Additionally, other longitudinal

studies, such as HS&B and B&B, have collected

data on respondents’ experiences in the workplace. Information on labor market participation,

however, has been limited primarily to data on

types of labor market outcomes—for example, earnings, duration of employment,

and types of occupation—rather than systematically examining how learning

occurs in the workplace or the degree of congruence between learning goals in

schools and education requirements on the job.

Comprehensively surveying learning that occurs outside the classroom is a tall

order, and NCES should approach this task incrementally. One place to focus an

expanded examination of informal learning is on the workplace and the strategies

adults use to maintain and upgrade the knowledge and skills needed to remain

productive, actively engaged workers.17 Such a focus is more important than

ever, given the changes that are occurring in today’s work world. These changes

include not only rapidly developing new technologies but also major shifts in the

20 From Data to Information

lthough elementary, 

secondary, and postsecondary classrooms

are the centers of formal

education in America, it

is widely understood that

much learning also takes

place outside the classroom in the home, the

workplace, and the 

community at large.

However, we know relatively little about what or

how learning occurs in

these settings.

A
attachments and relationships between employers

and employees. As the likelihood of lasting

employment with a single employer becomes

increasingly tenuous in the modern economy,

individuals must assume ever greater responsibility for nurturing their own careers and continuing

employability. How working adults will meet this

responsibility in the future poses important new

challenges for the nation’s systems of education.

Increased attention to learning through and for

work could begin with the following steps. First, it

is important to learn more about the knowledge

and skills needed for long-term success in the labor market. Are these requirements consistent with the academic and vocational goals of formal education, and

how well does the formal education system produce the desired prerequisites?

Second, NCES could pay closer attention to how learning occurs in the workplace; whether the process differs in important ways from learning in the classroom, and whether the two complement or reinforce one another. Third, NCES

could enrich work-related data in its current longitudinal surveys, concentrating

especially on better information about what people do on the job, what contributes to their successes and failures, and how they use or do not use schoolbased learning to perform and advance. 

As part of its own mission, NCES could independently address all three of these

issues. Alternatively, the agency may want to explore opportunities for collaborating with the Department of Labor and its surveys of employers and employees. Data collected by the Labor Department tend to provide greater detail on

labor market participation, while being relatively deficient on education variables. Better coordination or integration of the two Departments’ survey efforts

could yield some important benefits.

Information for Practice

Most teachers and administrators are accustomed to viewing data as something

to be reported to others. For example, they report daily attendance to central

offices to document federal, state, and local funding systems. They submit grades

for report cards to students and parents and for recording on student transcripts,

which in turn are reported to postsecondary admissions offices. They administer

standardized achievement tests for state assessments and college admissions.

Rarely, however, do teachers and administrators use data directly themselves to

improve their own programs and practices. One consequence of this outlook on

data is that most practitioners do not make much use of the information provided by NCES. An important challenge for NCES, therefore, is significantly

increasing the value and utility of its data for local teachers and administrators.

From Data to Information 21

ne place to focus an

expanded examination of

informal learning is on

the workplace and the

strategies adults use to

maintain and upgrade

the knowledge and skills

needed to remain productive, actively

engaged workers.

O
At least three strategies for providing better information for practice offer some important opportunities for NCES:

1) Benchmarking—helping local practitioners make

comparisons against established norms;

2) Networking—linking practitioners with other

practitioners and helping them discover more

quickly who is doing what and where;

3) Practitioner-Based Research and Self-Reflection—

engaging practitioners in systematic inquiry

through NCES surveys and related research.

Benchmarking

“How well am I performing?” This is an appropriate question for any professional

concerned with improving practice and increasing students’ mastery of knowledge

and skills. For most educators, however, it is a difficult question to answer in any

way other than in an impressionistic or anecdotal fashion. Until recently, education

has not had enough success with helping schools, programs, and faculty to monitor

their accomplishments or to use the results to improve what they do.

Fortunately, as more and more states and localities develop new strategies for

tracking performance and promoting school improvement, this situation is

changing. “School report cards” are now produced annually in many states.

Other states have developed systems of performance measures and standards,

along with procedures for school improvement plans in districts that perform

below state norms. Moreover, “keeping score” and using the results to assess the

relative effectiveness of different kinds of school improvement strategies are core

operating principles of several large consortiums, such as High Schools That Work

under the auspices of the Southern Regional Education Board.

NCES could make an important contribution to the continued development of

these practices by improving the utility of its survey results as benchmarks for

states and localities interested in knowing how their performance measures up in

relation to others. A local school or school district, for example, could find out

how well its record on student attendance or high school completion compares

with a national or state norm. It could then further refine the comparison by

examining such measures in a subset of districts or schools that are similar with

respect to size or student demographics. In addition to making comparisons at a

particular point in time, a local school or school district might also monitor its relative performance over time. For example, is its success in reducing dropout rates

proceeding at a faster or slower pace than in comparable districts or schools?

22 From Data to Information

ost teachers and administrators are accustomed to

viewing data as something to be reported to

others. Rarely, however,

do teachers and administrators use data directly

themselves to improve

their own programs and

practices.

M
There is nothing to prevent localities, or even individual teachers,

from using current NCES data to

establish these kinds of benchmarks. However, they need to

work rather hard to do it. Finding

the right data is not always easy,

nor is determining whether the

NCES estimate is comparable

with a local statistic. Tailoring an

NCES estimate to yield a comparison of “likes with likes” requires

a knowledge of NCES data sets,

as well as analysis techniques,

that most practitioners do not

have. Thus, there are significant

barriers to transforming NCES

data into useful benchmarks at

the local level.

However, there are at least three

steps that NCES could take to

make benchmarking easier for

states and localities. First, in collaboration with potential state and local user-practitioners, NCES could systematically review its current dissemination activities with specific attention to

how some aspects of the dissemination process could be modified to facilitate

benchmarking. For example, there may be consensus on a relatively small set

of indicators that NCES could publish annually in a succinct, accessible form

with widespread local distribution. Such a publication might be similar to The

Pocket Condition of Education, which NCES now produces annually, but it could

be designed with local benchmarking specifically in mind.

Second, NCES could explicitly consider local benchmarking when designing selected surveys, including customized reporting of results to survey participants. At present, survey participants receive little or no direct benefit from taking part in NCES

surveys, and the burden of doing so is often not trivial. Providing participants with

a summary of where they stand on selected variables relative to others in the survey

could be a useful service.18 Such a summary could take the form of a traditional

printed report. Alternatively, NCES might want to explore new electronic strategies

that could actually distribute some limited analytic capacity along with the data (see

discussion below on technological innovations).

Third, as NCES increases its capacity to provide information “on-line,” it should

consider strategies for developing and distributing analytic packages that enable

state and local benchmarking. In other words, instead of simply making data availFrom Data to Information 23

MEDIAN TOTAL TIME TO DOCTORATE DEGREES AMONG U.S.

CITIZENS AND PERMANENT RESIDENTS, 1970–93

0

6

8

10

12

14

16

18

20

'70 '75 '80 '85 '90 '93

Year of doctorate

Median number

of years

Education Humanities

Natural sciences

Computer science

and engineering

Social and

behavioral sciences

SOURCE: U.S. Department of Education, National Center for Education

Statistics, The Condition of Education 1995, Washington, D.C., 46–47.
able, NCES would also provide a menu of data

analysis programs or routines that would enable

practitioners to generate their own statistics

quickly and easily, without requiring a sophisticated knowledge of the underlying methodology.

Such a strategy would build on NCES’s current

practice of providing users with “table generators,” increasing both the kinds of analysis that

users could perform and the ease of using the analytic software.

Networking

Local teachers and administrators often want to

know who else has experience with a particular

school improvement strategy, type of curriculum, or teaching practice. Yet, systematically

locating and communicating with other knowledgeable practitioners can be quite difficult;

often it is not easy to find out who these individuals are or how to contact them. If NCES were to

assume a greater role in monitoring the implementation of more specific education policies and practices (see earlier section on

Implementation Indicators), it could also facilitate networking among practitioners. In addition to providing practitioners with information about the frequency

with which a particular reform is being implemented and where it is being

attempted, NCES could also match up interested parties and help them share

information about their experiences.

This kind of knowledge brokering would represent a new function for NCES, one

that may not be completely in keeping with traditional perspectives on the appropriate role of a statistical agency. Nevertheless, as NCES develops its presence on

the Internet and the World Wide Web, this kind of service would be an obvious

extension of its capacity to transform data into information valued by practitioners. Moreover, when providers of data also have a direct use for similar information from others, they are much more likely to respond to NCES’s requests in an

accurate and timely fashion. Thus, NCES’s ability to monitor implementation for

policy purposes could well be enhanced by its also using the information to provide an important service to teachers and administrators.

Practitioner-Based Research and Self-Reflection

Until recently, surprisingly little has been known about the specific elements of

high-quality teaching (and by extension, high-quality teachers). This lack of

knowledge has contributed to much misinformation and misunderstanding

about what it takes to be a good teacher, as well as confusion in the public policy

24 From Data to Information

hree steps that NCES could

take to make benchmarking easier for states and

localities:

1. Review its current 

dissemination activities

with specific attention to

how some aspects of the

dissemination process

could be modified to 

facilitate benchmarking.

2. Explicitly consider

local benchmarking when

designing selected 

surveys.

3. Consider strategies for

developing and distributing

analytic packages that

enable state and local

benchmarking.

T
arena over the role of professional development in education reform. The status

of national data on teachers reflects this state of affairs, with facts limited mainly

to demographic characteristics and scant information available on the quality of

practice or practitioners.

A very promising development, therefore, is the recent effort on the part of the

teaching profession to begin a systematic, sustained examination of what constitutes good teaching—specifically what teachers should know and be able to do to

help students master high levels of proficiency. Exemplified by the work of the

National Board for Professional Teaching Standards (NBPTS), as well as other

organizations and state-level initiatives, this effort is forging some consensus

about appropriate standards for defining advanced high-quality teaching. This

work has led to the establishment of a voluntary system of certification for early

childhood, elementary, middle, and secondary school teachers, including differentiation among a range of academic disciplines (for example, math, science,

history and the social sciences, English, and vocational education). 

These developments create an important opportunity for NCES not only to improve the data it

gathers on the nation’s teachers but also to contribute more directly to strengthening teaching.19

This opportunity can be realized in two ways.

First, as the work on teaching standards and certification continues to evolve, NCES should be

able to define a larger array of indicators of

teaching quality to include in national surveys.

At a minimum, these indicators should focus on

measuring teachers’ command of the knowledge

bases and teaching methods that are being identified as reflective of high-quality practice.

Gathering such data could rely on traditional

methods of written assessment or self-report.

Alternatively, if NCES opts for further developing video observation techniques, these methods could significantly enrich

information on the condition of teaching nationwide.20 Furthermore, as more

teachers choose to pursue national certification and as more certificates are

awarded, national counts of teachers participating in and successfully completing the process will assume greater value as indicators of teacher quality.

Second, as NCES pursues this first strategy for improving data on the quality of

practice, it could actively engage practitioners in this process and create opportunities for more interactive research and development. For example, if written

examinations (in the style of NAEP) or video observation become part of NCES’s

strategy for monitoring and reporting to the nation on teacher quality, this

process could be designed to simultaneously benefit individual teachers participating in the surveys. This might be accomplished in several ways. The

From Data to Information 25

s the work on teaching

standards and certification continues to evolve,

NCES should be able to

define a larger array of

indicators of teaching

quality to include in

national surveys. NCES

could actively engage

practitioners in this

process and create

opportunities for more

interactive research and

development.

A
results of written assessments could be returned to individual teachers. Groups

of teachers could be assembled to review and constructively critique video segments. Further, if the data gathering process included collecting mini-portfolios

submitted by teachers, these could be systematically evaluated, with examples of

best practices culled from the data and disseminated to teachers and teacher

education institutions. In short, the business of collecting national data could

begin to play a more direct role in professional development and the strategic

improvement of teaching and schools.

Further Considerations About Methodology and

Technology

Increasing the contribution of NCES to policy, research, and practice depends

in part on closer attention to a number of methodological and technological

opportunities. Some of these, such as imbedding experimental designs in

national surveys or collecting prequantified data through video and optical

scanning, have already been discussed. There are some additional considerations, however, that deserve special mention, including: 1) developments in

using administrative records, 2) promising techniques for obtaining hard-to-get

information or producing more finely tuned estimates, and 3) effective use of

the Internet and World Wide Web.

Administrative Records

Much of the information sought by national surveys already exists, at least in

an approximate form, in records maintained for administrative purposes by

schools, postsecondary institutions, district offices, state agencies, and other

public and private offices. Transcripts, for example, provide detailed information on courses attempted and completed, grades, credits earned, and scores

from standardized tests. Personnel records contain data on teaching assignments, salaries, demographics, qualifications, and experience. And budget and

accounting offices maintain extensive records on revenues and expenditures.

To the extent that surveys can access and use these administrative records,

they often can obtain information that is more accurate than the responses provided by survey participants, often at significantly less cost. 

Several NCES surveys already rely heavily on administrative records for

information. Some good examples are CCD, IPEDS, and the NPSAS. There are,

however, two types of problems that have limited the usefulness of administrative records. First, the contents of the records may not meet acceptable standards

of accuracy, consistency, and comparability. Second, access to administrative

records is often problematic, for a variety of reasons ranging from concerns about

confidentiality to technical problems that may be as mundane as locating the

right filing cabinet in the right office.

26 From Data to Information
Technological advances in computing and electronic networking promise to

reduce both of these problems considerably over the next decade, and NCES

should be alert to opportunities to exploit new developments.21 First, electronic

administrative records maintained in easy-to-use relational databases will

increasingly become accepted practice among the nation’s elementary, secondary, and postsecondary education systems,22 since they will be building

administrative databases to satisfy their own needs and uses for data.

Consequently, collecting data will less often be viewed as an externally imposed

burden and cost. Whether providing these data to national surveys will be seen as

burdensome, however, will depend critically on the ease with which data can be

transmitted to those requesting information. To facilitate transmission, NCES will

need to pay particular attention to assisting with the standardization of data elements and with the development of cheap scannable forms and other strategies

for promoting electronic access and transfer.

Second, as local educators and administrators

become more sophisticated users of data (rather

than just providers), the business of designing surveys, collecting and analyzing data, and reporting

results is likely to become much more interactive.

The traditional model in which NCES assumes

primary responsibility for all of these functions is

likely to yield to much more decentralized, distributed models in which the respective roles of

surveyor and respondent become less distinct and

more intertwined. For example, respondents who

are also users of data may play a much greater role

in defining survey questions and data elements.

They may also develop specialized analyses

(including analytic routines) that are shared with

other respondent/users. NCES, in turn, may

assume more responsibility for coordinating and brokering surveys, analyses, and

reporting, rather than unilaterally directing and conducting all of these activities.

One possible implication of these trends is a reversal in the respective roles of

independent surveys and administrative records in providing national data for

education. To date, administrative records have mainly been adjuncts to largescale surveys; they have supplemented data collected through written questionnaires or telephone interviews. In the not-too-distant future, administrative

records may become the basic building blocks of national data systems, with

smaller targeted questionnaires designed as supplements.

From Data to Information 27

ne possible implication

of these trends is a

reversal in the respective roles of independent

surveys and administrative records in providing

national data for education. Administrative

records may become the

basic building blocks of

national data systems,

with smaller targeted

questionnaires designed

as supplements.

O
Methods for Producing Better Statistical Estimates

To provide good information for policy, research, and practice, NCES relies on a

wide variety of survey design, data collection, and analytic methods—some relatively simple and widely known, others extraordinarily complex or reflecting

recent advances in specialized fields. For purposes of this synthesis, a thorough

discussion of survey and analytic methods is neither possible nor appropriate.

The papers produced for this project, however, raised and discussed a number of

methodological issues and developments. Four of these deserve special mention

for careful consideration by NCES in charting its future course.23

First, as mentioned previously, NCES should

exploit opportunities to combine well-designed,

targeted controlled experiments within the national surveys that have been the hallmark of its data

collection activities. These experiments must be

compatible with the mission and conduct of the

larger survey effort, and a particular experiment

should not be undertaken if it risks jeopardizing

the nationally representative and descriptive

power of the survey in which it is imbedded.

However, if these criteria can be satisfied, imbedded experiments are promising examples of constructing a “whole exceeding the

sum of its parts.” Such experiments could contribute significantly to knowledge

about what works and why in the nation’s classrooms.

Second, survey questions that elicit information on sensitive topics must always

be considered with great care. National surveys about education are no exception, and it is important that NCES does not avoid issues simply because they are

sensitive or controversial. Methodological developments can help reduce some of

the concern surrounding this issue. For example, one promising strategy called

“network-based estimating,” in which respondents are asked about the behavior

of unidentified acquaintances in their social network, has been developed by

quantitative anthropologists. There is growing evidence that this procedure produces indirect but reliable information on sensitive topics, without depending on the respondent

to report directly on his or her own personal experience. NCES should explore the feasibility of

using this or similar techniques in future surveys.

Third, and related to the second issue, there are

promising new developments in methods for generating indirect estimates of statistics at subnational levels or for intervening periods of time

between surveys. Traditionally, producing estimates for smaller units of analysis—states, for

28 From Data to Information

CES should exploit

opportunities to combine

well-designed, targeted

controlled experiments

within the national 

surveys that have been

the hallmark of its data

collection activities.

N

here are promising new

developments in 

methods for generating

indirect estimates of 

statistics at subnational 

levels or for intervening

periods of time between

surveys.

T
example, or institutions within states—has

depended primarily on increasing sample size.

Similarly, obtaining estimates more frequently—

say, every 5 years rather than every 10—typically

requires administering the survey more frequently.

Both of these strategies are usually quite expensive.

An alternative method being developed uses auxiliary data (from ongoing administrative records, for

example) along with the survey data to produce

indirect estimates for smaller units of analysis or

intervening periods of time. Successfully adapting these techniques to some NCES

surveys could yield more finely grained estimates at a modest cost.

Fourth, there are long-standing calls for better linking and integrating the databases produced by NCES surveys. As noted earlier, a better understanding of

how learning occurs in the classroom will require simultaneous access to data on

curriculum content, teaching practices, student behavior, and student outcomes.

It has been rare to find a single database with rich information on all of these

attributes for a sufficiently large sample, however. If NCES were able to significantly improve the connections between its surveys, it is likely that opportunities

for better, more focused research would be enhanced. However, in order to

achieve this long sought-after objective, NCES must do substantial work, both

conceptually and methodologically, to determine precisely what is meant by

“linking” and “integrating.”

Finally, NCES must continue to actively promote methodological developments

and adaptations suited to its mission. Most researchers, whether engaged in a

particular substantive pursuit or methodological advancement, are occupied primarily with their own interests and agendas; they are not paying much attention

to the relevance of their work for NCES. Consequently, NCES needs to provide

for the orderly acquisition and screening of methodological and technological

applications to surveys and analysis. There are many strategies for doing this,

including advisory groups, grants, conferences, commissioned papers, and so on.

Whatever strategy is chosen, however, the basic objective must be an explicit and

high-priority item on the agenda of NCES.

Internet and the World Wide Web

No discussion of future technological developments would be complete without

some mention of the Internet and the World Wide Web.24 However, the pace and

variety with which these are evolving make any effort to forecast precisely their

role in the work of NCES quite difficult, if not simply foolish. Perhaps, the most

useful approach is to use the evolution of the information highway as a metaphor

From Data to Information 29

f NCES were able to 

significantly improve the

connections between its

surveys, it is likely that

opportunities for better,

more focused research

would be enhanced.

I
for changes in communication

and interaction between NCES

and the public it serves. In some

respects, the Internet and the

World Wide Web will facilitate

and hasten these changes, but in

others, they are simply reflective

of larger forces at work in contemporary society.

Presently, NCES is in the second

stage of a three-stage evolution in

how many organizations typically

interact with their clients. In Stage

One, to accomplish its mission,

NCES dominates the relationship

between itself and those who

either provide or use the data it

gathers. Communication tends to

be mainly one-way and follows

well-established paths. NCES

designs the surveys, administers

questionnaires or interviews, collects and cleans data, conducts its own analysis,

and produces and disseminates reports. Other analysts of NCES data pursue their

research independently; they do not feed back results to NCES, at least in any

systematic fashion. Stage One loosely represents the pre-Internet, pre-Web world.

It is history.

In Stage Two, which coincides with the advent and initial development of electronic networks, relations between NCES and providers and users of data become

more interactive (though still predominantly one-way), and in some instances the

distinction between data provider and user begins to blur. In this stage, surveys

begin to make more use of electronically stored administrative records, and, consequently, questions or data elements may be tailored to a particular respondent.

Selected providers and users may be authorized limited on-line access data to

update or correct information. Although NCES continues to generate substantial

analyses on its own, it also begins to pay closer attention to the analytic objectives

of users. In addition to distributing data files, it also disseminates analysis files

designed to facilitate specific types of research—the relationship between education and labor market participation, for example. Additionally, NCES may provide analysts with software to accelerate their analyses or to ensure that those

who conduct external analyses of NCES data adopt appropriate statistical techniques. Reports are made available in electronic form, and specialized electronic

user groups or technical review panels begin to form on the network. Currently,

NCES is already well immersed in Stage Two.

30 From Data to Information

PERCENTAGE OF STUDENTS WHO USED A COMPUTER AT SCHOOL OR

AT HOME, BY FAMILY INCOME AND YEAR, GRADES 7–12:

1984, 1989, AND 1993

0

20

40

60

80

100

Low Middle High

Percent

1984 1989 1993

Used a

computer

at school

Used a

computer

at home

Used a

computer

at school

Used a

computer

at home

Used a

computer

at school

Used a

computer

at home

SOURCE: U.S. Department of Education, National Center for Education

Statistics, The Condition of Education 1995, Washington, D.C., 34–35.
In Stage Three, which will emerge more clearly

and strongly with greater access to electronic networks and with deeper understanding about how

to use them effectively, relationships between

NCES and its data providers and users will

become truly two-way and continuous. Any data

user, who could also be a data provider (a state

office, for example), might send NCES a small software program that initiates a customized database

search, adds the results to NCES’s data library, and

returns a tailored report to the original requester.

Conversely, NCES may be constantly developing

small software programs that go out over the network and retrieve data needed to respond to specific inquiries from Congress, researchers, educators, or the public at large.25

Surveys may assume the form of database development, with specifications designed interactively

by users and providers coordinated by NCES. The scale of written questionnaires or telephone interviews will diminish considerably or be limited to highly

focused inquiries. Much of this design process will occur on-line through electronic conferencing among NCES, data users, and data providers. Even though

NCES will probably still produce many of its own reports, electronic versions

of these documents will contain numerous electronic links to other data sets,

technical references, and related reports. They may also contain interactive software that will permit users to perform “what if” analyses while perusing a

report and to generate customized tables or graphics. Alternatively, users will

generate their own electronic reports and analyses and create links to NCES

documents residing in electronic networks. Precisely what is generated by

NCES and what is generated by others may become less easy to distinguish.

Stage Three is not here yet, and it will probably look quite different from this

admittedly inchoate prediction. However, this stage will probably arrive much

sooner than expected. The more NCES can anticipate and help shape these developments, the more likely it will be able to use them effectively to report on the

condition of education in the United States and other countries.

From Data to Information 31

ven though NCES will

probably still produce

many of its own reports,

electronic versions of

these documents will

contain numerous 

electronic links to other

data sets, technical 

references, and related

reports. They may also

contain interactive 

software that will permit

users to perform “what

if” analyses while 

perusing a report and to

generate customized

tables or graphics.

E
CONCLUSION

From data to information—transforming quantitative facts about education into

knowledge useful to policymakers, researchers, practitioners, and the general

public—this aim has always been central to the mission of the National Center for

Education Statistics. In and of itself, this objective is not a new direction for the

agency. However, what constitutes useful information and how it gets produced,

distributed, and used are changing. To keep pace with these changes, indeed to

stay out in front and help shape their development, NCES must chart some new

directions.

Probably the most fundamental change that NCES

will need to address is its emergence as a provider

of information services and systems, rather than a

primary collector and provider of data per se. In

today’s climate of growing demands for information, but limited resources to produce it, NCES will

need to pay particular attention to assuming new

roles as a facilitator, broker, translator, linkage, filter, and pathfinder in a complex web of providers

and users of education data. To these new roles,

the agency can bring a strong foundation of standards for high-quality data and analysis, as well as a firm understanding of the

kinds of information that are most relevant to deliberating national policy for

education.

As these new roles develop, NCES may find itself shedding or at least de-emphasizing old functions. Data collection that occurs independently of front-line

administrative and teaching systems and their own information needs is likely to

diminish significantly. This change, in combination with technological advances,

may lead to data collection systems that are far more decentralized, interactive,

and operating in “real time” than the systems that have traditionally supported

national surveys. It is even possible that eventually NCES may find that it is no longer in the data

collection business, as this function has traditionally been defined. Instead, it will be primarily a systems manager and analyst, a producer and broker

of information for ongoing nationally oriented

assessments, as well as thousands of state and local

customized queries. Data collection and storage,

however, may occur largely outside of the immediate domain of NCES.

32 From Data to Information

he most fundamental

change that NCES will

need to address is its

emergence as a provider

of information services

and systems, rather than

a primary collector and

provider of data per se. 

T

ata collection that

occurs independently of

front-line administrative

and teaching systems

and their own information needs is likely to

diminish significantly.

D
“Reporting statistics and information showing the condition and progress of education in the United States and other nations in order to promote and accelerate

the improvement of American education”—this charge is a lasting mission for the

National Center for Education Statistics. Fulfilling it successfully will require

careful attention to changing national priorities, a strong commitment to improving education research and practice, and an openness to recognizing and adopting important advances in methods and technology.

From Data to Information 33
NOTES

1. U.S. Department of Education, National Center for Education Statistics, “Common Core of

Data” and “Financial Statistics of Institutions of Higher Education,” surveys and unpublished

data, FY 94–95.

2. Section 402(b) of the National Education Statistics Act of 1994 (20 U.S.C. 9001). 

3. Indeed, there are now standards for all of NCES major activities—survey planning and testing,

statistical processing, data provision and analysis, evaluation and documentation, and contract

management and operations. See U.S. Department of Education. 1992. NCES Statistical

Standards. Washington, D.C.: National Center for Education Statistics.

4. These follow the principles for a federal statistical agency developed in Martin and Straf, eds.

(1992). 

5. For a full description of the current program of work at NCES, see U.S. Department of

Education. 1995a. Programs and Plans: 1995 Edition. Washington, D.C.: National Center for

Education Statistics.

6. SASS also surveys private school teachers in a sample of schools drawn from the Private School

Universe file maintained by NCES. 

7. At present, legislation requires that NAEP assess reading and mathematics every 2 years; science and writing at least every 4 years; and history, geography, and other subjects determined

by the National Assessment Governing Board at least every 6 years. 

8. In particular, see the following papers in From Data to Information: New Directions for the

National Center for Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S.

Department of Education, National Center for Education Statistics, 1996): D. Brewer and C.

Stasz, “Enhancing Opportunity to Learn Measures in NCES Data,” pp. 3-1–3-28; D. Mandel,

“Teacher Education, Training, and Staff Development: Implications for National Surveys,” pp.

3-29–3-42; M. McPherson and M. Schapiro, “Tracking the Costs and Benefits of Postsecondary

Education: Implications for National Surveys,” pp. 6-1–6-12; D. Breneman and F. Galloway,

“Special Issues in Postsecondary Education and Lifelong Learning,” pp. 6-13–6-28; P. Cappelli,

“Education and Work: Curriculum, Performance, and Job-Related Outcomes,” pp. 8-1–8-34. 

9. See especially the following papers in From Data to Information: New Directions for the National

Center for Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of

Education, National Center for Education Statistics, 1996): J. Jennings and D. Stark, “Tracking

Education Reform: What Type of National Data Should Be Collected Through 2010?” pp. 2-

1–2-11; C. Cross and A. Stempel, “Where Are We Going? Policy Implications for Data

Collection Through 2010,” pp. 2-12–2-18.

10. See the following papers in From Data to Information: New Directions for the National Center for

Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of Education,

National Center for Education Statistics, 1996): C. Metcalf, “Incorporating Experimental

Designs Into New NCES Data Collection Methodologies,” pp. 5-1–5-18; R. Boruch and G.

Terhanian, “‘So What?’ The Implications of New Analytic Methods for Designing NCES

Surveys,” pp. 4-1–4-115. 

11. See the discussion in Boruch and Terhanian, “‘So What?’.” These authors also suggest that

NCES consider adopting a “satellite” policy that would permit including controlled experimental studies in national surveys in a fashion similar to the way NASA allows adjuncts to

space missions for astrophysicists and others. 

12. See Brewer and Stasz, “Enhancing Opportunities,” and Mandel, “Teacher Education.” 

34 From Data to Information
13. For a summary of trends in technological capacity to store, retrieve, and analyze data, see G.

Ligon, “New Developments in Technology: Implications for Collecting, Storing, Retrieving,

and Disseminating National Data for Education,” pp. 9-32–9-65 in From Data to Information:

New Directions for the National Center for Education Statistics, ed. Gary Hoachlander

(Washington, D.C.: U.S. Department of Education, National Center for Education Statistics,

1996). 

14. See J. Stigler, “Large-Scale Video Surveys for the Study of Classroom Processes,” pp. 7-1–7-29

in From Data to Information: New Directions for the National Center for Education Statistics, ed. Gary

Hoachlander (Washington, D.C.: U.S. Department of Education, National Center for Education

Statistics, 1996). 

15. See McPherson and Schapiro, “Tracking the Costs.” 

16. While SASS provides additional data on salaries of administrative and instructional personnel

at the elementary and secondary level, NPSAS provides additional postsecondary information

on tuition and costs. 

17. See Cappelli, “Education and Work.” 

18. There are important confidentiality considerations that must be addressed if this kind of service were to be provided. However, with explicit attention to benchmarking at the outset of a

survey, problems surrounding confidentiality could be reduced. 

19. See Mandel, “Teacher Education.” 

20. Video already plays an important role in the certification process used by NBPTS, and NCES

could build on the experience of the Board, as well as that of other researchers developing this

technology. 

21. See the following papers in From Data to Information: New Directions for the National Center for

Education Statistics, ed. Gary Hoachlander (Washington, D.C.: U.S. Department of Education,

National Center for Education Statistics, 1996): G. Ligon, “New Developments in Technology:

Implications for Collecting, Storing, Retrieving, and Disseminating National Data for

Education,” pp. 9-32–9-65; F. Scheuren, “Administrative Record Opportunities in Education

Survey Research,” pp. 9-1–9-31. 

22. Electronic recordkeeping is still far from universal, especially at the elementary and secondary

levels; paper files are still the norm in many places. 

23. These are developed in more detail in the papers by Boruch and Terhanian, “‘So What?’,”

Metcalf, “Incorporating Experimental Designs,” and Scheuren, “Administrative Record

Opportunities.”

24. For more information about specific opportunities for NCES to use electronic networks, see the

papers by Boruch and Terhanian, “‘So What?’,” Ligon, “New Developments,” and Scheuren,

“Administrative Record Opportunities.”

25. The continuing development of “object technology,” a technique for more rapidly constructing

software programs out of many small object modules, should hasten the explosion of this sort

of interactive dissemination and sharing of programs. 

From Data to Information 35
REFERENCES

Levine, D. B. (Ed.). 1986. Creating a Center for Education Statistics: A Time for Action. Washington, DC:

National Academy Press. 

Martin, M. E., and Straf M. S. (Eds.). 1992. Principles and Practices for a Federal Statistical Agency, p. 2

Washington, DC: National Academy Press. 

U.S. Department of Commerce. 1995. Statistical Abstract, table 699, p. 451. Washington, DC: Bureau

of the Census. 

U.S. Department of Education. 1992. NCES Statistical Standards. Washington, DC: National Center

for Education Statistics. 

U.S. Department of Education. 1995a. Programs and Plans: 1995 Edition. Washington, DC: National

Center for Education Statistics. 

U.S. Department of Education. 1995b. Digest of Education Statistics 1995, table 3, p. 12. Washington,

DC: National Center for Education Statistics. 

36 From Data to Information