U.S. DEPARTMENT OF EDUCATION 

Use of Education Data at the Local Level 

From Accountability to Instructional Improvement 

 
 

 
 

Use of Education Data at the Local Level 

From Accountability to Instructional Improvement

U.S. Department of Education 

Office of Planning, Evaluation and Policy Development 

Prepared by: 

Barbara Means 

Christine Padilla 

Larry Gallagher 

SRI International 

2010 
This report was prepared for the U.S. Department of Education under Contract number ED-01-

CO-0040 Task 0002 with SRI International. Bernadette Adams Yates served as the project 

manager. The views expressed herein do not necessarily represent the positions or policies of the 

Department of Education. No official endorsement by the U.S. Department of Education is 

intended or should be inferred. 

U.S. Department of Education 

Arne Duncan 

Secretary 

Office of Planning, Evaluation and Policy Development 

Carmel Martin 

Assistant Secretary 

Policy and Program Studies Service 

Alan Ginsburg 

Director 

Office of Educational Technology 

Karen Cator 

Director 

January 2010 

This report is in the public domain. Authorization to reproduce this report in whole or in part is 

granted. While permission to reprint this publication is not necessary, the suggested citation is: 

U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Use of 

Education Data at the Local Level From Accountability to Instructional Improvement,

Washington, D.C., 2010. 

This report is available on the Department’s Web site at 

http://www.ed.gov/about/offices/list/opepd/ppss/reports.html#edtech 

On request, this publication is available in alternate formats, such as Braille, large print, or 

computer diskette. For more information, please contact the Department’s Alternate Format 

Center at 202-260-0852 or 202-260-0818. 

ii
Contents 

List of Exhibits ......................................................................................................................... iv 

Acknowledgments................................................................................................................... vii 

Executive Summary ................................................................................................................. ix 

1. Introduction and Approach ....................................................................................................1 

Data Systems: A Prerequisite for Data-driven Decision Making ..........................................2 

Beyond Data Systems: Need for a Systemic Approach ........................................................2 

Data-driven Decision-making Framework ............................................................................2 

Data Sources for the Report ..................................................................................................5 

Contents of the Report ...........................................................................................................6 

2. District Data System Features and District Use of Data for Decision Making .....................9 

A Profile of District Data Systems ......................................................................................10 

District Data System Features .............................................................................................13 

Districts’ Use of Data Systems ............................................................................................23 

Variations Across Districts ..................................................................................................26 

Summary .............................................................................................................................27 

3. District Supports for Data-driven Decision Making in Schools ..........................................29 

District Data-driven Decision-making Support Index .........................................................29 

District Strategies for Increasing Use of Data Systems ......................................................33 

Challenges to District Efforts to Spread Data-driven Decision Making .............................42 

Strategies for Addressing Implementation Challenges .......................................................54 

Summary .............................................................................................................................57 

4. School Use of Data and Supports for Data-driven Decision Making ..................................59 

Nature and Frequency of School Use of Data and Data Systems .......................................59 

Year-to-Year Growth in Data Use .......................................................................................62 

Stages in Developing a Data-using Culture ........................................................................64 

School Supports for Data Use .............................................................................................68 

Barriers to School Data Use ................................................................................................76 

Perceived Effects of Data Use .............................................................................................80 

Summary .............................................................................................................................81 

iii
5. Conclusion and Policy Recommendations...........................................................................83 

Supports Needed for Data-driven Decision Making ...........................................................83 

Data-driven Decision Making as a Systemic Reform .........................................................86 

Recommendations ...............................................................................................................87 

References ................................................................................................................................93 

Appendix A: Methods ........................................................................................................... A-1 

Appendix B: 2007–08 District Survey and Frequencies ........................................................B-1 

Appendix C: Terminology and Additional Data ....................................................................C-1 

Exhibits 

Exhibit ES-1 Types of Electronic Student Data Systems ........................................................x 

Exhibit ES-2 Districts With Student Data Systems, by Type ..................................................x 

Exhibit ES-3: District Data System Query Capabilities in 2007–08 ..................................... xii 

Exhibit ES-4: District Perceptions of Needed Examples of Good Practice ......................... xiv 

Exhibit ES-5: Uses of Data Described by Case Study Schools ..............................................xv 

Exhibit ES-6: Categories of Data-driven Decision Making ................................................. xvi 

Exhibit ES-7: Percentage of Case Study Schools Reporting Each Decision-making 

Category ......................................................................................................... xvi 

Exhibit ES-8: Teacher Perceptions of the Support They Receive for Data Use ................. xviii 

Exhibit 1-1: Conceptual Framework for Data-driven Decision Making ...............................3 

Exhibit 2-1: Types of Electronic Student Data Systems ........................................................9 

Exhibit 2-2: Profile of District Data System Elements in 2007–08 .....................................11 

Exhibit 2-3: District History With Student Data Systems, by Type ....................................15 

Exhibit 2-4: District Implementation of Student Data Systems ...........................................16 

Exhibit 2-5: Types of Information That Districts Maintained Electronically in 2007–08 ...20 

Exhibit 2-6: District Data System Query Capabilities in 2007–08 ......................................22 

Exhibit 2-7: District Data System Features and Tools in 2007–08 ......................................22 

Exhibit 2-8: Districts Using Data Systems for Selected Purposes .......................................24 

Exhibit 3-1: District Data-driven Decision-making Support Profile ...................................31 

Exhibit 3-2: District Strategies to Promote Data-driven Decision Making 

From 2005 to 2007 ...........................................................................................33 

Exhibit 3-3: Systemic Change Guides District Implementation of Data-driven 

Decision-making Practices...............................................................................34 

Exhibit 3-4: District-provided Training on the Use of Data and Data Systems ...................36 

Exhibit 3-5: District-provided Supports for School-level Use of Data to Improve

Instruction ........................................................................................................39 

iv
Exhibit 3-6: District Policies and Practices Encouraging Schools’ Use of Data .................40 

Exhibit 3-7: District Administrators’ Perceptions of Barriers to Increased Use

of Data Systems ...............................................................................................43 

Exhibit 3-8: District Leadership Support for Data-driven Decision Making .......................46 

Exhibit 3-9: District Perceptions of Needed Examples of Good Practice ...........................48 

Exhibit 3-10: District Administrators’ Perceptions of Barriers to Increased Use

of Data Systems, by District Size ....................................................................50 

Exhibit 3-11: Case Study District Supports for School Use of Student Data ........................52 

Exhibit 3-12: District Perceptions of Major Needs and Barriers, by Longevity of 

Data-driven Decision-making Efforts ..............................................................53 

Exhibit 3-13: Piloting Data-driven Decision-making Efforts ................................................57 

Exhibit 4-1: Teachers Who Reported Using a Student Data System at

Least a Few Times a Year for a Specific Function in 2007 .............................60 

Exhibit 4-2: Uses of Data Described by Case Study Schools ..............................................61 

Exhibit 4-3: Assessment Terminology .................................................................................63 

Exhibit 4-4: Categories of Data-driven Decision Making ...................................................66 

Exhibit 4-5: Percentage of Case Study Schools Reporting Each Decision-making 

Category ...........................................................................................................67 

Exhibit 4-6: A School at Stage 3 in Data Use ......................................................................68 

Exhibit 4-7: Teacher Reported Supports for Use of Student Data to Guide

Instruction, by Type of Support in 2007 ..........................................................68 

Exhibit 4-8: Teacher Perceptions of the Support They Receive for Data Use .....................75 

Exhibit A-1: Distribution of Districts and Student Population, by District Size ............... A-3 

Exhibit A-2: Distribution of Districts and Student Population, by District

Poverty Rate .................................................................................................. A-4 

Exhibit A-3: Number of Districts in the Universe and Sample Size, by Stratum .............. A-5 

Exhibit A-4: Case Study Districts and School Sample in 2007–08 ................................... A-7 

Exhibit C-1: Definitions of Terms ......................................................................................C-3 

Exhibit C-2: Top 15 Systems for Each Component ...........................................................C-5 

Exhibit C-3: Profile of District Data System Elements in 2007–08, by District Size ........C-6 

Exhibit C-4: District Data System Query Capabilities in 2007–08, by District Size .........C-8 

Exhibit C-5: District Data System Features and Tools in 2007–08, by District Size .........C-9 

Exhibit C-6a: Districts Using Data Systems for Selected Purposes, by 

District Size ..................................................................................................C-10 

Exhibit C-6b: Districts Not Conducting Selected Activities, by District Size ...................C-12 

Exhibit C-7: District Data-informed Decision-making Support Index, by

District Size ..................................................................................................C-13 

Exhibit C-8: District-provided Training on the Use of Data and Data Systems ...............C-15 

Exhibit C-9: District-provided Supports for School-level Use of Data to 

Improve Instruction ......................................................................................C-16 

v
vi

Exhibit C-10: District Policies and Practices Encouraging Schools’ Use of Data .............C-17 

Exhibit C-11: District Administrators’ Perceptions of Barriers to Increased Use 

of Data Systems ...........................................................................................C-18 

Exhibit C-12: District Perceptions of Needed Examples of Good Practice .......................C-20 

Exhibit C-13: Longevity of District Engagement in Helping Schools Use Data, 

by District Size.............................................................................................C-21 

Exhibit C-14: District Administrators’ Perceptions of Need for Examples of 

Good Practice, by District Size ....................................................................C-22 

Exhibit C-15: District Administrators’ Perceptions of Barriers to Increased Use

of Data Systems, by District Size ................................................................C-23 

Exhibit C-16: Districts Conducting Selected Activities, by Percent of Schools Not

Making AYP ................................................................................................C-26 
Acknowledgments 

Many individuals contributed to the completion of this final report. We are particularly grateful 

to the district and school-level staff who took time out of their busy schedules to respond to our 

requests for information: in particular, the 12 districts and 36 case study schools were generous 

with both their time and attention to this evaluation work. Without their efforts, this report would 

not have been possible, and we deeply appreciate their assistance. 

We would like to acknowledge the thoughtful contributions of the members of our Technical 

Work Group in reviewing study materials and prioritizing issues to investigate. The advisors 

consisted of Katherine Conoly of Corpus Christi ISD, Marty Daybell of Washington Schools 

Information Processing, Aimee Guidera of the National Center for Educational Accountability, 

Glynn Ligon of ESP Solutions, Ellen Mandinach of CNA Corporation, Jim Pellegino of the 

University of Illinois-Chicago, Arie van der Ploeg of Learning Point Associates, and Jeff 

Wayman at the University of Texas at Austin. 

Many U.S. Department of Education staff contributed to the completion of this report. 

Bernadette Adams Yates served as project manager and provided valuable substantive guidance 

and support throughout the design, implementation and reporting phases of this study. We would 

also like to acknowledge the assistance of other Department staff in reviewing this report and 

providing useful comments and suggestions, including David Goodwin, Daphne Kaplan, 

Vicotria Hammer, Vicki Robinson, Maureen Dowling, Lee Hoffman, Kashka Kubdzela, Paul 

Strasbery, and Larry Cohen. 

We appreciate the assistance and support of all of the above individuals; any errors in judgment 

or fact are, of course, the responsibility of the authors. 

The Study of Education Data Systems and Decision Making is supported by a large project team 

at SRI. Among the staff who contributed to the research were Maria Abasi, Marianne Bakia, 

Janeula Burt, Sara Carriere, Lauren Cassidy, Eva Chen, Angela DeBarger, Larry Gallagher, 

Marilyn Gillespie, Torie Gorges, Ann House, Harold Javitz, Karla Jones, Aasha Joshi, Carlin 

Llorente, Bladimir Lopez-Prado, Patrik Lundh, Nicolette Mayes, Natalie Nielsen, Christina Park, 

Angeline Reyes, Elizabeth Rivera, Corinne Singleton, Tina Stanford, Edith Yang, Kaily Yee, and 

Viki Young. Layout and editing were performed by Eileen Behr and Meredith Itner. 

vii

Executive Summary 

The use of student data systems to improve education and help students succeed is a national 

priority. The Elementary and Secondary Education Act, as reauthorized in 2002, calls for the 

collection, analysis, and use of student achievement data to improve school outcomes. Data 

systems are expected to play an integral role in improving educational decision making at all 

levels—including that of the classroom teacher. The U.S. Department of Education has provided 

support for major improvements in the quality of state data systems to enable longitudinal 

analysis of student data and linkage between student outcomes and other education system 

variables. These improved systems are supporting educational research and decision making at 

the state level, but at the local level, district and school staff work with district rather than state 

data systems. If data-driven decision making is to become an effective tool for improving the 

instruction provided to students, policymakers need a clear understanding of these data systems 

used at the local level and of the decision-making processes in schools and districts. 

Since 2006 the national Study of Education Data Systems and Decision Making, sponsored 

by the U.S. Department of Education’s Policy and Program Studies Service, has been examining 

both the implementation of student data systems per se and the broader set of practices involving 

the use of data to improve instruction, regardless of whether or not the data are stored in and 

accessed through an electronic system. The study’s data collections included a national survey of 

districts in spring 2007 and site visits during school years 2006–07 and 2007–08 to a purposive 

sample of districts and schools selected on the basis of their active involvement in the use of data 

for instructional improvement. The study team also conducted secondary analyses of national 

teacher survey responses to questions concerning data system access and use. 

Earlier study reports have documented a dramatic increase in the proportion of teachers with 

access to a student data system between 2005 and 2007 and described school practices with 

respect to data use and the challenges that are part of student data system implementation. This 

final report builds on the picture of local practices in implementing data-driven decision making 

provided in the earlier reports by presenting data from the national district survey as well as from 

site visits conducted during 2007–08 at 36 schools in 12 districts. 

Key findings from surveys and site visits with respect to district data systems and strategies 

for supporting the use of data in instructional decision making are described below. 

District Data Systems and Use of Data for Decision Making 

Districts are still in the process of building their data system technology capacity. An 

examination of district capacity with respect to data systems needs to take into account the 

multiple types of systems containing data concerning students and other aspects of the education 

system (see Exhibit ES-1). Nearly all school districts have an electronic student information 

system providing real-time access to information such as enrollment and attendance. According 

to district survey respondents, the majority of districts (70 percent) have had this type of system 

ix
for six or more years (see Exhibit ES-2). More recently, districts are acquiring other types of 

electronic data systems: 79 percent report having an assessment system that organizes and 

analyzes benchmark assessment data, 77 percent report having a data warehouse that provides 

access to current and historical data on students as well as data on other aspects of district 

functioning, and 64 percent report having an instructional or curriculum management system to 

support access to curriculum and instructional resources. 

Exhibit ES-1. Types of Electronic Student Data Systems 

1. Student information systems provide real-time access to student data such as attendance, 

demographics, test scores, grades and schedules. 

2. Data warehouses are electronic data collection and storage systems that provide access to 

current and historical data on students, personnel, finances and so on. 

3. Instructional or curriculum management systems provide a unifying framework to support 

access to curriculum and instructional resources such as planning tools, model lesson 

plans, creation of benchmark assessments, linkage to state content or performance 

standards, communication and collaboration tools (e.g., threaded discussion forums). 

4. Assessment systems support rapid organization and analysis of benchmark assessment 

data. 

Source: Wayman (2005). 

Exhibit ES-2. Districts With Electronic Student Data Systems, by Type 

79%

77%

64%

0% 20% 40% 60% 80% 100%

Student information system

Assessment system

Data warehouse

Instructional/curriculum

management system

Percent of districts that have this type of system

System Type

>99%

Exhibit reads: In 2007–08, more than 99 percent of districts reported 

having a student information system. 

Source: 2007–district survey question 4. 

x
The district survey found no significant relationships between district size or the proportion 

of a district’s students living in poverty and the likelihood of having a particular type of student 

data system.1

 The few districts without any electronic data system (10 out of 427 districts in the 

survey sample) tended to be smaller in size than other districts in the survey sample but similar 

in terms of the percentage of students in poverty and the percentage of their schools that are in 

Title I or not making AYP. 

At this point in time, districts are looking for a way to effectively link their multiple 

data systems since there is no “single solution” data system and there is no simple recipe 

for effective system implementation. Most districts have multiple, distinct data systems. The 

number of electronic data systems being used to support decisions about instruction in the case 

study districts ranged from three to seven. Although not a problem in principle, the use of 

multiple systems can be a problem in practice. On the district survey, over 60 percent of districts 

reported that lack of interoperability across data systems was a current barrier to expanded use of 

data-driven decision making. 

Districts’ initial acquisition of data systems and use of data has been driven by 

accountability requirements. The Elementary and Secondary Education Act requirements that 

states and districts report on the progress of every student subgroup toward academic proficiency 

and close the proficiency gap among student subgroups have both motivated districts to acquire 

data systems and shaped the nature of those systems. Districts are much more likely to have 

electronic systems with data such as student demographics and test scores than to have the ability 

to combine data from different types of systems or to link instructional resources to achievement 

data. Over 90 percent of the districts surveyed reported having electronically stored data on 

student demographics and attendance, student grades, student test scores on statewide 

assessments, and student course enrollment histories. In contrast, less than half of districts have 

electronic data systems that allow them to link outcomes to processes as required for 

continuous improvement. For example, only 42 percent of districts can generate data reports 

showing student performance linked to participation in specific instructional programs, and just 

over a third (38 percent) can execute queries concerning student performance linked to teacher 

characteristics. 

To support better decisions about instruction, data systems should make available data on the 

same student or group of students over time and support looking at the performance of students 

with different educational experiences (i.e., different teachers or instructional programs). Exhibit 

ES-3 shows the percentage of districts reporting that their systems include such tools for data 

use. 

 

 

1

 For the remainder of this report, the percentages reported will be for districts indicating that they have an 

electronic student information data system (which is greater than 99 percent of all surveyed districts) unless 

otherwise specified. 

xi
Exhibit ES-3. District Data System Query Capabilities in 2007–08 

Type of Query 

Percent of 

Districts 

With This 

System 

Capability 

Percent of 

Students 

Represented 

by These 

Districts 

Individual student history over time (e.g., cumulative grades)* 83 88 

Drill-down capability (ability to query a school-level finding to efficiently 

examine a subset of data at the grade, classroom or student level) 76 85 

Individual student assessment performance over time** 72 85 

Student performance linked to specific teachers** 67 78 

Student performance linked to specific instructional programs 42 50 

Student performance linked to teacher information or characteristics 38 45 

Exhibit reads: In 2007–08, 83 percent of districts had electronic data systems that had the capacity to 

support queries about individual student histories over time. These districts serve approximately 

88 percent of the nations’ public school students. 

Note: Asterisks indicate that the proportion of districts that report having this query capability varies 

significantly by district size (* p < .05 and ** p < .01). 

Source: 2007–08 district survey questions 7 and 8. 

District Supports for Data-driven Decision Making in Schools 

Both the survey and case study data suggest that districts are taking steps to improve the 

capacity of their schools to use data in decision making. 

One of the most commonly reported district policies to encourage schools’ use of data is 

to incorporate this practice into school improvement planning. Sixty-nine percent of districts 

reported requiring all or some of their schools to follow specific data-driven decision making 

practices in formulating their school improvement plans, and 65 percent of districts provide 

teachers with specific processes for how they should use data for instructional purposes. 

The most common district strategies for building school capacity for using data are 

professional development activities, providing support positions for system 

implementation, and the development of tools for generating data and tools for acting on 

data. Over 90 percent of districts responding to the survey reported that they have provided at 

least some school staff with training designed to enhance the school’s capacity to use data in 

ways that improve instruction (e.g., training principals or other school administrators on using 

the data system to analyze student achievement and to provide leadership for data-driven 

decision making in their school). But in many cases training has not been extended to all of the 

district’s schools. Training for teachers on how to use the data system to analyze student 

achievement or on how to use data to change their instructional practices are the two types of 

training least likely to have been given to every school (both provided to all their schools by 53 

percent of districts). 

xii
Another common district support is providing technical experts in systems, networks or 

databases who can help school staff get access to data from electronic systems. Eighty 

percent of districts say they have provided their schools with this kind of technical expertise, and 

65 percent say they have made such technical expertise available for all of their schools. In 

contrast, making data analysis experts (sometimes called “data coaches”) available to school 

staff is one of the least common supports. Still, 50 percent of districts say they have done this for 

at least some of their schools, and 32 percent say that they have done this for all of their schools. 

Other supports cited by districts in the case study sample are: 

• Providing full- or part-time positions for staff who help teachers work with data, 

• Creating easy-to-read data “dashboards” to help teachers get information from data 

systems that are not user friendly, 

• Developing benchmark and formative assessments to provide teachers with more timely 

data to assess student progress and adjust instruction to meet student needs, and 

• Participating in a range of partnerships to support their efforts to implement data-driven 

decision making in their schools. 

It is important to keep in mind that case study districts were selected for their leadership in 

data use. Survey responses from the national district survey suggest that many districts believe 

that they themselves need examples of good practice in order to guide their schools’ 

implementation of data-driven decision making. The greatest perceived area of need among 

districts is for models of how to connect student data to instructional practice (see Exhibit 

ES-4). Districts want examples of how to identify which practices work best for which students 

and how to adapt instructional strategies to meet the needs of individual students. 

xiii
Exhibit ES-4. District Perceptions of Needed Examples of Good Practice 

14%

21%

21%

26%

33%

35%

37%

49%

50%

59%

54%

48%

49%

47%

0% 20% 40% 60% 80% 100%

Communicating with parents about student 

progress

Using assessment data to identify gaps in student 

achievement

Structuring the district organization and practices to 

support data-driven decision making

Collaborating and sharing ideas with colleagues 

regarding data inquiry and analysis issues

Developing curriculum-embedded formative 

assessments

Adapting instructional activities to meet students’ 

individual needs

Examining student data to identify which practices 

work best for which students

Percentage of Districts

Examples of Good Practice

Great Need Some Need

Exhibit reads: In 2007–08, 80 percent of districts reported that they had some or a great need for 

examples of good practice regarding the examination of data to identify which practices work best for 

which students. 

Source: 2007–08 district survey question 24. 

Use of Data Systems and Data-driven Decision Making in Schools 

Site visits suggest that in districts that are leaders in data-driven decision making, the use of 

data in schools is encouraged not through extensive formal professional development but 

rather through ongoing support from colleagues and instructional or data coaches who help 

teachers examine data for their students and develop instructional plans to meet student needs. 

Staff in case study schools described a higher level of support for data-driven decision making 

than did teacher respondents to the national teacher surveys administered in 2005 and 2007. As a 

group, the 36 case study schools appeared to be especially well-supported in the area of funding 

for time for teachers to meet together during the school day to work with data and the provision 

of school-based positions for coaching teachers on how to connect data to instructional 

strategies.

Even in districts that are actively promoting the use of data, however, school staff 

provided relatively few examples of teachers using data to diagnose areas in which they 

could improve the way they teach. The most common school-level uses of data described by 

teachers and school leaders in the case study sample are school improvement planning, 

xiv
curriculum decisions, and placement or grouping of students for instruction or support services.

During school site visits, principals, coaches, and teachers were asked to describe specific 

examples of using data to make instructional decisions. From the 36 schools visited in 2007–08, 

188 examples of using data to inform instruction were obtained. Exhibit ES-5 shows the different 

purposes for which school staff described using data and the number of distinct citations of each 

use. The data in Exhibit ES-5 suggest that in districts considered leaders in data-driven decision 

making, most schools are using data to develop goals for school improvement and to do 

curriculum planning. In contrast, school interviewees provided only eight examples of teachers 

using data to determine which aspects of their teaching are working well or poorly. 

Exhibit ES-5. Uses of Data Described by Case Study Schools 

Data Use Frequency 

School improvement planning, including setting of quantitative goals 35 

Curriculum planning based on item or subscale analysis 25 

Student placement in classes or special services 22 

Grouping or regrouping of students within a class 21 

Tailoring instruction to the skill needs of individuals or small groups 15 

Deciding whether or what to reteach 13 

Identifying teachers with more successful strategies in order to emulate 

their instructional approach 11 

Referring students from classroom for supports or services 9 

Determining what aspects of your teaching are working well/poorly 8 

Evaluating teacher performance 7 

Exhibit reads: School improvement planning was the most common data use described by 

staff at the 36 case study schools. 

Source: Case study schools 2007–08. 

In site visit schools, the use of data by teachers to improve their teaching practice 

emerged later than uses such as school improvement planning or student placement. The 

relative frequency of reports of different uses of data within the 36 case study schools suggests 

that different uses emerge over time. Analysts developed three categories of school-level datadriven decisions as shown in Exhibit ES-6. The first category covers a range of accountabilitydriven uses of data, the second encompasses matching teaching content to standards or tests and 

giving students adequate time to master the content, and the third category involves using data to 

explore the relative effectiveness of different teaching methods or interventions. 

 

xv
Exhibit ES-6. Categories of Data-driven Decision Making 

Decision Type 

Category 1: Staff examine data for whole grade or school to ascertain areas for school 

improvement; examine data for individual students for purposes of class placement or 

assignment to services, including identifying “bubble kids” whose growth is likely to affect the 

school’s AYP status. 

Category 2: Teachers analyze performance of students in their class on individual items or 

standards for purposes of better aligning their content coverage with the accountability test or 

deciding what to reteach or how to group students within the class. 

Category 3: Staff examine data for different teachers or for different methods dealing with the 

same content to derive insights for improving the way they teach. Staff use comparative data 

to evaluate the effectiveness of specific instructional strategies. 

When the examples of data use described by case study school staff were classified in terms 

of the three data-use categories described in Exhibit ES-6, analysts found that most case study 

schools provided examples of Category 1 and Category 2 uses of data but less than half of all 

schools provided a Category 3 example. Specifically, as shown in Exhibit ES-7, 35 of 36 case 

study schools (97 percent) provided one or more Category 1 examples; 30 of 36 (83 percent) 

provided one or more Category 2 examples; and 17 of 36 (47 percent) provided one or more 

Category 3 examples. 

ES-7. Percent of Case Study Schools Reporting 

Each Decision-making Category 

47%

83%

97%

0% 20% 40% 60% 80% 100%

Category 3

Category 2

Category 1

Data‐driven decision making categories

Percent of Schools

Exhibit reads: Among case study schools, 97 percent provided examples of 

Category 1 uses of data. 

Source: Case study schools 2007–08. 

xvi
One of the strongest levers that districts can use to increase their schools’ use of data 

systems is to provide timely interim assessment data on those systems. Eighteen of the 36 

schools visited by researchers during school year 2007–08 had been visited a year earlier as part 

of 2006–07 data collection activities. Researchers analyzed site visit reports of examples of 

teacher and school leader data use to ascertain whether or not there were changes in data-use 

practices at those schools between the 2006–07 and 2007–08 school years. Schools in districts 

involved in implementing a system of districtwide interim assessments were more likely than 

other schools to show an increase in data use from year to year, and also provided the most 

striking examples of positive changes in teacher data use practices. In 13 of the 18 schools, there 

was evidence of an increase in data use over this one-year period; in four schools there was no 

indication of a change in the frequency or nature of data use.2

 

Having a set of common assessments that everyone teaching the same content gives to 

their students at about the same time encourages teachers to sit down and share both their 

data and their teaching strategies. When multiple teachers have all given the same recent 

assessment to their students, they can compare their results to identify strengths and weaknesses 

at the class level, something that is not possible if teachers assess different content at different 

times. Researchers found at least some teachers using common assessments and comparing their 

assessment data with each other as a way of comparing and reflecting on their practice in close to 

half of the case study schools. 

Actions that principals can take to encourage teacher use of data include designing and 

implementing regular activities involving the examination of student data and the 

establishment of an organizational climate of trust and mutual respect. Principals encourage 

data use by setting an example through their own activities, designating all or part of teacher 

planning or professional development time as occasions for examining and reflecting on data, 

and communicating expectations around data use. In 25 of the 36 case study schools, researchers 

judged data use to be an important tool in the principal’s assumption of the role of instructional 

leader. Principals at 18 case study schools led whole-school or grade-level meetings dedicated to 

the analysis of data. Principals at several schools maintained schoolwide lists of students who 

were in danger of failing to attain state proficiency standards in order to keep on top of efforts 

being made to support these students during the year. At three of the case study schools 

principals met with teachers individually on a regular basis to discuss their students’ needs and 

the teacher’s plan for the class. 

School staffs’ perceptions of barriers to greater use of data include a sense of lack of 

time,3

 system usability issues, the perception that the data in the system are not useful, and 

district policies around curriculum coverage or pacing that prohibit modifying learning 

 

2

 One school’s change in the use of data could not be classified. 

3

 District survey respondents also reported that lack of time for school staff to conduct data-driven decision making 

activities was the greatest barrier to spreading data-driven decision making practices throughout their district (92 

percent of districts cited lack of time as a barrier). 

xvii
time to match student needs. Nationally representative responses on the teacher survey (see 

Exhibit ES-8) suggest that teachers who have access to a data system view their colleagues as a 

resource for data use but have to do much of their work with data on their own time. 

Exhibit ES-8. Teacher Perceptions of the Support 

They Receive for Data Use 

12%

5%

13%

29%

54%

59%

71%

70%

42%

37%

23%

14%

I can turn to someone for help

I am comfortable having colleagues with me

My school has improved through Data Management 

System use

Available data not helpful in deciding what to teach (R)

Time is available as part of regular day

If I use the Data Management System, it has to be my own 

time (R)

Teacher Perception

Negative Positive

Exhibit reads: Among teacher survey respondents with access to an electronic data 

system, 71 percent agreed with the statement “I can turn to someone for help.” 

Note: (R) denotes an item that is reverse coded; for these items a disagree 

response indicates more support for data use. 

Source: NETTS teacher survey, 2007. 

At case study schools, staff were much more likely to criticize the quality or timeliness of the 

tests for which data are available in systems than to criticize the usability of the system per se. 

Teacher criticisms of the quality of the data available to them included delay issues, lack of 

alignment with standards, lack of alignment with the school’s instructional approach, and 

xviii
the fact that they received only cross-sectional data rather than longitudinal data for the 

same set of students over time. 

State achievement tests are typically a once-a-year spring event with results unavailable until 

the next fall when students have moved into a new grade level. Staff at many case study schools 

described looking at these data at the time they became available, often using them for school 

improvement planning and for placing students into classes or special services, as noted above. 

Individual teachers look at the prior spring’s scores for the students coming into their class to 

help do planning at the first of the year but soon find more recent and fine-grained information 

from classroom assessments and informal observations guiding their instructional decisions.

Recommendations 

The surveys and case studies provide a portrait of system and organizational capacity for data 

use in schools and districts. A comparison of this portrait with the ideal of all schools using data 

systems to improve the educational experiences and outcomes for their students suggests a 

number of implications for policies and actions at the school, district, state, and national levels. 

The recommendations below are those of the study team. 

Recommendations for Schools 

• Set clear expectations around the use of student data as the basis for decisions. 

• Integrate collaborative exploration of data into existing structures for joint teacher 

planning and reflection on teaching. 

• Provide a safe environment for teacher examination of their students’ performance. 

• Support teachers in making the link between data and alternate instructional strategies. 

Recommendations for Districts 

• Think of data-driven decision making as an ongoing systemic process rather than a onetime event centered on the acquisition of a data system. 

• Model decision making based on data and present decision-relevant data when 

announcing new policies. 

• Train principals in how to integrate the use of data into school improvement planning 

and promote their teachers’ use of data for making instructional decisions. 

• Integrate the use of data-driven decision-making practices with district initiatives for 

improving instruction in specific areas. 

• Support time within the work week for teachers to meet with colleagues for planning, 

informal professional development, and data use. 

• Make sure that the district has a data system that gives teachers data that is both timely 

and relevant to their instructional decisions. 

xix
xx

• Provide resources and construct policies so that teachers have access to data relevant to 

the students they are teaching when and where they want it. 

Recommendations for State and National Policy 

• Complement efforts to improve state data systems with investments helping districts 

improve both their data systems and their organizational supports for using data to 

improve instruction. 

• Improve the turnaround time for state assessment data so that schools receive student 

results in time to inform academic-year planning. 

• Promote linkages between local data systems that contain interim assessment data and 

state systems with instructional resources geared to standards. 

• Encourage districts to invest in developing data literacy among district staff in all 

departments. 

• Encourage or require school administrator preparation programs to incorporate 

assessment and data literacy concepts to foster continuous improvement activities 

informed by data as elements of school leadership training. 

• Encourage or require teacher preparation programs to incorporate assessment concepts 

and the use of data for instructional decision making into their teaching methods courses

(science methods, language arts methods, and so on). 

• Provide districts with good examples of practices that support the development of a datause culture within schools.
1. Introduction and Approach 

Proponents of data-driven decision making call on educators to adopt a continuousimprovement perspective, with an emphasis on goal setting, measurement, and feedback loops so 

that they can reflect on their programs and processes, relate them to student outcomes, and make 

refinements suggested by the outcome data (CoSN 2004; Dattnow, Park, and Wohlstetter 2007; 

Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). For example, a 2009 forum of 

state and federal policymakers and education leaders, including U.S. Secretary of Education 

Arne Duncan, emphasized the need to use data systems for continuous improvement (Data 

Quality Campaign 2009b). But until fairly recently, the use of data to make decisions at the 

district, school and classroom levels has been the exception rather than the rule. The lack of 

systems providing user-friendly access to timely, relevant information was one major 

impediment (Wayman 2005). Another was a culture more accustomed to making decisions on 

the basis of educational philosophy or political necessity rather than data (Coburn and Talbert 

2006; Coburn, Toure, and Yamashita 2009). 

In the last decade, however, several forces have converged to make data-driven decision 

making at all levels of the education system a priority. Improved data systems capable of 

tracking individual students’ progress from year to year have become available and have been 

implemented in an increasing number of states and districts (Data Quality Campaign 2008). 

Moreover, some districts are instituting data warehouses that allow them to combine data from 

different systems (for example, student achievement and teacher qualifications) in ways that 

allow investigating education quality issues (Wayman, Stringfield, and Yakimoski 2004). The 

Elementary and Secondary Education Act (ESEA), as reauthorized in 2002, has instituted 

requirements for achievement data reporting by student subgroup, requiring many districts to 

obtain or upgrade their student data systems, and has made schools and districts responsible for 

student achievement. As districts and schools have looked for strategies to help raise 

achievement, the use of data to predict and enhance student performance has emerged as perhaps 

the dominant improvement strategy. Studies of district and school achievement have documented 

a relationship between active use of data and increases in achievement (Datnow, Park, and 

Wohlstetter 2007; Snipes, Doolittle and Herlihy 2002). 

To understand the role of data systems and the supports necessary for teachers to use data 

from any source (electronic and nonelectronic) to inform educational practice, the U.S. 

Department of Education’s Policy and Program Studies Service (PPSS) sponsored a national 

study of Education Data Systems and Decision Making. The study addressed a set of basic 

questions: 

1. What kinds of systems are available to support district and school data-driven decision 

making? Within these systems, how prevalent are tools for generating and acting on data? 

2. How prevalent are organizational supports for school use of data systems to inform 

instruction? 

3. How are school staff using data systems? Do they know how to interpret student data? 

How is school staffs’ use of data systems and of data more broadly influencing 

instruction? 

1
This report describes findings from the study’s second round of data collection and analysis, 

conducted during the 2007–08 school year. 

Data Systems: A Prerequisite for Data-driven Decision Making 

For many years, local education agencies (LEAs) have developed and used multiple data 

systems for different purposes (for example, separate systems for finance data, personnel data, 

required accountability information for special education students, school lunch data, enrollment 

and attendance, assessment data). Historically, these data systems were so complex and poorly 

aligned that their use by school staff was not feasible (Wayman and Cho 2009). Moreover, the 

lack of persistent student and teacher identifiers in many of these systems made it impossible to 

obtain a longitudinal view of students’ history within the system or to follow students if they 

transferred schools. Advances in technology and recent policy emphasis on data use have 

resulted in much improved data infrastructures in many districts (Wayman and Cho 2009). 

Beyond Data Systems: Need for a Systemic Approach 

Policymakers envision wide-ranging educational data systems that collect, analyze, and use 

relevant data at every level—from the U.S. Department of Education to the individual teacher 

making instructional decisions for his or her classroom. The mere presence of a student data 

system within a district is a prerequisite but clearly insufficient to bring about change in the way 

that decisions impacting instruction get made. 

Within a school district, many offices (or departments) as well as its schools need to 

participate in order to bring about such a fundamental change in educational decision making 

(Datnow, Park and Wohlstetter 2007; Wayman, Cho, and Johnston 2007). The district needs to 

fund and acquire (or build) a data system capable of storing and manipulating the kinds of 

information needed for educational decisions. Provisions need to be made by the district to 

ensure that intended data system users (district staff, school leaders, instructional coaches, and 

teachers) have ready access to the data in a form they can comprehend and manipulate. 

Assessment and curriculum departments need to make sure that there is up-to-date information 

from assessments linked to the local curriculum available for teacher use. Professional 

development offerings need to include opportunities to learn how to make instructional decisions 

based on data and, if applicable, how to get data out of the system and analyze it to fit one’s 

needs. 

Data-driven Decision-making Framework 

Exhibit 1-1 shows the stages in a data-driven continuous-improvement process: plan, 

implement, assess, analyze data, and reflect (as a precursor to more planning and a refined 

implementation). As the graphic suggests, components of data-driven decision making are part of 

a continuous cycle. The starting point may vary, and there is no fixed end point. 

2
Exhibit 1-1. Conceptual Framework for Data-driven Decision Making

2. Leadership for 

improvement 

and use of 

data 

3: Tools for 

generating 

data 

4. Social 

structures and 

time set aside 

for reflection on 

data 

6. Tools for 

acting on data 

5. Professional 

development 

and technical 

support for data 

interpretation 

Prerequisites and Supporting Conditions 

1. Data systems 

A major cultural change is required if educators are to make the continuous-improvement 

perspective and the processes of data-driven decision making part of the way in which they 

function. Such a change will not occur without leadership, effort, and well-designed supports. 

The bottom portion of Exhibit 1-1 identifies six major types of prerequisites and supports for 

data-driven decision making that are part of the study’s conceptual framework: 

• State, district, and school data systems 

• Leadership for educational improvement and the use of data 

• Tools for generating actionable data 

• Social structures and supported time for analyzing and interpreting data 

• Professional development and technical support for data interpretation 

• Tools for acting on data 

3
Data Systems 

ESEA has stimulated an unprecedented level of state activity aimed at improving education 

data systems. Federal requirements for reporting schools’ year-to-year progress in raising the 

percent proficient overall and for specific student categories have led to an examination of 

information system adequacy and the adoption or development of new software systems in many 

states and districts. For example, in 2005, 36 states used unique student identification numbers 

statewide so that students could be followed if they changed districts. In 2008, 48 states did so 

(Data Quality Campaign 2008). As of 2008, state data systems typically include student 

enrollment information, basic demographic data, special program designation (if applicable), and 

scores on state-mandated achievement tests (in most cases, an annual spring testing in language 

arts and mathematics and often a proficiency or “exit” examination required for a high school 

diploma). 

At the local level, it is district systems that are most likely to be used by teachers (U.S. 

Department of Education 2009). These systems typically include student scores on statemandated tests, which are obtained from the state or from the state-designated vendor. District 

systems are also likely to include student contact and demographic information, scores on district 

tests, attendance, grades, disciplinary infractions, and course enrollment. 

Leadership for Education Improvement and Use of Data 

Pioneering efforts to promote data-driven decision making within districts and schools have 

found that the active promotion of the effort on the part of the superintendent or principal is vital 

(Marsh, Pane, and Hamilton 2006; Supovitz and Klein 2003; Wayman and Stringfield 2006; 

Young 2006). District and school leaders issue the “call to arms” for improving education and 

using data as a tool to bring about that improvement. Typically, they play a major role in framing 

targets for educational improvement, setting expectations for staff participation in data-driven 

decision making, and making resources, such as supported time, available to support the 

enterprise. 

Tools for Generating Actionable Data 

Increasingly, student achievement data are available at the school level in a form that can be 

disaggregated by student category (ethnicity, free or reduced-price lunch status, special 

education status, etc.). Software systems to support data-driven decision making all generate 

standard student achievement reports, and many also produce custom reports for user-designated 

student groups (an important feature for school staff who want to examine the effects of locally 

developed services for specific student groups). Research indicates, however, that often school 

staff do not find the kinds of data these systems provide particularly useful for guiding 

instruction (Brunner et al. 2005). School staff are frustrated by the fact that the data available to 

them are typically performance on a state achievement test taken six or more months earlier. 

Teachers want up-to-date information on their current group of students, not the students in the 

same grade level the prior year. They also want a greater level of detail concerning individual 

students’ strengths and weaknesses than they can get from standardized test scores (Mandinach 

et al. 2006; Thorn 2002). Although far less common than systems that provide data from prior 

testing, there are examples of systems that produce additional information for decision making 

through tools such as formative assessments that students may take online. In addition, some 

4
system designers are working on educational information systems that will integrate data on a 

broad range of transactions, such as daily school attendance, grades, and even library book 

checkouts, with the ultimate goal of automatically recording each interaction a student has with 

the school and student’s assessment and program participation data. 

Social Structures and Supported Time for Analyzing and Interpreting Data 

The most sophisticated data warehouse in the world will have no effect on instruction if no 

one has—or takes—the time to look at the data, reflect on them, and draw inferences for 

instructional planning. Given that time is one of the most basic resources in any organization, 

there need to be strong expectations that administrators will provide time to teachers, and 

educators will take the time to examine data and use it to guide improvements in their programs 

and practices. Such expectations have not been business-as-usual in most schools and districts. 

Case studies of schools active in data-driven decision making suggest that organizational 

structures that include supported time for reviewing and discussing data in small groups greatly 

increase the likelihood that the examination of data will be conducted and will lead to wellinformed decisions (Copland 2003; Datnow, Park, and Wholstetter 2007; Wayman, Cho, and 

Johnston 2007; Wayman and Stringfield 2006). 

Professional Development and Technical Support for Data Interpretation 

Teacher training generally has not included data analysis skills or data-driven decisionmaking processes in the past (Mandinach et al. 2005; Massell and Goertz 2002). Few 

administrators have this kind of training either. Moreover, the measurement issues affecting the 

interpretation of assessment data—and certainly the comparison of data across years, schools, or 

different student subgroups—are complicated. Data misinterpretation is a real concern (Confrey 

and Makar 2005). For this reason, districts and schools are devoting increasing amounts of 

professional development time to the topic of data-driven decision making. Many argue that the 

practice of bringing teachers together to examine data on their students and relate those data to 

their practices is a valuable form of professional development in its own right (Feldman and 

Tung 2001; Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). 

Tools for Acting on Data 

The examination of data is not an end in itself but rather a means to improving decisions 

about instructional programs, placements and methods. Once data have been analyzed to reveal 

weaknesses in certain parts of the education program or to identify students who have not 

attained the expected level of proficiency, educators need to reflect on the aspects of their 

processes that may contribute to less-than-desired outcomes and to generate options for 

addressing the identified weaknesses. Some of the data-driven decision-making systems 

incorporate resources that teachers can use in planning what to do differently. These resources 

are typically organized around state content standards and may include lesson plans, instructional 

materials, or descriptions of best practices (Palaich, Good, and van der Ploeg 2004). Resources 

for differentiated instruction can help teachers adapt their instructional approach to students with 

differing strengths and weaknesses. 

Data Sources for the Report 

Findings from this report are drawn from both survey and case study data. The primary data 

set for this report consists of responses of a sample of 529 districts to a survey administered 

5
between October 2007 and February 2008. The sampling plan for the district survey was 

developed with the primary goal of providing a nationally representative sample of districts to 

assess the prevalence of district support for data-driven decision making. A secondary goal was 

to provide numbers of districts adequate to support analyses focused on subgroups of districts. 

Of the 529 districts in the final sample of the Study of Education Data Systems and Decision 

Making, 427 responded for a response rate of 81 percent. (Additional information on sampling is 

provided in Appendix A.) 

The case study districts were purposefully selected to include districts that have been active 

in using student data to guide instruction (as a result, these are not typical districts). By focusing 

fieldwork on districts in which many teachers could be expected to be actively looking at student 

data, the study team increased the likelihood of seeing the effects of data use on practice, 

compared with a sample of schools drawn at random. For the 2007–08 case study districts, the 

site selection process was two-staged. The first stage involved the selection of six of the nine 

2006–07 case study districts to be visited for a second round of data collection. The research 

team decided that these districts warranted a second visit because they had been in the process of 

implementing new data systems or activities to support the use of student data at the school level 

during the first round of data collection in 2006–07 and could provide a longitudinal perspective 

on implementation activities.4

 The second stage involved identifying an additional group of six 

districts that have been active in data-driven decision making. These districts were drawn from 

the pool of districts that remained after the initial selection of 10 districts in 2006, supplemented 

by additional districts identified as active data users. For each district visited, respondents 

included key staff involved in the district’s data-informed decision-making activities (e.g., chief 

information officers, directors of curriculum and instruction, directors of research and evaluation, 

directors of accountability, directors of professional development). Within each school, the 

principal, an instructional or data coach (if applicable), and six teachers were interviewed. 

A second survey data set from the U.S. Department of Education’s National Educational 

Technology Trends Study (NETTS) has also been included to provide a national picture of 

teacher perspectives on data use. These data consist of responses of a random sample of K–12 

teachers to a survey administered to 2,509 teachers in spring 2007. The teachers were clustered 

in schools sampled from districts participating in the NETTS study. Teachers were asked to 

report on activities during the 2006–07 school year.5

Contents of the Report 

This final report builds on findings from the interim report (U.S. Department of Education 

2009) that described the types of data available to school staff, how school staff use electronic 

data systems, school practices with respect to data-driven decision making, and the supports and 

challenges for school use of student data in planning and implementing instruction. This report 

presents district survey data that provides a national picture of data-driven decision-making 

practices and additional case study data. It also incorporates some of the NETTS teacher survey 

 

4

 For additional information on the district identification process and characteristics of the first set of case study 

districts, see Implementing Data-Informed Decision Making in Schools—Teacher Access, Supports and Use (U.S. 

Department of Education 2009). 5

 The response rate for the NETTS teacher survey was 86 percent. 

6
7

 

data presented previously in the brief entitled Teachers’ Use of Student Data Systems to Improve 

Instruction–2005 to 2007 (U.S. Department of Education 2007). 

Earlier study reports have documented a dramatic increase in the proportion of teachers with 

access to a student data system between 2005 and 2007. They also described school practices 

with respect to data use and the challenges that are part of student data system implementation. 

In the case study districts visited during the 2006–07 school year, school leaders demonstrated 

their support for school’s use of data by purchasing data systems, modeling data use, and 

providing school-based support positions. Teachers who use data from a student data system do 

so not only on their own but also in collaboration with colleagues. But both teachers and district 

staff members express concerns about teachers’ ability to understand data—many appear to lack 

data literacy skills. 

The data in this final report are primarily descriptive; they do not address the effects of datadriven decision making on student outcomes. At the same time, the findings go beyond most 

prior research that has tended to focus on case studies of individual districts and schools to 

provide a broader picture of data-driven decision-making implementation efforts in a dozen 

districts across the country.6

 Study findings lay a foundation on which to build future research 

efforts and will assist policymakers in understanding how data are being used at the local level, 

the conditions affecting use (both positively and negatively), and other issues that arise during 

implementation. 

The remainder of this report is divided into four chapters. Chapter 2 describes the data 

systems that districts are employing to support data collection, storage, retrieval, and analysis; 

the type of data that districts maintain electronically and the ways that districts use data to focus 

on accountability and instructional improvement. Chapter 3 explores district efforts to promote 

and support the use of student data and data systems within their schools, including the 

challenges this effort entails and the strategies that districts are using to overcome obstacles. 

Chapter 4 focuses on school-level implementation of data systems and data-driven decision 

making. Chapter 5 discusses cross-cutting themes and draws implications for policy in this area. 

6

 As noted on page 6, the 2006−07 case study sample for the current study included an additional three districts that 

were not revisited in 2007–08, which also helped inform the second year of data collection. 

2. District Data System Features and District Use of Data 

for Decision Making 

Fundamental to the concept of an information infrastructure to support data-driven decision 

making is the ability to share data across levels of the system (Wayman, Stringfield, and 

Yakimoski 2004). A data system infrastructure is composed primarily of hardware and software 

but also relies on people to gather data, extract the data and use it. Typically, district data 

systems are made up of multiple elements that have a unique role or set of attributes, and the key 

to maximizing the utility of these systems is the ability to integrate and share data files. To get a 

better picture of the information technology available to districts, district survey respondents, 

along with case study sites, were asked by the research team to provide information on the 

electronic student data system or systems that are driving instructional improvement in their 

districts. The study focused on electronic systems that contain data on students but also sought 

information on other electronic systems containing data that might be relevant to instruction (for 

example, data on the professional development received by teachers). The major types of 

electronic student data systems in widespread use are defined in Exhibit 2-1 (Wayman 2005) and 

were used to organize information drawn from the district survey regarding the kinds of data 

systems available to support district and school data-driven decision making (a copy of the 

survey can be found in Appendix B). 

Exhibit 2-1. Types of Electronic Student Data Systems 

1. Student information systems provide real-time access to student data such as attendance, 

demographics, test scores, grades and schedules. 

2. Data warehouses are electronic data collection and storage systems that provide access 

to current and historical data on students, personnel, finance and so on. 

3. Instructional or curriculum management systems provide a unifying framework to support 

access to curriculum and instructional resources such as planning tools, model lesson 

plans, creation of benchmark assessments, linkage to state content or performance 

standards, communication and collaboration tools (e.g., threaded discussion forums). 

4. Assessment systems support rapid organization and analysis of benchmark assessment 

data.

Two themes emerged from district survey responses and analysis of the case study data: 

• Currently, districts are looking for a way to effectively link their multiple data systems 

since there is no “single solution” data system and there is no simple recipe for effective 

implementation. 

• Districts’ initial acquisition of data systems and use of data has been driven by 

accountability requirements, most recently the ESEA requirements that states and districts 

report on the progress of student subgroups in aggregate toward achieving academic 

proficiency by 2014 and to close the proficiency gap among student subgroups. 

9
10

A Profile of District Data Systems 

To provide an overview or snapshot of the electronic data systems that U.S. school districts 

employ and those systems’ capabilities, responses from multiple survey items were combined 

into a handful of key data system elements to assess the extent to which district data systems are 

providing information to support better decisions about instruction (see Data Quality Campaign 

2008 for a similar approach to depicting state data systems). Data systems with a greater variety 

of data and tools that support looking at the performance of students with different educational 

experiences are much better equipped to support the kind of inquiry necessary for continuous 

improvement activities—the ongoing examination of student data to assess the effectiveness of 

education activities and to refine programs and practices to improve student outcomes. 

Analysts organized survey responses to examine ten elements of quality for district data 

systems (Exhibit 2-2): 

• Student information system 

• Data linkages 

• Instructional or curriculum management system 

• Assessment system 

• Student performance data to measure academic growth 

• Student-level enrollment, demographic and program participation information 

• Teacher-level data 

• Student-level graduation, post-graduation and dropout data 

• College readiness data 

• Data quality assurance 

The profile provides information on the percentage of districts with all of the subelements 

present for a particular element (i.e., districts responded positively to all of the survey items 

comprising that element). It was fairly unusual for districts to have all of the subelements. While 

65 percent of districts have all the subelements for a student information system only 44 percent 

have all the subelements for student performance data to measure academic growth. 

Each element score in Exhibit 2-2 (right-hand column) represents the median number of 

subelements present across districts. The number in parentheses represents the percentage of 

districts having at least the median number of subelements in place. So, for example, when it 

comes to linking data, 64 percent of districts have the capacity to conduct two out of the five 

subelement linkages that make up this element (primarily student performance linked to AYP 

subgroups and linked to specific teachers). Seventy-four percent of districts report having one 

out of the two subelements under teacher-level data (primarily teacher qualifications). 
Exhibit 2-2. Profile of District Data System Elements in 2007–08 

Data System Element

Percent of 

Districts 

With 

Subelement 

Percent of 

Districts 

With All 

Subelements 

Median No. of 

Subelements 

Across All 

Districts (Percent 

of Districts With at 

Least Median No.) 

1. Student information system (3 subelements) 65 3 

(65) 

Have system (Q4a) 100 

Ability to generate standard accountability reports or 

district report card and school report cards (Q8a) 66 

Transaction capture (Q8b) 92 

2. Data linkages (5 subelements) 15 2 

(64) 

Linking school performance and finance data (Q7g) 24 

Student performance linked to teacher information or 

characteristics (Q7c) 36 

Student performance linked to AYP subgroups (Q7a) 65 

Student performance linked to specific teachers (Q7b) 64 

Student performance linked to specific instructional 

programs (Q7d) 40 

3. Instructional/curriculum management system (2 

subelements)

47 1 

(79) 

Have system (Q4c) 64 

Links to curricular resources (Q8f) 62 

4. Assessment system (2 subelements) 43 1 

(81) 

Have system (Q4d) 79 

Assessments available in reading, mathematics, or 

other core subject areas that students take online 

(Q8d) 

46 

5. Student performance data to measure academic 

growth (4 subelements)

44 3 

(71) 

Student test scores on statewide assessments (Q5a) 93 

Student test scores on district-administered 

assessments (Q5b) 72 

Drill-down capability (Q8c) 72 

Individual student assessment performance over time 

(Q7e) 70 

6. Student-level enrollment, demographic and 

program participation information (8 subelements)

59 8 

(59) 

Student grades (Q5e) 95 

Student course enrollment histories (Q5f) 92 

Prior school(s) attended within the district (Q5h) 86 

Student demographics (Q5g) 98 

Student attendance (Q5k) 98 

Student behavior (Q5l) 87 

Student special education information (Q5i) 84 

Individual student history over time (Q7f) 81 

Exhibit 2-2 continues on next page 

11 
Exhibit 2-2. Profile of District Data System Elements in 2007–08 (continued) 

Data System Element

Percent of 

Districts By 

Subelement 

Percent of 

Districts 

With All 

Sublements 

Median No. of 

Subelements 

Across All Districts 

(Percent of 

Districts With at 

Least Median No.) 

7. Teacher-level data (2 subelements) 47 1 

(74) 

Teacher qualifications (Q5p) 73 

Teacher professional development (Q5q) 47 

8. Student-level graduation, post-graduation and 

dropout data (3 subelements)

38 2 

(86) 

Differential codes for students no longer enrolled 

(Q5m) 93 

Student graduation status (Q5n) 90 

Student status after graduation (Q5o) 34 

9. College readiness (1 subelement) 57 1 

(57) 

Student test scores on SAT, ACT, and Advanced 

Placement tests (Q5d) 57 

10. Assessment of data quality (3 subelements) 46 2 

(83) 

District/state has disseminated data collection 

guidelines & recommended data information 

management and security practices to schools (Q12) 

71 

District has staff or outside source responsible for 

receiving & preparing files from outside sources to 

load into the student data system (Q13) 

89 

Greater than 90% of data captured by the district’s 

student data system(s) that drive instructional 

improvement are accurate (Q14) 

65 

Exhibit reads: In 2007–08, 65 percent of districts with electronic data systems reported having all three of 

the subelements for student information systems (whereas 100 percent reported having the first 

subelement). The median number of subelements present is three. 

Source: 2007–08 district survey questions 4, 5, 7, 8, 12-14. 

The profile suggests that district access to robust data systems is still limited. Such 

limitations may impede district achievement of three of the education goals outlined in the 

American Recovery and Reinvestment Act (ARRA) of 2009: (a) establishing pre-K to college and 

career data systems that track student progress, (b) providing and assessing effective 

interventions for the lowest-performing schools, and (c) assessing teacher effectiveness and the 

equitable distribution of qualified teachers for all students (particularly students who are most in 

need). 

As was true in prior years, districts are maintaining more administrative data than data 

that are targeted to individual student performance (U.S. Department of Education 2009). 

Fifty-nine percent of districts have all eight of the subelements under student-level enrollment, 

demographic and program participation information (element 6), and 65 percent have all three 

subelements associated with a student information system (element 1), whereas only 44 percent 

have all four subelements for student performance data to measure academic growth (element 5). 

12 
An even larger impediment appears to be districts’ inability to link data across systems 

(element 2), with just 15 percent of districts capable of all of the linkages described by the 

subelements. The fact that only 15 percent of districts report having all five subelements 

related to data linkages points to the challenges in achieving the efficiencies that technology 

was envisioned to offer. Lack of interoperability between systems complicates data analysis, 

and differences in system interfaces increase training requirements for district and school staff. 

As discussed later in this chapter, districts are using multiple data systems and over 60 percent 

reported that lack of interoperability across their data systems was a current barrier to expanded 

use of data driven decision making. 

Data quality is of particular concern to policymakers given their focus on accountability, but 

the issue of data quality is part of a broader context. The district data system profile indicates that 

the average district carries out two of the three activities associated with assessing data quality 

(element 10). About two-thirds of districts (65 percent) reported that 90 percent or greater of the 

data captured by their student data system or other system that drives instructional improvement 

is accurate. Case study and survey respondents indicated that data with the biggest accuracy 

problems tend to be items such as student demographics that rely on self-report or are provided 

by parents or else data that often changes frequently (e.g., student schedules, students receiving 

tutoring outside the school day). The case studies suggested that the concept of data accuracy is 

subject to interpretation and is more nuanced than the statistic based on survey responses may 

suggest. Data for accountability purposes (e.g., state test scores) have consequences attached, 

and therefore accuracy is critical. Data used for ongoing adjustments to instructional practice 

tend to be generated more frequently, with any one data entry having less serious consequences, 

justifying a lower level of effort for quality control. 

The remainder of this chapter provides a more indepth look at the components that make up 

district data systems, where these components come from, the types of data maintained in district 

data systems, the tools available to carry out more sophisticated analyses, and how districts use 

their data systems. 

District Data System Features 

As discussed in the introduction, the standards movement and state and federal accountability 

requirements have placed a greater emphasis on using data to monitor progress. Advances in 

technology have made it possible to link multiple datasets, to track change over time, and engage 

in much more sophisticated analysis activities. But there is often a gap between what is 

technically possible and what districts are equipped to do. One of the study’s goals was to 

ascertain the extent to which districts have access to electronic data systems with the capacity to 

realize these benefits. To this end, district survey respondents were asked to report on the 

elements of their current student data system. The application of sampling weights to the district 

survey data produced a nationally representative portrait of the educational data systems used by 

districts in school year 2007–08. 

Very few districts responding to the survey (10 out of 427) indicated that they did not 

have any kind of electronic student data system or tools to enhance educational decision 

making. The ten districts without electronic data systems tend to be smaller than other districts 

13 
in the survey sample. They were similar to the survey sample in terms of the percentage of 

students in poverty and the percentage of their schools that are Title I or not making AYP.7

The remaining 417 districts all indicated that they had an electronic student information 

system. More than three-quarters of these districts indicated that they had an online assessment 

system, and 77 percent reported having a data warehouse. A somewhat smaller proportion, 64 

percent, reported having an instructional or curriculum management system. Analysis of whether 

district size or the proportion of a district’s students living in poverty affected the likelihood of 

having a particular type of student data system found a statistically significant relationship 

between district size and the likelihood of having an online assessment system.8 While 74 

percent of small districts reported having an online assessment system, 91 percent of large 

districts reported having such a system.9

 There was no statistically significant association 

between district poverty level and the type of student data system a district employs. 

Across districts, about half (48 percent) reported having one to three electronic data systems 

and another third (36 percent) reported an average of four systems. Large districts reported a 

significantly greater mean number of data systems (4.1) when compared to smaller districts 

(3.5).10 

Exhibit 2-3 shows the length of time for which districts reported having had various types of 

electronic student data systems. For Exhibit 2-3 and the remainder of this report, the percentages 

reported will be for districts indicating that they have an electronic student data system unless 

otherwise specified. The data suggest that student information systems are not only nearly 

universal but have been around for six or more years in over two-thirds of districts. None of the 

other system types is this widespread or has this long a history. 

 

7

 These 10 districts varied significantly from those with data systems in terms of district size. There were no large 

districts represented among the 10, whereas one-third of districts with data systems were large. There were no 

significant differences by percentage of students in poverty, in the percentage of their schools that are Title I (58 

percent compared to 62 percent) or the percentage of their schools not making AYP (15 percent compared to 26 

percent for districts with a data system). 8

 In this report the word “significant” should be interpreted as “statistically significant.” The term “statistically 

significant” means that the observed difference in a pair of values is not likely due to random variation in the 

sample. The annotations p < .05, p < .01, etc., indicate the probability that these differences are due to random 

chance (i.e., less than a 5 percent probability, a 1 percent probability, etc.). When differences are not marked with 

an asterisk, we cannot reliably rule out the possibility that they are due to random variation in the sample. This 

report uses the standard convention of *p < .05, **p < .01 and ***p < .001. 9

 Unless otherwise stated, when testing to see whether a key measure varies with a categorical variable such as 

district size we employ two common tests, both adjusted to account for the stratified cluster sampling method used 

to collect the survey data. When the measure is continuous, we test the equivalence of the mean value over the 

levels of the categorical variable. When the measure is categorical (such as whether a district utilizes a particular 

type of data system or not), we conduct an adjusted test of independence between the two variables. If this first 

omnibus is significant, we report that the measure varies by the categorical variable. When the categorical variable 

reflects more than two levels (as in district size, with small, medium, and large levels), we then examine pair-wise 

contrasts across all category levels and report on statistically significant contrasts. For this comparison F(1,411) = 

9.86, p < .01. All statistical tests were adjusted to account for the stratified cluster sampling method used to collect 

the survey data. 10 F(1,411) = 12.09, p < .001. 

14 
Exhibit 2-3. District History With Student Data Systems, by Type 

System Type 

Not Applicable 

(Do Not Have 

This Type of 

System) 

Number of Years Have Had This Type of Data 

System 

Less Than 

1 Year 

1 to 2 

Years 

3 to 5 

Years 

6 or More 

Years 

Student information system 0 3 6 21 70 

Assessment system 21 10 15 38 16 

Data warehouse 23 9 9 21 39 

Instructional/curriculum 

management system 36 10 13 24 17 

Exhibit reads: Among districts with a student data system, all reported having a student information 

system in 2007–08; 3 percent have had this type of system for less than a year, 

6 percent for one to two years, 21 percent for three to five years, and 70 percent for six or more years. 

Source: 2007–08 district survey question 4. 

The survey data also suggest that districts are in the process of building their 

technology capacity. Districts are more likely to have electronic access to data such as student 

demographics, attendance, grades and test scores than they are to have the ability to combine 

data from different types of systems or to be able to link assessments and instructional resources 

to achievement data. As illustrated in Exhibit 2-2, only 15 percent of districts have systems that 

can perform all five data linkages. Sixty-four percent of districts report that one of the barriers to 

spreading data-driven decision-making practices throughout their district is information located 

in multiple databases and not linked (barriers will be discussed further in the next chapter). An 

administrator in one of the case study districts stated: “We have a lot of data. One of the 

problems is that we have a lot of data that is in too many places. And that’s been a complaint that 

you have to go to [system] X, Y, Z [to get data].” 

The case study districts, identified for their high data use, maintained three to seven data 

systems each. Districts had multiple systems, even of the same type, because of ongoing 

technology demands that challenged system capacity or systems that did not perform as 

expected, requiring districts to add or refine data system components. About half of the case 

study districts were in the process of adding new system components that they did not have 

previously. For example, three districts had just added a new online assessment system and 

another had added a Web-based system that makes student grades and schedules accessible to 

teachers and parents. Eight of the districts were upgrading their current system components (e.g., 

acquiring the next generation of their current student information system) or enhancing the 

features of current system components (e.g., new reporting features or data elements). One of 

these districts was acquiring a second data warehouse because the current data warehouse had 

limited reporting features that could not be expanded. 

The case study districts also illustrate that data systems that save teachers’ time and 

provide them with information that they can act on immediately are uncommon. Districts 

have spent years trying to develop or acquire systems that support instructional decisions and not 

just accountability and enhanced data access. The stories of two large districts provide examples 

of the challenges faced in effectively integrating multiple systems to meet data needs (see 

Exhibit 2-4). 

15 
Exhibit 2-4. District Implementation of Student Data Systems 

One large, suburban district has spent over a decade trying to make data available to its schools. The 

district’s development of a data system to support these efforts has been an iterative process that has 

included ongoing development of new data systems to replace outdated ones and the incorporation of 

new technology as it is developed. Their current student information system (SIS) was developed locally 

over seven years ago. When the system was first implemented, information from the data warehouse 

was exported into the SIS to make it more accessible to teachers. Unfortunately, teachers did not use 

the system because it did not meet their needs and was not user friendly. The same result occurred with 

a commercial assessment system. According to the executive director of information systems and 

support: “Technically you can make most anything work. [We had a] good vision of where we wanted to 

go, but when it came down to sitting down with the individuals [vendors], it was difficult to get an end 

product. ... It took from 1999–2000 to 2008 to get a tool that was teacher-friendly. That was our objective 

up front, but now it has finally evolved into a true teacher tool. The district has always had lots of data, 

but there was a challenge turning data into information.” From these experiences district staff learned 

that they needed to involve subject-matter experts to develop data system solutions. 

Through an ongoing process of soliciting feedback from users, this first district has begun to replace the 

current SIS with a new system that utilizes commercial software to make data more intuitive and teacherfriendly (teachers have drill down capabilities for their own classes and individual students). The district 

maintains two commercial data warehouse systems—one is a legacy system used for state reporting 

and keeping records of students with special needs. Within the last two years, the district has acquired 

an assessment system that was implemented districtwide during the 2007–08 school year. This system 

contains district benchmark data that is made available to teachers five times a year within 24 hours after 

test administration. The assessment system also contains links to district standards, pacing guides, and 

an item bank aligned with state tests. In 2006–07 the district implemented a locally developed Webbased portal so that teachers can access data anywhere any time; the portal also helps to support the 

interoperability of the districts’ various data systems. Over the next few years, the district will continue to 

manage the transfer of student data to its new SIS, try to improve the linkages between the assessment 

system and the curriculum management system, and link electronic teacher gradebooks with the SIS. 

The district will also try to find a way of storing portfolio-based information that current data systems do 

not support. 

One of the strengths of the second large district has been its capacity to build its own data systems 

customized to the needs of district and school staff. The district’s data warehouse was locally developed. 

Using this system, district and school staff can generate standard accountability reports or district and 

school report cards, and record daily class attendance and disciplinary actions. The data system is the 

primary tool used by schools for drafting and revising school improvement plans. Their data warehouse 

has the capability to link student performance data to student subgroups so that school and district staff 

are able to disaggregate data to perform different data queries, and student performance data can be 

linked to specific teachers, teacher characteristics, and specific instructional programs so that staff can 

examine student performance in different classrooms or programs. When the district found that their offthe-shelf assessment system did not fully meet their needs, they decided to develop their own. District 

staff are working in conjunction with a commercial firm to design an assessment system that includes an 

item bank (items that are aligned with state standards and the district pacing guides) and provides links 

to instructional resources. The assessment system will be linked to the data warehouse to support 

aggregating data and longitudinal analysis, and a new interface will highlight for teachers and principals 

where potential problem areas might be and let them drill down to access the relevant information. This 

district is also in search of a data system that is capable of storing student portfolios and other 

nontraditional forms of data.

16 
System Sources 

Information in the first round of case study data collection (U.S. Department of Education 

2009) indicated that most of the data available to educators come from local data systems. Given 

the considerable effort that is being put into building data systems at the state level (e.g., $265 

million investment in state longitudinal data systems between 2005 and 2009 to improve the 

management and use of education data),11 the district survey gathered information on the source 

of data systems that districts use to guide their data-driven decision making. Districts report 

that they rely primarily on data systems they purchase or develop themselves for the 

purposes of instructional improvement. Recent federal resource investments made in 

improving the quality of state data systems are showing promise in increasing the capacity of 

these systems,12 but two years into the grant cycle, they did not as yet appear to have influenced 

data use at the local level. 

In 2008, the Data Quality Campaign (DQC), an organization focused on improving state 

longitudinal data systems, partnered with APQC Education to explore how states can support 

districts and data-driven decision making at all levels. Their survey of districts suggested that 

before the potential from investments in state data systems are realized, “[T]he cultural and 

technical differences that exist between state and district data systems must be addressed” (Data 

Quality Campaign 2009a, pg. 6).13 That is, there is currently a misalignment between the types 

of data that districts feel are key for improving student achievement and the types of data that are 

being requested by the state for accountability purposes. Districts need student-level information 

to inform instruction and what is sent to the state is aggregated student achievement data, 

attendance, and student counts. When the state sends achievement results to the districts, the time 

lag in receiving these results means that teachers are not able to use the information to inform 

instruction (responses similar to those of our case study districts). Half of the districts in the 

DQC/APQC benchmarking study reported that they have minimal communication or 

 

11 The $265 million investment refers to the total cost of the grants to date, including continuations to be paid out of 

FY 2010 and FY 2011 funds. The total appropriations include $24.8 million in FY 2005, $24.552 million in 

FY 2006, $24.552 million in FY 2007, $48.293 million in FY 2008, and $65 million in FY 2009, plus $250 

million in 2009 ARRA funds. The long-term goal of the State Longitudinal Data Systems Grant Program 

sponsored by the Institute of Education Sciences (IES) is to enable all states to create comprehensive P–20 

systems that permit the generation and use of accurate data to support analysis and informed decision making at 

all levels of the education system, increase the efficiency with which data may be analyzed to support continuous 

improvement, facilitate research, support education accountability systems, and simplify reporting. Since 2005, 41 

states and the District of Columbia have received these multi-year, competitive grants (see the IES Web site at 

http://nces.ed.gov/programs/slds/). 12 States have made progress in developing longitudinal data systems according to the 2008 survey by the DQC (see 

http://www.DataQualityCampaign.org/survey_results/). In 2005, no state reported having all 10 essential elements 

of a state longitudinal data system and in 2008 only six states had them (47 states plan to have eight or more of the 

10 elements in place within three years). In 2008, 48 states have five or more of the 10 elements. At the same 

time, many states currently lack elements that would allow them to conduct analysis that could facilitate 

instructional decision making. For example, only 21 states have a teacher identifier system that provides them 

with the ability to match teachers to students and only 17 states collect student-level course completion and 

transcript information. States report that it is not a lack of technological know-how that is keeping them from 

building a more robust data system but rather lack of political will and resources to implement the elements and 

change the culture around data use (Data Quality Campaign 2008). 13 The DQC/APQC benchmarking project surveyed 57 districts in 2008 (five of the districts in their survey sample 

overlap with the current case study sample). 

17 
collaboration with their state education agencies (SEAs) on state technology planning, district 

technology planning or data training opportunities. Conversely, when states do provide training 

or conferences on technology and data issues, districts cite this type of collaboration as valuable. 

Responses to our district survey administered in 2007–08 indicate that most (88 percent) of 

the student information systems used by districts were commercially developed. There are some 

well-publicized cases of large districts developing their own data systems (e.g., Broward County, 

Fairfax County, San Diego, Tucson), but this choice is not representative of districts as a whole. 

Among the other types of data systems used by districts to shape instruction, commercial 

systems are also dominant but use of state systems is more common for these other systems than 

it is for student information systems. Of those districts with a data warehouse, a majority 

(56 percent) of data warehouses were obtained from commercial sources, but almost a fourth 

(23 percent) of districts report using a data warehouse supplied by their state. Among districts 

using an instructional or curriculum management system, 42 percent report obtaining it from a 

commercial vendor, 18 percent developed it locally, and 14 percent obtained it from their state. 

The pattern was similar for online assessment systems (47 percent from commercial vendors, 17 

percent state developed, 12 percent locally developed). Analysis of the specific systems named 

by survey respondents who said they had a system from their state found that many of these 

systems are commercially developed (e.g., SASI, Edusoft, Cognos) and as such are better 

thought of as “state supplied” rather than “state developed” systems. (A list of the most 

frequently identified systems is provided in Appendix C, Exhibit C-2.) 

Analyses were run to explore the relationship between district size and poverty level and the 

source of a district’s student data systems. High-poverty districts were more likely than other 

districts to be using a student information system designed locally or by their state.14 There was 

no relationship between district poverty level and the source of other kinds of student data 

systems, and there was no relationship between district size and the source of any of the four 

types of data systems. 

Among the case study districts, commercial systems predominated, but there were six 

districts (primarily large) that developed their own systems that were sometimes used in 

conjunction with commercial systems. Half of the case study districts also used state data 

systems or systems made available to them by their state. For example, through Reading First 

funding, one state is contracting with a commercial software company to provide teachers with 

immediate diagnostic results—data are entered by the teachers and reports are immediately 

generated that indicate student mastery by each skill area. One of the districts noted strong 

support from their state in making data available to teachers: “The state department of education 

is forward thinking. They take data we give them and they are creating tools for teachers. [They 

are thinking] how can we more frequently give them [teachers] data?” A medium-sized district 

indicated that they have been part of a cooperative zone of seven neighboring districts created by 

the state to learn how to use data generally as well as data from the statewide electronic 

information system. The regional cooperatives contribute to the development of data tools as 

well as professional development for district and school staff. Conversely, another medium-sized 

district commented that it did not have the resources to develop a data system to meet its needs 

 

14 Twenty-four percent of high-poverty districts used a student information system designed locally or by the state, 

whereas only 13 percent of low-poverty districts used such a system. F(1,411)=4.32, p < .05. 

18 
(it is struggling with commercial systems that have not performed as hoped) and wished that the 

state were able to provide more technology resources. Currently, the SEA is piloting a statewide 

system, but this district did not participate in the competitive grant competition held by the SEA 

because it had already invested heavily in its own data system. 

Types of Information Maintained Electronically 

Data is an essential element in any education reform or inquiry process (Datnow, Park and 

Wholstetter 2007; Wayman and Cho 2009). Some researchers have likened data use to a road trip 

(e.g., Love 2002): without data it is like driving a car with no gauges or windows and without a 

map—you cannot tell how much gas you have, how fast you are traveling, where you are, and 

whether you are even headed in the right direction. To get a finer-grained picture of the kinds of 

data that districts maintain and that are available to support analysis activities, survey 

respondents were asked about the availability of specific types of information. The results are 

shown in Exhibit 2-5. 

The types of data maintained in district data systems did not change between 2006 and 

2008. Over 90 percent of districts reported having electronic data on (a) student demographics 

and student attendance (both 98 percent of districts), (b) student grades (95 percent), (c) student 

test scores on statewide assessments and differential codes for students no longer enrolled (both 

93 percent), and (d) student course enrollment histories (92 percent). These percentages are 

similar to those reported on the 2006–07 NETTS district survey (U.S. Department of Education 

2008) and are consistent with districts’ near-universal reports that they have student information 

systems. 15

Fewer districts maintain electronic records of district-administered assessments (72 percent) 

or of school-administered assessments (52 percent) which could provide more detailed 

information on student performance relative to content standards during the school year while 

there is still time to take corrective action where needed. Less than half of districts (47 percent) 

report keeping electronic data on the professional development taken by their teachers. Districts 

do maintain electronic records of students’ graduation status (90 percent), but fewer districts 

keep electronic data on other measures of their system’s output. Just over half (57 percent) keep 

electronic data on students’ scores on college entrance or Advanced Placement examinations, 

and only about a third (34 percent) have electronic records of their students’ status after 

graduation (e.g., whether attending college, working). 

 

15 The NETTS district surveys indicated that the types of data most frequently stored electronically were 

attendance, student demographics, special education information, student grades, and course enrollment histories. 

The 2007 survey conducted as part of the study reported here showed that the most frequent types of data 

maintained in electronic form over time (the same format for three years or longer) were student demographics 

(70 percent), course enrollment histories (69 percent), attendance and special education information (both 

68 percent), and student grades (67 percent). See Teachers’ Use of Student Data Systems to Improve Instruction: 

2005 to 2007 (U.S. Department of Education 2008) available at http:/www.ed.gov/rschstat/eval/tech/teachersdata-use-2005-2007/teachers-data-use-2005-2007/-intro.html. 

19 
Exhibit 2-5. Types of Information That Districts Maintained Electronically in 2007–08 

Type of Information 

Percent 

Having Data 

Available 

Electronically

Percent With Longitudinal Data 

Stored for 3 Years or More 

Districts With 

This Data Type All Districts 

Student demographics (e.g., campus of 

enrollment, grade level, gender, English 

language learner—ELL status, economically 

disadvantaged status, migrant status) 

98 90 88 

Student attendance (e.g., daily attendance, 

tardies) 98 87 85 

Student grades (i.e., end of course, quarter or 

semester grades) 95 86 81 

Student test scores on statewide 

assessments 93 85 79 

Student course enrollment histories (e.g., 

course completion information) 92 88 81 

Student graduation status (i.e., whether or 

not each student graduated) 90 88 79 

Student behavior data (e.g., counselor 

reports, referrals, discipline) 87 73 64 

Student participation in educational programs 

(e.g., Title I, gifted and talented, special 

education, after school learning programs) 

86 78 67 

Student special education information (e.g., 

diagnostic data) 84 79 66 

Teacher qualifications (e.g., certification, 

education) 73 75 55 

Student test scores on district-administered 

assessments (e.g., benchmark, diagnostic) 72 57 41 

Student test scores on school-administered 

assessments (e.g., end of unit test) 52 43 22 

Teacher professional development (e.g., 

workshops attended, courses taken) 47 50 23 

Exhibit reads: In 2007–08, 98 percent of districts reported that they stored student demographic data 

in electronic form and 90 percent had stored this type of data in the same format for three years or 

longer (representing 88 percent of all districts, with or without an electronic data system). 

Source: 2007–08 district survey question 5. 

Access to longitudinal data has also remained constant over time. To examine trends or 

to investigate the effectiveness of new programs and policies, districts need to be able to track a 

consistent set of measures over multiple years. District survey respondents were asked for each 

type of data that they store electronically, whether they had three years or more of longitudinal 

data stored in the same format. Their responses are shown in the right-hand columns of 

Exhibit 2-5. The pattern of responses overall suggests that districts have longitudinal information 

for the kinds of data most commonly found in student information systems (e.g., student 

20 
demographics, course enrollment, attendance, grades) but that other types of information (e.g., 

scores on district-administered assessments, teacher professional development, student status 

after graduation), even if available for the current student cohort, were typically not available in 

district systems for three years or more at the time of the survey in 2007–08. 

System Capabilities and Tools 

The utility of a student data system depends not just on the types of information it contains 

but also on the system’s capability to support data requests or “queries” related to issues of 

practical importance for education decision makers. The ability to query different datasets allows 

teachers and administrators to go beyond standardized reports that have a set format and cannot 

be manipulated or altered. District survey respondents were asked to indicate their systems’ 

capability to support key query types. The results are shown in Exhibit 2-6. 

While districts have the capacity to conduct some types of inquiry, few have electronic 

data systems that allow them to link outcomes to processes as required for continuous 

improvement. Roughly three-quarters (76 percent) of all districts report that their systems have 

drill-down capabilities and 83 percent report that they can generate longitudinal student histories 

of information such as schools attended or grade point average. Two-thirds (66 percent) can 

generate data reports showing student performance by Adequate Yearly Progress (AYP) student 

subgroups. On the other hand, fewer districts have electronic data systems that support inquiry 

into the factors that districts can actually influence to try to raise student achievement. Just 42 

percent of districts can generate data reports showing student performance linked to participation 

in specific instructional programs; just over a third (38 percent) can execute queries concerning 

student performance linked to teacher characteristics; and just over a fourth (27 percent) can 

generate data reports linking school performance to finance data. These limitations were also 

evident regarding other specific features or tools of district electronic systems for generating and 

organizing data to support instructional improvement (e.g., online assessments, links to 

curricular materials). District responses concerning these capabilities are shown in Exhibit 2-7. 

21 
Exhibit 2-6. District Data System Query Capabilities in 2007–08 

Type of Query 

Percent of 

Districts 

With This 

System 

Capability 

Percent of 

Students 

Represented 

by These 

Districts 

Individual student history over time (e.g., cumulative grades)* 83 88 

Drill-down capability (ability to query a school-level finding to efficiently 

examine a subset of data at the grade, classroom, or student level) 76 85 

Individual student assessment performance over time** 72 85 

Student performance linked to specific teachers** 67 78 

Student performance linked to Adequate Yearly Progress subgroups** 66 78 

Student performance linked to specific instructional programs 42 50 

Student performance linked to teacher information or characteristics 38 45 

School performance linked to finance data 27 34 

Exhibit reads: In 2007–08, 83 percent of districts had electronic data systems that had the capacity to 

support queries about individual student histories over time. These districts serve approximately 

88 percent of the nations’ public school students. 

Note: Asterisks indicate statistically significant differences in percent of districts reporting this query 

capability by district size (small, medium, large), *p <. 05 and **p < .01. 

Source: 2007–08 district survey questions 7 and 8. 

Exhibit 2-7. District Data System Features and Tools in 2007–08 

Feature or Tool 

Percent of 

Districts With This 

System Feature 

Percent of 

Students 

Represented by 

These Districts 

Transaction capture (e.g., daily and class attendance) 95 94 

Tools for communicating with parents around an individual 

student’s performance 78 78 

Ability to generate standard accountability reports or 

district or school report cards 70 79 

Links to curriculum resources 67 65 

Assessments available in reading, mathematics or other 

core subject areas that students take online 52 55 

Exhibit reads: In 2007–08, 95 percent of districts had electronic data systems that could capture 

transactional data. These districts serve approximately 94 percent of the nations’ public school 

students. 

Source: 2007–08 district survey question 8. 

22 
23 

Districts’ Use of Data Systems 

In addition to asking district respondents to describe the capabilities of their data systems, the 

survey sought information on the various purposes for which districts use their electronic data 

systems. Exhibit 2-8 displays their responses. The survey reports suggest that districts’ use of 

data systems is closely linked to their efforts to track student achievement for 

accountability purposes. Over 90 percent of districts use their data system at least once a year 

for the purposes of (a) analyzing student achievement by grade-level, districtwide or by school; 

(b) analyzing student achievement data over time; (c) examining achievement gaps among 

student subgroups; (d) tracking individual school performance; and (e) meeting accountability 

reporting requirements. The only comparably widespread use of data systems not involving 

achievement test scores is the tracking of student attendance that is used to generate school 

funding. The picture is quite different when it comes to how school staff use data, which will be 

discussed in Chapter 4. 

Some of the purposes for which districts use system data are primarily once-a-year activities, 

as might be expected. These include activities associated with accountability (such as examining 

achievement gaps among student subgroups, tracking achievement by school, tracking 

graduation rates by school, meeting accountability reporting requirements, tracking teacher 

qualifications); instruction-related activities (such as identifying promising instructional 

programs, examining resource allocations); and other evaluative activities (such as examining 

district or school climate data; evaluating principal performance). District uses of system data 

that are most likely to occur more frequently (from twice a year to monthly or more) include 

monitoring student attendance and informing parents about student progress, and instructionrelated activities such as tracking measures of student progress (e.g., benchmark or diagnostic 

tests), informing teachers about individual student’s instructional needs, and deciding what 

professional development to offer. 
24 

Exhibit 2-8. Districts Using Data Systems for Selected Purposes

0% 20% 40% 60% 80% 

Evaluate principal performance*

Evaluate teacher performance

Examine district or school climate data*

OTHER ACTIVITIES

Target individual teachers for specific training*

Identify promising instructional programs*

Inform resource allocation to improve instruction*

Inform instructional practice*

Inform teachers about individual student's needs

Develop benchmark assessments aligned with curriculum*

Inform professional development offerings for school staff

Inform student placement*

Guide curricular changes/curriculum development*

Track other measures of student progress*

INSTRUCTION-RELATED ACTIVITIES

Track teacher qualifications*

Track student readiness for promotion/graduation

Inform parents about student progress

Track graduation rates by school*

Meet accountability reporting requirements*

Analyze student achievement data over time*

Examine achievement gaps between groups of students*

Monitor student attendance

Analyze student achievement by grade-level cohorts*

Track school performance*

ACCOUNTABILITY ACTIVITIES

Percentage of districts

Exhibit reads: In 2007–08, 96 percent of districts used their electronic data systems at least once a year to track s

Note: Asterisks indicate the proportion of districts carrying out a particular activity vary significantly by district size 

see Appendix C, Exhibit C-6a. Source: 2007–08 district survey questions 18 and 19. 
The case study districts not only used data to meet accountability requirements but also 

engaged in data analysis activities that influenced a wide range of instructional and 

administrative decisions. 

• Inform instruction. District 10 is a large urban district that had gathered both 

quantitative and qualitative data that showed that writing instruction was lacking in their 

schools. After discussions with principals and teachers, the district office recruited a 

cohort of 15 schools to pilot a new writing program. Principals from these schools are 

observing a writing program sponsored by Columbia University. Each principal has 

agreed to train teachers using this approach to see what effect it will have on writing 

scores. As part of the program, a coach from the Columbia program visits the schools so 

that teachers can observe the coach conducting instruction; the coach also serves as a 

“critical friend.” The district has hired a specialist to conduct monthly meetings with 

school-level coaches to help them design professional development for their schools 

around the new writing program. 

• Inform student placement. The executive director for instructional services from a case 

study district that uses data as part of its continuous improvement efforts noted, “[I]n 

looking at middle school data we noticed that African American students weren’t taking 

Algebra in middle school at nearly the rates of their peers. Why not? Turns out the 

criteria for eligibility were teacher recommendations.” The district decided to replace the 

recommendation system with an objective math skills inventory. They gave the test to all 

seventh- and eighth-grade students starting in 2006–07, created a cut point that 

determined math class assignments, and continued to collect teacher recommendations. 

“Turns out that 200 students we assigned to Algebra [using this approach] would not 

have been recommended by teachers. We increased the African American and Latino 

enrollment. Since then, we’ve tracked three semesters of grades for Algebra and 

geometry. What’s really great is we’ve seen an increase in student performance (10 

percent). This finding was consistent across all middle school sites.” 

Another case study district uses its academic data browser to identify students for 

intervention and retention. Prior district research had shown that if students had not 

reached proficiency in reading and math by third grade they would not succeed in school. 

As part of the district’s retention policies, all elementary schools must now assess student 

proficiency levels by a certain point and the electronic data system helps teachers analyze 

data to determine when to administer the district benchmark assessments. 

• Guide program evaluation. Over the past two years, this large case study district has 

been looking at research from other districts and states to inform its transition from halfday to full-day kindergarten. The process began with a few schools as the district 

gathered data on results and conducted ongoing conversations with kindergarten teachers. 

According to the executive director for instructional services: “We look at the data 

they’re collecting, from district assessments, DRA [Diagnostic Reading Assessment], and 

samples of student work. We’ve met with first-grade teachers too. All the while looking 

at the data to see where kids are.” As a result, the district was able to create a full-day 

kindergarten program that met the needs of all students. This approach is representative 

of how the district prioritizes its allocation of resources, as described by the assistant 

superintendent: “When a federal or state initiative comes forward, related to instruction, 

25 
assessment, professional development, what have you, we look at the data. [We] do a 

thorough field test, aggregate groups, subgroups. We can tell if that resource, practice, 

whatever is working for the largest number of children.” 

• Evaluate teachers and principals. As part of its results-based approach to decision 

making, this case study district requires both schools and teachers to develop 

instructional goals and plans to meet those goals. School staff review their goals in 

schoolwide meetings held a few days after each school’s benchmark data are received 

and then discuss the results further in team meetings to make adjustments in instruction 

and professional development plans. Teachers set individual and team goals and are 

evaluated based on their students’ performance on benchmark and state assessments. The 

aim of district leaders is to make this into a process for professional self-reflection and 

growth for both teachers and principals. 

• Inform professional development. The director of assessment in one of the case study 

districts is collecting data on professional development for the 500 plus classes offered to 

certified and classified staff, including union programs for administrative staff, 

transportation staff, and custodians: “[The] purpose is to make sure employees are 

building skills that will help them move in district if they wish, keep them happy, keep 

them here.” 

Variations Across Districts 

District survey data indicate that the availability of electronic student data systems is 

almost universal among districts, but districts are at varying stages of building their 

technology capacity. While there were few differences identified in the composition of district 

data systems (i.e., the type and number of data systems), analyses of survey responses uncovered 

variations in system query capabilities and how districts use their data systems. It appears that 

capacity is positively related to district size, with larger districts having systems that support a 

wider range of analysis activities, and, as observed in the case study districts, these larger 

districts also have greater resources to acquire or develop data systems to meet their needs and to 

employ staff who have the expertise to conduct analysis activities. An administrator in one of 

these large districts noted that they have staff to customize reports for their schools: “We can do 

report writing at the district level [write the program scripts], but we’re an urban district. We 

have enough people where we can hire programmers whereas in a little bitty school district, they 

don’t.” 

In general, larger districts are more likely than small ones to have more robust data 

systems. There were significant differences by district size in the number of query capabilities 

outlined in the survey (as described in Exhibit 2-6), with a greater percentage of large and 

26 
medium districts than small ones reporting having four of the eight query capabilities.16 District 

size was not a predictor of the inclusion of the features or tools listed in Exhibit 2-7.17

In looking across the key system elements profiled in Exhibit 2-2, both medium and large 

districts reported higher numbers of subelements compared with small districts for six of the 10 

data system elements. Medium and large districts have significantly more of the subelements 

associated with system capacity, including those under the following elements:18 

• Data linkages 

• Assessment system 

• Student performance data to measure academic growth 

• Teacher-level data 

• Student-level graduation, post-graduation, and dropout rates 

• College readiness 

• Assessment of data quality 

Finally, there are also some significant differences in districts’ use of data depending on their 

size and the proportion of a district’s schools not making AYP. A smaller percentage of small 

districts than large districts ever perform 12 of the 23 data use functions in Exhibit 2-8 (see 

Appendix C, Exhibit C-6a). Other significant differences were associated with a district’s 

performance with respect to AYP, with those districts with a larger proportion of their schools 

failing to make AYP reporting more data use (see Appendix C, Exhibit C-16). The proportion of 

a district’s schools not making AYP was associated with using data to examine achievement 

gaps between groups of students, to track school performance, to guide curricular changes or 

curriculum development, and to inform instructional practice (i.e., districts that engaged in these 

data use practices had a higher proportion of their schools not making AYP than did districts that 

never engaged in these practices). 

Summary 

The data in this chapter highlight both the progress that districts have achieved in developing 

an information infrastructure to support data-driven decision making and the unmet needs that 

still exist. Many of the education data systems in current use are not able to share information 

due to incompatabilities in their hardware and software (a barrier reported by over 60 percent of 

districts). Districts also lack the human capacity to make use of the information resources 

available to them. Together, these two factors inhibit the ability of districts to conduct 

 

16 Following a finding that the proportion of districts with a specific capability varied by district size, pair-wise 

contrasts across all possible combinations of small, medium, and large districts were examined. For each of the 

four capabilities that varied significantly by district size we found that smaller districts were significantly less 

likely to have this capability, when compared to large and medium districts (p < .05). Data by district size for 

Exhibits 2-6 and 2-7 can be found in Appendix C, Exhibits C-4 and C-5. 17 Similarly, analyses contrasting high- and low-poverty districts found no significant differences in their systems’ 

query capabilities or inclusion of the tools identified in Exhibit 2-6. 18 Differences by subelement are presented in Appendix C, Exhibit C-3 (p < .05 to p < .001). 

27 
28 

meaningful analyses to better support instructional decision making. But the landscape is not 

uniform. Generally speaking, large districts are more likely than small ones to have more robust 

data systems and to conduct a wider range of data analysis activities. 
3. District Supports for Data-driven Decision Making in Schools 

The previous chapter described districts’ electronic data systems and the ways in which 

district staff are using data. This chapter describes districts’ activities to encourage and support 

the use of student data and data systems within schools and the challenges that these efforts 

encounter. Both the survey and case study data suggest that districts understand that addressing 

the “people” element is just as important as hardware and software issues in promoting datadriven decision making. Improving the capacity of schools to provide high-quality instruction 

and supporting their efforts to effectively use data (e.g., through leadership, policies, 

partnerships, professional development, and fostering supportive school cultures) are critical to 

success—having the human capacity to make use of the data provided. Implementation of datadriven decision making also requires strategic thinking so that all parts of the system are working 

in the same direction. Several themes emerged from the analysis of the survey and case study 

data: 

• To spread the use of data and data systems to the classroom level, districts must connect 

their systems, professional development, and supports to instructional needs. 

• Data-driven decision making is an ongoing, overlapping set of processes that take many 

years to come to fruition. 

• Despite the fact that implementation is challenging, there are districts that are addressing 

the implementation challenges and their hard work is starting to pay off in terms of 

teacher adoption of data-driven decision-making practices to support instruction. 

• Strategies used by these districts to implement data-driven decision making include 

integrating data use with school planning and instructional coaching, developing district 

benchmark assessments and making benchmark assessment data and instructional 

resources available on the system, designing data-driven decision-making-related 

professional development, and adopting a continuous improvement process with respect 

to their own activities around data and data systems. 

District Data-driven Decision-making Support Index 

The model of supports for data-driven decision making in schools presented in Chapter 1 

included both a user-friendly student data system and a set of tools, policies, and organizational 

supports that enhance the likelihood that school staff can actually use data to improve 

instruction. Chapter 2 presented a Data System Profile based on responses to the district survey 

regarding the capacity of districts’ data systems to provide data and analysis tools to support 

instructional decision making. A second district profile, presented below, examines the extent to 

which districts are providing the other components in the conceptual model: 

• Leadership for improvement and use of data 

• Tools such as online assessments for generating data 

• Social structures and time set aside for reflecting on data 

29
• Professional development and technical support for using data 

• Tools such as curriculum resources for acting on data 

The district support profile suggests that district efforts to provide supports for datadriven decision making are mixed. Exhibit 3-1 presents the composite Data-driven Decisionmaking Support Index for districts responding to the survey—each element score (right-hand 

column) reflects the median number of subelements present across districts.19 The percentage of 

districts with all of the subelements present and the percentage of districts that has each 

subelement is also provided. While the district responses to individual survey items about their 

support of data use indicate that data-informed decision making has become a district priority, 

only a minority of districts have all of the elements of any of the support dimensions in place 

other than data system access (for which 75 percent of districts have all subelements in place) 

and leadership for improvement and use of data (51 percent of districts have all of these 

subelements in place). At the same time, districts are undertaking a variety of activities to 

increase data use as evidenced by the percentage of districts reporting that they carry out various 

subelements. These data are consistent with responses to survey items concerning the strategies 

that districts have been emphasizing between 2005 and 2007 to promote data-driven decision 

making—see Exhibit 3.2. 

The remainder of this chapter explores the strategies that districts are employing to support 

data-driven decision making in their schools—professional development, technical assistance 

and onsite resources, and district policies in support of data use. 

 

19 The index was created from responses from multiple district survey items combined into a handful of key data 

supports as was done in Exhibit 2-2 for data system elements. 

30
Exhibit 3-1. District Data-driven Decision-making Support Profile

Support Element 

Percent of 

Districts By 

Subelement 

Percent of 

Districts 

With All 

Subelements 

Median No. of 

Subelements 

Across All Districts 

(Percent of 

Districts With At 

Least Median No.) 

1. Data system access (2 subelements) 75 2 

(75) 

Principal or specialists access to all or most data for 

students in their school (Q9) 92 

Teacher access to data on students in their classroom 

(Q10a&b) 81 

2. Leadership for improvement and use of data 

(3 subelements)

51 3 

(51) 

Training school administrators on how to provide 

leadership for data-driven decision-making practices in 

their school (Q21f) 

82 

Requiring all or particular schools to follow specific datadriven decision-making practices in their school 

improvement plans (Q22k) 

69 

Following up to determine if schools have implemented 

instructional changes prescribed as a result of data 

analysis activities (Q22m) 

60 

3. Tools for generating data (3 subelements) 21 2 

(62) 

Have assessment system (Q4d) 79 

Assessments available in reading, mathematics, or 

other core subject areas that students take online (Q8d) 46 

Providing Web-accessible library of diagnostic or 

benchmark assessments (for downloading) linked to 

academic standards (Q22f) 

47 

4. Social structures and time set aside for data use 

(3 subelements)

17 2 

(53) 

Paying for incentives for teachers to use or obtain 

training in data-driven decision making (e.g., paying for 

dedicated time for school staff to review data) (Q22i) 

30 

Requiring “data conferences” between individual 

principals and their supervisors (Q22l) 48 

Implementing policies and requirements to use data or 

providing incentives for data use (Q23c) 81 

Exhibit 3-1 continues on next page 

 

31
Exhibit 3-1. District Data-driven Decision-making Support Profile (continued) 

Support Element 

Percent of 

Districts By 

Subelement 

Percent of 

Districts 

With All 

Subelements 

Median No. of 

Subelements 

Across All Districts 

(Percent of 

Districts With At 

Least Median No.) 

5. Professional development and technical support 

for data interpretation (8 subelements) 26 7 

(54) 

Training school staff on the basic functions of the data 

system (Q21a) 90 

Training principals or other building administrators on 

using the data system to analyze student achievement 

(Q21d) 

91 

Training principals on using data to change 

instructional practice (Q21e) 86 

Training teachers on using the data system to analyze 

student achievement (Q21g) 81 

Training teachers on using data to change instructional 

practice (Q21h) 82 

Making technical experts (in systems, networks, 

databases) available to schools to support system use 

(Q22a) 

80 

Making data analysis experts available to school staff 

such as data coaches (Q22c) 50 

Requiring instructional coaches to explicitly incorporate 

data use and train teachers in data use as part of their 

job (Q22h) 

44 

6. Tools for acting on data (5 subelements) 23 3 

(70) 

Providing models to schools to illustrate how to use 

data in allocating resources and designing school 

improvement activities (Q22b) 

57 

Providing teachers with processes or tools to 

effectively utilize data for instructional purposes (Q22j) 65 

Links to curricular resources (Q8f) 62 

Providing teachers research-based guidance on 

differentiating instruction on the basis of student 

assessment data (Q22g) 

74 

Providing an online database of lesson plans and 

planning resources linked to academic standards and 

assessment results. (Q22e) 

60 

Exhibit reads: In 2007–08, 75 percent of districts reported having both of the subelements for data system 

access (whereas 92 percent of districts had the first subelement and 81 percent had the second 

subelement). At least 75 percent of district reported having the median number of two subelements for 

data system access. 

Note: At least a fifth of district respondents to the district survey were not sure about the level of teacher 

access to the district’s information system. 

Source: 2007–08 district survey questions 4, 8-10, 21-23. 

32
District Strategies for Increasing Use of Data Systems 

Both the survey and case study data suggest that districts are taking steps to improve 

the capacity of their schools to use data in decision making, but the degree to which these 

efforts are part of an integrated approach to data-driven decision making is still unclear. 

Survey responses of the national district sample indicate that a majority of districts (58 percent) 

have put a major emphasis on establishing a process for continuous improvement (e.g., 

developing measurable goals, measuring progress, reflecting, and making refinements) between 

2005 and 2007 (Exhibit 3-2). Less than half (46 percent) report putting a major emphasis on 

designing or acquiring information management systems to support data analysis, and just 43 

percent describe building district staff capacity for data analysis and interpretation as a major 

emphasis. 

Exhibit 3-2. District Strategies to Promote Data-driven Decision Making 

From 2005 to 2007 

District Strategies 

Percent 

No 

Emphasis 

Percent 

Some 

Emphasis 

Percent 

Major 

Emphasis 

Establishing a process for continuous improvement (e.g., 

developing measurable goals, measuring progress, 

reflection, making refinements). 

4 38 58 

Designing professional development and capacity-building 

activities for schools’ data-driven decision making (e.g., 

school-level professional development, modeling data use, 

providing time to discuss data, providing tools to interpret 

and act on data). 

6 41 53 

Establishing a culture of data use at the school level (e.g., 

explicit norms and expectations regarding data use, 

creating a safe climate for data use, mutual accountability 

among staff). 

5 46 50 

How to design/acquire an information management system 

to support data analysis (that generates timely data and 

makes useful data accessible to staff at all levels of the 

system—user friendly system). 

9 45 46 

Building district staff capacity for data analysis and 

interpretation. 10 47 43 

Exhibit reads: Between 2005 and 2007, 96 percent of districts placed at least some emphasis on 

establishing a process for continuous improvement as one of their strategies to promote data-driven 

decision making. 

Source: 2007–08 district survey question 20. 

The case studies provide a better understanding of the systemic nature of developing a 

culture of data use and the time involved in establishing these practices. Exhibit 3-3 describes 

two districts each of which has spent a decade or more creating or identifying data systems that 

will support instructional decision making and adopting organizational practices that encourage 

the use of data as a problem-solving tool (not the norm among districts). Each district has 

provided strong leadership to guide the adoption and integration of data into a process of 

continuous improvement. All levels of the system are held accountable for using data to guide 

33
decision making, but school staff feel that they have a voice in the process given established 

feedback loops and opportunities to review data and to receive training to support data analysis 

activities. These two districts have also made efforts to improve the usability of their data 

systems (since the structures and functions of the technology tools can either facilitate or impede 

their utility to users) and to include the kinds of data that support instructional decisions (e.g., 

interim district benchmark assessments that allow school staff to track student progress on a 

more frequent basis than annual state-level assessments). The issue of systemic change is 

discussed further as part of the description of the challenges posed to district efforts to spread 

data-driven decision-making practices. 

Exhibit 3-3. Systemic Change Guides District Implementation of Data-driven 

Decision-making Practices 

District 12 is located in the suburbs of a large urban area. The district’s administrators are dedicated 

to building a world-class school system with data as the foundation for driving all decisions and have 

spent the past 12 years implementing a Results-Based Educational System (RBES), an organizing 

structure for measuring and monitoring the improvement process at the district, school and teacher 

levels with data as the foundation. The process began when the superintendent noticed that the 

district had tools only for technical staff to access data at the same time that demands for data by 

school staff were increasing. He stated: “I want the information in teachers’ hands and I want it easy 

to access.” The district has invested significant fiscal resources both in purchasing commercially 

available data systems and in creating their own, and in soliciting feedback from users to support 

developing and refining their data systems. 

The RBES process rates schools in large part based on improvement in student performance. At the 

school level, each school is evaluated based on a system of weighted indicators of school 

performance (e.g., student achievement, initiatives to improve, customer satisfaction based on parent 

and staff surveys, school management). District benchmark tests are currently offered in math and 

reading, and these along with state tests and other student data (e.g., Individualized Education Plans 

and English Language Learner data) are heavily used for instructional decision making and to select 

topics for professional development in which students are weakest. Teachers have weekly team 

meetings (same subject area and grade level) and cross-team meetings every two weeks in which 

discussions of data play a key role. Teachers are able to analyze data by student groups and to drill 

down in the data to the individual student level. While the district has invested a great deal of 

resources in making their data systems more intuitive and providing each school with a technology 

coordinator who assists staff with both technical (hardware) issues and the use of the data systems, 

the trade off has been less resources spent on teacher professional development on use of data. 

Despite the district’s ongoing struggle to meet performance standards in the midst of ongoing 

demographic changes, there appeared to be a good deal of respect overall for the consistent, 

continuous improvement approach of the superintendent and a sense that he puts teachers and 

students first in his vision for change. There is a perceived balance between accountability for results, 

school level flexibility, and district supports provided to schools to meet their needs. The stability of 

district staff and their efforts to listen to feedback from teachers and school leaders seems to have 

been an asset in the district’s improvement process. 

Exhibit 3-3 continues on next page 

 

34
Exhibit 3-3. Systemic Change Guides District Implementation of Data-driven 

Decision-making Practices (continued) 

District 10 has developed a culture of data use within the district office and there is an expectation 

that all levels of the system will use data to inform practice. The district operates under a performance 

management system that includes setting performance standards and goals and measuring 

performance towards those goals, a quality improvement process, and a system for reporting on 

progress. Plan, do, study, act (PDSA) is the approach to everything including how students evaluate 

their own work. The district has spent 10 years creating systems within the organization to support 

data-driven decision-making goals including the development of its own electronic data systems to 

make them user-friendly and inclusion of the kinds of data that support instructional decisions, 

professional development on the use of data, and active engagement of a broad range of 

stakeholders in the design and implementation of practices and programs. The assistant 

superintendent for research and evaluation stated: “I think the way I would characterize our data use 

is we look at data as a problem-solving tool rather than an evaluation tool. So we say, ‘We have a 

problem. Where? At the district level, at the school level, at the classroom level or at the department 

level?’ So we all try to understand the pieces and that helps teachers alleviate all these fears about ‘Is 

this going to be used for my evaluation?'” 

The district has an office that offers 22 courses for administrators, specialists and teachers on how to 

use data (e.g., goal setting and action planning, using data in the classroom, creating short-cycle 

assessments to check in on subgoals, engaging students in goal setting). The office was launched 

with a $3 million grant from the Broad Foundation to try and identify critical data and make it userfriendly. The office director stated: “We had tons of data. It was never an issue of not having data. 

...There is a difference in having tons of data and having useful data and being able to look at it and 

knowing what your kids need as a result.” The district moved from focusing on state data to districtlevel and site-level authentic assessments, “Something that tells you where the kids are right now, 

this time and place versus where they were at the end of last year.” The assistant superintendent for 

elementary education stated that the district views continuous improvement as the foundation for 

good instruction: “As a district we are easily critical of ourselves because we are always looking at the 

next way to get better. We’re always looking for what will move our kids to the next step and what 

perspectives can we bring to it. Whether it’s data analysis or teaching teachers about vocabulary 

development or a content area, we’re always just searching for a better way to do our work.”

Training on Data Systems and Use of Data to Improve Instruction 

Hardware and software are the vehicles for delivering data, but they are not sufficient to 

ensure good use of the data they contain: there is a need to train staff to use these information 

resources to make informed decisions. Over 90 percent of districts responding to the survey 

reported that they have provided at least some school staff with training designed to enhance the 

school’s capacity to use data in ways that improve instruction. A series of survey items probed 

districts on the nature of the training they have provided and the proportion of their schools that 

have participated (see Exhibit 3-4; see also Appendix C, Exhibit C-8 for additional data). 

35
Exhibit 3-4. District-provided Training on the Use of Data and Data Systems 

56%

53%

53%

60%

63%

64%

67%

73%

20%

28%

29%

22%

21%

23%

23%

18%

0% 20% 40% 60% 80% 100%

Training school staff on data management and security

Training teachers on using the data system to analyze student 

achievement

Training teachers on using data to change instructional practice

Training principals and other building administrators on how to provide 

leadership for data-driven decision making in their school

Training school staff on data entry to improve data accuracy.

Training principals or other building administrators on using data to 

change instructional practice.

Training school staff on the basic functions of the data system.

Training principals or other building administrators on using the data 

system to analyze student achievement.

Proportion of Schools Involved

Type of Training

All schools Some schools

Exhibit reads: In 2007–08, 73 percent of districts reported that they had provided training to principals 

or other building administrators in all of their schools on using the data system to analyze student 

achievement and 18 percent reported providing this training to some of their schools. 

Source: 2007–08 district survey question 21. 

Although the great majority of districts have offered a wide variety of training (76 to 91 

percent of districts report having given the eight types of training listed on the survey), the 

training has not necessarily been extended to all of the district’s schools. For each training type, 

53 to 73 percent of districts report having given the training to all of their schools. Training for 

principals or other school administrators on how to use data to analyze student 

achievement is most likely to have been given to all schools (in 73 percent of districts), 

whereas training for teachers on how to use the data system to analyze student achievement 

or on how to use data to change their instructional practice are the two types of training 

least likely to have been given to every school (both by 53 percent of districts). The latter 

may be due to the fact that some districts are relying on staff such as data facilitators to help 

teachers access the data system or may rely on a user-friendly data system interface that district 

staff hope will require less professional development, as was seen in some of the case study 

districts. Another reason for limiting professional development activities to a few schools in case 

study districts was a scarcity of professional development resources. This constraint was cited as 

a major barrier by 35 percent of districts nationally (discussed further under challenges to 

implementation). 

Data from the case study districts provide more detailed information on the kinds of training 

provided on the use of data and data systems. 

• Training school staff on the district data system. District 4 has provided a great many 

professional development opportunities for its school staff around the use of their data 

36
systems and provides on-demand training as requested. The district keeps a record of all 

the phone calls it receives requesting technical support and periodically evaluates these 

calls to determine areas of need and update its professional development courses 

accordingly. 

• Training principals and other school leaders. Two of the case study districts have 

identified the principal or vice-principal as the data leader for their schools and have 

supported them in this role through professional development. One of these districts 

offers a wide array of data analysis training courses for principals, and data use is part of 

the district’s principal training program. The program includes a year-long apprenticeship 

during which aspiring principals meet one-on-one with the assistant superintendent and 

must bring with them their data goals focusing on monitoring and evaluating student 

achievement. Current principals meet together every four to six weeks to look at data on 

key results in their schools. Another case study district has designated the vice principal 

as the data coach and vice principals meet with teachers in teams, providing teachers with 

schoolwide data that they do not have direct access to. The vice principal receives 

assistance from a school-based technology coordinator who is trained to use the district 

data systems. The district’s director of student accountability provides school leaders 

with professional development on how to conduct training with their staff on using the 

data systems and using data to guide instruction. (Other case study districts use 

instructional coaches to work with teachers in this way.) 

• Embedding professional development on data use. The use of instructional coaches is 

one of the ways that districts have provided embedded professional development on datainformed decision making. Another approach to this type of training is the requirement 

by one of the case study districts that every school designate a data team to receive 

training from the district. When data teams meet with district staff, they talk about datarelated issues such as reviewing data from a school that is excelling. The data teams are 

then supposed to bring back what they have learned to share with the rest of their staff (a 

process that had yet to be universally adopted when their district was visited). 

• Mandating teacher training. Two of the case study districts indicated that all new 

teachers are required to receive basic training in how to use the district’s data systems 

(being able to access data is the first step). One of these districts also requires secondyear teachers to receive intensive training on good assessment practices. 

• Providing numerous training opportunities. A medium-sized district that is very 

committed to data-driven decision making has carved out a number of opportunities for 

school staff to receive professional development on data use and to conduct ongoing 

discussions around the use of data. The district has used the state requirement that 

teachers get 60 hours of professional development a year as an impetus for teachers to get 

training on using data. During the district’s summer institute, principals are trained on 

using data (e.g., data disaggregation) and on conducting classroom walk-throughs. This 

administrator training has been coordinated with the implementation of Professional 

Learning Communities (PLCs) for teachers. Data from the principal walk-throughs is one 

of many types of data collected by this district. The district also sponsors semi-annual 

“data days” in which school staff and groups of teachers from the same grade level meet 

to discuss performance in individual classrooms as well as across classrooms. Time 

37
during the school day is set aside for the PLCs to meet to help “regularize” discussions 

informed by data. There is an expectation that the PLC meetings are to focus on data: 

“The teachers meet no less than once a week in their PLCs. But they can meet more. On 

that day, they bring data with them to drive instruction,” according to the executive 

director of education. In addition, the district’s director of data and testing provides oneon-one support to teachers and school staff that request assistance. 

• Supplementing district training resources. Several of the case study districts have used 

system vendors and external consultants to supplement their professional development 

training or were members of collaboratives that offered data-related training. For 

example, a small district has received support from its county curriculum supervisor who 

has held training sessions for teachers on how to use assessment data and the state Web 

site that contains both data and instructional resources. The supervisor also spends two 

days a week at the district office providing assistance with curriculum, data analysis (e.g., 

item analysis of state test data, developing easy-to-use spreadsheets to support analysis 

activities) and professional development for administrators. In addition, the state has 

trained regional specialists to support teachers in reviewing the state value-added reports 

in reading and math and has created guiding questions for looking at these data; school 

districts pay a fee for these services. 

While professional development is one of the most common district strategies for 

building school capacity for using data, the quality of this training is uneven. The district 

survey did not examine the quality level of district investments in professional development, but 

data from the case study districts suggest that not all of the training has been of high quality nor 

sufficiently robust to achieve desired results. School staff in one case study district, for example, 

described the training they received from their district on collaboration to use formative 

assessment data as very helpful for learning how to work with data, but data use was not 

supported afterward and the training resulted in just another binder on the shelf. Some of the 

challenges to providing effective professional development are discussed later in this chapter. 

Resources and Assistance for School Use of Data 

Training is not the only kind of support that districts are offering to encourage data-informed 

decision making in their schools. Exhibit 3-5 shows the proportion of districts reporting that they 

provide seven additional kinds of technical assistance and resources. 

In addition to training, one of the most common supports provided by districts is access 

to technical experts in systems, networks or databases who can support school use of 

electronic data systems. Eighty percent of districts say they have provided this technical 

expertise and 65 percent say they have made it available for all of their schools. In contrast, 

making data analysis experts available to school staff (for example, in the form of a data coach) 

is one of the least common supports. Still, 50 percent of districts say they have done this for at 

least some of their schools and 32 percent say that they have done this for all of their schools. 

Other less common types of district-provided resources are a Web-accessible library of 

diagnostic or benchmark assessments linked to academic standards (provided by 47 percent), 

models of how a school can use data to allocate resources and design school improvement 

38
activities (provided by 57 percent), and an online database of lesson plans and planning 

resources (60 percent). (See Appendix C, Exhibit C-9 for additional data.) 

Exhibit 3-5. District-provided Supports for School-level Use of Data 

to Improve Instruction 

34%

32%

43%

41%

51%

50%

65%

13%

18%

15%

19%

23%

26%

16%

0% 20% 40% 60% 80% 100%

Providing a Web-accessible library of diagnostic/benchmark 

assessments linked to academic standards

Making data analysis experts available to school staff such as data 

coaches

Providing models to schools to illustrate how to use data in 

allocating resources and designing school improvement activities

Providing an online database of lesson plans and planning 

resources linked to academic standards and assessment results.

Providing teachers research-based guidance on differentiating 

instruction on the basis of student assessment data.

Using data to identify professional development activities for 

schools identified for improvement.

Making technical experts available to schools to support system 

use.

Proportion of Schools Involved

Type of Support

All schools Some schools

Exhibit reads: In 2007–08, 65 percent of districts reported they had made technical experts available to 

all of their schools to support system use, and 16 percent had provided such technical experts to some 

of their schools. 

Source: 2007–08 district survey question 22. 

Data from the case study districts reinforce district survey data that indicate that districts are 

providing leadership and supports for schools’ use of data. For example, several of the case study 

districts supply school-based staff who provide technical expertise around the operations of the 

district’s electronic data systems. So-called data facilitators or technology coordinators can be 

regular classroom teachers or technology specialists who are provided with additional training by 

the district on their electronic data systems. Their role is generally to assist other school staff 

with both the hardware and the use of data systems (e.g., accessing reports). These roles can be 

critical in making data available to school staff when the district’s electronic data system is not 

user friendly, as discussed in the next chapter. 

Other supports have included providing more timely data linked to diagnostic assessments 

and resources to assist school staff in tailoring instruction to student needs. One case study 

district, for example, was in the process of developing its own curriculum management and 

assessment database consisting of three components: a curriculum repository to be used by 

curriculum specialists to load and maintain standards and benchmarks for state assessments, an 

item bank and district benchmark assessments aligned with state standards, and the district’s 

curriculum pacing guides and instructional resources. Using this system, teachers will be able to 

generate their own assessments, review results that pinpoint potential problem areas, and identify 

39
resources for student remediation. Another case study district has adopted a commercial 

assessment system to provide similar resources to its teachers. 

District Policies in Support of School Use of Data 

District policies and practices can signal expectations around schools’ use of data (see 

Exhibit 3-6). Of the policies and practices included on the district survey, the most common 

are requiring all or some of their schools to follow specific data-driven decision-making 

practices in formulating their school improvement plans (69 percent of districts do so) and 

providing teachers with specific processes for how they should use data for instructional 

purposes (65 percent of districts do so). The least common practices are actually keeping track 

of the extent to which teachers use the data system and paying incentives for teachers to use or 

obtain training in data-driven decision making: only 30 percent of surveyed districts reported 

doing this. Requiring instructional coaches to incorporate data use in their work was also among 

the less common practices reported by districts (44 percent). (See Appendix C, Exhibit C-10 for 

additional data.) 

Exhibit 3-6. District Policies and Practices Encouraging Schools’ Use of Data 

18%

21%

26%

36%

39%

40%

51%

12%

9%

18%

12%

21%

25%

18%

0% 20% 40% 60% 80% 100%

Paying for incentives for teachers to use or obtain training in datadriven decision making.

Tracking teacher use of the data system.

Requiring instructional coaches to explicitly incorporate data use 

and train teachers in data use as part of their job.

Requiring "data conferences" between individual principals and their 

supervisors.

Following up to determine if schools have implemented instructional 

changes prescribed as a result of data analysis activities.

Providing teachers with processes or tools to effectively utilize data 

for instructional purposes.

Requiring all or particular schools to follow specific data-driven 

decision making practices in their school improvement plans.

Proportion of Schools Involved

District Policy or Practice

All schools Some schools

Exhibit reads: In 2007–08, 51 percent of districts reported requiring all of their schools to follow specific 

data-driven decision-making practices in their school improvement plans, and 18 percent reported 

requiring this of some of their schools. 

Source: 2007–08 district survey question 22. 

The survey data provide a general picture of district practices and policies to encourage 

schools’ use of data and the case study districts provided an opportunity to examine these 

practices in greater depth given their focus on data-driven decision making. Two-thirds of the 

40
case study districts provide data analysis support to their schools through instructional and other 

types of school-based coaches (compared to 44 percent of districts overall). These districts have 

also adopted processes and tools to help school staff effectively use data. The leadership role that 

instructional coaches play in facilitating data use at the school level is discussed in greater detail 

in the next chapter. 

One of the case study districts, concerned that there was a disparity across schools in the 

degree to which teachers were accessing and using the district’s data systems to inform 

instruction, turned to the use of professional learning communities (PLCs) as an alternative to 

relying solely on principal support for data use at the school level. In 2007–08 the district 

focused on establishing PLCs to promote teacher collaboration around data use. PLC meetings 

were formalized and integrated within the school day. There were PLC meetings dedicated to 

language arts and math staff development during which reading or math specialists planned and 

led the meetings to discuss student data and instructional strategies. Other PLC meetings were 

led by team members to monitor student improvement, develop common assessments, and share 

best practices. The district holds schools accountable for establishing PLCs by monitoring their 

implementation and rating their quality. Staff at some of the case study schools indicated that 

staff can become too dependent on specialists so this district’s next step is going to be to 

encourage teachers to take more ownership and responsibility as a team for the PLC meetings. In 

addition, the district is keeping track of teachers’ use of the data system. A log is kept to record 

what teachers do with the data system and principals can then talk to those teachers who are not 

using the system to find out why. 

Several case study districts are putting a greater emphasis on using data to guide school-level 

planning. To increase school-level accountability for improving student achievement, one of the 

medium-sized case study districts is placing a greater emphasis on using data to monitor school 

progress against school improvement plan goals—requiring school staff to develop monthly 

action plans using a common template so that staff can make the connections between their plan, 

the data and the process of improvement. District professional development has emphasized that 

school improvement is more than a yearly activity; it is not something that is done at the 

beginning of the year and then filed away in a drawer. The director of school improvement and 

accountability said, “School improvement is actually about improving schools, it is not a 

document. It’s an active process....The whole use of data on a regular basis to drive changes 

within your school to get you where you ultimately want to go.” This district requires 

instructional coaches to focus on the action plans when they work with school staff. This process 

includes reviews of student work to help teachers understand the requirements in the state 

standards and tailor their instruction based on student needs to assist each student to meet the 

standards. Another district adopted a Data Action Cycle process as a way to look at data at both 

the district and school level. Principals are encouraged to use the process as a way to guide the 

thinking of school leadership teams to analyze data for school improvement planning. The Data 

Action Cycle is a five-step data protocol to support a collaborative and reflective dialogue 

around data that focuses on a continuous improvement approach (e.g., where to get the right data 

to address a topic, how to read the data, how to apply what is learned, review results). 

While data can provide opportunities for dialogue, reflection and learning, sometimes there is 

the temptation to use data to justify prior assumptions or jump to conclusions (Honig and Coburn 

41
2005; Knapp et al. 2006). Therefore, it is important to structure opportunities for staff to 

effectively engage in data discussions, and tools for organizing these discussions can be useful. 

As part of one case study district’s efforts to help train school staff to implement improvement 

processes, the district identified a number of tools and planning strategies and compiled them 

into a resource guide. The guide defines each tool and provides an example of how it can be 

used. The process for implementing each tool is outlined and frequently asked questions (FAQs) 

are addressed to help educators problem solve as they begin using the tools. For example, the 

Fishbone diagram provides a picture of various system elements that may contribute to a 

problem or topic. This tool can be used to help identify the root cause of a problem, provide a 

visual representation of the cause and effect relationships that are occurring, and help organize 

ideas during a brainstorming session. 

Challenges to District Efforts to Spread Data-driven Decision Making 

Because the gathering and examination of data is just the starting point in developing a 

culture and system of continuous improvement, it is important to think about the implementation 

of data-driven decision making as a systems change (Mandinach et al. 2006). District and school 

adoption of data-driven decision-making practices requires a focused coordination of elements of 

the larger education system, including such things as policy change, widespread professional 

development planning, and reliable and affordable support mechanisms. In other words, it is 

important to consider how the process will function across the entire district and will be used by 

participants at all levels of the system. To better understand the systems challenges faced by 

districts, survey respondents were asked to rate a range of potential technology, logistical and 

other barriers in terms of the extent to which they were currently inhibiting expanded use of datadriven decision making within their district. Their responses, shown in Exhibit 3-7, suggest that 

a majority of districts report that they have the hardware needed to support a data system. 

In addition, fewer than half report that there are problems with the quality of the data in 

their systems.20 The most commonly cited system and technology barriers were (see 

Appendix C, Exhibit C-11 for additional data): 

• Lack of funding for expanded use of data-informed decision making 

• Difficulty linking disparate databases for purposes of analysis (lack of interoperability) 

• Lack of trained technical staff available for product and service acquisition, installation 

or equipment maintenance 

• Data stored in forms that are difficult to access, manage and interpret. 

All of these issues are seen as barriers by over half of districts. 

According to districts, the greatest logistical barrier to spreading data-driven decisionmaking practices throughout their district is the lack of time for school staff to conduct 

data-driven decision-making activities: 92 percent of districts cited lack of time as a barrier, 

and 51 percent identified it as a “major” barrier. A good deal of literature exists on the 

importance of collaboration and dialogue around decision making, and researchers such as 

 

20 As discussed in Chapter 2, data from the case studies suggest that the concept of data accuracy is not onedimensional—where data have consequences attached (e.g., accountability) accuracy is critical, but data used to 

make ongoing adjustments to instruction do not require as much quality control. 

42
Exhibit 3-7. District Administrators’ Perceptions of Barriers to Increased Use 

of Data Systems 

3%

5%

6%

9%

12%

23%

23%

36%

42%

31%

14%

21%

27%

38%

38%

29%

47%

28%

34%

 to provide adequate safeguards and security 

for the data*

districtwide unique student identification 

numbers that are consistent from year to year

districtwide unique teacher identification 

numbers that are consistent from year to year

inaccurate data in system*

that are not fast or reliable 

enough

sufficient hardware

Out-of-date hardware

are difficult to access, 

manage, and interpret*

Lack of trained technical staff available for product and 

service acquisition, installation, or equipment ...

located in multiple disparate databases 

that make it difficult to link data for analyses*

Lack of funding to expand or improve the data system

4%

Inability

Lack of 

Lack of 

Unreliable or 

Internet connections 

Lack of 

Data stored in forms that 

Information 

4%

9%

13%

10%

15%

16%

20%

29%

35%

51%

33%

30%

50%

56%

49%

53%

69%

56%

46%

41%

0% 20% 40% 60% 80% 100%

Policies that preclude direct access by school staff to data systems or 

idual student-level data

of district leadership support for data-driven decision making

of incentives for data used by district or school staff

communication or sharing of data across departments within the 

district

Lack of a clear vision or strategic plan for data-driven decision making 

Lack of building administrator preparation on how to use data for data-driven 

of technical skills of school staff to access or utilize data systems

Lack of teacher preparation on how to use data for instructional decision 

of resources to train school staff on how to use data to support 

ement

of time for school staff to conduct data-driven decision making activities 

Percentage of Districts

tical and Other Barriers

indiv

Lack 

Lack 

Lack of 

decision making

Lack 

making

Lack 

instructional improv

Lack 

Logis

Major Barrier Minor Barrier

Lack of sufficient hardware 

Internet connections that are not fast or reliable enough 

Unreliable or inaccurate data in system* 

Lack of districtwide unique teacher identification numbers 

Lack of districtwide unique student identification numbers 

Inability to provide adequate safeguards and security for the 

data* 

 Logistical and Other Barriers 

Lack of time for school staff to conduct data-driven decision making 

Lack of resources to train school staff on how to use data 

Lack of teacher preparation on how to use data 

Lack of technical skills of school staff to use data systems 

Lack of building administrator preparation on how to use data 

Lack of a clear vision or strategic plan for data-driven decision 

making 

Lack of sharing of data across departments within the district 

Lack of incentives for data used by district or school staff 

Lack of district leadership support for data-driven decision making 

Out-of-date hardware 

Data stored in forms that are difficult to access, manage, 

interpret* 

Lack of trained technical staff* 

Information located in multiple data bases and not linked* 

Lack of funding to expand or improve the data system 

Data System and Technology Barriers 

Exhibit reads: In 2007–08, 42 percent of districts reported that lack of funding to expand or improve the data 

system was a major barrier to expanding the use of data-driven decision making in their district; 35 percent 

reported it was a minor barrier. 

Note: Asterisks indicate significant differences by district size (p < .05 and p < .01). 

Source: 2007–08 district survey question 25. 

43
Elmore (2004) and Wilhelm (2007) have expressed concern that teachers often are not given the 

time or support to allow such activity. Even among case study districts selected as high data 

users, lack of time frequently emerged as a major barrier to data use that was manifested in 

multiple ways (e.g., lack of time for analysis, to collaborate around data, to receive training). 

Administrators in three of the case study districts indicated that both district and school staff 

are faced with many competing priorities and, as a result, not all of them can be achieved or 

achieved well. One administrator stated: 

I think it just comes to time. It might be that technology, the infrastructure of 

technology is not moving at the speed that it needs to be. It might be that I would 

really be glad to do that, but I’m implementing the new SIS system, the new Web site 

system, the new curriculum, the new RTI model...all the other innovations that we’re 

doing as a district. Priorities are called priorities for a reason. You can’t have five of 

them.... People need time not only to make meaning [from the data] individually but 

to collaborate to make meaning as a group. And I think often times the time is not 

there because we are always doing something...do, do, do instead of stopping and 

thinking. 

A superintendent stated, “Every program that you implement takes time. As a result, you’re 

taking time away from something they [teachers] used to do or adding time to their already long 

day.” Similarly, lack of time for professional development on how to use the district’s data 

system or how to analyze data, particularly for principals, can serve as a barrier to data use at the 

school level. 

District ratings of other organizational and logistic barriers to spreading data-driven 

decision making and ratings of data system barriers suggest that districts perceive the 

organizational issues as a bigger challenge than the data system issues. In addition to time, 

other organizational issues seen as a barrier by a majority of districts include (a) school staff’s 

lack of the technical skills to access or use electronic data systems (89 percent), (b) lack of 

teacher preparation on how to use data for instructional decision making (85 percent), (c) lack of 

resources to train school staff (81 percent), (d) lack of building administrator preparation on how 

to use data to inform decision making (69 percent), (e) lack of communication or information 

sharing across departments (66 percent), and (f) lack of a clear vision or strategic plan for datadriven decision making (65 percent). These data also suggest that despite the widespread district 

provision of professional development related to the use of data to inform instruction discussed 

earlier, districts believe that school staff need further training and support; a conclusion 

supported by teacher survey data (U.S. Department of Education 2009).21

All of the major barriers cited by survey respondents were also in evidence in the case study 

districts and attest to the numerous challenges that districts face as they try to implement data-

 

21 In the NETTS 2007 national teacher survey, teachers expressed a desire for more professional development in 

using data, particularly for training in how to develop diagnostic assessments and how to adjust instruction based 

on diagnostic data. They also indicated that they lacked knowledge in using data systems (e.g., forming data 

queries, finding the information they want) and that the electronic data system available to them was hard to use 

(U.S. Department of Education 2009). 

44
driven decision making. There appear to be some significant differences in perceived barriers by 

district size. (These will be discussed in greater detail later in this chapter.) 

In the case study districts, lack of resources to train school staff on how to use data to support 

instructional improvement took several forms: not having sufficient training staff and funds, 

reductions in school-based staff who provide embedded professional development on data use as 

part of their day-to-day work with teachers on instructional issues, and a limited range of 

training. In a medium-sized district, administrators indicated that there were many meetings in 

which teachers and principals look at data but far fewer opportunities to be trained in how to 

analyze data. They acknowledged that in addition to offering more professional development, 

they also needed to change their approach so that teachers become better data analysts: “[It’s] 

almost sensory overload. We’ve put so much [data] out there. We don’t practice what we preach. 

If a teacher were to say, ‘I’ve covered that curriculum but the children haven’t learned....’ We do 

the same thing. We’ve covered [data and assessment] in workshops, but teachers haven’t 

assimilated it; [they] can’t in a sophisticated, confident way use that information.” 

Related to the challenge of providing adequate professional development to school staff 

is a lack of teacher and principal preparation on how to use data. Respondents in several 

case study districts commented on the lack of fundamental knowledge about assessment among 

their teachers and principals, and school staff’s lack of knowledge concerning how to take 

assessment data and determine appropriate interventions. As one district administrator stated, 

“We’re data rich and resource poor. Rank order report by skill or ability; draw a line to make 

groups. But beyond that, there’s not a great deal [of knowledge]. ...Teachers need to know more 

about how to interpret assessments....Every time I do a workshop [teachers] don’t understand 

assessment very well. Don’t understand how to look at data, how assessments are constructed.” 

For some teachers it’s even more basic—there is a general lack of comfort with using data 

systems and data—that has led to mediated data system access in some districts, a slower 

learning curve for some teachers, a concern about how data will be used, and a general resistance 

to relying on data that is not generated in the classroom. 

As the conceptual framework suggests, leadership for educational improvement and use 

of data is a vital component in promoting data-driven decision making within districts and 

schools. The case study districts demonstrate that operationalizing this type of leadership is 

a complex undertaking. To be successful in spreading the use of data and data systems to the 

classroom level, districts must find ways to connect their data systems, professional 

development, and supports to address instructional needs—to think systemically about how to 

infuse data within the culture of district and school practice. An issue underlying the use of data 

systems is the tradeoff between standardization and customization raised in the interim report 

(U.S. Department of Education 2009). District investments in data systems to store, manipulate 

and report on students are predicated on the assumption of a uniform set of practices across 

schools. But standardized implementation of teaching practices is not the norm, and school-based 

staff often feel that the kinds of information available in the district system do not address the 

questions and decisions most important to them. Accountability requirements have heightened 

this tension. Some of the case study districts have resisted top-down mandates and relied on 

voluntary adoption of data-related practices by staff as they see the value of new technologies to 

support their work. Other districts have been much more directive, balancing requirements with 

45
communication loops that help the district identify problem areas and garner support by 

engaging stakeholders up and down the system, and identifying how to redirect resources and 

refine policies and practices. A few examples from the case study districts illustrate how district 

leadership can influence data-driven decision-making implementation both negatively and 

positively (Exhibit 3-8). 

Exhibit 3-8. District Leadership Support for Data-driven Decision Making 

The impetus for developing district benchmark assessments in one district was to generate a common 

language to talk about data as part of the standards-based system and as a way to link what was going 

on in the classroom with student performance (as opposed to just state testing). This district has 

designated the principal as the data leader for its schools and there is an expectation that data will guide 

conversations between principals and their staff with a goal of continuous improvement. The district has 

created an extensive array of professional development courses around data use, including preservice 

requirements for teachers and principals. This district has a fairly mobile population, so the benchmark 

data are available to teachers no matter where the student is served within the district. One of the district 

administrators who advocates strong district leadership stated: 

I’m a firm believer in autonomy...but I think that there needs to be a good balance between 

centralized and decentralized. And in some things, if it’s not implemented consistently and you’re 

not seeing gains, then I think it needs to be more centralized. I think there needs to be more 

expectation that “show me how you’re using the data to drive instruction to make decisions; to 

move kids forward versus [just] saying they’re doing it.”...As a district we’re big believers that if a 

child attends a school in a certain part of town and receives a certain curriculum, that if that child 

moves then he should absolutely have the same [curriculum]. And then how does a teacher 

know? If you have a group of teachers from a variety of different schools get into one room then 

they can have a conversation about what they’re seeing and how they know....Long gone are 

the days of going into your classroom, closing the door, and teaching. It’s not about that any 

more and rightly so. We all get better I think when we can work with each other and get new 

ideas. 

Administrators in other districts commented on the challenges they face in getting all of their principals to 

provide leadership for data use: “[Data-driven decision making] is about leadership. The district needs to 

give more direction to reluctant principals and schools.” Alternatively, another administrator in the same 

district stated: “They [teachers] could use better progress monitoring assessments, common 

assessments. I’m more worried at the school level [with] principals having the time and capacity to really 

go in there [the data system] and mess with the data. Some don’t have time to learn.” 

Similar concerns about the expertise and willingness of district administrators to prioritize data-driven 

decision making were expressed in other districts. In one large district, the support for schools to improve 

their capacity to use data varies with the area superintendents who have significant latitude to determine 

what to do with their schools. District administrators indicated that the district has two or three area 

superintendents who are really strong, while others are struggling. There is also a sense that there is no 

real strategy or push from the new superintendent to prioritize data-driven decision making with the area 

superintendents. So even though the district has changed its policies to provide teachers with more 

direct access to the district’s data system so that they can take advantage of new district benchmark 

assessment data, the structural limitations of the data system to centrally collect, easily sort, aggregate 

and report data to all school staff, coupled with no clear signals that school staff should use this data, 

makes systematic use of data challenging. 

46
Another aspect of the tension between standardization and customization can reside at the 

district level. Some districts engaged in creating a data-driven culture have focused on issues of 

professional development and support for school staff to use data but have not addressed 

technical integration or data sharing issues across their own departments. As discussed in 

Chapter 2, sometimes in order for districts to meet federal and state reporting requirements they 

have had to maintain separate data systems that are not fully interoperable. In other instances, the 

multitude of stakeholders involved in data-driven decision-making activities can lead to 

incompatible systems if there is a lack of communication or centralized plan. 

District Needs for Examples of Good Practice 

Another perspective on the areas in which districts are struggling is provided by a set of 

survey questions (see Exhibit 3-9) asking districts to indicate how much they need various types 

of examples of good practice. The greatest perceived area of need among districts is for 

models of how to connect student data to instructional practice. The most frequently cited 

needs are for examples of how to perform the following (see Appendix C, Exhibit C-12 for 

additional data): 

• Adapt instructional activities to meet students’ individual needs (cited by 84 percent 

of districts) 

• Examine data to identify which practices work best for which students (83 percent) 

• Develop curriculum-embedded formative assessments (81 percent). 

The area in which districts felt least in need of good examples was communicating with 

parents about student progress (63 percent), a practice that teacher responses on an earlier survey 

suggest is quite common (U.S. Department of Education 2008). 

 

47
Exhibit 3-9. District Perceptions of Needed Examples of Good Practice 

14%

21%

21%

26%

33%

35%

37%

49%

50%

59%

54%

48%

49%

47%

0% 20% 40% 60% 80% 100%

Communicating with parents about student 

progress

Using assessment data to identify gaps in student 

achievement

Structuring the district organization and practices to 

support data-driven decision making

Collaborating and sharing ideas with colleagues 

regarding data inquiry and analysis issues

Developing curriculum-embedded formative 

assessments

Adapting instructional activities to meet students’ 

individual needs

Examining student data to identify which practices 

work best for which students

Percentage of Districts

Examples of Good Practice

Great Need Some Need

Exhibit reads: In 2007–08, 47 percent of districts reported that they had some need for examples of 

good practice regarding the examination of data to identify which practices work best for which 

students, and 37 reported a great need. 

Source: 2007–08 district survey question 24. 

Variations in Perceived Barriers and Needs 

District survey and case study data indicate that the use of educational data to make decisions 

and foster improvement has been increasing, but progress has been uneven. Smaller districts 

have gotten a later start in implementing data-driven decision-making strategies and they tend to 

provide fewer supports. District size has also generated differences in perceived barriers (all 

related to technology) and perceived needs. When analysts compared the number of years that 

districts have been implementing or supporting data-driven decision-making strategies, some 

interesting patterns emerged. 

Smaller districts reported getting a later start in supporting data-driven decisionmaking activities in their schools in terms of providing professional development programs for 

teachers and principals on the use of data to improve instructional practices, providing resources 

such as consultants to support the use of data, and implementing policies and requirements to use 

data or providing incentives for data use. A greater percentage of smaller districts are either 

planning to begin to provide these supports or have been doing so for one to two years, whereas 

most larger districts have been providing these supports for at least three years (see Appendix C, 

Exhibit C-13 for data tables). 

48
Districts of different sizes also vary significantly in terms of some of the perceived 

barriers to implementing data-driven decision-making practices but only for data system 

and technology-related barriers and not for other types of barriers. Exhibit 3-10 shows that 

a significantly greater percentage of small districts than medium and large districts report that 

they face a barrier in the terms of data stored in forms that are difficult to access and manage. On 

the other hand, fewer small districts report facing barriers in terms of unreliable or inaccurate 

data in their systems or providing adequate safeguards and security for system data when 

compared to medium and large districts—perhaps due to the lower volume of data. A greater 

percentage of both small and medium districts than large districts report facing the barrier of 

information stored in multiple disparate databases that are not interoperable. A greater 

percentage of both small and large districts than medium districts report that lack of trained 

technical staff for product and system acquisition and maintenance is a barrier. The latter finding 

is contrary to what was observed in the large case study districts, which tended to have greater 

resources to employ staff to develop and maintain their data systems, although a few were 

beginning to experience cutbacks due to shrinking resources (see Appendix C, Exhibit C-15). 

Districts of different sizes varied significantly in terms of one area of needed model 

practices—using data to identify gaps in student achievement. Three quarters (75 percent) of 

small districts reported having some to a great need in this area compared with about half of 

medium-sized (54 percent) and large districts (53 percent). This difference may reflect 

differences between large and small districts in the frequency with which they carry out this 

activity—a greater percentage of small districts report never examining achievement gaps 

between subgroups (7 percent compared to 5 percent of medium-sized and 1 percent of large 

districts), and their data systems are less likely to have query tools to look at student performance 

by AYP subgroups or over time (see Exhibit 2-5; see Appendix C, Exhibits C-4 and C-14 for 

differences by district size). 

District size also is a factor in the number of supports that districts provide to schools to 

encourage data-driven decision making (the proportion of districts with a particular 

subelement varies significantly by district size).22 In looking across the support index profiled in 

Exhibit 3-1 earlier in this chapter, both medium and large districts reported higher numbers of 

subelements for four of the six elements compared with small districts. Medium and large 

districts had more of the elements associated with (a) leadership for improvement and use of 

data, (b) social structures and time set aside for data use, (c) professional development and 

technical support for data interpretation, and (d) tools for acting on data. 

 

 

22 Differences by subelement are presented in Appendix C, Exhibit C-7, p < .01 to p < .001. 

49
Exhibit 3-10. District Administrators’ Perceptions of Barriers to Increased Use of 

Data Systems, by District Size 

Exhibit reads: In 2007–08, large and medium districts were more likely than small districts to report 

that unreliable or inaccurate data in the system was a barrier. Among small districts, 4 percent 

reported this to be a major barrier compared with 7 percent of medium districts and 6 percent of large 

districts. Eighteen percent of small districts reported that unreliable or inaccurate data in the system 

was a minor barrier compared with 36 percent of medium and 42 percent of large districts. 

Large district

Medium district

Small district

acquisition, installation, or equipment maintenance*

Lack of trained technical staff available for product and service 

Large district

Medium district

Small district

Inability to provide adequate safeguards/security for data**

Large district

Medium district

Small district

it difficult to link data for analysis*

Information located in multiple disparate data bases that make 

Large district

Medium district

Small district

interpret*

Data stored in forms that are difficult to access, manage, 

Large district

Medium district

Small district

Unreliable or inaccurate data in the system**

Percentage of Districts

Perceived Barriers

Major Barrier Minor Barrier

0% 20% 40% 60% 80% 100%

Note: Asterisks indicate extent to which this was a barrier varies significantly by district size (*p < .05 

and ** p < .01). 

Source: 2007–08 district survey question 25. 

Data-driven decision making is clearly a complex process, combining technical, pedagogical 

and social elements, and requiring both top-down and bottom-up participation and support. As 

districts undertake this effort, they would like to have a realistic perspective on the amount of 

time that it is likely to take and the amount of progress that can be expected after one year, three 

years, and so on. Both the case studies and the responses to the district survey provide clues to 

the trajectory that districts can expect. 

Exhibit 3-11 shows the case study districts’ classification in terms of the strength and 

consistency of supports with respect to data-informed decision making arrayed against the 

number of years that districts have been actively engaged in the effort and their size (from largest 

to smallest). For each type of support, a district was given a rating: the support was widely 

available in the district (coded as 2); the support was partially present, for example, instructional 

coaches available in some schools but not all (coded as 1); or the support was not currently 

present in the district (coded as 0). The larger districts (those with student enrollments of 25,800 

50
51

or more) provide a somewhat greater number of supports than the medium-sized or small 

districts—a similar pattern to that found for the 2006–07 case study sample (U.S. Department of 

Education 2009). The mean number of widely present supports in large districts is 5.29 

compared with an average of 4.0 in medium-sized districts. But there are exceptions to the 

general pattern. The length of time that the case study districts have been actively engaged in 

data use did not necessarily translate into the presence of a greater number of supports. 

The perceived barriers to implementing data-driven decision making differ 

significantly based on the number of years a district has been implementing data-driven 

decision making. Using the district survey data, analysts were also able to identify districts that 

have been trying to spread data-informed decision making to their schools for one to two years, 

three to five years, or six years or more and compare their responses as to the perceived 

seriousness of implementation barriers and their need for examples of good practice. A greater 

percentage of districts in their first year or two of providing resources for data-driven decision 

making to schools than of districts that have a longer history of such efforts reported major 

barriers in the areas of (a) lack of teacher preparation on how to use data for instructional 

decision making (29 percent), (b) lack of sufficient hardware (17 percent), (c) lack of a clear 

vision or strategic plan for data-driven decision making (11 percent), (d) lack of building 

administrator preparation on how to use data (11 percent), and (e) lack of district leadership 

support for data-driven decision making (9 percent). The launch of a new endeavor can often run 

into problems with inadequate direction, staff buy-in, staff preparation, and resources. As shown 

in Exhibit 3-12, a smaller proportion of the districts that have been trying to spread data-driven 

decision making to their schools for three to five years perceive these issues as major barriers. 

Teacher preparation and district leadership appear to re-emerge as perceived major barriers for 

districts that have been engaged in spreading data-driven decision making for six years or 

more—perhaps reflecting staff turnover in both schools and district offices. 
 52 Exhibit 3-11. Case Study District Supports for School Use of Student Data District Number 4 12 11 3 10 1 16 9 13 

Years of Data-driven Decision Making Implementation 8 12 7 10 10 10 10 10 7 

District Size L L L L L L L M M 

Supports Strong district leadership for data use 2 2 2 1 2 2 2 2 2 Availability of data disaggregated by student groups and skill levels 2 2 2 2 2 2 2 2 2 Districtwide data-driven decision-making activities 0 0 2 0 1 2 0 0 0 Professional development and support for data system implementation 2 2 1 1 1 2 0 2 1 Provides school-based staff to support data-driven decision making 2 2 2 0 2 2 2 0 1 Gives teachers direct access to extract data from system 2 2 1 1 2 2 0 2 2 LEA benchmark or formative assessment data on the system 2 2 2 2 2 2 2 1 2 System includes links from assessment results to instructional resources 

1 2 1 0 1 0 1 0 1 

Students can take assessments online 2 1 0 0 0 0 0 0 0 

Total number of supports fully 

present 7 7 5 2 5 7 4 4 4 

Exhibit reads: In District 1, strong district leadership for data use was fully present in 2007–08. This large district has been driven decision-making practices for at least 10 years. Notes: District size categories include large (L) with 25,800 or more students, medium (M) with enrollment from 5,444 to 25enrollment from 300 to 5,443. Codes indicate degree to which each support was present in the district: 0 = not present, 1 = 

2 = fully present/widely available. 
Exhibit 3-12. District Perceptions of Major Needs and Barriers, by 

Longevity of Data-driven Decision-making Efforts 

0% 5% 10% 15% 20% 25% 30% 35%

Policiesthat preclude direct access by school staff

to data systems or individual student‐level data

Lack of district leadership support for data‐driven

decision making*

Unreliable or inaccurate data in the system.

Lack of building administrator preparation on

how to use data for data driven decision

making***

Lack of a clear vision or strategic plan for data‐

driven decision making***

Lack of sufficient hardware (servers, computers,

peripheral devices, etc.).*

Lack of teacher preparation on how to use data

for instructional decision*

Structuring the district organization and practices

to support*

Collaborating and sharing ideas with colleagues

regarding data inquiry**

Perception of Needs and Barriers

6 or more yrs 3‐5 yrs 1‐2 yrs

Exhibit reads: Among districts that have been providing resources to schools for one to 

two years to support the use of data, 28 percent reported a great need for examples of 

good practice about collaborating and sharing ideas with colleagues regarding data 

inquiry. For districts that have been providing these resources for three to five years, 11 

percent reported a great need compared with 27 percent of districts that have been 

providing these resources for six or more years. 

Note: Asterisks indicate extent to which perceived need or barrier varies significantly by years 

of data-driven decision-making efforts (*p < .05 and ** p < .01). 

Source: 2007–08 district survey questions 23-25. 

In a related vein, districts in their first year or two of trying to support data-driven 

decision making in their schools are particularly likely to report that they have a need for 

examples of good practice with respect to how to structure the district to support datadriven decision making (28 percent report a “great need”) and how to collaborate and 

 53
share data inquiry ideas with colleagues (28 percent). Perceptions that these are great needs 

are less common among districts that have spent three to five years providing resources to 

promote data-driven decision making (11 percent and 10 percent, respectively) but are fairly high 

among those that have been involved for six years or more (27 percent and 15 percent). 

Finally, analysts explored the relationship between the number of strategies districts have 

employed over the past two years to promote data-driven decision making and the frequency

with which they analyze data. Using the survey data, a scale was constructed representing 

District Capacity Building Strategies by assigning each of the strategies listed for survey 

question 20 a score of 0 (no emphasis), 1 (some emphasis), or 2 (major emphasis). The total 

score for the five items in question 20 was calculated resulting in a number ranging from 0 to 10. 

This scale was in turn used to predict the survey responses on individual items in questions 18 

and 19 about how frequently staff in the district office conduct data analysis activities using a 

logistic regression model.23 The result was that the vast majority of items in questions 18 and 19 

were significantly predicted by the capacity building score based on question 20. In other words, 

districts that are engaging in multiple capacity building strategies appear to be analyzing 

data more frequently.

Strategies for Addressing Implementation Challenges 

The preceding discussions have addressed the many challenges that districts face as they 

have moved toward a greater reliance on data-driven decisions. Despite the multiplicity of 

challenges, a number of districts are finding ways to promote data-driven decision making. In 

particular, the districts included in the case study sample are employing a variety of strategies to 

facilitate the adoption of data-driven decision-making practices at the school level. It has taken 

them many years to develop policies and practices designed around the ongoing examination of 

data aimed at improving outcomes for students and it has not always been a smooth transition. 

These districts continue to struggle with many implementation issues but serve as examples of 

what can be accomplished. 

As mentioned previously, when data systems have not been user friendly, case study districts 

have relied on data mediators (e.g., technology coordinators, instructional coaches). 

Unfortunately, when resource constraints necessitate the loss of these staff, it can leave a school 

or district with little technology support, particularly if training on use of the district’s data 

systems has not been widespread. For these and other reasons, five of the case study districts 

have begun to embrace the use of data dashboards that provide school staff with frequent 

 

 

23 The responses to questions 18 and 19 were recoded such that a 0 indicates never performing the activity, while a 1 

indicates performing the activity at least annually. These recoded responses were regressed on the district capacity 

building score using a weighted logistic regression model with adjustments for the stratified sampling design. A 

statistically significant coefficient for the capacity building score was interpreted to indicate an association 

between the capacity building score and the specific activity. 

 54
updates in an easily digestible format on data that is of most interest to them.24 For one of 

these districts, the move toward the creation of a data dashboard represented a change in thinking 

about how to make data accessible and more useful to school staff: 

Maybe our approach of putting the data at the fingertips of every person was a pretty 

grandiose idea. Not that we are saying the opposite of that. Make it available to 

everybody, but recognize this generation of data users. They went into education to 

teach and design lessons and engage in meaningful work with kids. And so what is my 

inherent propensity to go to the computer and pull that data? But if you bring it to me 

and it tells me more about my children, tells me more about my instruction...yeah I’m 

interested. Let’s talk. 

The district is working with a commercial vendor to create a data dashboard that already has 

reports created so that all teachers have to do is press a button. 

Most of what we see is that people have too much data not too little. We see more 

data drowning than data scarcity and too much data doesn’t do much good 

either....Certainly we don’t want to take away people’s personalization of the product 

if they want to create custom reports, but we know there are four or five reports that 

we want everyone looking at. When we have district meetings, when we have 

department chairs from content areas from campuses together, when we bring 

principals together having some of those streamlined, similar reports will help 

expedite conversation and planning and that kind of thing. 

Other case study districts were changing over to more user-friendly systems to increase 

data usage so that there would be less need for data facilitators and professional 

development on how to use the data system, with mixed results. In one of these districts, the 

district has invested its resources in making their data systems more intuitive, with the 

presumption that schools and teachers will “discover” and teach each other how to use the 

systems (e.g., teachers were provided with an hour to a half day of formal training on the new 

student information system). One of the reasons for uneven use of data among school staff in this 

district was due to differences in their comfort level with using the technology—not all teachers 

are the kind of “self-starters” the district expects and some mentioned the need for additional 

professional development. 

A number of the case study districts use districtwide benchmark assessments to provide 

both district and school staff with formative data that allow them to more frequently assess 

student progress on district and state standards and to adjust instruction to meet 

individual student needs. Several of these districts mentioned involving teachers in the 

development of these assessments as a way to build capacity and to achieve teacher buy-in for 

 

24 The purpose of dashboards is to distill large amounts of data into a simple format to highlight problems and 

successes. Data dashboards include a series of key indicators at the district or school level that are tracked on a 

regular basis (e.g., academic performance, safe schools, attendance, accountability measures). The dashboard 

provides a snapshot at a specified point in time so that educators have a clear and timely picture of how their 

district or school fares on each measure so that they can take actions as needed—similar to a dashboard in an 

automobile with gauges that help the driver to monitor, at a glance, the speed of the car, the gas tank, oil pressure, 

and distance traveled. 

 55
use of the assessments. When these data are made accessible through the district’s data system, 

teachers are able to analyze the data by student groups and drill down in the data to the student 

level. At the same time, one of these districts indicated that some of the benchmark items written 

by teachers were not always of high quality, and the district subsequently contracted with some 

outside companies to write test items. These items were then reviewed by the district and by 

teachers, and used to construct the district benchmark tests. Concerns expressed by district 

administrators about teachers’ lack of knowledge about assessment mentioned earlier contribute 

to these limitations. 

The use of interim assessments both at the district and school level are also meeting the 

demands by school staff for more timely data (e.g., results of assessments given every six to nine 

weeks) that are linked to the local curriculum and sometimes to instructional resources to help 

inform instructional practice. Occasionally, it has taken more than one attempt to develop a 

benchmark assessment that was aligned with instruction and to devise a system for rapid 

deployment of the data back to teachers. For example, in two case study districts that had 

initially arranged for central processing of their benchmark data (student results were recorded 

on forms that were scanned at the central office because there was no capacity for online testing), 

they are now making provisions for school-based processing (scanners placed in selected 

locations) at the request of school staff so that results can be generated much more quickly. 

Another case study district created a benchmark assessment and reporting tool two years ago 

using a commercial product, but the math items were not well aligned with the district math 

curriculum. This system is being replaced with another assessment system (also developed in 

collaboration with a vendor) that contains assessment items that are better aligned with the 

curriculum and linked to more instructional resources for teachers. 

Piloting new data systems and data-related practices at a few schools was an approach 

mentioned by several of the case study districts (see Exhibit 3-13). 

Three of the case study districts have provided laptops to teachers to encourage data use. One 

of these districts provides laptop computers with Internet access to all of their teachers to provide 

anytime, anywhere access to data. One of the case study schools within this district has set up a 

school server that teachers can access through their laptops where they can share instructional 

materials, assessment data, and student work. The server also holds student data that is not 

available through the district data system (such as the reading assessment data for students in 

lower grades). This access to data has promoted data dialogues among teachers from different 

grades and changed teachers’ perceptions of what constitutes data. 

 

 56
Exhibit 3-13. Piloting Data-driven Decision-making Efforts 

• In one large case study district that has been developing its own curriculum assessment 

resource tool, 30 schools are piloting the system to aide in its continued refinement. In the prior 

year, five schools had piloted the assessment component within this system. 

• Another case study district has the philosophy that it is better to “put the systems out there,” 

even if they are not perfect, and solicit feedback from schools regarding how to make 

improvements. For example, the district piloted benchmark tests in reading and math for a year 

or two in Title I schools and then made them available districtwide (because of Title I funding 

these schools had additional support staff to help teachers use data). The plan is to gradually 

add more tests (e.g., social studies and science in the next year) and hire teachers in the 

summer to write a user’s guide on how to access benchmark data from the new assessment 

system. 

• A third district likes to build from within—seeking out information, practices and tools and then 

bringing them back and making them their own. This philosophy is evident in the practice of 

piloting programs to address problems as they emerge and employing a Teacher Council (with 

representatives from every school organized by school level) to serve as an advisory group to 

talk about new initiatives and data issues. District administrators talked about listening to “noise 

in the system” to identify problems and then piloting a program or practice to address it. If it 

shows promise, then they use word of mouth to build “buzz” to develop buy-in from school staff. 

Piloting may be a slower adoption process, but it has worked well for this district. 

A number of the case study districts have developed or participated in a range of 

partnerships to support their efforts to use data. For example, one of the large case study 

districts has been supported by two privately funded organizations to provide training in 

collaborative planning, teaching and learning as part of an effort to develop model districts 

across the state. The district has also contributed funds so that a broader range of teachers can 

participate in data-driven decision-making training. Another district noted that they have used 

the service of their regional consortia (state-supported regions) to discuss special education data: 

“The Education Service District has been working monthly with a group of us from the district. 

With me, we’re working off of the 14 major [federal] indicators of special education, doing data 

carousels. We look at the data, looking for patterns and trends. Trying to figure out how best to 

provide services to these kids that will make a difference.” Another case study district has 

applied for grants from foundations as well as the federal government for data-related activities 

(e.g., Reading First, Small Learning Communities). This same district has partnered with the 

local university to develop a teacher training program that includes the use of data as part of 

some nonnegotiable research-based instructional strategies that the district identified. These 

strategies are also the focus of the professional development that new teachers receive for the 

first three years they are on the job. 

Summary 

Data from the district survey and case studies illustrate the interrelationships between data 

systems and district activities to encourage and support the use of data at the school level. The 

dynamic and interacting factors that influence the structure and functioning of districts must be 

coordinated and brought to bear to achieve successful data-driven decision making. This often 

requires change in practices among district administrators and ultimately at the school level. 

Beyond the presence of data systems with the capacity to store and analyze data, survey 

responses indicate that districts are demonstrating their support and leadership for schools’ use of 

 57
 58

data through a number of actions corresponding to the supporting conditions outlined in the 

study’s conceptual framework. Districts are providing leadership to develop a culture of data use 

at both the district and school levels. Strategies include establishing a process for continuous 

improvement guided by data (e.g., instituting policies and requirements for principals and 

teachers to use data in their planning activities), providing support positions for system 

implementation (and tracking usage), and designing professional development and capacitybuilding activities for schools. 

Districts are also beginning to develop and employ tools for generating data (e.g., benchmark 

assessments in core subject areas that are aligned with curriculum standards to track student 

progress, Web-accessible library of resources that teachers can access if their students are 

struggling with a particular standard) as well as providing their schools with tools for acting on 

data (e.g., models for school staff to illustrate how to use data to design school improvement 

activities). But district progress is uneven and it appears, cyclical, with challenges varying over 

time. 

Many districts, including those identified as high data users, are struggling with logistical 

barriers to implementing data-driven decision making in their schools and the lack of 

administrator and teacher preparation to use data effectively. At the same time, there are school 

staff who are supporting the development of data-driven practices within their schools and 

generating data required for instructional decision making when district data systems fall short. 

These issues are discussed in greater detail in the next chapter. 

.
4. Use of Data Systems and Data-driven Decision-making in Schools 

The prior chapter described ways in which districts have been promoting the use of data 

systems and data-driven decision processes in their schools. In this chapter, we examine datadriven decision making from the school perspective. The data sources for the chapter are 

secondary analyses of a teacher survey administered in 2007 as part of the National Educational 

Technology Trends Study and school staff interviews and focus groups conducted during the 

2007–08 school year at 36 schools in the 12 case study districts described in Chapters 2 and 3. 

Major themes coming out of the analysis of school-level data were the following: 

• The most common school-level uses of data described by case study school staff are 

school improvement planning, curriculum decisions, and placement or grouping of 

students for instruction or support services. 

• Use of data to support inquiry into how one can improve one’s teaching appears to 

emerge later than use of data to set school improvement targets, to identify students for 

special services, or to fine-tune curriculum coverage to match accountability tests. 

• The most powerful school-level catalysts for teacher use of data are school leader 

promotion of these activities and the establishment of an organizational climate of trust 

and mutual respect. 

• School staffs’ perceptions of barriers to greater use of data include a sense of lack of 

time, system usability issues, the perception that the data in the system are not useful, and 

district policies around pacing that prohibit modifying learning time to match student 

needs. 

Nature and Frequency of School Use of Data and Data Systems 

On the NETTS teacher survey (U.S. Department of Education 2009) the majority of 

teachers with access to student data systems reported that they use the systems to provide 

information to parents (68 percent), track individual test scores, and monitor student 

progress (both 65 percent). Less common uses of data—reported by just over a third of the 

teacher survey respondents—were identifying promising practices (39 percent), informing 

student placement in courses or special programs (37 percent), and assessing test taking needs 

(36 percent). Exhibit 4-1 shows the 10 data uses most often reported by respondents in the 2007 

NETTS Teacher Survey. 

The present study was designed to complement survey reports with qualitative data from the 

sample of case study schools. School staff descriptions obtained from over 200 respondents 

during the school site visits provide a more detailed and interpretable view of uses of data and 

data systems in the case study schools. In viewing descriptions of data use practices in case study 

schools, it should be kept in mind that these schools are in districts that are not representative of 

school districts as a whole. Teachers in the case study schools had more electronic data available 

to them and more supports for data use than did teachers in U.S. schools as a whole during the 

2006–07 and 2007–08 school years. 

59
Exhibit 4-1. Teachers Who Reported Using a Student Data System at 

Least a Few Times a Year for a Specific Function in 2007 

Data Use Activities Percent 

Inform parents about student progress 68 

Track individual student test scores 65 

Estimate whether your students are making adequate progress 65 

Track other measures of student progress 59 

Identify skill gaps for individual students so that you can give each 

student materials tailored to his/her skill profile 55 

Determine whether your class or individual students are ready to move 

on to the next instructional unit 44 

Track standardized test scores by grade 44 

Inform curriculum changes 43 

Evaluate promising classroom practices 39 

Inform student placement in courses or special programs 37 

Exhibit reads: Among teacher survey respondents with access to a data system, 

68 percent reported that they use the system data in talking to parents about their 

student’s progress. 

Source: NETTS teacher survey, 2007. 

During school site visits, principals, coaches and teachers were asked to describe specific 

incidents of using data to make instructional decisions. From the 36 schools visited in 2007–08, 

188 examples of using data to inform instruction were obtained.25 Exhibit 4-2 shows the 

different purposes for which staff in case study schools described using data and the number of 

distinct citations of each use (a school could report the same kind of use on more than one 

occasion or with different kinds of data or groups of students). 

As shown in the exhibit, the 10 most common uses of data described by case study school 

staff were for school improvement planning (35 citations), curriculum planning (25), and 

decisions about student placement in classes or special services (22). Informing parents about 

student progress, the data use most commonly selected by the nationally representative sample of 

teacher survey respondents when viewing a menu of options, was described only three times in 

the 36 site visits to case study schools. One possible interpretation of this apparent discrepancy 

across the two data sources is that teachers do in fact commonly show students’ data to their 

parents (for example, during parent-teacher conferences) but do not think of this activity as 

“using data” unless prompted to do so (as they were by the NETTS survey question). 

 

25 When multiple informants from the same school described the same incident of data use, the incident was counted 

only once. 

60
Exhibit 4-2. Uses of Data Described by Case Study Schools 

Data Use Frequency 

School improvement planning, including setting of quantitative goals 35 

Curriculum planning based on item or subscale analysis 25 

Student placement in classes or special services 22 

Grouping or regrouping of students within a class 21 

Tailoring instruction to the skill needs of individuals or small groups 15 

Deciding whether or what to reteach 13 

Identifying teachers with more successful strategies in order to emulate 

their instructional approach 

11 

Referring students from classroom for supports or services 9 

Determining what aspects of your teaching are working well/poorly 8 

Evaluating teacher performance 7 

Exhibit reads: School improvement planning was the most common data use described by 

staff at the 36 case study schools. 

The data in Exhibit 4-2 suggest that in districts considered leaders in data-driven 

decision making, most schools are using data to develop goals for school improvement and 

to do curriculum planning. At one elementary school, for example, staff spend August and 

September analyzing data to determine whether they met the prior year’s goals and to develop 

new goals and make adjustments based on this analysis. These case study school reports are 

consistent with the responses of surveyed districts (Exhibit 3-6): 69 percent of districts reported 

requiring all or particular schools to follow specific data-driven decision-making practices in 

developing their school improvement plans. 

Specific examples of data-driven changes from the case studies include an elementary school 

in which the instructional coach reviewed fall data with the third-grade teachers and determined 

that many of their students had difficulty identifying the main idea of a passage leading to the 

decision that third-grade teachers needed to focus more instruction on that skill. At another 

elementary school a teacher reported: 

We looked at our test scores as a first-grade team and found that we were very weak 

in one area—phonemic awareness. We spoke to [the literacy coach about possible 

interventions]. She suggested bringing in more rhymes to help students discriminate 

different sounds. 

Some of the case study schools had become quite sophisticated about analyzing their state 

accountability test data and their students’ performance in ways that would enable them to raise 

test scores. In one school, for example, a review of students’ performance on the district’s 

benchmark tests revealed that students were doing poorly on items asking them to check the 

correctness of the grammar of a provided sentence. In response, the school staff took sentences 

61
from the benchmark assessments for use in classroom instruction and started instructing students 

using the same method of indicating an answer (e.g., underlining the predicate in a sentence 

rather than circling it) as the test did. Thus, the school responded to the assessment data both by 

increasing its coverage of grammar and by aligning the mechanics of its coverage to the specifics 

of the test. At the same school an item analysis of student data revealed that their students were 

not doing well on the vocabulary section of the state test. They noted that their teachers were 

spending almost all of their reading time working on comprehension, but that the language arts 

section of the state test had 10 to 15 vocabulary items compared to 18 to 19 comprehension 

items. The next year teachers spent more time on vocabulary instruction and saw a big jump in 

test scores.26

Year-to-Year Growth in Data Use 

An examination of changes in data use practices between the first and second round of 

school site visits suggest that the strongest levers districts have for increasing schools’ use 

of data systems are (a) providing timely data that school staff view as relevant to 

instructional decisions and (b) making sure that data systems are accessible and easy to 

use. 

Eighteen of the 36 schools visited by researchers during school year 2007–08 had been 

visited a year earlier as part of the 2006–07 data collection. Researchers reviewed descriptions of 

data use provided by teachers and school leaders in schools visited twice to ascertain whether or 

not there were changes in data-use practices at those schools between the 2006–07 and 2007–08 

school years. In 13 of the 18 schools, there was some evidence of an increase in data use over 

this one-year period; in four schools there was no indication of a change in the frequency or 

nature of data use; one school’s report could not be classified. 

Districts are generating more timely assessment data that teachers can use in 

instructional planning by implementing districtwide interim (either formative or 

benchmark) assessments. Exhibit 4-3 provides definitions for these and related assessment 

terms. Schools in districts involved in implementing a major system of districtwide interim 

assessments were the most likely to show an increase in data use from year to year, and also 

provided the most striking examples of changes in teacher data use practices, such as the 

institution of collaboration around the review of assessment results to derive implications for 

instruction. Staff at six case study schools reported that the introduction of interim assessments 

and the incorporation of the results of these assessments into an electronic data system had 

increased their use of data. 

 

26 Some critics charge that ESEA has placed so much focus on annual achievement test scores that schools are 

“teaching to the test” rather than helping students learn more broadly (Koretz 2008). Often times school practices 

fall into a gray area. Some might view these practices as an appropriate focus on important content, whereas 

others might see an inappropriate emphasis on becoming test wise. 

62
Exhibit 4-3. Assessment Terminology 

Assessment for Learning—See formative assessment. 

Benchmark Assessment—Benchmark assessments are used to measure progress along the way 

as learners work toward a target level of proficiency. The purpose of a benchmark assessment 

is to predict which students will attain the target proficiency on a future assessment, assuming 

progress continues at the same rate as in the past. 

Formative Assessment—Sometimes referred to as “assessments for learning.” Assessments 

given during the school year to inform future instruction. Assessment experts (Pellegrino, in 

press) note that you cannot tell whether an assessment is formative by examining the 

assessment items; you need to see how the assessment results are used in practice to 

determine whether or not it is formative. 

Interim Assessment—Assessments given during the school year, prior to an end-of-year or 

end-of-course examination. This term is neutral with respect to the assessment’s purpose 

(formative or predictive). 

Standardized Assessment—An assessment that has been administered to a large norming 

population so that a student’s score can be used to estimate where that student would rank 

within the norming population on the competencies or knowledge being measured. 

Criterion-Referenced Assessment—An assessment organized around specified levels of 

achievement; designed to compare an examinee’s performance to pre-specified achievement 

criteria rather than to the performance of other examinees. 

Summative Assessment—Assessment given at the end of an educational unit or experience to 

measure a student’s level of achievement; often used as the basis for awarding a grade or 

course credit. 

For example, in one case study district the introduction of benchmark assessments has had an 

impact on instructional practices because teachers now have more detailed data on student 

performance. Many teachers in this district lauded the fact that they can see how individual 

students perform on particular test items. They are also able to compare their own students’ 

performance with performance in other classes, the school as a whole, and the district as a whole. 

Because they can compare their class with other classes in their own school, teachers can see 

whether their students’ performance on particular items is due to their instructional practices or 

the wording of the test item, which affects students in all classes. This helps them decide whether 

they need to reteach or make changes to their own teaching practices. This kind of comparison of 

assessment results across different classes was described by teachers in eight of the case study 

schools.27

In another case study district, the introduction of districtwide assessments in language arts 

and social studies between the first and second site visits provided a forum for teacher 

discussions of student performance on specific test items and topics. Teachers began using their 

common planning time to compare performance of their respective classes on the common test 

and discuss the relationship between different instructional approaches and student performance. 

One such example came from a pair of social studies teachers who taught the same content but 

 

27 Good practice with respect to student privacy would require that teachers aggregate their students’ data to the 

class level or mask student names so that only a student’s teacher or teachers have access to his or her 

performance. 

63
had students with quite different performance on the benchmark test. After the meeting, the 

teacher whose students had performed better began coaching his colleague 

Some teachers have found it useful to share results of interim assessments not just with 

colleagues but with their students. One teacher noted, “The kids even like to look at it [the 

interim assessment report]. That’s so cool. We talk about how we need to analyze the question.” 

In case study districts in which interim assessments were not being introduced, the 

changes in school staffs’ use of data generally would be described as incremental rather 

than dramatic. Three case study schools reported looking at more data because their district had 

a new system or an enhancement to an existing system that provided greater functionality. At one 

of these schools, staff reported that in 2007–08 they could get data from the system directly 

themselves instead of going through the principal or a district staff person and this change had 

increased their frequency of data use. Two schools reported increased data use related to more 

teacher experience with a system introduced in 2006–07 or as a result of staff training. Teachers 

at one school reported that they were no longer satisfied with the standard reports they received 

from their assessment data system and had begun asking the district data staff for custom reports 

to address more specific questions about their students. Another school reported increased 

collaboration around data in the middle grades. 

School-level data underscore the complex relationship between system usability, data 

quality, and teacher training and experience noted previously in Chapter 3. The change in 

teacher attitudes and system use at an elementary school in a case study district that had 

introduced an interim assessment system during 2006–07 (the year of the first site visit) 

illustrates the inter-connectedness of these factors. During the first year the assessment system 

was implemented, teachers expressed discomfort and dissatisfaction with the system. They were 

required to give interim math assessments (intended to be formative) from the system four times 

a year, and they complained that the assessments did not match their curriculum and hence the 

data were not very useful. Also during this first year of system use teachers relied on the 

principal to pull assessment data reports for them because they did not have direct access to the 

system that contained the data. During the 2007–08 site visit, in contrast, both individual 

teachers and grade-level teams were making more use of data from the districtwide interim 

assessments. Though the assessments were still not a perfect match to the school’s instructional 

sequence, the teachers had learned how to create reports of the subset of assessment items that 

did match what they had taught. Teachers looked at these data to determine which skills had 

been mastered and which should be retaught. The teachers reported that the assessment system 

itself had been improved with the addition of links to instructional activities and lesson plans tied 

to standards. Most teachers described the system as more user friendly than the one teachers had 

the prior year, and they reported feeling comfortable using it without the assistance of the 

technology coordinator or content specialist. 

Stages in Developing a Data-using Culture 

Based on the first round of school site visits in 2006–07, the research team hypothesized that 

school staff go through a developmental progression in their data use activities. Early uses of 

data are often for the purposes of school improvement planning (in many cases, a use mandated 

by the district and required by ESEA for failing schools) and for assigning students to special 

64
services or to classes at different levels. This activity typically is conducted by school, gradelevel or department teams. In the next stage, a school’s teachers start using data for making 

decisions about what happens within their individual classrooms. At this stage, teachers look at 

data for the students in their classroom and do item analysis in order to make decisions about 

what to stress in their curriculum, how to group students with similar instructional needs, and 

whether or not to reteach certain material. This stage corresponds to the kind of progress 

monitoring that is often used in special education (Fuchs and Fuchs 2006). The third stage of 

data use development occurs when teachers reach the point where they begin looking at data and 

consulting others to help them make decisions not just about their students’ skill levels and what 

they should teach but about the way that they teach. In this stage teachers begin comparing the 

gains that their students have made in specific areas to the gains made by students in other 

teachers’ classes and start sharing the practices that have produced the best student gains. It is at 

this point that the school has evolved to a data using culture with mechanisms for continuing 

self-improvement based on the combination of data use and collaboration. Using data has 

become second nature to staff at schools that have attained this third stage and created a culture 

of data use. 

Exhibit 4-4 presents the three categories of school data use hypothesized as stages along with 

case study examples of activity in each category. The examples illustrate the attainment of the 

defining characteristics of the category, but readers should note that the hypothesized stages are 

cumulative rather than discrete: When a school begins engaging in Category 2 activities, it 

continues Category 1 activities, and when school staff begin the kind of collaboration around 

instruction that distinguishes Category 3, they still engage in Category 1 and 2 activities as well. 

65
Exhibit 4-4. Categories of Data-driven Decision Making 

Category Examples 

Category 1: Staff examine data for whole grade 

or school to ascertain areas for school 

improvement; examine data for individual 

students for purposes of class placement, or 

assignment to services, including identifying 

“bubble kids” whose growth is likely to affect 

the school’s AYP status. 

Across multiple dimensions, data is used for 

school improvement at elementary school. 

Teachers are given grade assignments and 

professional development goals based on data. 

Students are assigned to particular classes, 

receive additional instruction, and are publicly 

acknowledged based on data. 

Category 2: Teachers analyze performance of 

students in their class on individual items or 

standards for the purpose of better aligning 

their content coverage with the accountability 

test, or deciding what to reteach or how to 

group students within the class. 

At the first of the year, the first and second grade 

teachers meet as a grade level for a day. They 

use data from the state test to find out which skills 

the grade-level team needs to work on as a whole. 

The instructional coach gave another example in 

which she asked the grade-level teams to bring 

pretest data to the meeting, that is, data assessing 

what students know about the unit teachers were 

about to teach. The third-grade teachers were 

surprised by how many kids did not know main 

idea and decided to focus more on it. 

In some cases, teachers changed their pacing to 

address weaknesses identified in the data. For 

example, a fourth-grade teacher said, “Last time 

we looked at data, we saw that the majority of 

students struggled with elapsed time and long 

division so in addition to the work being done in 

the intervention groups, I also tried to review it in 

the classroom as much as I could even though I 

was on a different concept.” 

Category 3: Staff examine data for different 

teachers or for different methods dealing with 

the same content to derive insights for 

improving the way they teach. Staff use 

comparative data to evaluate the effectiveness 

of specific instructional strategies. 

Teachers at an elementary school use the data to 

identify areas in which they could improve their 

instruction, because, for example, they are not 

obtaining the same assessment results as other 

teachers in their school or because many of their 

students did not reach proficiency on a particular 

topic. Teachers spoke highly of the item-level 

analysis available through the data system to help 

focus attention on individual and group needs. 

A teacher offered, “When we did go over our data, 

we would see which teacher had more strength in 

math and ask, ‘What are you doing?’” 

These categories of data use, developed by analysts based on the first round of site visits, 

were used as a framework for analyzing the specific examples of using data supplied by school 

staff during the second round of site visits. When the examples of data use described by case 

study school staff were classified in terms of the three data-use categories described in 

Exhibit 4-4, analysts found that most case study schools provided examples of Category 1 and 

Category 2 uses of data, but less than half of all schools gave a Category 3 example. Specifically, 

66
35 of 36 (97 percent) case study schools provided one or more Category 1 examples; 30 of 36 

(83 percent) provided one or more Category 2 examples; and 17 of 36 (47 percent) provided one 

or more Category 3 examples. Because the great majority of case study schools described 

instances of both school-level (Category 1) and classroom-level (Category 2) data-based 

decisions, there is no support for the study’s hypothesis that using data at the school level is 

typically the first step in becoming a data-using school. It does seem clear, however, that using 

data to reflect on how one teaches and not just what one teaches or to whom (Category 3) is 

a more advanced stage in the building of a data-using culture. Exhibit 4-5 shows the relative 

frequency of data-based decisions in these three categories. Exhibit 4-6 describes the activities of 

a middle school that had moved into this third stage of data use. 

Exhibit 4-5. Percent of Case Study Schools Reporting 

Each Decision-making Category 

47%

83%

97%

0% 20% 40% 60% 80% 100%

Category 3

Category 2

Category 1

Data‐driven decision making categories

Percent of Schools

Exhibit reads: Among case study schools, 97 percent provided examples of 

Category 1 uses of data. 

Source: Case study schools 2007–08. 

67
Exhibit 4-6. A School at Stage 3 in Data Use 

Case study researchers judged teachers at this middle school to be highly sophisticated data 

users. These teachers expressed a sense of urgency to improve scores because of the changing 

demographics of the schools student population. Over the past four or five years, the school had 

seen a considerable shift from a primarily white, middle-income population to a primarily minority, 

low-income student body. School year 2007–08 was the first year in the last four that they had 

made AYP. Teachers were enthusiastic about the ability of the new district assessment system to 

provide scores from just-in-time benchmark tests and were using multiple data and assessment 

systems to group students, link attendance and discipline data to student achievement measures, 

plan remediation, and improve lesson plans. Teachers, individually and as a group, had learned to 

make data queries and analyze test scores for student subgroups as well as to carefully analyze 

patterns in incorrect answers in order to help determine areas in which their instruction was weak. 

Some teachers exported data into Excel for further analysis. In weekly team meetings the school’s 

assistant principal provided teachers with test data for all members of the team (e.g., all English 

and language arts teachers for Grade 8), and teachers examined the data, looking for colleagues 

whose students did well on a particular test item or item strand so that they could ask that teacher 

to share teaching strategies. Teachers in these groups jointly wrote and tried out at least one new 

lesson plan every three weeks, usually in an area of weakness. 

School Supports for Data Use 

Chapter 3 described the kinds of activities that districts undertake in order to encourage their 

schools to use data in making instructional decisions. This section discusses data use supports 

implemented at the school level. The 2007 NETTS teacher survey provided a nationally 

representative snapshot of school supports by asking teachers to indicate which of a set of 

potential supports for data-driven decision making they had experienced. The most commonly 

provided supports, according to the teacher survey, were professional development at the 

school (58 percent), principal encouragement (56 percent), and support from a consultant 

or mentor teacher skilled in data analysis (25 percent), as shown in Exhibit 4-7. 

Exhibit 4-7. Teacher-Reported Supports for Use of Student Data to Guide 

Instruction, by Type of Support in 2007 

Type of Support Percent 

Professional development on data-driven decision making provided at school 58 

Principal’s encouragement for using data in instructional decision making 56 

Support from a consultant or mentor teacher skilled in data analysis 25 

Professional development on data-driven decision making provided outside 

school 20 

Paid time set aside for examining student data and using the data to guide 

decision about practice 12 

Formal coursework covering data-driven decision making 6 

Exhibit reads: Among teacher survey respondents with access to a student data system, 

58 percent indicated that they had received professional development on data-driven 

decision making at their school. 

Source: NETTS teacher survey, 2007. 

68
Site visits to case study schools found many examples of the school supports mentioned most 

often in the teacher survey (professional development, principal encouragement, support from a 

coach or mentor teacher). 

Professional Development 

Formal professional development that districts offer on the use of data systems was described 

in Chapter 3. Case study data suggest that this formal instruction on how to use a data 

system or apply data to instructional decisions, though widespread, is usually quite short in 

duration. In one case study district, for example, teachers reported receiving only a single hour 

of formal training on how to use the two electronic systems available to them. Typically, the 

district offers teachers just a single training session on how to get data out of the system. The 

research on professional development suggests that such “one-shot” training experiences are 

generally ineffective for changing practice (Loucks-Horsley et al. 2003; Garet et al. 2001). Site 

visits to case study schools in active data-using districts found that they were supporting 

teachers’ use of data not through additional formal professional development but rather through 

ongoing interactions around data with school leaders, mentor teachers or coaches, and 

colleagues, as described below. 

Assistance From a Mentor Teacher or Coach 

Having a staff member on-site who can support not just the interpretation of data but 

also the connection of data to instructional decisions appears to be a critical support. Such 

individuals have credibility because of their instructional expertise and can encourage and 

facilitate teachers’ engagement with data. In some ways, mentor teachers or instructional coaches 

have an advantage over principals in this role because they are not responsible for evaluating 

teachers, thus removing an element of teacher anxiety. Mentor teachers or instructional coaches 

can provide teachers with the confidence to engage with data in an environment that feels low 

risk. A first-grade teacher at an elementary school implementing Reading First, for example, 

described how that program’s data coordinator led grade-level meetings to discuss literacy data. 

We hear about other teachers, their students’ struggles. [The Reading First coach] 

runs the meetings. It’s nice to just have someone at the helm. It helps to know 

someone is there seeing the same issues I am seeing. We talk about ‘Let’s try these 

things.’ We’re good with that with literacy. 

The principal at the same elementary school noted that teachers there did not feel 

comfortable interpreting data by themselves: 

When teachers are working individually, if you don’t have the background, there is a 

very low comfort level with understanding what it [the data] means and says. 

Because of that discomfort people aren’t sound and secure in making judgments. 

They definitely need support. 

Another good example of teachers working with instructional coaches came from an 

elementary school in which teachers look at data with content area coaches on an ongoing basis. 

During weekly cluster meetings, math and literacy coaches alternate leading the discussion, 

which generally involves data. At one meeting, for example, the math coach may lead the 

teachers in a discussion about student performance and then in brainstorming about strategies for 

69
addressing their students’ needs. At another meeting the literacy coach may lead teachers in 

looking at all the performance data available on one particular student. At this kind of meeting, a 

teacher will bring a student’s literacy folder, which includes state and district assessments, 

writing samples, teacher-student interviews, and other data. The teachers and the coach 

brainstorm together about how to work with that particular student based on this data. 

Similarly teachers at an elementary school in another district described how working with 

their instructional coach was increasing the amount of time they spent talking about data and the 

extent to which they are using it. Previously, teachers met every three months to discuss data, but 

with the instructional coach they began meeting every month to go over their monthly action 

plan and DIBELS data. Said one teacher, 

I think I’ve always been good at collecting data, but I’ve never really known what to 

do with it. I think that has really changed this year. Instead of wasting my time 

collecting all this stuff for a cute little grade book at the end of the year, I’m actually 

using it [data] more. 

The case study observations suggest that the support being provided for data use by 

mentor teachers or coaches falls primarily into the area of linking data to instructional 

decisions. There were some cases of mentor teachers or educational technology coordinators 

helping teachers with system use issues per se, but these were far less common than reports of 

assistance related to instructional practice. 

Structures and Dedicated Time for Data Use 

Teacher survey data suggest that the majority of U.S. schools are not giving teachers time 

during the regular work week to focus on data use (U.S. Department of Education 2009),28 but 

site visitors found that time set aside for activities that included the examination of data was 

present at 30 of the 36 case study schools (83 percent).29 

• The middle school described in Exhibit 4-6, for example, has multiple opportunities for 

teachers to get together and discuss data built into its schedule because of district policies 

to make regularly scheduled time available to them. In addition to monthly grade-level 

meetings, there are weekly departmental meetings and meetings of teacher teams from 

different departments who share the same students. An innovative strategy at this school 

is the expectation that teachers create common lesson plans, addressing needs identified 

through the district benchmark assessment—every three weeks. Site visitors noted that 

teachers expressed enthusiasm for this opportunity to work together to address student 

needs. 

 

28 This finding is similar to district reports on the national district survey, in which lack of time for school staff to 

conduct data-driven decision making activities was the most frequent barrier reported by districts to spreading 

data-driven decision-making practices throughout their district (92 percent of districts cited lack of time as a 

barrier). 

29 The likelihood of having time set aside for data use did not appear to vary by school level within the case study 

sample. Among case study schools, 11 of 13 middle schools had dedicated time for data use as did 19 of 26 

elementary schools. 

70
• At an elementary school in another district, the assistant principal prints out student 

results after each interim assessment. She reviews the data with the principal, and gives 

each teacher data for his or her students. She asks teachers to follow up through e-mail 

with descriptions of what they will do in response to the data and encourages them to use 

the data on an ongoing basis. The assistant principal also gives all the grade-level data to 

team leaders. The grade-level teams hold monthly meetings at which they analyze their 

data. The assistant principal conducts item-level data analysis with the teams “so that we 

can see how many students choose a wrong answer and think about why.” 

• At another elementary school the principal decided in 2007–08 to ask each teacher to 

select achievement goals in reading, writing and math on specific assessments for their 

students rather than setting grade-level goals as in prior years. Each teacher was 

responsible for monitoring his or her students’ performance on the related assessments in 

the fall and winter leading up to the year-end measure. After each round of assessment, 

the principal had a one-on-one conference with each teacher in which the teacher was 

asked, “Where are you in relation to the achievement goals you set in August? What are 

the one or two things between now and the end of year that you will do to measurably 

impact achievement goals? What is something you need from me to ensure that your 

goals are met at the end of the year?” 

• District funding for grade-level meeting times and school improvement funds helped 

support multiple structures for interacting around data at an elementary school serving a 

high percentage of economically disadvantaged students. Teachers meet with a gradelevel partner to discuss their students’ data and their instructional approaches once a 

week. Every other week, they participate in a grade-level team meeting at which data are 

discussed. Finally, each teacher meets with the principal one-on-one three or four times a 

year. At these meetings, the teacher presents both their data and an action plan based on 

the data. 

Climate of Collective Responsibility and Mutual Trust 

A persistent theme across schools in which teachers look at data collaboratively is the 

importance of their sense that their colleagues respect them and their trust that the data 

will not be used to punish them. As noted in studies of other school reform efforts (Bryk and 

Schneider 2002), mutual trust and a sense that they are jointly responsible for the success of their 

students appear to be important factors in teachers’ adoption of data-use practices. 

As a teacher at one middle school said, “Lucky that nobody here has an ego. Everybody is 

very open about sharing. No fear that they will think I’m a bad teacher.” At another middle 

school, researchers noted the “we’re in this together” philosophy of administrators and staff. At 

this school names of teachers were included in data reports so that if one teacher’s class was 

doing particularly well in an area, other teachers could approach the successful teacher for 

strategies and advice. The data administrator at this school described the teachers as having 

“shared ownership” in the success of their students and an understanding “It’s not just about me; 

it’s about the whole school.” 

The principal at an elementary school expressed the belief that looking at data with their 

colleagues motivates teachers to pay more attention to it: 

71
Collaboration helps teachers’ motivation. Their kids are struggling. They see they 

aren’t the only ones having difficulties. They get advice from another colleague. It’s a 

motivation to get them to do it [use data to reflect on practice]. Internalize it more, 

pay more attention than if [you] work with data by yourself. 

Building this kind of mutual trust can be difficult, however. The principal quoted above 

noted that building trust took two years at his school and was still uneven across grades. The 

principal at a middle school commented, “Having that trust as a team sounds easy on paper, but 

teaching is so isolated. Teachers have never co-labored the way that we are asking them to 

co-labor now.” 

Strategies that case study school leaders used to foster the development of mutual trust 

included having teachers meet in small groups and starting with nonthreatening uses of 

data, for example by showing classroom-level assessment results without teacher names. 

Teachers at an elementary school expressed their preference for small groups when they look at 

student data. They said that they like to be grouped with other teachers they know well. As one 

first grade teacher explained: 

You feel more confident presenting your student [in a small group of teachers you 

know well]. You can really say what you are struggling in. You don’t feel as shy as 

you would if you were with a whole group of teachers you don’t know. There you are 

afraid to present something in case you are not doing your job well. 

At a middle school, the assistant principal first began preparing reports of benchmark test 

results by class with teachers’ names left off in order to preserve anonymity. When the staff held 

meetings to discuss the data, they got to the point where the teachers wanted to identify the 

colleague whose class was the one with the best performance so that they could share 

instructional strategies. Teachers were spending time at these meetings going around to each 

other to try to assign the unlabelled class assessment results to teachers. At this school it was the 

teachers themselves who asked the assistant principal to just put their names on the class-level 

assessment reports to make their collaboration around data more efficient. 

The principal at another elementary school offered the following advice to other schools 

embarking upon bringing teachers on board with data-driven decision making: 

Start slow with something they’re comfortable with. Teachers will be at different 

levels. [You] need to let each teacher move at their pace. Find someplace where 

there’s success so each teacher can see it’s working. Do it consistently. Make sure 

it’s meaningful. Don’t do it [look at data] just for its own sake. 

Principal Leadership 

Prior case studies of data use in schools point to principal leadership as a key enabler 

(Supovitz and Klein 2003; Wayman and Stringfield 2006). Principals encourage data use by 

setting an example through their own activities, providing time in the school schedule for 

teachers to examine and reflect on data, and by communicating expectations around data use. 

72
School leaders also set a tone for how to think about data. Just as the actions of principals 

and other school leaders convey the importance that they place on data, the way in which they 

frame data discussions provides a perspective on the meaning of data. A principal at a case study 

elementary school provided a striking example. In a year when the school had not expected to 

see increases in their state assessment results, but scores had in fact risen, the principal talked to 

the staff before showing them their scores. He asked the staff to think about the reasons why the 

scores might be what they were. The teachers responded by listing reasons that students might 

have low scores: that year they had many transient students, and many special education and 

English language learner students. The principal noted that the school had done some good 

things that year and enumerated them. He then showed the test scores the students had achieved, 

and after the teachers stopped cheering, he made the point that they could never use the nature of 

their students as an excuse again. They had shown that they could achieve progress with the 

students they had. The principal reported that this event was a turning point with the school 

environment shifting to one in which “it’s the students” was not acceptable as an explanation for 

poor performance. 

Beyond setting expectations for data use, principals at some of the case study schools 

provided tangible supports for this activity in the form of tools that teachers could use and 

a regular schedule of data reflection and decision-making activities. At one of the elementary 

schools, for example, the principal holds monthly meetings with grade-level teams for which she 

prepares charts and spreadsheets to illustrate student performance. The principal then challenges 

the teacher teams to think hard about what might be causing the patterns they observe in the data. 

Whenever teachers at this school propose a new instructional plan, the principal reminds them to 

use data to shape their proposal. 

Similarly, the principal at a middle school explained that when she asks a teacher how his or 

her students are doing, she expects an answer that includes a description of student data. The 

principal also tries to communicate the idea that the purpose of looking at data is not to judge 

individual teachers; rather, the point is the school’s collective responsibility for figuring out what 

can be done. 

In many of the case study schools, data use was an important tool in the principal’s 

assumption of the role as instructional leader. In 25 of the 36 case study schools visited in 

2007–08, researchers judged that principals were promoting data use in a way that gave 

them increased influence over the quality of instruction. Principals at 18 of the 36 schools 

led whole-school or grade-level meetings devoted to the analysis of data. Principals at 

several schools maintained schoolwide lists of students who were in danger of failing to 

attain state proficiency standards in order to keep on top of efforts being made to support 

those students during the year. 

Principals at three schools met with teachers individually to discuss their students’ needs 

and the teacher’s plan for the class. At one elementary school, for example, the principal 

was regularly working with individual teachers to help them interpret interim assessment 

data. He stated, “We pull [system] data right after formative assessments. We pull teachers 

in and make [color coded] printouts for them.” The principal refers to these 11 x 17 inch 

laminated printouts as “placemats.” These are designed for easy reading, with green for 

73
mastered skills or concepts and pink for areas in which students did not perform well. Using 

the “placemat” data, the principal works with the teacher to decide if remediation needs to 

be given to the entire class, a small group or an individual, and then the principal, teacher 

and intervention specialist work together to develop an instructional plan for standards on 

which one or more students need help. 

The first round of site visits conducted for the study suggested that leadership for data use did 

not have to come from a school’s principal: staff in other positions, such as a vice principal or 

instructional coach, could provide this leadership effectively, at least in schools within districts 

that support data use (U.S. Department of Education 2009). In the second round of case study 

site visits (half of which were to schools visited in the earlier round of data collection and half of 

which were to new schools), researchers saw more examples of strong principal leadership 

around data use, but again saw staff in other roles (largely assistant principals or data or 

instructional coaches) providing leadership and support for data activities. 

Supports Related to Data Use 

Perhaps because they are in districts that put an emphasis on data use, the teachers in case 

study schools described a level of support for data use that appeared stronger than that perceived 

by teachers nationally. The responses of teachers on the NETTS 2007 teacher survey 

suggested that most teachers with access to a student data system feel that they have 

someone who can help them use the system (71 percent) and that they are comfortable 

looking at data with colleagues (70 percent).30 On the other hand, on the NETTS teacher 

survey, only 23 percent of teachers reported that time was available for working with data 

during the regular school day, and 59 percent of survey respondents agreed with the statement, 

“If I use the student data system, it has to be on my own time.” As noted above, the case study 

schools by and large had time set aside for data examination and use. They also were likely to 

have school-based coaches who could help teachers relate student data to instructional strategies. 

Of the 36 case study schools, there were instructional coaches (usually for literacy or 

mathematics) who helped teachers work with data in 17 and a data coach in seven. 

The relationship among responses to different items on the NETTS teacher survey data 

suggests that supports such as those in case study schools have an influence on teacher 

behavior. When teacher perceptions of the support items shown in Exhibit 4-8 were put into a 

scale (internal reliability = .61) and correlated with a scale of data use frequency items (internal 

reliability = .76),31 the correlation between perceived support and frequency of data use 

 

30 Teachers responded to survey items using a Likert response scale calling for selection of one of five categories: 

strongly disagree, disagree, neither disagree nor agree, agree, or strongly agree. For the majority of these items, 

“strongly disagree” and “disagree” responses were summed to estimate negative perceptions, and “strongly agree” 

and “agree” responses were summed to estimate positive perceptions. For items with reverse coding, “strongly 

disagree” and “disagree” responses were summed to estimate positive perceptions, and “strongly agree” and 

“agree” responses were summed to estimate negative perceptions. 31 The internal reliability of a scale is a measure of how consistently the questions in that scale refer to the same 

underlying idea and is expressed as a number ranging from 0 (no consistency among questions) to 1 (perfect 

consistency or essentially identical questions). Scales with higher reliability are able to measure finer-grained 

distinctions in levels of the underlying idea then are scales with lower reliability. For items such as “I have trouble 

finding information,” disagreeing with the statement would indicate positive attitude indicated with (R). The 

percentage of teachers indicating a positive attitude is shown to the right of the vertical center line in Exhibit 4-8. 

74
was 0.40.32 A correlation of this size is considered indicative of a moderately strong relationship 

in social science research . 

Exhibit 4-8. Teacher Perceptions of the Support 

They Receive for Data Use 

12%

5%

13%

29%

54%

59%

71%

70%

42%

37%

23%

14%

I can turn to someone for help

I am comfortable having colleagues with me

My school has improved through Data Management 

System use

Available data not helpful in deciding what to teach (R)

Time is available as part of regular day

If I use the Data Management System, it has to be my own 

time (R)

Teacher Perception

Negative Positive

Exhibit reads: Among teacher survey respondents with access to an electronic data 

system, 71 percent agreed with the statement “I can turn to someone for help.” 

Note: (R) denoted an item that is reverse coded; for these items a disagree 

response indicates more support for data use. 

Source: NETTS teacher survey, 2007. 

 

32 A correlation of this size means that 16 percent of the variance in frequency of data use is explained by the level 

of support for data-driven decision making. The size of the correlation is attenuated by the reliability of the two 

measures. 

75
Barriers to School Data Use 

Despite general district encouragement for data use and the kinds supports for the practice of 

data use described above, implementation of data-driven decision making and establishment of a 

data-using culture are not easy. School site visits identified a number of factors inhibiting growth 

of the practice, as described below. 

System Usability Issues 

Wayman (2005) has pointed out that for many years data systems were simply too difficult to 

use for implementation within schools to be feasible. Wayman suggests that major advances in 

this area as one of the drivers of the current movement toward school use of data. Even with 

recent advances, however, many schools and districts are finding their data systems 

difficult for school staff to use because of either interface issues or because of limited 

system functionality. District perceptions of data system usability issues were described in 

Chapter 3. This section discusses the usability issues raised by school staff. 

A sample criticism from one case study teacher was “[You] can’t use the back button in [the 

system]. . . There are lots of precise procedures.” The teacher went on to explain that if you misenter something, you have to start over from the beginning and “[the system] takes a while to 

load.” To make matters worse, “Printing from [the system] is a nightmare.”33

In a number of cases, school staff complained about the lack of flexibility in the system 

available to them. There were other cases, however, when having a great many options proved to 

be overwhelming: 

Because there’s such freedom, there’s so much information, there are a thousandand-one pull-down menus and a thousand-and-one options for whether you want 

to know which six weeks a kid scored in, if you want to know just the Hispanic 

kids or the kids who are wearing blue that day, you can get any information you 

want, but it’s almost overload. Sometimes you just want a simple report, and you 

find yourself pointing and clicking everywhere and you don’t get what you want 

anyway. 

Teachers in a medium-sized case study district have access to a student information system 

and a “data mart.” Teachers input attendance and upload assessment results into these systems, 

but they do not use the systems to generate reports. They told researchers that the systems are so 

cumbersome to use that such efforts are not worthwhile. The director of school improvement and 

accountability in this district commented, 

[The system] is not an easy system to use, and we can’t make it easier. People are 

already turned off by it, so they don’t really want to try again. So it’s more than a 

training issue, it’s kind of a convincing issue. And, honestly, we don’t have someone 

 

33 The schools in the 12 case study districts use over three dozen data systems, some of which were district 

developed and some of which came from commercial sources. System names have been deleted from staff 

criticisms of systems because the veracity of staff critiques were not tested independently by research staff. 

76
who has time to convince and cajole people to use [it], particularly a system that is 

not user friendly. 

Teachers often decide whether or not to even try using a data system on the basis of reports 

from their colleagues. In one case study district, a representative teacher comment was “I have 

heard how laborious it is to enter test items on [system].” When school staff criticized a data 

system’s interface or usability, their comments were typically quite general (e.g., “It’s not user 

friendly). 

Perceived Lack of Timeliness and Relevance of System Data 

School staff were much more likely to criticize the quality or timeliness of the tests for 

which data are available in systems than to criticize the usability of the system per se.

Teacher criticisms of the quality of the data available to them included delay issues, lack of 

alignment with standards, lack of alignment with the school’s instructional approach, and the fact 

that they received only cross-sectional data rather than longitudinal data for the same set of 

students over time. 

State achievement tests are typically a once-a-year spring event with results unavailable until 

the next fall when students have moved into a new grade level. Staff at many case study schools 

described looking at these data at the time they became available, often using them for school 

improvement planning and for placing students into classes or special services, as noted above. 

Individual teachers look at the prior spring’s scores for the students coming into their class to 

help do planning at the first of the year but soon find more recent and fine-grained information 

from classroom assessments and informal observations guiding their instructional decisions. As 

one middle school teacher said, “We use numbers [from standardized tests] when we don’t know 

them [the students], and when we do know them [the students], we don’t use numbers.” 

In a large district, teachers pointed out that the state assessment results are not available until 

the following fall, after they no longer have the students who were tested the prior spring in their 

classes. Although the district gives benchmark tests six times during the school year, the results 

of these tests take six weeks to come back to teachers—at which point they have moved well 

beyond the assessed topics. 

Teachers in several districts criticized the districtwide assessments that provided much of the 

data in the systems available to them. Teachers noted cases in which content on the district 

assessment did not match content standards or did not line up with what they taught in their 

classes. Also, in several cases teachers noted (correctly) that a content standard was assessed by 

only a single item on the test, making it impossible to make valid, reliable conclusions about an 

individual student’s mastery status with respect to that standard. 

Teachers at an innovative school in another case study district expressed distrust of the 

standardized test data in their system because it reflected a learning theory so different from the 

one that was part of the school’s mission. 

A middle school teacher expressed frustration at not being able to get longitudinal data for 

the same group of students over time from the district’s system: 

77
We would like to see the data that reflects each child’s accomplishment over the year 

that you have taught him or her. We don’t have access to that information. We would 

like to access that information. We have to jump through hoops to get that kind of 

data. 

Lack of Time to Examine and Reflect on Data 

On the NETTS teacher survey, only 23 percent of responding teachers agreed with the 

statement, “Time is available [to look at data] as part of the regular day.” As noted above, 

the case study schools appeared to be much more likely than schools in general to set aside time 

for teachers to look at data together, but there were still references in a number of the case study 

districts to not having enough time set aside for these activities. A teacher described her need for 

designated time for data use: “I would like to have an afternoon a month. In Florida, we had a 

half a day every month for teachers to get together to plan or look at data. If we were given time 

during the school day, I’d be able to do it.” 

Lack of time appeared to be most problematic in the one small case study district. That 

district did not allow its schools to build release time for analyzing the data into the school year. 

The lack of time problem was exacerbated because training on data system use in this district 

was limited to a brief in-service session. Informal support was provided by an unofficial “data 

coach,” who was actually a full-time fifth-grade teacher with many other responsibilities. 

The case study data also illustrate that lack of time is actually a subjective perception 

and quite complex in practice. For example, lack of time was reported as a barrier to data use at 

a middle school in a district that provided extensive time during the school week for teacher 

meetings. Because the meetings were not used for discussing data and most teachers were not 

working with data on their own time, teachers at the school had the perception that there was no 

time for such pursuits. At a middle school in a different district, the teachers similarly reported 

not having enough time to examine student data. The school’s principal expressed the opinion 

that the teachers were not seeing the value of being proactive in their approach to using data, and 

for this reason they were not making the time to analyze student and classroom level data in their 

meetings. Administrators at another school in this district noted that teachers there saw data use 

as an “add on” rather than integral aspect of instructional practice. 

District Pacing Requirements 

At a middle school in a large case study district teachers give interim assessments several 

times a quarter and look at interim assessment results during time set aside for professional 

learning community activities. The items on the tests are aligned with specific goals and teachers 

can do item analysis. It does not appear that teachers at this school look at the interim assessment 

data outside of professional learning community time; however, because they have to move on 

through the curriculum, regardless of what the data reveal about student mastery, to conform to 

the district’s pacing calendar. Responses on the NETTS teacher survey suggest that this barrier is 

quite commonplace. Sixty percent of teacher survey respondents agreed with the statement, 

“No matter what the interim assessments for my students indicate, I have to keep up with 

the pacing plan for the year because of the state and district mandates.”

78
Lack of Training 

In case study schools, the need for training in how to interpret data and connect it to 

instructional practice appeared greater than that for additional training on system use per 

se. As noted above, the amount of time spent on formal district-provided training on data system 

use tended to be quite limited in the case study districts. Staff at case study schools described the 

training with words such as “brief” and “a few hours of information shoved into a half hour of 

time.” Some districts used a strategy of training the principal or a lead teacher at each school and 

then relying on that person to train his or her colleagues. Although only a minority of staff 

interviewed at case study schools indicated a desire for more formal professional development, at 

least one staff member did so at 17 of the 36 case study schools. 

One district was implementing a new data system and had provided limited or no 

professional development to school staff. An elementary school principal explained, “We need 

training on how to easily get your teacher roster [from the system] and how to group those kids 

for what they need to know in [the system], and we need the time to do this training that’s not at 

3:30 in the afternoon.” 

In another district a middle school teacher offered a similar opinion about the need for more 

training, “We do have a staff development to help us use the basics, but we still don’t know how 

to do everything.” A math teacher at the same school remarked, 

We now know how to access more general class information but now...we need to 

access more specific information and we are kind of on our own clicking and looking 

around and I was able to find item-level analysis and some other really useful 

functions....There are a few extra steps we have to do, but I’m really excited when I 

figure out how to create a really good chart. 

This sentiment was echoed by a sixth-grade teacher at another school: 

[There is] not enough training about how to interpret data. Some math whizzes get it, 

[They] understand things like the variability in a score. But we’re not given enough 

PD [professional development]. We’re trained how to access the system, how to input 

data, but that’s mostly grades and attendance. Maybe we were told how to access test 

scores. I had to figure it out mostly on my own. 

At another district in its first year of implementing a new system with interim assessment 

data, teachers felt that they needed more professional development. Some teachers did not know 

how to use the system and others could extract data from it but did not know what to do with the 

data: “I wouldn’t say that everyone is absolutely certain what to do with the data that’s available. 

It’s an excellent resource, but I think we’re still all learning how to use and maximize it. We 

need more time and more...professional development.” 

In some cases the more technologically savvy among a school’s staff are stepping in and 

providing support for their colleagues who need more training. One such teacher reported, “[We] 

had some training this summer where we were introduced to [system]. We had one-on-one 

sessions. [The principal] would pull data onto the big screen.” This teacher went on to explain: 

 

79
A lot of it is jumping in and figuring out which reports to run. I’m comfortable with 

technology so it works for me. [It] might not be enough for others if they aren’t as 

comfortable.” In fact, this teacher has become so skilled at using her district’s data 

system that she was asked to provide training to her colleagues. This teacher stated: 

“I like to put [the system data] into Excel and run some formulas to see the 

percentage of standards mastered and standards hit least. That’s easier for me to see 

because the colors [to indicate mastery on the standard system report] stress me out. 

My husband has made some macros for me to use in Excel. After Christmas break, I 

taught a tech class. I taught the third- and fourth-grade teachers how to use the 

macro. They were excited. 

Perceived Effects of Data Use 

The present study did not include a test of the effects of data use on student achievement 

outcomes. It did, however, collect school staff perceptions of the effects that their data use 

activities were having. Among a majority of case study schools, staff perceptions of the 

impacts of data use were quite positive. 

• A middle school reported that, since it started using data in instructional planning, student 

performance in general, and that of their English language learners and special education 

students in particular, had improved. The principal explained that the data have given the 

teachers a better understanding of students’ strengths and weaknesses, “Data frees up 

time for teachers to focus on instruction rather than trying to figure out where each 

student is.” 

• Teachers at a middle school in another district expressed their belief that the use of data 

had improved student achievement at their school. The data had helped them focus in on 

the math learning needs of students with disabilities, a group the school had not served 

well in the past. As a result, they were able to make AYP for the first time in three years. 

• Staff at an elementary school that has achieved state recognition as an exemplary school 

despite serving large proportions of low-income, minority, and special education students 

attribute their success to years of examining data. The principal stated, “I really think we 

would not be able to perform as well as we do without that data. The data that [system] 

provides as well as what [the instructional advisor] does has been the core of the success 

of our school. If you didn’t have data, you work blindly. Data provides direction for 

instruction, intervention, everything.” 

• A principal in another school district was able to give a specific example of the impact of 

data on student achievement. Several years ago, the school noted that its social studies 

test scores were low compared to other district schools with similar students. The 

principal sought advice on how to teach social studies from the principal of a school with 

similar student demographics and high social studies scores. After implementing the 

other school’s approach to social studies, the students’ test scores rose. 

• A principal in an elementary school serving mostly low-income students expressed his 

belief that the use of data combined with “some solid direct instruction” turned that 

elementary school around. He explained, “I think student achievement has gone up 

80
81

because teachers are now teaching to kids’ skill deficits. They’re remediating instead of 

just teaching to the middle [achievement level] the whole year.” 

• The principal at an elementary school designated in need of improvement expressed the 

importance of data use: 

When I came in last year, it was black and white. I used the data for dialogue 

with faculty, to be specific about the areas where we needed to improve. We were 

under improvement; it was very clear using the [state test] scores where we 

needed to improve. This October we made every target the state set for us, in 

every subcategory; we were very pleased. We are still in corrective action, as we 

have to maintain the scores for two years. [But] there is no question mark; we 

know what we need to do. The data makes this clear. 

Summary 

School staff reports underscore the interrelated nature of the components of implementing 

data-driven decision making. Such efforts require user-friendly software, but more than that, 

there needs to be data in the system that teachers regard as worth looking at if teachers are going 

to use it. Training on how to use the system is particularly important if a system is not user 

friendly, but a lack of formal training can to some extent be compensated for through the efforts 

of an on-site staff person with the system skills to help colleagues. 

School-level leadership appears to be an important ingredient in moving teachers from forced 

compliance to avid use. Principals can make the incorporation of data a standard for plans 

developed by school staff and for presentation of options for consideration. Principals can also 

make sure that joint reflection on data is on the agenda for regularly scheduled meetings of 

groups of teachers. 

Key challenges to school data use are developing teachers’ interest in student data and 

fostering their comfort in looking at data with colleagues so that they can seek help from those 

colleagues whose students are profiting most from instruction. Having a set of common 

assessments that all of a school’s teachers teaching the same content give to their students at 

about the same time provides one of the essential conditions for teachers to be able to sit down 

and share both their data and their teaching strategies. We found this practice among at least 

some of the teachers in close to half of the case study schools. 

5. Conclusion and Policy Recommendations 

The survey and case study findings presented in this report suggest that educational data 

systems are improving, but the combination of technical and human resources available in 

schools has yet to constitute a capacity for routine use of data to support instructional decisions. 

Three broad implementation themes with respect to improving local use of educational data 

systems come out of this work: 

• A good data system is not enough: Use of data to inform instruction requires 

leadership and systemic realignment. Data-driven decision making requires leader 

initiative to align curriculum and assessment practices, professional development, and 

data systems. 

• Efforts to promote data-driven decision making are more successful when they are 

not treated as a separate innovation but are implemented as part of district, school and 

teacher efforts to provide better instruction for every student. 

• Timely, credible, interim assessment data is key to motivating teachers to use data 

systems. Teachers’ inclination to use data systems is affected by the nature of the system 

interface and the amount of training and support they receive, but these supports are 

largely futile if the system itself does not contain information that teachers consider 

relevant to their practice. 

The more detailed study findings, organized by the study’s conceptual framework, are 

summarized below, followed by presentation of a set of policy recommendations derived from 

those findings. 

Supports Needed for Data-driven Decision Making 

The conceptual framework that guided instrument development and data analysis for this 

project emphasizes practices that support data use for the purpose of informing instruction, as 

described in Chapter 1. The NETTS teacher survey findings described in Chapter 4 are 

consistent with that conceptual framework: Teachers who report having more of the supports 

enumerated in the framework also report making more frequent use of data. Major findings with 

respect to each component of the framework are presented below. 

Data Systems 

At this point, nearly all school districts have some kind of electronic student data system, 

according to responses to the district survey. But districts do not necessarily have systems that 

provide the kinds of information needed to guide instruction. District systems are more likely to 

have electronic data on student demographics, attendance, grades and state test scores than they 

are to have data from interim assessments that can be used to improve instruction during the 

course of the school year. Districts are making progress in this area, however. Although only 41 

percent of districts reported having had data on district-administered assessments for three years 

or more, 72 percent of districts said that they now have such data in electronic form (suggesting 

that 31 percent of districts began putting interim assessment data into an electronic system within 

the last two years). 

83
The majority of districts lack data systems adequate to support routine evaluation of 

district practices and decisions. An area in which districts are making less rapid progress than 

they are with interim assessments is in obtaining the capability to combine data from different 

types of systems in order to link assessment results with instructional resources. Just 42 percent 

of districts have systems that can generate reports showing student performance linked to 

participation in specific instructional programs. Only 27 percent of districts have systems that 

will support queries linking student performance to school finance data. 

The case studies demonstrated that acquiring and maintaining a student data system is 

neither simple nor a one-time event. Among the 12 case study districts, nominated for their 

leadership in the area of data use, all had multiple data systems (with a range from three to seven 

systems). A number of the districts described cumbersome processes for moving data between 

the various systems being used, and system upgrades and changes were common. In fact, about 

half of the case study districts were in the process of adding new system components. Several of 

the data systems were described as difficult to use, a condition that prompted some districts to 

decide that they would not ask teachers to pull down system data for themselves. Standard 

reports created by district staff, the school’s principal, or a data coach were the strategy of choice 

in such cases. 

Leadership for Educational Improvement and Use of Data 

Leadership for data use is important at both the district and the school levels. District 

superintendents set the tone for the amount of priority to be given to data-driven decision 

making. They decide both about resources (such as system acquisition and professional 

development on system and data use) and about processes that will be used within district offices 

and schools. A number of the case study districts had instituted a school improvement plan 

development process that included looking at data to set goals and quantitative targets for 

improvement. At the school level, some principals similarly set up processes whereby teachers 

were expected to use data to set targets for their grade’s or their individual class’ achievement. 

At both the district and the school levels, leaders also set general expectations about the role of 

data. Strong leadership for data use includes making it clear that any proposal from a staff 

member needs to be backed by data and will be evaluated on the basis of data. 

Tools for Generating Actionable Data 

Teachers commonly describe the state test scores they receive in the fall for the students their 

grade level had the prior spring as “dead on arrival.” The teachers now have a new batch of 

students so the specifics of the performance of a student they had the prior year is no longer their 

central concern. Aggregate data from the prior year does provide some insight into areas where 

students in the grade level in which they work typically do well and poorly, and teachers and 

school leaders do use this information to look for curriculum areas needing more emphasis. But 

state tests necessarily give little coverage to any particular content standard because of the sheer 

number of standards they are trying to assess. Thus, many content standards are not covered on 

the state test and in some cases student performance by standard is not available to teachers. 

To get performance data for their current students at a level of detail that informs what they 

teach next week or next month, teachers need interim assessments that are scored and returned to 

them as quickly as possible. Eleven of the 12 case study districts had interim assessment systems 

84
in place, at least for a subset of subject areas and grade levels. In some cases, the assessments 

could be taken online so that teachers got results immediately. In other cases, districts had 

provided schools with scanners, and test results were turned around within a matter of days. 

Based on case study data, interim assessments for generating actionable data were among 

the most powerful strategies districts had for getting teachers to use data.

Social Structures and Time Set Aside for Analyzing and Interpreting Data 

On the district survey, 92 percent of respondents identified lack of time as a barrier to the use 

of data for decision making. Teachers too expressed frustration with this lack of time for data 

use: on the NETTS teacher survey only 12 percent of teachers with access to an electronic 

student data system said that there was time set aside for them to examine student data and use it 

to guide decisions about practice. In contrast to the nationally representative data from the 

NETTS survey, at the case study schools in districts with a reputation for data use, teachers in 30 

of 36 schools talked about regularly scheduled times, such as planning and grade-level meetings, 

when they could examine and discuss student data. Hence, setting time aside within the 

regular work week for the examination of student data and the development of 

instructional decision informed by data appears to be a key characteristic of schools and 

districts that are leaders in data-driven decision-making. 

Professional Development and Technical Support for Data Interpretation 

Both district and teacher surveys indicate that most districts have provided teachers with 

professional development on data-driven decision making, and the case study districts were no 

exception. A distinction can be made, however, between training that focuses on system use and 

training that focuses on how to interpret data for the purpose of making instructional decisions. 

District-provided training in the case study sample was often of the former type and was usually 

quite limited in duration. However, many of the case study districts that did not provide teachers 

with formal professional development or technical support in the area of data interpretation did 

make support available to teachers in the form of embedded professional development as 

teachers worked with an instructional coach or with colleagues on the collaborative interpretation 

of data. Embedded, informal professional development provided as part of coaching or 

teacher collaboration around instruction appeared to offer the strongest support for 

teachers’ use of data in deciding what and how to teach. 

Districts themselves are seeking guidance on how to link data to instructional decisions. On 

the district survey, the three areas in which districts perceived the greatest need of examples of 

good practice were adapting instruction to meet students’ individual needs, examining data to 

identify which practices work best for which students, and developing curriculum-embedded 

formative assessments.

85
Tools for Acting on Data 

Once teachers have examined data for their students and drawn inferences about instructional 

needs, the next step is a plan of action that includes both an instructional approach and 

instructional materials. Teachers vary in terms of their knowledge of alternative approaches for 

teaching specific concepts and skills. Some districts and states have data systems that include 

instructional resources and lesson plans geared to the same content standards as the student 

assessments. On the district survey, 60 percent of districts reported providing this kind of online 

resource. In these cases, teachers have easily accessible resources to help them implement 

instruction based on data. In the great majority of case study districts, school staff reported that 

such instructional resources linked to assessment results were not provided online, but in many 

case study schools an instructional coach was available who could model or describe 

instructional strategies and provide teachers with appropriate materials. 

Data-driven Decision Making as a Systemic Endeavor 

Beyond providing support for the study’s conceptual framework, the qualitative data 

collected through site visits suggest that using data in a continuous improvement process requires 

major systemic change. Systems change or systemic reform, as defined by Smith and O’Day 

(1991), is not the same as scaling up an innovation (getting more schools to adopt a program). It 

requires coordination and alignment of all of the elements of the larger education system. In the 

case of data-driven decision making, systems change means that district offices that have been 

accustomed to functioning as separate silos—such as Information Technology, Professional 

Development Services, Curriculum and Instruction, and Assessment and Evaluation—need to 

coordinate their efforts. This challenge plays out both within district offices and within the 

schools that need to implement policies and requirements coming from various district offices. If 

districts really want to see data-driven decision making within their schools, they need to make 

sure that the student data system made available to teachers includes timely data that teachers 

view as relevant to instructional decision making. 

Within schools, processes of school improvement and teacher collaboration and planning 

need to be integrated with data use. It is within the schoolhouse that the important link between 

data and instructional decisions gets made (Herman et al. 2008). Making this connection 

requires both expertise around data and assessment and knowledge of instructional strategies 

within the area being taught. The use of data in decision making cannot have a positive 

impact on instruction without a linkage to effective instructional practices.

The case studies revealed more detail about the interdependencies among these different 

aspects of school and district functioning as they relate to the use of data for making instructional 

decisions. The case studies illustrated the benefits of aligning and articulating goals within the 

district. Districts that merged their goals for improving instruction in specific areas (such as 

early literacy or mathematics) with the goal of promoting teacher use of data were able to use 

school-based instructional coaches to promote data use. These coaches were in a position 

both to encourage teachers to collect and use interim assessment data and to help teachers 

interpret student assessment results—making the important link between student data and 

instructional decisions. 

86
The case studies suggested that districts wishing to promote data use among teachers would 

do well to make sure that they have interim assessment tools and practices that foster data use. 

As noted above, many of the case study districts set time aside for teachers to examine data and 

were in the process of putting districtwide interim assessment and reporting systems into place. 

In some cases, though, district policies are contradictory: they require teachers to administer 

interim assessments, but pacing requirements prohibit teachers from going back to reteach 

content that the students have not yet mastered. In still more cases, there is no district pacing 

requirement per se, but teachers feel pressure to cover the great breadth of material in the district 

curriculum. These are cases in which different parts of the district system are not well-aligned, 

and such tensions between competing priorities should be recognized and addressed by district 

leadership. 

Recommendations 

These findings are the basis for researchers’ recommendations for actions and policies at the 

school district, state, and national levels. 

Recommendations for Schools 

• Set clear expectations around the use of student data as the basis for decisions. Case 

studies illustrated the important role that school leaders play in modeling the use of data 

and in developing school practices in which teachers are expected to use data to guide 

their decisions. 

• Integrate collaborative exploration of data into existing structures for joint teacher 

planning and reflection on teaching. Lack of time to examine and reflect on data is the 

greatest barrier to data-driven decision making according to both teacher and district 

survey respondents. Given the constraints of educational budgets, funding for additional 

time for teachers to engage in data use activities is likely to remain relatively unusual. 

But the case study schools illustrate how data use can become an integral part of teacher 

planning and collaboration time that is already funded in many districts, provided there is 

district and school leadership for doing so. 

• Provide a safe environment for teacher examination of their students’ performance.

Reports from case study schools suggest that teachers benefit from opportunities to 

examine student data with their colleagues, but that they only want to do so if they feel 

confident that they will not be opening themselves up to harsh judgments. Small groups 

of teachers who typically work together as part of a grade-level, department, or project 

team appear to be the most suitable for collaborative inspection of data. Keeping data 

reflection activities separate from performance management activities (which could affect 

salary or job status) helps to create a climate of trust. 

• Support teachers in making the link between data and alternative instructional strategies.

Just having student data is not sufficient if teachers do not have ideas about how to teach 

differently based on student performance. Grouping students by their performance level 

and giving additional emphasis to content on which a majority of students did poorly are 

responses to data considered by most teachers. It was less common to find teachers who 

had ideas about different ways to teach content depending on student assessment results. 

Instructional coaches, however, especially in the area of early literacy, were able to bring 

this kind of insight to teachers in a number of the case study schools. Coaching that 

87
combines training in research-based instructional strategies and the use of student data as 

a basis for selecting and evaluating instructional strategies should be encouraged. 

Recommendations for Districts 

• Think of data-driven decision making as an ongoing process rather than a one-time event 

centered on the acquisition of a data system. The case study districts considered by 

researchers to be most advanced in their data-use practices had been actively engaged in 

this area for an average of more than eight years. The case studies suggest that districts 

will get more out of their investments in electronic data systems if they think about datadriven decision making as a systemwide innovation and develop a long-term strategy for 

its implementation as part of a continuous improvement process. This perspective entails 

treating professional development around data use as an ongoing effort rather than a onetime event. 

• Model decision making based on data and present decision-relevant data when 

announcing new policies. Implementing a culture that strives for continuous 

improvement and uses data to drive that improvement is a major undertaking. It is 

unlikely that many of a district’s schools will develop this kind of culture if the district 

itself is not an exemplar. The shift to a data use culture within the district will require 

obtaining interoperability between data systems for operations such as human resources 

and finance and those for student acheivement. 

• Train principals in how to integrate the use of data into school improvement planning 

and promote their teachers’ use of data for making instructional decisions. Case studies 

underscore the importance of the principal or other school leader in setting up the 

expectation for teachers to use data and in helping them do so. The ability to lead their 

schools in the use of continuous improvement processes informed by data should be a 

hiring criterion for school leaders. In addition, school leaders should receive ongoing 

training in how to connect data to school improvement and instructional decision making 

and in how to motivate their staff to engage in these practices. 

• Integrate the use of data-driven decision-making practices with district initiatives for 

improving instruction in specific areas; when possible fund school-based coaches who 

support teachers in both data interpretation and linking data to instructional decisions. 

Data use is a means to an end rather than an end in itself. Integrating the use of data with 

core school improvement and reform activities gives school staff motivation for looking 

at data and means that resources for improvement and data use can be pooled. 

• Support time within the work week for teachers to meet with colleagues for planning, 

informal professional development, and data use. If the use of data is to become an 

accepted part of teachers’ professional practice, it needs to be built into their regular 

schedule. As noted above, this does not mean that periods designated solely for data use 

need to be instituted. Case study schools illustrated how discussions around data can 

become an accepted part of teacher professional development, planning, and 

collaboration time. When districts fund noninstructional time so teachers can engage in 

these activities, they should hold school leaders responsible for making sure that the time 

is well used. 

88
• Make sure that the district has a data system that gives teachers data that is both timely 

and relevant to their instructional decisions. This entails developing or acquiring good 

interim assessment tools and securing an infrastructure that supports the rapid turnaround 

of assessment results to teachers. A number of case study districts involved teachers in 

developing interim or end-of-course assessments for inclusion in the data systems. The 

use of online assessment systems or school-based scanners can ensure that teachers can 

get assessment results within minutes or days rather than weeks or months. 

• Provide resources and construct policies so that teachers have access to data relevant to 

the students they are teaching when and where they want it. During the two years of the 

study, case study districts increased their teachers’ direct access to student data. The 

provision of data system access from home sends an important message to teachers about 

the centrality of student data to the profession of teaching. 

Recommendations for State and National Policy 

• Complement efforts to improve state data systems with investments to help districts 

improve both their data systems and their organizational supports for using data to 

improve instruction. In case study interviews, district and school staff explain that they 

do not make extensive use of state data systems because the information in those systems 

consists primarily of student scores on state tests given once a year. If the goal is to 

improve instructional practices within schools and classrooms, state and national 

policymakers should work toward improving district data systems, both in terms of the 

instructional relevance of the data they contain and in terms of system functionalities. 

Analyses reported in Chapter 2 showed that data systems in small districts have fewer 

query capabilities than those in larger districts, suggesting that small districts should be a 

special focus for state and federal assistance. Equally important, state and national 

policymakers should support districts in developing the organizational support systems 

that surround use of data by school staff. 

• Improve the turnaround time for state assessment data so that schools receive student 

results in time to inform academic-year planning. State assessment data from the prior 

academic year is useful to school staff mainly at the start of the school year when plans 

for curriculum coverage and school improvement are being made. State data may be used 

annually in school improvement planning, but this once-a-year activity is hindered when 

test data are unavailable until October or November—after students have been placed, 

summer professional development completed, and curriculum planned. School staff 

sessions for academic planning are typically held in August, prior to school opening. 

Local use of state test data and the systems that contain it will remain limited if states and 

test vendors cannot get the data back to schools in time to inform their planning activities. 

• Promote linkages between local data systems that contain interim assessment data and 

state systems with instructional resources geared to standards. Many states have 

assembled collections of digitized resources for planning and implementing instruction 

around their state standards. In some cases, these collections are integrated with state data 

systems, but teachers are much more likely to use district data systems than state data 

systems. It should be possible for teachers using a district data system to examine 

89
students’ performance on a specific standard to be just a click or two away from statecollected instructional resources for that same standard. 

• Encourage districts to invest in developing data literacy among district staff in all 

departments. An essential component of the systemic alignment and integration called for 

above is the development of knowledge of the district’s data resources and competencies 

in using data to identify needs and evaluate practices among district leaders in all 

departments. 

• Encourage or require school administrator preparation programs to incorporate 

assessment and data literacy concepts to foster continuous improvement activities 

informed by data as elements of school leadership training. Becoming a data-informed 

instructional leader requires an understanding of assessment concepts and basic data 

analysis concepts. Fostering these skills and their exercise among school staff requires 

skill in organizational change. These competencies should be a central part of 

administrator preparation programs. Such programs should include exposure to good 

models of school and classroom-level data use. 

• Encourage or require teacher preparation programs to incorporate assessment concepts 

and the use of data for instructional decision making into their teaching methods courses

(science methods, language arts methods, and so on). Teachers’ data-driven decisionmaking skills are likely to require development and support if teachers are to use data 

effectively in their practice (U.S. Department of Education, in press). Teacher preparation 

programs should incorporate modeling of good use of data for instructional decision 

making. In the short-term, specially designated coaches can help with ongoing 

professional development and technical assistance, but longer term, assessment 

interpretation and data use skills should be covered more extensively as part of teacher 

preparation. 

• Provide districts with good examples of practices that support the development of a datause culture within schools. Responses to the district survey indicate that district staff 

believe that they need exemplary models of how to connect student data to instructional 

decisions. Districts will have a hard time supporting staff at their schools in this area if 

they themselves lack a concrete understanding of how to do it. The areas in which district 

staff indicated having the greatest need for good models were examining student data to 

determine which practices work best for which students, adapting instructional practices 

to meet students’ individual needs, and developing curriculum-embedded formative 

assessments. 

In the years since the reauthorization of the Elementary and Secondary Education Act in 

2002, both federal policy and private advocacy partnerships (Data Quality Campaign 2006) have 

put heavy emphasis on the improvement of states’ student data systems. Efforts began in 2005 to 

support states in improving the quality of the data in their systems and acquiring or developing 

systems capable of tracking longitudinal changes in achievement for individual students and 

student subgroups. The Institute of Education Sciences (IES) began awarding three- to five-year 

grants to states for developing or expanding longitudinal data systems in FY2006. 

States have made extensive progress in improving their systems in these areas, and now are 

being asked to make their systems useful not just for accountability purposes but also for 

90
91

continuous improvement (Data Quality Campaign 2009b). The FY2009 IES call for state 

applications for student longitudinal data system grants, for example, stipulates, “The system 

should provide data that can be used in education decision-making at multiple levels, from policy 

to classroom instruction” (Institute of Education Sciences June 2008, p. 3). 

Findings reported here make it clear that at the time that the data for this study were 

collected (2005–08), school and district staff were making little use of state systems to support 

their decision making. But it should be kept in mind that efforts to make state systems useful for 

such purposes were in their infancy. As the first of these efforts to make state systems useful to 

practitioners mature, it will be important to reexamine local data use practices in states that have 

received federal funding for data system improvement, with a special focus on small districts 

with limited local capacity for data system acquisition and support, in order to judge whether 

state systems and associated supports can fill local needs. 

The findings reported here make it clear that the data systems themselves, the assessment 

data that populate district data systems, and school and district practices around data use are all 

changing rapidly. This field warrants continued study to identify emerging best practices and test 

the causal linkage between data use practices and improvements in student learning. 

References 

Brunner, C., C. Fasca, J. Heinze, M. Honey, D. Light, E. Mandinach, and D. Wexler. (2005). 

Linking data and learning: The Grow Network study. Journal of Education for Students 

Placed At Risk, 10, no. 3: 241–267. 

Bryk, A. S., and B. Schneider. (2002). Trust in schools: A core resource for improvement. New 

York: Russell Sage Foundation. 

Coburn, C. E., J. Toure, and M. Yamashita. (2009). Evidence, interpretation, and persuasion: 

Instructional decision making at the district central office. Teachers College Record. 

Coburn, C. E., and J. E. Talbert. (2006). Conceptions of evidence-based practice in school 

districts: Mapping the terrain. American Journal of Education, 112, no. 4: 469–495. 

Confrey, J., and K. M. Makar. (2005). Critiquing and improving the use of data from high-stakes 

tests with the aid of dynamic statistics software. In Scaling up success: Lessons learned from 

technology-based educational improvement, eds. C. Dede, J. P. Honan, and L. C. Peters, 

198–226. San Francisco, Calif: Jossey-Bass. 

Copland, M. A. (2003). Leadership of inquiry: Building and sustaining capacity for school 

improvement. Educational Evaluation and Policy Analysis, 25, no. 4: 375–395. 

Consortium for School Networking. (2004). Vision to know and do: The power of data as a tool 

in educational decision making. Washington, D.C.: Author. 

Data Quality Campaign. (April 2009a). Data-driven districts: Building the culture and capacity to 

improve student achievement. DQC Brief. 

http://www.dataqualitycampaign.org/files/DQCbrief_FINAL-lowres_2_.pdf (accessed 

Aug. 4, 2009). 

Data Quality Campaign. (April 2009b). DQC brings together education leaders to urge states to 

use data systems for continuous improvement. DQC Newsletter, 3, Issue 7. 

http://www.dataqualitycampaign.org/files/DQCNewsletterApr09.pdf (accessed May 1, 

2009). 

Data Quality Campaign. (2008). Results of 2008 NCES survey of state P–12 data collection 

issues related to longitudinal analysis. http://www.dataqualitycampaign.org/survey_results/ 

(accessed Nov. 18, 2008). 

Datnow, A., V. Park, and P. Wholstetter. (2007). Achieving with data: How high-performing 

school systems use data to improve instruction for elementary students. Los Angeles, Calif: 

University of Southern California, Center on Educational Governance. 

Elmore, R. (2004). School reform: From the inside out. Cambridge, Mass.: Harvard Education 

Press. 

Feldman, J., and R. Tung. (2001). Using data-based inquiry and decision-making to improve 

instruction. ERS Spectrum, 19, no. 3: 10–19. 

Fuchs, D., and L. S. Fuchs. (January/February/March 2006). Introduction to response to 

intervention: What, why, and how valid is it? Reading Research Quarterly. 

http://www.reading.org/Library/Retrieve.cfm?D=10.1598/RRQ.41.1.4&F=RRQ-41-1-

Fuchs.html (accessed Jan. 8, 2009). 

93
Garet, M. S., A. C. Porter, L. Desimone, B. F. Birman, and K. S. Yoon. (2001). What makes 

professional development effective? Results from a national sample of teachers. American 

Educational Research Journal, 38, no. 4: 915-945. 

Herman, R., P. Dawson, T. Dee, J. Greene, R. Maynard, S. Redding, and M. Darwin. (2008). 

Turning around chronically low-performing schools: A practice guide (NCEE #2008-4020). 

Washington, D.C.: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences. http://ies.ed.gov/ncee/wwc/practiceguides (accessed June 12, 

2008). 

Hoffman, J. (2007). Numbers and types of public elementary and secondary education agencies 

from the Common Core of Data: School year 2005–06 (NCES 2007–353). U.S. Department 

of Education. Washington, D.C.: National Center for Education Statistics. 

http://neces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007353 (accessed Aug. 14, 2008). 

Honig, M. I., and C. E. Coburn. (Winter 2005). When districts use evidence to improve 

instruction: What do we know and where do we go from here? Voices in Urban Education, 6. 

http://www.annenberinstitute.org/ VUE/winter05/Honig.html (accessed Nov. 13, 2007). 

Institute of Education Sciences. (June 26, 2008). Statewide Longitudinal Data System Grants.

CFDA No. 84.372A. http://nces.ed.gov/Programs/SLDS/grant_information.asp 

Knapp, M. S., J. A. Swinnerton, M. A. Copland, and J. Monpas-Huber. (October 2006). Datainformed leadership in education. Seattle, Wash.: Center for the Study of Teaching and 

Learning. 

Koretz, S. (2008). Further steps toward the development of an accountability-oriented science of 

measurement. In The future of test-based educational accountability, eds. K. E. Ryan and L. 

A. Shepard, 71–91. New York: Routledge, Taylor and Francis. 

Loucks-Horsley, S., N. Love, K. E. Stiles, P. W. Hewson, and S. Mundry. (2003). Designing 

professional development for teachers of science and mathematics. Newbury Park, Calif.: 

Corwin Press. 

Love, N. (2002). Using data/getting results: A practical guide for school improvement in 

mathematics and science. Norwood, Mass.: Christopher-Gordon Publishers, Inc. 

Mandinach, E. B., L. Rivas, D. Light, and C. Heinze. (April 2006). The impact of data-driven 

decision making tools on educational practice: A systems analysis of six school districts. 

Paper presented at the annual meeting of the American Educational Research Association, 

San Francisco. 

http://cct.edc.org/reports.asp?numStartCount=6&numTopicID=25&numPubTypeCatID=2 

(accessed Aug. 3, 2009). 

Mandinach, E. B., M. Honey, D. Light, J. Heinze, and L. Rivas. (2005). Technology-based tools 

that facilitate data-driven decision making. In Toward sustainable and scalable educational 

innovations informed by the learning sciences, eds. C. K. Looi, D. Jonassen, and M. Ideda, 

267–274. Amsterdam: IOS Press. 

Marsh, J. A., J. F. Pane, and L. S. Hamilton, (2006). Making sense of data-driven decision 

making in education. Santa Monica, Calif.: RAND. 

94
Massell, D., and M. E. Goertz, (2002). District strategies for building instructional capacity. In 

School districts and instructional renewal, eds. A. M. Hightower, M. S. Knapp, J. A. Marsh, 

and M. W. McLaughlin, 43–60. New York: Teachers College Press. 

Palaich, R. M., D. G. Good, and A. van der Ploeg, (2004). State education data systems that 

increase learning and improve accountability. Policy Issues, No. 16. Naperville, Ill.: Learning 

Point Associates. 

Pellegrino, J. W. (in press). Technology and formative assessment. In International encyclopedia 

of education, 3rd ed., eds. E. Baker, B. McGaw, and P. Peterson. Oxford, U.K.: Elsevier. 

Smith, M. S., and J. O’Day. (1991). Systemic school reform. In The politics of curriculum and 

testing, eds. S. H. Fuhrman and B. Malen, 233–267. New York: Falmer. 

Snipes, J., F. Doolittle, and C. Herlihy. (2002). Foundations for success: Case studies of how 

urban school systems improve student achievement. Washington, D.C.: Council of Great 

City Schools. 

Supovitz, J. A., and V. Klein. (November 2003). Mapping a course for improved student 

learning: How innovative schools systems actually use student performance data to guide 

improvement. Philadelphia, Pa.: Consortium for Policy Research in Education, University of 

Pennsylvania. 

Thorn, C. A. (2002). Data use in the classroom: The challenges of implementing data-based 

decisionmaking at the school level. Paper presented at the annual meeting of the American 

Educational Research Association, New Orleans. 

Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement, 

International Journal of Public Opinion Research, 15, no. 1: 3–7. 

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (in 

press). Supporting teachers’ ability to use data to inform instruction. Washington, D.C.: 

Author. 

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2009). 

Implementing data-informed decision making in school —Teacher access, supports and use. 

Washington, D.C.: Author. 

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2007). 

Teachers’ use of student data systems to improve instruction 2005 to 2007. Washington, 

D.C.: Author. 

Wayman, J. C. (2005). Involving teachers in data-driven decision making: Using computer data 

systems to support teacher inquiry and reflection. Journal of Education for Students Placed 

At Risk, 10, no. 3: 295–308. 

Wayman, J. C., and V. Cho, (2009). Preparing educators to effectively use student data systems. 

In Handbook of data-based decision making in education, ed. T. J. Kowalski and T. J. Lasley 

II, 89–104. New York, N.Y.: Routledge. 

Wayman, J. C., V. Cho, and M. T. Johnson. (2007). The data-informed district: A district-wide 

evaluation of data use in the Natrona County School District. Austin, Texas: The University 

of Texas. 

95
96

Wayman, J. C., and S. Stringfield, (2006). Technology-supported involvement of entire faculties 

in examination of student data for instructional improvement. American Journal of 

Education, 112, no. 4: 549–571. 

Wayman, J. C., S. Stringfield, and M. Yakimoski. (2004). Software enabling school 

improvement through analysis of student data. CRESPAR Technical Report No. 67. 

Baltimore, Md.: Johns Hopkins University. 

Wilhelm, J. (2007). Engaging readers and writers with inquiry. New York: Scholastic. 

Young, V. M. (2006). Teachers’ use of data: Loose coupling, agenda setting, and team norms. 

American Journal of Education, 112, no. 4, 521–548. 

Zavadsky. H. (2009). Building data-driven district systems: Examples from three award-winning 

urban systems. In Handbook of data-based decision making in education, eds. T. J. Kowalski 

and T. J. Lasley II, 89–104. New York, N.Y.: Routledge. 
Appendix A 

Methods 

A-1

District Survey Sample 

The National Technology Activities (NTA) district survey was administered between 

October 2007 and February 2008. Of the 529 districts in the final sample, 427 responded for a 

response rate of 81 percent. The response rate within each cell of the sampling frame was 72 

percent or higher. To conduct analyses with reasonable statistical precision, a six-cell sampling 

frame stratified by district size and poverty rate was created. The process for identifying the 

sampling strata is described below. 

The national population of districts was sorted into three size categories such that each 

category contained districts serving approximately one-third of the nation’s public school 

students, based on enrollment data provided by the National Center for Education Statistics’ 

Common Core of Data (CCD): 

• Large (estimated enrollment 25,800 or greater). These are either districts in large urban 

centers or large county systems, which typically are organizationally complex and often 

are broken up into subdistricts. 

• Medium (estimated enrollment from 5,444 to 25,799). These are districts set in small to 

medium-sized cities or are large county systems. They also are organizationally complex, 

but these systems tend to be centralized. 

• Small (estimated enrollment from 300 to 5,443). The small district group typically 

includes suburban districts, districts in large rural towns, small county systems, and small 

rural districts. These districts tend to have more limited organizational capacity. 

Districts with 299 or fewer students were excluded from the study. Such districts account for 

approximately only 0.1 percent of all public school students. The distribution of districts among 

the three size strata and the proportion of public school students accounted for by each stratum 

are displayed in Exhibit A-1. The proportion of districts among the three size strata in the district 

sample (excluding districts with 299 or fewer students) are large (2.2 percent), medium 

(13.5 percent), and small (84.3 percent). 

Exhibit A-1 

Distribution of Districts and Student Population, by District Size 

Enrollment Size Category 

Number of 

Districts 

Percent of 

Districts 

Number of 

Students 

(000s) 

Percent of 

Students 

Large (>25,800) 249 2.2 15,834 33.3 

Medium (5,444–25,799) 1,497 13.5 15,843 33.3 

Small (300–5,443) 9,378 84.3 15,853 33.3 

Very small (299 or less) 2,956 21.0 478 0.1 

TOTAL 14,080 100.0 47,530 100.0 

Source: Based on 2004–05 NCES Common Core of Data (CCD). 

A-3
As a measure of district poverty, the percentage of children ages 5 to 17 who are living in 

poverty, as reported by the U.S. Census Bureau and applied to districts by the National Center 

for Education Statistics, was used. The distribution of districts among strata and the proportion of 

students accounted for by each stratum are displayed in Exhibit A-2. 

Exhibit A-2 

Distribution of Districts and Student Population, by District Poverty Rate 

District 

Poverty Rate 

Number of 

Districts 

Percent of 

Districts 

Number of 

Students (000s) 

Percent of 

Students 

Lower(≤20%) 8,555 76.9 33,370 70.2 

Higher (>20%) 2,569 23.1 14,160 29.8 

TOTAL 11,124 100.0 47,530 100.0 

Note: Distribution excludes districts with 299 or fewer students. 

Source: Based on data from the 2003 U.S. Census for the percentage of children ages 5 to 17 who 

are living in poverty and applied to districts by NCES. 

A sample of 529 districts was drawn with the goal of obtaining approximately 500 

respondents. The two variables of district size and poverty rate were used to generate a six-cell 

grid into which the universe of districts (excluding very small districts) could be fit. From the 

population of districts in each cell, 89 districts were sampled (except in the cell representing 

large high-poverty districts in which only 84 districts were available after removing districts 

participating in case studies). Exhibit A-3 shows the strata, distribution of the number of districts 

in each stratum, and the sample size in each cell. 

A-4
Exhibit A-3 

Number of Districts in the Universe and Sample Size, by Stratum 

District Poverty Rate 

District Size 

Lower 

(≤ 20%) 

Higher 

(> 20%) Total 

Large 

No. of respondents 

(response rate) 

Sample 

Universe

77 

86.5) 

89

160 

64 

(76.2) 

84 

89 

141 

(81.5) 

173 

249

Medium 

No. of respondents 

(response rate) 

Sample 

Universe

69 

(77.5) 

89 

1,155 

80 

(89.9) 

89 

342 

149 

(83.7) 

178 

1,497 

Small 

No. of respondents 

(response rate) 

Sample 

Universe

64 

(71.9) 

89 

7,240 

73 

(82,0) 

89

1,155 

137 

(77.0) 

178 

9,378 

TOTAL 

No. of respondents 

(response rate) 

Sample 

Universe

210 

(78.6) 

267

8,555 

217 

(82.8) 

262 

2,569 

427 

(80.7) 

529 

11,124

Note: A total of 89 districts in each cell were sampled except for large, higher-poverty districts of 

which only 84 remained after eliminating case study districts. As a result, the final sample consisted 

of 529 districts. 

Case Study Sample 

A two-stage sample process was used to select the 2007–08 case study districts. First, six 

districts were selected for a second round of data collection because they had been in the process 

of implementing new data systems or activities to support the use of student data at the school 

level and warranted another visit. The second stage involved identifying an additional group of 

six districts that have been active in data-driven decision making. These districts were drawn 

from the pool of districts that remained after the initial selection of 10 districts in 2006, 

supplemented by additional districts identified as active data users. These included Broad prize 

winners and nominees, and selected districts participating in focus groups at the U.S. Department 

of Education to discuss issues related to education technology (i.e., districts using data to support 

instructional decision making). 

A-5
For both groups of districts, the research team worked with district administrators to identify 

three appropriate schools, based on the following criteria:34

• One elementary or middle school that the district considers high in its data use 

practices. 

• One school that has shown improvement in its use of data to guide instruction 

(emerging). 

• One school that is typical of the district with respect to use of data systems. 

Researchers asked the district to recommend, to the extent possible, three schools serving 

demographically similar students at the same grade level (either elementary or middle school). 

Within each school, the principal, an instructional or data coach (if applicable), and six teachers 

were interviewed. 

The demographic data for the 12 districts and 36 schools in the 2007–08 site visit sample are 

shown in Exhibit A-4. For each district visited, respondents included key staff involved in the 

district’s data-informed decision-making activities (e.g., chief information officers, directors of 

curriculum and instruction, directors of research and evaluation, directors of accountability, 

directors of professional development). Among the districts visited for a second time, questions 

focused on the impact of changes since the first round of data collection. 

 

34 The 2007–08 case study sample includes a small district for which an exception was made to the requirement of 

having three schools from the same district (the technical work group for the study felt it was important to 

research the experiences of small districts because they serve approximately a third of public school students; 

Hoffman 2007). A third school from a neighboring district in which the sampled district was providing datadriven decision-making assistance was included. Additional information on the sample selection process for the 

case study districts and schools can be found in Appendix A. 

A-6
Exhibit A-4. Case Study Districts and School Sample in 2007–08 

District District Demographics School Type and Size 

District 4 Student enrollment = 164,295 (large) 

Percentage minority = 50% 

Percentage poverty = 20% 

No. of schools = 238 

Elementary School 1 = 732 students 

Elementary School 2 = 455 students 

Elementary School 3 = 684 students 

District 12 Student enrollment = 151,421 (large) 

Percentage minority = 63% 

Percentage poverty = 40% 

No. of schools = 101 

Middle School 1 = 2,066 students 

Middle School 2 = 1,082 students 

Middle School 3 = 1,784 students 

District 11 Student enrollment = 134,002 (large) 

Percentage minority = 47% 

Percentage poverty = 28% 

No. of schools = 132 

Middle School 1 = 1,368 students 

Middle School 2 = 1,105students 

Middle School 3 = 490 students 

District 3 Student enrollment = 132,482 (large) 

Percentage minority = 74% 

Percentage poverty = 62% 

No. of schools = 219 

Middle School 1 = 1,018 students 

Middle School 2 = 1,330 students 

Middle School 3 = 1,070 students 

District 10 Student enrollment = 90,663 (large) 

Percentage minority = 83% 

Percentage poverty = 61% 

No. of schools = 89 

Elementary School 1 = 464 students 

Elementary School 2 = 720 students 

Elementary School 3 = 919 students 

District 1 Student enrollment = 39,213 (large) 

Percentage minority = 82% 

Percentage poverty = 64% 

No. of schools = 63 

Elementary School 1 = 384 students 

Elementary School 2 = 349 students 

Middle School = 585 students 

District 16 Student enrollment = 27,211 (large) 

Percentage minority = 47% 

Percentage poverty = 38% 

No. of schools = 42 

Elementary School 1 = 548 students 

Elementary School 2 = 502 students 

Elementary School 3 = 495 students 

District 9 Student enrollment = 22,174 (medium) 

Percentage minority = 12% 

Percentage poverty = 13% 

No. of schools = 29 

Middle School 1 = 1,014 students 

Middle School 2 = 833 students 

Middle School 3 = 787 students 

District 13 Student enrollment = 11,862 (medium) 

Percentage minority = 17% 

Percentage poverty = 26% 

No. of schools = 12 

Elementary School 1 = 667 students 

Elementary School 2 = 537 students 

Elementary School 3 = 550 students 

Exhibit A-4 continues on next page 

A-7
Exhibit A-4. Case Study Districts and School Sample in 2007–08 (continued) 

District District Demographics School Type and Size 

District 7 Student enrollment = 10,780 (medium) 

Percentage minority = 71% 

Percentage poverty = 62% 

No. of schools = 24 

Elementary School 1 = 355 students 

Elementary School 2 = 430 students 

Elementary School 3 = 339 students 

District 5 Student enrollment = 5,599 (medium) 

Percentage minority = 64% 

Percentage poverty = 43% 

No. of schools = 14 

Elementary School 1 = 365 students 

Elementary School 2 = 260 students 

Elementary School 3 = 399 students 

District 15* Student enrollment = 1,275 (small) 

Percentage minority = 1% 

Percentage poverty = 14% 

No. of schools = 3 

Elementary School 1 = 568 students 

Elementary School 2 = 316 students 

Middle School = 308 students 

Notes: Numbers have been used to label districts and schools for confidentiality reasons. To support 

comparisons of data, the same number for a district is used throughout the report (for those districts 

visited a second time, the number corresponds to that used in the interim report). District size 

categories are based on those for the district survey sample. The asterisk denotes that one of the 

elementary schools included in the school sample is from a neighboring district (enrollment of 681) 

because this case study district had only one elementary and one middle school. 

Teachers to be interviewed were nested within the purposive sample of schools. Even though 

schools were selected because they used student data systems in instructional decision making, 

variability was expected across teachers with respect to how they were using data for 

instructional decision making. Such variability is typical for educational reforms and was found 

for data-driven decision making in earlier school case studies (Marsh, Pane, and Hamilton 2006). 

Therefore, project staff requested that the principal of each case study school nominate three 

active practitioners of data-driven decision making and three teachers who represented average 

users. In this way, the study expected to capture “best practices” within the school but still 

maintain a realistic perspective with regard to the pervasiveness of those practices.35

NETTS Survey Data 

As part of the National Educational Technology Trends Study (NETTS) 1,039 district 

technology directors were surveyed during spring 2007, and 2,509 teachers were surveyed in 

spring 2007. The teachers were clustered in schools sampled from districts participating in the 

NETTS district survey.36 The NETTS district sample of 1,039 districts was nationally 

representative with respect to poverty status, student enrollments, and location (urban or rural 

status). The 60 largest urban school districts across the country were selected with certainty (i.e., 

included in the sample from the outset). Districts composed entirely of special education schools 

and vocational-technical schools, as well as independent charter schools that are their own 

 

35 It is quite possible that the teachers whom principals described as typical in terms of data use were in fact better 

than average for the school. When this potential bias is considered along with the fact that the study conducted 

case studies in districts considered leaders in the instructional use of data, the reader should be aware that the 

teachers’ understanding of data described herein is likely to be better than that of average teachers. 36 Districts were sampled from among the 12,483 districts that received federal Enhancing Education Through 

Technology (EETT) funds in 2003, as well as an additional 2,239 districts that had not received EETT funds. 

A-8
A-9

districts, were excluded from the district sampling frame because of their dissimilarity to 

“typical” districts. The teacher sample was created by drawing a probability sample of 975 

schools from respondents to the district survey, stratified by school type (elementary or 

secondary), and poverty level (high or low). Schools were randomly sampled in proportion to the 

number of teachers and in inverse proportion to district size to produce a sample of schools 

whose selection probabilities were roughly independent of the size of their district’s enrollment. 

Higher-poverty schools were oversampled (233 of the 975 schools) to obtain more precise 

data about their technology use. For schools, “higher poverty” was defined as above a specified 

cutoff in terms of the percentage of students who were eligible for the free or reduced-price 

lunch program. The dividing line between higher-poverty and lower-poverty schools was 

selected to ensure that for each school type (elementary, middle, or high school), there would be 

the same number of teachers in the higher-poverty and the lower-poverty groups, as reported in 

the National Center for Education Statistics Common Core of Data (CCD). Elementary schools 

with 29.7 percent of their students eligible for free or reduced-price lunches were classified as 

higher poverty. For middle and high schools, the poverty thresholds were 24.3 percent and 15.9 

percent, respectively. 

NETTS researchers obtained teacher rosters for the 975 schools within the districts selected 

for the NETTS district survey. To be eligible for the teacher sample, a teacher had to be teaching 

at the same school in the school year prior to survey administration (i.e., teachers new to the 

school were excluded). Teachers who did not teach core academic subjects also were omitted 

from the sample. Targets of four teachers from each of the schools were randomly selected for 

the teacher sample. The final teacher sample in 2007 consisted of 1,779 teachers from 865 

schools. 

Response rates were 94 percent for the district survey in 2007 and 85 percent for the teacher 

survey. Sampling weights were applied to the teacher data to obtain nationally representative 

estimates. The district and teacher surveys were initially administered during the 2004-05 school 

year to the same sample of districts and schools that provide the basis of comparison between 

2007 and 2005 for the teacher survey data. In fall 2005, 6,017 teachers were surveyed with a 

response rate of 82 percent; the larger sample was designed to provide robust, school-level 

estimates of technology use. For additional information on the comparisons between 2007 and 

2005 teacher survey responses, see Teachers’ Use of Student Data Systems to Improve 

Instruction 2005 to 2007 (U.S. Department of Education 2008). 
 
Appendix B 

2007–08 District Survey and Frequencies 

B-1 

Appendix C 

Terminology and Additional Data 

Note: Exhibits C-3 through C-16 provide additional information on data discussed in the text. 

Where applicable, the corresponding exhibit in the text is identified next to the exhibit number 

(e.g., Exhibit C-3 provides information on variation by district size for data presented in 

Exhibit 2-2 in Chapter 2). 

C-1
 
Exhibit C-1 

Definitions of Terms 

Term Definition

Data-driven decision 

making 

A process that integrates the analysis of educational data, typically 

stored in educational data systems, to support decisions intended to 

improve teaching and learning at the school and classroom levels. 

The practice entails regular data collection and ongoing 

implementation of improvements. Some researchers prefer the term 

“data-informed” decision making to acknowledge the role of factors 

other than data in shaping educators’ decisions. 

Data query A request for specific records from a data system. Some researchers 

prefer the term “data-informed” decision making to acknowledge the 

role of factors other than data in shaping educators’ decisions. 

Data systems Electronic information systems to assist in the organization and 

management of data. They consist of hardware and software that 

provide many different functions to users, such as storing current and 

historical data, rapidly organizing and analyzing data (e.g., examining 

relationships within data, specifying data subgroups), and developing 

presentation formats or interfaces. 

DIBELS The Dynamic Indicators of Basic Early Literacy are a set of 

standardized, individually administered measures of early literacy 

development. They are designed to be short (one minute) fluency 

measures used to regularly monitor the development of pre-reading 

and early reading skills. 

Electronic 

communication tools 

Tools to facilitate communication of information such as use of e-mail, 

Web sites, and electronic discussion groups or “message boards” that 

users from multiple locations can use to discuss issues. 

Electronic grade books Online tools to help teachers manage classroom activities (e.g., 

generating seating charts, recording grades and test scores, 

attendance management, tracking skills and standards, lesson 

planning, and generating report cards). 

Electronic portfolios Online storage of student work samples. 

Formative assessment Assessment conducted during instruction in order to provide feedback 

that can be used to adjust ongoing teaching and learning to improve 

student outcomes. Formative assessment can be contrasted with 

summative assessment, which takes place after a period of 

instruction in order to judge how much learning has occurred. 

Interoperability The ability of different data systems or software packages to 

communicate with one another. 

Longitudinal student 

data 

Data on individual students collected over time that allows users to 

compile an academic history for each student. This type of data 

enables more robust analyses of student performance to help 

differentiate instruction and improve student achievement. 

Query tool Software that allows for customized and ad-hoc data requests, such 

as “drill down” capability to efficiently examine a subset of data at a 

grade, classroom, or student level. 

Exhibit C-1 continues on next page 

C-3
Exhibit C-1 (continued) 

Definitions of Terms 

Term Definition

Reading First Under this federally funded reading program, states and districts 

receive support to apply scientifically based reading research—and 

the proven instructional and assessment tools consistent with this 

research—to ensure that all children learn to read well by the end of 

third grade. The program provides formula grants to states that 

submit an approved application. States award subgrants to eligible 

districts on a competitive basis. Funds are allocated to states 

according to the proportion of children age 5 to 17 who reside within 

the state and who are from families with incomes below the poverty 

line. 

Software applications Software applications consist of a wide range of specialized products 

to facilitate access to data, data analysis and interpretation, and 

presentations of data (e.g., formatted reports, graphing functions). 

Transaction capture Real-time accounting of daily school functions, such as attendance 

and school lunch counts. 

C-4
Exhibit C-2 

Top 15 Systems for Each Component* 

Student Information System Data Warehouses 

1. Penatamation Eschool (31) 1. SASI (20) 

2. SASI (26) 2. Pentamation (14) 

3. PowerSchool (24) 3. TetraData (11) 

4. Infinite Campus (20) 4. PowerSchool (10) 

5. AERIES (16) 5. STI (10) 

6. STI (15) 6. AERIES (10) 

7. CIMS (14) 7. Skyward (9) 

8. Skyward (13) 8. Educational Data Warehouse (8) 

9. STAR (10) 9. CIMS (7) 

10. TERMS (9) 10. COGNOS (7) 

11. Chancery (9) 11. JPAMS (6) 

12. JPAMS (8) 12. DataDirector (5) 

13. ESIS (7) 13. Otis Ed (4) 

14. Gradespeed (6) 14. ProSoft (4) 

15. Zangle and SIMS (both 6) 

Instructional/Curriculum Management 

System Assessment System 

1. Edusoft (16) 1. Edusoft (35) 

2. Moodle (8) 2. AEIS IT (17) 

3. SchoolNet (7) 3. DataDirector (12) 

4. Blackboard (7) 4. DIBELS (10) 

5. Curriculum Mapper (7) 5. NWEA (9) 

6. Angel (6) 6. MAP (7) 

7. DataDirector (6) 7. SCANTRON (7) 

8. FirstClass (6) 8. THINKLINK (7) 

9. OnCourse (6) 9. PERFORMANCE TRACKER (6) 

10. STI (5) 10. OARS (5) 

11. CSCOPE (5) 11. TRIAND (5) 

12. Eduphoria (5 12. AIMS (4) 

13. Rubicon (4) 13. Princeton Review (4) 

14. Sharepoint (3) 14. SchoolNet (4) 

15. THINKLINK and Triand (both 3) 

Note: Not all components include 15 systems because of duplications. 

* Cited by survey respondents. These categories may not correspond to the purposes for which the 

systems were developed. 

C-5
Exhibit C-3 (Exhibit 2-2 in Text) 

Profile of District Data System Elements in 2007–08, by District Size 

Data System Element 

Percent 

of All 

Districts 

Percent 

of Small 

Districts 

Percent of 

Medium 

Districts 

Percent 

of Large 

Districts 

1. Student information system 100 100 100 100 

Have system (Q4a) 100 100 100 100 

Ability to generate standard accountability 

reports or district report card and school report 

cards (Q8a) 

66 64** 77** 86** 

Transaction capture (Q8b) 92 92 95 97 

2. Data linkages 40 40 60 60 

Linking school performance and finance data 

(Q7g) 24 23 29 31 

Student performance linked to teacher 

information or characteristics (Q7c) 36 35 39 38 

Student performance linked to AYP subgroups 

(Q7a) 65 62** 81** 83** 

Student performance linked to specific teachers 

(Q7b) 64 61*** 82*** 85*** 

Student performance linked to specific 

instructional programs (Q7d) 40 38 50 48 

3. Instructional/curriculum management system 50 50 100 100 

Have system (Q4c) 64 63 68 72 

Links to curricular resources (Q8f) 62 62 62 71 

4. Assessment system 50 50 100 100 

Have system (Q4d) 79 77** 87** 93** 

Assessments available in reading, mathematics, 

or other core subject areas that students take 

online (Q8d) 

46 44 52 61 

5. Student performance data to measure 

academic growth 75 75 100 100 

Student test scores on statewide assessments 

(Q5a) 93 92* 99* 99* 

Student test scores on district-administered 

assessments (Q5b) 72 69*** 83*** 95*** 

Drill-down capability (Q8c) 72 70* 83* 86* 

Individual student assessment performance over 

time (Q7e) 70 66*** 88*** 88*** 

6. Student-level enrollment, demographic and 

program participation information 100 100 100 100 

Student grades (Q5e) 95 95 97 98 

Student course enrollment histories (Q5f) 92 90 96 98 

Prior school(s) attended within the district (Q5h) 86 84** 96** 99** 

Student demographics (Q5g) 98 98 99 99 

Student attendance (Q5k) 98 98 97 99 

Student behavior (Q5l) 87 86** 90** 98** 

Student special education information (Q5i) 84 82* 93* 96* 

Individual student history over time (Q7f) 81 79* 94* 90* 

7. Teacher-level data 50 50 100 100 

Teacher qualifications (Q5p) 73 70*** 87*** 96*** 

Teacher professional development (Q5q) 47 43*** 63*** 85*** 

Exhibit C-3 continues on next page. 

C-6
Exhibit C-3 (continued) 

Profile of District Data System Elements in 2007–08, by District Size 

Data System Element 

Percent 

of All 

Districts 

Percent 

of Small 

Districts 

Percent of 

Medium 

Districts 

Percent 

of Large 

Districts 

8. Student-level graduation, post-graduation 

and dropout data 67 67 67 67 

Differential codes for students no longer enrolled 

(Q5m) 93 93* 95* 99* 

Student graduation status (Q5n) 90 89* 95* 98* 

Student status after graduation (Q5o) 34 35 30 27 

9. College readiness 100 100 100 100 

Student test scores on SAT, ACT, and 

Advanced Placement tests (Q5d) 57 55*** 68*** 85*** 

10. Assessment of data quality 67 67 100 100 

District/state has disseminated data collection 

guidelines & recommended data information 

management and security practices to schools 

(Q12) 

71 68** 85** 87** 

District has staff or outside source responsible 

for receiving & preparing files from outside 

sources to load into the student data system 

(Q13) 

89 88* 97* 97* 

Greater than 90 percent of data captured by the 

district’s student data system(s) that drive 

instructional improvement are accurate (Q14) 

65 64 70 68 

Note: Shaded numbers represent the median percent of subelements present. Numbers for each 

of the subelements are the percent of districts that have that subelement. Asterisks indicate the 

proportion of districts with a particular subelement vary significantly by district size (*p < .05, 

**p < .01 and ***p < .001). 

Source: 2007–08 district survey questions 4, 5, 7, 8, 12-14. 

C-7
Exhibit C-4 (Exhibit 2-6 in Text)

District Data System Query Capabilities in 2007–08, by District Size 

Type of Query 

Percent of Districts With 

This System Capability 

Percent of Students 

Represented by These 

Districts 

Small Medium Large Small Medium Large 

Individual student history over time 

(e.g., cumulative grades)* 81 94 90 82 89 92 

Drill-down capability (ability to 

query a school-level finding to 

efficiently examine a subset of data 

at the grade, classroom, or student 

level) 

74 84 86 79 84 90 

Individual student assessment 

performance over time** 69 88 89 75 88 92 

Student performance linked to 

specific teachers** 38 39 39 49 43 45 

Student performance linked to 

Adequate Yearly Progress 

subgroups** 

63 81 84 66 81 88 

Student performance linked to 

specific instructional programs 41 50 48 45 52 53 

Student performance linked to 

teacher information or 

characteristics 

38 39 39 49 43 45 

School performance linked to 

finance data 26 29 31 32 31 41 

Note: Asterisks indicate the proportion of districts that have this query capability vary significantly by 

district size (*p < .05 and **p <. 01). 

Source: 2007–08 district survey questions 7 and 8. 

C-8
Exhibit C-5 (Exhibit 2-7 in Text) 

District Data System Features and Tools in 2007–08, by District Size 

Feature or Tool 

Percent of Districts With 

This System Feature 

Percent of Students 

Represented by These 

Districts 

Small Medium Large Small Medium Large 

Transaction capture (e.g., daily 

and class attendance) 95 96 98 92 92 98 

Tools for communicating with 

parents around an individual 

student’s performance 

78 81 74 78 76 80 

Ability to generate standard 

accountability reports or district or 

school report cards 

68 77 87 69 78 90 

Links to curriculum resources 68 62 73 66 57 73 

Assessments available in reading, 

mathematics or other core subject 

areas that students take online 

52 53 62 52 52 61 

Source: 2007–08 district survey question 8. 

C-9
Exhibit C-6a (Exhibit 2-8 in Text) 

Districts Using Data Systems for Selected Purposes, by District Size 

Use of Data System 

Percent of 

All Districts 

Percent of 

Small 

Districts 

Percent of 

Medium 

Districts 

Percent of 

Large 

Districts 

Accountability Activities 

Track school performance (e.g., 

to estimate AYP for schools) 96* 96 95 100 

Analyze student achievement by 

grade-level, districtwide or by 

school 

94* 93 98 100 

Monitor student attendance 94 94 96 99 

Examine achievement gaps 

between groups of students 93* 93 95 99 

Analyze student achievement 

data over time (i.e., to identify 

trends) 

92** 91 98 100 

Meet accountability reporting 

requirements 92* 91 97 99 

Track graduation rates by school 90** 89 92 99 

Inform parents about student 

progress 

88 87 93 92 

Track student readiness for 

promotion or graduation (e.g., 

percent on track to graduate) 

83 82 89 93 

Track teacher qualifications (e.g., 

to determine highly qualified 

status) 

80+++ 81 69 89 

Instruction-related Activities 

Track other measures of student 

progress (e.g., benchmark and 

diagnostic tests) 

87** 85 96 99 

Guide curricular changes or 

curriculum development 87*** 86 89 98 

Inform student placement in 

courses or special programs or 

support services 

84* 83 92 95 

Inform professional development 

offerings for school staff 84 83 87 93 

Develop benchmark assessments 

aligned with curriculum 82*** 81 82 95 

Inform teachers about individual 

student’s instructional needs in 

terms of specific skills 

81 80 90 84 

Inform instructional practice (e.g., 

tailoring instruction to meet 

student needs) 

80* 79 82 92 

Inform resource allocation to 

improve instruction 

78++ 80 70 88 

Exhibit C-6a continues on next page. 

C-10
Exhibit C-6a (continued) 

Districts Using Data Systems for Selected Purposes, by District Size 

Use of Data System 

Percent of 

All Districts 

Percent of 

Small 

Districts 

Percent of 

Medium 

Districts 

Percent of 

Large 

Districts 

Identify promising instructional 

programs 

76** 74 82 92 

Target individual teachers for 

specific professional development 70+ 73 54 64 

Other Activities 

Examine district or school climate 

data (e.g., student perceptions) 70** 67 81 88 

Evaluate teacher performance 65 66 61 68 

Evaluate principal performance 65*** 65 63 86 

Note: Asterisks indicate the proportion of districts that carry out a particular activity vary significantly by 

district size (*p < .05, **p < .01). 

Source: 2007–08 district survey questions 18 and 19. 

C-11
Exhibit C-6b 

Districts Not Conducting Selected Activities, by District Size 

Use of Data System 

Percent of 

Small 

Districts 

Percent of 

Medium 

Districts 

Percent of 

Large 

Districts 

Accountability Activities 

Track school performance (e.g., to estimate AYP 

for schools)* 4 5 0 

Analyze student achievement by grade-level, 

districtwide or by school* 7 2 0 

Examine achievement gaps between groups of 

students* 7 5 1 

Analyze student achievement data over time 

(i.e., to identify trends)** 9 2 0 

Meet accountability reporting requirements* 9 3 1 

Track graduation rates by school** 11 8 1 

Track teacher qualifications (e.g., to determine 

highly qualified status)** 19 31 11 

Instruction-related Activities 

Track other measures of student progress (e.g., 

benchmark and diagnostic tests)** 15 4 1 

Guide curricular changes or curriculum 

development** 14 11 2 

Inform student placement in courses or special 

programs or support services* 17 8 5 

Develop benchmark assessments aligned with 

curriculum** 19 18 5 

Inform instructional practice (e.g., tailoring 

instruction to meet student needs)* 21 18 8 

Inform resource allocation to improve 

instruction** 

20 30 12 

Identify promising instructional programs** 26 18 8 

Target individual teachers for specific 

professional development* 

27 46 36 

Other Activities 

Examine district or school climate data (e.g., 

student perceptions)** 33 19 12 

Evaluate principal performance*** 35 37 14 

Note: Asterisks indicate the proportion of districts that report they never carry out a particular 

activity vary significantly by district size (*p < .05, **p < .01, and ***p<.001). 

Source: 2007–08 district survey questions 18 and 19. 

C-12
Exhibit C-7 (Exhibit 3-1 in Text) 

District Data-informed Decision-making Support Index, by District Size 

Percent of 

All 

Districts 

Percent 

of Small 

Districts 

Percent of 

Medium 

Districts 

Percent 

of Large 

Districts 

1. Data system access 100 100 100 100 

Principal or specialists access to all or most 

data for students in their school (Q9) 92 92 93 93 

Teacher access to data on students in their 

classroom (Q10a&b) 81 83 72 77 

2. Leadership for improvement and use of data 100 67 100 100 

Training school administrators on how to 

provide leadership for data-driven decision 

making practices in their school (Q21f) 

82 80** 93** 97** 

Requiring all or particular schools to follow 

specific data-driven decision-making practices 

in their school improvement plans (Q22k) 

69 65*** 88*** 91*** 

Following up to determine if schools have 

implemented instructional changes prescribed 

as a result of data analysis activities (Q22m) 

60 56*** 77*** 84*** 

3. Tools for generating data 67 67 67 67 

Have assessment system (Q4d) 79 77** 87** 93** 

Assessments available in reading, mathematics 

or other core subject areas that students take 

online (Q8d) 

46 44 52 61 

Providing Web-accessible library of diagnostic 

or benchmark assessments (for downloading) 

linked to academic standards (Q22f) 

47 46* 49* 64* 

4. Social structures and time set aside for data 

use 

67 33 67 67 

Paying for incentives for teachers to use or 

obtain training in data-driven decision making 

(e.g., paying for dedicated time for school staff 

to review data) (Q22i) 

30 28* 38* 47* 

Requiring “data conferences” between 

individual principals and their supervisors (Q22l) 48 43*** 73*** 81*** 

Implementing policies and requirements to use 

data or providing incentives for data use (Q23c) 81 80 85 87 

5. Professional development and technical 

support for data interpretation 88 75 100 100 

Training school staff on the basic functions of 

the data system (Q21a) 90 89 96 97 

Training principals or other building 

administrators on using the data system to 

analyze student achievement (Q21d) 

91 90* 94* 98* 

Training principals on using data to change 

instructional practice (Q21e) 86 85*** 91*** 99*** 

Training teachers on using the data system to 

analyze student achievement (Q21g) 81 80* 89* 94* 

Training teachers on using data to change 

instructional practice (Q21h) 82 80** 89** 96** 

Making technical experts (in systems, networks, 

databases) available to schools to support 

system use (Q22a) 

80 79* 89* 93* 

Exhibit C-7 continues on next page. 

C-13
Exhibit C-7 (continued) 

District Data-informed Decision-making Support Index, by District Size 

Percent of 

All 

Districts 

Percent 

of Small 

Districts 

Percent of 

Medium 

Districts 

Percent 

of Large 

Districts 

Making data analysis experts available to 

school staff such as data coaches (Q22c) 50 44*** 74*** 84*** 

Requiring instructional coaches to explicitly 

incorporate data use and train teachers in data 

use as part of their job (Q22h) 

44 37*** 76*** 87*** 

6. Tools for acting on data 60 60 80 100 

Providing models to schools to illustrate how to 

use data in allocating resources and designing 

school improvement activities (Q22b) 

57 53*** 80*** 90*** 

Providing teachers with processes or tools to 

effectively utilize data for instructional purposes 

(Q22j) 

65 62** 79** 84** 

Links to curricular resources (Q8f) 62 62 62 71 

Providing teachers research-based guidance on 

differentiating instruction on the basis of student 

assessment data (Q22g) 

74 72* 83* 88* 

Providing an online database lesson plans and 

planning resources linked to academic 

standards and assessment results. (Q22e) 

60 62** 51** 73** 

Note: Asterisks indicate proportion of districts with a particular element varies significantly by 

district size (*p < .05, ** p < .01 and *** p < .001). 

Source: 2007–08 district survey questions 4, 8-10, 21-23. 

C-14
Exhibit C-8 (Exhibit 3-4 in Text)

District-provided Training on the Use of Data and Data Systems 

District-Supported Training to Increase 

School-level Capacity 

District 

Has Not 

Provided 

This 

Training 

Percent of Schools Involved 

Some But 

Not All 

Schools All Schools 

Training principals or other building administrators on 

using the data system to analyze student 

achievement. 

9 18 73 

Training school staff on the basic functions of the 

data system (e.g., accessing and downloading data, 

data queries). 

10 23 67 

Training principals or other building administrators on 

using data to change instructional practice (e.g., 

tools for translating data into practice). 

14 23 64 

Training school staff on data entry to improve data 

accuracy. 

16 21 63 

Training principals and other building administrators 

on how to provide leadership for data-driven 

decision-making practices in their school (e.g., 

modeling data use, leading data discussions). 

18 22 60 

Training teachers on using data to change 

instructional practice (e.g., tools for translating data 

into practice, collaborating with colleagues to discuss 

data). 

18 29 53 

Training teachers on using the data system to 

analyze student achievement. 19 28 53 

Training school staff on data management and 

security. 24 20 56 

Source: 2007–08 district survey question 21. 

C-15
Exhibit C-9 (Exhibit 3-5 in Text) 

District-provided Supports for School-level Use of Data to Improve Instruction 

District Activities to Increase School-level 

Capacity 

District 

Has Not 

Done 

This 

Activity 

Percent of Schools Involved 

Some But 

Not All 

Schools All Schools 

Making technical experts (in systems, networks, 

databases) available to schools to support system 

use. 

20 16 65 

Using data to identify professional development 

activities for schools identified for improvement (e.g., 

to address school improvement needs). 

24 26 50 

Providing teachers research-based guidance on 

differentiating instruction on the basis of student 

assessment data. 

26 23 51 

Providing an online database of lesson plans and 

planning resources linked to academic standards 

and assessment results. 

40 19 41 

Providing models to schools to illustrate how to use 

data in allocating resources and designing school 

improvement activities (e.g., school improvement 

template, providing assistance in analyzing and 

revising the school budget). 

43 15 43 

Making data analysis experts available to school staff 

such as data coaches. 50 18 32 

Providing a Web-accessible library of diagnostic or 

benchmark assessments (for downloading) linked to 

academic standards. 

53 13 34 

Source: 2007–08 district survey question 22. 

 

C-16
Exhibit C-10 (Exhibit 3-6 in Text) 

District Policies and Practices Encouraging Schools’ Use of Data 

District Activities to Increase 

School-level Capacity 

District 

Has Not 

Done 

This 

Activity 

Percent of Schools Involved 

Some But 

Not All 

Schools All Schools 

Requiring all or particular schools to follow 

specific data-driven decision-making practices in 

their school improvement plans (e.g., identifying 

targets, monitoring their data). 

31 18 51 

Providing teachers with processes or tools to 

effectively utilize data for instructional purposes 

(e.g., template for how to talk about data with 

colleagues). 

35 25 40 

Following up to determine whether schools have 

implemented instructional changes prescribed as 

a result of data analysis activities. 

40 21 39 

Requiring “data conferences” between individual 

principals and their supervisors. 52 12 36 

Requiring instructional coaches to explicitly 

incorporate data use and train teachers in data 

use as part of their job. 

56 18 26 

Tracking teacher use of the data system. 70 9 21 

Paying for incentives for teachers to use or obtain 

training in data-driven decision making (e.g., 

paying for dedicated time for school staff to 

review data). 

70 12 18 

Source: 2007–08 district survey question 22. 

C-17
Exhibit C-11 (Exhibit 3-7 in Text) 

District Administrators’ Perceptions of Barriers to Increased Use of 

Data Systems 

Barriers 

Percent 

Reporting 

Major Barrier 

Percent 

Reporting 

Minor 

Barrier 

Percent 

Reporting 

Not a Barrier 

Data System and Technology Barriers 

Lack of funding to expand or improve the data 

system 42 34 24 

Information located in multiple disparate 

databases that make it difficult to link data for 

analyses (i.e., lack of interoperability)* 

36 28 36 

Lack of trained technical staff available for 

product and service acquisition, installation or 

equipment maintenance* 

23 47 30 

Data stored in forms that are difficult to access, 

manage and interpret* 23 29 48 

Out-of-date hardware 12 38 51 

Lack of sufficient hardware (servers, 

computers, peripheral devices, etc.) 9 38 53 

Internet connections that are not fast or reliable 

enough 6 27 68 

Unreliable or inaccurate data in system** 5 21 74 

Lack of districtwide unique teacher

identification numbers that are consistent from 

year to year 

3 14 83 

Lack of districtwide unique student

identification numbers that are consistent from 

year to year 

1 4 95 

Inability to provide adequate safeguards and 

security for the data** 1 31 68 

Logistical and Other Barriers 

Lack of time for school staff to conduct datadriven decision-making activities (e.g., to reflect 

on or use data) 

51 41 8 

Lack of resources to train school staff on how 

to use data to support instructional 

improvement 

35 46 19 

Lack of teacher preparation on how to use data 

for instructional decision making (e.g., data 

interpretation skills) 

29 56 16 

Lack of technical skills of school staff to access 

or utilize data systems (e.g., technical 

proficiencies) 

20 69 11 

Lack of building administrator preparation on 

how to use data for data-driven decision 

making 

16 53 31 

Lack of a clear vision or strategic plan for datadriven decision making 15 49 36 

Exhibit C-11 continues on next page. 

C-18
Exhibit C-11 (continued) 

District Administrators’ Perceptions of Barriers to Increased Use of 

Data Systems 

Barriers 

Percent 

Reporting 

Major Barrier 

Percent 

Reporting 

Minor 

Barrier 

Percent 

Reporting 

Not a Barrier 

Lack of communication or sharing of data 

across departments within the district 10 56 34 

Lack of incentives for data used by district or 

school staff 13 50 37 

Lack of district leadership support for datadriven decision making 9 30 61 

Policies that preclude direct access by school 

staff to data systems or individual student-level 

data 

4 33 62 

Note: Asterisks indicate significant differences by district size (*p < .05 and ** p < .01). 

Source: 2007–08 district survey question 25. 

C-19
Exhibit C-12 (Exhibit 3-9 in Text)

District Perceptions of Needed Examples of Good Practice 

Areas of Need 

Percent 

Reporting 

Little Need 

(We know 

how to do 

this) 

Percent 

Reporting 

Some 

Need 

Percent 

Reporting 

Great 

Need 

Examining student data to identify which practices 

work best for which students (e.g., comparing the 

performance of students receiving instructional 

programs). 

17 47 37 

Adapting instructional activities to meet students’ 

individual needs (e.g., modifying lesson plans to teach 

students at different ability levels). 

16 49 35 

Developing curriculum-embedded formative 

assessments (e.g., designing assessments to use 

with instruction). 

19 48 33 

Collaborating and sharing ideas with colleagues 

regarding data inquiry and analysis issues (e.g., group 

facilitation techniques). 

20 54 26 

Structuring the district organization and practices to 

support data-driven decision making. 20 59 21 

Using assessment data to identify gaps in student 

achievement (e.g., standards that individual students 

or groups of students don’t meet). 

29 50 21 

Communicating with parents about student progress. 37 49 14 

Source: 2007–08 district survey question 24. 

C-20
Exhibit C-13

Longevity of District Engagement in Helping Schools Use Data, by District Size 

District Support Activities 

Percent of 

All Districts 

Percent of 

Small 

Districts 

Percent of 

Medium 

Districts 

Percent 

of Large 

Districts 

Providing professional development for teachers and principals on the use of data to improve 

instructional practices. 

No plans 4 5 1 1 

Planning, but not started 17 20 9 <1 

1 to 2 years 30 31 26 17 

3 to 5 years 27 26 30 41 

6 or more years 21 19 34 40 

Providing resources (such as models, consultants) to schools to support the use of data to inform 

instruction. 

No plans 9 10 3 3 

Planning, but not started 18 19 13 2 

1 to 2 years 25 27 18 19 

3 to 5 years 28 26 37 44 

6 or more years 20 18 29 32 

Implementing policies and requirements to use data or providing incentives for data use. 

No plans 19 20 15 13 

Planning, but not started 31 33 22 9 

1 to 2 years 17 17 17 19 

3 to 5 years 17 14 28 34 

6 or more years 16 15 18 25 

Source: 2007–08 district survey question 23. 

C-21
Exhibit C-14 

District Administrators’ Perceptions of Need for Examples of Good Practice, 

by District Size

Perceived Need 

District 

Size 

Percent 

Reporting 

Great 

Need 

Percent 

Reporting 

Little or 

No Need 

a. Using assessment data to identify gaps in student 

achievement (e.g., standards that individual students 

or groups of students don’t meet).** 

All 21 29 

Small 22 25 

Medium 15 46 

Large 14 47 

b. Adapting instructional activities to meet students’ 

individual needs (e.g., modifying lesson plans to 

teach students at different ability levels). 

All 35 16 

Small 35 16 

Medium 34 17 

Large 31 18 

c. Developing curriculum-embedded formative 

assessments (e.g., designing assessments to use 

with instruction). 

All 33 19 

Small 35 17 

Medium 27 23 

Large 26 31 

d. Examining student data to identify which practices 

work best for which students (e.g., comparing the 

performance of students receiving instructional 

programs). 

All 37 17 

Small 38 16 

Medium 29 20 

Large 26 20 

e. Collaborating and sharing ideas with colleagues 

regarding data inquiry and analysis issues (e.g., 

group facilitation techniques). 

All 26 20 

Small 27 18 

Medium 23 31 

Large 19 27 

f. Communicating with parents about student progress. All 14 37 

Small 15 35 

Medium 8 46 

Large 17 38 

g. Structuring the district organization and practices to 

support data-driven decision making. All 21 20 

Small 23 17 

Medium 16 32 

Large 15 26 

Note: Asterisks indicate the extent to which this need varies significantly by district size (**p < .01). 

Source: 2007–08 district survey question 24. 

C-22
Exhibit C-15 (Exhibit 3-10 in Text) 

District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, 

by District Size

Perceived Barriers 

District 

Size 

Percent 

Reporting 

Major 

Barrier 

Percent 

Reporting 

Minor 

Barrier 

Data System and Technology Barriers 

a. Lack of sufficient hardware All 9 38 

Small 9 40 

Medium 9 26 

Large 19 36 

b. Out-of-date hardware All 12 38 

Small 12 39 

Medium 7 34 

Large 19 33 

c. Internet connections that are not fast or reliable 

enough All 6 27 

Small 5 28 

Medium 9 21 

Large 14 25 

d. Unreliable or inaccurate data in the system** All 5 21 

Small 4 18 

Medium 7 36 

Large 6 42 

e. Data stored in forms that are difficult to access, 

manage and interpret* All 23 29 

Small 24 27 

Medium 19 40 

Large 15 57 

f. Information located in multiple disparate databases 

that make it difficult to link data for analysis* All 36 28 

Small 36 26 

Medium 37 41 

Large 33 45 

g. Lack of districtwide unique student identification 

numbers that are consistent from year-to-year All 1 4 

Small 2 4 

Medium 0 5 

Large 2 5 

h. Lack of districtwide unique teacher identification 

numbers that are that are consistent from year-toyear 

All 3 14 

Small 3 13 

Medium 2 17 

Large 5 17 

Exhibit C-15 continues on next page. 

C-23
Exhibit C-15 (continued) 

District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, 

by District Size

Perceived Barriers 

District 

Size 

Percent 

Reporting 

Major 

Barrier 

Percent 

Reporting 

Minor 

Barrier 

i. Inability to provide adequate safeguards and security 

for the data** All 1 31 

Small 0 33 

Medium 3 20 

Large 3 19 

j. Lack of funding to expand or improve the student 

data system All 42 34 

Small 42 35 

Medium 42 30 

Large 31 46 

k. Lack of trained technical staff available for product 

and service acquisition, installation, or equipment 

maintenance* 

All 23 47 

Small 25 47 

Medium 12 45 

Large 22 46 

Logistical and Other Barriers 

l. Lack of resources to train school staff on how to use 

data to support instructional improvement All 35 46 

Small 36 46 

Medium 29 44 

Large 33 44 

m. Lack of teacher preparation on how to use data for 

instructional decision making All 29 56 

Small 28 56 

Medium 33 52 

Large 39 52 

n. Lack of technical skills of school staff to access or 

utilize electronic data systems All 20 70 

Small 18 72 

Medium 30 56 

Large 29 55 

o. Lack of building administrator preparation on how to 

use data for data-driven decision making All 16 53 

Small 15 52 

Medium 19 60 

Large 16 58 

Exhibit C-15 continues on next page. 

C-24
Exhibit C-15 (continued) 

District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, 

by District Size

Perceived Barriers 

District 

Size 

Percent 

Reporting 

Major 

Barrier 

Percent 

Reporting 

Minor 

Barrier 

p. Lack of time for school staff to conduct data-driven 

decision-making activities All 51 41 

Small 50 41 

Medium 54 40 

Large 49 43 

q. Lack of a clear vision or strategic plan for data-driven 

decision making All 15 50 

Small 16 50 

Medium 9 47 

Large 9 49 

r. Lack of district leadership support for data-driven 

decision making All 9 30 

Small 10 31 

Medium 6 26 

Large 3 28 

s. Lack of communication or sharing of data across 

departments within the district All 10 56 

Small 10 57 

Medium 12 52 

Large 7 63 

t. Policies that preclude direct access by school staff to 

data systems or individual student-level data All 4 33 

Small 4 35 

Medium 5 26 

Large 4 26 

u. Lack of incentives for data use by district or school 

staff All 13 50 

Small 13 51 

Medium 16 42 

Large 10 41 

Note: Asterisks indicate the extent to which this barrier varies significantly by district size (*p < .05 

and **p < .01). 

Source: 2007–08 district survey question 25. 

C-25
C-26

Exhibit C-16 

Districts Conducting Selected Activities, by Percent of Schools Not Making AYP 

Activity 

Percent of 

Districts That 

Sometimes 

Conduct Activity 

Percent of 

Districts That 

Never Conduct 

Activity 

Examine achievement gaps between groups of 

students* 18 7 

Track school performance* 17 2 

Guide curricular changes or curriculum development** 19 8 

Inform instructional practice* 19 10 

Note: Asterisks indicate the proportion of schools not making AYP within a district varies significantly 

by district size for whether or not a district carries out a particular activity (*p < .05 and **p < .01). 

Source: 2007–08 district survey questions 18 and 19.