**Use of Education Data at the Local Level From Accountability to Instructional Improvement**

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Prepared by:

Barbara Means

Christine Padilla

Larry Gallagher

SRI International

2010

This report was prepared for the U.S. Department of Education under Contract number ED-01-CO-0040 Task 0002 with SRI International. Bernadette Adams Yates served as the project manager. The views expressed herein do not necessarily represent the positions or policies of the Department of Education. No official endorsement by the U.S. Department of Education is intended or should be inferred.

**U.S. Department of Education**Arne Duncan*Secretary*

**Office of Planning, Evaluation and Policy Development**Carmel Martin*Assistant Secretary*

**Policy and Program Studies Service**Alan Ginsburg*Director*

**Office of Educational Technology**Karen Cator*Director*

January 2010

This report is in the public domain. Authorization to reproduce this report in whole or in part is granted. While permission to reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, *Use of Education Data at the Local Level From Accountability to Instructional Improvement,* Washington, D.C., 2010.

This report is available on the Department’s Web site at http://www.ed.gov/about/offices/list/opepd/ppss/reports.html#edtech

On request, this publication is available in alternate formats, such as Braille, large print, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818.

# Contents

List of Exhibitsiv

Acknowledgmentsvii

Executive Summaryix

1. Introduction and Approach1

Data Systems: A Prerequisite for Data-driven Decision Making2

Beyond Data Systems: Need for a Systemic Approach2

Data-driven Decision-making Framework2

Data Sources for the Report5

Contents of the Report6

2. District Data System Features and District Use of Data for Decision Making9

A Profile of District Data Systems10

District Data System Features13

Districts’ Use of Data Systems23

Variations Across Districts26

Summary27

3. District Supports for Data-driven Decision Making in Schools29

District Data-driven Decision-making Support Index29

District Strategies for Increasing Use of Data Systems33

Challenges to District Efforts to Spread Data-driven Decision Making42

Strategies for Addressing Implementation Challenges54

Summary57

4. School Use of Data and Supports for Data-driven Decision Making59

Nature and Frequency of School Use of Data and Data Systems59

Year-to-Year Growth in Data Use62

Stages in Developing a Data-using Culture64

School Supports for Data Use68

Barriers to School Data Use76

Perceived Effects of Data Use80

Summary81

5. Conclusion and Policy Recommendations83

Supports Needed for Data-driven Decision Making83

Data-driven Decision Making as a Systemic Reform86

Recommendations87

References93

Appendix A: MethodsA-1

Appendix B: 2007–08 District Survey and FrequenciesB-1

Appendix C: Terminology and Additional DataC-1

# Exhibits 

Exhibit ES-1Types of Electronic Student Data Systemsx

Exhibit ES-2Districts With Student Data Systems, by Typex

Exhibit ES-3:District Data System Query Capabilities in 2007–08xii

Exhibit ES-4:District Perceptions of Needed Examples of Good Practicexiv

Exhibit ES-5:Uses of Data Described by Case Study Schoolsxv

Exhibit ES-6:Categories of Data-driven Decision Makingxvi

Exhibit ES-7:Percentage of Case Study Schools Reporting Each Decision-making ****Categoryxvi

Exhibit ES-8:Teacher Perceptions of the Support They Receive for Data Usexviii

Exhibit 1-1:Conceptual Framework for Data-driven Decision Making3

Exhibit 2-1:Types of Electronic Student Data Systems9

Exhibit 2-2:Profile of District Data System Elements in 2007–0811

Exhibit 2-3:District History With Student Data Systems, by Type15

Exhibit 2-4:District Implementation of Student Data Systems16

Exhibit 2-5:Types of Information That Districts Maintained Electronically in 2007–0820

Exhibit 2-6:District Data System Query Capabilities in 2007–0822

Exhibit 2-7:District Data System Features and Tools in 2007–0822

Exhibit 2-8:Districts Using Data Systems for Selected Purposes24

Exhibit 3-1:District Data-driven Decision-making Support Profile31

Exhibit 3-2:District Strategies to Promote Data-driven Decision Making From 2005 to 200733

Exhibit 3-3:Systemic Change Guides District Implementation of Data-driven Decision-making Practices34

Exhibit 3-4:District-provided Training on the Use of Data and Data Systems36

Exhibit 3-5:District-provided Supports for School-level Use of Data to Improve****Instruction39

Exhibit 3-6:District Policies and Practices Encouraging Schools’ Use of Data40

Exhibit 3-7:District Administrators’ Perceptions of Barriers to Increased Use****of Data Systems43

Exhibit 3-8:District Leadership Support for Data-driven Decision Making46

Exhibit 3-9:District Perceptions of Needed Examples of Good Practice48

Exhibit 3-10:District Administrators’ Perceptions of Barriers to Increased Use****of Data Systems, by District Size50

Exhibit 3-11:Case Study District Supports for School Use of Student Data52

Exhibit 3-12:District Perceptions of Major Needs and Barriers, by Longevity of ****Data-driven Decision-making Efforts53

Exhibit 3-13:Piloting Data-driven Decision-making Efforts57

Exhibit 4-1:Teachers Who Reported Using a Student Data System at****Least a Few Times a Year for a Specific Function in 200760

Exhibit 4-2:Uses of Data Described by Case Study Schools61

Exhibit 4-3:Assessment Terminology63

Exhibit 4-4:Categories of Data-driven Decision Making66

Exhibit 4-5:Percentage of Case Study Schools Reporting Each Decision-making ****Category67

Exhibit 4-6:A School at Stage 3 in Data Use68

Exhibit 4-7:Teacher Reported Supports for Use of Student Data to Guide****Instruction, by Type of Support in 200768

Exhibit 4-8:Teacher Perceptions of the Support They Receive for Data Use75

Exhibit A-1:Distribution of Districts and Student Population, by District SizeA-3

Exhibit A-2:Distribution of Districts and Student Population, by District****Poverty RateA-4

Exhibit A-3:Number of Districts in the Universe and Sample Size, by StratumA-5

Exhibit A-4:Case Study Districts and School Sample in 2007–08A-7

Exhibit C-1:Definitions of TermsC-3

Exhibit C-2:Top 15 Systems for Each ComponentC-5

Exhibit C-3:Profile of District Data System Elements in 2007–08, by District SizeC-6

Exhibit C-4:District Data System Query Capabilities in 2007–08, by District SizeC-8

Exhibit C-5:District Data System Features and Tools in 2007–08, by District SizeC-9

Exhibit C-6a:Districts Using Data Systems for Selected Purposes, by ****District SizeC-10

Exhibit C-6b:Districts Not Conducting Selected Activities, by District SizeC-12

Exhibit C-7:District Data-informed Decision-making Support Index, by****District SizeC-13

Exhibit C-8:District-provided Training on the Use of Data and Data SystemsC-15

Exhibit C-9:District-provided Supports for School-level Use of Data to ****Improve InstructionC-16

Exhibit C-10:District Policies and Practices Encouraging Schools’ Use of DataC-17

Exhibit C-11:District Administrators’ Perceptions of Barriers to Increased Use of Data SystemsC-18

Exhibit C-12:District Perceptions of Needed Examples of Good PracticeC-20

Exhibit C-13: Longevity of District Engagement in Helping Schools Use Data, by****District SizeC-21

Exhibit C-14:District Administrators’ Perceptions of Need for Examples of Good Practice, by District SizeC-22

Exhibit C-15:District Administrators’ Perceptions of Barriers to Increased Use****of Data Systems, by District SizeC-23

Exhibit C-16:Districts Conducting Selected Activities, by Percent of Schools Not****Making AYPC-26

# Acknowledgments

Many individuals contributed to the completion of this final report. We are particularly grateful to the district and school-level staff who took time out of their busy schedules to respond to our requests for information: in particular, the 12 districts and 36 case study schools were generous with both their time and attention to this evaluation work. Without their efforts, this report would not have been possible, and we deeply appreciate their assistance.

We would like to acknowledge the thoughtful contributions of the members of our Technical Work Group in reviewing study materials and prioritizing issues to investigate. The advisors consisted of Katherine Conoly of Corpus Christi ISD, Marty Daybell of Washington Schools Information Processing, Aimee Guidera of the National Center for Educational Accountability, Glynn Ligon of ESP Solutions, Ellen Mandinach of CNA Corporation, Jim Pellegino of the University of Illinois-Chicago, Arie van der Ploeg of Learning Point Associates, and Jeff Wayman at the University of Texas at Austin. 

Many U.S. Department of Education staff contributed to the completion of this report. Bernadette Adams Yates served as project manager and provided valuable substantive guidance and support throughout the design, implementation and reporting phases of this study. We would also like to acknowledge the assistance of other Department staff in reviewing this report and providing useful comments and suggestions, including David Goodwin, Daphne Kaplan, Vicotria Hammer, Vicki Robinson, Maureen Dowling, Lee Hoffman, Kashka Kubdzela, Paul Strasbery, and Larry Cohen.

We appreciate the assistance and support of all of the above individuals; any errors in judgment or fact are, of course, the responsibility of the authors.

The Study of Education Data Systems and Decision Making is supported by a large project team at SRI. Among the staff who contributed to the research were Maria Abasi, Marianne Bakia, Janeula Burt, Sara Carriere, Lauren Cassidy, Eva Chen, Angela DeBarger, Larry Gallagher, Marilyn Gillespie, Torie Gorges, Ann House, Harold Javitz, Karla Jones, Aasha Joshi, Carlin Llorente, Bladimir Lopez-Prado, Patrik Lundh, Nicolette Mayes, Natalie Nielsen, Christina Park, Angeline Reyes, Elizabeth Rivera, Corinne Singleton, Tina Stanford, Edith Yang, Kaily Yee, and Viki Young. Layout and editing were performed by Eileen Behr and Meredith Itner.

# Executive Summary

The use of student data systems to improve education and help students succeed is a national priority. The *Elementary and Secondary Education Act*, as reauthorized in 2002, calls for the collection, analysis, and use of student achievement data to improve school outcomes. Data systems are expected to play an integral role in improving educational decision making at all levels—including that of the classroom teacher. The U.S. Department of Education has provided support for major improvements in the quality of state data systems to enable longitudinal analysis of student data and linkage between student outcomes and other education system variables. These improved systems are supporting educational research and decision making at the state level, but at the local level, district and school staff work with district rather than state data systems. If data-driven decision making is to become an effective tool for improving the instruction provided to students, policymakers need a clear understanding of these data systems used at the local level and of the decision-making processes in schools and districts.

Since 2006 the national Study of Education Data Systems and Decision Making, sponsored by the U.S. Department of Education’s Policy and Program Studies Service, has been examining both the implementation of student data systems per se and the broader set of practices involving the use of data to improve instruction, regardless of whether or not the data are stored in and accessed through an electronic system. The study’s data collections included a national survey of districts in spring 2007 and site visits during school years 2006–07 and 2007–08 to a purposive sample of districts and schools selected on the basis of their active involvement in the use of data for instructional improvement. The study team also conducted secondary analyses of national teacher survey responses to questions concerning data system access and use. 

Earlier study reports have documented a dramatic increase in the proportion of teachers with access to a student data system between 2005 and 2007 and described school practices with respect to data use and the challenges that are part of student data system implementation. This final report builds on the picture of local practices in implementing data-driven decision making provided in the earlier reports by presenting data from the national district survey as well as from site visits conducted during 2007–08 at 36 schools in 12 districts. 

Key findings from surveys and site visits with respect to district data systems and strategies for supporting the use of data in instructional decision making are described below.

**District Data Systems and Use of Data for Decision Making**

**Districts are still in the process of building their data system technology capacity.** An examination of district capacity with respect to data systems needs to take into account the multiple types of systems containing data concerning students and other aspects of the education system (see Exhibit ES-1). Nearly all school districts have an electronic student information system providing real-time access to information such as enrollment and attendance. According to district survey respondents, the majority of districts (70 percent) have had this type of system for six or more years (see Exhibit ES-2). More recently, districts are acquiring other types of electronic data systems: 79 percent report having an assessment system that organizes and analyzes benchmark assessment data, 77 percent report having a data warehouse that provides access to current and historical data on students as well as data on other aspects of district functioning, and 64 percent report having an instructional or curriculum management system to support access to curriculum and instructional resources. 

**Exhibit ES-1. Types of Electronic Student Data Systems**

**Exhibit ES-2. Districts With Electronic Student Data Systems, by Type**

Exhibit reads: In 2007–08, more than 99 percent of districts reported having a student information system.

Source: 2007–district survey question 4.

The district survey found no significant relationships between district size or the proportion of a district’s students living in poverty and the likelihood of having a particular type of student data system. The few districts without any electronic data system (10 out of 427 districts in the survey sample) tended to be smaller in size than other districts in the survey sample but similar in terms of the percentage of students in poverty and the percentage of their schools that are in Title I or not making AYP.

**At this point in time, districts are looking for a way to effectively link their multiple data systems since there is no “single solution” data system and there is no simple recipe for effective system implementation.**Most districts have multiple, distinct data systems. The number of electronic data systems being used to support decisions about instruction in the case study districts ranged from three to seven. Although not a problem in principle, the use of multiple systems can be a problem in practice. On the district survey, over 60 percent of districts reported that lack of interoperability across data systems was a current barrier to expanded use of data-driven decision making. 

**Districts’ initial acquisition of data systems and use of data has been driven by accountability requirements.** The *Elementary and Secondary Education Act* requirements that states and districts report on the progress of every student subgroup toward academic proficiency and close the proficiency gap among student subgroups have both motivated districts to acquire data systems and shaped the nature of those systems. Districts are much more likely to have electronic systems with data such as student demographics and test scores than to have the ability to combine data from different types of systems or to link instructional resources to achievement data. Over 90 percent of the districts surveyed reported having electronically stored data on student demographics and attendance, student grades, student test scores on statewide assessments, and student course enrollment histories. In contrast, **less than half of districts have electronic data systems that allow them to link outcomes to processes as required for continuous improvement.**For example, only 42 percent of districts can generate data reports showing student performance linked to participation in specific instructional programs, and just over a third (38 percent) can execute queries concerning student performance linked to teacher characteristics. 

To support better decisions about instruction, data systems should make available data on the same student or group of students over time and support looking at the performance of students with different educational experiences (i.e., different teachers or instructional programs). Exhibit ES-3 shows the percentage of districts reporting that their systems include such tools for data use. 

**Exhibit ES-3. District Data System Query Capabilities in 2007–08**

Exhibit reads: In 2007–08, 83 percent of districts had electronic data systems that had the capacity to support queries about individual student histories over time. These districts serve approximately 88 percent of the nations’ public school students.

Note: Asterisks indicate that the proportion of districts that report having this query capability varies significantly by district size (* p < .05 and ** p < .01).

Source: 2007–08 district survey questions 7 and 8.

**District Supports for Data-driven Decision Making in Schools**

Both the survey and case study data suggest that districts are taking steps to improve the capacity of their schools to use data in decision making. 

**One of the most commonly reported district policies to encourage schools’ use of data is to incorporate this practice into school improvement planning.** Sixty-nine percent of districts reported requiring all or some of their schools to follow specific data-driven decision making practices in formulating their school improvement plans, and 65 percent of districts provide teachers with specific processes for how they should use data for instructional purposes. 

**The most common district strategies for building school capacity for using data are professional development activities, providing support positions for system implementation, and the development of tools for generating data and tools for acting on data.** Over 90 percent of districts responding to the survey reported that they have provided at least some school staff with training designed to enhance the school’s capacity to use data in ways that improve instruction (e.g., training principals or other school administrators on using the data system to analyze student achievement and to provide leadership for data-driven decision making in their school). But in many cases training has not been extended to all of the district’s schools. Training for teachers on how to use the data system to analyze student achievement or on how to use data to change their instructional practices are the two types of training least likely to have been given to every school (both provided to all their schools by 53 percent of districts). 

**Another common district support is providing technical experts in systems, networks or databases who can help school staff get access to data from electronic systems.** Eighty percent of districts say they have provided their schools with this kind of technical expertise, and 65 percent say they have made such technical expertise available for all of their schools. In contrast, making data analysis experts (sometimes called “data coaches”) available to school staff is one of the least common supports. Still, 50 percent of districts say they have done this for at least some of their schools, and 32 percent say that they have done this for all of their schools. 

Other supports cited by districts in the case study sample are: 

· Providing full- or part-time positions for staff who help teachers work with data, 

· Creating easy-to-read data “dashboards” to help teachers get information from data systems that are not user friendly, 

· Developing benchmark and formative assessments to provide teachers with more timely data to assess student progress and adjust instruction to meet student needs, and 

· Participating in a range of partnerships to support their efforts to implement data-driven decision making in their schools.

It is important to keep in mind that case study districts were selected for their leadership in data use. Survey responses from the national district survey suggest that many districts believe that they themselves need examples of good practice in order to guide their schools’ implementation of data-driven decision making. **The greatest perceived area of need among districts is for models of how to connect student data to instructional practice** (see Exhibit ES-4). Districts want examples of how to identify which practices work best for which students and how to adapt instructional strategies to meet the needs of individual students.

**Exhibit ES-4. District Perceptions of Needed Examples of Good Practice**

Exhibit reads: In 2007–08, 80 percent of districts reported that they had some or a great need for examples of good practice regarding the examination of data to identify which practices work best for which students. 

Source: 2007–08 district survey question 24.

**Use of Data Systems and Data-driven Decision Making in Schools**

Site visits suggest that in districts that are leaders in data-driven decision making, the **use of data in schools is encouraged not through extensive formal professional development but rather through ongoing support from colleagues and instructional or data coaches** who help teachers examine data for their students and develop instructional plans to meet student needs. Staff in case study schools described a higher level of support for data-driven decision making than did teacher respondents to the national teacher surveys administered in 2005 and 2007. As a group, the 36 case study schools appeared to be especially well-supported in the area of funding for time for teachers to meet together during the school day to work with data and the provision of school-based positions for coaching teachers on how to connect data to instructional strategies.****

**Even in districts that are actively promoting the use of data, however, school staff provided relatively few examples of teachers using data to diagnose areas in which they could improve the way they teach.**The most common school-level uses of data described by teachers and school leaders in the case study sample are school improvement planning, curriculum decisions, and placement or grouping of students for instruction or support services.****During school site visits, principals, coaches, and teachers were asked to describe specific examples of using data to make instructional decisions. From the 36 schools visited in 2007–08, 188 examples of using data to inform instruction were obtained. Exhibit ES-5 shows the different purposes for which school staff described using data and the number of distinct citations of each use. The data in Exhibit ES-5 suggest that in districts considered leaders in data-driven decision making, most schools are using data to develop goals for school improvement and to do curriculum planning. In contrast, school interviewees provided only eight examples of teachers using data to determine which aspects of their teaching are working well or poorly. 

**Exhibit ES-5. Uses of Data Described by Case Study Schools**

Exhibit reads: School improvement planning was the most common data use described by staff at the 36 case study schools.

Source: Case study schools 2007–08.

**In site visit schools, the use of data by teachers to improve their teaching practice emerged later than uses such as school improvement planning or student placement.**The relative frequency of reports of different uses of data within the 36 case study schools suggests that different uses emerge over time. Analysts developed three categories of school-level data-driven decisions as shown in Exhibit ES-6. The first category covers a range of accountability-driven uses of data, the second encompasses matching teaching content to standards or tests and giving students adequate time to master the content, and the third category involves using data to explore the relative effectiveness of different teaching methods or interventions.

**Exhibit ES-6. Categories of Data-driven Decision Making**

When the examples of data use described by case study school staff were classified in terms of the three data-use categories described in Exhibit ES-6, analysts found that most case study schools provided examples of Category 1 and Category 2 uses of data but less than half of all schools provided a Category 3 example. Specifically, as shown in Exhibit ES-7, 35 of 36 case study schools (97 percent) provided one or more Category 1 examples; 30 of 36 (83 percent) provided one or more Category 2 examples; and 17 of 36 (47 percent) provided one or more Category 3 examples. 

**ES-7. Percent of Case Study Schools Reporting Each Decision-making Category**

Exhibit reads: Among case study schools, 97 percent provided examples of Category 1 uses of data.

Source: Case study schools 2007–08.

**One of the strongest levers that districts can use to increase their schools’ use of data systems is to provide timely interim assessment data on those systems.**Eighteen of the 36 schools visited by researchers during school year 2007–08 had been visited a year earlier as part of 2006–07 data collection activities. Researchers analyzed site visit reports of examples of teacher and school leader data use to ascertain whether or not there were changes in data-use practices at those schools between the 2006–07 and 2007–08 school years. Schools in districts involved in implementing a system of districtwide interim assessments were more likely than other schools to show an increase in data use from year to year, and also provided the most striking examples of positive changes in teacher data use practices. In 13 of the 18 schools, there was evidence of an increase in data use over this one-year period; in four schools there was no indication of a change in the frequency or nature of data use. 

**Having a set of common assessments that everyone teaching the same content gives to their students at about the same time encourages teachers to sit down and share both their data and their teaching strategies.** When multiple teachers have all given the same recent assessment to their students, they can compare their results to identify strengths and weaknesses at the class level, something that is not possible if teachers assess different content at different times. Researchers found at least some teachers using common assessments and comparing their assessment data with each other as a way of comparing and reflecting on their practice in close to half of the case study schools.

**Actions that principals can take to encourage teacher use of data include designing and implementing regular activities involving the examination of student data and the establishment of an organizational climate of trust and mutual respect.**Principals encourage data use by setting an example through their own activities, designating all or part of teacher planning or professional development time as occasions for examining and reflecting on data, and communicating expectations around data use. In 25 of the 36 case study schools, researchers judged data use to be an important tool in the principal’s assumption of the role of instructional leader. Principals at 18 case study schools led whole-school or grade-level meetings dedicated to the analysis of data. Principals at several schools maintained schoolwide lists of students who were in danger of failing to attain state proficiency standards in order to keep on top of efforts being made to support these students during the year.  At three of the case study schools principals met with teachers individually on a regular basis to discuss their students’ needs and the teacher’s plan for the class.****

**School staffs’ perceptions of barriers to greater use of data include a sense of lack of time,****system usability issues, the perception that the data in the system are not useful, and district policies around curriculum coverage or pacing that prohibit modifying learning time to match student needs.**Nationally representative responses on the teacher survey (see Exhibit ES-8) suggest that teachers who have access to a data system view their colleagues as a resource for data use but have to do much of their work with data on their own time. 

**Exhibit ES-8. Teacher Perceptions of the SupportThey Receive for Data Use**

Exhibit reads: Among teacher survey respondents with access to an electronic data system, 71 percent agreed with the statement “I can turn to someone for help.”

Note: (R) denotes an item that is reverse coded; for these items a disagree response indicates more support for data use.

Source: NETTS teacher survey, 2007.

At case study schools, staff were much more likely to criticize the quality or timeliness of the tests for which data are available in systems than to criticize the usability of the system per se. **Teacher criticisms of the quality of the data available to them included delay issues, lack of alignment with standards, lack of alignment with the school’s instructional approach, and the fact that they received only cross-sectional data rather than longitudinal data for the same set of students over time.**

State achievement tests are typically a once-a-year spring event with results unavailable until the next fall when students have moved into a new grade level. Staff at many case study schools described looking at these data at the time they became available, often using them for school improvement planning and for placing students into classes or special services, as noted above. Individual teachers look at the prior spring’s scores for the students coming into their class to help do planning at the first of the year but soon find more recent and fine-grained information from classroom assessments and informal observations guiding their instructional decisions.

**Recommendations**

The surveys and case studies provide a portrait of system and organizational capacity for data use in schools and districts. A comparison of this portrait with the ideal of all schools using data systems to improve the educational experiences and outcomes for their students suggests a number of implications for policies and actions at the school, district, state, and national levels. The recommendations below are those of the study team.

**Recommendations for Schools**

· *Set clear expectations around the use of student data as the basis for decisions.*

· *Integrate collaborative exploration of data into existing structures for joint teacher planning and reflection on teaching.*

· *Provide a safe environment for teacher examination of their students’ performance.*

· *Support teachers in making the link between data and alternate instructional strategies.*

**Recommendations for Districts**

· *Think of data-driven decision making as an ongoing systemic process rather than a one-time event centered on the acquisition of a data system.*

· *Model decision making based on data and present decision-relevant data when announcing new policies.*

· *Train principals in how to integrate the use of data into school improvement planning and promote their teachers’ use of data for making instructional decisions.*

· *Integrate the use of data-driven decision-making practices with district initiatives for improving instruction in specific areas.*

· *Support time within the work week for teachers to meet with colleagues for planning, informal professional development, and data use.*

· *Make sure that the district has a data system that gives teachers data that is both timely and relevant to their instructional decisions.*

· *Provide resources and construct policies so that teachers have access to data relevant to the students they are teaching when and where they want it.*

**Recommendations for State and National Policy**

· *Complement efforts to improve state data systems with investments helping districts improve both their data systems and their organizational supports for using data to improve instruction.*

· *Improve the turnaround time for state assessment data so that schools receive student results in time to inform academic-year planning.*

· *Promote linkages between local data systems that contain interim assessment data and state systems with instructional resources geared to standards*. 

· *Encourage districts to invest in developing data literacy among district staff in all departments.*

· *Encourage or require school administrator preparation programs to incorporate assessment and data literacy concepts to foster continuous improvement activities informed by data as elements of school leadership training.*

· *Encourage or require teacher preparation programs to incorporate assessment concepts and the use of data for instructional decision making into their teaching methods courses* (science methods, language arts methods, and so on). 

· *Provide districts with good examples of practices that support the development of a data-use culture within schools.*

**1. Introduction and Approach**

Proponents of data-driven decision making call on educators to adopt a continuous-improvement perspective, with an emphasis on goal setting, measurement, and feedback loops so that they can reflect on their programs and processes, relate them to student outcomes, and make refinements suggested by the outcome data (CoSN 2004; Dattnow, Park, and Wohlstetter 2007; Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). For example, a 2009 forum of state and federal policymakers and education leaders, including U.S. Secretary of Education Arne Duncan, emphasized the need to use data systems for continuous improvement (Data Quality Campaign 2009b). But until fairly recently, the use of data to make decisions at the district, school and classroom levels has been the exception rather than the rule. The lack of systems providing user-friendly access to timely, relevant information was one major impediment (Wayman 2005). Another was a culture more accustomed to making decisions on the basis of educational philosophy or political necessity rather than data (Coburn and Talbert 2006; Coburn, Toure, and Yamashita 2009). 

In the last decade, however, several forces have converged to make data-driven decision making at all levels of the education system a priority. Improved data systems capable of tracking individual students’ progress from year to year have become available and have been implemented in an increasing number of states and districts (Data Quality Campaign 2008). Moreover, some districts are instituting data warehouses that allow them to combine data from different systems (for example, student achievement and teacher qualifications) in ways that allow investigating education quality issues (Wayman, Stringfield, and Yakimoski 2004). The *Elementary and Secondary Education Act (ESEA)*, as reauthorized in 2002, has instituted requirements for achievement data reporting by student subgroup, requiring many districts to obtain or upgrade their student data systems, and has made schools and districts responsible for student achievement. As districts and schools have looked for strategies to help raise achievement, the use of data to predict and enhance student performance has emerged as perhaps the dominant improvement strategy. Studies of district and school achievement have documented a relationship between active use of data and increases in achievement (Datnow, Park, and Wohlstetter 2007; Snipes, Doolittle and Herlihy 2002). 

To understand the role of data systems and the supports necessary for teachers to use data from any source (electronic and nonelectronic) to inform educational practice, the U.S. Department of Education’s Policy and Program Studies Service (PPSS) sponsored a national study of Education Data Systems and Decision Making. The study addressed a set of basic questions:

1. What kinds of systems are available to support district and school data-driven decision making? Within these systems, how prevalent are tools for generating and acting on data?

2. How prevalent are organizational supports for school use of data systems to inform instruction?

3. How are school staff using data systems? Do they know how to interpret student data? How is school staffs’ use of data systems and of data more broadly influencing instruction? 

This report describes findings from the study’s second round of data collection and analysis, conducted during the 2007–08 school year.

**Data Systems: A Prerequisite for Data-driven Decision Making**

For many years, local education agencies (LEAs) have developed and used multiple data systems for different purposes (for example, separate systems for finance data, personnel data, required accountability information for special education students, school lunch data, enrollment and attendance, assessment data). Historically, these data systems were so complex and poorly aligned that their use by school staff was not feasible (Wayman and Cho 2009). Moreover, the lack of persistent student and teacher identifiers in many of these systems made it impossible to obtain a longitudinal view of students’ history within the system or to follow students if they transferred schools. Advances in technology and recent policy emphasis on data use have resulted in much improved data infrastructures in many districts (Wayman and Cho 2009). 

**Beyond Data Systems: Need for a Systemic Approach**

Policymakers envision wide-ranging educational data systems that collect, analyze, and use relevant data at every level—from the U.S. Department of Education to the individual teacher making instructional decisions for his or her classroom. The mere presence of a student data system within a district is a prerequisite but clearly insufficient to bring about change in the way that decisions impacting instruction get made. 

Within a school district, many offices (or departments) as well as its schools need to participate in order to bring about such a fundamental change in educational decision making (Datnow, Park and Wohlstetter 2007; Wayman, Cho, and Johnston 2007). The district needs to fund and acquire (or build) a data system capable of storing and manipulating the kinds of information needed for educational decisions. Provisions need to be made by the district to ensure that intended data system users (district staff, school leaders, instructional coaches, and teachers) have ready access to the data in a form they can comprehend and manipulate. Assessment and curriculum departments need to make sure that there is up-to-date information from assessments linked to the local curriculum available for teacher use. Professional development offerings need to include opportunities to learn how to make instructional decisions based on data and, if applicable, how to get data out of the system and analyze it to fit one’s needs. 

**Data-driven Decision-making Framework**

Exhibit 1-1 shows the stages in a data-driven continuous-improvement process: plan, implement, assess, analyze data, and reflect (as a precursor to more planning and a refined implementation). As the graphic suggests, components of data-driven decision making are part of a continuous cycle. The starting point may vary, and there is no fixed end point.

**Exhibit 1-1. Conceptual Framework for Data-driven Decision Making**

A major cultural change is required if educators are to make the continuous-improvement perspective and the processes of data-driven decision making part of the way in which they function. Such a change will not occur without leadership, effort, and well-designed supports. The bottom portion of Exhibit 1-1 identifies six major types of prerequisites and supports for data-driven decision making that are part of the study’s conceptual framework: 

· State, district, and school data systems 

· Leadership for educational improvement and the use of data 

· Tools for generating actionable data 

· Social structures and supported time for analyzing and interpreting data 

· Professional development and technical support for data interpretation 

· Tools for acting on data 

**Data Systems**

*ESEA* has stimulated an unprecedented level of state activity aimed at improving education data systems. Federal requirements for reporting schools’ year-to-year progress in raising the percent proficient overall and for specific student categories have led to an examination of information system adequacy and the adoption or development of new software systems in many states and districts. For example, in 2005, 36 states used unique student identification numbers statewide so that students could be followed if they changed districts. In 2008, 48 states did so (Data Quality Campaign 2008). As of 2008, state data systems typically include student enrollment information, basic demographic data, special program designation (if applicable), and scores on state-mandated achievement tests (in most cases, an annual spring testing in language arts and mathematics and often a proficiency or “exit” examination required for a high school diploma). 

**At the local level, it is district systems that are most likely to be used by teachers** (U.S. Department of Education 2009). These systems typically include student scores on state-mandated tests, which are obtained from the state or from the state-designated vendor. District systems are also likely to include student contact and demographic information, scores on district tests, attendance, grades, disciplinary infractions, and course enrollment.

**Leadership for Education Improvement and Use of Data**

Pioneering efforts to promote data-driven decision making within districts and schools have found that the active promotion of the effort on the part of the superintendent or principal is vital (Marsh, Pane, and Hamilton 2006; Supovitz and Klein 2003; Wayman and Stringfield 2006; Young 2006). District and school leaders issue the “call to arms” for improving education and using data as a tool to bring about that improvement. Typically, they play a major role in framing targets for educational improvement, setting expectations for staff participation in data-driven decision making, and making resources, such as supported time, available to support the enterprise.

**Tools for Generating Actionable Data**

Increasingly, student achievement data are available at the school level in a form that can be disaggregated by student category (ethnicity, free or reduced-price lunch status, special education status, etc.). Software systems to support data-driven decision making all generate standard student achievement reports, and many also produce custom reports for user-designated student groups (an important feature for school staff who want to examine the effects of locally developed services for specific student groups). Research indicates, however, that often school staff do not find the kinds of data these systems provide particularly useful for guiding instruction (Brunner et al. 2005). School staff are frustrated by the fact that the data available to them are typically performance on a state achievement test taken six or more months earlier. Teachers want up-to-date information on their current group of students, not the students in the same grade level the prior year. They also want a greater level of detail concerning individual students’ strengths and weaknesses than they can get from standardized test scores (Mandinach et al. 2006; Thorn 2002). Although far less common than systems that provide data from prior testing, there are examples of systems that produce additional information for decision making through tools such as formative assessments that students may take online. In addition, some system designers are working on educational information systems that will integrate data on a broad range of transactions, such as daily school attendance, grades, and even library book checkouts, with the ultimate goal of automatically recording each interaction a student has with the school and student’s assessment and program participation data.

**Social Structures and Supported Time for Analyzing and Interpreting Data**

The most sophisticated data warehouse in the world will have no effect on instruction if no one has—or takes—the time to look at the data, reflect on them, and draw inferences for instructional planning. Given that time is one of the most basic resources in any organization, there need to be strong expectations that administrators will provide time to teachers, and educators will take the time to examine data and use it to guide improvements in their programs and practices. Such expectations have not been business-as-usual in most schools and districts. Case studies of schools active in data-driven decision making suggest that organizational structures that include supported time for reviewing and discussing data in small groups greatly increase the likelihood that the examination of data will be conducted and will lead to well-informed decisions (Copland 2003; Datnow, Park, and Wholstetter 2007; Wayman, Cho, and Johnston 2007; Wayman and Stringfield 2006).

**Professional Development and Technical Support for Data Interpretation**

Teacher training generally has not included data analysis skills or data-driven decision-making processes in the past (Mandinach et al. 2005; Massell and Goertz 2002). Few administrators have this kind of training either. Moreover, the measurement issues affecting the interpretation of assessment data—and certainly the comparison of data across years, schools, or different student subgroups—are complicated. Data misinterpretation is a real concern (Confrey and Makar 2005). For this reason, districts and schools are devoting increasing amounts of professional development time to the topic of data-driven decision making. Many argue that the practice of bringing teachers together to examine data on their students and relate those data to their practices is a valuable form of professional development in its own right (Feldman and Tung 2001; Supovitz and Klein 2003; Wayman, Cho, and Johnston 2007). 

**Tools for Acting on Data**

The examination of data is not an end in itself but rather a means to improving decisions about instructional programs, placements and methods. Once data have been analyzed to reveal weaknesses in certain parts of the education program or to identify students who have not attained the expected level of proficiency, educators need to reflect on the aspects of their processes that may contribute to less-than-desired outcomes and to generate options for addressing the identified weaknesses. Some of the data-driven decision-making systems incorporate resources that teachers can use in planning what to do differently. These resources are typically organized around state content standards and may include lesson plans, instructional materials, or descriptions of best practices (Palaich, Good, and van der Ploeg 2004). Resources for differentiated instruction can help teachers adapt their instructional approach to students with differing strengths and weaknesses.

**Data Sources for the Report**

Findings from this report are drawn from both survey and case study data. The primary data set for this report consists of responses of a sample of 529 districts to a survey administered between October 2007 and February 2008. The sampling plan for the district survey was developed with the primary goal of providing a nationally representative sample of districts to assess the prevalence of district support for data-driven decision making. A secondary goal was to provide numbers of districts adequate to support analyses focused on subgroups of districts. Of the 529 districts in the final sample of the Study of Education Data Systems and Decision Making, 427 responded for a response rate of 81 percent. (Additional information on sampling is provided in Appendix A.)

The case study districts were purposefully selected to include districts that have been active in using student data to guide instruction (as a result, these are not typical districts). By focusing fieldwork on districts in which many teachers could be expected to be actively looking at student data, the study team increased the likelihood of seeing the effects of data use on practice, compared with a sample of schools drawn at random. For the 2007–08 case study districts, the site selection process was two-staged. The first stage involved the selection of six of the nine 2006–07 case study districts to be visited for a second round of data collection. The research team decided that these districts warranted a second visit because they had been in the process of implementing new data systems or activities to support the use of student data at the school level during the first round of data collection in 2006–07 and could provide a longitudinal perspective on implementation activities. The second stage involved identifying an additional group of six districts that have been active in data-driven decision making. These districts were drawn from the pool of districts that remained after the initial selection of 10 districts in 2006, supplemented by additional districts identified as active data users. For each district visited, respondents included key staff involved in the district’s data-informed decision-making activities (e.g., chief information officers, directors of curriculum and instruction, directors of research and evaluation, directors of accountability, directors of professional development). Within each school, the principal, an instructional or data coach (if applicable), and six teachers were interviewed.

A second survey data set from the U.S. Department of Education’s National Educational Technology Trends Study (NETTS) has also been included to provide a national picture of teacher perspectives on data use. These data consist of responses of a random sample of K–12 teachers to a survey administered to 2,509 teachers in spring 2007. The teachers were clustered in schools sampled from districts participating in the NETTS study. Teachers were asked to report on activities during the 2006–07 school year.

**Contents of the Report**

This final report builds on findings from the interim report (U.S. Department of Education 2009) that described the types of data available to school staff, how school staff use electronic data systems, school practices with respect to data-driven decision making, and the supports and challenges for school use of student data in planning and implementing instruction. This report presents district survey data that provides a national picture of data-driven decision-making practices and additional case study data. It also incorporates some of the NETTS teacher survey data presented previously in the brief entitled *Teachers’ Use of Student Data Systems to Improve Instruction*–*2005 to 2007* (U.S. Department of Education 2007).

Earlier study reports have documented a dramatic increase in the proportion of teachers with access to a student data system between 2005 and 2007. They also described school practices with respect to data use and the challenges that are part of student data system implementation. In the case study districts visited during the 2006–07 school year, school leaders demonstrated their support for school’s use of data by purchasing data systems, modeling data use, and providing school-based support positions. Teachers who use data from a student data system do so not only on their own but also in collaboration with colleagues. But both teachers and district staff members express concerns about teachers’ ability to understand data—many appear to lack data literacy skills.

The data in this final report are primarily descriptive; they do not address the effects of data-driven decision making on student outcomes. At the same time, the findings go beyond most prior research that has tended to focus on case studies of individual districts and schools to provide a broader picture of data-driven decision-making implementation efforts in a dozen districts across the country. Study findings lay a foundation on which to build future research efforts and will assist policymakers in understanding how data are being used at the local level, the conditions affecting use (both positively and negatively), and other issues that arise during implementation. 

The remainder of this report is divided into four chapters. Chapter 2 describes the data systems that districts are employing to support data collection, storage, retrieval, and analysis; the type of data that districts maintain electronically and the ways that districts use data to focus on accountability and instructional improvement. Chapter 3 explores district efforts to promote and support the use of student data and data systems within their schools, including the challenges this effort entails and the strategies that districts are using to overcome obstacles. Chapter 4 focuses on school-level implementation of data systems and data-driven decision making. Chapter 5 discusses cross-cutting themes and draws implications for policy in this area.

**2. District Data System Features and District Use of Data for Decision Making**

Fundamental to the concept of an information infrastructure to support data-driven decision making is the ability to share data across levels of the system (Wayman, Stringfield, and Yakimoski 2004). A data system infrastructure is composed primarily of hardware and software but also relies on people to gather data, extract the data and use it. Typically, district data systems are made up of multiple elements that have a unique role or set of attributes, and the key to maximizing the utility of these systems is the ability to integrate and share data files. To get a better picture of the information technology available to districts, district survey respondents, along with case study sites, were asked by the research team to provide information on the electronic student data system or systems that are driving instructional improvement in their districts. The study focused on electronic systems that contain data on students but also sought information on other electronic systems containing data that might be relevant to instruction (for example, data on the professional development received by teachers). The major types of electronic student data systems in widespread use are defined in Exhibit 2-1 (Wayman 2005) and were used to organize information drawn from the district survey regarding the kinds of data systems available to support district and school data-driven decision making (a copy of the survey can be found in Appendix B).  

**Exhibit 2-1. Types of Electronic Student Data Systems**

Two themes emerged from district survey responses and analysis of the case study data:

· Currently, districts are looking for a way to effectively link their multiple data systems since there is no “single solution” data system and there is no simple recipe for effective implementation.

· Districts’ initial acquisition of data systems and use of data has been driven by accountability requirements, most recently the *ESEA* requirements that states and districts report on the progress of student subgroups in aggregate toward achieving academic proficiency by 2014 and to close the proficiency gap among student subgroups.

**A Profile of District Data Systems**

To provide an overview or snapshot of the electronic data systems that U.S. school districts employ and those systems’ capabilities, responses from multiple survey items were combined into a handful of key data system elements to assess the extent to which district data systems are providing information to support better decisions about instruction (see Data Quality Campaign 2008 for a similar approach to depicting state data systems). Data systems with a greater variety of data and tools that support looking at the performance of students with different educational experiences are much better equipped to support the kind of inquiry necessary for continuous improvement activities—the ongoing examination of student data to assess the effectiveness of education activities and to refine programs and practices to improve student outcomes.

Analysts organized survey responses to examine ten elements of quality for district data systems (Exhibit 2-2):

· Student information system

· Data linkages

· Instructional or curriculum management system

· Assessment system

· Student performance data to measure academic growth

· Student-level enrollment, demographic and program participation information

· Teacher-level data

· Student-level graduation, post-graduation and dropout data

· College readiness data

· Data quality assurance

The profile provides information on the percentage of districts with all of the subelements present for a particular element (i.e., districts responded positively to all of the survey items comprising that element). It was fairly unusual for districts to have all of the subelements. While 65 percent of districts have all the subelements for a student information system only 44 percent have all the subelements for student performance data to measure academic growth. 

Each element score in Exhibit 2-2 (right-hand column) represents the median number of subelements present across districts. The number in parentheses represents the percentage of districts having at least the median number of subelements in place. So, for example, when it comes to linking data, 64 percent of districts have the capacity to conduct two out of the five subelement linkages that make up this element (primarily student performance linked to AYP subgroups and linked to specific teachers). Seventy-four percent of districts report having one out of the two subelements under teacher-level data (primarily teacher qualifications). 

**Exhibit 2-2. Profile of District Data System Elements in 2007–08**

Exhibit 2-2 continues on next page

**Exhibit 2-2. Profile of District Data System Elements in 2007–08 (continued)**

Exhibit reads: In 2007–08, 65 percent of districts with electronic data systems reported having all three of the subelements for student information systems (whereas 100 percent reported having the first subelement). The median number of subelements present is three. 

Source: 2007–08 district survey questions 4, 5, 7, 8, 12-14.

**The profile suggests that district access to robust data systems is still limited.**Such limitations may impede district achievement of three of the education goals outlined in the *American Recovery and Reinvestment Act (ARRA) of 2009*: (a) establishing pre-K to college and career data systems that track student progress, (b) providing and assessing effective interventions for the lowest-performing schools, and (c) assessing teacher effectiveness and the equitable distribution of qualified teachers for all students (particularly students who are most in need).

**As was true in prior years, districts are maintaining more administrative data than data that are targeted to individual student performance** (U.S. Department of Education 2009). Fifty-nine percent of districts have all eight of the subelements under student-level enrollment, demographic and program participation information (element 6), and 65 percent have all three subelements associated with a student information system (element 1), whereas only 44 percent have all four subelements for student performance data to measure academic growth (element 5). 

An even larger impediment appears to be districts’ inability to link data across systems (element 2), with just 15 percent of districts capable of all of the linkages described by the subelements. **The fact that only 15 percent of districts report having all five subelements related to data linkages points to the challenges in achieving the efficiencies that technology was envisioned to offer.** Lack of interoperability between systems complicates data analysis, and differences in system interfaces increase training requirements for district and school staff. As discussed later in this chapter, districts are using multiple data systems and over 60 percent reported that lack of interoperability across their data systems was a current barrier to expanded use of data driven decision making.

Data quality is of particular concern to policymakers given their focus on accountability, but the issue of data quality is part of a broader context. The district data system profile indicates that the average district carries out two of the three activities associated with assessing data quality (element 10). About two-thirds of districts (65 percent) reported that 90 percent or greater of the data captured by their student data system or other system that drives instructional improvement is accurate. Case study and survey respondents indicated that data with the biggest accuracy problems tend to be items such as student demographics that rely on self-report or are provided by parents or else data that often changes frequently (e.g., student schedules, students receiving tutoring outside the school day). The case studies suggested that the concept of data accuracy is subject to interpretation and is more nuanced than the statistic based on survey responses may suggest. Data for accountability purposes (e.g., state test scores) have consequences attached, and therefore accuracy is critical. Data used for ongoing adjustments to instructional practice tend to be generated more frequently, with any one data entry having less serious consequences, justifying a lower level of effort for quality control. 

The remainder of this chapter provides a more indepth look at the components that make up district data systems, where these components come from, the types of data maintained in district data systems, the tools available to carry out more sophisticated analyses, and how districts use their data systems.

**District Data System Features**

As discussed in the introduction, the standards movement and state and federal accountability requirements have placed a greater emphasis on using data to monitor progress. Advances in technology have made it possible to link multiple datasets, to track change over time, and engage in much more sophisticated analysis activities. But there is often a gap between what is technically possible and what districts are equipped to do. One of the study’s goals was to ascertain the extent to which districts have access to electronic data systems with the capacity to realize these benefits. To this end, district survey respondents were asked to report on the elements of their current student data system. The application of sampling weights to the district survey data produced a nationally representative portrait of the educational data systems used by districts in school year 2007–08.

**Very few districts responding to the survey (10 out of 427) indicated that they did not have *any* kind of electronic student data system or tools to enhance educational decision making.** The ten districts without electronic data systems tend to be smaller than other districts in the survey sample. They were similar to the survey sample in terms of the percentage of students in poverty and the percentage of their schools that are Title I or not making AYP.

The remaining 417 districts all indicated that they had an electronic student information system. More than three-quarters of these districts indicated that they had an online assessment system, and 77 percent reported having a data warehouse. A somewhat smaller proportion, 64 percent, reported having an instructional or curriculum management system. Analysis of whether district size or the proportion of a district’s students living in poverty affected the likelihood of having a particular type of student data system found a statistically significant relationship between district size and the likelihood of having an online assessment system. While 74 percent of small districts reported having an online assessment system, 91 percent of large districts reported having such a system. There was no statistically significant association between district poverty level and the type of student data system a district employs.

Across districts, about half (48 percent) reported having one to three electronic data systems and another third (36 percent) reported an average of four systems. Large districts reported a significantly greater mean number of data systems (4.1) when compared to smaller districts (3.5). 

Exhibit 2-3 shows the length of time for which districts reported having had various types of electronic student data systems. For Exhibit 2-3 and the remainder of this report, the percentages reported will be for districts indicating that they have an electronic student data system unless otherwise specified. The data suggest that student information systems are not only nearly universal but have been around for six or more years in over two-thirds of districts. None of the other system types is this widespread or has this long a history. 

**Exhibit 2-3. District History With Student Data Systems, by Type**

Exhibit reads: Among districts with a student data system, all reported having a student information system in 2007–08; 3 percent have had this type of system for less than a year, 6 percent for one to two years, 21 percent for three to five years, and 70 percent for six or more years.

Source: 2007–08 district survey question 4.

**The survey data also suggest that districts are in the process of building their technology capacity.** Districts are more likely to have electronic access to data such as student demographics, attendance, grades and test scores than they are to have the ability to combine data from different types of systems or to be able to link assessments and instructional resources to achievement data. As illustrated in Exhibit 2-2, only 15 percent of districts have systems that can perform all five data linkages. Sixty-four percent of districts report that one of the barriers to spreading data-driven decision-making practices throughout their district is information located in multiple databases and not linked (barriers will be discussed further in the next chapter). An administrator in one of the case study districts stated: “We have a lot of data. One of the problems is that we have a lot of data that is in too many places. And that’s been a complaint that you have to go to [system] X, Y, Z [to get data].”

The case study districts, identified for their high data use, maintained three to seven data systems each. Districts had multiple systems, even of the same type, because of ongoing technology demands that challenged system capacity or systems that did not perform as expected, requiring districts to add or refine data system components. About half of the case study districts were in the process of adding new system components that they did not have previously. For example, three districts had just added a new online assessment system and another had added a Web-based system that makes student grades and schedules accessible to teachers and parents. Eight of the districts were upgrading their current system components (e.g., acquiring the next generation of their current student information system) or enhancing the features of current system components (e.g., new reporting features or data elements). One of these districts was acquiring a second data warehouse because the current data warehouse had limited reporting features that could not be expanded.  

**The case study districts also illustrate that data systems that save teachers’ time and provide them with information that they can act on immediately are uncommon.** Districts have spent years trying to develop or acquire systems that support instructional decisions and not just accountability and enhanced data access. The stories of two large districts provide examples of the challenges faced in effectively integrating multiple systems to meet data needs (see Exhibit 2-4).

**Exhibit 2-4. District Implementation of Student Data Systems**

**System Sources**

Information in the first round of case study data collection (U.S. Department of Education 2009) indicated that most of the data available to educators come from local data systems. Given the considerable effort that is being put into building data systems at the state level (e.g., $265 million investment in state longitudinal data systems between 2005 and 2009 to improve the management and use of education data), the district survey gathered information on the source of data systems that districts use to guide their data-driven decision making.**Districts report that they rely primarily on data systems they purchase or develop themselves for the purposes of instructional improvement.** Recent federal resource investments made in improving the quality of state data systems are showing promise in increasing the capacity of these systems, but two years into the grant cycle, they did not as yet appear to have influenced data use at the local level. 

In 2008, the Data Quality Campaign (DQC), an organization focused on improving state longitudinal data systems, partnered with APQC Education to explore how states can support districts and data-driven decision making at all levels. Their survey of districts suggested that before the potential from investments in state data systems are realized, “[T]he cultural and technical differences that exist between state and district data systems must be addressed” (Data Quality Campaign 2009a, pg. 6). That is, there is currently a misalignment between the types of data that districts feel are key for improving student achievement and the types of data that are being requested by the state for accountability purposes. Districts need student-level information to inform instruction and what is sent to the state is aggregated student achievement data, attendance, and student counts. When the state sends achievement results to the districts, the time lag in receiving these results means that teachers are not able to use the information to inform instruction (responses similar to those of our case study districts). Half of the districts in the DQC/APQC benchmarking study reported that they have minimal communication or collaboration with their state education agencies (SEAs) on state technology planning, district technology planning or data training opportunities. Conversely, when states do provide training or conferences on technology and data issues, districts cite this type of collaboration as valuable.

Responses to our district survey administered in 2007–08 indicate that most (88 percent) of the student information systems used by districts were commercially developed. There are some well-publicized cases of large districts developing their own data systems (e.g., Broward County, Fairfax County, San Diego, Tucson), but this choice is not representative of districts as a whole. Among the other types of data systems used by districts to shape instruction, commercial systems are also dominant but use of state systems is more common for these other systems than it is for student information systems. Of those districts with a data warehouse, a majority (56 percent) of data warehouses were obtained from commercial sources, but almost a fourth (23 percent) of districts report using a data warehouse supplied by their state. Among districts using an instructional or curriculum management system, 42 percent report obtaining it from a commercial vendor, 18 percent developed it locally, and 14 percent obtained it from their state. The pattern was similar for online assessment systems (47 percent from commercial vendors, 17 percent state developed, 12 percent locally developed). Analysis of the specific systems named by survey respondents who said they had a system from their state found that many of these systems are commercially developed (e.g., SASI, Edusoft, Cognos) and as such are better thought of as “state supplied” rather than “state developed” systems. (A list of the most frequently identified systems is provided in Appendix C, Exhibit C-2.) 

Analyses were run to explore the relationship between district size and poverty level and the source of a district’s student data systems. High-poverty districts were more likely than other districts to be using a student information system designed locally or by their state. There was no relationship between district poverty level and the source of other kinds of student data systems, and there was no relationship between district size and the source of any of the four types of data systems.  

Among the case study districts, commercial systems predominated, but there were six districts (primarily large) that developed their own systems that were sometimes used in conjunction with commercial systems. Half of the case study districts also used state data systems or systems made available to them by their state. For example, through Reading First funding, one state is contracting with a commercial software company to provide teachers with immediate diagnostic results—data are entered by the teachers and reports are immediately generated that indicate student mastery by each skill area. One of the districts noted strong support from their state in making data available to teachers: “The state department of education is forward thinking. They take data we give them and they are creating tools for teachers. [They are thinking] how can we more frequently give them [teachers] data?” A medium-sized district indicated that they have been part of a cooperative zone of seven neighboring districts created by the state to learn how to use data generally as well as data from the statewide electronic information system. The regional cooperatives contribute to the development of data tools as well as professional development for district and school staff. Conversely, another medium-sized district commented that it did not have the resources to develop a data system to meet its needs (it is struggling with commercial systems that have not performed as hoped) and wished that the state were able to provide more technology resources. Currently, the SEA is piloting a statewide system, but this district did not participate in the competitive grant competition held by the SEA because it had already invested heavily in its own data system.

**Types of Information Maintained Electronically**

Data is an essential element in any education reform or inquiry process (Datnow, Park and Wholstetter 2007; Wayman and Cho 2009). Some researchers have likened data use to a road trip (e.g., Love 2002): without data it is like driving a car with no gauges or windows and without a map—you cannot tell how much gas you have, how fast you are traveling, where you are, and whether you are even headed in the right direction. To get a finer-grained picture of the kinds of data that districts maintain and that are available to support analysis activities, survey respondents were asked about the availability of specific types of information. The results are shown in Exhibit 2-5. 

**The types of data maintained in district data systems did not change between 2006 and 2008.** Over 90 percent of districts reported having electronic data on (a) student demographics and student attendance (both 98 percent of districts), (b) student grades (95 percent), (c) student test scores on statewide assessments and differential codes for students no longer enrolled (both 93 percent), and (d) student course enrollment histories (92 percent). These percentages are similar to those reported on the 2006–07 NETTS district survey (U.S. Department of Education 2008) and are consistent with districts’ near-universal reports that they have student information systems. 

Fewer districts maintain electronic records of district-administered assessments (72 percent) or of school-administered assessments (52 percent) which could provide more detailed information on student performance relative to content standards during the school year while there is still time to take corrective action where needed. Less than half of districts (47 percent) report keeping electronic data on the professional development taken by their teachers. Districts do maintain electronic records of students’ graduation status (90 percent), but fewer districts keep electronic data on other measures of their system’s output. Just over half (57 percent) keep electronic data on students’ scores on college entrance or Advanced Placement examinations, and only about a third (34 percent) have electronic records of their students’ status after graduation (e.g., whether attending college, working). 

**Exhibit 2-5. Types of Information That Districts Maintained Electronically in 2007–08**

Exhibit reads: In 2007–08, 98 percent of districts reported that they stored student demographic data in electronic form and 90 percent had stored this type of data in the same format for three years or longer (representing 88 percent of all districts, with or without an electronic data system).

Source: 2007–08 district survey question 5.

**Access to longitudinal data has also remained constant over time.** To examine trends or to investigate the effectiveness of new programs and policies, districts need to be able to track a consistent set of measures over multiple years. District survey respondents were asked for each type of data that they store electronically, whether they had three years or more of longitudinal data stored in the same format. Their responses are shown in the right-hand columns of Exhibit 2-5. The pattern of responses overall suggests that districts have longitudinal information for the kinds of data most commonly found in student information systems (e.g., student demographics, course enrollment, attendance, grades) but that other types of information (e.g., scores on district-administered assessments, teacher professional development, student status after graduation), even if available for the current student cohort, were typically not available in district systems for three years or more at the time of the survey in 2007–08. 

**System Capabilities and Tools**

The utility of a student data system depends not just on the types of information it contains but also on the system’s capability to support data requests or “queries” related to issues of practical importance for education decision makers. The ability to query different datasets allows teachers and administrators to go beyond standardized reports that have a set format and cannot be manipulated or altered. District survey respondents were asked to indicate their systems’ capability to support key query types. The results are shown in Exhibit 2-6. 

**While districts have the capacity to conduct some types of inquiry, few have electronic data systems that allow them to link outcomes to processes as required for continuous improvement.** Roughly three-quarters (76 percent) of all districts report that their systems have drill-down capabilities and 83 percent report that they can generate longitudinal student histories of information such as schools attended or grade point average. Two-thirds (66 percent) can generate data reports showing student performance by Adequate Yearly Progress (AYP) student subgroups. On the other hand, fewer districts have electronic data systems that support inquiry into the factors that districts can actually influence to try to raise student achievement. Just 42 percent of districts can generate data reports showing student performance linked to participation in specific instructional programs; just over a third (38 percent) can execute queries concerning student performance linked to teacher characteristics; and just over a fourth (27 percent) can generate data reports linking school performance to finance data. These limitations were also evident regarding other specific features or tools of district electronic systems for generating and organizing data to support instructional improvement (e.g., online assessments, links to curricular materials). District responses concerning these capabilities are shown in Exhibit 2-7.

**Exhibit 2-6. District Data System Query Capabilities in 2007–08**

Exhibit reads: In 2007–08, 83 percent of districts had electronic data systems that had the capacity to support queries about individual student histories over time. These districts serve approximately 88 percent of the nations’ public school students.

Note: Asterisks indicate statistically significant differences in percent of districts reporting this query capability by district size (small, medium, large), **p* <. 05 and ***p* < .01.

Source: 2007–08 district survey questions 7 and 8.

**Exhibit 2-7. District Data System Features and Tools in 2007–08**

Exhibit reads: In 2007–08, 95 percent of districts had electronic data systems that could capture transactional data. These districts serve approximately 94 percent of the nations’ public school students.

Source: 2007–08 district survey question 8.

**Districts’ Use of Data Systems**

In addition to asking district respondents to describe the capabilities of their data systems, the survey sought information on the various purposes for which districts use their electronic data systems. Exhibit 2-8 displays their responses. The survey reports suggest that **districts’ use of data systems is closely linked to their efforts to track student achievement for accountability purposes.**Over 90 percent of districts use their data system at least once a year for the purposes of (a) analyzing student achievement by grade-level, districtwide or by school; (b) analyzing student achievement data over time; (c) examining achievement gaps among student subgroups; (d) tracking individual school performance; and (e) meeting accountability reporting requirements. The only comparably widespread use of data systems not involving achievement test scores is the tracking of student attendance that is used to generate school funding. The picture is quite different when it comes to how school staff use data, which will be discussed in Chapter 4.

Some of the purposes for which districts use system data are primarily once-a-year activities,as might be expected. These include activities associated with accountability (such as examining achievement gaps among student subgroups, tracking achievement by school, tracking graduation rates by school, meeting accountability reporting requirements, tracking teacher qualifications); instruction-related activities (such as identifying promising instructional programs, examining resource allocations); and other evaluative activities (such as examining district or school climate data; evaluating principal performance). District uses of system data that are most likely to occur more frequently (from twice a year to monthly or more) include monitoring student attendance and informing parents about student progress, and instruction-related activities such as tracking measures of student progress (e.g., benchmark or diagnostic tests), informing teachers about individual student’s instructional needs, and deciding what professional development to offer. 

**Exhibit 2-8. Districts Using Data Systems for Selected Purposes**

Exhibit reads: In 2007–08, 96 percent of districts used their electronic data systems at least once a year to track school performance.

Note: Asterisks indicate the proportion of districts carrying out a particular activity vary significantly by district size (*p* < .05 to *p* < .001); see Appendix C, Exhibit C-6a. 

Source: 2007–08 district survey questions 18 and 19. 

The case study districts not only used data to meet accountability requirements but also engaged in data analysis activities that influenced a wide range of instructional and administrative decisions.

· **Inform instruction.** District 10 is a large urban district that had gathered both quantitative and qualitative data that showed that writing instruction was lacking in their schools. After discussions with principals and teachers, the district office recruited a cohort of 15 schools to pilot a new writing program. Principals from these schools are observing a writing program sponsored by Columbia University. Each principal has agreed to train teachers using this approach to see what effect it will have on writing scores. As part of the program, a coach from the Columbia program visits the schools so that teachers can observe the coach conducting instruction; the coach also serves as a “critical friend.” The district has hired a specialist to conduct monthly meetings with school-level coaches to help them design professional development for their schools around the new writing program. 

· **Inform student placement.** The executive director for instructional services from a case study district that uses data as part of its continuous improvement efforts noted, “[I]n looking at middle school data we noticed that African American students weren’t taking Algebra in middle school at nearly the rates of their peers. Why not? Turns out the criteria for eligibility were teacher recommendations.” The district decided to replace the recommendation system with an objective math skills inventory. They gave the test to all seventh- and eighth-grade students starting in 2006–07, created a cut point that determined math class assignments, and continued to collect teacher recommendations. “Turns out that 200 students we assigned to Algebra [using this approach] would not have been recommended by teachers. We increased the African American and Latino enrollment. Since then, we’ve tracked three semesters of grades for Algebra and geometry. What’s really great is we’ve seen an increase in student performance (10 percent). This finding was consistent across all middle school sites.” 

Another case study district uses its academic data browser to identify students for intervention and retention. Prior district research had shown that if students had not reached proficiency in reading and math by third grade they would not succeed in school. As part of the district’s retention policies, all elementary schools must now assess student proficiency levels by a certain point and the electronic data system helps teachers analyze data to determine when to administer the district benchmark assessments.

· **Guide program evaluation.** Over the past two years, this large case study district has been looking at research from other districts and states to inform its transition from half-day to full-day kindergarten. The process began with a few schools as the district gathered data on results and conducted ongoing conversations with kindergarten teachers. According to the executive director for instructional services: “We look at the data they’re collecting, from district assessments, DRA [Diagnostic Reading Assessment], and samples of student work. We’ve met with first-grade teachers too. All the while looking at the data to see where kids are.” As a result, the district was able to create a full-day kindergarten program that met the needs of all students. This approach is representative of how the district prioritizes its allocation of resources, as described by the assistant superintendent: “When a federal or state initiative comes forward, related to instruction, assessment, professional development, what have you, we look at the data. [We] do a thorough field test, aggregate groups, subgroups. We can tell if that resource, practice, whatever is working for the largest number of children.” 

· **Evaluate teachers and principals.** As part of its results-based approach to decision making, this case study district requires both schools and teachers to develop instructional goals and plans to meet those goals. School staff review their goals in schoolwide meetings held a few days after each school’s benchmark data are received and then discuss the results further in team meetings to make adjustments in instruction and professional development plans. Teachers set individual and team goals and are evaluated based on their students’ performance on benchmark and state assessments. The aim of district leaders is to make this into a process for professional self-reflection and growth for both teachers and principals.

· **Inform professional development.** The director of assessment in one of the case study districts is collecting data on professional development for the 500 plus classes offered to certified and classified staff, including union programs for administrative staff, transportation staff, and custodians: “[The] purpose is to make sure employees are building skills that will help them move in district if they wish, keep them happy, keep them here.”

**Variations Across Districts**

**District survey data indicate that the availability of electronic student data systems is almost universal among districts, but districts are at varying stages of building their technology capacity.**While there were few differences identified in the composition of district data systems (i.e., the type and number of data systems), analyses of survey responses uncovered variations in system query capabilities and how districts use their data systems. It appears that capacity is positively related to district size, with larger districts having systems that support a wider range of analysis activities, and, as observed in the case study districts, these larger districts also have greater resources to acquire or develop data systems to meet their needs and to employ staff who have the expertise to conduct analysis activities. An administrator in one of these large districts noted that they have staff to customize reports for their schools: “We can do report writing at the district level [write the program scripts], but we’re an urban district. We have enough people where we can hire programmers whereas in a little bitty school district, they don’t.”

**In general, larger districts are more likely than small ones to have more robust data systems.** There were significant differences by district size in the number of query capabilities outlined in the survey (as described in Exhibit 2-6), with a greater percentage of large and medium districts than small ones  reporting having four of the eight query capabilities. District size was not a predictor of the inclusion of the features or tools listed in Exhibit 2-7.

In looking across the key system elements profiled in Exhibit 2-2, both medium and large districts reported higher numbers of subelements compared with small districts for six of the 10 data system elements. Medium and large districts have significantly more of the subelements associated with system capacity, including those under the following elements: 

· Data linkages

· Assessment system

· Student performance data to measure academic growth

· Teacher-level data

· Student-level graduation, post-graduation, and dropout rates

· College readiness

· Assessment of data quality

Finally, there are also some significant differences in districts’ use of data depending on their size and the proportion of a district’s schools not making AYP. A smaller percentage of small districts than large districts  ever perform 12 of the 23 data use functions in Exhibit 2-8 (see Appendix C, Exhibit C-6a). Other significant differences were associated with a district’s performance with respect to AYP, with those districts with a larger proportion of their schools failing to make AYP reporting more data use (see Appendix C, Exhibit C-16). The proportion of a district’s schools not making AYP was associated with using data to examine achievement gaps between groups of students, to track school performance, to guide curricular changes or curriculum development, and to inform instructional practice (i.e., districts that engaged in these data use practices had a higher proportion of their schools not making AYP than did districts that never engaged in these practices).

**Summary**

The data in this chapter highlight both the progress that districts have achieved in developing an information infrastructure to support data-driven decision making and the unmet needs that still exist. Many of the education data systems in current use are not able to share information due to incompatabilities in their hardware and software (a barrier reported by over 60 percent of districts). Districts also lack the human capacity to make use of the information resources available to them. Together, these two factors inhibit the ability of districts to conduct meaningful analyses to better support instructional decision making. But the landscape is not uniform. Generally speaking, large districts are more likely than small ones to have more robust data systems and to conduct a wider range of data analysis activities.

**3. District Supports for Data-driven Decision Making in Schools**

The previous chapter described districts’ electronic data systems and the ways in which district staff are using data. This chapter describes districts’ activities to encourage and support the use of student data and data systems within schools and the challenges that these efforts encounter. Both the survey and case study data suggest that districts understand that addressing the “people” element is just as important as hardware and software issues in promoting data-driven decision making. Improving the capacity of schools to provide high-quality instruction and supporting their efforts to effectively use data (e.g., through leadership, policies, partnerships, professional development, and fostering supportive school cultures) are critical to success—having the human capacity to make use of the data provided. Implementation of data-driven decision making also requires strategic thinking so that all parts of the system are working in the same direction. Several themes emerged from the analysis of the survey and case study data:

· To spread the use of data and data systems to the classroom level, districts must connect their systems, professional development, and supports to instructional needs.

· Data-driven decision making is an ongoing, overlapping set of processes that take many years to come to fruition.

· Despite the fact that implementation is challenging, there are districts that are addressing the implementation challenges and their hard work is starting to pay off in terms of teacher adoption of data-driven decision-making practices to support instruction. 

· Strategies used by these districts to implement data-driven decision making include integrating data use with school planning and instructional coaching, developing district benchmark assessments and making benchmark assessment data and instructional resources available on the system, designing data-driven decision-making-related professional development, and adopting a continuous improvement process with respect to their own activities around data and data systems.

**District Data-driven Decision-making Support Index**

The model of supports for data-driven decision making in schools presented in Chapter 1 included both a user-friendly student data system and a set of tools, policies, and organizational supports that enhance the likelihood that school staff can actually use data to improve instruction. Chapter 2 presented a Data System Profile based on responses to the district survey regarding the capacity of districts’ data systems to provide data and analysis tools to support instructional decision making. A second district profile, presented below, examines the extent to which districts are providing the other components in the conceptual model:

· Leadership for improvement and use of data

· Tools such as online assessments for generating data 

· Social structures and time set aside for reflecting on data

· Professional development and technical support for using data

· Tools such as curriculum resources for acting on data 

**The district support profile suggests that district efforts to provide supports for data-driven decision making are mixed.** Exhibit 3-1 presents the composite Data-driven Decision-making Support Index for districts responding to the survey—each element score (right-hand column) reflects the median number of subelements present across districts. The percentage of districts with all of the subelements present and the percentage of districts that has each subelement is also provided. While the district responses to individual survey items about their support of data use indicate that data-informed decision making has become a district priority, only a minority of districts have all of the elements of any of the support dimensions in place other than data system access (for which 75 percent of districts have all subelements in place) and leadership for improvement and use of data (51 percent of districts have all of these subelements in place). At the same time, districts are undertaking a variety of activities to increase data use as evidenced by the percentage of districts reporting that they carry out various subelements. These data are consistent with responses to survey items concerning the strategies that districts have been emphasizing between 2005 and 2007 to promote data-driven decision making—see Exhibit 3.2.

The remainder of this chapter explores the strategies that districts are employing to support data-driven decision making in their schools—professional development, technical assistance and onsite resources, and district policies in support of data use.

**Exhibit 3-1. District Data-driven Decision-making Support Profile**

Exhibit 3-1 continues on next page

**Exhibit 3-1. District Data-driven Decision-making Support Profile (continued)**

Exhibit reads: In 2007–08, 75 percent of districts reported having both of the subelements for data system access (whereas 92 percent of districts had the first subelement and 81 percent had the second subelement). At least 75 percent of district reported having the median number of two subelements for data system access. 

Note: At least a fifth of district respondents to the district survey were not sure about the level of teacher access to the district’s information system.

Source: 2007–08 district survey questions 4, 8-10, 21-23.

**District Strategies for Increasing Use of Data Systems**

**Both the survey and case study data suggest that districts are taking steps to improve the capacity of their schools to use data in decision making, but the degree to which these efforts are part of an integrated approach to data-driven decision making is still unclear.**Survey responses of the national district sample indicate that a majority of districts (58 percent) have put a major emphasis on establishing a process for continuous improvement (e.g., developing measurable goals, measuring progress, reflecting, and making refinements) between 2005 and 2007 (Exhibit 3-2). Less than half (46 percent) report putting a major emphasis on designing or acquiring information management systems to support data analysis, and just 43 percent describe building district staff capacity for data analysis and interpretation as a major emphasis. 

**Exhibit 3-2. District Strategies to Promote Data-driven Decision Making From 2005 to 2007**

Exhibit reads: Between 2005 and 2007, 96 percent of districts placed at least some emphasis on establishing a process for continuous improvement as one of their strategies to promote data-driven decision making.

Source: 2007–08 district survey question 20.

The case studies provide a better understanding of the systemic nature of developing a culture of data use and the time involved in establishing these practices. Exhibit 3-3 describes two districts each of which has spent a decade or more creating or identifying data systems that will support instructional decision making and adopting organizational practices that encourage the use of data as a problem-solving tool (not the norm among districts). Each district has provided strong leadership to guide the adoption and integration of data into a process of continuous improvement. All levels of the system are held accountable for using data to guide decision making, but school staff feel that they have a voice in the process given established feedback loops and opportunities to review data and to receive training to support data analysis activities. These two districts have also made efforts to improve the usability of their data systems (since the structures and functions of the technology tools can either facilitate or impede their utility to users) and to include the kinds of data that support instructional decisions (e.g., interim district benchmark assessments that allow school staff to track student progress on a more frequent basis than annual state-level assessments). The issue of systemic change is discussed further as part of the description of the challenges posed to district efforts to spread data-driven decision-making practices.

**Exhibit 3-3. Systemic Change Guides District Implementation of Data-driven Decision-making Practices**

**Training on Data Systems and Use of Data to Improve Instruction**

Hardware and software are the vehicles for delivering data, but they are not sufficient to ensure good use of the data they contain: there is a need to train staff to use these information resources to make informed decisions. Over 90 percent of districts responding to the survey reported that they have provided at least some school staff with training designed to enhance the school’s capacity to use data in ways that improve instruction. A series of survey items probed districts on the nature of the training they have provided and the proportion of their schools that have participated (see Exhibit 3-4; see also Appendix C, Exhibit C-8 for additional data). 

**Exhibit 3-4. District-provided Training on the Use of Data and Data Systems**

Exhibit reads: In 2007–08, 73 percent of districts reported that they had provided training to principals or other building administrators in all of their schools on using the data system to analyze student achievement and 18 percent reported providing this training to some of their schools.

Source: 2007–08 district survey question 21.

Although the great majority of districts have offered a wide variety of training (76 to 91 percent of districts report having given the eight types of training listed on the survey), the training has not necessarily been extended to all of the district’s schools. For each training type, 53 to 73 percent of districts report having given the training to all of their schools. **Training for principals or other school administrators on how to use data to analyze student achievement is most likely to have been given to all schools (in 73 percent of districts), whereas training for teachers on how to use the data system to analyze student achievement or on how to use data to change their instructional practice are the two types of training least likely to have been given to every school (both by 53 percent of districts)**. The latter may be due to the fact that some districts are relying on staff such as data facilitators to help teachers access the data system or may rely on a user-friendly data system interface that district staff hope will require less professional development, as was seen in some of the case study districts. Another reason for limiting professional development activities to a few schools in case study districts was a scarcity of professional development resources. This constraint was cited as a major barrier by 35 percent of districts nationally (discussed further under challenges to implementation).

Data from the case study districts provide more detailed information on the kinds of training provided on the use of data and data systems. 

· **Training school staff on the district data system.** District 4 has provided a great many professional development opportunities for its school staff around the use of their data systems and provides on-demand training as requested. The district keeps a record of all the phone calls it receives requesting technical support and periodically evaluates these calls to determine areas of need and update its professional development courses accordingly.

· **Training principals and other school leaders.** Two of the case study districts have identified the principal or vice-principal as the data leader for their schools and have supported them in this role through professional development. One of these districts offers a wide array of data analysis training courses for principals, and data use is part of the district’s principal training program. The program includes a year-long apprenticeship during which aspiring principals meet one-on-one with the assistant superintendent and must bring with them their data goals focusing on monitoring and evaluating student achievement. Current principals meet together every four to six weeks to look at data on key results in their schools. Another case study district has designated the vice principal as the data coach and vice principals meet with teachers in teams, providing teachers with schoolwide data that they do not have direct access to. The vice principal receives assistance from a school-based technology coordinator who is trained to use the district data systems. The district’s director of student accountability provides school leaders with professional development on how to conduct training with their staff on using the data systems and using data to guide instruction. (Other case study districts use instructional coaches to work with teachers in this way.)

· **Embedding professional development on data use.** The use of instructional coaches is one of the ways that districts have provided embedded professional development on data-informed decision making. Another approach to this type of training is the requirement by one of the case study districts that every school designate a data team to receive training from the district. When data teams meet with district staff, they talk about data-related issues such as reviewing data from a school that is excelling. The data teams are then supposed to bring back what they have learned to share with the rest of their staff (a process that had yet to be universally adopted when their district was visited).

· **Mandating teacher training.** Two of the case study districts indicated that all new teachers are required to**receive basic training in how to use the district’s data systems (being able to access data is the first step). One of these districts also requires second-year teachers to receive intensive training on good assessment practices. 

· **Providing numerous training opportunities.** A medium-sized district that is very committed to data-driven decision making has carved out a number of opportunities for school staff to receive professional development on data use and to conduct ongoing discussions around the use of data. The district has used the state requirement that teachers get 60 hours of professional development a year as an impetus for teachers to get training on using data. During the district’s summer institute, principals are trained on using data (e.g., data disaggregation) and on conducting classroom walk-throughs. This administrator training has been coordinated with the implementation of Professional Learning Communities (PLCs) for teachers. Data from the principal walk-throughs is one of many types of data collected by this district. The district also sponsors semi-annual “data days” in which school staff and groups of teachers from the same grade level meet to discuss performance in individual classrooms as well as across classrooms. Time during the school day is set aside for the PLCs to meet to help “regularize” discussions informed by data. There is an expectation that the PLC meetings are to focus on data: “The teachers meet no less than once a week in their PLCs. But they can meet more. On that day, they bring data with them to drive instruction,” according to the executive director of education. In addition, the district’s director of data and testing provides one-on-one support to teachers and school staff that request assistance.

· **Supplementing district training resources.** Several of the case study districts have used system vendors and external consultants to supplement their professional development training or were members of collaboratives that offered data-related training. For example, a small district has received support from its county curriculum supervisor who has held training sessions for teachers on how to use assessment data and the state Web site that contains both data and instructional resources. The supervisor also spends two days a week at the district office providing assistance with curriculum, data analysis (e.g., item analysis of state test data, developing easy-to-use spreadsheets to support analysis activities) and professional development for administrators. In addition, the state has trained regional specialists to support teachers in reviewing the state value-added reports in reading and math and has created guiding questions for looking at these data; school districts pay a fee for these services.

**While professional development is one of the most common district strategies for building school capacity for using data, the quality of this training is uneven.** The district survey did not examine the quality level of district investments in professional development, but data from the case study districts suggest that not all of the training has been of high quality nor sufficiently robust to achieve desired results. School staff in one case study district, for example, described the training they received from their district on collaboration to use formative assessment data as very helpful for learning how to work with data, but data use was not supported afterward and the training resulted in just another binder on the shelf. Some of the challenges to providing effective professional development are discussed later in this chapter.

**Resources and Assistance for School Use of Data**

Training is not the only kind of support that districts are offering to encourage data-informed decision making in their schools. Exhibit 3-5 shows the proportion of districts reporting that they provide seven additional kinds of technical assistance and resources. 

**In addition to training, one of the most common supports provided by districts is access to technical experts in systems, networks or databases who can support school use of electronic data systems.** Eighty percent of districts say they have provided this technical expertise and 65 percent say they have made it available for all of their schools. In contrast, making data analysis experts available to school staff (for example, in the form of a data coach) is one of the least common supports. Still, 50 percent of districts say they have done this for at least some of their schools and 32 percent say that they have done this for all of their schools. Other less common types of district-provided resources are a Web-accessible library of diagnostic or benchmark assessments linked to academic standards (provided by 47 percent), models of how a school can use data to allocate resources and design school improvement activities (provided by 57 percent), and an online database of lesson plans and planning resources (60 percent). (See Appendix C, Exhibit C-9 for additional data.) 

**Exhibit 3-5. District-provided Supports for School-level Use of Data to Improve Instruction**

Exhibit reads: In 2007–08, 65 percent of districts reported they had made technical experts available to all of their schools to support system use, and 16 percent had provided such technical experts to some of their schools.

Source: 2007–08 district survey question 22.

Data from the case study districts reinforce district survey data that indicate that districts are providing leadership and supports for schools’ use of data. For example, several of the case study districts supply school-based staff who provide technical expertise around the operations of the district’s electronic data systems. So-called data facilitators or technology coordinators can be regular classroom teachers or technology specialists who are provided with additional training by the district on their electronic data systems. Their role is generally to assist other school staff with both the hardware and the use of data systems (e.g., accessing reports). These roles can be critical in making data available to school staff when the district’s electronic data system is not user friendly, as discussed in the next chapter.

Other supports have included providing more timely data linked to diagnostic assessments and resources to assist school staff in tailoring instruction to student needs. One case study district, for example, was in the process of developing its own curriculum management and assessment database consisting of three components: a curriculum repository to be used by curriculum specialists to load and maintain standards and benchmarks for state assessments, an item bank and district benchmark assessments aligned with state standards, and the district’s curriculum pacing guides and instructional resources. Using this system, teachers will be able to generate their own assessments, review results that pinpoint potential problem areas, and identify resources for student remediation. Another case study district has adopted a commercial assessment system to provide similar resources to its teachers. 

**District Policies in Support of School Use of Data**

District policies and practices can signal expectations around schools’ use of data (see Exhibit 3-6). **Of the policies and practices included on the district survey, the most common are requiring all or some of their schools to follow specific data-driven decision-making practices in formulating their school improvement plans (69 percent of districts do so) and providing teachers with specific processes for how they should use data for instructional purposes (65 percent of districts do so).** The least common practices are actually keeping track of the extent to which teachers use the data system and paying incentives for teachers to use or obtain training in data-driven decision making: only 30 percent of surveyed districts reported doing this. Requiring instructional coaches to incorporate data use in their work was also among the less common practices reported by districts (44 percent). (See Appendix C, Exhibit C-10 for additional data.)

**Exhibit 3-6. District Policies and Practices Encouraging Schools’ Use of Data**

Exhibit reads: In 2007–08, 51 percent of districts reported requiring all of their schools to follow specific data-driven decision-making practices in their school improvement plans, and 18 percent reported requiring this of some of their schools.

Source: 2007–08 district survey question 22.

The survey data provide a general picture of district practices and policies to encourage schools’ use of data and the case study districts provided an opportunity to examine these practices in greater depth given their focus on data-driven decision making. Two-thirds of the case study districts provide data analysis support to their schools through instructional and other types of school-based coaches (compared to 44 percent of districts overall). These districts have also adopted processes and tools to help school staff effectively use data. The leadership role that instructional coaches play in facilitating data use at the school level is discussed in greater detail in the next chapter. 

One of the case study districts, concerned that there was a disparity across schools in the degree to which teachers were accessing and using the district’s data systems to inform instruction, turned to the use of professional learning communities (PLCs) as an alternative to relying solely on principal support for data use at the school level. In 2007–08 the district focused on establishing PLCs to promote teacher collaboration around data use. PLC meetings were formalized and integrated within the school day. There were PLC meetings dedicated to language arts and math staff development during which reading or math specialists planned and led the meetings to discuss student data and instructional strategies. Other PLC meetings were led by team members to monitor student improvement, develop common assessments, and share best practices. The district holds schools accountable for establishing PLCs by monitoring their implementation and rating their quality. Staff at some of the case study schools indicated that staff can become too dependent on specialists so this district’s next step is going to be to encourage teachers to take more ownership and responsibility as a team for the PLC meetings. In addition, the district is keeping track of teachers’ use of the data system. A log is kept to record what teachers do with the data system and principals can then talk to those teachers who are not using the system to find out why.

Several case study districts are putting a greater emphasis on using data to guide school-level planning. To increase school-level accountability for improving student achievement, one of the medium-sized case study districts is placing a greater emphasis on using data to monitor school progress against school improvement plan goals—requiring school staff to develop monthly action plans using a common template so that staff can make the connections between their plan, the data and the process of improvement. District professional development has emphasized that school improvement is more than a yearly activity; it is not something that is done at the beginning of the year and then filed away in a drawer. The director of school improvement and accountability said, “School improvement is actually about improving schools, it is not a document. It’s an active process....The whole use of data on a regular basis to drive changes within your school to get you where you ultimately want to go.” This district requires instructional coaches to focus on the action plans when they work with school staff. This process includes reviews of student work to help teachers understand the requirements in the state standards and tailor their instruction based on student needs to assist each student to meet the standards. Another district adopted a Data Action Cycle process as a way to look at data at both the district and school level. Principals are encouraged to use the process as a way to guide the thinking of school leadership teams to analyze data for school improvement planning. The Data Action Cycle is a five-step data protocol to support a collaborative and reflective dialogue around data that focuses on a continuous improvement approach (e.g., where to get the right data to address a topic, how to read the data, how to apply what is learned, review results).

While data can provide opportunities for dialogue, reflection and learning, sometimes there is the temptation to use data to justify prior assumptions or jump to conclusions (Honig and Coburn 2005; Knapp et al. 2006). Therefore, it is important to structure opportunities for staff to effectively engage in data discussions, and tools for organizing these discussions can be useful. As part of one case study district’s efforts to help train school staff to implement improvement processes, the district identified a number of tools and planning strategies and compiled them into a resource guide. The guide defines each tool and provides an example of how it can be used. The process for implementing each tool is outlined and frequently asked questions (FAQs) are addressed to help educators problem solve as they begin using the tools. For example, the Fishbone diagram provides a picture of various system elements that may contribute to a problem or topic. This tool can be used to help identify the root cause of a problem, provide a visual representation of the cause and effect relationships that are occurring, and help organize ideas during a brainstorming session.

**Challenges to District Efforts to Spread Data-driven Decision Making**

Because the gathering and examination of data is just the starting point in developing a culture and system of continuous improvement, it is important to think about the implementation of data-driven decision making as a systems change (Mandinach et al. 2006). District and school adoption of data-driven decision-making practices requires a focused coordination of elements of the larger education system, including such things as policy change, widespread professional development planning, and reliable and affordable support mechanisms. In other words, it is important to consider how the process will function across the entire district and will be used by participants at all levels of the system. To better understand the systems challenges faced by districts, survey respondents were asked to rate a range of potential technology, logistical and other barriers in terms of the extent to which they were currently inhibiting expanded use of data-driven decision making within their district. Their responses, shown in Exhibit 3-7, suggest that **a majority of districts report that they have the hardware needed to support a data system. In addition, fewer than half report that there are problems with the quality of the data in their systems.** The most commonly cited system and technology barriers were (see Appendix C, Exhibit C-11 for additional data): 

· Lack of funding for expanded use of data-informed decision making 

· Difficulty linking disparate databases for purposes of analysis (lack of interoperability)

· Lack of trained technical staff available for product and service acquisition, installation or equipment maintenance

· Data stored in forms that are difficult to access, manage and interpret. 

All of these issues are seen as barriers by over half of districts. 

**According to districts, the greatest logistical barrier to spreading data-driven decision-making practices throughout their district is the lack of time for school staff to conduct data-driven decision-making activities**: 92 percent of districts cited lack of time as a barrier, and 51 percent identified it as a “major” barrier. A good deal of literature exists on the importance of collaboration and dialogue around decision making, and researchers such as 

**Exhibit 3-7. District Administrators’ Perceptions of Barriers to Increased Use of Data Systems**

Exhibit reads: In 2007–08, 42 percent of districts reported that lack of funding to expand or improve the data system was a major barrier to expanding the use of data-driven decision making in their district; 35 percent reported it was a minor barrier.

Note: Asterisks indicate significant differences by district size (*p* < .05 and *p* < .01).

Source: 2007–08 district survey question 25.

Elmore (2004) and Wilhelm (2007) have expressed concern that teachers often are not given the time or support to allow such activity. Even among case study districts selected as high data users, lack of time frequently emerged as a major barrier to data use that was manifested in multiple ways (e.g., lack of time for analysis, to collaborate around data, to receive training). 

Administrators in three of the case study districts indicated that both district and school staff are faced with many competing priorities and, as a result, not all of them can be achieved or achieved well. One administrator stated: 

*I think it just comes to time. It might be that technology, the infrastructure of technology is not moving at the speed that it needs to be. It might be that I would really be glad to do that, but I’m implementing the new SIS system, the new Web site system, the new curriculum, the new RTI model...all the other innovations that we’re doing as a district. Priorities are called priorities for a reason. You can’t have five of them.... People need time not only to make meaning [from the data] individually but to collaborate to make meaning as a group. And I think often times the time is not there because we are always doing something...do, do, do instead of stopping and thinking.*

A superintendent stated, “Every program that you implement takes time. As a result, you’re taking time away from something they [teachers] used to do or adding time to their already long day.” Similarly, lack of time for professional development on how to use the district’s data system or how to analyze data, particularly for principals, can serve as a barrier to data use at the school level. 

**District ratings of other organizational and logistic barriers to spreading data-driven decision making and ratings of data system barriers suggest that districts perceive the organizational issues as a bigger challenge than the data system issues.** In addition to time, other organizational issues seen as a barrier by a majority of districts include (a) school staff’s lack of the technical skills to access or use electronic data systems (89 percent), (b) lack of teacher preparation on how to use data for instructional decision making (85 percent), (c) lack of resources to train school staff (81 percent), (d) lack of building administrator preparation on how to use data to inform decision making (69 percent), (e) lack of communication or information sharing across departments (66 percent), and (f) lack of a clear vision or strategic plan for data-driven decision making (65 percent). These data also suggest that despite the widespread district provision of professional development related to the use of data to inform instruction discussed earlier, districts believe that school staff need further training and support; a conclusion supported by teacher survey data (U.S. Department of Education 2009).

All of the major barriers cited by survey respondents were also in evidence in the case study districts and attest to the numerous challenges that districts face as they try to implement data-driven decision making. There appear to be some significant differences in perceived barriers by district size. (These will be discussed in greater detail later in this chapter.) 

In the case study districts, lack of resources to train school staff on how to use data to support instructional improvement took several forms: not having sufficient training staff and funds, reductions in school-based staff who provide embedded professional development on data use as part of their day-to-day work with teachers on instructional issues, and a limited range of training. In a medium-sized district, administrators indicated that there were many meetings in which teachers and principals look at data but far fewer opportunities to be trained in how to analyze data. They acknowledged that in addition to offering more professional development, they also needed to change their approach so that teachers become better data analysts: “[It’s] almost sensory overload. We’ve put so much [data] out there. We don’t practice what we preach. If a teacher were to say, ‘I’ve covered that curriculum but the children haven’t learned....’ We do the same thing. We’ve covered [data and assessment] in workshops, but teachers haven’t assimilated it; [they] can’t in a sophisticated, confident way use that information.”

**Related to the challenge of providing adequate professional development to school staff is a lack of teacher and principal preparation on how to use data.** Respondents in several case study districts commented on the lack of fundamental knowledge about assessment among their teachers and principals, and school staff’s lack of knowledge concerning how to take assessment data and determine appropriate interventions. As one district administrator stated, “We’re data rich and resource poor. Rank order report by skill or ability; draw a line to make groups. But beyond that, there’s not a great deal [of knowledge]. ...Teachers need to know more about how to interpret assessments....Every time I do a workshop [teachers] don’t understand assessment very well. Don’t understand how to look at data, how assessments are constructed.” For some teachers it’s even more basic—there is a general lack of comfort with using data systems and data—that has led to mediated data system access in some districts, a slower learning curve for some teachers, a concern about how data will be used, and a general resistance to relying on data that is not generated in the classroom.

As the conceptual framework suggests, **leadership for educational improvement and use of data is a vital component in promoting data-driven decision making within districts and schools. The case study districts demonstrate that operationalizing this type of leadership is a complex undertaking.**To be successful in spreading the use of data and data systems to the classroom level, districts must find ways to connect their data systems, professional development, and supports to address instructional needs—to think systemically about how to infuse data within the culture of district and school practice. An issue underlying the use of data systems is the tradeoff between standardization and customization raised in the interim report (U.S. Department of Education 2009). District investments in data systems to store, manipulate and report on students are predicated on the assumption of a uniform set of practices across schools. But standardized implementation of teaching practices is not the norm, and school-based staff often feel that the kinds of information available in the district system do not address the questions and decisions most important to them. Accountability requirements have heightened this tension. Some of the case study districts have resisted top-down mandates and relied on voluntary adoption of data-related practices by staff as they see the value of new technologies to support their work. Other districts have been much more directive, balancing requirements with communication loops that help the district identify problem areas and garner support by engaging stakeholders up and down the system, and identifying how to redirect resources and refine policies and practices. A few examples from the case study districts illustrate how district leadership can influence data-driven decision-making implementation both negatively and positively (Exhibit 3-8).

**Exhibit 3-8. District Leadership Support for Data-driven Decision Making**

Another aspect of the tension between standardization and customization can reside at the district level. Some districts engaged in creating a data-driven culture have focused on issues of professional development and support for school staff to use data but have not addressed technical integration or data sharing issues across their own departments. As discussed in Chapter 2, sometimes in order for districts to meet federal and state reporting requirements they have had to maintain separate data systems that are not fully interoperable. In other instances, the multitude of stakeholders involved in data-driven decision-making activities can lead to incompatible systems if there is a lack of communication or centralized plan. 

**District Needs for Examples of Good Practice**

Another perspective on the areas in which districts are struggling is provided by a set of survey questions (see Exhibit 3-9) asking districts to indicate how much they need various types of examples of good practice. **The greatest perceived area of need among districts is for models of how to connect student data to instructional practice.** The most frequently cited needs are for examples of how to perform the following (see Appendix C, Exhibit C-12 for additional data):

· Adapt instructional activities to meet students’ individual needs (cited by 84 percent of districts)

· Examine data to identify which practices work best for which students (83 percent)

· Develop curriculum-embedded formative assessments (81 percent). 

The area in which districts felt least in need of good examples was communicating with parents about student progress (63 percent), a practice that teacher responses on an earlier survey suggest is quite common (U.S. Department of Education 2008). 

**Exhibit 3-9. District Perceptions of Needed Examples of Good Practice**

Exhibit reads: In 2007–08, 47 percent of districts reported that they had some need for examples of good practice regarding the examination of data to identify which practices work best for which students, and 37 reported a great need.

Source: 2007–08 district survey question 24.

**Variations in Perceived Barriers and Needs**

District survey and case study data indicate that the use of educational data to make decisions and foster improvement has been increasing, but progress has been uneven. Smaller districts have gotten a later start in implementing data-driven decision-making strategies and they tend to provide fewer supports. District size has also generated differences in perceived barriers (all related to technology) and perceived needs. When analysts compared the number of years that districts have been implementing or supporting data-driven decision-making strategies, some interesting patterns emerged.

**Smaller districts reported getting a later start in supporting data-driven decision-making activities in their schools** in terms of providing professional development programs for teachers and principals on the use of data to improve instructional practices, providing resources such as consultants to support the use of data, and implementing policies and requirements to use data or providing incentives for data use. A greater percentage of smaller districts are either planning to begin to provide these supports or have been doing so for one to two years, whereas most larger districts have been providing these supports for at least three years (see Appendix C, Exhibit C-13 for data tables).

**Districts of different sizes also vary significantly in terms of some of the perceived barriers to implementing data-driven decision-making practices but only for data system and technology-related barriers and not for other types of barriers.** Exhibit 3-10 shows that a significantly greater percentage of small districts than medium and large districts report that they face a barrier in the terms of data stored in forms that are difficult to access and manage. On the other hand, fewer small districts report facing barriers in terms of unreliable or inaccurate data in their systems or providing adequate safeguards and security for system data when compared to medium and large districts—perhaps due to the lower volume of data. A greater percentage of both small and medium districts than large districts report facing the barrier of information stored in multiple disparate databases that are not interoperable. A greater percentage of both small and large districts than medium districts report that lack of trained technical staff for product and system acquisition and maintenance is a barrier. The latter finding is contrary to what was observed in the large case study districts, which tended to have greater resources to employ staff to develop and maintain their data systems, although a few were beginning to experience cutbacks due to shrinking resources (see Appendix C, Exhibit C-15).

**Districts of different sizes varied significantly in terms of one area of needed model practices—using data to identify gaps in student achievement.** Three quarters (75 percent) of small districts reported having some to a great need in this area compared with about half of medium-sized (54 percent) and large districts (53 percent). This difference may reflect differences between large and small districts in the frequency with which they carry out this activity—a greater percentage of small districts report never examining achievement gaps between subgroups (7 percent compared to 5 percent of medium-sized and 1 percent of large districts), and their data systems are less likely to have query tools to look at student performance by AYP subgroups or over time (see Exhibit 2-5; see Appendix C, Exhibits C-4 and C-14 for differences by district size).

**District size also is a factor in the number of supports that districts provide to schools to encourage data-driven decision making**(the proportion of districts with a particular subelement varies significantly by district size). In looking across the support index profiled in Exhibit 3-1 earlier in this chapter, both medium and large districts reported higher numbers of subelements for four of the six elements compared with small districts. Medium and large districts had more of the elements associated with (a) leadership for improvement and use of data, (b) social structures and time set aside for data use, (c) professional development and technical support for data interpretation, and (d) tools for acting on data. 

**Exhibit 3-10. District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, by District Size**

Exhibit reads: In 2007–08, large and medium districts were more likely than small districts to report that unreliable or inaccurate data in the system was a barrier. Among small districts, 4 percent reported this to be a major barrier compared with 7 percent of medium districts and 6 percent of large districts. Eighteen percent of small districts reported that unreliable or inaccurate data in the system was a minor barrier compared with 36 percent of medium and 42 percent of large districts.

Note: Asterisks indicate extent to which this was a barrier varies significantly by district size (**p* < .05 and ** *p*< .01).

Source: 2007–08 district survey question 25.

Data-driven decision making is clearly a complex process, combining technical, pedagogical and social elements, and requiring both top-down and bottom-up participation and support. As districts undertake this effort, they would like to have a realistic perspective on the amount of time that it is likely to take and the amount of progress that can be expected after one year, three years, and so on. Both the case studies and the responses to the district survey provide clues to the trajectory that districts can expect. 

Exhibit 3-11 shows the case study districts’ classification in terms of the strength and consistency of supports with respect to data-informed decision making arrayed against the number of years that districts have been actively engaged in the effort and their size (from largest to smallest). For each type of support, a district was given a rating: the support was widely available in the district (coded as 2); the support was partially present, for example, instructional coaches available in some schools but not all (coded as 1); or the support was not currently present in the district (coded as 0). The larger districts (those with student enrollments of 25,800 or more) provide a somewhat greater number of supports than the medium-sized or small districts—a similar pattern to that found for the 2006–07 case study sample (U.S. Department of Education 2009). The mean number of widely present supports in large districts is 5.29 compared with an average of 4.0 in medium-sized districts. But there are exceptions to the general pattern. The length of time that the case study districts have been actively engaged in data use did not necessarily translate into the presence of a greater number of supports.

**The perceived barriers to implementing data-driven decision making differ significantly based on the number of years a district has been implementing data-driven decision making.** Using the district survey data, analysts were also able to identify districts that have been trying to spread data-informed decision making to their schools for one to two years, three to five years, or six years or more and compare their responses as to the perceived seriousness of implementation barriers and their need for examples of good practice. A greater percentage of districts in their first year or two of providing resources for data-driven decision making to schools than of districts that have a longer history of such efforts reported major barriers in the areas of (a) lack of teacher preparation on how to use data for instructional decision making (29 percent), (b) lack of sufficient hardware (17 percent), (c) lack of a clear vision or strategic plan for data-driven decision making (11 percent), (d) lack of building administrator preparation on how to use data (11 percent), and (e) lack of district leadership support for data-driven decision making (9 percent). The launch of a new endeavor can often run into problems with inadequate direction, staff buy-in, staff preparation, and resources. As shown in Exhibit 3-12, a smaller proportion of the districts that have been trying to spread data-driven decision making to their schools for three to five years perceive these issues as major barriers. Teacher preparation and district leadership appear to re-emerge as perceived major barriers for districts that have been engaged in spreading data-driven decision making for six years or more—perhaps reflecting staff turnover in both schools and district offices.

**Exhibit 3-11. Case Study District Supports for School Use of Student Data**

Exhibit reads: In District 1, strong district leadership for data use was fully present in 2007–08. This large district has been implementing data-driven decision-making practices for at least 10 years.

Notes: District size categories include large (L) with 25,800 or more students, medium (M) with enrollment from 5,444 to 25,799, and small (S) with enrollment from 300 to 5,443. Codes indicate degree to which each support was present in the district: 0 = not present, 1 = partially present, 2 = fully present/widely available. 

**Exhibit 3-12. District Perceptions of Major Needs and Barriers, by Longevity of Data-driven Decision-making Efforts**

Exhibit reads: Among districts that have been providing resources to schools for one to two years to support the use of data, 28 percent reported a great need for examples of good practice about collaborating and sharing ideas with colleagues regarding data inquiry. For districts that have been providing these resources for three to five years, 11 percent reported a great need compared with 27 percent of districts that have been providing these resources for six or more years.

Note: Asterisks indicate extent to which perceived need or barrier varies significantly by years of data-driven decision-making efforts (**p*< .05 and ** *p* < .01).

Source: 2007–08 district survey questions 23-25.

In a related vein, **districts in their first year or two of trying to support data-driven decision making in their schools are particularly likely to report that they have a need for examples of good practice with respect to how to structure the district to support data-driven decision making (28 percent report a “great need”) and how to collaborate and share data inquiry ideas with colleagues (28 percent).**Perceptions that these are great needs are less common among districts that have spent three to five years providing resources to promote data-driven decision making (11 percent and 10 percent, respectively) but are fairly high among those that have been involved for six years or more (27 percent and 15 percent).

Finally, analysts explored the relationship between the number of strategies districts have employed over the past two years to promote data-driven decision making and the frequency**with which they analyze data. Using the survey data, a scale was constructed representing District Capacity Building Strategies by assigning each of the strategies listed for survey question 20 a score of 0 (no emphasis), 1 (some emphasis), or 2 (major emphasis). The total score for the five items in question 20 was calculated resulting in a number ranging from 0 to 10. This scale was in turn used to predict the survey responses on individual items in questions 18 and 19 about how frequently staff in the district office conduct data analysis activities using a logistic regression model. The result was that the vast majority of items in questions 18 and 19 were significantly predicted by the capacity building score based on question 20. In other words, **districts that are engaging in multiple capacity building strategies appear to be analyzing data more frequently.**

**Strategies for Addressing Implementation Challenges**

The preceding discussions have addressed the many challenges that districts face as they have moved toward a greater reliance on data-driven decisions. Despite the multiplicity of challenges, a number of districts are finding ways to promote data-driven decision making. In particular, the districts included in the case study sample are employing a variety of strategies to facilitate the adoption of data-driven decision-making practices at the school level. It has taken them many years to develop policies and practices designed around the ongoing examination of data aimed at improving outcomes for students and it has not always been a smooth transition. These districts continue to struggle with many implementation issues but serve as examples of what can be accomplished. 

As mentioned previously, when data systems have not been user friendly, case study districts have relied on data mediators (e.g., technology coordinators, instructional coaches). Unfortunately, when resource constraints necessitate the loss of these staff, it can leave a school or district with little technology support, particularly if training on use of the district’s data systems has not been widespread. For these and other reasons, **five of the****case study districts have begun to embrace the use of data dashboards that provide school staff with frequent**

**updates in an easily digestible format on data that is of most interest to them.** For one of these districts, the move toward the creation of a data dashboard represented a change in thinking about how to make data accessible and more useful to school staff: 

*Maybe our approach of putting the data at the fingertips of every person was a pretty grandiose idea. Not that we are saying the opposite of that. Make it available to everybody, but recognize this generation of data users. They went into education to teach and design lessons and engage in meaningful work with kids. And so what is my inherent propensity to go to the computer and pull that data? But if you bring it to me and it tells me more about my children, tells me more about my instruction...yeah I’m interested. Let’s talk.*

The district is working with a commercial vendor to create a data dashboard that already has reports created so that all teachers have to do is press a button.

*Most of what we see is that people have too much data not too little. We see more data drowning than data scarcity and too much data doesn’t do much good either....Certainly we don’t want to take away people’s personalization of the product if they want to create custom reports, but we know there are four or five reports that we want everyone looking at. When we have district meetings, when we have department chairs from content areas from campuses together, when we bring principals together having some of those streamlined, similar reports will help expedite conversation and planning and that kind of thing.*

**Other case study districts were changing over to more user-friendly systems to increase data usage so that there would be less need for data facilitators and professional development on how to use the data system, with mixed results.** In one of these districts, the district has invested its resources in making their data systems more intuitive, with the presumption that schools and teachers will “discover” and teach each other how to use the systems (e.g., teachers were provided with an hour to a half day of formal training on the new student information system). One of the reasons for uneven use of data among school staff in this district was due to differences in their comfort level with using the technology—not all teachers are the kind of “self-starters” the district expects and some mentioned the need for additional professional development.

**A number of the case study districts use districtwide benchmark assessments to provide both district and school staff with formative data that allow them to more frequently assess student progress on district and state standards and to adjust instruction to meet individual student needs.** Several of these districts mentioned involving teachers in the development of these assessments as a way to build capacity and to achieve teacher buy-in for use of the assessments. When these data are made accessible through the district’s data system, teachers are able to analyze the data by student groups and drill down in the data to the student level. At the same time, one of these districts indicated that some of the benchmark items written by teachers were not always of high quality, and the district subsequently contracted with some outside companies to write test items. These items were then reviewed by the district and by teachers, and used to construct the district benchmark tests. Concerns expressed by district administrators about teachers’ lack of knowledge about assessment mentioned earlier contribute to these limitations.

The use of interim assessments both at the district and school level are also meeting the demands by school staff for more timely data (e.g., results of assessments given every six to nine weeks) that are linked to the local curriculum and sometimes to instructional resources to help inform instructional practice. Occasionally, it has taken more than one attempt to develop a benchmark assessment that was aligned with instruction and to devise a system for rapid deployment of the data back to teachers. For example, in two case study districts that had initially arranged for central processing of their benchmark data (student results were recorded on forms that were scanned at the central office because there was no capacity for online testing), they are now making provisions for school-based processing (scanners placed in selected locations) at the request of school staff so that results can be generated much more quickly. Another case study district created a benchmark assessment and reporting tool two years ago using a commercial product, but the math items were not well aligned with the district math curriculum. This system is being replaced with another assessment system (also developed in collaboration with a vendor) that contains assessment items that are better aligned with the curriculum and linked to more instructional resources for teachers.

**Piloting new data systems and data-related practices at a few schools was an approach mentioned by several of the case study districts** (see Exhibit 3-13). 

Three of the case study districts have provided laptops to teachers to encourage data use. One of these districts provides laptop computers with Internet access to all of their teachers to provide anytime, anywhere access to data. One of the case study schools within this district has set up a school server that teachers can access through their laptops where they can share instructional materials, assessment data, and student work. The server also holds student data that is not available through the district data system (such as the reading assessment data for students in lower grades). This access to data has promoted data dialogues among teachers from different grades and changed teachers’ perceptions of what constitutes data. 

**Exhibit 3-13. Piloting Data-driven Decision-making Efforts**

**A number of the case study districts have developed or participated in a range of partnerships to support their efforts to use data.** For example, one of the large case study districts has been supported by two privately funded organizations to provide training in collaborative planning, teaching and learning as part of an effort to develop model districts across the state. The district has also contributed funds so that a broader range of teachers can participate in data-driven decision-making training. Another district noted that they have used the service of their regional consortia (state-supported regions) to discuss special education data: “The Education Service District has been working monthly with a group of us from the district. With me, we’re working off of the 14 major [federal] indicators of special education, doing data carousels. We look at the data, looking for patterns and trends. Trying to figure out how best to provide services to these kids that will make a difference.” Another case study district has applied for grants from foundations as well as the federal government for data-related activities (e.g., Reading First, Small Learning Communities). This same district has partnered with the local university to develop a teacher training program that includes the use of data as part of some nonnegotiable research-based instructional strategies that the district identified. These strategies are also the focus of the professional development that new teachers receive for the first three years they are on the job.

**Summary**

Data from the district survey and case studies illustrate the interrelationships between data systems and district activities to encourage and support the use of data at the school level. The dynamic and interacting factors that influence the structure and functioning of districts must be coordinated and brought to bear to achieve successful data-driven decision making. This often requires change in practices among district administrators and ultimately at the school level. Beyond the presence of data systems with the capacity to store and analyze data, survey responses indicate that districts are demonstrating their support and leadership for schools’ use of data through a number of actions corresponding to the supporting conditions outlined in the study’s conceptual framework. Districts are providing leadership to develop a culture of data use at both the district and school levels. Strategies include establishing a process for continuous improvement guided by data (e.g., instituting policies and requirements for principals and teachers to use data in their planning activities), providing support positions for system implementation (and tracking usage), and designing professional development and capacity-building activities for schools.

Districts are also beginning to develop and employ tools for generating data (e.g., benchmark assessments in core subject areas that are aligned with curriculum standards to track student progress, Web-accessible library of resources that teachers can access if their students are struggling with a particular standard) as well as providing their schools with tools for acting on data (e.g., models for school staff to illustrate how to use data to design school improvement activities). But district progress is uneven and it appears, cyclical, with challenges varying over time. 

Many districts, including those identified as high data users, are struggling with logistical barriers to implementing data-driven decision making in their schools and the lack of administrator and teacher preparation to use data effectively. At the same time, there are school staff who are supporting the development of data-driven practices within their schools and generating data required for instructional decision making when district data systems fall short. These issues are discussed in greater detail in the next chapter.

.

**4. Use of Data Systems and Data-driven Decision-making in Schools**

The prior chapter described ways in which districts have been promoting the use of data systems and data-driven decision processes in their schools. In this chapter, we examine data-driven decision making from the school perspective. The data sources for the chapter are secondary analyses of a teacher survey administered in 2007 as part of the National Educational Technology Trends Study and school staff interviews and focus groups conducted during the 2007–08 school year at 36 schools in the 12 case study districts described in Chapters 2 and 3.

Major themes coming out of the analysis of school-level data were the following:

· The most common school-level uses of data described by case study school staff are school improvement planning, curriculum decisions, and placement or grouping of students for instruction or support services. 

· Use of data to support inquiry into how one can improve one’s teaching appears to emerge later than use of data to set school improvement targets, to identify students for special services, or to fine-tune curriculum coverage to match accountability tests.

· The most powerful school-level catalysts for teacher use of data are school leader promotion of these activities and the establishment of an organizational climate of trust and mutual respect.

· School staffs’ perceptions of barriers to greater use of data include a sense of lack of time, system usability issues, the perception that the data in the system are not useful, and district policies around pacing that prohibit modifying learning time to match student needs.

**Nature and Frequency of School Use of Data and Data Systems**

**On the NETTS teacher survey (U.S. Department of Education 2009) the majority of teachers with access to student data systems reported that they use the systems to provide information to parents (68 percent), track individual test scores, and monitor student progress (both 65 percent).**Less common uses of data—reported by just over a third of the teacher survey respondents—were**identifying promising practices (39 percent), informing student placement in courses or special programs (37 percent), and assessing test taking needs (36 percent). Exhibit 4-1 shows the 10 data uses most often reported by respondents in the 2007 NETTS Teacher Survey. 

The present study was designed to complement survey reports with qualitative data from the sample of case study schools. School staff descriptions obtained from over 200 respondents during the school site visits provide a more detailed and interpretable view of uses of data and data systems in the case study schools. In viewing descriptions of data use practices in case study schools, it should be kept in mind that these schools are in districts that are not representative of school districts as a whole. Teachers in the case study schools had more electronic data available to them and more supports for data use than did teachers in U.S. schools as a whole during the 2006–07 and 2007–08 school years.

**Exhibit 4-1. Teachers Who Reported Using a Student Data System at Least a Few Times a Year for a Specific Function in 2007**

Exhibit reads: Among teacher survey respondents with access to a data system, 68 percent reported that they use the system data in talking to parents about their student’s progress.

Source: NETTS teacher survey, 2007.

During school site visits, principals, coaches and teachers were asked to describe specific incidents of using data to make instructional decisions. From the 36 schools visited in 2007–08, 188 examples of using data to inform instruction were obtained. Exhibit 4-2 shows the different purposes for which staff in case study schools described using data and the number of distinct citations of each use (a school could report the same kind of use on more than one occasion or with different kinds of data or groups of students). 

As shown in the exhibit, the 10 most common uses of data described by case study school staff were for school improvement planning (35 citations), curriculum planning (25), and decisions about student placement in classes or special services (22). Informing parents about student progress, the data use most commonly selected by the nationally representative sample of teacher survey respondents when viewing a menu of options, was described only three times in the 36 site visits to case study schools. One possible interpretation of this apparent discrepancy across the two data sources is that teachers do in fact commonly show students’ data to their parents (for example, during parent-teacher conferences) but do not think of this activity as “using data” unless prompted to do so (as they were by the NETTS survey question).

**Exhibit 4-2. Uses of Data Described by Case Study Schools**

Exhibit reads: School improvement planning was the most common data use described by staff at the 36 case study schools.

**The data in Exhibit 4-2 suggest that in districts considered leaders in data-driven decision making, most schools are using data to develop goals for school improvement and to do curriculum planning.**At one elementary school, for example, staff spend August and September analyzing data to determine whether they met the prior year’s goals and to develop new goals and make adjustments based on this analysis. These case study school reports are consistent with the responses of surveyed districts (Exhibit 3-6): 69 percent of districts reported requiring all or particular schools to follow specific data-driven decision-making practices in developing their school improvement plans.

Specific examples of data-driven changes from the case studies include an elementary school in which the instructional coach reviewed fall data with the third-grade teachers and determined that many of their students had difficulty identifying the main idea of a passage leading to the decision that third-grade teachers needed to focus more instruction on that skill. At another elementary school a teacher reported: 

*We looked at our test scores as a first-grade team and found that we were very weak in one area—phonemic awareness. We spoke to [the literacy coach about possible interventions]. She suggested bringing in more rhymes to help students discriminate different sounds.*

Some of the case study schools had become quite sophisticated about analyzing their state accountability test data and their students’ performance in ways that would enable them to raise test scores. In one school, for example, a review of students’ performance on the district’s benchmark tests revealed that students were doing poorly on items asking them to check the correctness of the grammar of a provided sentence. In response, the school staff took sentences 

from the benchmark assessments for use in classroom instruction and started instructing students using the same method of indicating an answer (e.g., underlining the predicate in a sentence rather than circling it) as the test did. Thus, the school responded to the assessment data both by increasing its coverage of grammar and by aligning the mechanics of its coverage to the specifics of the test. At the same school an item analysis of student data revealed that their students were not doing well on the vocabulary section of the state test. They noted that their teachers were spending almost all of their reading time working on comprehension, but that the language arts section of the state test had 10 to 15 vocabulary items compared to 18 to 19 comprehension items. The next year teachers spent more time on vocabulary instruction and saw a big jump in test scores.

**Year-to-Year Growth in Data Use**

**An examination of changes in data use practices between the first and second round of school site visits suggest that the strongest levers districts have for increasing schools’ use of data systems are (a) providing timely data that school staff view as relevant to instructional decisions and (b) making sure that data systems are accessible and easy to use.**

Eighteen of the 36 schools visited by researchers during school year 2007–08 had been visited a year earlier as part of the 2006–07 data collection. Researchers reviewed descriptions of data use provided by teachers and school leaders in schools visited twice to ascertain whether or not there were changes in data-use practices at those schools between the 2006–07 and 2007–08 school years. In 13 of the 18 schools, there was some evidence of an increase in data use over this one-year period; in four schools there was no indication of a change in the frequency or nature of data use; one school’s report could not be classified. 

**Districts are generating more timely assessment data that teachers can use in instructional planning by implementing districtwide interim (either formative or benchmark) assessments.** Exhibit 4-3 provides definitions for these and related assessment terms. Schools in districts involved in implementing a major system of districtwide interim assessments were the most likely to show an increase in data use from year to year, and also provided the most striking examples of changes in teacher data use practices, such as the institution of collaboration around the review of assessment results to derive implications for instruction. Staff at six case study schools reported that the introduction of interim assessments and the incorporation of the results of these assessments into an electronic data system had increased their use of data. 

**Exhibit 4-3. Assessment Terminology**

For example, in one case study district the introduction of benchmark assessments has had an impact on instructional practices because teachers now have more detailed data on student performance. Many teachers in this district lauded the fact that they can see how individual students perform on particular test items. They are also able to compare their own students’ performance with performance in other classes, the school as a whole, and the district as a whole. Because they can compare their class with other classes in their own school, teachers can see whether their students’ performance on particular items is due to their instructional practices or the wording of the test item, which affects students in all classes. This helps them decide whether they need to reteach or make changes to their own teaching practices. This kind of comparison of assessment results across different classes was described by teachers in eight of the case study schools.

In another case study district, the introduction of districtwide assessments in language arts and social studies between the first and second site visits provided a forum for teacher discussions of student performance on specific test items and topics. Teachers began using their common planning time to compare performance of their respective classes on the common test and discuss the relationship between different instructional approaches and student performance. One such example came from a pair of social studies teachers who taught the same content but had students with quite different performance on the benchmark test. After the meeting, the teacher whose students had performed better began coaching his colleague 

Some teachers have found it useful to share results of interim assessments not just with colleagues but with their students. One teacher noted, “The kids even like to look at it [the interim assessment report]. That’s so cool. We talk about how we need to analyze the question.”

**In case study districts in which interim assessments were not being introduced, the changes in school staffs’ use of data generally would be described as incremental rather than dramatic.** Three case study schools reported looking at more data because their district had a new system or an enhancement to an existing system that provided greater functionality. At one of these schools, staff reported that in 2007–08 they could get data from the system directly themselves instead of going through the principal or a district staff person and this change had increased their frequency of data use. Two schools reported increased data use related to more teacher experience with a system introduced in 2006–07 or as a result of staff training. Teachers at one school reported that they were no longer satisfied with the standard reports they received from their assessment data system and had begun asking the district data staff for custom reports to address more specific questions about their students. Another school reported increased collaboration around data in the middle grades.

**School-level data underscore the complex relationship between system usability, data quality, and teacher training and experience** noted previously in Chapter 3. The change in teacher attitudes and system use at an elementary school in a case study district that had introduced an interim assessment system during 2006–07 (the year of the first site visit) illustrates the inter-connectedness of these factors. During the first year the assessment system was implemented, teachers expressed discomfort and dissatisfaction with the system. They were required to give interim math assessments (intended to be formative) from the system four times a year, and they complained that the assessments did not match their curriculum and hence the data were not very useful. Also during this first year of system use teachers relied on the principal to pull assessment data reports for them because they did not have direct access to the system that contained the data. During the 2007–08 site visit, in contrast, both individual teachers and grade-level teams were making more use of data from the districtwide interim assessments. Though the assessments were still not a perfect match to the school’s instructional sequence, the teachers had learned how to create reports of the subset of assessment items that did match what they had taught. Teachers looked at these data to determine which skills had been mastered and which should be retaught. The teachers reported that the assessment system itself had been improved with the addition of links to instructional activities and lesson plans tied to standards. Most teachers described the system as more user friendly than the one teachers had the prior year, and they reported feeling comfortable using it without the assistance of the technology coordinator or content specialist.

**Stages in Developing a Data-using Culture**

Based on the first round of school site visits in 2006–07, the research team hypothesized that school staff go through a developmental progression in their data use activities. Early uses of data are often for the purposes of school improvement planning (in many cases, a use mandated by the district and required by *ESEA* for failing schools) and for assigning students to special services or to classes at different levels. This activity typically is conducted by school, grade- level or department teams. In the next stage, a school’s teachers start using data for making decisions about what happens within their individual classrooms. At this stage, teachers look at data for the students in their classroom and do item analysis in order to make decisions about what to stress in their curriculum, how to group students with similar instructional needs, and whether or not to reteach certain material. This stage corresponds to the kind of progress monitoring that is often used in special education (Fuchs and Fuchs 2006). The third stage of data use development occurs when teachers reach the point where they begin looking at data and consulting others to help them make decisions not just about their students’ skill levels and what they should teach but about the *way* that they teach. In this stage teachers begin comparing the gains that their students have made in specific areas to the gains made by students in other teachers’ classes and start sharing the practices that have produced the best student gains. It is at this point that the school has evolved to a data using culture with mechanisms for continuing self-improvement based on the combination of data use and collaboration. Using data has become second nature to staff at schools that have attained this third stage and created a culture of data use.

Exhibit 4-4 presents the three categories of school data use hypothesized as stages along with case study examples of activity in each category. The examples illustrate the attainment of the defining characteristics of the category, but readers should note that the hypothesized stages are cumulative rather than discrete: When a school begins engaging in Category 2 activities, it continues Category 1 activities, and when school staff begin the kind of collaboration around instruction that distinguishes Category 3, they still engage in Category 1 and 2 activities as well.

**Exhibit 4-4. Categories of Data-driven Decision Making**

These categories of data use, developed by analysts based on the first round of site visits, were used as a framework for analyzing the specific examples of using data supplied by school staff during the second round of site visits. When the examples of data use described by case study school staff were classified in terms of the three data-use categories described in Exhibit 4-4, analysts found that most case study schools provided examples of Category 1 and Category 2 uses of data, but less than half of all schools gave a Category 3 example. Specifically, 35 of 36 (97 percent) case study schools provided one or more Category 1 examples; 30 of 36 (83 percent) provided one or more Category 2 examples; and 17 of 36 (47 percent) provided one or more Category 3 examples. Because the great majority of case study schools described instances of both school-level (Category 1) and classroom-level (Category 2) data-based decisions, there is no support for the study’s hypothesis that using data at the school level is typically the first step in becoming a data-using school. It does seem clear, however, that **using data to reflect on how one teaches and not just what one teaches or to whom (Category 3) is a more advanced stage in the building of a data-using culture**. Exhibit 4-5 shows the relative frequency of data-based decisions in these three categories. Exhibit 4-6 describes the activities of a middle school that had moved into this third stage of data use.

**Exhibit 4-5. Percent of Case Study Schools Reporting Each Decision-making Category**

Exhibit reads: Among case study schools, 97 percent provided examples of Category 1 uses of data. 

Source: Case study schools 2007–08.

**Exhibit 4-6. A School at Stage 3 in Data Use**

**School Supports for Data Use**

Chapter 3 described the kinds of activities that districts undertake in order to encourage their schools to use data in making instructional decisions. This section discusses data use supports implemented at the school level. The 2007 NETTS teacher survey provided a nationally representative snapshot of school supports by asking teachers to indicate which of a set of potential supports for data-driven decision making they had experienced. **The most commonly provided supports, according to the teacher survey, were professional development at the school (58 percent), principal encouragement (56 percent), and support from a consultant or mentor teacher skilled in data analysis (25 percent)**, as shown in Exhibit 4-7.

**Exhibit 4-7. Teacher-Reported Supports for Use of Student Data to Guide Instruction, by Type of Support in 2007**

Exhibit reads: Among teacher survey respondents with access to a student data system, 58 percent indicated that they had received professional development on data-driven decision making at their school.

Source: NETTS teacher survey, 2007.

Site visits to case study schools found many examples of the school supports mentioned most often in the teacher survey (professional development, principal encouragement, support from a coach or mentor teacher).  

**Professional Development**

Formal professional development that districts offer on the use of data systems was described in Chapter 3. **Case study data suggest that this formal instruction on how to use a data system or apply data to instructional decisions, though widespread, is usually quite short in duration.**In one case study district, for example, teachers reported receiving only a single hour of formal training on how to use the two electronic systems available to them. Typically, the district offers teachers just a single training session on how to get data out of the system. The research on professional development suggests that such “one-shot” training experiences are generally ineffective for changing practice (Loucks-Horsley et al. 2003; Garet et al. 2001). Site visits to case study schools in active data-using districts found that they were supporting teachers’ use of data not through additional formal professional development but rather through ongoing interactions around data with school leaders, mentor teachers or coaches, and colleagues, as described below. 

**Assistance From a Mentor Teacher or Coach**

**Having a staff member on-site who can support not just the interpretation of data but also the *connection of data to instructional decisions* appears to be a critical support.** Such individuals have credibility because of their instructional expertise and can encourage and facilitate teachers’ engagement with data. In some ways, mentor teachers or instructional coaches have an advantage over principals in this role because they are not responsible for evaluating teachers, thus removing an element of teacher anxiety. Mentor teachers or instructional coaches can provide teachers with the confidence to engage with data in an environment that feels low risk. A first-grade teacher at an elementary school implementing Reading First, for example, described how that program’s data coordinator led grade-level meetings to discuss literacy data. 

*We hear about other teachers, their students’ struggles. [The Reading First coach] runs the meetings. It’s nice to just have someone at the helm. It helps to know someone is there seeing the same issues I am seeing. We talk about ‘Let’s try these things.’ We’re good with that with literacy.*

The principal at the same elementary school noted that teachers there did not feel comfortable interpreting data by themselves: 

*When teachers are working individually, if you don’t have the background, there is a very low comfort level with understanding what it [the data] means and says. Because of that discomfort people aren’t sound and secure in making judgments. They definitely need support.*

Another good example of teachers working with instructional coaches came from an elementary school in which teachers look at data with content area coaches on an ongoing basis. During weekly cluster meetings, math and literacy coaches alternate leading the discussion, which generally involves data. At one meeting, for example, the math coach may lead the teachers in a discussion about student performance and then in brainstorming about strategies for addressing their students’ needs. At another meeting the literacy coach may lead teachers in looking at all the performance data available on one particular student. At this kind of meeting, a teacher will bring a student’s literacy folder, which includes state and district assessments, writing samples, teacher-student interviews, and other data. The teachers and the coach brainstorm together about how to work with that particular student based on this data.

Similarly teachers at an elementary school in another district described how working with their instructional coach was increasing the amount of time they spent talking about data and the extent to which they are using it. Previously, teachers met every three months to discuss data, but with the instructional coach they began meeting every month to go over their monthly action plan and DIBELS data. Said one teacher, 

*I think I’ve always been good at collecting data, but I’ve never really known what to do with it. I think that has really changed this year. Instead of wasting my time collecting all this stuff for a cute little grade book at the end of the year, I’m actually using it [data] more.*

**The case study observations suggest that the support being provided for data use by mentor teachers or coaches falls primarily into the area of linking data to instructional decisions.**There were some cases of mentor teachers or educational technology coordinators helping teachers with system use issues per se, but these were far less common than reports of assistance related to instructional practice.

**Structures and Dedicated Time for Data Use**

Teacher survey data suggest that the majority of U.S. schools are not giving teachers time during the regular work week to focus on data use (U.S. Department of Education 2009), but site visitors found that time set aside for activities that included the examination of data was present at 30 of the 36 case study schools (83 percent). 

· The middle school described in Exhibit 4-6, for example, has multiple opportunities for teachers to get together and discuss data built into its schedule because of district policies to make regularly scheduled time available to them. In addition to monthly grade-level meetings, there are weekly departmental meetings and meetings of teacher teams from different departments who share the same students. An innovative strategy at this school is the expectation that teachers create common lesson plans, addressing needs identified through the district benchmark assessment—every three weeks. Site visitors noted that teachers expressed enthusiasm for this opportunity to work together to address student needs.

· At an elementary school in another district, the assistant principal prints out student results after each interim assessment. She reviews the data with the principal, and gives each teacher data for his or her students. She asks teachers to follow up through e-mail with descriptions of what they will do in response to the data and encourages them to use the data on an ongoing basis. The assistant principal also gives all the grade-level data to team leaders. The grade-level teams hold monthly meetings at which they analyze their data. The assistant principal conducts item-level data analysis with the teams “so that we can see how many students choose a wrong answer and think about why.”

· At another elementary school the principal decided in 2007–08 to ask each teacher to select achievement goals in reading, writing and math on specific assessments for their students rather than setting grade-level goals as in prior years. Each teacher was responsible for monitoring his or her students’ performance on the related assessments in the fall and winter leading up to the year-end measure. After each round of assessment, the principal had a one-on-one conference with each teacher in which the teacher was asked, “Where are you in relation to the achievement goals you set in August? What are the one or two things between now and the end of year that you will do to measurably impact achievement goals? What is something you need from me to ensure that your goals are met at the end of the year?”

· District funding for grade-level meeting times and school improvement funds helped support multiple structures for interacting around data at an elementary school serving a high percentage of economically disadvantaged students. Teachers meet with a grade-level partner to discuss their students’ data and their instructional approaches once a week. Every other week, they participate in a grade-level team meeting at which data are discussed. Finally, each teacher meets with the principal one-on-one three or four times a year. At these meetings, the teacher presents both their data and an action plan based on the data.

**Climate of Collective Responsibility and Mutual Trust**

**A persistent theme across schools in which teachers look at data collaboratively is the importance of their sense that their colleagues respect them and their trust that the data will not be used to punish them.** As noted in studies of other school reform efforts (Bryk and Schneider 2002), mutual trust and a sense that they are jointly responsible for the success of their students appear to be important factors in teachers’ adoption of data-use practices.

As a teacher at one middle school said, “Lucky that nobody here has an ego. Everybody is very open about sharing. No fear that they will think I’m a bad teacher.” At another middle school, researchers noted the “we’re in this together” philosophy of administrators and staff. At this school names of teachers were included in data reports so that if one teacher’s class was doing particularly well in an area, other teachers could approach the successful teacher for strategies and advice. The data administrator at this school described the teachers as having “shared ownership” in the success of their students and an understanding “It’s not just about me; it’s about the whole school.”

The principal at an elementary school expressed the belief that looking at data with their colleagues motivates teachers to pay more attention to it: 

*Collaboration helps teachers’ motivation. Their kids are struggling. They see they aren’t the only ones having difficulties. They get advice from another colleague. It’s a motivation to get them to do it [use data to reflect on practice]. Internalize it more, pay more attention than if [you] work with data by yourself.*

Building this kind of mutual trust can be difficult, however. The principal quoted above noted that building trust took two years at his school and was still uneven across grades. The principal at a middle school commented, “Having that trust as a team sounds easy on paper, but teaching is so isolated. Teachers have never co-labored the way that we are asking them to co-labor now.”

**Strategies that case study school leaders used to foster the development of mutual trust included having teachers meet in small groups and starting with nonthreatening uses of data, for example by showing classroom-level assessment results without teacher names.**Teachers at an elementary school expressed their preference for small groups when they look at student data. They said that they like to be grouped with other teachers they know well. As one first grade teacher explained: 

*You feel more confident presenting your student [in a small group of teachers you know well]. You can really say what you are struggling in. You don’t feel as shy as you would if you were with a whole group of teachers you don’t know. There you are afraid to present something in case you are not doing your job well.*

At a middle school, the assistant principal first began preparing reports of benchmark test results by class with teachers’ names left off in order to preserve anonymity. When the staff held meetings to discuss the data, they got to the point where the teachers wanted to identify the colleague whose class was the one with the best performance so that they could share instructional strategies. Teachers were spending time at these meetings going around to each other to try to assign the unlabelled class assessment results to teachers. At this school it was the teachers themselves who asked the assistant principal to just put their names on the class-level assessment reports to make their collaboration around data more efficient.

The principal at another elementary school offered the following advice to other schools embarking upon bringing teachers on board with data-driven decision making: 

*Start slow with something they’re comfortable with. Teachers will be at different levels. [You] need to let each teacher move at their pace. Find someplace where there’s success so each teacher can see it’s working. Do it consistently. Make sure it’s meaningful. Don’t do it [look at data] just for its own sake.*

**Principal Leadership**

Prior case studies of data use in schools point to principal leadership as a key enabler (Supovitz and Klein 2003; Wayman and Stringfield 2006). Principals encourage data use by setting an example through their own activities, providing time in the school schedule for teachers to examine and reflect on data, and by communicating expectations around data use.

School leaders also set a tone for how to think about data. Just as the actions of principals and other school leaders convey the importance that they place on data, the way in which they frame data discussions provides a perspective on the meaning of data. A principal at a case study elementary school provided a striking example. In a year when the school had not expected to see increases in their state assessment results, but scores had in fact risen, the principal talked to the staff before showing them their scores. He asked the staff to think about the reasons why the scores might be what they were. The teachers responded by listing reasons that students might have low scores: that year they had many transient students, and many special education and English language learner students. The principal noted that the school had done some good things that year and enumerated them. He then showed the test scores the students had achieved, and after the teachers stopped cheering, he made the point that they could never use the nature of their students as an excuse again. They had shown that they could achieve progress with the students they had. The principal reported that this event was a turning point with the school environment shifting to one in which “it’s the students” was not acceptable as an explanation for poor performance.

**Beyond setting expectations for data use, principals at some of the case study schools provided tangible supports for this activity in the form of tools that teachers could use and a regular schedule of data reflection and decision-making activities.** At one of the elementary schools, for example, the principal holds monthly meetings with grade-level teams for which she prepares charts and spreadsheets to illustrate student performance. The principal then challenges the teacher teams to think hard about what might be causing the patterns they observe in the data. Whenever teachers at this school propose a new instructional plan, the principal reminds them to use data to shape their proposal.

Similarly, the principal at a middle school explained that when she asks a teacher how his or her students are doing, she expects an answer that includes a description of student data. The principal also tries to communicate the idea that the purpose of looking at data is not to judge individual teachers; rather, the point is the school’s collective responsibility for figuring out what can be done.

**In many of the case study schools, data use was an important tool in the principal’s assumption of the role as instructional leader.** In 25 of the 36 case study schools visited in 2007–08, researchers judged that principals were promoting data use in a way that gave them increased influence over the quality of instruction. Principals at 18 of the 36 schools led whole-school or grade-level meetings devoted to the analysis of data. Principals at several schools maintained schoolwide lists of students who were in danger of failing to attain state proficiency standards in order to keep on top of efforts being made to support those students during the year.   

Principals at three schools met with teachers individually to discuss their students’ needs and the teacher’s plan for the class. At one elementary school, for example, the principal was regularly working with individual teachers to help them interpret interim assessment data. He stated, “We pull [system] data right after formative assessments. We pull teachers in and make [color coded] printouts for them.” The principal refers to these 11 x 17 inch laminated printouts as “placemats.” These are designed for easy reading, with green for mastered skills or concepts and pink for areas in which students did not perform well. Using the “placemat” data, the principal works with the teacher to decide if remediation needs to be given to the entire class, a small group or an individual, and then the principal, teacher and intervention specialist work together to develop an instructional plan for standards on which one or more students need help. 

The first round of site visits conducted for the study suggested that leadership for data use did not have to come from a school’s principal: staff in other positions, such as a vice principal or instructional coach, could provide this leadership effectively, at least in schools within districts that support data use (U.S. Department of Education 2009). In the second round of case study site visits (half of which were to schools visited in the earlier round of data collection and half of which were to new schools), researchers saw more examples of strong principal leadership around data use, but again saw staff in other roles (largely assistant principals or data or instructional coaches) providing leadership and support for data activities. 

**Supports Related to Data Use**

Perhaps because they are in districts that put an emphasis on data use, the teachers in case study schools described a level of support for data use that appeared stronger than that perceived by teachers nationally. **The responses of teachers on the NETTS 2007 teacher survey suggested that most teachers with access to a student data system feel that they have someone who can help them use the system (71 percent) and that they are comfortable looking at data with colleagues (70 percent). On the other hand, on the NETTS teacher survey, only 23 percent of teachers reported that time was available for working with data during the regular school day**, and 59 percent of survey respondents agreed with the statement, “If I use the student data system, it has to be on my own time.” As noted above, the case study schools by and large had time set aside for data examination and use. They also were likely to have school-based coaches who could help teachers relate student data to instructional strategies. Of the 36 case study schools, there were instructional coaches (usually for literacy or mathematics) who helped teachers work with data in 17 and a data coach in seven. 

**The relationship among responses to different items on the NETTS teacher survey data suggests that supports such as those in case study schools have an influence on teacher behavior.**When teacher perceptions of the support items shown in Exhibit 4-8 were put into a scale (internal reliability = .61) and correlated with a scale of data use frequency items (internal reliability = .76), the correlation between perceived support and frequency of data use was 0.40. A correlation of this size is considered indicative of a moderately strong relationship in social science research .

**Exhibit 4-8. Teacher Perceptions of the SupportThey Receive for Data Use**

Exhibit reads: Among teacher survey respondents with access to an electronic data system, 71 percent agreed with the statement “I can turn to someone for help.”

Note: (R) denoted an item that is reverse coded; for these items a disagree response indicates more support for data use.

Source: NETTS teacher survey, 2007.

**Barriers to School Data Use**

Despite general district encouragement for data use and the kinds supports for the practice of data use described above, implementation of data-driven decision making and establishment of a data-using culture are not easy. School site visits identified a number of factors inhibiting growth of the practice, as described below.

**System Usability Issues**

Wayman (2005) has pointed out that for many years data systems were simply too difficult to use for implementation within schools to be feasible. Wayman suggests that major advances in this area as one of the drivers of the current movement toward school use of data. **Even with recent advances, however, many schools and districts are finding their data systems difficult for school staff to use because of either interface issues or because of limited system functionality.** District perceptions of data system usability issues were described in Chapter 3. This section discusses the usability issues raised by school staff. 

A sample criticism from one case study teacher was “[You] can’t use the back button in [the system]. . . There are lots of precise procedures.” The teacher went on to explain that if you mis-enter something, you have to start over from the beginning and “[the system] takes a while to load.” To make matters worse, “Printing from [the system] is a nightmare.”

In a number of cases, school staff complained about the lack of flexibility in the system available to them. There were other cases, however, when having a great many options proved to be overwhelming:

*Because there’s such freedom, there’s so much information, there are a thousand-and-one pull-down menus and a thousand-and-one options for whether you want to know which six weeks a kid scored in, if you want to know just the Hispanic kids or the kids who are wearing blue that day, you can get any information you want, but it’s almost overload. Sometimes you just want a simple report, and you find yourself pointing and clicking everywhere and you don’t get what you want anyway.*

Teachers in a medium-sized case study district have access to a student information system and a “data mart.” Teachers input attendance and upload assessment results into these systems, but they do not use the systems to generate reports. They told researchers that the systems are so cumbersome to use that such efforts are not worthwhile. The director of school improvement and accountability in this district commented, 

*[The system] is not an easy system to use, and we can’t make it easier. People are already turned off by it, so they don’t really want to try again. So it’s more than a training issue, it’s kind of a convincing issue. And, honestly, we don’t have someone who has time to convince and cajole people to use [it], particularly a system that is not user friendly.*

Teachers often decide whether or not to even try using a data system on the basis of reports from their colleagues. In one case study district, a representative teacher comment was “I have heard how laborious it is to enter test items on [system].” When school staff criticized a data system’s interface or usability, their comments were typically quite general (e.g., “It’s not user friendly). 

**Perceived Lack of Timeliness and Relevance of System Data**

**School staff were much more likely to criticize the quality or timeliness of the tests for which data are available in systems than to criticize the usability of the system per se.** Teacher criticisms of the quality of the data available to them included delay issues, lack of alignment with standards, lack of alignment with the school’s instructional approach, and the fact that they received only cross-sectional data rather than longitudinal data for the same set of students over time.

State achievement tests are typically a once-a-year spring event with results unavailable until the next fall when students have moved into a new grade level. Staff at many case study schools described looking at these data at the time they became available, often using them for school improvement planning and for placing students into classes or special services, as noted above. Individual teachers look at the prior spring’s scores for the students coming into their class to help do planning at the first of the year but soon find more recent and fine-grained information from classroom assessments and informal observations guiding their instructional decisions. As one middle school teacher said, “We use numbers [from standardized tests] when we don’t know them [the students], and when we do know them [the students], we don’t use numbers.”

In a large district, teachers pointed out that the state assessment results are not available until the following fall, after they no longer have the students who were tested the prior spring in their classes. Although the district gives benchmark tests six times during the school year, the results of these tests take six weeks to come back to teachers—at which point they have moved well beyond the assessed topics.

Teachers in several districts criticized the districtwide assessments that provided much of the data in the systems available to them. Teachers noted cases in which content on the district assessment did not match content standards or did not line up with what they taught in their classes. Also, in several cases teachers noted (correctly) that a content standard was assessed by only a single item on the test, making it impossible to make valid, reliable conclusions about an individual student’s mastery status with respect to that standard. 

Teachers at an innovative school in another case study district expressed distrust of the standardized test data in their system because it reflected a learning theory so different from the one that was part of the school’s mission. 

A middle school teacher expressed frustration at not being able to get longitudinal data for the same group of students over time from the district’s system: 

*We would like to see the data that reflects each child’s accomplishment over the year that you have taught him or her. We don’t have access to that information. We would like to access that information. We have to jump through hoops to get that kind of data.*

**Lack of Time to Examine and Reflect on Data**

**On the NETTS teacher survey, only 23 percent of responding teachers agreed with the statement, “Time is available [to look at data] as part of the regular day.”** As noted above, the case study schools appeared to be much more likely than schools in general to set aside time for teachers to look at data together, but there were still references in a number of the case study districts to not having enough time set aside for these activities. A teacher described her need for designated time for data use: “I would like to have an afternoon a month. In Florida, we had a half a day every month for teachers to get together to plan or look at data. If we were given time during the school day, I’d be able to do it.”

Lack of time appeared to be most problematic in the one small case study district. That district did not allow its schools to build release time for analyzing the data into the school year.  The lack of time problem was exacerbated because training on data system use in this district was limited to a brief in-service session. Informal support was provided by an unofficial “data coach,” who was actually a full-time fifth-grade teacher with many other responsibilities.  

**The case study data also illustrate that lack of time is actually a subjective perception and quite complex in practice.** For example, lack of time was reported as a barrier to data use at a middle school in a district that provided extensive time during the school week for teacher meetings. Because the meetings were not used for discussing data and most teachers were not working with data on their own time, teachers at the school had the perception that there was no time for such pursuits. At a middle school in a different district, the teachers similarly reported not having enough time to examine student data. The school’s principal expressed the opinion that the teachers were not seeing the value of being proactive in their approach to using data, and for this reason they were not making the time to analyze student and classroom level data in their meetings. Administrators at another school in this district noted that teachers there saw data use as an “add on” rather than integral aspect of instructional practice.

**District Pacing Requirements**

At a middle school in a large case study district teachers give interim assessments several times a quarter and look at interim assessment results during time set aside for professional learning community activities. The items on the tests are aligned with specific goals and teachers can do item analysis. It does not appear that teachers at this school look at the interim assessment data outside of professional learning community time; however, because they have to move on through the curriculum, regardless of what the data reveal about student mastery, to conform to the district’s pacing calendar. Responses on the NETTS teacher survey suggest that this barrier is quite commonplace. **Sixty percent of teacher survey respondents agreed with the statement, “No matter what the interim assessments for my students indicate, I have to keep up with the pacing plan for the year because of the state and district mandates.”**

**Lack of Training**

**In case study schools, the need for training in how to interpret data and connect it to instructional practice appeared greater than that for additional training on system use per se.** As noted above, the amount of time spent on formal district-provided training on data system use tended to be quite limited in the case study districts. Staff at case study schools described the training with words such as “brief” and “a few hours of information shoved into a half hour of time.” Some districts used a strategy of training the principal or a lead teacher at each school and then relying on that person to train his or her colleagues. Although only a minority of staff interviewed at case study schools indicated a desire for more formal professional development, at least one staff member did so at 17 of the 36 case study schools. 

One district was implementing a new data system and had provided limited or no professional development to school staff. An elementary school principal explained, “We need training on how to easily get your teacher roster [from the system] and how to group those kids for what they need to know in [the system], and we need the time to do this training that’s not at 3:30 in the afternoon.”

In another district a middle school teacher offered a similar opinion about the need for more training, “We do have a staff development to help us use the basics, but we still don’t know how to do everything.” A math teacher at the same school remarked, 

*We now know how to access more general class information but now...we need to access more specific information and we are kind of on our own clicking and looking around and I was able to find item-level analysis and some other really useful functions....There are a few extra steps we have to do, but I’m really excited when I figure out how to create a really good chart.*

This sentiment was echoed by a sixth-grade teacher at another school: 

*[There is] not enough training about how to interpret data. Some math whizzes get it, [They] understand things like the variability in a score. But we’re not given enough PD [professional development]. We’re trained how to access the system, how to input data, but that’s mostly grades and attendance. Maybe we were told how to access test scores.  I had to figure it out mostly on my own.*

At another district in its first year of implementing a new system with interim assessment data, teachers felt that they needed more professional development. Some teachers did not know how to use the system and others could extract data from it but did not know what to do with the data: “I wouldn’t say that everyone is absolutely certain what to do with the data that’s available. It’s an excellent resource, but I think we’re still all learning how to use and maximize it. We need more time and more...professional development.”

In some cases the more technologically savvy among a school’s staff are stepping in and providing support for their colleagues who need more training. One such teacher reported, “[We] had some training this summer where we were introduced to [system]. We had one-on-one sessions. [The principal] would pull data onto the big screen.” This teacher went on to explain: 

*A lot of it is jumping in and figuring out which reports to run. I’m comfortable with technology so it works for me. [It] might not be enough for others if they aren’t as comfortable.” In fact, this teacher has become so skilled at using her district’s data system that she was asked to provide training to her colleagues. This teacher stated: “I like to put [the system data] into Excel and run some formulas to see the percentage of standards mastered and standards hit least. That’s easier for me to see because the colors [to indicate mastery on the standard system report] stress me out. My husband has made some macros for me to use in Excel. After Christmas break, I taught a tech class. I taught the third- and fourth-grade teachers how to use the macro. They were excited.*

**Perceived Effects of Data Use**

The present study did not include a test of the effects of data use on student achievement outcomes. It did, however, collect school staff perceptions of the effects that their data use activities were having. **Among a majority of case study schools, staff perceptions of the impacts of data use were quite positive.**

· A middle school reported that, since it started using data in instructional planning, student performance in general, and that of their English language learners and special education students in particular, had improved. The principal explained that the data have given the teachers a better understanding of students’ strengths and weaknesses, “Data frees up time for teachers to focus on instruction rather than trying to figure out where each student is.”

· Teachers at a middle school in another district expressed their belief that the use of data had improved student achievement at their school. The data had helped them focus in on the math learning needs of students with disabilities, a group the school had not served well in the past. As a result, they were able to make AYP for the first time in three years.

· Staff at an elementary school that has achieved state recognition as an exemplary school despite serving large proportions of low-income, minority, and special education students attribute their success to years of examining data. The principal stated, “I really think we would not be able to perform as well as we do without that data. The data that [system] provides as well as what [the instructional advisor] does has been the core of the success of our school. If you didn’t have data, you work blindly. Data provides direction for instruction, intervention, everything.”

· A principal in another school district was able to give a specific example of the impact of data on student achievement. Several years ago, the school noted that its social studies test scores were low compared to other district schools with similar students. The principal sought advice on how to teach social studies from the principal of a school with similar student demographics and high social studies scores. After implementing the other school’s approach to social studies, the students’ test scores rose.

· A principal in an elementary school serving mostly low-income students expressed his belief that the use of data combined with “some solid direct instruction” turned that elementary school around. He explained, “I think student achievement has gone up because teachers are now teaching to kids’ skill deficits. They’re remediating instead of just teaching to the middle [achievement level] the whole year.”

· The principal at an elementary school designated in need of improvement expressed the importance of data use:

*When I came in last year, it was black and white. I used the data for dialogue with faculty, to be specific about the areas where we needed to improve. We were under improvement; it was very clear using the [state test] scores where we needed to improve. This October we made every target the state set for us, in every subcategory; we were very pleased. We are still in corrective action, as we have to maintain the scores for two years. [But] there is no question mark; we know what we need to do. The data makes this clear.*

**Summary**

School staff reports underscore the interrelated nature of the components of implementing data-driven decision making. Such efforts require user-friendly software, but more than that, there needs to be data in the system that teachers regard as worth looking at if teachers are going to use it. Training on how to use the system is particularly important if a system is not user friendly, but a lack of formal training can to some extent be compensated for through the efforts of an on-site staff person with the system skills to help colleagues.

School-level leadership appears to be an important ingredient in moving teachers from forced compliance to avid use. Principals can make the incorporation of data a standard for plans developed by school staff and for presentation of options for consideration. Principals can also make sure that joint reflection on data is on the agenda for regularly scheduled meetings of groups of teachers. 

Key challenges to school data use are developing teachers’ interest in student data and fostering their comfort in looking at data with colleagues so that they can seek help from those colleagues whose students are profiting most from instruction. Having a set of common assessments that all of a school’s teachers teaching the same content give to their students at about the same time provides one of the essential conditions for teachers to be able to sit down and share both their data and their teaching strategies. We found this practice among at least some of the teachers in close to half of the case study schools.

**5. Conclusion and Policy Recommendations**

The survey and case study findings presented in this report suggest that educational data systems are improving, but the combination of technical and human resources available in schools has yet to constitute a capacity for routine use of data to support instructional decisions. Three broad implementation themes with respect to improving local use of educational data systems come out of this work: 

· **A good data system is not enough: Use of data to inform instruction requires leadership and systemic realignment.** Data-driven decision making requires leader initiative to align curriculum and assessment practices, professional development, and data systems. 

· **Efforts to promote data-driven decision making are more successful when they are not treated as a separate innovation** but are implemented as part of district, school and teacher efforts to provide better instruction for every student.

· **Timely, credible, interim assessment data is key to motivating teachers to use data systems.** Teachers’ inclination to use data systems is affected by the nature of the system interface and the amount of training and support they receive, but these supports are largely futile if the system itself does not contain information that teachers consider relevant to their practice.

The more detailed study findings, organized by the study’s conceptual framework, are summarized below, followed by presentation of a set of policy recommendations derived from those findings.

**Supports Needed for Data-driven Decision Making**

The conceptual framework that guided instrument development and data analysis for this project emphasizes practices that support data use for the purpose of informing instruction, as described in Chapter 1. The NETTS teacher survey findings described in Chapter 4 are consistent with that conceptual framework: Teachers who report having more of the supports enumerated in the framework also report making more frequent use of data. Major findings with respect to each component of the framework are presented below.

**Data Systems**

At this point, nearly all school districts have some kind of electronic student data system, according to responses to the district survey. But districts do not necessarily have systems that provide the kinds of information needed to guide instruction. District systems are more likely to have electronic data on student demographics, attendance, grades and state test scores than they are to have data from interim assessments that can be used to improve instruction during the course of the school year. Districts are making progress in this area, however. Although only 41 percent of districts reported having had data on district-administered assessments for three years or more, 72 percent of districts said that they now have such data in electronic form (suggesting that 31 percent of districts began putting interim assessment data into an electronic system within the last two years).

**The majority of districts lack data systems adequate to support routine evaluation of district practices and decisions.** An area in which districts are making less rapid progress than they are with interim assessments is in obtaining the capability to combine data from different types of systems in order to link assessment results with instructional resources. Just 42 percent of districts have systems that can generate reports showing student performance linked to participation in specific instructional programs. Only 27 percent of districts have systems that will support queries linking student performance to school finance data.

**The case studies demonstrated that acquiring and maintaining a student data system is neither simple nor a one-time event.** Among the 12 case study districts, nominated for their leadership in the area of data use, all had multiple data systems (with a range from three to seven systems). A number of the districts described cumbersome processes for moving data between the various systems being used, and system upgrades and changes were common. In fact, about half of the case study districts were in the process of adding new system components. Several of the data systems were described as difficult to use, a condition that prompted some districts to decide that they would not ask teachers to pull down system data for themselves. Standard reports created by district staff, the school’s principal, or a data coach were the strategy of choice in such cases.

**Leadership for Educational Improvement and Use of Data**

Leadership for data use is important at both the district and the school levels. District superintendents set the tone for the amount of priority to be given to data-driven decision making. They decide both about resources (such as system acquisition and professional development on system and data use) and about processes that will be used within district offices and schools. A number of the case study districts had instituted a school improvement plan development process that included looking at data to set goals and quantitative targets for improvement. At the school level, some principals similarly set up processes whereby teachers were expected to use data to set targets for their grade’s or their individual class’ achievement. At both the district and the school levels, leaders also set general expectations about the role of data. Strong leadership for data use includes making it clear that any proposal from a staff member needs to be backed by data and will be evaluated on the basis of data.  

**Tools for Generating Actionable Data**

Teachers commonly describe the state test scores they receive in the fall for the students their grade level had the prior spring as “dead on arrival.” The teachers now have a new batch of students so the specifics of the performance of a student they had the prior year is no longer their central concern. Aggregate data from the prior year does provide some insight into areas where students in the grade level in which they work typically do well and poorly, and teachers and school leaders do use this information to look for curriculum areas needing more emphasis. But state tests necessarily give little coverage to any particular content standard because of the sheer number of standards they are trying to assess. Thus, many content standards are not covered on the state test and in some cases student performance by standard is not available to teachers. 

To get performance data for their current students at a level of detail that informs what they teach next week or next month, teachers need interim assessments that are scored and returned to them as quickly as possible. Eleven of the 12 case study districts had interim assessment systems in place, at least for a subset of subject areas and grade levels. In some cases, the assessments could be taken online so that teachers got results immediately. In other cases, districts had provided schools with scanners, and test results were turned around within a matter of days. **Based on case study data, interim assessments for generating actionable data were among the most powerful strategies districts had for getting teachers to use data.**

**Social Structures and Time Set Aside for Analyzing and Interpreting Data**

On the district survey, 92 percent of respondents identified lack of time as a barrier to the use of data for decision making. Teachers too expressed frustration with this lack of time for data use: on the NETTS teacher survey only 12 percent of teachers with access to an electronic student data system said that there was time set aside for them to examine student data and use it to guide decisions about practice. In contrast to the nationally representative data from the NETTS survey, at the case study schools in districts with a reputation for data use, teachers in 30 of 36 schools talked about regularly scheduled times, such as planning and grade-level meetings, when they could examine and discuss student data. Hence, **setting time aside within the regular work week for the examination of student data and the development of instructional decision informed by data appears to be a key characteristic of schools and districts that are leaders in data-driven decision-making.**

**Professional Development and Technical Support for Data Interpretation**

Both district and teacher surveys indicate that most districts have provided teachers with professional development on data-driven decision making, and the case study districts were no exception. A distinction can be made, however, between training that focuses on system use and training that focuses on how to interpret data for the purpose of making instructional decisions. District-provided training in the case study sample was often of the former type and was usually quite limited in duration. However, many of the case study districts that did not provide teachers with formal professional development or technical support in the area of data interpretation did make support available to teachers in the form of embedded professional development as teachers worked with an instructional coach or with colleagues on the collaborative interpretation of data. **Embedded, informal professional development provided as part of coaching or teacher collaboration around instruction appeared to offer the strongest support for teachers’ use of data in deciding what and how to teach.**

Districts themselves are seeking guidance on how to link data to instructional decisions. On the district survey, the three areas in which districts perceived the greatest need of examples of good practice were adapting instruction to meet students’ individual needs, examining data to identify which practices work best for which students, and developing curriculum-embedded formative assessments.

**Tools for Acting on Data**

Once teachers have****examined data for their students and drawn inferences about instructional needs, the next step is a plan of action that includes both an instructional approach and instructional materials. Teachers vary in terms of their knowledge of alternative approaches for teaching specific concepts and skills. Some districts and states have data systems that include instructional resources and lesson plans geared to the same content standards as the student assessments. On the district survey, 60 percent of districts reported providing this kind of online resource. In these cases, teachers have easily accessible resources to help them implement instruction based on data. In the great majority of case study districts, school staff reported that such instructional resources linked to assessment results were not provided online, but in many case study schools an instructional coach was available who could model or describe instructional strategies and provide teachers with appropriate materials.

**Data-driven Decision Making as a Systemic Endeavor**

Beyond providing support for the study’s conceptual framework, the qualitative data collected through site visits suggest that using data in a continuous improvement process requires major systemic change. Systems change or systemic reform, as defined by Smith and O’Day (1991), is not the same as scaling up an innovation (getting more schools to adopt a program). It requires coordination and alignment of all of the elements of the larger education system. In the case of data-driven decision making, systems change means that district offices that have been accustomed to functioning as separate silos—such as Information Technology, Professional Development Services, Curriculum and Instruction, and Assessment and Evaluation—need to coordinate their efforts. This challenge plays out both within district offices and within the schools that need to implement policies and requirements coming from various district offices. If districts really want to see data-driven decision making within their schools, they need to make sure that the student data system made available to teachers includes timely data that teachers view as relevant to instructional decision making. 

Within schools, processes of school improvement and teacher collaboration and planning need to be integrated with data use. It is within the schoolhouse that the important link between data and instructional decisions gets made (Herman et al. 2008).  Making this connection requires both expertise around data and assessment and knowledge of instructional strategies within the area being taught. **The use of data in decision making cannot have a positive impact on instruction without a linkage to effective instructional practices.**

The case studies revealed more detail about the interdependencies among these different aspects of school and district functioning as they relate to the use of data for making instructional decisions. The case studies illustrated the benefits of aligning and articulating goals within the district. **Districts that merged their goals for improving instruction in specific areas** (such as early literacy or mathematics) **with the goal of promoting teacher use of data were able to use school-based instructional coaches to promote data use.** These coaches were in a position both to encourage teachers to collect and use interim assessment data and to help teachers interpret student assessment results—making the important link between student data and instructional decisions.

The case studies suggested that districts wishing to promote data use among teachers would do well to make sure that they have interim assessment tools and practices that foster data use. As noted above, many of the case study districts set time aside for teachers to examine data and were in the process of putting districtwide interim assessment and reporting systems into place. In some cases, though, district policies are contradictory: they require teachers to administer interim assessments, but pacing requirements prohibit teachers from going back to reteach content that the students have not yet mastered. In still more cases, there is no district pacing requirement per se, but teachers feel pressure to cover the great breadth of material in the district curriculum. These are cases in which different parts of the district system are not well-aligned, and such tensions between competing priorities should be recognized and addressed by district leadership.  

**Recommendations**

These findings are the basis for researchers’ recommendations for actions and policies at the school district, state, and national levels. 

**Recommendations for Schools**

· *Set clear expectations around the use of student data as the basis for decisions.* Case studies illustrated the important role that school leaders play in modeling the use of data and in developing school practices in which teachers are expected to use data to guide their decisions.

· *Integrate collaborative exploration of data into existing structures for joint teacher planning and reflection on teaching.* Lack of time to examine and reflect on data is the greatest barrier to data-driven decision making according to both teacher and district survey respondents. Given the constraints of educational budgets, funding for additional time for teachers to engage in data use activities is likely to remain relatively unusual. But the case study schools illustrate how data use can become an integral part of teacher planning and collaboration time that is already funded in many districts, provided there is district and school leadership for doing so.

· *Provide a safe environment for teacher examination of their students’ performance.* Reports from case study schools suggest that teachers benefit from opportunities to examine student data with their colleagues, but that they only want to do so if they feel confident that they will not be opening themselves up to harsh judgments. Small groups of teachers who typically work together as part of a grade-level, department, or project team appear to be the most suitable for collaborative inspection of data. Keeping data reflection activities separate from performance management activities (which could affect salary or job status) helps to create a climate of trust. 

· *Support teachers in making the link between data and alternative instructional strategies.* Just having student data is not sufficient if teachers do not have ideas about how to teach differently based on student performance. Grouping students by their performance level and giving additional emphasis to content on which a majority of students did poorly are responses to data considered by most teachers. It was less common to find teachers who had ideas about different ways to teach content depending on student assessment results. Instructional coaches, however, especially in the area of early literacy, were able to bring this kind of insight to teachers in a number of the case study schools. Coaching that combines training in research-based instructional strategies and the use of student data as a basis for selecting and evaluating instructional strategies should be encouraged.

**Recommendations for Districts**

· *Think of data-driven decision making as an ongoing process rather than a one-time event centered on the acquisition of a data system.* The case study districts considered by researchers to be most advanced in their data-use practices had been actively engaged in this area for an average of more than eight years. The case studies suggest that districts will get more out of their investments in electronic data systems if they think about data-driven decision making as a systemwide innovation and develop a long-term strategy for its implementation as part of a continuous improvement process. This perspective entails treating professional development around data use as an ongoing effort rather than a one-time event.

· *Model decision making based on data and present decision-relevant data when announcing new policies.* Implementing a culture that strives for continuous improvement and uses data to drive that improvement is a major undertaking. It is unlikely that many of a district’s schools will develop this kind of culture if the district itself is not an exemplar. The shift to a data use culture within the district will require obtaining interoperability between data systems for operations such as human resources and finance and those for student acheivement.

· *Train principals in how to integrate the use of data into school improvement planning and promote their teachers’ use of data for making instructional decisions.* Case studies underscore the importance of the principal or other school leader in setting up the expectation for teachers to use data and in helping them do so. The ability to lead their schools in the use of continuous improvement processes informed by data should be a hiring criterion for school leaders. In addition, school leaders should receive ongoing training in how to connect data to school improvement and instructional decision making and in how to motivate their staff to engage in these practices.

· *Integrate the use of data-driven decision-making practices with district initiatives for improving instruction in specific areas*; when possible fund school-based coaches who support teachers in both data interpretation and linking data to instructional decisions. Data use is a means to an end rather than an end in itself. Integrating the use of data with core school improvement and reform activities gives school staff motivation for looking at data and means that resources for improvement and data use can be pooled.

· *Support time within the work week for teachers to meet with colleagues for planning, informal professional development, and data use.* If the use of data is to become an accepted part of teachers’ professional practice, it needs to be built into their regular schedule. As noted above, this does not mean that periods designated solely for data use need to be instituted. Case study schools illustrated how discussions around data can become an accepted part of teacher professional development, planning, and collaboration time. When districts fund noninstructional time so teachers can engage in these activities, they should hold school leaders responsible for making sure that the time is well used.

· *Make sure that the district has a data system that gives teachers data that is both timely and relevant to their instructional decisions.* This entails developing or acquiring good interim assessment tools and securing an infrastructure that supports the rapid turnaround of assessment results to teachers. A number of case study districts involved teachers in developing interim or end-of-course assessments for inclusion in the data systems. The use of online assessment systems or school-based scanners can ensure that teachers can get assessment results within minutes or days rather than weeks or months.

· *Provide resources and construct policies so that teachers have access to data relevant to the students they are teaching when and where they want it.* During the two years of the study, case study districts increased their teachers’ direct access to student data. The provision of data system access from home sends an important message to teachers about the centrality of student data to the profession of teaching.

**Recommendations for State and National Policy**

· *Complement efforts to improve state data systems with investments to help districts improve both their data systems and their organizational supports for using data to improve instruction.* In case study interviews, district and school staff explain that they do not make extensive use of state data systems because the information in those systems consists primarily of student scores on state tests given once a year. If the goal is to improve instructional practices within schools and classrooms, state and national policymakers should work toward improving district data systems, both in terms of the instructional relevance of the data they contain and in terms of system functionalities. Analyses reported in Chapter 2 showed that data systems in small districts have fewer query capabilities than those in larger districts, suggesting that small districts should be a special focus for state and federal assistance. Equally important, state and national policymakers should support districts in developing the organizational support systems that surround use of data by school staff. 

· *Improve the turnaround time for state assessment data so that schools receive student results in time to inform academic-year planning.*State assessment data from the prior academic year is useful to school staff mainly at the start of the school year when plans for curriculum coverage and school improvement are being made. State data may be used annually in school improvement planning, but this once-a-year activity is hindered when test data are unavailable until October or November—after students have been placed, summer professional development completed, and curriculum planned. School staff sessions for academic planning are typically held in August, prior to school opening. Local use of state test data and the systems that contain it will remain limited if states and test vendors cannot get the data back to schools in time to inform their planning activities.

· *Promote linkages between local data systems that contain interim assessment data and state systems with instructional resources geared to standards*. Many states have assembled collections of digitized resources for planning and implementing instruction around their state standards. In some cases, these collections are integrated with state data systems, but teachers are much more likely to use district data systems than state data systems. It should be possible for teachers using a district data system to examine students’ performance on a specific standard to be just a click or two away from state-collected instructional resources for that same standard.   

· *Encourage districts to invest in developing data literacy among district staff in all departments.* An essential component of the systemic alignment and integration called for above is the development of knowledge of the district’s data resources and competencies in using data to identify needs and evaluate practices among district leaders in all departments. 

· *Encourage or require school administrator preparation programs to incorporate assessment and data literacy concepts to foster continuous improvement activities informed by data as elements of school leadership training.* Becoming a data-informed instructional leader requires an understanding of assessment concepts and basic data analysis concepts. Fostering these skills and their exercise among school staff requires skill in organizational change. These competencies should be a central part of administrator preparation programs. Such programs should include exposure to good models of school and classroom-level data use.

· *Encourage or require teacher preparation programs to incorporate assessment concepts and the use of data for instructional decision making into their teaching methods courses* (science methods, language arts methods, and so on). Teachers’ data-driven decision-making skills are likely to require development and support if teachers are to use data effectively in their practice (U.S. Department of Education, in press). Teacher preparation programs should incorporate modeling of good use of data for instructional decision making. In the short-term, specially designated coaches can help with ongoing professional development and technical assistance, but longer term, assessment interpretation and data use skills should be covered more extensively as part of teacher preparation.

· *Provide districts with good examples of practices that support the development of a data-use culture within schools.*Responses to the district survey indicate that district staff believe that they need exemplary models of how to connect student data to instructional decisions. Districts will have a hard time supporting staff at their schools in this area if they themselves lack a concrete understanding of how to do it. The areas in which district staff indicated having the greatest need for good models were examining student data to determine which practices work best for which students, adapting instructional practices to meet students’ individual needs, and developing curriculum-embedded formative assessments.

In the years since the reauthorization of the *Elementary and Secondary Education Act* in 2002, both federal policy and private advocacy partnerships (Data Quality Campaign 2006) have put heavy emphasis on the improvement of states’ student data systems. Efforts began in 2005 to support states in improving the quality of the data in their systems and acquiring or developing systems capable of tracking longitudinal changes in achievement for individual students and student subgroups. The Institute of Education Sciences (IES) began awarding three- to five-year grants to states for developing or expanding longitudinal data systems in FY2006. 

States have made extensive progress in improving their systems in these areas, and now are being asked to make their systems useful not just for accountability purposes but also for continuous improvement (Data Quality Campaign 2009b). The FY2009 IES call for state applications for student longitudinal data system grants, for example, stipulates, “The system should provide data that can be used in education decision-making at multiple levels, from policy to classroom instruction” (Institute of Education Sciences June 2008, p. 3). 

Findings reported here make it clear that at the time that the data for this study were collected (2005–08), school and district staff were making little use of state systems to support their decision making. But it should be kept in mind that efforts to make state systems useful for such purposes were in their infancy. As the first of these efforts to make state systems useful to practitioners mature, it will be important to reexamine local data use practices in states that have received federal funding for data system improvement, with a special focus on small districts with limited local capacity for data system acquisition and support, in order to judge whether state systems and associated supports can fill local needs. 

The findings reported here make it clear that the data systems themselves, the assessment data that populate district data systems, and school and district practices around data use are all changing rapidly. This field warrants continued study to identify emerging best practices and test the causal linkage between data use practices and improvements in student learning. 

# References

Brunner, C., C. Fasca, J. Heinze, M. Honey, D. Light, E. Mandinach, and D. Wexler. (2005). Linking data and learning: The Grow Network study. *Journal of Education for Students Placed At Risk,*10, no.**3: 241–267.

Bryk, A. S., and B. Schneider. (2002). *Trust in schools: A core resource for improvement.* New York: Russell Sage Foundation. 

Coburn, C. E., J. Toure, and M. Yamashita. (2009). Evidence, interpretation, and persuasion: Instructional decision making at the district central office. *Teachers College Record*.

Coburn, C. E., and J. E. Talbert. (2006). Conceptions of evidence-based practice in school districts: Mapping the terrain. *American Journal of Education*,**112, no. 4: 469–495.

Confrey, J., and K. M. Makar. (2005). Critiquing and improving the use of data from high-stakes tests with the aid of dynamic statistics software. In *Scaling up success: Lessons learned from technology-based educational improvement,* eds. C. Dede, J. P. Honan, and L. C. Peters, 198–226. San Francisco, Calif: Jossey-Bass.

Copland, M. A. (2003). Leadership of inquiry: Building and sustaining capacity for school improvement. *Educational Evaluation and Policy Analysis,*25, no. 4: 375–395.

Consortium for School Networking. (2004). *Vision to know and do: The power of data as a tool in educational decision making.* Washington, D.C.: Author.

Data Quality Campaign. (April 2009a). Data-driven districts: Building the culture and capacity to improve student achievement. *DQC Brief*. http://www.dataqualitycampaign.org/files/DQCbrief_FINAL-lowres_2_.pdf (accessed Aug. 4, 2009).

Data Quality Campaign. (April 2009b). DQC brings together education leaders to urge states to use data systems for continuous improvement. *DQC Newsletter*, 3, Issue 7. http://www.dataqualitycampaign.org/files/DQCNewsletterApr09.pdf (accessed May 1, 2009).

Data Quality Campaign. (2008). *Results of 2008 NCES survey of state P–12 data collection issues related to longitudinal analysis*. http://www.dataqualitycampaign.org/survey_results/ (accessed Nov. 18, 2008).

Datnow, A., V. Park, and P. Wholstetter. (2007). *Achieving with data: How high-performing school systems use data to improve instruction for elementary students*. Los Angeles, Calif: University of Southern California, Center on Educational Governance.

Elmore, R. (2004). *School reform: From the inside out*. Cambridge, Mass.: Harvard Education Press.

Feldman, J., and R. Tung. (2001). Using data-based inquiry and decision-making to improve instruction. *ERS Spectrum,*19, no. 3: 10–19.

Fuchs, D., and L. S. Fuchs. (January/February/March 2006). Introduction to response to intervention: What, why, and how valid is it? *Reading Research Quarterly*. http://www.reading.org/Library/Retrieve.cfm?D=10.1598/RRQ.41.1.4&F=RRQ-41-1-Fuchs.html (accessed Jan. 8, 2009).

Garet, M. S., A. C. Porter, L. Desimone, B. F. Birman, and K. S. Yoon. (2001). What makes professional development effective? Results from a national sample of teachers. *American Educational Research Journal,*38, no. 4: 915-945.

Herman, R., P. Dawson, T. Dee, J. Greene, R. Maynard, S. Redding, and M. Darwin. (2008). *Turning around chronically low-performing schools: A practice guide* (NCEE #2008-4020). Washington, D.C.: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences. http://ies.ed.gov/ncee/wwc/practiceguides (accessed June 12, 2008).

Hoffman, J. (2007). *Numbers and types of public elementary and secondary education agencies from the Common Core of Data: School year 2005–06 (NCES 2007–353)*. U.S. Department of Education. Washington, D.C.: National Center for Education Statistics. http://neces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007353 (accessed Aug. 14, 2008).

Honig, M. I., and C. E. Coburn. (Winter 2005). When districts use evidence to improve instruction: What do we know and where do we go from here? *Voices in Urban Education,*6. http://www.annenberinstitute.org/ VUE/winter05/Honig.html (accessed Nov. 13, 2007).

Institute of Education Sciences. (June 26, 2008). *Statewide Longitudinal Data System Grants.* CFDA No. 84.372A. http://nces.ed.gov/Programs/SLDS/grant_information.asp

Knapp, M. S., J. A. Swinnerton, M. A. Copland, and J. Monpas-Huber. (October 2006). *Data-informed leadership in education.* Seattle, Wash.: Center for the Study of Teaching and Learning.

Koretz, S. (2008). Further steps toward the development of an accountability-oriented science of measurement. In *The future of test-based educational accountability,* eds. K. E. Ryan and L. A. Shepard, 71–91. New York: Routledge, Taylor and Francis.

Loucks-Horsley, S., N. Love, K. E. Stiles, P. W. Hewson, and S. Mundry. (2003). *Designing professional development for teachers of science and mathematics.* Newbury Park, Calif.: Corwin Press.

Love, N. (2002). *Using data/getting results:**A practical guide for school improvement in mathematics and science*. Norwood, Mass.: Christopher-Gordon Publishers, Inc.

Mandinach, E. B., L. Rivas, D. Light, and C. Heinze. (April 2006). *The impact of data-driven decision making tools on educational practice: A systems analysis of six school districts*. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. http://cct.edc.org/reports.asp?numStartCount=6&numTopicID=25&numPubTypeCatID=2 (accessed Aug. 3, 2009).

Mandinach, E. B., M. Honey, D. Light, J. Heinze, and L. Rivas. (2005). Technology-based tools that facilitate data-driven decision making. In *Toward sustainable and scalable educational innovations informed by the learning sciences,* eds. C. K. Looi, D. Jonassen, and M. Ideda, 267–274. Amsterdam: IOS Press.

Marsh, J. A., J. F. Pane, and L. S. Hamilton, (2006). *Making sense of data-driven decision making in education.* Santa Monica, Calif.: RAND.

Massell, D., and M. E. Goertz, (2002). District strategies for building instructional capacity. In *School districts and instructional renewal,* eds. A. M. Hightower, M. S. Knapp, J. A. Marsh, and M. W. McLaughlin, 43–60. New York: Teachers College Press. 

Palaich, R. M., D. G. Good, and A. van der Ploeg, (2004). State education data systems that increase learning and improve accountability. *Policy Issues,* No. 16. Naperville, Ill.: Learning Point Associates.

Pellegrino, J. W. (in press). Technology and formative assessment. In *International encyclopedia of education,* 3rd ed., eds. E. Baker, B. McGaw, and P. Peterson. Oxford, U.K.: Elsevier.

Smith, M. S., and J. O’Day. (1991). Systemic school reform. In *The politics of curriculum and testing,* eds. S. H. Fuhrman and B. Malen, 233–267. New York: Falmer.

Snipes, J., F. Doolittle, and C. Herlihy. (2002). Foundations for success: Case studies of how urban school systems improve student achievement. Washington, D.C.: Council of Great City Schools.

Supovitz, J. A., and V. Klein. (November 2003). *Mapping a course for improved student learning: How innovative schools systems actually use student performance data to guide improvement*. Philadelphia, Pa.: Consortium for Policy Research in Education, University of Pennsylvania.

Thorn, C. A. (2002). Data use in the classroom: The challenges of implementing data-based decisionmaking at the school level. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.

Tourangeau, R. (2003). Cognitive aspects of survey measurement and mismeasurement, *International Journal of Public Opinion Research,*15, no. 1: 3–7.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (in press). *Supporting teachers’ ability to use data to inform instruction.* Washington, D.C.: Author.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2009). *Implementing data-informed decision making in school —Teacher access, supports and use*. Washington, D.C.: Author.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development. (2007). *Teachers’ use of student data systems to improve instruction 2005 to 2007*. Washington, D.C.: Author.

Wayman, J. C. (2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. *Journal of Education for Students Placed At Risk,*10, no. 3: 295–308.

Wayman, J. C., and V. Cho, (2009). Preparing educators to effectively use student data systems. In *Handbook of data-based decision making in education,* ed. T. J. Kowalski and T. J. Lasley II, 89–104. New York, N.Y.: Routledge.

Wayman, J. C., V. Cho, and M. T. Johnson. (2007). The data-informed district: A district-wide evaluation of data use in the Natrona County School District. Austin, Texas: The University of Texas.

Wayman, J. C., and S. Stringfield, (2006). Technology-supported involvement of entire faculties in examination of student data for instructional improvement. *American Journal of Education,*112, no. 4: 549–571. 

Wayman, J. C., S. Stringfield, and M. Yakimoski. (2004). Software enabling school improvement through analysis of student data. *CRESPAR Technical Report* No. 67. Baltimore, Md.: Johns Hopkins University.

Wilhelm, J. (2007). *Engaging readers and writers with inquiry*. New York: Scholastic.

Young, V. M. (2006). Teachers’ use of data: Loose coupling, agenda setting, and team norms. *American Journal of Education,*112, no. 4, 521–548.

Zavadsky. H. (2009). Building data-driven district systems: Examples from three award-winning urban systems. In *Handbook of data-based decision making in education,* eds. T. J. Kowalski and T. J. Lasley II, 89–104. New York, N.Y.: Routledge.

**Appendix A**

**Methods**

# District Survey Sample 

The National Technology Activities (NTA) district survey was administered between October 2007 and February 2008. Of the 529 districts in the final sample, 427 responded for a response rate of 81 percent. The response rate within each cell of the sampling frame was 72 percent or higher. To conduct analyses with reasonable statistical precision, a six-cell sampling frame stratified by district size and poverty rate was created. The process for identifying the sampling strata is described below.

The national population of districts was sorted into three size categories such that each category contained districts serving approximately one-third of the nation’s public school students, based on enrollment data provided by the National Center for Education Statistics’ Common Core of Data (CCD):

· **Large** (estimated enrollment 25,800 or greater). These are either districts in large urban centers or large county systems, which typically are organizationally complex and often are broken up into subdistricts.  

· **Medium** (estimated enrollment from 5,444 to 25,799). These are districts set in small to medium-sized cities or are large county systems. They also are organizationally complex, but these systems tend to be centralized.

· **Small** (estimated enrollment from 300 to 5,443). The small district group typically includes suburban districts, districts in large rural towns, small county systems, and small rural districts. These districts tend to have more limited organizational capacity.

Districts with 299 or fewer students were excluded from the study. Such districts account for approximately only 0.1 percent of all public school students. The distribution of districts among the three size strata and the proportion of public school students accounted for by each stratum are displayed in Exhibit A-1. The proportion of districts among the three size strata in the district sample (excluding districts with 299 or fewer students) are large (2.2 percent), medium (13.5 percent), and small (84.3 percent).

**Exhibit A-1**

**Distribution of Districts and Student Population, by District Size**

Source: Based on 2004–05 NCES Common Core of Data (CCD).

As a measure of district poverty, the percentage of children ages 5 to 17 who are living in poverty, as reported by the U.S. Census Bureau and applied to districts by the National Center for Education Statistics, was used. The distribution of districts among strata and the proportion of students accounted for by each stratum are displayed in Exhibit A-2.

**Exhibit A-2**

**Distribution of Districts and Student Population, by District Poverty Rate**

Note: Distribution excludes districts with 299 or fewer students. 

Source: Based on data from the 2003 U.S. Census for the percentage of children ages 5 to 17 who are living in poverty and applied to districts by NCES.

A sample of 529 districts was drawn with the goal of obtaining approximately 500 respondents. The two variables of district size and poverty rate were used to generate a six-cell grid into which the universe of districts (excluding very small districts) could be fit. From the population of districts in each cell, 89 districts were sampled (except in the cell representing large high-poverty districts in which only 84 districts were available after removing districts participating in case studies). Exhibit A-3 shows the strata, distribution of the number of districts in each stratum, and the sample size in each cell. 

**Exhibit A-3**

**Number of Districts in the Universe and Sample Size, by Stratum**

Note: A total of 89 districts in each cell were sampled except for large, higher-poverty districts of which only 84 remained after eliminating case study districts. As a result, the final sample consisted of 529 districts. 

# Case Study Sample

A two-stage sample process was used to select the 2007–08 case study districts. First, six districts were selected for a second round of data collection because they had been in the process of implementing new data systems or activities to support the use of student data at the school level and warranted another visit. The second stage involved identifying an additional group of six districts that have been active in data-driven decision making. These districts were drawn from the pool of districts that remained after the initial selection of 10 districts in 2006, supplemented by additional districts identified as active data users. These included Broad prize winners and nominees, and selected districts participating in focus groups at the U.S. Department of Education to discuss issues related to education technology (i.e., districts using data to support instructional decision making).

For both groups of districts, the research team worked with district administrators to identify three appropriate schools, based on the following criteria:

· One elementary or middle school that the district considers *high* in its data use practices.

· One school that has shown improvement in its use of data to guide instruction (*emerging*).

· One school that is *typical*of the district with respect to use of data systems.

Researchers asked the district to recommend, to the extent possible, three schools serving demographically similar students at the same grade level (either elementary or middle school). Within each school, the principal, an instructional or data coach (if applicable), and six teachers were interviewed.

The demographic data for the 12 districts and 36 schools in the 2007–08 site visit sample are shown in Exhibit A-4. For each district visited, respondents included key staff involved in the district’s data-informed decision-making activities (e.g., chief information officers, directors of curriculum and instruction, directors of research and evaluation, directors of accountability, directors of professional development). Among the districts visited for a second time, questions focused on the impact of changes since the first round of data collection.

**Exhibit A-4. Case Study Districts and School Sample in 2007–08**

Exhibit A-4 continues on next page

**Exhibit A-4. Case Study Districts and School Sample in 2007–08 (continued)**

Notes: Numbers have been used to label districts and schools for confidentiality reasons. To support comparisons of data, the same number for a district is used throughout the report (for those districts visited a second time, the number corresponds to that used in the interim report). District size categories are based on those for the district survey sample. The asterisk denotes that one of the elementary schools included in the school sample is from a neighboring district (enrollment of 681) because this case study district had only one elementary and one middle school.

Teachers to be interviewed were nested within the purposive sample of schools. Even though schools were selected because they used student data systems in instructional decision making, variability was expected across teachers with respect to how they were using data for instructional decision making. Such variability is typical for educational reforms and was found for data-driven decision making in earlier school case studies (Marsh, Pane, and Hamilton 2006). Therefore, project staff requested that the principal of each case study school nominate three active practitioners of data-driven decision making and three teachers who represented average users. In this way, the study expected to capture “best practices” within the school but still maintain a realistic perspective with regard to the pervasiveness of those practices.

# NETTS Survey Data

As part of the National Educational Technology Trends Study (NETTS) 1,039 district technology directors were surveyed during spring 2007, and 2,509 teachers were surveyed in spring 2007. The teachers were clustered in schools sampled from districts participating in the NETTS district survey. The NETTS district sample of 1,039 districts was nationally representative with respect to poverty status, student enrollments, and location (urban or rural status). The 60 largest urban school districts across the country were selected with certainty (i.e., included in the sample from the outset). Districts composed entirely of special education schools and vocational-technical schools, as well as independent charter schools that are their own districts, were excluded from the district sampling frame because of their dissimilarity to “typical” districts. The teacher sample was created by drawing a probability sample of 975 schools from respondents to the district survey, stratified by school type (elementary or secondary), and poverty level (high or low). Schools were randomly sampled in proportion to the number of teachers and in inverse proportion to district size to produce a sample of schools whose selection probabilities were roughly independent of the size of their district’s enrollment. 

Higher-poverty schools were oversampled (233 of the 975 schools) to obtain more precise data about their technology use. For schools, “higher poverty” was defined as above a specified cutoff in terms of the percentage of students who were eligible for the free or reduced-price lunch program. The dividing line between higher-poverty and lower-poverty schools was selected to ensure that for each school type (elementary, middle, or high school), there would be the same number of teachers in the higher-poverty and the lower-poverty groups, as reported in the National Center for Education Statistics Common Core of Data (CCD). Elementary schools with 29.7 percent of their students eligible for free or reduced-price lunches were classified as higher poverty. For middle and high schools, the poverty thresholds were 24.3 percent and 15.9 percent, respectively.

NETTS researchers obtained teacher rosters for the 975 schools within the districts selected for the NETTS district survey. To be eligible for the teacher sample, a teacher had to be teaching at the same school in the school year prior to survey administration (i.e., teachers new to the school were excluded). Teachers who did not teach core academic subjects also were omitted from the sample. Targets of four teachers from each of the schools were randomly selected for the teacher sample. The final teacher sample in 2007 consisted of 1,779 teachers from 865 schools. 

Response rates were 94 percent for the district survey in 2007 and 85 percent for the teacher survey. Sampling weights were applied to the teacher data to obtain nationally representative estimates. The district and teacher surveys were initially administered during the 2004-05 school year to the same sample of districts and schools that provide the basis of comparison between 2007 and 2005 for the teacher survey data. In fall 2005, 6,017 teachers were surveyed with a response rate of 82 percent; the larger sample was designed to provide robust, school-level estimates of technology use. For additional information on the comparisons between 2007 and 2005 teacher survey responses, see *Teachers’ Use of Student Data Systems to Improve Instruction 2005 to 2007* (U.S. Department of Education 2008).

**Appendix B**

**2007–08 District Survey and Frequencies**

**Appendix C**

**Terminology and Additional Data**

Note: Exhibits C-3 through C-16 provide additional information on data discussed in the text. Where applicable, the corresponding exhibit in the text is identified next to the exhibit number (e.g., Exhibit C-3 provides information on variation by district size for data presented in Exhibit 2-2 in Chapter 2).

**Exhibit C-1**

**Definitions of Terms**

Exhibit C-1 continues on next page

**Exhibit C-1 (continued)**

**Definitions of Terms**

**Exhibit C-2**

**Top 15 Systems for Each Component***

Note: Not all components include 15 systems because of duplications.

*  Cited by survey respondents. These categories may not correspond to the purposes for which the systems were developed.

**Exhibit C-3 (Exhibit 2-2 in Text)**

**Profile of District Data System Elements in 2007–08, by District Size**

**Exhibit C-3 (continued)**

**Profile of District Data System Elements in 2007–08, by District Size**

Note: Shaded numbers represent the median percent of subelements present. Numbers for each of the subelements are the percent of districts that have that subelement. Asterisks indicate the proportion of districts with a particular subelement vary significantly by district size (**p* < .05, ***p*< .01 and ****p* < .001).

Source: 2007–08 district survey questions 4, 5, 7, 8, 12-14.

**Exhibit C-4 (Exhibit 2-6 in Text)**

**District Data System Query Capabilities in 2007–08, by District Size**

Note: Asterisks indicate the proportion of districts that have this query capability vary significantly by district size (**p* < .05 and ***p* <. 01).

Source: 2007–08 district survey questions 7 and 8.

**Exhibit C-5 (Exhibit 2-7 in Text)**

**District Data System Features and Tools in 2007–08, by District Size**

Source: 2007–08 district survey question 8.

**Exhibit C-6a (Exhibit 2-8 in Text)**

**Districts Using Data Systems for Selected Purposes, by District Size**

**Exhibit C-6a (continued)**

**Districts Using Data Systems for Selected Purposes, by District Size**

Note: Asterisks indicate the proportion of districts that carry out a particular activity vary significantly by district size (*p < .05, ***p*< .01).

Source: 2007–08 district survey questions 18 and 19.

**Exhibit C-6b**

**Districts Not Conducting Selected Activities, by District Size**

Note: Asterisks indicate the proportion of districts that report they never carry out a particular activity vary significantly by district size (*p < .05, ***p*< .01, and ***p<.001).

Source: 2007–08 district survey questions 18 and 19.

**Exhibit C-7 (Exhibit 3-1 in Text)**

**District Data-informed Decision-making Support Index, by District Size**

**Exhibit C-7 (continued)**

**District Data-informed Decision-making Support Index, by District Size**

Note: Asterisks indicate proportion of districts with a particular element varies significantly by district size (**p* < .05, ** *p* < .01 and *** *p* < .001).

Source: 2007–08 district survey questions 4, 8-10, 21-23.

**Exhibit C-8 (Exhibit 3-4 in Text)**

**District-provided Training on the Use of Data and Data Systems**

Source: 2007–08 district survey question 21.

**Exhibit C-9 (Exhibit 3-5 in Text)**

**District-provided Supports for School-level Use of Data to Improve Instruction**

Source: 2007–08 district survey question 22.

**Exhibit C-10 (Exhibit 3-6 in Text)**

**District Policies and Practices Encouraging Schools’ Use of Data**

Source: 2007–08 district survey question 22.

**Exhibit C-11 (Exhibit 3-7 in Text)**

**District Administrators’ Perceptions of Barriers to Increased Use of Data Systems**

**Exhibit C-11 (continued)**

**District Administrators’ Perceptions of Barriers to Increased Use of Data Systems**

Note: Asterisks indicate significant differences by district size (**p*< .05 and ** *p*< .01).

Source: 2007–08 district survey question 25.

**Exhibit C-12 (Exhibit 3-9 in Text)**

**District Perceptions of Needed Examples of Good Practice**

Source: 2007–08 district survey question 24.

**Exhibit C-13**

**Longevity of District Engagement in Helping Schools Use Data, by District Size**

Source: 2007–08 district survey question 23.

**Exhibit C-14**

**District Administrators’ Perceptions of Need for Examples of Good Practice, by District Size**

Note: Asterisks indicate the extent to which this need varies significantly by district size (***p* < .01).

Source: 2007–08 district survey question 24.

**Exhibit C-15 (Exhibit 3-10 in Text)**

**District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, by District Size**

**Exhibit C-15 (continued)**

**District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, by District Size**

**Exhibit C-15 (continued)**

**District Administrators’ Perceptions of Barriers to Increased Use of Data Systems, by District Size**

Note: Asterisks indicate the extent to which this barrier varies significantly by district size (**p*< .05 and ***p*< .01).

Source: 2007–08 district survey question 25.

**Exhibit C-16**

**Districts Conducting Selected Activities, by Percent of Schools Not Making AYP**

Note: Asterisks indicate the proportion of schools not making AYP within a district varies significantly by district size for whether or not a district carries out a particular activity (**p* < .05 and ***p*< .01).

Source: 2007–08 district survey questions 18 and 19.

From Accountability to Instructional Improvement

Use of Education Data at the Local Level

U.S. Department of Education

1. Data systems

2. Leadership for improvement and use of data

3: Tools for generating data

4. Social structures and time set aside for reflection on data

6. Tools for acting on data

5. Professional development and technical support for data interpretation

Prerequisites and Supporting Conditions

	24

Data System and Technology Barriers

Lack of funding to expand or improve the data system

Information located in multiple data bases and not linked*

Lack of trained technical staff*

Data stored in forms that are difficult to access, manage, interpret*

Out-of-date hardware

Lack of sufficient hardware

Internet connections that are not fast or reliable enough

Unreliable or inaccurate data in system*

Lack of districtwide unique teacher identification numbers

Lack of districtwide unique student identification numbers

Inability to provide adequate safeguards and security for the data*

          Logistical and Other Barriers

Lack of time for school staff to conduct data-driven decision making 

Lack of resources to train school staff on how to use data

Lack of teacher preparation on how to use data

Lack of technical skills of school staff to use data systems

Lack of building administrator preparation on how to use data

Lack of a clear vision or strategic plan for data-driven decision making

Lack of sharing of data across departments within the district

Lack of incentives for data used by district or school staff

Lack of district leadership support for data-driven decision making

Policies that preclude direct access by school staff to data

Exhibit C-3 continues on next page.

Exhibit C-6a continues on next page.

Exhibit C-7 continues on next page.

Exhibit C-11 continues on next page.

Exhibit C-15 continues on next page.

Exhibit C-15 continues on next page.

�  For the remainder of this report, the percentages reported will be for districts indicating that they have an electronic student information data system (which is greater than 99 percent of all surveyed districts) unless otherwise specified. 

�  One school’s change in the use of data could not be classified.

�  District survey respondents also reported that lack of time for school staff to conduct data-driven decision making activities was the greatest barrier to spreading data-driven decision making practices throughout their district (92 percent of districts cited lack of time as a barrier).

�  For additional information on the district identification process and characteristics of the first set of case study districts, see Implementing Data-Informed Decision Making in Schools—Teacher Access, Supports and Use (U.S. Department of Education 2009).

�  The response rate for the NETTS teacher survey was 86 percent. 

� 	As noted on page 6, the 2006(07 case study sample for the current study included an additional three districts that were not revisited in 2007–08, which also helped inform the second year of data collection.

�  These 10 districts varied significantly from those with data systems in terms of district size. There were no large districts represented among the 10, whereas one-third of districts with data systems were large. There were no significant differences by percentage of students in poverty, in the percentage of their schools that are Title I (58 percent compared to 62 percent) or the percentage of their schools not making AYP (15 percent compared to 26 percent for districts with a data system).

�  In this report the word “significant” should be interpreted as “statistically significant.” The term “statistically significant” means that the observed difference in a pair of values is not likely due to random variation in the sample. The annotations p < .05, p < .01, etc., indicate the probability that these differences are due to random chance (i.e., less than a 5 percent probability, a 1 percent probability, etc.). When differences are not marked with an asterisk, we cannot reliably rule out the possibility that they are due to random variation in the sample. This report uses the standard convention of *p < .05, **p < .01 and ***p < .001.

�  Unless otherwise stated, when testing to see whether a key measure varies with a categorical variable such as district size we employ two common tests, both adjusted to account for the stratified cluster sampling method used to collect the survey data. When the measure is continuous, we test the equivalence of the mean value over the levels of the categorical variable. When the measure is categorical (such as whether a district utilizes a particular type of data system or not), we conduct an adjusted test of independence between the two variables. If this first omnibus is significant, we report that the measure varies by the categorical variable. When the categorical variable reflects more than two levels (as in district size, with small, medium, and large levels), we then examine pair-wise contrasts across all category levels and report on statistically significant contrasts. For this comparison F(1,411) = 9.86, p < .01. All statistical tests were adjusted to account for the stratified cluster sampling method used to collect the survey data. 

�  F(1,411) = 12.09, p < .001.

� The $265 million investment refers to the total cost of the grants to date, including continuations to be paid out of FY 2010 and FY 2011 funds. The total appropriations include $24.8 million in FY 2005, $24.552 million in FY 2006, $24.552 million in FY 2007, $48.293 million in FY 2008, and $65 million in FY 2009, plus $250 million in 2009 ARRA funds. The long-term goal of the State Longitudinal Data Systems Grant Program sponsored by the Institute of Education Sciences (IES) is to enable all states to create comprehensive P–20 systems that permit the generation and use of accurate data to support analysis and informed decision making at all levels of the education system, increase the efficiency with which data may be analyzed to support continuous improvement, facilitate research, support education accountability systems, and simplify reporting. Since 2005, 41 states and the District of Columbia have received these multi-year, competitive grants (see the IES Web site at http://nces.ed.gov/programs/slds/).

� States have made progress in developing longitudinal data systems according to the 2008 survey by the DQC (see http://www.DataQualityCampaign.org/survey_results/). In 2005, no state reported having all 10 essential elements of a state longitudinal data system and in 2008 only six states had them (47 states plan to have eight or more of the 10 elements in place within three years). In 2008, 48 states have five or more of the 10 elements. At the same time, many states currently lack elements that would allow them to conduct analysis that could facilitate instructional decision making. For example, only 21 states have a teacher identifier system that provides them with the ability to match teachers to students and only 17 states collect student-level course completion and transcript information. States report that it is not a lack of technological know-how that is keeping them from building a more robust data system but rather lack of political will and resources to implement the elements and change the culture around data use (Data Quality Campaign 2008).

� The DQC/APQC benchmarking project surveyed 57 districts in 2008 (five of the districts in their survey sample overlap with the current case study sample).

� Twenty-four percent of high-poverty districts used a student information system designed locally or by the state, whereas only 13 percent of low-poverty districts used such a system. F(1,411)=4.32, p < .05.

�  The NETTS district surveys indicated that the types of data most frequently stored electronically were attendance, student demographics, special education information, student grades, and course enrollment histories. The 2007 survey conducted as part of the study reported here showed that the most frequent types of data maintained in electronic form over time (the same format for three years or longer) were student demographics (70 percent), course enrollment histories (69 percent), attendance and special education information (both 68 percent), and student grades (67 percent). See Teachers’ Use of Student Data Systems to Improve Instruction: 2005 to 2007 (U.S. Department of Education 2008) available at http:/www.ed.gov/rschstat/eval/tech/teachers-data-use-2005-2007/teachers-data-use-2005-2007/-intro.html.

� Following a finding that the proportion of districts with a specific capability varied by district size, pair-wise contrasts across all possible combinations of small, medium, and large districts were examined. For each of the four capabilities that varied significantly by district size we found that smaller districts were significantly less likely to have this capability, when compared to large and medium districts (p < .05). Data by district size for Exhibits 2-6 and 2-7 can be found in Appendix C, Exhibits C-4 and C-5.

� Similarly, analyses contrasting high- and low-poverty districts found no significant differences in their systems’ query capabilities or inclusion of the tools identified in Exhibit 2-6. 

� Differences by subelement are presented in Appendix C, Exhibit C-3 (p < .05 to p < .001). 

� The index was created from responses from multiple district survey items combined into a handful of key data supports as was done in Exhibit 2-2 for data system elements.

� As discussed in Chapter 2, data from the case studies suggest that the concept of data accuracy is not one-dimensional—where data have consequences attached (e.g., accountability) accuracy is critical, but data used to make ongoing adjustments to instruction do not require as much quality control.

� In the NETTS 2007 national teacher survey, teachers expressed a desire for more professional development in using data, particularly for training in how to develop diagnostic assessments and how to adjust instruction based on diagnostic data. They also indicated that they lacked knowledge in using data systems (e.g., forming data queries, finding the information they want) and that the electronic data system available to them was hard to use (U.S. Department of Education 2009).

� Differences by subelement are presented in Appendix C, Exhibit C-7, p < .01 to p < .001.

� The responses to questions 18 and 19 were recoded such that a 0 indicates never performing the activity, while a 1 indicates performing the activity at least annually. These recoded responses were regressed on the district capacity building score using a weighted logistic regression model with adjustments for the stratified sampling design. A statistically significant coefficient for the capacity building score was interpreted to indicate an association between the capacity building score and the specific activity. 

� The purpose of dashboards is to distill large amounts of data into a simple format to highlight problems and successes. Data dashboards include a series of key indicators at the district or school level that are tracked on a regular basis (e.g., academic performance, safe schools, attendance, accountability measures). The dashboard provides a snapshot at a specified point in time so that educators have a clear and timely picture of how their district or school fares on each measure so that they can take actions as needed—similar to a dashboard in an automobile with gauges that help the driver to monitor, at a glance, the speed of the car, the gas tank, oil pressure, and distance traveled.

� When multiple informants from the same school described the same incident of data use, the incident was counted only once. 

�  Some critics charge that ESEA has placed so much focus on annual achievement test scores that schools are “teaching to the test” rather than helping students learn more broadly (Koretz 2008). Often times school practices fall into a gray area. Some might view these practices as an appropriate focus on important content, whereas others might see an inappropriate emphasis on becoming test wise.

� Good practice with respect to student privacy would require that teachers aggregate their students’ data to the class level or mask student names so that only a student’s teacher or teachers have access to his or her performance.

� This finding is similar to district reports on the national district survey, in which lack of time for school staff to conduct data-driven decision making activities was the most frequent barrier reported by districts to spreading data-driven decision-making practices throughout their district (92 percent of districts cited lack of time as a barrier).

� The likelihood of having time set aside for data use did not appear to vary by school level within the case study sample. Among case study schools, 11 of 13 middle schools had dedicated time for data use as did 19 of 26 elementary schools. 

� Teachers responded to survey items using a Likert response scale calling for selection of one of five categories: strongly disagree, disagree, neither disagree nor agree, agree, or strongly agree. For the majority of these items, “strongly disagree” and “disagree” responses were summed to estimate negative perceptions, and “strongly agree” and “agree” responses were summed to estimate positive perceptions. For items with reverse coding, “strongly disagree” and “disagree” responses were summed to estimate positive perceptions, and “strongly agree” and “agree” responses were summed to estimate negative perceptions.

� The internal reliability of a scale is a measure of how consistently the questions in that scale refer to the same underlying idea and is expressed as a number ranging from 0 (no consistency among questions) to 1 (perfect consistency or essentially identical questions). Scales with higher reliability are able to measure finer-grained distinctions in levels of the underlying idea then are scales with lower reliability. For items such as “I have trouble finding information,” disagreeing with the statement would indicate positive attitude indicated with (R). The percentage of teachers indicating a positive attitude is shown to the right of the vertical center line in Exhibit 4-8.

� A correlation of this size means that 16 percent of the variance in frequency of data use is explained by the level of support for data-driven decision making. The size of the correlation is attenuated by the reliability of the two measures.

�  The schools in the 12 case study districts use over three dozen data systems, some of which were district developed and some of which came from commercial sources. System names have been deleted from staff criticisms of systems because the veracity of staff critiques were not tested independently by research staff.

�  The 2007–08 case study sample includes a small district for which an exception was made to the requirement of having three schools from the same district (the technical work group for the study felt it was important to research the experiences of small districts because they serve approximately a third of public school students; Hoffman 2007). A third school from a neighboring district in which the sampled district was providing data-driven decision-making assistance was included. Additional information on the sample selection process for the case study districts and schools can be found in Appendix A.

�  It is quite possible that the teachers whom principals described as typical in terms of data use were in fact better than average for the school. When this potential bias is considered along with the fact that the study conducted case studies in districts considered leaders in the instructional use of data, the reader should be aware that the teachers’ understanding of data described herein is likely to be better than that of average teachers.

� Districts were sampled from among the 12,483 districts that received federal Enhancing Education Through Technology (EETT) funds in 2003, as well as an additional 2,239 districts that had not received EETT funds.