Education's Data Management Initiative: Significant Progress Made, but Better Planning Needed to Accomplish Project Goals
Published: 2005-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As a condition of receiving federal funding for elementary and secondary education programs, states each year provide vast amounts of data to Education. While the need for information that informs evaluation is important (particularly with the No Child Left Behind Act), Education's data gathering has heretofore presented some problems. It has been burdensome to states because there are multiple and redundant requests administered by a number of offices. In addition, the resulting data supplied by states has not been accurate, timely, or conducive to assessing program performance. To improve the information by which it evaluates such programs and also to ease states' reporting burden, Education in 2002 initiated an ambitious, multiyear plan to consolidate elementary and secondary data collections into a single, department-wide system focused on performance. Given its importance, we prepared a study, under the authority of the Comptroller General, to provide Congress with information on its progress.
section_title: What GAO Found
paragraphs: Through its Performance-Based Data Management Initiative (PBDMI), Education has consolidated and defined much of the data it anticipates collecting under a unified system. Education reports that many data definitions have been agreed-to and data redundancies eliminated. PBDMI officials also said that to date, however, it has not been able to resolve all remaining differences among the program offices that manage many of the different data collections. PBDMI officials have conducted extensive outreach to the states to advance the initiative. The outreach to states involved regional conferences, two rounds of site visits, and according to officials, $100,000 in grants to most states to help offset their costs. State data providers responding to our survey expressed general satisfaction with the department's outreach, but some were not optimistic that the initiative would ease their reporting burden or enhance their own analytic capacity. The states were not able to produce enough data during test submissions in 2003 and 2004 to enable data quality verification or phasing out the department's multiple data collections. With regard to the lack of sufficient data from many states, Education officials said some lack the technical capacity needed to produce new performance data requirements. State data providers reported having competing demands for their time and resources, given other federal initiatives. Education officials have decided to proceed with the undertaking and have developed a draft interim strategy for moving forward. But they currently have no formal plan for how they would overcome obstacles such as the lack of state data and other technical and training delays to the initiative.

section_title: Letter
paragraphs: Each year, state education agencies provide vast amounts of information  to the U.S. Department of Education (Education) in order to fulfill  reporting requirements for federal programs supporting elementary and  secondary education. While this information is important for managing  programs, it has been accompanied by some problems. Reporting has been  burdensome for the state data providers because the department makes its  data requests through multiple, ongoing, and uncoordinated data  collections. By Education’s own account, there are currently 200 active  data collections for elementary and secondary programs—each resulting  in approximately 10,000 “person hours” for design, administration,  collection, and reporting. From the vantage point of the department and its  program offices, the information it receives has customarily been  compromised because the schools, districts, and states reporting data  employ their own definitions and, in some cases, report data that is  inaccurate, incomplete, and not timely. Finally, in terms of program  evaluation, much of the data that Education has traditionally requested  has not necessarily focused on program performance. Yet the need for  evaluative data has grown, particularly with passage of laws such as the  No Child Left Behind Act of 2001, which requires states receiving  assistance under the act to report on, among other things, the achievement  of their students on academic assessments required under that law.
To address these problems and better evaluate its programs, Education in  2002 began an initiative to consolidate and improve the information it  requests from states on elementary and secondary education and to seek  more consistency and quality in the data states supply. The Performance  Based Data Management Initiative (PBDMI) is a large-scale effort within  the department to combine more than a dozen separate data collections  into a single collection system, and better focus the information Education  requests from states by eliminating duplication, conflicting definitions, and  information that is not useful for the evaluation of its programs. The  PBDMI represents an important step forward for Education in its efforts to  monitor the performance of the nation’s elementary and secondary  schools. The initiative is also a large-scale undertaking for state education  agencies, which are volunteering to help develop uniform data and test the  new data collection system while they continue to meet their ongoing  reporting requirements. The PBDMI was scheduled to begin phasing out  the old data collections by September 2005, following final testing of the  new system and training of department staff.
In view of its importance and the inherent challenges, therefore, we have  prepared a study under the authority of the Comptroller General to  provide Congress with information about Education’s progress with the  PBDMI. We have examined Education’s work to (1) define what  performance-related data it will collect from states on behalf of the  program offices, (2) assist states in their efforts to submit quality  information, and (3) utilize performance-related data to provide enhanced  analytic capacity within the program offices.
To address our objectives we reviewed relevant documents, including  Education’s business plans, information collected by Education on states’  capacity to supply data, various contracts for key pieces of the initiative,  Education’s submissions to the Office of Management and Budget (OMB)  justifying the various data collections, the department’s concept of  operations, and other information related to the development of PBDMI.  We also interviewed Education officials overseeing PBDMI, officials from  most of the participating program offices, and key stakeholders in PBDMI,  including a standards-setting organization, an advocacy group, and  contractors, to obtain their perspectives on the progress of the initiative  and to verify the information we reviewed. Finally, we surveyed 52 state  data coordinators, including the District of Columbia and Puerto Rico,  about their experiences with PBDMI, and we received 50 responses. We  performed this work between April 2004 and September 2005 in  accordance with generally accepted government auditing standards. See  appendix I for additional information on our scope and methodology.
subsections: section_title: Background
paragraphs: The Department of Education annually administers data collections to  gather information from states about elementary and secondary education  programs receiving federal assistance. When it administers a data  collection, Education, like most federal agencies, is required to follow the  provisions of the Paperwork Reduction Act (PRA) in order to maximize  the utility of information to the federal agency and minimize the level of  burden incurred by the states and agencies from whom it solicits the  information. Traditionally, the department’s program offices, which have  responsibility for the administration and oversight of federal education  programs, have developed and operated similar data collections  independent of one another, in a continuous year-round process. In  addition, much of the data requested from states has been focused on  compliance and procedural matters, and overlooked performance and the  impact of programs in the classroom. Moreover, the collection of this data  has been complex and prone to error, given that it typically passes from  about 94,000 public schools to more than 14,000 school districts and then  to state education agencies before Education receives it.
Collecting data can be both time-intensive and costly. Education estimated  that in 2004, for example, that states spent approximately 45,000 hours and  nearly $1.2 million responding to the department’s requests for certain  elementary and secondary education data. (See fig. 1.) Data collections are  costly for Education also. Over $5 million was spent in 2004 administering  certain data collections that included allocating federal funds for both the  staff to administer the collections and in many instances for contractors to  analyze these data.
Initiated in 2002, the Education’s PBDMI has four goals: to improve the  quality of the data Education collects about elementary and secondary  education in terms of accuracy, consistency, and timeliness; to reduce the  burden that states incur in reporting data to the department; to improve  the focus of data analysis on program performance; and to improve  Education’s data-sharing relationship with the states. While this initiative  is not the department’s first attempt to overhaul the way it collects data, it  nonetheless represents a fundamental change to its data management in  that it is agencywide as opposed to program specific. As envisioned, the  new collection would consolidate 16 separate collections heretofore  conducted by seven program offices. Given the additional reporting effort  that development and testing of the system would require of states,  Education sought and received OMB approval to collect data from the  states through PBDMI. (See table 1 for a list of the separate collections  the PBDMI is designed to supplant.)
In addition to defining the information to be collected, the initiative  involves the development of a Web-based, data exchange network that will  provide states and others with the ability to submit school-based data into  one unified system to be stored in a data repository. The network will  comprise three separate, but interrelated systems—the first system, the  submission system, developed in late 2004, is used to collect data from  states, check data for quality, and store the data in the data repository. The  second system, the survey tool, which was also developed in 2004, enables  Education to collect supplemental data from states and others that is also  stored in the data repository. The third system, the data analysis and  reporting system, which is not yet operational, will allow users (i.e.,  program office staff and the public) to among other things, query the data  repository to analyze retrieved data and generate ad hoc reports.  Education envisions that states and school districts would be able to use  the data to assess their own program performance while also providing an  opportunity for them to verify the quality of data submitted through the  system. Figure 2 depicts the system design for the data network.
Education had originally planned to have all components of the data  exchange network fully operational in the spring of 2005 following the  completion of key activities, such as (1) defining the data to be collected  through in-depth consultations with department program offices and with  state data providers,(2) populating the database with school-based data  submitted by the states so that the quality of the stored data can be  checked, and (3) training program staff on how to use the new network.
PBDMI’s efforts to define what data were to be collected included forging  agreements among Education’s individual program offices about which  data would be essential to administration and oversight, particularly as  performance indicators, and also developing common definitions for those  elements that had been redundant. As a collaborative project, this involved  developing consensus and receiving feedback from many parties–program  offices, state policymakers and data providers, and organizations that  develop data standards in the field of Education. Within the department,  the office responsible for the day-to-day work of the project and for  ensuring its success is the Strategic Accountability Service, which also has  responsibility for developing and disseminating agencywide performance  indicators. However, a number of other offices and boards within the  department have been charged with providing oversight and guidance: a  steering committee convened to share information on the development of  the initiative consisting of the PBDMI managers and other senior officials  within the participating program offices, the Chief Information Officer  (CIO), a data information working group, and Education’s investment  review board. The data information working group, which is headed by  Education’s CIO, has responsibility for ensuring the consistency and  quality of new data collections and for facilitating the integration and  sharing of information between program offices. The department’s  investment review board has overall responsibility for reviewing and  approving and prioritizing department investments in technology,  including the new network. As voluntary participants, stakeholders such  as data coordinators from each of the 50 state education agencies, the  District of Columbia, and Puerto Rico were provided with opportunities to  give their input and feedback on the development of the initiative. The  Education Information Advisory Committee established by Council for  Chief State School Officers facilitates this exchange. Figure 3 depicts the  various groups involved in the initiative.
Once departmental data requirements were identified, Education planned  a series of data collections to be followed by extensive testing of the  quality of that data by the program offices. Specifically, Education planned  to have states submit the newly defined data for the 2002-2003 and 2003- 2004 school years. (States would voluntarily make these submissions to  PBDMI while also maintaining their current multiple reporting obligations  under Education’s program offices.) In conjunction with the program  offices, PBDMI officials then anticipated validating and verifying the  quality of the new data submitted using a number of checks and  evaluations. Also at this time the development of the system that staff  would use to analyze data and generate reports was to be finalized. Once  these activities were completed, the program offices were to assess  whether the new system would be an adequate substitute for their existing  data collections.
Education has projected that it would spend just over $30 million through  2005 and initial estimates indicate that the data network will cost— beginning in 2006—just over $4 million annually to maintain. See figure 4  for project time frames and projected costs through 2009.
section_title: Education Has Made Progress Defining Data to Be Collected under a Consolidated System, but Project Officials are Unable to Reconcile All Differences
paragraphs: Education officials spearheading PBDMI told us they have made progress  defining the data to be collected. To do this, project officials worked with  the program offices to identify their existing data needs. They also worked  with program offices to translate these needs into performance-related  data, such as math and reading achievement scores for different groups of  students. Officials told us they had eliminated data elements collected by  the program offices that are more indicative of process than performance.  PBDMI officials encouraged program offices to identify performance- related data by using requirements specified in laws such as the No Child  Left Behind Act and using the goals in the department’s strategic plans.
PBDMI officials also worked with the program offices to reach agreement  on common definitions for the data elements selected and to eliminate  redundancy. For example, some programs needed information on charter  schools, and PBDMI officials coordinated efforts within the department to  develop one standard definition for them. The end result of these efforts is  a unified body of data elements that includes definitions for each of the  data elements and identifies the program with primary stewardship over  decisions about that element. According to one department official  managing the initiative, this collection will improve the quality of the data  by assuring more consistency in what states provide.
Although PBDMI officials reported progress in identifying performance- related data and establishing common data definitions, project officials  have not fully documented these achievements by establishing a baseline  and thus cannot be certain of the full extent of the progress made toward  achieving their goal to enhance the department’s focus on outcomes and  accountability. For example, while PBDMI officials were able to provide a  list of 161 data elements focused on performance they were unable to  provide us with a comprehensive list of “process-oriented” elements that  had been eliminated. Similarly, while PBDMI managers reported that the  program offices had agreed to definitions for the bulk of the data  elements—one official estimated that they reached agreement for about 90  percent of the data—they could not provide us with a complete list of  redundant elements that had been eliminated or those that remain because  they had not tracked them.
While PBDMI officials could not provide a full list of disputed data  elements, they reported that some differences still remain among program  offices. Although PBDMI officials encouraged the use of strategic plans  and statutory requirements to justify the selection of performance-based  data, they told us that program offices had final say over what data to  collect. For example, one office uses similar although somewhat broader  criteria that allow it to collect “data that can be reliably obtained from  states or that Education has a documented need for.” Additionally,  according to initiative officials, some differences remain due to differences  in legislative requirements for the particular programs, while others  resulted from preferences of some offices to continue using the same  definitions as in the past.
Officials responsible for carrying out the PBDMI told us they were unable  to reconcile all differences. Officials told us they were working with the  program offices to reach agreements, but said the programs maintain  primary control for defining their data needs and would make final  decisions. Additionally, we were told by Education’s CIO, who is required  to review all data collections and who has a primary role within the Data  Information Working Group, that this office does not have a role in  resolving data disputes between program offices in order to ensure  uniformity. However, an official also said that any differences that could  not be resolved between the program offices would ultimately be  arbitrated at the assistant secretary level within Education.
section_title: PBDMI Officials Have Worked Extensively with States on Data Preparation and Submissions, but Most States Cannot Produce the Requested Data
paragraphs: PBDMI officials have conducted extensive outreach to the states to help  unify their data definitions and upgrade their collection and submission  systems. State data providers responding to our survey expressed general  satisfaction with the department’s outreach. However, the majority  thought that the burden of data collection and reporting would either  increase or remain the same with implementation of the PBDMI. In  addition, less than half expected the initiative to improve their ability to  conduct their own in-state analyses only somewhat. Despite the extensive  outreach, the states were not able to produce enough data during test  submissions in 2002-2003 and again in 2003-2004 for the department to  validate its quality and consider phasing out its standing collection  systems.
subsections: section_title: Education Has Conducted Extensive Outreach to States to Improve Data Quality
paragraphs: In order to ensure that states could meet Education’s requests for quality  data required as part of PBDMI, officials conducted extensive outreach to  state agencies, their data providers, and to data standards organizations.  After Education developed its body of data elements, it consulted in 2002  with a task force consisting of a small number of state data providers to  advise the department on the availability of the data it intended to collect.  The department then conducted site visits beginning in April 2003 to 50  states, the District of Columbia, and Puerto Rico to obtain feedback on the  ability of states to provide needed data and to prepare for testing the  states’ ability to submit data. Education officials said they also made  $50,000 grants to all 52 states to offset costs of overhauling information  systems or obtaining additional staff. At the culmination of these visits,  Education originally planned for states to transmit 2002-2003 school year  data that could be tested for quality.
However, Education scaled back the scope of this first data collection  after recognizing that states would not, as yet, be able to offer certain  types of data, such as data needed to meet requirements of the NCLBA.  Consequently, Education delayed its plans to assess the quality of the data  states submitted and focused instead on the ability of states to  electronically transmit as much PBDMI data as they could to the  department. Also, Education decided to remove from PBDMI’s prospective  collection some data elements that states reported were not available at  that time. Under this transmission pilot test, 50 states, including the  District of Columbia and Puerto Rico, were able to submit some data to  Education demonstrating that PBDMI was technically feasible.
After establishing this technical feasibility, Education began preparing in  2004 for its data collection of 2003-2004 school year by providing  additional outreach to the states. Project officials conducted a second  round of site visits beginning in April and provided further guidance to  help states align their data definitions with PBDMI standards. By aligning  definitions with PBDMI, Education attempted to minimize possible  confusion about what data to submit and when, further assisting the  department’s efforts to improve data quality. Department officials have  said that establishing a unified body of data elements across the  department and states—so that all involved parties use the same  “language” when analyzing and sharing data—is a priority. Education  officials attribute the lack of quality in the data it currently collects from  states and others to a variety of reasons, such as the lack of common data  definitions that developed over time in response to the specific  information needs of the program offices and data requirements arising at  the state level.
Officials with the initiative also conducted a limited number of quality  assessments of state information systems to identify better ways of  collecting and reporting data to the department. To serve states on a  broader scale, Education conducted regional meetings, providing them  with updates and feedback on the progress of the initiative. Officials also  established a call center to answer states’ questions about the data to be  submitted. Most states also received another $50,000 in grants for their  continued participation in the initiative. Education began collecting 2003- 2004 school year data in November 2004.
To increase the likelihood that its definitions would be adopted by states  and other data providers, PBDMI officials also collaborated with advocacy  groups that establish data and influence the development of technical  standards. For example, PBDMI officials contracted with the Council of  Chief State School Officers to coordinate PBDMI conferences, help states  prepare and submit data, and provide feedback as PBDMI developed data  definitions. Education also collaborated with the Schools Interoperability  Framework, a group that develops data-sharing standards and software  primarily designed for schools and districts. By working with the Schools  Framework, Education officials said they could improve data quality by  increasing the likelihood that departmental definitions and other  standards would be incorporated into software used by schools and  districts. This interaction with the Schools Framework is Education’s  primary attempt to deal with the long-standing problem of poor data  provided by schools and districts.  (See table 2 for a list of some of  Education’s outreach activities.)
section_title: Most States Expressed Satisfaction with Education’s Outreach, but Had Mixed Views on PBDMI’s Potential Benefits
paragraphs: States were generally satisfied with Education’s outreach activities. (See  table 3.) Most state data providers—72 percent—rated Education’s site  visits effective in improving the partnership with the states. One state data  provider characterized his exchanges with the department as open and  non-defensive, and further reported that the department had been  responsive. More than half rated as effective or very effective Education’s  technical assistance (57 percent) and regional meetings (52 percent).
While most states thought Education’s activities to improve its partnership  with states were effective, some suggested areas for improvements. For  example, 72 percent thought the site visits provided only some or little  information on successes achieved in other states.
In their survey responses half of the states expressed the view that  reducing their reporting burden was the most important PBDMI goal;  however, fewer than a third of the states said they believe the initiative  will do so. (See table 4.) Some states emphasized their burden had  increased in the short term as they continued dual reporting in order to  meet the still ongoing data collection requirements of the program offices.  Three states reported to us their cost estimates of systems development  projects needed to support PBDMI, which ranged from approximately  $120,000 to as much as $5 million. Moreover, about 75 percent of the states  reported that they thought the burden to collect data would remain the  same or increase once PBDMI was implemented. Some state respondents  expressed the opinion that until there is a firm commitment by Education  to halt multiple data collections their reporting burden would not likely  lessen. “We are asked from the federal government for more and more  information….  opens the flood gate for more and more reporting,”  noted one official, adding that it is currently “hard to see the benefit at this  time.”
Some states also had reservations about the benefits of PBDMI for  evaluation. One respondent cautioned, for example, that support within  his state had weakened because of the lack of perceived benefits. Only  about 20 percent of states expected PBDMI to improve or greatly improve  their analytic capacity—that is the ability to meet their own state reporting  requirements, analyze program effectiveness, analyze student outcomes,  and to compare outcomes within states. Their reasons varied. For  example, five states reported that they would continue to use their own  systems. A few elaborated that their own information systems allow more  detailed analyses of state performance than the information to be  collected through PBDMI. Additionally, an almost equal number of states  saw PBDMI as an effective tool to inform stakeholders as not. Table 5 lists  the extent to which state data providers expect PBDMI to enhance their  analytical capacity in a variety of areas.
section_title: Many States Are Not Prepared to Meet Education’s Data Requests
paragraphs: As of June 3, 2005, only 9 states had submitted more than half of the  requested 2003-2004 school year data, while 29 states had submitted less  than 20 percent (see fig. 5). Although PBDMI officials said they will wait  until August 2005 for states to submit the 2003-2004 data, they also  acknowledged that many states would not be able to provide significant  portions. The lack of state data is particularly acute in some programmatic  areas. For example, many states have been unable to provide data on  homeless and migrant students or students with limited English  proficiency. States told Education officials early in the process that  changes to state data collection processes, systems, and definitions would  be needed to provide these types of information.
We found that there were various reasons why states could not provide  data. Some states reported that they wanted better documentation from  the department in areas such as clarifying established data definitions and  file format specifications needed to transmit data. States needed to make  major modifications to their existing data collection and reporting  processes in order to provide new information required by PBDMI. States  also reported that they would not provide certain data elements that were  inapplicable, hard to collect, or available elsewhere. Some also reported  that there was still some confusion over multiple or unclear definitions.  Department officials said that many states had initially overestimated their  capabilities and that the data states said would be available differed  greatly from what they have produced thus far. States have also noted  competing demands for their time and resources stemming from NCLBA.  Some states reported they lacked resources, such as staff and money, to  implement changes specific to the initiative. Specifically, 56 percent of the  state survey respondents said that all or a portion of the $50,000 in grants  they received from Education were used to contract for additional  personnel, a quarter of the states said that these funds were used to  improve their information systems. Some states noted, however, that these  funds were insufficient to make changes necessary for their participation  in PBDMI.
Recognizing that obtaining state data has been problematic, Education has  recently developed a preliminary strategy for working more closely with  states to ensure that it obtains 100 percent of data from all the states.  While not finalized, Education is currently considering actions such as  issuing regulations requiring states to submit PBDMI data and allowing  those states that provide acceptable amounts of “high quality” data under  PBDMI to be exempt from existing data collections. For example, states  that submit data to PBDMI that are also currently collected through the  Consolidated State Report—one of many data collections required under  the NCLBA—would not have to submit the same data under this data  collection. Officials have also tentatively proposed collecting data of lesser  quality that are readily available and obtaining data through other systems  to supplement what has been provided thus far. It is not clear the extent to  which this proposal would undermine efforts to improve data quality and  maintain program office buy-in. Another option under consideration at  Education is to target departmental resources, such as $25 million in  grants for system improvements from the Institute of Education Sciences,  at states that actively participate in PBDMI.
section_title: Education Is Proceeding with Implementation despite the Data Shortage and without a Detailed Plan of Action
paragraphs: Education is proceeding with efforts toward full implementation of  PBDMI—using the data for analysis and reporting—despite the limited  amount of data collected. To do so, program offices decide whether the  quality of the data (in terms of accuracy, consistency, timeliness, and  utility) collected through PBDMI meets their needs. Once program offices  validate the quality of the data, Education would begin to phase out  existing data collections. Additionally staff will be trained on how to  access and use the data collected to date. Originally Education expected  to complete all of these activities by the spring of 2004. To the degree that  it has been able to proceed, the department has developed a set of quality  checks designed to ensure the accuracy and completeness of the data  states submit.
Nevertheless, two program offices, which as members of the seven  principle offices included in the initiative and have a role in determining  whether the data are accurate and complete for their purposes, expressed  concern that PBDMI’s procedures to ensure data quality may not be  adequate. An official in the Office of Special Education and Rehabilitative  Services (OSERS), which has collected almost 30 years of longitudinal  data about the effectiveness of the nation’s special education programs,  told us that PBDMI had been provided with information about the nearly  200 data quality checks used in special education collections, but was not  sure that PBDMI adopted them all. PBDMI officials said they adopted  those that were universally relevant. Further, this official expressed  concern that PBDMI would not meet its special needs. Specifically, unlike  other program offices, OSERS programs bases student assessment on age  as opposed to grade level attained. Additionally, this official was  concerned about the timeliness of the data collected through PBDMI  because that office generated a number of congressionally mandated  reports at specific times of the year. Consequently, this office plans to  compare the quality of its own data with the data collected through  PBDMI. Officials in the Office for Civil Rights also expressed similar  reservations with PBDMI’s administration of its large elementary and  secondary survey of schools and districts used to assess compliance with  civil rights laws and identify trends. Historically, district superintendents  have responded to this survey in large enough numbers allowing  Education to generalize on any findings with a high degree of confidence.  However, when PBDMI administered the survey, fewer superintendents  responded and, according to the Office for Civil Rights, PBDMI did not  have a readily available plan that adequately outlined steps needed to raise  the response rate. As of June 10, 2005, the response rate for this survey  was lower than previous surveys.
Final implementation has also been hampered by delays in training and  delivery of the analysis and reporting system. Both are more than a year  behind schedule. An official responsible for overseeing the training efforts  told us they could not focus on the delays because considerable time was  spent addressing state problems submitting data through PBDMI. The data  analysis and reporting system is more than a year behind scheduled due to  the lack of data and the failure of Education’s contractor to meet its  scheduled delivery of the system. Education officials now expect to fully  implement the system by March 31, 2006. In lieu of developing its data  analysis and reporting system and training, PBDMI has offered  presentations of these tools as a preview for staff to see the new system’s  capabilities and to keep them apprised of the initiative’s progress.
Despite the many obstacles confronting the PBDMI, Education officials  said they expect to proceed with implementation of the initiative, albeit  with some activities postponed. In August, project officials developed a  preliminary strategy designed to address the problem of collecting data  from the states, such as providing exemptions from certain reporting  requirements for some states. However, this strategy has not been  finalized, and Education has not developed a specific plan of action for  how they might (1) help states that are deficient, (2) deal with state  expectations for phasing out the multiple data collections, or (3) meet the  expectations of their own program offices.
section_title: Conclusions
paragraphs: The PBDMI represents an important step forward for the Department of  Education in its efforts to monitor the performance of the nation’s  elementary and secondary schools. By developing the ability to collect  data that are more accurate, timely, consistent, and focused on key  national performance indicators, Education will be much better informed  to make its many policy and programmatic decisions. The initiative, by  asking for a clearly defined set of information that is to be submitted only  one time, has the potential to substantially reduce state reporting burden  for elementary and secondary programs as well as to help states to  develop better data systems. However, PBDMI is an ambitious and risky  undertaking that requires the continued cooperation of a number of  internal and external stakeholders.
In order for PBDMI to be successful, the department must rely on states to  provide new information at a time when they are busy implementing large  new federal initiatives, such as the No Child Left Behind Act. While some  states have been able to provide significant amounts of data, others  continue to lag far behind. In order for PBDMI to be successful, it is  important for all states to submit timely, reliable, accurate, and consistent  data. Consequently, it is important for the department to have a clear plan  for addressing states with problems providing data and to continue to  provide a proper combination of support and incentives for states to  participate. By having worked closely with the states on their collection  systems, PBDMI officials have the information they would need to develop  a plan of action to help move them forward.
Because PBDMI represents a significant change in the way the Department  of Education conducts business, it can only be accompanied effectively  and efficiently by a change in management practices. However, program  offices still retain much discretion over what data they will collect, how  they will define it, and whether or not PBDMI’s data will meet their needs.  While it is the initiative’s responsibility to make sure it collects data that  meets the program offices’ requirements, PBDMI is also responsible for  developing a data collection system focused on program performance and  quality data. To the extent that programmatic differences, such as those  over data definitions, inhibit PBDMI’s goals there should be a clear  process for reconciling those differences. If PBDMI truly represents a new  way of doing business, Education should be able to ensure that its  organizational units go along. It is difficult to see PBDMI achieving its full  potential without a clear process for furthering the initiative’s goals.
Fundamental to any large, complex effort’s success is a well thought out  plan that tracks its progress against a set of clearly defined and  measurable goals. PBDMI has not put in place such a planning and  tracking system. State governments and Education’s program offices have  devoted much time, effort, and money participating in PBDMI with the  idea that they would see benefits as a result. A lack of demonstrated  progress and benefits potentially erodes state support, undermining the  viability of this important initiative. Some states are already beginning to  lose sight of the potential benefits of PBDMI. As the department goes past  its original completion deadline, it is important for it to lay out a clear plan  for how it will now proceed.
section_title: Recommendations for Executive Action
paragraphs: To address the issues we have identified with regard to planning, decision- making, and improving data quality, we recommend that the Secretary of  Education develop    a strategy to help states improve their ability to provide quality data  given the challenges that many states face in providing data;    a clear process for reconciling differences between the program offices  and the PBDMI oversight office to ensure that decisions critical to the  success of PBDMI are made; and    a clear plan for completing final aspects of PBDMI, including specific  time frames and indicators of progress toward the initiative’s goals.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the  Department of Education. Education agreed with our findings and  recommendations and stated that it has devoted additional resources to  the initiative and plan to issue a detailed project plan that outlines the  steps needed to complete the initiative. These comments are reprinted in  appendix II.
Education also provided technical corrections and comments that we  incorporated where appropriate.
We are sending copies of this report to the Secretary of Education, the  Office of Strategic Accountability Services, the Director of the Office of  Management and Budget, and appropriate congressional committees.  Copies will also be made available to other interested parties upon  request. Additional copies can be obtained at no cost from our Web site at  www.gao.gov.
If you or your staff should have any questions, please call me at 415-904- 2272 or bellisd@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.
GAO staff who made major contributions to this report are listed in  appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: The objective of our review of the Performance Based Data Management  Initiative (PBDMI) was to assess the progress Education has made in its  implementation of the initiative, particularly with regard to (1) defining  what performance-related data it will collect from states on behalf of the  program offices, (2) assisting states in their efforts to submit quality  information, and (3) utilizing performance-related data to provide  enhanced analytic capacity within the program offices. We conducted our  review between April 2004 and September 2005 in accordance with  generally accepted government auditing standards.
subsections: section_title: Overall Approach
paragraphs: To assess the department’s progress in each of these areas, we reviewed  documents relating to the implementation of the initiative, relevant laws,  and information provided by the office responsible for PBDMI—the  Strategic Accountability Service (SAS) and others. We interviewed key  staff responsible for the initiative as well as officials in each of the offices  that are participating in PBDMI. We also interviewed senior-level  Education officials to determine their role in the implementation of  PBDMI. To gain insight into state perspectives on the initiative, we  administered a Web-based survey to state officials responsible for  providing these data to Education. We received responses from 50 states  including Puerto Rico. We also interviewed a variety of external  stakeholders, a data standards organization, and three contractors  involved in the initiative, including an official from the Council of Chief  State School Officers. We also reviewed previously issued reports by  Education’s Office of the Inspector General (IG) as well as GAO reports  and testimonies.
In addition to interviewing departmental officials, we also reviewed  documentation on the initiative to gain a better understanding of what  actions Education was undertaking to implement the goals of the  initiative, including its data quality contract, data dictionary, its business  plans as well as justification reports to the Office of Management and  Budget (OMB) required under the Paperwork Reduction Act to collect  data. We also reviewed summary information about state performance  data that was obtained as a result of site visits to states conducted in 2004  in order to analyze what data was obtained from states as a result of their  efforts.
Education provided information on states’ submission of requested data  elements to PBDMI as of June 3, 2005. States were expected to provide  data for 64 data elements ranging from dropout rates, student performance  on reading, science, and writing assessments, teacher certification, and  many others. For each of these elements, Education determined whether  each state had submitted the information, had not submitted the data, or  did not collect the information. We incorporated into our report  Education’s calculated percentages of elements submitted for each state.  We determined that these data were sufficient for the purposes of this  engagement.
In order to document the burden hours associated with certain elementary  and secondary data collections, we accessed 14 data collection  justifications authored by each of the department’s program offices and  submitted to the chief information officer. These reports had received  OMB approval or were seeking approval to collect data from states and  others. We talked with an official responsible for maintaining these  documents at the department’s Web site to verify that these were the most  recent data available for analysis. From each document we obtained the  estimated state burden hours and costs and federal administrative costs  associated with each data collection. Each estimate was based on a  formula that we adjusted to reflect these costs for the 52 states  participating in the initiative. In some instances where an average was  used, we assumed that the 52 states were similar in characteristics to the  overall population of states included in Education’s estimates. However,  we did not find it feasible to prorate the formulas for the federal  administrative costs (based on 52 states) for each of the collections. A  statistician verified each of the calculated estimates for accuracy.
We also surveyed all 52 state data coordinators using a Web-based survey  instrument in order to obtain their perspectives on various aspects of the  initiative. Our survey instrument was developed based on information  obtained during interviews with state data coordinators in Pennsylvania,  Virginia, Washington, and Oregon. Additionally, other internal  stakeholders specializing in technology and education were asked to  review and comment on our draft survey instrument. The survey was pre- tested with Wyoming, North Carolina, and Illinois to determine if the  questions were clear and unbiased and whether the terms were accurate  and precise. We included these three states in our pretests because they  varied in size and technical capacity for data transmission as determined  by an earlier Education survey. Based on their comments, we refined the  questionnaire as appropriate.
Our final survey instrument asked a combination of questions that allowed  for closed-ended as well as open-ended responses and included questions  about state perspectives on PBDMI’s ability to achieve its goals. The  survey was conducted using self-administered electronic questionnaire  posted on the Internet. We sent e-mail notifications about the upcoming  survey to all 52 state data coordinators (50 states, the District of Columbia,  and Puerto Rico) on November 15, 2004, and activated the survey shortly  thereafter. Each potential respondent was provided a unique password  and username by e-mail to limit participation to members of the target  population. To encourage respondents to complete the questionnaire, we  sent an e-mail message to prompt each non-respondent approximately 2  weeks after the survey was activated and followed up by e-mail or phone  with each non-respondent several times thereafter. We closed the survey  on January 21, 2005, after the 50th respondent had replied.
Because this was not a sample survey, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as non-sampling errors. For example,  difficulties in how a particular question is interpreted, in the sources of  information that are available to respondents, or in how the data are  entered into a database or were analyzed can introduce unwanted  variability into the survey results. We took steps in the development of the  survey instrument, the data collection, and the data analysis to minimize  these non-sampling errors. For example, a survey specialist designed the  survey instrument in collaboration with GAO staff with subject matter  expertise. Then, as stated earlier, it was pre-tested to ensure that the  questions were clear, unbiased, and accurate. When the data were  analyzed, a second, independent analyst checked all computer programs.  Because this was a Web-based survey, respondents entered their answers  directly into the electronic questionnaire, eliminating the need to have the  data keyed into a database, thus removing an additional source of error.
section_title: Appendix II: Comments from the Department of Education
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above the following individuals made  important contributions to this report: Bryon Gordon, Assistant Director;  Carla Craddock, Analyst-in-Charge; Susan Bernstein; David Dornisch;  Mary Dorsey; Kimberly Gianopoulos; Brandon Haller; Stuart Kaufman;  Jonathan McMurray; Valerie Melvin; James Rebbe; Gloria Hernandez  Saunders; Kimberly Siegel; Michelle Verbrugge; and Elias Walsh.