Education's Data Management Initiative: Significant Progress Made, but Better Planning Needed to Accomplish Project Goals
Released: 2005-10-28T00:00:00

Letter

Background

The Department of Education annually administers data collections to gather information from states about elementary and secondary education programs receiving federal assistance. When it administers a data collection, Education, like most federal agencies, is required to follow the provisions of the Paperwork Reduction Act (PRA) in order to maximize the utility of information to the federal agency and minimize the level of burden incurred by the states and agencies from whom it solicits the information. Traditionally, the department’s program offices, which have responsibility for the administration and oversight of federal education programs, have developed and operated similar data collections independent of one another, in a continuous year-round process. In addition, much of the data requested from states has been focused on compliance and procedural matters, and overlooked performance and the impact of programs in the classroom. Moreover, the collection of this data has been complex and prone to error, given that it typically passes from about 94,000 public schools to more than 14,000 school districts and then to state education agencies before Education receives it.

Collecting data can be both time-intensive and costly. Education estimated that in 2004, for example, that states spent approximately 45,000 hours and nearly $1.2 million responding to the department’s requests for certain elementary and secondary education data. (See fig. 1.) Data collections are costly for Education also. Over $5 million was spent in 2004 administering certain data collections that included allocating federal funds for both the staff to administer the collections and in many instances for contractors to analyze these data.

Initiated in 2002, the Education’s PBDMI has four goals: to improve the quality of the data Education collects about elementary and secondary education in terms of accuracy, consistency, and timeliness; to reduce the burden that states incur in reporting data to the department; to improve the focus of data analysis on program performance; and to improve Education’s data-sharing relationship with the states. While this initiative is not the department’s first attempt to overhaul the way it collects data, it nonetheless represents a fundamental change to its data management in that it is agencywide as opposed to program specific. As envisioned, the new collection would consolidate 16 separate collections heretofore conducted by seven program offices. Given the additional reporting effort that development and testing of the system would require of states, Education sought and received OMB approval to collect data from the states through PBDMI. (See table 1 for a list of the separate collections the PBDMI is designed to supplant.)

In addition to defining the information to be collected, the initiative involves the development of a Web-based, data exchange network that will provide states and others with the ability to submit school-based data into one unified system to be stored in a data repository. The network will comprise three separate, but interrelated systems—the first system, the submission system, developed in late 2004, is used to collect data from states, check data for quality, and store the data in the data repository. The second system, the survey tool, which was also developed in 2004, enables Education to collect supplemental data from states and others that is also stored in the data repository. The third system, the data analysis and reporting system, which is not yet operational, will allow users (i.e., program office staff and the public) to among other things, query the data repository to analyze retrieved data and generate ad hoc reports. Education envisions that states and school districts would be able to use the data to assess their own program performance while also providing an opportunity for them to verify the quality of data submitted through the system. Figure 2 depicts the system design for the data network.

Education had originally planned to have all components of the data exchange network fully operational in the spring of 2005 following the completion of key activities, such as (1) defining the data to be collected through in-depth consultations with department program offices and with state data providers,(2) populating the database with school-based data submitted by the states so that the quality of the stored data can be checked, and (3) training program staff on how to use the new network.

PBDMI’s efforts to define what data were to be collected included forging agreements among Education’s individual program offices about which data would be essential to administration and oversight, particularly as performance indicators, and also developing common definitions for those elements that had been redundant. As a collaborative project, this involved developing consensus and receiving feedback from many parties–program offices, state policymakers and data providers, and organizations that develop data standards in the field of Education. Within the department, the office responsible for the day-to-day work of the project and for ensuring its success is the Strategic Accountability Service, which also has responsibility for developing and disseminating agencywide performance indicators. However, a number of other offices and boards within the department have been charged with providing oversight and guidance: a steering committee convened to share information on the development of the initiative consisting of the PBDMI managers and other senior officials within the participating program offices, the Chief Information Officer (CIO), a data information working group, and Education’s investment review board. The data information working group, which is headed by Education’s CIO, has responsibility for ensuring the consistency and quality of new data collections and for facilitating the integration and sharing of information between program offices. The department’s investment review board has overall responsibility for reviewing and approving and prioritizing department investments in technology, including the new network. As voluntary participants, stakeholders such as data coordinators from each of the 50 state education agencies, the District of Columbia, and Puerto Rico were provided with opportunities to give their input and feedback on the development of the initiative. The Education Information Advisory Committee established by Council for Chief State School Officers facilitates this exchange. Figure 3 depicts the various groups involved in the initiative.

Once departmental data requirements were identified, Education planned a series of data collections to be followed by extensive testing of the quality of that data by the program offices. Specifically, Education planned to have states submit the newly defined data for the 2002-2003 and 2003- 2004 school years. (States would voluntarily make these submissions to PBDMI while also maintaining their current multiple reporting obligations under Education’s program offices.) In conjunction with the program offices, PBDMI officials then anticipated validating and verifying the quality of the new data submitted using a number of checks and evaluations. Also at this time the development of the system that staff would use to analyze data and generate reports was to be finalized. Once these activities were completed, the program offices were to assess whether the new system would be an adequate substitute for their existing data collections.

Education has projected that it would spend just over $30 million through 2005 and initial estimates indicate that the data network will cost— beginning in 2006—just over $4 million annually to maintain. See figure 4 for project time frames and projected costs through 2009.

Education Has Made Progress Defining Data to Be Collected under a Consolidated System, but Project Officials are Unable to Reconcile All Differences

Education officials spearheading PBDMI told us they have made progress defining the data to be collected. To do this, project officials worked with the program offices to identify their existing data needs. They also worked with program offices to translate these needs into performance-related data, such as math and reading achievement scores for different groups of students. Officials told us they had eliminated data elements collected by the program offices that are more indicative of process than performance. PBDMI officials encouraged program offices to identify performance- related data by using requirements specified in laws such as the No Child Left Behind Act and using the goals in the department’s strategic plans.

PBDMI officials also worked with the program offices to reach agreement on common definitions for the data elements selected and to eliminate redundancy. For example, some programs needed information on charter schools, and PBDMI officials coordinated efforts within the department to develop one standard definition for them. The end result of these efforts is a unified body of data elements that includes definitions for each of the data elements and identifies the program with primary stewardship over decisions about that element. According to one department official managing the initiative, this collection will improve the quality of the data by assuring more consistency in what states provide.

Although PBDMI officials reported progress in identifying performance- related data and establishing common data definitions, project officials have not fully documented these achievements by establishing a baseline and thus cannot be certain of the full extent of the progress made toward achieving their goal to enhance the department’s focus on outcomes and accountability. For example, while PBDMI officials were able to provide a list of 161 data elements focused on performance they were unable to provide us with a comprehensive list of “process-oriented” elements that had been eliminated. Similarly, while PBDMI managers reported that the program offices had agreed to definitions for the bulk of the data elements—one official estimated that they reached agreement for about 90 percent of the data—they could not provide us with a complete list of redundant elements that had been eliminated or those that remain because they had not tracked them.

While PBDMI officials could not provide a full list of disputed data elements, they reported that some differences still remain among program offices. Although PBDMI officials encouraged the use of strategic plans and statutory requirements to justify the selection of performance-based data, they told us that program offices had final say over what data to collect. For example, one office uses similar although somewhat broader criteria that allow it to collect “data that can be reliably obtained from states or that Education has a documented need for.” Additionally, according to initiative officials, some differences remain due to differences in legislative requirements for the particular programs, while others resulted from preferences of some offices to continue using the same definitions as in the past.

Officials responsible for carrying out the PBDMI told us they were unable to reconcile all differences. Officials told us they were working with the program offices to reach agreements, but said the programs maintain primary control for defining their data needs and would make final decisions. Additionally, we were told by Education’s CIO, who is required to review all data collections and who has a primary role within the Data Information Working Group, that this office does not have a role in resolving data disputes between program offices in order to ensure uniformity. However, an official also said that any differences that could not be resolved between the program offices would ultimately be arbitrated at the assistant secretary level within Education.

PBDMI Officials Have Worked Extensively with States on Data Preparation and Submissions, but Most States Cannot Produce the Requested Data

PBDMI officials have conducted extensive outreach to the states to help unify their data definitions and upgrade their collection and submission systems. State data providers responding to our survey expressed general satisfaction with the department’s outreach. However, the majority thought that the burden of data collection and reporting would either increase or remain the same with implementation of the PBDMI. In addition, less than half expected the initiative to improve their ability to conduct their own in-state analyses only somewhat. Despite the extensive outreach, the states were not able to produce enough data during test submissions in 2002-2003 and again in 2003-2004 for the department to validate its quality and consider phasing out its standing collection systems.

Education Has Conducted Extensive Outreach to States to Improve Data Quality

In order to ensure that states could meet Education’s requests for quality data required as part of PBDMI, officials conducted extensive outreach to state agencies, their data providers, and to data standards organizations. After Education developed its body of data elements, it consulted in 2002 with a task force consisting of a small number of state data providers to advise the department on the availability of the data it intended to collect. The department then conducted site visits beginning in April 2003 to 50 states, the District of Columbia, and Puerto Rico to obtain feedback on the ability of states to provide needed data and to prepare for testing the states’ ability to submit data. Education officials said they also made $50,000 grants to all 52 states to offset costs of overhauling information systems or obtaining additional staff. At the culmination of these visits, Education originally planned for states to transmit 2002-2003 school year data that could be tested for quality.

However, Education scaled back the scope of this first data collection after recognizing that states would not, as yet, be able to offer certain types of data, such as data needed to meet requirements of the NCLBA. Consequently, Education delayed its plans to assess the quality of the data states submitted and focused instead on the ability of states to electronically transmit as much PBDMI data as they could to the department. Also, Education decided to remove from PBDMI’s prospective collection some data elements that states reported were not available at that time. Under this transmission pilot test, 50 states, including the District of Columbia and Puerto Rico, were able to submit some data to Education demonstrating that PBDMI was technically feasible.

After establishing this technical feasibility, Education began preparing in 2004 for its data collection of 2003-2004 school year by providing additional outreach to the states. Project officials conducted a second round of site visits beginning in April and provided further guidance to help states align their data definitions with PBDMI standards. By aligning definitions with PBDMI, Education attempted to minimize possible confusion about what data to submit and when, further assisting the department’s efforts to improve data quality. Department officials have said that establishing a unified body of data elements across the department and states—so that all involved parties use the same “language” when analyzing and sharing data—is a priority. Education officials attribute the lack of quality in the data it currently collects from states and others to a variety of reasons, such as the lack of common data definitions that developed over time in response to the specific information needs of the program offices and data requirements arising at the state level.

Officials with the initiative also conducted a limited number of quality assessments of state information systems to identify better ways of collecting and reporting data to the department. To serve states on a broader scale, Education conducted regional meetings, providing them with updates and feedback on the progress of the initiative. Officials also established a call center to answer states’ questions about the data to be submitted. Most states also received another $50,000 in grants for their continued participation in the initiative. Education began collecting 2003- 2004 school year data in November 2004.

To increase the likelihood that its definitions would be adopted by states and other data providers, PBDMI officials also collaborated with advocacy groups that establish data and influence the development of technical standards. For example, PBDMI officials contracted with the Council of Chief State School Officers to coordinate PBDMI conferences, help states prepare and submit data, and provide feedback as PBDMI developed data definitions. Education also collaborated with the Schools Interoperability Framework, a group that develops data-sharing standards and software primarily designed for schools and districts. By working with the Schools Framework, Education officials said they could improve data quality by increasing the likelihood that departmental definitions and other standards would be incorporated into software used by schools and districts. This interaction with the Schools Framework is Education’s primary attempt to deal with the long-standing problem of poor data provided by schools and districts. (See table 2 for a list of some of Education’s outreach activities.)

Most States Expressed Satisfaction with Education’s Outreach, but Had Mixed Views on PBDMI’s Potential Benefits

States were generally satisfied with Education’s outreach activities. (See table 3.) Most state data providers—72 percent—rated Education’s site visits effective in improving the partnership with the states. One state data provider characterized his exchanges with the department as open and non-defensive, and further reported that the department had been responsive. More than half rated as effective or very effective Education’s technical assistance (57 percent) and regional meetings (52 percent).

While most states thought Education’s activities to improve its partnership with states were effective, some suggested areas for improvements. For example, 72 percent thought the site visits provided only some or little information on successes achieved in other states.

In their survey responses half of the states expressed the view that reducing their reporting burden was the most important PBDMI goal; however, fewer than a third of the states said they believe the initiative will do so. (See table 4.) Some states emphasized their burden had increased in the short term as they continued dual reporting in order to meet the still ongoing data collection requirements of the program offices. Three states reported to us their cost estimates of systems development projects needed to support PBDMI, which ranged from approximately $120,000 to as much as $5 million. Moreover, about 75 percent of the states reported that they thought the burden to collect data would remain the same or increase once PBDMI was implemented. Some state respondents expressed the opinion that until there is a firm commitment by Education to halt multiple data collections their reporting burden would not likely lessen. “We are asked from the federal government for more and more information…. opens the flood gate for more and more reporting,” noted one official, adding that it is currently “hard to see the benefit at this time.”

Some states also had reservations about the benefits of PBDMI for evaluation. One respondent cautioned, for example, that support within his state had weakened because of the lack of perceived benefits. Only about 20 percent of states expected PBDMI to improve or greatly improve their analytic capacity—that is the ability to meet their own state reporting requirements, analyze program effectiveness, analyze student outcomes, and to compare outcomes within states. Their reasons varied. For example, five states reported that they would continue to use their own systems. A few elaborated that their own information systems allow more detailed analyses of state performance than the information to be collected through PBDMI. Additionally, an almost equal number of states saw PBDMI as an effective tool to inform stakeholders as not. Table 5 lists the extent to which state data providers expect PBDMI to enhance their analytical capacity in a variety of areas.

Many States Are Not Prepared to Meet Education’s Data Requests

As of June 3, 2005, only 9 states had submitted more than half of the requested 2003-2004 school year data, while 29 states had submitted less than 20 percent (see fig. 5). Although PBDMI officials said they will wait until August 2005 for states to submit the 2003-2004 data, they also acknowledged that many states would not be able to provide significant portions. The lack of state data is particularly acute in some programmatic areas. For example, many states have been unable to provide data on homeless and migrant students or students with limited English proficiency. States told Education officials early in the process that changes to state data collection processes, systems, and definitions would be needed to provide these types of information.

We found that there were various reasons why states could not provide data. Some states reported that they wanted better documentation from the department in areas such as clarifying established data definitions and file format specifications needed to transmit data. States needed to make major modifications to their existing data collection and reporting processes in order to provide new information required by PBDMI. States also reported that they would not provide certain data elements that were inapplicable, hard to collect, or available elsewhere. Some also reported that there was still some confusion over multiple or unclear definitions. Department officials said that many states had initially overestimated their capabilities and that the data states said would be available differed greatly from what they have produced thus far. States have also noted competing demands for their time and resources stemming from NCLBA. Some states reported they lacked resources, such as staff and money, to implement changes specific to the initiative. Specifically, 56 percent of the state survey respondents said that all or a portion of the $50,000 in grants they received from Education were used to contract for additional personnel, a quarter of the states said that these funds were used to improve their information systems. Some states noted, however, that these funds were insufficient to make changes necessary for their participation in PBDMI.

Recognizing that obtaining state data has been problematic, Education has recently developed a preliminary strategy for working more closely with states to ensure that it obtains 100 percent of data from all the states. While not finalized, Education is currently considering actions such as issuing regulations requiring states to submit PBDMI data and allowing those states that provide acceptable amounts of “high quality” data under PBDMI to be exempt from existing data collections. For example, states that submit data to PBDMI that are also currently collected through the Consolidated State Report—one of many data collections required under the NCLBA—would not have to submit the same data under this data collection. Officials have also tentatively proposed collecting data of lesser quality that are readily available and obtaining data through other systems to supplement what has been provided thus far. It is not clear the extent to which this proposal would undermine efforts to improve data quality and maintain program office buy-in. Another option under consideration at Education is to target departmental resources, such as $25 million in grants for system improvements from the Institute of Education Sciences, at states that actively participate in PBDMI.

Education Is Proceeding with Implementation despite the Data Shortage and without a Detailed Plan of Action

Education is proceeding with efforts toward full implementation of PBDMI—using the data for analysis and reporting—despite the limited amount of data collected. To do so, program offices decide whether the quality of the data (in terms of accuracy, consistency, timeliness, and utility) collected through PBDMI meets their needs. Once program offices validate the quality of the data, Education would begin to phase out existing data collections. Additionally staff will be trained on how to access and use the data collected to date. Originally Education expected to complete all of these activities by the spring of 2004. To the degree that it has been able to proceed, the department has developed a set of quality checks designed to ensure the accuracy and completeness of the data states submit.

Nevertheless, two program offices, which as members of the seven principle offices included in the initiative and have a role in determining whether the data are accurate and complete for their purposes, expressed concern that PBDMI’s procedures to ensure data quality may not be adequate. An official in the Office of Special Education and Rehabilitative Services (OSERS), which has collected almost 30 years of longitudinal data about the effectiveness of the nation’s special education programs, told us that PBDMI had been provided with information about the nearly 200 data quality checks used in special education collections, but was not sure that PBDMI adopted them all. PBDMI officials said they adopted those that were universally relevant. Further, this official expressed concern that PBDMI would not meet its special needs. Specifically, unlike other program offices, OSERS programs bases student assessment on age as opposed to grade level attained. Additionally, this official was concerned about the timeliness of the data collected through PBDMI because that office generated a number of congressionally mandated reports at specific times of the year. Consequently, this office plans to compare the quality of its own data with the data collected through PBDMI. Officials in the Office for Civil Rights also expressed similar reservations with PBDMI’s administration of its large elementary and secondary survey of schools and districts used to assess compliance with civil rights laws and identify trends. Historically, district superintendents have responded to this survey in large enough numbers allowing Education to generalize on any findings with a high degree of confidence. However, when PBDMI administered the survey, fewer superintendents responded and, according to the Office for Civil Rights, PBDMI did not have a readily available plan that adequately outlined steps needed to raise the response rate. As of June 10, 2005, the response rate for this survey was lower than previous surveys.

Final implementation has also been hampered by delays in training and delivery of the analysis and reporting system. Both are more than a year behind schedule. An official responsible for overseeing the training efforts told us they could not focus on the delays because considerable time was spent addressing state problems submitting data through PBDMI. The data analysis and reporting system is more than a year behind scheduled due to the lack of data and the failure of Education’s contractor to meet its scheduled delivery of the system. Education officials now expect to fully implement the system by March 31, 2006. In lieu of developing its data analysis and reporting system and training, PBDMI has offered presentations of these tools as a preview for staff to see the new system’s capabilities and to keep them apprised of the initiative’s progress.

Despite the many obstacles confronting the PBDMI, Education officials said they expect to proceed with implementation of the initiative, albeit with some activities postponed. In August, project officials developed a preliminary strategy designed to address the problem of collecting data from the states, such as providing exemptions from certain reporting requirements for some states. However, this strategy has not been finalized, and Education has not developed a specific plan of action for how they might (1) help states that are deficient, (2) deal with state expectations for phasing out the multiple data collections, or (3) meet the expectations of their own program offices.

Conclusions

The PBDMI represents an important step forward for the Department of Education in its efforts to monitor the performance of the nation’s elementary and secondary schools. By developing the ability to collect data that are more accurate, timely, consistent, and focused on key national performance indicators, Education will be much better informed to make its many policy and programmatic decisions. The initiative, by asking for a clearly defined set of information that is to be submitted only one time, has the potential to substantially reduce state reporting burden for elementary and secondary programs as well as to help states to develop better data systems. However, PBDMI is an ambitious and risky undertaking that requires the continued cooperation of a number of internal and external stakeholders.

In order for PBDMI to be successful, the department must rely on states to provide new information at a time when they are busy implementing large new federal initiatives, such as the No Child Left Behind Act. While some states have been able to provide significant amounts of data, others continue to lag far behind. In order for PBDMI to be successful, it is important for all states to submit timely, reliable, accurate, and consistent data. Consequently, it is important for the department to have a clear plan for addressing states with problems providing data and to continue to provide a proper combination of support and incentives for states to participate. By having worked closely with the states on their collection systems, PBDMI officials have the information they would need to develop a plan of action to help move them forward.

Because PBDMI represents a significant change in the way the Department of Education conducts business, it can only be accompanied effectively and efficiently by a change in management practices. However, program offices still retain much discretion over what data they will collect, how they will define it, and whether or not PBDMI’s data will meet their needs. While it is the initiative’s responsibility to make sure it collects data that meets the program offices’ requirements, PBDMI is also responsible for developing a data collection system focused on program performance and quality data. To the extent that programmatic differences, such as those over data definitions, inhibit PBDMI’s goals there should be a clear process for reconciling those differences. If PBDMI truly represents a new way of doing business, Education should be able to ensure that its organizational units go along. It is difficult to see PBDMI achieving its full potential without a clear process for furthering the initiative’s goals.

Fundamental to any large, complex effort’s success is a well thought out plan that tracks its progress against a set of clearly defined and measurable goals. PBDMI has not put in place such a planning and tracking system. State governments and Education’s program offices have devoted much time, effort, and money participating in PBDMI with the idea that they would see benefits as a result. A lack of demonstrated progress and benefits potentially erodes state support, undermining the viability of this important initiative. Some states are already beginning to lose sight of the potential benefits of PBDMI. As the department goes past its original completion deadline, it is important for it to lay out a clear plan for how it will now proceed.

Recommendations for Executive Action

To address the issues we have identified with regard to planning, decision- making, and improving data quality, we recommend that the Secretary of Education develop a strategy to help states improve their ability to provide quality data given the challenges that many states face in providing data; a clear process for reconciling differences between the program offices and the PBDMI oversight office to ensure that decisions critical to the success of PBDMI are made; and a clear plan for completing final aspects of PBDMI, including specific time frames and indicators of progress toward the initiative’s goals.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from the Department of Education. Education agreed with our findings and recommendations and stated that it has devoted additional resources to the initiative and plan to issue a detailed project plan that outlines the steps needed to complete the initiative. These comments are reprinted in appendix II.

Education also provided technical corrections and comments that we incorporated where appropriate.

We are sending copies of this report to the Secretary of Education, the Office of Strategic Accountability Services, the Director of the Office of Management and Budget, and appropriate congressional committees. Copies will also be made available to other interested parties upon request. Additional copies can be obtained at no cost from our Web site at www.gao.gov.

If you or your staff should have any questions, please call me at 415-904- 2272 or bellisd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report.

GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

The objective of our review of the Performance Based Data Management Initiative (PBDMI) was to assess the progress Education has made in its implementation of the initiative, particularly with regard to (1) defining what performance-related data it will collect from states on behalf of the program offices, (2) assisting states in their efforts to submit quality information, and (3) utilizing performance-related data to provide enhanced analytic capacity within the program offices. We conducted our review between April 2004 and September 2005 in accordance with generally accepted government auditing standards.

Overall Approach

To assess the department’s progress in each of these areas, we reviewed documents relating to the implementation of the initiative, relevant laws, and information provided by the office responsible for PBDMI—the Strategic Accountability Service (SAS) and others. We interviewed key staff responsible for the initiative as well as officials in each of the offices that are participating in PBDMI. We also interviewed senior-level Education officials to determine their role in the implementation of PBDMI. To gain insight into state perspectives on the initiative, we administered a Web-based survey to state officials responsible for providing these data to Education. We received responses from 50 states including Puerto Rico. We also interviewed a variety of external stakeholders, a data standards organization, and three contractors involved in the initiative, including an official from the Council of Chief State School Officers. We also reviewed previously issued reports by Education’s Office of the Inspector General (IG) as well as GAO reports and testimonies.

In addition to interviewing departmental officials, we also reviewed documentation on the initiative to gain a better understanding of what actions Education was undertaking to implement the goals of the initiative, including its data quality contract, data dictionary, its business plans as well as justification reports to the Office of Management and Budget (OMB) required under the Paperwork Reduction Act to collect data. We also reviewed summary information about state performance data that was obtained as a result of site visits to states conducted in 2004 in order to analyze what data was obtained from states as a result of their efforts.

Education provided information on states’ submission of requested data elements to PBDMI as of June 3, 2005. States were expected to provide data for 64 data elements ranging from dropout rates, student performance on reading, science, and writing assessments, teacher certification, and many others. For each of these elements, Education determined whether each state had submitted the information, had not submitted the data, or did not collect the information. We incorporated into our report Education’s calculated percentages of elements submitted for each state. We determined that these data were sufficient for the purposes of this engagement.

In order to document the burden hours associated with certain elementary and secondary data collections, we accessed 14 data collection justifications authored by each of the department’s program offices and submitted to the chief information officer. These reports had received OMB approval or were seeking approval to collect data from states and others. We talked with an official responsible for maintaining these documents at the department’s Web site to verify that these were the most recent data available for analysis. From each document we obtained the estimated state burden hours and costs and federal administrative costs associated with each data collection. Each estimate was based on a formula that we adjusted to reflect these costs for the 52 states participating in the initiative. In some instances where an average was used, we assumed that the 52 states were similar in characteristics to the overall population of states included in Education’s estimates. However, we did not find it feasible to prorate the formulas for the federal administrative costs (based on 52 states) for each of the collections. A statistician verified each of the calculated estimates for accuracy.

We also surveyed all 52 state data coordinators using a Web-based survey instrument in order to obtain their perspectives on various aspects of the initiative. Our survey instrument was developed based on information obtained during interviews with state data coordinators in Pennsylvania, Virginia, Washington, and Oregon. Additionally, other internal stakeholders specializing in technology and education were asked to review and comment on our draft survey instrument. The survey was pre- tested with Wyoming, North Carolina, and Illinois to determine if the questions were clear and unbiased and whether the terms were accurate and precise. We included these three states in our pretests because they varied in size and technical capacity for data transmission as determined by an earlier Education survey. Based on their comments, we refined the questionnaire as appropriate.

Our final survey instrument asked a combination of questions that allowed for closed-ended as well as open-ended responses and included questions about state perspectives on PBDMI’s ability to achieve its goals. The survey was conducted using self-administered electronic questionnaire posted on the Internet. We sent e-mail notifications about the upcoming survey to all 52 state data coordinators (50 states, the District of Columbia, and Puerto Rico) on November 15, 2004, and activated the survey shortly thereafter. Each potential respondent was provided a unique password and username by e-mail to limit participation to members of the target population. To encourage respondents to complete the questionnaire, we sent an e-mail message to prompt each non-respondent approximately 2 weeks after the survey was activated and followed up by e-mail or phone with each non-respondent several times thereafter. We closed the survey on January 21, 2005, after the 50th respondent had replied.

Because this was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as non-sampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or in how the data are entered into a database or were analyzed can introduce unwanted variability into the survey results. We took steps in the development of the survey instrument, the data collection, and the data analysis to minimize these non-sampling errors. For example, a survey specialist designed the survey instrument in collaboration with GAO staff with subject matter expertise. Then, as stated earlier, it was pre-tested to ensure that the questions were clear, unbiased, and accurate. When the data were analyzed, a second, independent analyst checked all computer programs. Because this was a Web-based survey, respondents entered their answers directly into the electronic questionnaire, eliminating the need to have the data keyed into a database, thus removing an additional source of error.

Appendix II: Comments from the Department of Education

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above the following individuals made important contributions to this report: Bryon Gordon, Assistant Director; Carla Craddock, Analyst-in-Charge; Susan Bernstein; David Dornisch; Mary Dorsey; Kimberly Gianopoulos; Brandon Haller; Stuart Kaufman; Jonathan McMurray; Valerie Melvin; James Rebbe; Gloria Hernandez Saunders; Kimberly Siegel; Michelle Verbrugge; and Elias Walsh.