Career and Technical Education: States Have Broad Flexibility in Implementing Perkins IV
Released: 2009-07-29T00:00:00

Letter

Background

Under Perkins IV, Education Allocates Funds for Career and Technical Education to States in order to Improve Local CTE Programs

The principal source of federal funding for CTE, Perkins IV authorizes federal grant funds for the enhancement of CTE for secondary and postsecondary students. In fiscal year 2008, Congress appropriated $1.2 billion for the improvement of local CTE programs. Education’s Office of Vocational and Adult Education allocates the funds to states, which retain up to 15 percent of the funds for administration and state leadership of CTE programs, before passing at least 85 percent of the funds on to local recipients of funds, such as local school districts and community colleges. States determine the percentage of funds that will be allocated to the secondary and postsecondary levels. The majority of funds allocated to the secondary level are passed on to local recipients based on the school district’s share of students from families below the poverty level for the preceding fiscal year. Postsecondary funds are primarily allocated based on the institution’s share of Pell Grant recipients.

Perkins IV Established Performance Measures for Secondary and Postsecondary Levels and Requires States and Local Recipients to Report on Program Outcomes

Perkins IV established six student performance measures at the secondary level and five performance measures at the postsecondary level. These measures represent a range of student outcomes, such as attainment of technical skills and placement in employment or further education following the completion of CTE programs. In addition, the measures include the nontraditional participation and completion of students from an underrepresented gender in programs with significant gender disparities (such as women participating in auto repair), among others (see tables 1 and 2 for a description of the Perkins IV performance measures). To ease states’ transition to the new provisions in Perkins IV, Education permitted states to submit a 1-year transition plan that covered only the first program year of Perkins IV implementation, 2007-2008. Accordingly, states were required only to implement and report performance on two secondary performance measures for the 2007-2008 program year: academic attainment and student graduation rates. These two measures are based on the same academic attainment and student graduation rate measures required by Title I of the Elementary and Secondary Education Act. Beginning in the 2008-2009 program year, states are required to report on student outcomes for all of the performance measures. States will report these outcomes to Education in December 2009.

Perkins IV requires states to negotiate specific performance targets with Education and to annually report their performance to Education. It also requires local recipients to negotiate performance targets with the states and to annually report to the state their progress toward meeting these targets. Perkins IV established additional accountability requirements for states and local recipients, including actions to address states that do not meet all of their performance targets. Under Perkins IV, if a state does not meet at least 90 percent of its targets for one or more of the performance measures, it is required to develop and implement a program improvement plan that describes how it will address its failing performance targets. Prior to Perkins IV, states were only required to develop and implement a program improvement plan if they failed to meet their targets in all of their performance measures, not just one measure. States can also face financial sanctions. For example, Education can withhold all or a portion of funds if a state does not implement a program improvement plan, show improvement in meeting its failing performance measure, or meet the target for the same performance measure for 3 consecutive years. Local recipients that do not meet at least 90 percent of their performance targets have the same program improvement requirements as the state and face similar sanctions from the state. In the event of financial sanctions, Education is required to use the withheld funds to provide technical assistance to the state for improving its performance on the measures and the state is to use funds withheld from local recipients to provide CTE services and activities to students.

Education Developed Nonregulatory Guidance to Assist States with Perkins IV

In order to implement the performance measurement requirements of Perkins IV, states must define which students will be included in the measures and collect data for each of the performance measures at the secondary and postsecondary levels. For example, states define the minimum requirements, such as a certain number of CTE credits that a student would need to obtain in order to be identified as a student concentrating in CTE. Education has taken a range of actions to help states with these activities. For example, in January 2007, Education began issuing nonregulatory guidance to states to help them develop their student definitions and data collection approaches for the performance measures. Education also issued guidance to states on the information states must include in their state Perkins plans and in the annual reports that they submit to Education. In the state plans, states must detail how they intend to implement the performance measures, and in the annual reports states must describe their progress in meeting the negotiated performance targets.

In Addition to Implementing the Perkins Performance Measures, Perkins IV Also Requires States to Annually Evaluate Their Local CTE Programs

In addition to implementing performance measures, states are required to evaluate programs, services, and activities supported with Perkins funds and to report to Education in their state plans how they intend to conduct these evaluations. To meet this requirement, states describe the approaches, such as the use of state-developed standards, they will use to evaluate local CTE programs. In addition, Education requires states to include a description of how they used Perkins funds to evaluate their local CTE programs in their annual reports.

States Are Implementing Some Performance Measures Using Different Approaches and Report That the Greatest Challenge Is Collecting Data on Technical Skill Attainment and Student Placement

Flexibility in Law and Guidance Allows for Differences in How States Implement Some Performance Measures and Results in Variation in the Student Outcome Data Education Will Collect

A key feature of Perkins IV—to enhance state and local flexibility in developing, implementing, and improving career and technical education—allows for considerable variation in how states implement some performance measures. While Perkins IV was designed to strengthen accountability for results at the state and local levels, it also allows states to establish their own accountability systems, including their own data collection methods for the performance measures. Of the 11 performance measures, the secondary and postsecondary levels have 3 measures in common: technical skill attainment, student placement, and participation in and completion of nontraditional programs (see fig. 1). States may also include additional, state-developed performance measures in their accountability systems. For example, Washington state added three performance measures—earnings, employer satisfaction, and CTE student satisfaction—to its accountability system.

Consistent with Perkins IV, Education’s guidance to states also allows for flexibility. Education issued nonregulatory guidance that proposed specific definitions that could be adopted by states to develop each of the secondary and postsecondary performance measures. It also identified preferred approaches for collecting data for certain measures such as student technical skill attainment. However, Education noted that in accordance with Perkins IV, states could propose other definitions and approaches to collect data for the required performance measures if they meet the requirements of the law.

We found through our surveys of state CTE directors that states vary considerably in the extent to which they plan to follow Education’s guidance—specifically with regard to the technical skill attainment and secondary school completion measures. As a result, Education will collect student outcome data that vary across states for the same measures. This can create challenges for Education to aggregate student outcomes at the national level. For example, a majority of states reported that they will use technical assessments—the approach recommended in Education’s guidance—to measure student attainment of skills at the secondary and postsecondary levels. These include assessments leading to industry-ba certificates or state licenses. However, a number of states will rely on other approaches to collect data for the performance measure, including grade poi nt average (GPA), program completion, or other methods (see table 3).

Officials in the states we visited provided a variety of reasons for their use of alternate methods to measure students’ attainment of technical skills. For example, postsecondary state officials in California said that a CTE instructor’s overall evaluation of a student’s technical skill proficiency, in the form of a final grade, is a better measure of technical skill attainmentthan third-party technical assessments, and can more effectivel program improvement. They questioned the value of technical assessments, in part because assessments often cannot keep pace with technology and changing CTE program curricula, such as curricula for digital animation. A Washington state official told us that the state plans to use program completion to measure technical skills at the postsecondary level, noting that each postsecondary CTE program incorporates industry- recognized standards into the curriculum. He noted that a national system se it of third-party assessments may not be adequate or appropriate, becau would not necessarily incorporate the same standards. Local school officials in Minnesota said that they will report on CTE course complet view by for this measure. Because CTE courses undergo curriculum re teachers as well as industry advisors, and align with relevant postsecondary programs in the area, school officials told us cours completion i attainment. s sufficient to satisfy the definition of technical skill Education’s guidance also allows for considerable variation in the types of technical assessments states can use and when they can administer them.

Most states at the secondary level reported in our survey that they plan to use industry-developed certificates or credentials most often administere at the end of a program, such as a certificate awarded for an automotive at the end of a program, such as a certificate awarded for an automotive technician. At the postsecondary level, states plan to most often rely upon technician. At the postsecondary level, states plan to most often rely upon the results of assessments for state lice the results of assessments for state licenses, such as state nursing licenses, to measure technical skills (see fig. 2). nses, such as state nursing licenses, to measure technical skills (see fig. 2).

However, we found that while a majority of states plan to use assessment to report to Education, the assessments are not currently in widespread use. For example, more than half of states at the secondary and postsecondary levels reported that they plan to use these assessments to report on few to none of their state-approved CTE programs in the 2008- s 2009 program year. Some states at the secondary level reported will use a combination of methods—including GPA o r program completion—to report on technical skill attainment.

We also found that states differ in whether they plan to report student dat on GED credentials, part of the secondary school completion measure. Thirty states reported through our survey that they do not plan to report GED data to Education for the 2008-2009 program year, while 18 reported that they would. About one-third of all states cited their ability to access accurate GED data as a great or very great challenge. For example, state officials we interviewed said states face difficulty tracking the students that leave secondary education and return, sometimes several years lat to earn a GED credential. An Education official said that the agency is aware of the challenges and limitations states face in collecting GED data ed to provide technical assistance to states on and that the agency may ne ways to collect these data.

States Face the Most Challenge Collecting Data on Student Technical Skill Attainment and Placement Measures because of Cost and Data Concerns

States reported in our surveys that they face the most difficulty in collecting student data for two of the performance measures: technical skill attainment and student placement (see fig. 3 and fig. 4). Thirty-eight states at the secondary level reported that they face great or very great challenges in collecting data on student technical skill attainment, while, similarly, 14 said they face challenges collecting data on student placement. The results were similar at the postsecondary level: 39 states reported great or very great challenges with the technical skill attainment measure and 11 cited a similar level of difficulty with student placement.

States reported that the technical skill attainment measure at the secondary and postsecondary levels was most challenging to implement because of costs and the states’ ability to collect accurate and complete student data. Specifically, states reported that the costs of state-developed assessments and third-party technical assessments—such as those for industry certifications—are high and often too expensive for many districts, institutions, or students. Several state CTE directors commented in our surveys that their Perkins funds are inadequate to pay for these assessments and additional funds would be necessary to cover the costs. Another CTE director stated that economically disadvantaged students cannot afford the cost of assessments. In addition to challenges due to cost, states are limited in their ability to access accurate and complete data. For example, a state official said that Washington state does not have data-sharing agreements with assessment providers to receive the results of student assessments. As a result, the state will have to rely largely on students to self-report the results of their assessments, which raises concerns of data quality. Challenges such as these likely contribute to some states’ use of other data—such as GPA or program completion—to collect and report information for this key student performance measure.

Some states also reported difficulty collecting data on CTE students after they leave the school system. States at the secondary and postsecondary levels reported that their greatest challenge with the student placement measure is collecting data on students that are employed out of state. As we previously reported, state wage records, such as Unemployment Insurance data, track employment-related outcomes only within a state, not across states. A number of states commented in our surveys on challenges in tracking students because of the lack of data sharing across states. We found that states face challenges in tracking students employed out of state regardless of the method they most commonly use to collect student placement data. Thirty-eight states at the secondary level will use student survey data from the state, school district, or a third party to track student placement and report to Education, while 41 states at the postsecondary level will rely on state wage record data, despite potential gaps in student data (see fig. 5). g. 5).

States also cited other challenges in obtaining data on student placement for CTE students. At the secondary level, states reported that their next greatest challenge is linking secondary and postsecondary data systems in order to track students that pursue higher education after graduation. To help overcome this challenge, Minnesota—one of the states we visited— recently passed legislation to allow data sharing between the secondary and postsecondary levels. Our survey also found that states’ next greatest challenge at the postsecondary level was collecting data on students who are self-employed after leaving postsecondary institutions. Community college officials in California said that while they rely on Unemployment Insurance wage record data, the data are incomplete and do not capture information on the self-employed, a group that is important for the measurement of CTE outcomes at the postsecondary level.

States face similar challenges of cost and ability to access accurate data for the remaining performance measures. For example, states at the secondary level commented on data challenges for the academic attainment and student graduation rate measures. Specifically, several states cited problems in obtaining data from separate student data systems containing academic and CTE information. This can be particularly challenging for states that are trying to match student data from different systems in order to track required CTE student outcomes. In addition, at the postsecondary level, states cited challenges in tracking student retention in postsecondary education or student transfer to a baccalaureate degree program. In particular, accessing student data from out-of-state and private institutions and the high costs required to track these students were identified as the most challenging issues. States most often reported that they will track these students through their state postsecondary data systems.

Education Uses Risk- Based Monitoring to Ensure Implementation of the Performance Measures and Supports States through Technical Assistance and Guidance

Education Uses Risk- Based Monitoring and Reviews State Annual Reports to Ensure Implementation of the Performance Measures

As we have previously reported, effective monitoring is a critical component of grant management. The Domestic Working Group’s suggested grant practices state that financial and performance monitoring is important to ensure accountability and attainment of performance goals. Additionally, GAO recently reported on the importance of using a risk-based strategy to monitor grants and noted that it is important to identify, prioritize, and manage potential at-risk grant recipients, given the large number of grants awarded by federal agencies. Education’s approach to monitoring Perkins is consistent with these suggested grant practices. According to its Perkins monitoring plan, Education selects which states to monitor based on a combination of risk factors and monitors states in two ways: through on-site visits and off-site reviews of state plans, budgets, and annual reports for those states not visited in a given year. To determine which states it will visit for on-site monitoring, Education uses a combination of risk factors, such as grant award size, issues identified through reviews of state Perkins plans, and time elapsed since Education’s last monitoring visit. Education officials told us that their goal is to visit each state at least once every 5 years and reported that they have conducted on-site monitoring visits to 28 states since 2006. Education officials also told us that the same monitoring team performs both on-site and off-site reviews, which officials said helps to ensure continuity between the reviews. Furthermore, when conducting the off- site reviews, the monitoring team looks for trends in state data and for any problems with state data validity and reliability. The team uses a checklist to match performance data to the data states report in their required annual reports.

According to Education’s inventory of open monitoring findings, as of May 2009, 9 of the 28 open findings were related to accountability and states failing to submit complete or reliable data. For example, in a February 2008 monitoring visit, Education found that the monitored state’s data system had design limitations that affected the state’s ability to collect and assess data on career and technical education students. Specifically, Education found that the various data systems across the local secondary and postsecondary levels did not share data with the state-level CTE data system. This data-sharing issue raised doubts about the validity and reliability of the state’s Perkins data. Education tracks the findings from each state’s monitoring visit in a database and reviews the findings in an internal report that is updated monthly. Additionally, if a state has open findings, the state may be required to report corrective actions to Education in the state’s annual report. Officials told us that the amount of time it takes for a state to close out a finding depends upon the nature of the finding. For example, a finding related to accountability may take up to a year to resolve because a state may have to undertake extensive actions to address the deficiency. Education officials reported that their monitoring process emphasizes program improvement rather than focusing solely on compliance issues and that they use monitoring findings to guide the technical assistance they provide to the states.

To evaluate its monitoring process, Education sends a survey to the CTE directors of states that were monitored that year and asks them to rate the format and content of Education’s Perkins monitoring process. For example, the survey asks states to report on whether they received sufficient notice that the site visit was going to take place, whether the monitoring team provided on-site technical assistance, and whether the state received a written report within a reasonable time frame following the visit. We reviewed Education’s summaries of the state surveys and found that for 2004 and 2005, the results of these surveys were generally positive. For example, in a 2004 monitoring evaluation report, the 10 states that were surveyed all reported that they had received sufficient notice about the monitoring visit and that Education staff provided on-site technical assistance. According to our survey of secondary-level CTE directors, about half of states have had a monitoring visit within the last 3 years, and almost all of the states whose monitoring visit resulted in findings said that Education worked with them to ensure that the findings were addressed.

Education Supports States by Providing Technical Assistance and Guidance

Education provides states with guidance, technical assistance, and a variety of other resources and is taking actions to meet states’ need for additional help. Since Perkins IV was enacted, Education has issued guidance to states on topics such as instructions for developing the state Perkins plans and annual reports, as well as guidance related to the performance measures. For example, Education’s guidance provides clarification to states on what information each state has to submit to Education before it can receive its grant award for the next program year, such as any revisions a state wants to make to its definitions of student populations, measurement approaches, and proposed performance levels for each of the measures. Some of the guidance resulted from Education’s collaborative efforts with states. For example, Education’s guidance to states on student definitions and measurement approaches incorporated the input given by state CTE directors during national conference calls between states and Education. Other guidance addresses questions raised by states during national Perkins IV meetings, such as how a state should negotiate performance levels with its local recipients.

In addition to guidance, Education offers states technical assistance from Education staff—called Regional Accountability Specialists—and through a private contractor. Education officials told us that each Regional Accountability Specialist works with a specific group of states to negotiate state data collection approaches for the performance measures. In addition, each specialist maintains regular contact with his or her states throughout the year and provides assistance on other issues, such as reporting requirements and program improvement plans. In addition to the Regional Accountability Specialists, Education also provides states with technical assistance by using MPR Associates, a private contractor. MPR Associates provides technical assistance that generally includes on-site visits and follow-up discussions to help states improve their CTE programs and facilitate data collection for the performance measures. For example, MPR Associates met with one state to assist with developing population definitions and measurement approaches that aligned with Education’s guidance and helped another state with developing a plan for implementing secondary and postsecondary technical skill assessments. After providing technical assistance to a state, MPR Associates develops a summary report, which is then published on Education’s information- sharing Web site, the Peer Collaborative Resource Network. Education also offers states a range of other resources, including data work groups and monthly conference calls. See table 4 for a description of the various ways in which Education provides assistance to states.

Most states reported that the assistance provided by Education has helped them implement the performance measures, but that more assistance in the area of technical skill attainment would be helpful. In our survey, states responded positively about their Regional Accountability Specialist and all of Education’s other forms of assistance, including the Data Quality Institute and the Next Steps Work Group. States also reported that more nonregulatory guidance and more individual technical assistance would improve their ability to implement the performance measures. Of the states that provided additional information on the areas in which they wanted assistance, 4 of 16 states at the secondary level and 9 of 20 states at the postsecondary level said that they wanted assistance on the technical skill attainment measure. Specifically, some of the states that provided additional information said they would like Education to clarify its expectations for this measure, to provide states with a library of technical assessments, and to provide state-specific assistance with developing low-cost, effective technical assessments. States also raised issues regarding the performance measures and their state’s data collection challenges. For example, one state reported that it was unsure how a state should report technical skill attainment as a single measure for over 400 distinct CTE programs.

We found that Education officials were aware of states’ need for additional assistance and that Education has taken some actions to address these needs, particularly in the area of technical assessments. For example, through the Next Steps Work Group, Education facilitated a technical skills attainment subgroup that is led by state officials and a national research organization. The subgroup reviewed state Perkins plans and annual reports for technical skill assessment strategies that states reported to Education for consideration in upcoming guidance. Education also collaborated with MPR Associates to conduct a study on the feasibility of a national technical assessment clearinghouse and test item bank. The study, conducted with several CTE research organizations and state-level consortia, proposed national clearinghouse models for technical assessments. MPR Associates concluded that clarifying ownership, such as who is responsible for the development and management of the system, and securing start-up funding were the two most likely impediments to creating such a system. The report was presented to states at the October 2008 Data Quality Institute seminar, and Education officials reported that they are working with organizations such as the National Association for State Directors of Career and Technical Education Consortium and the Council of Chief State School Officers to implement next steps.

In addition to helping states with the technical skill attainment measure, Education also has taken actions to improve its information-sharing Web site, the Peer Collaborative Resource Network. Specifically, a Next Steps Work Group subcommittee surveyed states for suggested ways to improve the Web site and reported that states wanted to see the information on the site kept more current. The subcommittee reported in December 2008 that Education would use the survey results to develop a work plan to update the Web site. In May 2009, Education officials reported that they had implemented the work plan and were piloting the revamped site with selected state CTE directors before the department finalizes and formally launches the site.

Education Relies on the Performance Measures to Gauge the Success of State CTE Programs

State performance measures are the primary source of data available to Education for determining the effectiveness of CTE programs, and Education relies on student outcomes reported through these measures to gauge the success of states’ programs. While Perkins IV requires states to evaluate their programs supported with Perkins funds, it only requires states to report to Education—through their state plans—how they intend to evaluate the effectiveness of their CTE programs. It does not require states to report on the findings of their evaluations and does not provide any specific guidance on how states should evaluate their programs.

Because only 2 of 11 measures have been implemented and reported on thus far, Education has little information to date on program outcomes. In program year 2007-2008, Education required states to implement and report only the academic skill attainment and graduation rate measures. States are required to provide Education with outcome data for the remaining 9 secondary and postsecondary measures in December 2009. According to Education’s annual report for program year 2007-2008, 43 states met their targets for the academic attainment in reading/language arts measure, 38 states met their targets for the academic attainment in mathematics measure, and 46 states met their targets for the graduation rate measure.

We analyzed the state plans of all 50 states and the District of Columbia and found that, as required by Perkins IV, states provide a description to Education on how they are evaluating their CTE programs. The type of information that states provided varied. For example, some states described the databases they use to capture key data and others explained how they use state-developed performance measures to evaluate their programs. Perkins IV does not require that states include information on what their evaluations may have found in terms of the success of a program. In our surveys of state CTE directors, nearly half of states (23 states at the secondary level and 21 states at the postsecondary level) responded that they have conducted or sponsored a study, in the past 5 years, to examine the effectiveness of their CTE programs. In response to these survey results, we collected seven studies that states identified as evaluations of their program effectiveness. We developed an instrument for evaluating these studies and determined the type of evaluation and methodology used by states in these studies. We determined that four of the studies were outcome evaluations and the remaining three studies were not outcome, impact, or process evaluations. For example, one state found in its outcome evaluation that high school graduates who completed a CTE program of study entered postsecondary institutions directly after high school at the same rate as all graduates.

Concluding Observations

Perkins IV provides states with considerable flexibility in how they implement the required performance measures and how they evaluate the effectiveness of their CTE programs. While this flexibility enables states to structure and evaluate their programs in ways that work best for them, it may hinder Education’s ability to gain a broader perspective on the success of state CTE programs. Specifically, differences in how states collect data for some performance measures may challenge Education’s ability to aggregate student outcomes at a national level and compare student outcomes on a state-by-state basis. Further, Education is limited in what it knows about the effectiveness of state CTE programs, beyond what states report through the performance measures. Perkins only requires that states report on how they are evaluating their programs, and does not provide any guidance on how states should evaluate their programs or require that states report on the outcomes of their evaluations. Education is working with states to help them overcome challenges they face in collecting and reporting student outcomes, and over time, states may collect more consistent data for measures such as technical skill attainment. As states become more adept at implementing the Perkins performance measures, they will be better positioned to conduct more rigorous evaluations of their CTE programs. However this information may not be reported to Education. If policymakers are interested in obtaining information on state evaluations, they will need to weigh the benefits of Education obtaining this information with the burden of additional reporting requirements.

Agency Comments

We provided a draft of this report and the electronic supplement to the Department of Education for review and comment. Education provided technical comments on the report, which we incorporated as appropriate. Education had no comments on the electronic supplement.

We are sending copies of this report to appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about the report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made major contributions to this report are listed in appendix II.

Appendix I: Scope and Methodology

To obtain national-level information on states’ implementation of Perkins IV, we designed and administered two Web-based surveys, at the secondary and postsecondary levels, to state directors of career and technical education (CTE) in the 50 states and the District of Columbia. The surveys were conducted between January and April 2009, with 100 percent of state CTE directors responding to each survey. The surveys included questions about the types of data states collect for the student performance measures and challenges they face; the various kinds of technical assistance, guidance, and monitoring states received from Education; and how states evaluate their CTE programs. The surveys and a more complete tabulation of the results can be viewed at GAO-09-737SP.

Because this was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce nonsampling errors, such as variations in how respondents interpret questions and their willingness to offer accurate responses. We took steps to minimize nonsampling errors, including pretesting draft survey instruments and using a Web-based administration system. Specifically, during survey development, we pretested draft instruments with officials in Minnesota, Washington state, and Vermont in December 2008. We also conducted expert reviews with officials from the National Association of State Directors of Career and Technical Education Consortium and MPR Associates, who provided comments on the survey. In the pretests and expert reviews, we were generally interested in the clarity of the questions and the flow and layout of the survey. For example, we wanted to ensure that terms used in the surveys were clear and known to the respondents, categories provided in closed-ended questions were complete and exclusive, and the ordering of survey sections and the questions within each section were appropriate. On the basis of the pretests and expert reviews, the Web instruments underwent some revisions. A second step we took to minimize nonsampling errors was using Web-based surveys. By allowing respondents to enter their responses directly into an electronic instrument, this method automatically created a record for each respondent in a data file and eliminated the need for and the errors associated with a manual data entry process. When the survey data were analyzed, a second, independent analyst checked all computer programs to further minimize error.

While we did not fully validate all of the information that state officials reported through our surveys, we reviewed the survey responses overall to determine that they were complete and reasonable. We also validated select pieces of information by corroborating the information with other sources. For example, we compared select state responses with information submitted to Education in state Perkins plans. On the basis of our checks, we believe our survey data are sufficiently reliable for the purposes of our work.

Site Visits

To better understand Perkins IV implementation at the state and local levels, we conducted site visits to three states—California, Minnesota, and Washington state—between September 2008 and February 2009. In each state we spoke with secondary and postsecondary officials at the state level with CTE and Perkins responsibilities. We also interviewed officials from local recipients of Perkins funds—that is, school districts and postsecondary institutions. Through our interviews with state and local officials, we collected information on efforts to implement the Perkins performance measures and uses of Perkins funding, experiences with Education’s monitoring and technical assistance, and methods for CTE program evaluation. The states we selected represent variation across characteristics such as the type of state agency (i.e., state educational agencies or state college and university systems) eligible to receive Perkins funds, the amount of Perkins IV funds received in fiscal year 2008, and type of approach used to measure student attainment of technical skills. The localities selected for site visits provided further variation in geographic location (urban versus rural), number of CTE students served, and amount of Perkins funding received.

We conducted this performance audit from August 2008 to July 2009, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contact named above, Elizabeth Morrison (Assistant Director), Avani Locke, Robin Nye, Charlotte Gamble, Stephen Steigleder, Jessica Orr, Jean McSween, Christine San, and Jessica Botsford made key contributions to this report.