Career and Technical Education: States Have Broad Flexibility in Implementing Perkins IV
Published: 2009-07-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV) supports career and technical education (CTE) in high schools and postsecondary institutions, such as community colleges. Perkins IV established student performance measures at the secondary and postsecondary levels for state agencies, such as state educational agencies, and local recipients, such as school districts, eligible to receive funds. GAO examined (1) how states have implemented the Perkins IV performance measures and what, if any, challenges they have faced in implementing the measures; (2) to what extent the Department of Education (Education) has ensured that states are implementing the new performance measures and supported states in their efforts; and (3) what Education knows about the effectiveness of CTE programs. To collect national-level data, GAO surveyed state CTE directors in the 50 states and District of Columbia between January and April 2009, and received responses from all states and the District of Columbia. To view survey results, click on http://www.gao.gov/special.pubs/gao-09-737sp/index.html . We provided a draft copy of this report to Education for comment. We received technical comments, which we incorporated into the draft where appropriate.
section_title: What GAO Found
paragraphs: States are implementing some of the Perkins IV performance measures using different approaches and report that the greatest challenge is collecting data on technical skill attainment and student placement. Flexibility in Perkins IV and Education's guidance permits differences in how states implement the measures. According to our surveys, 34 states at the secondary level and 29 at the postsecondary level intend to adopt Education's recommended use of assessments--such as those for industry certifications--to measure technical skills. States reported that they face the most challenge collecting data on the technical skill attainment and student placement measures because of cost and concerns with their ability to access complete and accurate data. Education ensures states are implementing the Perkins IV accountability requirements through on-site monitoring and off-site document reviews, and supports states through technical assistance and guidance. Monitoring findings were most often related to states failing to submit complete or reliable data, and Education uses its findings to guide the technical assistance it provides to states. States reported that Education's assistance has helped them implement the performance measures, but that more assistance with technical skill attainment would be helpful. Education is aware of states' need for additional assistance and has taken actions to address this, including facilitating a state-led committee looking at technical assessment approaches. State performance measures are the primary source of data available to Education for determining the effectiveness of CTE programs, and Education relies on student outcomes reported through these measures to gauge the success of states' programs. Because only 2 of 11 measures (secondary and postsecondary have 3 measures in common) have been implemented and reported on thus far, Education has little information to date on program outcomes. In addition, Perkins IV does not require states to report to Education the findings of their program evaluations. In our surveys of state CTE directors, nearly half of states responded that they have conducted or sponsored a study to examine the effectiveness of their CTE programs. We reviewed 7 of these studies and found that only 4 were outcome evaluations.

section_title: Letter
paragraphs: The shift to a global economy and rapid advances in technology  underscore the importance of preparing our current and future workforce  for high-demand careers with 21st century skills, such as those that  emphasize problem solving and teamwork. In the 2006-2007 program year,  more than 15 million high school and college students nationwide  participated in career and technical education (CTE) programs, which are  designed to provide students with the academic and career and technical  skills to help them succeed in the workforce. As authorized by the Carl D.  Perkins Career and Technical Education Act of 2006 (Perkins IV),  Congress provided states with $1.2 billion in fiscal year 2008 to support  career and technical education in high schools and to support programs in  postsecondary institutions, such as community colleges. The U.S.  Department of Education (Education) estimates that approximately 5  percent of all funds that states use for CTE programs are federal funds,  with state and local funding generally covering the remainder. The  American Recovery and Reinvestment Act of 2009 provides additional  funds that states can use to help support their CTE programs. Federal  funds for CTE programs are likely to take on increasing importance as  states continue to confront mounting fiscal pressures that may lead them  to propose cuts to secondary and postsecondary education spending used  to support career and technical education.
Perkins IV aims to prepare students for current or emerging high-skill,  high-wage, or high-demand jobs by emphasizing rigorous student  academic and technical skill achievement, increased accountability for  student outcomes, and enhanced coordination between secondary and  postsecondary career and technical education. It also seeks to increase  state and local flexibility in providing career and technical education by  involving multiple groups such as students, parents, and local  administrators in planning and administration, and by allowing states  flexibility in the design of their accountability systems. To increase  accountability for student outcomes, Perkins IV established student  performance measures at the secondary and postsecondary levels for state  agencies, such as state educational agencies or state college and  university systems, as well as for local recipients of funds, such as school  districts. Key performance measures include student attainment of  academic content standards and student academic achievement standards,  as adopted by the state in accordance with the requirements of Title I of  the Elementary and Secondary Education Act. Overall, Perkins IV reflects  a shift from an emphasis on vocational education—once considered by  some to be an occupationally specific track for students with lower  academic skills—to an emphasis on preparing students for entry into high- demand occupations.
Education provides technical assistance and guidance to states regarding  their data collection and student definitions and measurement approaches.  States report annually to Education on their progress in meeting their  performance targets for the measures. In light of a governmentwide focus  on performance and accountability, you asked us to examine (1) how  states have implemented the Perkins IV performance measures, and what,  if any, challenges they have faced in implementing the measures; (2) to  what extent Education has ensured that states are implementing the new  performance measures and supported states in their efforts; and (3) what  Education knows about the effectiveness of CTE programs.
To answer our three research questions, we collected data through  multiple methods. First, to gather state-level information on Perkins IV  implementation, we collected information through two Web-based surveys  of state CTE directors, at the secondary and postsecondary levels, in the  50 states and the District of Columbia. The surveys obtained information  on the types of data states collect for the student performance measures  and challenges they face; technical assistance, guidance, and monitoring  states receive from Education; and how states evaluate their CTE  programs. We administered the surveys between January and April 2009  and received responses from all 50 states and the District of Columbia.  While we did not fully validate specific information that state officials  reported through our surveys, we reviewed the information to determine  that their responses were complete and reasonable and found the  information to be sufficiently reliable for the purposes of this report. This  report does not contain all the results from the surveys. The surveys and a  more complete tabulation of the results can be viewed online at  GAO-09-737SP. In addition to our surveys, we collected information from  site visits to California, Minnesota, and Washington state. These states  represent variation across characteristics such as the type of state agency  eligible to receive Perkins funds; amount of Perkins IV funds received in  fiscal year 2008; and type of approach used to assess how students attain  technical skills, a key program outcome. We interviewed secondary and  postsecondary officials at the state level and officials from local recipients  of Perkins funds—that is, school districts and postsecondary  institutions—that varied by number of CTE students served, amount of  Perkins funding received, and geographic location (urban versus rural).  We also reviewed relevant federal legislation and agency guidance and  interviewed Education officials to obtain information on how states have  implemented the performance measures, how Education has monitored  and supported states in their efforts to implement the performance  measures, and what Education knows about how states are evaluating  their local CTE programs. To analyze how states are evaluating CTE  programs, we reviewed state Perkins plans and annual reports submitted  to Education from the 50 states and the District of Columbia. See appendix  I for detailed information on our surveys and site visits.
We conducted this performance audit from August 2008 to July 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Under Perkins IV, Education Allocates Funds for Career and Technical Education to States in order to Improve Local CTE Programs
paragraphs: The principal source of federal funding for CTE, Perkins IV authorizes  federal grant funds for the enhancement of CTE for secondary and  postsecondary students. In fiscal year 2008, Congress appropriated $1.2  billion for the improvement of local CTE programs. Education’s Office of  Vocational and Adult Education allocates the funds to states, which retain  up to 15 percent of the funds for administration and state leadership of  CTE programs, before passing at least 85 percent of the funds on to local  recipients of funds, such as local school districts and community colleges.  States determine the percentage of funds that will be allocated to the  secondary and postsecondary levels. The majority of funds allocated to the  secondary level are passed on to local recipients based on the school  district’s share of students from families below the poverty level for the  preceding fiscal year. Postsecondary funds are primarily allocated based  on the institution’s share of Pell Grant recipients.
section_title: Perkins IV Established Performance Measures for Secondary and Postsecondary Levels and Requires States and Local Recipients to Report on Program Outcomes
paragraphs: Perkins IV established six student performance measures at the secondary  level and five performance measures at the postsecondary level. These  measures represent a range of student outcomes, such as attainment of  technical skills and placement in employment or further education  following the completion of CTE programs. In addition, the measures  include the nontraditional participation and completion of students from  an underrepresented gender in programs with significant gender  disparities (such as women participating in auto repair), among others  (see tables 1 and 2 for a description of the Perkins IV performance  measures). To ease states’ transition to the new provisions in Perkins IV,  Education permitted states to submit a 1-year transition plan that covered  only the first program year of Perkins IV implementation, 2007-2008.  Accordingly, states were required only to implement and report  performance on two secondary performance measures for the 2007-2008  program year: academic attainment and student graduation rates. These  two measures are based on the same academic attainment and student  graduation rate measures required by Title I of the Elementary and  Secondary Education Act. Beginning in the 2008-2009 program year, states  are required to report on student outcomes for all of the performance  measures. States will report these outcomes to Education in December  2009.
Perkins IV requires states to negotiate specific performance targets with  Education and to annually report their performance to Education. It also  requires local recipients to negotiate performance targets with the states  and to annually report to the state their progress toward meeting these  targets. Perkins IV established additional accountability requirements for  states and local recipients, including actions to address states that do not  meet all of their performance targets. Under Perkins IV, if a state does not  meet at least 90 percent of its targets for one or more of the performance  measures, it is required to develop and implement a program improvement  plan that describes how it will address its failing performance targets.  Prior to Perkins IV, states were only required to develop and implement a  program improvement plan if they failed to meet their targets in all of their  performance measures, not just one measure. States can also face  financial sanctions. For example, Education can withhold all or a portion  of funds if a state does not implement a program improvement plan, show  improvement in meeting its failing performance measure, or meet the  target for the same performance measure for 3 consecutive years. Local  recipients that do not meet at least 90 percent of their performance targets  have the same program improvement requirements as the state and face  similar sanctions from the state. In the event of financial sanctions,  Education is required to use the withheld funds to provide technical  assistance to the state for improving its performance on the measures and  the state is to use funds withheld from local recipients to provide CTE  services and activities to students.
section_title: Education Developed Nonregulatory Guidance to Assist States with Perkins IV
paragraphs: In order to implement the performance measurement requirements of  Perkins IV, states must define which students will be included in the  measures and collect data for each of the performance measures at the  secondary and postsecondary levels. For example, states define the  minimum requirements, such as a certain number of CTE credits that a  student would need to obtain in order to be identified as a student  concentrating in CTE. Education has taken a range of actions to help  states with these activities. For example, in January 2007, Education began  issuing nonregulatory guidance to states to help them develop their  student definitions and data collection approaches for the performance  measures. Education also issued guidance to states on the information  states must include in their state Perkins plans and in the annual reports  that they submit to Education. In the state plans, states must detail how  they intend to implement the performance measures, and in the annual  reports states must describe their progress in meeting the negotiated  performance targets.
section_title: In Addition to Implementing the Perkins Performance Measures, Perkins IV Also Requires States to Annually Evaluate Their Local CTE Programs
paragraphs: In addition to implementing performance measures, states are required to  evaluate programs, services, and activities supported with Perkins funds  and to report to Education in their state plans how they intend to conduct  these evaluations. To meet this requirement, states describe the  approaches, such as the use of state-developed standards, they will use to  evaluate local CTE programs. In addition, Education requires states to  include a description of how they used Perkins funds to evaluate their  local CTE programs in their annual reports.
section_title: States Are Implementing Some Performance Measures Using Different Approaches and Report That the Greatest Challenge Is Collecting Data on Technical Skill Attainment and Student Placement
subsections: section_title: Flexibility in Law and Guidance Allows for Differences in How States Implement Some Performance Measures and Results in Variation in the Student Outcome Data Education Will Collect
paragraphs: A key feature of Perkins IV—to enhance state and local flexibility in  developing, implementing, and improving career and technical  education—allows for considerable variation in how states implement  some performance measures. While Perkins IV was designed to  strengthen accountability for results at the state and local levels, it also  allows states to establish their own accountability systems, including their  own data collection methods for the performance measures. Of the 11  performance measures, the secondary and postsecondary levels have 3  measures in common: technical skill attainment, student placement, and  participation in and completion of nontraditional programs (see fig. 1).  States may also include additional, state-developed performance measures  in their accountability systems. For example, Washington state added  three performance measures—earnings, employer satisfaction, and CTE  student satisfaction—to its accountability system.
Consistent with Perkins IV, Education’s guidance to states also allows for  flexibility. Education issued nonregulatory guidance that proposed  specific definitions that could be adopted by states to develop each of the  secondary and postsecondary performance measures. It also identified  preferred approaches for collecting data for certain measures such as  student technical skill attainment. However, Education noted that in  accordance with Perkins IV, states could propose other definitions and  approaches to collect data for the required performance measures if they  meet the requirements of the law.
We found through our surveys of state CTE directors that states vary  considerably in the extent to which they plan to follow Education’s  guidance—specifically with regard to the technical skill attainment and  secondary school completion measures. As a result, Education will collect  student outcome data that vary across states for the same measures. This  can create challenges for Education to aggregate student outcomes at the  national level. For example, a majority of states reported that they will use  technical assessments—the approach recommended in Education’s  guidance—to measure student attainment of skills at the secondary and  postsecondary levels. These include assessments leading to industry-ba certificates or state licenses. However, a number of states will rely on  other approaches to collect data for the performance measure, including grade poi nt average (GPA), program completion, or other methods (see  table 3).
Officials in the states we visited provided a variety of reasons for their use of alternate methods to measure students’ attainment of technical skills.  For example, postsecondary state officials in California said that a CTE  instructor’s overall evaluation of a student’s technical skill proficiency, in  the form of a final grade, is a better measure of technical skill attainmentthan third-party technical assessments, and can more effectivel program improvement. They questioned the value of technical  assessments, in part because assessments often cannot keep pace with  technology and changing CTE program curricula, such as curricula for  digital animation. A Washington state official told us that the state plans to use program completion to measure technical skills at the postsecondary  level, noting that each postsecondary CTE program incorporates industry- recognized standards into the curriculum. He noted that a national system  se it  of third-party assessments may not be adequate or appropriate, becau would not necessarily incorporate the same standards. Local school  officials in Minnesota said that they will report on CTE course complet view by  for this measure. Because CTE courses undergo curriculum re teachers as well as industry advisors, and align with relevant  postsecondary programs in the area, school officials told us cours completion i attainment.  s sufficient to satisfy the definition of technical skill  Education’s guidance also allows for considerable variation in the types of  technical assessments states can use and when they can administer them.
Most states at the secondary level reported in our survey that they plan to  use industry-developed certificates or credentials most often administere at the end of a program, such as a certificate awarded for an automotive  at the end of a program, such as a certificate awarded for an automotive  technician. At the postsecondary level, states plan to most often rely upon  technician. At the postsecondary level, states plan to most often rely upon  the results of assessments for state lice the results of assessments for state licenses, such as state nursing licenses,  to measure technical skills (see fig. 2). nses, such as state nursing licenses,  to measure technical skills (see fig. 2).
However, we found that while a majority of states plan to use assessment to report to Education, the assessments are not currently in widespread  use. For example, more than half of states at the secondary and  postsecondary levels reported that they plan to use these assessments to  report on few to none of their state-approved CTE programs in the 2008- s  2009 program year. Some states at the secondary level reported  will use a combination of methods—including GPA o r program  completion—to report on technical skill attainment.
We also found that states differ in whether they plan to report student dat on GED credentials, part of the secondary school completion measure.  Thirty states reported through our survey that they do not plan to report  GED data to Education for the 2008-2009 program year, while 18 reported that they would. About one-third of all states cited their ability to access  accurate GED data as a great or very great challenge. For example, state officials we interviewed said states face difficulty tracking the students  that leave secondary education and return, sometimes several years lat to earn a GED credential. An Education official said that the agency is  aware of the challenges and limitations states face in collecting GED data  ed to provide technical assistance to states on  and that the agency may ne ways to collect these data.
section_title: States Face the Most Challenge Collecting Data on Student Technical Skill Attainment and Placement Measures because of Cost and Data Concerns
paragraphs: States reported in our surveys that they face the most difficulty in  collecting student data for two of the performance measures: technical  skill attainment and student placement (see fig. 3 and fig. 4). Thirty-eight  states at the secondary level reported that they face great or very great  challenges in collecting data on student technical skill attainment, while,  similarly, 14 said they face challenges collecting data on student  placement. The results were similar at the postsecondary level: 39 states  reported great or very great challenges with the technical skill attainment  measure and 11 cited a similar level of difficulty with student placement.
States reported that the technical skill attainment measure at the  secondary and postsecondary levels was most challenging to implement  because of costs and the states’ ability to collect accurate and complete  student data. Specifically, states reported that the costs of state-developed  assessments and third-party technical assessments—such as those for  industry certifications—are high and often too expensive for many  districts, institutions, or students. Several state CTE directors  commented in our surveys that their Perkins funds are inadequate to pay  for these assessments and additional funds would be necessary to cover  the costs. Another CTE director stated that economically disadvantaged  students cannot afford the cost of assessments. In addition to challenges  due to cost, states are limited in their ability to access accurate and  complete data. For example, a state official said that Washington state  does not have data-sharing agreements with assessment providers to  receive the results of student assessments. As a result, the state will have  to rely largely on students to self-report the results of their assessments,  which raises concerns of data quality. Challenges such as these likely  contribute to some states’ use of other data—such as GPA or program  completion—to collect and report information for this key student  performance measure.
Some states also reported difficulty collecting data on CTE students after  they leave the school system. States at the secondary and postsecondary  levels reported that their greatest challenge with the student placement  measure is collecting data on students that are employed out of state. As  we previously reported, state wage records, such as Unemployment  Insurance data, track employment-related outcomes only within a state,  not across states. A number of states commented in our surveys on  challenges in tracking students because of the lack of data sharing across  states. We found that states face challenges in tracking students  employed out of state regardless of the method they most commonly use  to collect student placement data. Thirty-eight states at the secondary  level will use student survey data from the state, school district, or a third  party to track student placement and report to Education, while 41 states  at the postsecondary level will rely on state wage record data, despite  potential gaps in student data (see fig. 5).  g. 5).
States also cited other challenges in obtaining data on student placement  for CTE students. At the secondary level, states reported that their next  greatest challenge is linking secondary and postsecondary data systems in  order to track students that pursue higher education after graduation. To  help overcome this challenge, Minnesota—one of the states we visited— recently passed legislation to allow data sharing between the secondary  and postsecondary levels. Our survey also found that states’ next greatest  challenge at the postsecondary level was collecting data on students who  are self-employed after leaving postsecondary institutions. Community  college officials in California said that while they rely on Unemployment  Insurance wage record data, the data are incomplete and do not capture  information on the self-employed, a group that is important for the  measurement of CTE outcomes at the postsecondary level.
States face similar challenges of cost and ability to access accurate data  for the remaining performance measures. For example, states at the  secondary level commented on data challenges for the academic  attainment and student graduation rate measures. Specifically, several  states cited problems in obtaining data from separate student data systems  containing academic and CTE information. This can be particularly  challenging for states that are trying to match student data from different  systems in order to track required CTE student outcomes. In addition, at  the postsecondary level, states cited challenges in tracking student  retention in postsecondary education or student transfer to a  baccalaureate degree program. In particular, accessing student data from  out-of-state and private institutions and the high costs required to track  these students were identified as the most challenging issues. States most  often reported that they will track these students through their state  postsecondary data systems.
section_title: Education Uses Risk- Based Monitoring to Ensure Implementation of the Performance Measures and Supports States through Technical Assistance and Guidance
subsections: section_title: Education Uses Risk- Based Monitoring and Reviews State Annual Reports to Ensure Implementation of the Performance Measures
paragraphs: As we have previously reported, effective monitoring is a critical  component of grant management. The Domestic Working Group’s  suggested grant practices state that financial and performance monitoring  is important to ensure accountability and attainment of performance  goals. Additionally, GAO recently reported on the importance of using a  risk-based strategy to monitor grants and noted that it is important to  identify, prioritize, and manage potential at-risk grant recipients, given the  large number of grants awarded by federal agencies. Education’s  approach to monitoring Perkins is consistent with these suggested grant  practices. According to its Perkins monitoring plan, Education selects  which states to monitor based on a combination of risk factors and  monitors states in two ways: through on-site visits and off-site reviews of  state plans, budgets, and annual reports for those states not visited in a  given year. To determine which states it will visit for on-site monitoring,  Education uses a combination of risk factors, such as grant award size,  issues identified through reviews of state Perkins plans, and time elapsed  since Education’s last monitoring visit. Education officials told us that  their goal is to visit each state at least once every 5 years and reported that  they have conducted on-site monitoring visits to 28 states since 2006.  Education officials also told us that the same monitoring team performs  both on-site and off-site reviews, which officials said helps to ensure  continuity between the reviews. Furthermore, when conducting the off- site reviews, the monitoring team looks for trends in state data and for any  problems with state data validity and reliability. The team uses a checklist  to match performance data to the data states report in their required  annual reports.
According to Education’s inventory of open monitoring findings, as of May  2009, 9 of the 28 open findings were related to accountability and states  failing to submit complete or reliable data. For example, in a February  2008 monitoring visit, Education found that the monitored state’s data  system had design limitations that affected the state’s ability to collect and  assess data on career and technical education students. Specifically,  Education found that the various data systems across the local secondary  and postsecondary levels did not share data with the state-level CTE data  system. This data-sharing issue raised doubts about the validity and  reliability of the state’s Perkins data. Education tracks the findings from  each state’s monitoring visit in a database and reviews the findings in an  internal report that is updated monthly. Additionally, if a state has open  findings, the state may be required to report corrective actions to  Education in the state’s annual report. Officials told us that the amount of  time it takes for a state to close out a finding depends upon the nature of  the finding. For example, a finding related to accountability may take up to  a year to resolve because a state may have to undertake extensive actions  to address the deficiency. Education officials reported that their  monitoring process emphasizes program improvement rather than  focusing solely on compliance issues and that they use monitoring findings  to guide the technical assistance they provide to the states.
To evaluate its monitoring process, Education sends a survey to the CTE  directors of states that were monitored that year and asks them to rate the  format and content of Education’s Perkins monitoring process. For  example, the survey asks states to report on whether they received  sufficient notice that the site visit was going to take place, whether the  monitoring team provided on-site technical assistance, and whether the  state received a written report within a reasonable time frame following  the visit. We reviewed Education’s summaries of the state surveys and  found that for 2004 and 2005, the results of these surveys were generally  positive. For example, in a 2004 monitoring evaluation report, the 10 states  that were surveyed all reported that they had received sufficient notice  about the monitoring visit and that Education staff provided on-site  technical assistance. According to our survey of secondary-level CTE  directors, about half of states have had a monitoring visit within the last 3  years, and almost all of the states whose monitoring visit resulted in  findings said that Education worked with them to ensure that the findings  were addressed.
section_title: Education Supports States by Providing Technical Assistance and Guidance
paragraphs: Education provides states with guidance, technical assistance, and a  variety of other resources and is taking actions to meet states’ need for  additional help. Since Perkins IV was enacted, Education has issued  guidance to states on topics such as instructions for developing the state  Perkins plans and annual reports, as well as guidance related to the  performance measures. For example, Education’s guidance provides  clarification to states on what information each state has to submit to  Education before it can receive its grant award for the next program year,  such as any revisions a state wants to make to its definitions of student  populations, measurement approaches, and proposed performance levels  for each of the measures. Some of the guidance resulted from Education’s  collaborative efforts with states. For example, Education’s guidance to  states on student definitions and measurement approaches incorporated  the input given by state CTE directors during national conference calls  between states and Education. Other guidance addresses questions raised  by states during national Perkins IV meetings, such as how a state should  negotiate performance levels with its local recipients.
In addition to guidance, Education offers states technical assistance from  Education staff—called Regional Accountability Specialists—and through  a private contractor. Education officials told us that each Regional  Accountability Specialist works with a specific group of states to negotiate  state data collection approaches for the performance measures. In  addition, each specialist maintains regular contact with his or her states  throughout the year and provides assistance on other issues, such as  reporting requirements and program improvement plans. In addition to the  Regional Accountability Specialists, Education also provides states with  technical assistance by using MPR Associates, a private contractor. MPR  Associates provides technical assistance that generally includes on-site  visits and follow-up discussions to help states improve their CTE programs  and facilitate data collection for the performance measures. For example,  MPR Associates met with one state to assist with developing population  definitions and measurement approaches that aligned with Education’s  guidance and helped another state with developing a plan for  implementing secondary and postsecondary technical skill assessments.  After providing technical assistance to a state, MPR Associates develops a  summary report, which is then published on Education’s information- sharing Web site, the Peer Collaborative Resource Network. Education  also offers states a range of other resources, including data work groups  and monthly conference calls. See table 4 for a description of the various  ways in which Education provides assistance to states.
Most states reported that the assistance provided by Education has helped  them implement the performance measures, but that more assistance in  the area of technical skill attainment would be helpful. In our survey,  states responded positively about their Regional Accountability Specialist  and all of Education’s other forms of assistance, including the Data Quality  Institute and the Next Steps Work Group. States also reported that more  nonregulatory guidance and more individual technical assistance would  improve their ability to implement the performance measures. Of the  states that provided additional information on the areas in which they  wanted assistance, 4 of 16 states at the secondary level and 9 of 20 states  at the postsecondary level said that they wanted assistance on the  technical skill attainment measure. Specifically, some of the states that  provided additional information said they would like Education to clarify  its expectations for this measure, to provide states with a library of  technical assessments, and to provide state-specific assistance with  developing low-cost, effective technical assessments. States also raised  issues regarding the performance measures and their state’s data  collection challenges. For example, one state reported that it was unsure  how a state should report technical skill attainment as a single measure for  over 400 distinct CTE programs.
We found that Education officials were aware of states’ need for additional  assistance and that Education has taken some actions to address these  needs, particularly in the area of technical assessments. For example,  through the Next Steps Work Group, Education facilitated a technical  skills attainment subgroup that is led by state officials and a national  research organization. The subgroup reviewed state Perkins plans and  annual reports for technical skill assessment strategies that states  reported to Education for consideration in upcoming guidance. Education  also collaborated with MPR Associates to conduct a study on the  feasibility of a national technical assessment clearinghouse and test item  bank. The study, conducted with several CTE research organizations and  state-level consortia, proposed national clearinghouse models for  technical assessments. MPR Associates concluded that clarifying  ownership, such as who is responsible for the development and  management of the system, and securing start-up funding were the two  most likely impediments to creating such a system. The report was  presented to states at the October 2008 Data Quality Institute seminar, and  Education officials reported that they are working with organizations such  as the National Association for State Directors of Career and Technical  Education Consortium and the Council of Chief State School Officers to  implement next steps.
In addition to helping states with the technical skill attainment measure,  Education also has taken actions to improve its information-sharing Web  site, the Peer Collaborative Resource Network. Specifically, a Next Steps  Work Group subcommittee surveyed states for suggested ways to improve  the Web site and reported that states wanted to see the information on the  site kept more current. The subcommittee reported in December 2008 that  Education would use the survey results to develop a work plan to update  the Web site. In May 2009, Education officials reported that they had  implemented the work plan and were piloting the revamped site with  selected state CTE directors before the department finalizes and formally  launches the site.
section_title: Education Relies on the Performance Measures to Gauge the Success of State CTE Programs
paragraphs: State performance measures are the primary source of data available to  Education for determining the effectiveness of CTE programs, and  Education relies on student outcomes reported through these measures to  gauge the success of states’ programs. While Perkins IV requires states to  evaluate their programs supported with Perkins funds, it only requires  states to report to Education—through their state plans—how they intend  to evaluate the effectiveness of their CTE programs. It does not require  states to report on the findings of their evaluations and does not provide  any specific guidance on how states should evaluate their programs.
Because only 2 of 11 measures have been implemented and reported on  thus far, Education has little information to date on program outcomes. In  program year 2007-2008, Education required states to implement and  report only the academic skill attainment and graduation rate measures.  States are required to provide Education with outcome data for the  remaining 9 secondary and postsecondary measures in December 2009.  According to Education’s annual report for program year 2007-2008, 43  states met their targets for the academic attainment in reading/language  arts measure, 38 states met their targets for the academic attainment in  mathematics measure, and 46 states met their targets for the graduation  rate measure.
We analyzed the state plans of all 50 states and the District of Columbia  and found that, as required by Perkins IV, states provide a description to  Education on how they are evaluating their CTE programs. The type of  information that states provided varied. For example, some states  described the databases they use to capture key data and others explained  how they use state-developed performance measures to evaluate their  programs. Perkins IV does not require that states include information on  what their evaluations may have found in terms of the success of a  program. In our surveys of state CTE directors, nearly half of states (23  states at the secondary level and 21 states at the postsecondary level)  responded that they have conducted or sponsored a study, in the past 5  years, to examine the effectiveness of their CTE programs. In response to  these survey results, we collected seven studies that states identified as  evaluations of their program effectiveness. We developed an instrument  for evaluating these studies and determined the type of evaluation and  methodology used by states in these studies. We determined that four of  the studies were outcome evaluations and the remaining three studies  were not outcome, impact, or process evaluations. For example, one  state found in its outcome evaluation that high school graduates who  completed a CTE program of study entered postsecondary institutions  directly after high school at the same rate as all graduates.
section_title: Concluding Observations
paragraphs: Perkins IV provides states with considerable flexibility in how they  implement the required performance measures and how they evaluate the  effectiveness of their CTE programs. While this flexibility enables states to  structure and evaluate their programs in ways that work best for them, it  may hinder Education’s ability to gain a broader perspective on the  success of state CTE programs. Specifically, differences in how states  collect data for some performance measures may challenge Education’s  ability to aggregate student outcomes at a national level and compare  student outcomes on a state-by-state basis. Further, Education is limited in  what it knows about the effectiveness of state CTE programs, beyond  what states report through the performance measures. Perkins only  requires that states report on how they are evaluating their programs, and  does not provide any guidance on how states should evaluate their  programs or require that states report on the outcomes of their  evaluations. Education is working with states to help them overcome  challenges they face in collecting and reporting student outcomes, and  over time, states may collect more consistent data for measures such as  technical skill attainment. As states become more adept at implementing  the Perkins performance measures, they will be better positioned to  conduct more rigorous evaluations of their CTE programs. However this  information may not be reported to Education. If policymakers are  interested in obtaining information on state evaluations, they will need to  weigh the benefits of Education obtaining this information with the  burden of additional reporting requirements.
section_title: Agency Comments
paragraphs: We provided a draft of this report and the electronic supplement to the  Department of Education for review and comment. Education provided  technical comments on the report, which we incorporated as appropriate.  Education had no comments on the electronic supplement.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Education, and other interested parties. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have any questions about the report, please contact me  at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff that made major contributions to this report are  listed in appendix II.
section_title: Appendix I: Scope and Methodology
paragraphs: To obtain national-level information on states’ implementation of Perkins  IV, we designed and administered two Web-based surveys, at the  secondary and postsecondary levels, to state directors of career and  technical education (CTE) in the 50 states and the District of Columbia.  The surveys were conducted between January and April 2009, with 100  percent of state CTE directors responding to each survey. The surveys  included questions about the types of data states collect for the student  performance measures and challenges they face; the various kinds of  technical assistance, guidance, and monitoring states received from  Education; and how states evaluate their CTE programs. The surveys and  a more complete tabulation of the results can be viewed at GAO-09-737SP.
Because this was not a sample survey, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  nonsampling errors, such as variations in how respondents interpret  questions and their willingness to offer accurate responses. We took steps  to minimize nonsampling errors, including pretesting draft survey  instruments and using a Web-based administration system. Specifically,  during survey development, we pretested draft instruments with officials  in Minnesota, Washington state, and Vermont in December 2008. We also  conducted expert reviews with officials from the National Association of  State Directors of Career and Technical Education Consortium and MPR  Associates, who provided comments on the survey. In the pretests and  expert reviews, we were generally interested in the clarity of the questions  and the flow and layout of the survey. For example, we wanted to ensure  that terms used in the surveys were clear and known to the respondents,  categories provided in closed-ended questions were complete and  exclusive, and the ordering of survey sections and the questions within  each section were appropriate. On the basis of the pretests and expert  reviews, the Web instruments underwent some revisions. A second step  we took to minimize nonsampling errors was using Web-based surveys. By  allowing respondents to enter their responses directly into an electronic  instrument, this method automatically created a record for each  respondent in a data file and eliminated the need for and the errors  associated with a manual data entry process. When the survey data were  analyzed, a second, independent analyst checked all computer programs  to further minimize error.
While we did not fully validate all of the information that state officials  reported through our surveys, we reviewed the survey responses overall to  determine that they were complete and reasonable. We also validated  select pieces of information by corroborating the information with other  sources. For example, we compared select state responses with  information submitted to Education in state Perkins plans. On the basis of  our checks, we believe our survey data are sufficiently reliable for the  purposes of our work.
subsections: section_title: Site Visits
paragraphs: To better understand Perkins IV implementation at the state and local  levels, we conducted site visits to three states—California, Minnesota, and  Washington state—between September 2008 and February 2009. In each  state we spoke with secondary and postsecondary officials at the state  level with CTE and Perkins responsibilities. We also interviewed officials  from local recipients of Perkins funds—that is, school districts and  postsecondary institutions. Through our interviews with state and local  officials, we collected information on efforts to implement the Perkins  performance measures and uses of Perkins funding, experiences with  Education’s monitoring and technical assistance, and methods for CTE  program evaluation. The states we selected represent variation across  characteristics such as the type of state agency (i.e., state educational  agencies or state college and university systems) eligible to receive  Perkins funds, the amount of Perkins IV funds received in fiscal year 2008,  and type of approach used to measure student attainment of technical  skills. The localities selected for site visits provided further variation in  geographic location (urban versus rural), number of CTE students served,  and amount of Perkins funding received.
We conducted this performance audit from August 2008 to July 2009, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Elizabeth Morrison (Assistant  Director), Avani Locke, Robin Nye, Charlotte Gamble, Stephen Steigleder,  Jessica Orr, Jean McSween, Christine San, and Jessica Botsford made key  contributions to this report.