Establishing a System to Evaluate Assessments of Student Occupational Skill Attainment

The state of Pennsylvania has been engaged in training students and assessing their Occupational Skills attainment for many decades (Kapes, 2001). Pennsylvania Department of Education (PDE) established a system of recognizing high achievement through a Skill Certificate Program which utilized 14 national and local tests. Initially the Commonwealth established a Pass/Fail decision process for one such test outlined in a report entitled: An Evaluation of Pennsylvania Occupational Competency Written Exams Administered During 1975-78 (Kapes and Funk, 1978). However there was no system in place for evaluating and approving tests. The project in this document outlines the process for establishing and implementing a system for evaluating potential tests that might be used to measure student occupational skill attainment and determining jobreadiness. Online Journal of Workforce Education and Development Volume II, Issue 4 – Fall 2007 3 Introduction For many years, Pennsylvania has been engaged in training and qualifying young people for employment in industry. As far back as 1984, Governor Richard Thornburgh proposed changes in his initiative “Turning the Tide: An Agenda for Excellence in Pennsylvania Public Education. ” (Commonwealth of Pennsylvania, Department of Education, 1983) This was in response to a national report to the Nation and the Secretary of Education, United States Department of Education by the National Commission on Excellence in Education, (April 1983) that found that the middle group of students was not adequately served by the educational system. In response to that plan, Jerry Olsen, State Director of Vocational Education, established a committee to review occupational testing available for use in vocational programs. The committee reviewed four or five testing programs in other states. Ohio’s testing program lacked security. The V-TECS had an open system item bank where teachers would select the items they wanted. It was determined that the National Occupational Competency Testing Institute (NOCTI) was the best alternative although they had only a limited number of tests available. At the cost of $250,000 the Bureau contracted NOCTI to develop a bank of items for 40 tests that coincided with the state’s program areas. The contract included administering, scoring and reporting the test results in the form of national norms for the 40 Student Occupational Competency Achievement Tests (SOCATS). The Pennsylvania Skills Certificate Program These tests were chosen because, among other qualities, they had three components. There was a cognitive portion of 80 multiple-choice items, a written portion with 150 to 200 multiple-choice items, and a performance hands-on portion of 3 to 10 tasks evaluated by an industry practitioner. The SOCAT tests were developed to reflect the program description as written by the National Center for Educational Statistics in a Classification of Instructional Programs (CIP). However, for the Pennsylvania Skills Certificate (PSC), Pennsylvania Department of Education program descriptions were also used to ensure that tests were relevant to the programs being offered in Pennsylvania. Because they were norm-referenced tests, individuals who performed at or above the national norm were awarded the PSC in recognition of high achievement. The seven purposes of the PSC program were to: 1. Provide a standardized method in all areas of “vocational/technical” education to determine the level of occupational competence of students who complete programs at the secondary level; 2. Provide a testing system based upon competencies that have been identified by workers in the respective occupations as essential for the entry-level worker; 3. Implement a two-part system of student assessment wherein the manipulative skills of the occupation are tested in addition to the theoretical concepts that are normally covered in competency evaluation; 4. Provide standards whereby teachers and administrators in career and technical education can compare the progress of a whole class or of an individual student with other students who have been enrolled in similar programs. Statistics are available by individual states and by the nation as a whole; Online Journal of Workforce Education and Development Volume II, Issue 4 – Fall 2007 4 5. Assist local educators in the interpretation of test results that can be used to improve curriculum and instructional programs; 6. Establish employer recognition of the PSC standard. Since personnel from the various occupations have been involved in the identification of competencies upon which the tests were constructed, it is important that each local teacher and administrator recognize the value of the SOCAT score when exploring placement with the prospective employer. SOCAT and the PSC were designed for job placement; and 7. Support and conduct research appropriate to substantiate the validity and reliability of the SOCAT tests and career and technical training. (PDE, 1987) When the Carl D. Perkins Vocational Education Act (PL 98-524) was passed, the Pennsylvania Council on Vocational Education (PACVE) was required to advise the State Board on policies that the board should pursue to strengthen vocational education in the state. The council made three recommendations that (a) the Board and the Department of Education expand the standardized process for measuring secondary vocational education occupational competencies as provided in the Pennsylvania Skills Certificate Program; (b) the Pennsylvania General Assembly approve the request of the governor to increase the reimbursement for adult vocational education programs offered in secondary education institutions as well as support for adult vocational-technical education through reimbursement of noncredit vocational Full-Time Equivalents at community agencies; and (c) the State Board and the Department of Education undertake an evaluation of the postsecondary technical education system to determine if there was a need for additional postsecondary technical education programs and services (Pennsylvania Council on Vocational Education, 1991). A review of the various states’ measures and standards of student occupational skill attainment indicated that every state had implemented some form of assessment. However there was no uniformity in developing measurement approaches that could be characterized as efficient or comprehensive. Some measured skill attainment using checklists that an instructor maintained; and at the end of the program the instructor would create a list of students who had demonstrated mastery of the required amount skills. Some states merely maintained an attendance record to determine whether a student had invested enough seat-time as an indicator of skill attainment. Others relied on locally developed and state approved tests that were presented as valid and reliable measures. Yet others utilized off-the-shelf tests of employability skills. Only a few states had invested efforts in choosing appropriate tests that were aligned with recognized industry standards. One state elected to develop tests based on the state programs. Statement of the Problem Under Perkins Act’s accountability all students who completed a career and technical education program were expected to attain the knowledge and skills that meet program identified as industry validated career and technical skill standards. This attainment was measured as the percentage of career and technical education program completers who achieve a level of competency at or above the national norm on the NOCTI Job-Ready Assessment. At first Job-Ready competency was determined on the basis of students passing the state’s Nurse Aide Training and Competency Evaluation Online Journal of Workforce Education and Development Volume II, Issue 4 – Fall 2007 5 Program test, or on a set of three out of eight Automotive Service Excellence tests. Later that number was increased to 13 other tests that could be administered in lieu of the NOCTI for the purpose of awarding the Pennsylvania Skills Certificate. The PSC was awarded through any of the following five ways (Commonwealth of Pennsylvania, Department of Education, 2007 Guide). First, by default, performance at or above the advanced level on the criterion-referenced NOCTI tests’ written and performance components. All NOCTI tests are routinely benchmarked by teaches and industry representatives using the Nedelsky method to establish cut scores for advanced, competent, basic levels. Second, passing state boards’ or national licensure and certification tests. This included the boards of cosmetology, licensed practical nursing, and nurse aide tests. Third, passing a national industry-credentialing test such as American Welding Society (AWS), Industry Competency Exam (ICE), the American Culinary Federation (ACF), the American Hotel and Lodging Association (AHLA), and Electronic Technician Associate (ETA). Fourth, passing selected bundles of student endof-program tests. This included the Computer Technology Industry Association (CompTIA), the National Automotive Technicians Education Foundation (NATEF), the National Institute for Automotive Service Excellence (AYES) and the corresponding Automotive Youth Educational Systems (AYES), and the National Institute for Metalworking Skills (NIMS) where four core tests were necessary to earn the certificate. Finally, one locally developed test by a national chapter was surprisingly allowed as an alternative to the NOCTI. There was no clear standard system for accepting or rejecting some tests and not others. Some educators complained that they were being held to a much higher standard of performance than on the NOCTI. Others wanted to utilize any industry test regardless of depth or other considerations. Pennsylvania also failed to meet its accountability obligations to report complete data on the Consolidated Annual Report (CAR) the number of program completers who performe