The AHELO Feasibility Study:
Study Results and the Conclusions of theTechnical Advisory Group (TAG)

AHELO 䝣䜱䞊䝆䝡䝸䝔䜱䞉䝇䝍䝕䜱:ㄪᰝ⤖ᯝ䛸ᢏ⾡ㅎၥ䜾䝹䞊䝥(TAG)䛾⤖ㄽ

Peter T. EwellPeter T. Ewell

National Center for Higher Education
Management Systems (NCHEMS)඲⡿㧗➼ᩍ⫱⤒Ⴀ䝅䝇䝔䝮◊✲ᡤ
(National Center for Higher Education
Management Systems, NCHEMS)

NIER Conferenceᅜ❧ᩍ⫱ᨻ⟇◊✲ᡤᩍ⫱ᨵ㠉ᅜ㝿䝅䞁䝫䝆䜴䝮Tokyo, Japanᮾி䠄᪥ᮏ䠅

December 10, 20132013 ᖺ12᭶10᪥

Background and Early Development ofAHELO

AHELO䛾⫼ᬒ䛸ึᮇ䛾ᒎ㛤 Parallel Efforts in PISA and PIACC PISA䛸PIACC䛻䛚䛡䜛ྠᵝ䛾ྲྀ⤌ Expert Meetings in 2007 in Washington, Paris, andKorea 2007ᖺ䚸ᑓ㛛ᐙ఍ྜ䠄䝽䝅䞁䝖䞁䚸䝟䝸䚸㡑ᅜ䠅 ㄪᰝศ㔝: ⤒῭Ꮫ䚸ᕤᏛ䚸୍⯡ⓗᢏ⬟䚸⫼ᬒ᝟ሗ Domains: Economics, Engineering, and Generic
Skills—As Well as a Contextual Dimension
 Decision to Launch a “Feasibility Study”

 䛂䝣䜱䞊䝆䝡䝸䝔䜱䞉䝇䝍䝕䜱䛃䜢ᐇ᪋䛩䜛䛣䛸䛜Ỵᐃ䛥䜜䛯Design of the AHELO Feasibility Study

AHELO 䝣䜱䞊䝆䝡䝸䝔䜱䞉䝇䝍䝕䜱䛾䝕䝄䜲䞁 Seventeen Countries (Jurisdictions) 17䛛ᅜ(ᆅᇦ)

 Six to Ten Institutions per Country ྛᅜ6䡚10䛾㧗➼ᩍ⫱ᶵ㛵䛜ཧຍ Two Hundred Students per Institution who are
Nearing the End of their Academic Programs Ꮫ఩䝥䝻䜾䝷䝮ಟ஢㛫㝿䛾Ꮫ⏕䚸ྛᶵ㛵200ே䜢ᑐ㇟ Ninety Minute to Two Hour Testing Period 䝔䝇䝖᫬㛫䛿90ศ䛛䜙2᫬㛫 Mix of Multiple Choice and Constructed Response ከ⫥㑅ᢥᘧ䞉グ㏙ᘧၥ㢟䜢⤌䜏ྜ䜟䛫䛶ᐇ᪋ Test Items “Spiraled” Across Test Takers

 Ꮫ⏕䛤䛸䛻␗䛺䜛ၥ㢟䝉䝑䝖䛻ゎ⟅䠖䛂䝇䝟䜲䝷䝹໬䛃

The AHELO Generic Skills Assessment

AHELO୍⯡ⓗᢏ⬟䛾ㄪᰝ Administered in Nine Countries 9䛛ᅜ䛷ᐇ᪋ Two Constructed Response Prompts Based on theCollegiate Learning Assessment (CLA) ኱Ꮫ⏕Ꮫ⩦ㄪᰝ (CLA) 䛻ᇶ䛵䛔䛶సᡂ䛥䜜䛯グ㏙ᘧၥ㢟2ၥ

 Multiple-Choice Items Drawn from the AustralianGraduate Skills Assessment ከ⫥㑅ᢥᘧၥ㢟䛿㇦ᕞ༞ᴗ⏕⬟ຊㄪᰝ䛛䜙᥇⏝ ྛᅜ䛻䛚䛔䛶㐺ṇ໬䛚䜘䜃⩻ヂ Contextualized and Translated Under Direction byEach Country
 Scored Locally in Each Country ྛᅜ䛻䛚䛔䛶᥇Ⅼ

Examples of Generic Skills
 Cognitive:୍⯡ⓗᢏ⬟䛸䛿ఱ䛛 ㄆ▱ⓗഃ㠃:• Communications (Written and Oral)• 䝁䝭䝳䝙䜿䞊䝅䝵䞁⬟ຊ䠄➹グ䛚䜘䜃ཱྀ㢌䠅• Critical Thinking and Problem Solving• ᢈุⓗᛮ⪃ຊ䛸ၥ㢟ゎỴ⬟ຊ Non-Cognitive 㠀ㄆ▱ⓗഃ㠃䠖• Empathy and Ethics• ඹឤ䛸೔⌮• Working with Others• ௚⪅䛸༠ྠ䛩䜛• Taking Responsibility• ㈐௵䜢ᢸ䛖

Graduate Attributes: Top Choices forEmployers༞ᴗ⏕䛾ᒓᛶ: 㞠⏝⪅䛜㔜ど䛩䜛䜒䛾Percent Reporting “Very” and “Somewhat” Important:䛂䛸䛶䜒㔜せ䛃䛂䛒䜛⛬ᗘ㔜せ䛃䛸ᅇ⟅䛧䛯㞠⏝⪅䛾ẚ⋡: Ethical Judgment and Integrity – 96%೔⌮ⓗุ᩿ຊ䛸ㄔᐇ䛥– 96%

 Comfortable with Diverse Backgrounds – 98%ከᵝᛶ䛻ᑐ䛩䜛ᐶᐜ䛥– 98%

 Demonstrated Capacity for Professional
Development – 94%ᑓ㛛ⓗ⫋⬟㛤Ⓨ䛾వᆅ– 94%

 Interest in Giving Back to Communities OurCompany Serves – 71%௻ᴗ䝃䞊䝡䝇䜢ᆅᇦ♫఍䛻㑏ඖ䛩䜛䛣䛸䜈䛾ពḧ– 71%

Teaching Generic Skills: CommonApproaches୍⯡ⓗᢏ⬟䛾ᩍᤵ: ㏻ᖖ䛾䜰䝥䝻䞊䝏 A “General Education” Block of Courses 䛂୍⯡ᩍ⫱䛃⛉┠⩌ Embed These Skills in Disciplinary Teaching Ꮫၥศ㔝ู䛾ᩍ⫱䛻䛂୍⯡ⓗᢏ⬟䛃䜢⤌䜏㎸䜐

 Particular Pedagogies: ≉Ṧ䛺ᩍᤵἲ:• Collaboration and Group Work• ༠ㄪᏛ⩦䛸䜾䝹䞊䝥Ꮫ⩦• Internships and Fieldwork• 䜲䞁䝍䞊䞁䝅䝑䝥䛸䝣䜱䞊䝹䝗䝽䞊䜽• Capstone Courses and Projects• ༞ᴗ◊✲䛸䝥䝻䝆䜵䜽䝖Ꮫ⩦• Role of the “Co-Curriculum”• 䛂ṇㄢእ䛃άື䛾ᙺ๭

The AHELO Economics Assessment

AHELO⤒῭Ꮫศ㔝䛾ㄪᰝ Administered in Seven Countries 7䛛ᅜ䛷ᐇ᪋ Constructed Response Prompts Based on Tuning
and UK Subject Benchmarks in Economics 䝏䝳䞊䝙䞁䜾䛚䜘䜃ⱥᅜศ㔝ูᏛ఩Ỉ‽ᇶᶆ䠄⤒῭Ꮫ䠅䛻ᇶ䛵䛔䛶సᡂ䛥䜜䛯グ㏙ᘧၥ㢟 Multiple-Choice Items Based on the ETS Graduate
Record Examination in Economics

 ETS䛜ᐇ᪋䛩䜛GREヨ㦂䠄⤒῭Ꮫ䠅䛻ᇶ䛵䛔䛶సᡂ䛥䜜䛯ከ⫥㑅ᢥᘧၥ㢟 Focused on Student Ability to “Think AboveContent”

 Ꮫ⏕䛾䛂⪃䛘䜛䛃ຊ䛻↔Ⅼ䜢ᙜ䛶䛯ၥ㢟 ྛᅜ䛻䛚䛔䛶᥇Ⅼ Scored Locally in Each Country

The AHELO Engineering Assessment
 Administered in Nine Countries
 Confined to Civil Engineering Constructed Response Prompts Based on TuningAHELOᕤᏛศ㔝䛾ㄪᰝ 9䛛ᅜ䛷ᐇ᪋ ᅵᮌᕤᏛ䛻㝈ᐃ 䝏䝳䞊䝙䞁䜾䛻ᇶ䛵䛔䛶సᡂ䛥䜜䛯グ㏙ᘧၥ㢟 Multiple-Choice Items Based on a JapaneseEngineering Licensure Examination ᪥ᮏ䛾ᅵᮌᏛ఍䛾ㄆᐃᅵᮌᢏ⾡⪅㈨᱁ヨ㦂䛚䜘䜃᪥ᮏᢏ⾡ኈ఍䛾ᢏ⾡ኈ➨୍ḟヨ㦂䛻ᇶ䛵䛔䛶సᡂ䛥䜜䛯ከ⫥㑅ᢥᘧၥ㢟 Focused on Student Ability to “Think Like anEngineer” Ꮫ⏕䛾䛂ᢏ⾡⪅䛾䜘䛖䛻⪃䛘䜛䛃⬟ຊ䛻↔Ⅼ䜢ᙜ䛶䛯ၥ㢟 Scored Locally in Each Country ྛᅜ䛻䛚䛔䛶᥇ⅬThe AHELO Contextual Dimension
 Designed to Gather Background Information inTerms of Which to Explain (and Adjust) CrossCountry Variation

AHELO⫼ᬒ᝟ሗㄪᰝ 䝔䝇䝖ᚓⅬ䛻䛚䛡䜛ྛᅜ㛫䛾ᕪ␗䜢ㄝ᫂䠄ㄪᩚ䠅䛩䜛䛯䜑䛾⫼ᬒ᝟ሗ䜢཰㞟䛩䜛䛣䛸䜢┠ⓗ䛸䛧䛶タィ ⫼ᬒ᝟ሗㄪᰝ䛾せ⣲ Elements of the Contextual Dimension• Ꮫ⏕䠄ཷ㦂⪅䠅ㄪᰝ• Student (Test Taker) Survey• ኱Ꮫᩍဨㄪᰝ• Faculty Survey• ᶵ㛵䛾⫼ᬒ᝟ሗㄪᰝ
• Institutional Background Survey• ྛᅜ䛻㛵䛩䜛⫼ᬒ᝟ሗ• Country Background Information

Some Findingsㄪᰝ⤖ᯝ All Assessments Performed at or Above Expected
Levels of Reliability 䛹䛾ศ㔝䛻䛚䛡䜛ㄪᰝ䛷䜒䚸ᮇᚅ䛥䜜䜛Ỉ‽௨ୖ䛾ಙ㢗ᛶ䛜☜ㄆ䛥䜜䛯 Constructed Response Items Less Reliable thanMultiple-Choice Items [as Expected]

 グ㏙ᘧၥ㢟䛿ከ⫥㑅ᢥᘧၥ㢟䜘䜚䜒䚸ಙ㢗ᛶ䛜ప䛛䛳䛯 [ண᝿㏻䜚] Constructed Response Items Clearly Tapped an
Independent Dimension of Student Knowledge

 グ㏙ᘧၥ㢟䛿᫂䜙䛛䛻Ꮫ⏕䛾▱㆑䛾⊂❧䛧䛯ഃ㠃䜢ᤊ䛘䛶䛔䛯 Some Evidence of Convergent Validity for AllAssessments 䛹䛾ศ㔝䛻䛚䛡䜛ㄪᰝ䛷䜒䚸୍ᐃ⛬ᗘ䛾཰᮰ⓗጇᙜᛶ䛜☜ㄆ䛥䜜䛯The Technical Advisory Group (TAG) Charged with Reviewing the Technical Adequacyof All Aspects of the AHELO Feasibility Study:ᢏ⾡ㅎၥ䜾䝹䞊䝥(TAG) 㻭㻴㻱㻸㻻䝣䜱䞊䝆䝡䝸䝔䜱䞉䝇䝍䝕䜱䛾඲ᒁ㠃䛻䛚䛔䛶䚸άື䛾㐺ษ䛥䜢ᢏ⾡ⓗほⅬ䛛䜙ᑂᰝ䛩䜛ᙺ๭䜢ᢸ䛳䛯㻦㻌• Assessment Frameworks and Instruments• Sampling and Test Administration Procedures• Analysis and Reporting Charged with Making Final Recommendations onFeasibility and the Future Conduct of the AHELOMain Study•ᐃ䛩䜛⬟ຊ䛾ᯟ⤌䛸䝔䝇䝖ၥ㢟㻌• 䝃䞁䝥䝹ᢳฟ䛸䝔䝇䝖ᐇ᪋䛾ᡭ㡰㻌• ศᯒ䛸ሗ࿌㻌• 㻭㻴㻱㻸㻻ㄪᰝ䛾ᐇ᪋ྍ⬟ᛶ䚸䛚䜘䜃ᑗ᮶ⓗ䛻ᮏㄪᰝ䜢ᐇ᪋䛩䜛䛛䛹䛖䛛䛻ಀ䜛᭱⤊ᥦゝ㻌The TAG’s View: What Went Well

TAG䛾ぢゎ:䛖䜎䛟䛔䛳䛯Ⅼ

 Assessment Administration ㄪᰝ䛾㐠Ⴀ Technical Aspects of the Data Analysis

 䝕䞊䝍ศᯒ䛾ᢏ⾡ⓗഃ㠃 Instrument Design for Purpose-Built Instruments

 AHELO䛾䛯䜑䛻సᡂ䛥䜜䛯䝔䝇䝖䛾タィ Overall Coordination ඲య䛾ㄪᩚ

The TAG’s View: What Did Not Go SoWell

TAG䛾ぢゎ: 䛒䜎䜚䛖䜎䛟䛔䛛䛺䛛䛳䛯Ⅼ

 Resources and Time ㈈※䛸᫬㛫 Uneven Student Cooperation Rates Ꮫ⏕䛾ཧຍ⋡䛾䜀䜙䛴䛝 Contextualization and Difficulty of ConstructedResponse Tasks

 グ㏙ᘧၥ㢟䛾㐺ṇ໬䛸㞴᫆ᗘ䛾タᐃ ዎ⣙ୖ䛾ྲྀỴ䜑 Contractual Arrangements

Lessons for the AHELO Main Study More Opportunities for Stakeholder Participation inAssessment Design and Analysis
 Full-Scale Try-Out of Instruments andAdministration Procedures

AHELOᮏㄪᰝ䛻ྥ䛡䛯ᩍカ ㄪᰝタィ䛸ศᯒ䛾㐣⛬䛷䚸䝇䝔䞊䜽䝩䝹䝎䞊䛾ཧຍ䛩䜛ᶵ఍䜢ᣑ඘䛩䜛ᚲせ䛜䛒䜛 ᮏㄪᰝ䛸ྠ➼䛾つᶍ䛷䚸䝔䝇䝖ၥ㢟䛸ᐇ᪋䛾ᡭ㡰䜢ヨ⾜䛩䜛ᚲせ䛜䛒䜛

 More Up-Front Information on Costs and Benefits ㈝⏝ᑐຠᯝ䛻㛵䛩䜛᝟ሗ䜢䚸ึᮇẁ㝵䛛䜙ᥦ౪䛩䜛ᚲせ䛜䛒䜛 Development of Country/Institution “ReadinessCriteria” to Assist Decisions to Participate ᅜ䜔኱Ꮫ䛜ཧຍ䛾᭷↓䜢Ỵ᩿䛩䜛㝿䛻ཧ↷䛩䜛䛣䛸䛾䛷䛝䜛䛂‽ഛ≧ἣ䛻ಀ䜛ᇶ‽䛃䜢㛤Ⓨ䛩䜛ᚲせ䛜䛒䜛

 More Work on the Assessment of Generic Skills ୍⯡ⓗᢏ⬟䛾ㄪᰝ䛻䛴䛔䛶䛿䚸᳨ウ䜢῝䜑䜛ᚲせ䛜䛒䜛Lessons for the AHELO Main Study(Continued)

AHELOᮏㄪᰝ䛻ྥ䛡䛯ᩍカ䠄⥆䛝䠅 Continue Mix of Item Types (Multiple-Choice andConstructed Response) ၥ㢟ᙧᘧ䜢⤌䜏ྜ䜟䛫䜛䠄ከ⫥㑅ᢥᘧ䛚䜘䜃グ㏙ᘧၥ㢟䠅᪉㔪䛿⥔ᣢ䛩䜛 Use Census or Probability Sample for AllInstitutions 䛹䛾ᶵ㛵䛻䛚䛔䛶䜒䚸䝃䞁䝥䝹䛾☜⋡ᢳฟ䜎䛯䛿඲ᩘㄪᰝ䜢ᐇ᪋䛩䜛 Better Integrate AHELO into International
Discussions of Quality in the Scholarly Community

 AHELO䜢㉁䛻㛵䛩䜛Ꮫ⾡ⓗ䞉ᅜ㝿ⓗ䛺㆟ㄽ䛾୰䛻఩⨨䛵䛡䜛 All Will Require More Time and Adequate
Resources 䛹䛾ㄢ㢟䛻ᑐฎ䛩䜛䛻䜒䚸䜘䜚ከ䛟䛾᫬㛫䛸༑ศ䛺㈨※䛜ᚲせMoving Forward?

௒ᚋ䛻ྥ䛡䛶

 OECD Has Concluded that AHELO is Feasible
and is Planning to Move Forward

 OECD䛿AHELO䛜ᐇ᪋ྍ⬟䛷䛒䜛䛸⤖ㄽ䛵䛡䚸ḟ䛾ẁ㝵䛻㐍䜐䛣䛸䜢ィ⏬䛧䛶䛔䜛 Country and Institutional Decisions About
Participating will Depend on Individual Decisions
About Costs and Benefits ᅜ䜔ᶵ㛵䛜ཧຍ䛩䜛䛛䛹䛖䛛䛿䚸㈝⏝ᑐຠᯝ䛻㛵䛩䜛ಶ䚻䛾ุ᩿䛻ጤ䛽䜙䜜䜛 Cost Side Known, Benefits Not Yet Assured ㈝⏝䛻䛴䛔䛶䛿᫂䜙䛛䛻䛺䛳䛶䛔䜛䛜䚸ຠᯝ䛻䛴䛔䛶䛿ᮍ䛰᫂䜙䛛䛷䛿䛺䛔 We Will Know within a Year Whether and How
AHELO Will Move Forward

 AHELO䛜ḟ䛾ẁ㝵䛻㐍䜐䛛䛹䛖䛛䚸䛹䛾䜘䛖䛺ᙧ䛷㐍䜐䛛䛿䚸1ᖺ௨ෆ䛻ุ᫂䛩䜛䛰䜝䛖