National Institute for Educational Policy ResearchInternational Symposium on Educational Reform 2013
Tuesday, December 10, 2013 13:00 17:30

Informing Universities
for Educational Improvement
The AHELO Feasibility Study Experience inJapan, Canada, and Australia

AHELO

Satoko Fukahori
National Institute for Educational Policy ResearchOECD-AHELO State of ProgressOECD-AHELO• OECD-AHELO• OECD-AHELO: An international test that aims to measure what
students in higher education know and can do upon graduation.
• OECD-AHELO Feasibility Study– Is the assessment scientifically possible?

• OECD-AHELO–• Measuring what?
Defining competence frameworks• How
Defining measurable learning outcomes that demonstrate masteryof a given competence, and developing instruments that measure attainment of
those learning outcomes.
• Scoring
Defining through scoring rubrics the scope and level of learning
outcomes to be attained, and sharing an understanding of the framework.

•••–– Is the assessment practically possible?

•• Getting universities and students to participate.

–– Can we verify the reliability and validity of the instruments?

•Satoko Fukahori• The conclusion of the Technical Advisory Group.
2013.12.10 NIER Symposium

Satoko Fukahori2013.12.10 NIER SymposiumOECD-AHELO

If there is a Future AHELO-For whom and for what?-• Who will benefit from a future AHELO andhow?

•– How will the results be analyzed, and how will
they be reported?

––In order for a future AHELO to be successful,we will need to share a clear vision on“for whom and for what”
Satoko Fukahori

2013.12.10 NIER Symposium

AHELO-FSSatoko Fukahori

2013.12.10 NIER Symposium

The AHELO-FS Research Design• A convenience sample of universities•– The university sample does not represent the country’s HE system.

–• The “university population” is not self evident – what is a university?
• Randomly sampling a university from the university population, and gaining their participation
at a high rate will not be easy.

••– AHELO-FS was not designed to compare countries.

• A random sample of students–– The student sample represents the university’s/program’s student population.
– Students responded to different sets of items based on the item response
theory model.

•––• Information such as the difficulty level and student distribution of each item are necessary inorder to standardize and make comparable student test scores. Since such information is not
yet available, it is not at this point possible to conduct a robust analysis with students as theunit of analysis.

•• The unit of analysis is the “university.”•– The purpose of AHELO is to provide institutions with information that will help
them improve their education.

–Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER SymposiumFor the Panel Discussion:
What kinds of information can an international
Assessment of Higher Education Learning Outcomes
provide that will help universities improve their education?

Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER Symposium

What are the kinds of information thatuniversities seek from AHELO?
• Consultation:–– MEXT Commissioned Study on the OECD AHELO Feasibility Study, hosted by the Tokyo Instituteof Technology (2009-2012).
– NIER Project Study on the Analysis of the AHELO-FS Results(2013).

–• 1. International benchmarking•– How are their students performing in relation to their international peers?
– Not ranking.

–•••• 2. Diagnosis of student competencies.
– What are the strengths and weaknesses of their students’ competencies?

•• 3. The relationship between teaching and learning environments
and student performance.

–•– How can programs and student support systems be improved?

–Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER SymposiumInternational Benchmarking & Diagnosis of Student Competences
Student Performance by Competence Clusters
80
60

The information that
universities seek.
- Hypothetical Data -Instruments need tobe designed in a waythat will allow for theanalysis ofcompetence clusters.

Basic EngineeringSciences
8060404020

Engineering GenericSkills

Engineering Analysis

2000

Engineering PracticeEngineering DesignInstitutions must agree in advance to share information on the names of institutions constituting the benchmark group .
Ideally, institutions should be able to choose their peers .

Ins A
Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

National Peers

International Peers

2013.12.10 NIER Symposium

The Relationship Between

Teaching and Learning Environments and Student Test Scores- an Exploratory Analysis -• The Central Council for Education (2012). A Substantive Reform ofUniversity Education Towards Constructing a New Future.
Recommendation 1: Induce qualitative change in Bachelor level
education through encouraging autonomous study.

•– Increasing time spent on active learning such as discussion and debate.
– Encouraging learning outside the classroom, such as internship.

––• Hypothesis 1: The lower the proportion of “lectures” and higher the proportion of“seminars” and “group-work,” the stronger the capacity for “autonomous study.”•Recommendation 2: Increase quality learning time.
– Securing sufficient total time for autonomous study – preparing for class,taking classes, and later developments.

–• Hypothesis2: The longer total time spent on study, the stronger the capacity for“autonomous study.”•Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER SymposiumTest Scores
Do they measure “ability to think autonomously?”• SWLE(Scaled Weighted Likelihood Estimate)• SWLE(Scaled Weighted Likelihood Estimate)– Combined total score of Multiple Choice Questions and Constructed Response Tasks.

–• Estimates of the student ability calculated based on the Item Response Theory model.
• Average 500, Standard Deviation 100.

••

• Student Comments to the Test.

•– Multiple Choice Questions–• Most of the contents were covered in class, so everyone should be able to solve them.
• The questions were sporadic, superficial, and unconnected. It would be better to delve moredeeply into the subject matter.

••– Constructed Response Tasks.

–• Interesting questions, asking what caused the problem and how to solve them. Useful in practice.
• In university, we deal mostly with theoretical and abstract problems. The test was interesting
because it dealt with real world problems. I liked the ethical questions.
• If constructed response tasks are important, we should have more classes with group-work andcase study. We have not had such experience, nor chances to learn approaches to solving thesetypes of questions.

•••
Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER Symposium

Sharing National Data
An MOU between Australia, Canada, and Japan• For the purposes of educational research each Party desires to discloseand share its respective national Data with each other Party (“thePurpose”).
• The Parties agree that no Party shall disclose to any third Party, publish orotherwise disseminate any other Party’s national Data disclosed to it
under this Memorandum of Understanding or findings or research basedon such Data without the prior written permission of the other relevant
Party or Parties.
• Any major public announcement in connection with this Memorandum ofUnderstanding must be agreed by the Parties before it is made, except if
required by law or a regulatory body, in which case the party required tomake an announcement must, to the extent practicable, first consult withand take into account the reasonable requirements of each other party.

•••

Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER SymposiumHypothesis 1: The lower the proportion of “lectures” and higher the proportion of“seminars” and “group-work,” the stronger the capacity for “autonomous study.”• The proportion of “lectures”•– 81 % of the students in theJapanese sample responded that“61-75 ”or “75 +”of theirclasses were of “lecture” type.
The test scores for these groups
were relatively high.
– A similar pattern was found in theCountry A sample.
– In the Country B sample, students
responding “41-60 ” and “61-75” scored relatively high on thetest.

76%+(51%)
HE Sample
in Japan76%+(51%)
61-75%(30%)
41-60%(14%)–

0-40%(4%)

61-75%(30%)
41-60%(14%)
0-40%(4%)

HE Sample
in Country A

76%+(46%)–

61-75%(30%)
41-60%(20%)
0-40%(5%)

HE Sample
in Country B

76%+(24%)61-75%(29%)•0-40%(18%)
450

500

550

Satoko Fukahori

600

41-60%(20%)
0-40%(5%)–

41-60%(29%)

76%+(46%)
61-75%(30%)

76%+(24%)61-75%(29%)
41-60%(29%)
0-40%(18%)

650

450

2013.12.10 NIER Symposium

500

550

Satoko Fukahori

600650• For the three country-samples,around 60% “lecture” type seems
to be the preferable proportion.

2013.12.10 NIER Symposium

For the Japanese sample, students spend relatively little time on active
learning activities. However, those taking more “seminar” or “group-work”
type classes do not necessarily score better.
The preferable proportion seems to be about 25%?
“Seminar”(54%)

41%+(28%)

41%+(36%)

25-40%(34%)

25-40%(28%)

25%

25%

(36%)

(35%)

41%+(38%)

41%+(40%)

25-40%(36%)

25-40%(30%)

25%

25%(25%)(1%)

Satoko Fukahori

(29%)

500

550

600

650

2013.12.10 NIER Symposium

none(4%)41%+(28%)
25-40%(34%)
less than 25%(36%)

41%+(36%)
25-40%(28%)
less than 25%(35%)
none(2%)41%+(38%)
25-40%(36%)
less than 25%(25%)

41%+(40%)
25-40%(30%)
less than 25%(29%)

none(1%)(1%)
450less than 25%(54%)

none(2%)(2%)

HE Sample
in Country B(2%)

less than 25%(59%)

25-40%(26%)

none(16%)(4%)

HE Sample
in Country A(16%)

HE Sample
in Japan

25%

(59%)

41%+(16%)

HE Sample
in Country A

25%

41%+(14%)25-40%(11%)

HE Sample
in Country B

41%+(16%)25-40%(26%)

HE Sample
in Japan41%+(14%)25-40%(11%)“Group work”

450

500

550

600

650

none(1%)450

Satoko Fukahori

500

550

600

650

2013.12.10 NIER Symposium

450

500

550

600

650

76%+(22%)
61-75%(43%)
41-60%(26%)
0-40%(9%)

76%+(16%)61-75%(63%)
41-60%(21%)
0-40%(0%)

76%+(22%)61-75%(43%)
41-60%(26%)
0-40%(9%)

41%+(0%)25-40%(0%)
25%
(87%)
(13%)

41%+(0%)25-40%(16%)
25%
(74%)
(11%)

41%+(0%)25-40%(0%)
less than 25%(87%)
none(13%)41%+(17%)25-40%(57%)
25%
(26%)
(0%)

41%+(5%)25-40%(53%)
25%
(42%)
(0%)

41%+(17%)25-40%(57%)
less than 25%(26%)
none(0%)300

400

500600

700

300

800

76%+(51%)
61-75%(26%)
41-60%(19%)
0-40%(4%)

76%+(25%)
61-75%(40%)
41-60%(30%)
0-40%(5%)

41%+(23%)
25-40%(18%)
25%
(50%)
(9%)

41%+(53%)
25-40%(18%)
25%
(28%)
(3%)

41%+(23%)
25-40%(20%)
25%
(51%)
(5%)

41%+(55%)
25-40%(25%)
25%
(20%)
(0%)
300

400

500

600

700

500

600

700

800

Institution 1:Lecture centered type

300

400

500

600

700

800

300

41%+(23%)
25-40%(18%)
less than 25%(50%)

700

800

600

700

Institution
4:
Mixed type

300

400

500

600

700

300

800

400

500

600

700

The ideal pattern and implications for improvement may vary among institutions.
Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER Symposium

Hypothesis2: The longer total time spent on study, the stronger
the capacity for “autonomous study.”
Number of Hours Spent on Activities
First Semester of Third Year(per 7 day-week)(average)
18.9

18.9
Attending formal classes

20.9

20.9

6.8

6.8
15.7

4.5
6.2
4.3

5.7

Preparing for class

15.7
11.3
5.7

Practical training

6.5

HE Sample
in Japan

6.5
5.0

5.0

HE Sample
in Country A

0.5

0.5
Paid work related to study

3.3

3.3

HE Sample
in Country B

5.0

5.0

10.5

10.5
Paid work unrelated to study

3.8

3.8

8.1

8.1

7.7

7.7
Co-corricular activities

6.8

6.8
3.4

3.4
0.0

Satoko Fukahori

Total time spent on
study per day.
Japan: 4.5 hours
A: 6.2 hours

13.7

13.7

11.3

5.0

10.0

15.0

2013.12.10 NIER Symposium

20.0

0.0

25.0

Satoko Fukahori

800

41%+(55%)
25-40%(25%)
less than 25%(20%)
none(0%)41%+(23%)
25-40%(20%)
less than 25%(51%)
none(5%)600

500

41%+(53%)
25-40%(18%)
less than 25%(28%)
none(3%)none(9%)500

400

76%+(25%)
61-75%(40%)
41-60%(30%)
0-40%(5%)

Institution
3:Mixed type41-60%(19%)
0-40%(4%)

400

Institution 2: Mixed type

41%+(0%)25-40%(16%)
less than 25%(74%)
none(11%)41%+(5%)25-40%(53%)
less than 25%(42%)
none(0%)76%+(51%)
61-75%(26%)

300

800

400

76%+(16%)61-75%(63%)
41-60%(21%)
0-40%(0%)

5.0

10.0

15.0

2013.12.10 NIER Symposium

20.0

25.0

800

Aim “11-15 hours” “preparing for class.”
Reduce time spent on “paid work unrelated to study.”
Preparing for class

Paid work unrelated to study

26+(2%)

26+(6%)

21-25(1%)

21-25(7%)

0-10(53%)

26+(16%)26+(1%)

21-25(11%)

21-25(3%)

16-20(18%)

16-20(5%)

11-15(24%)

11-15(5%)

0-10(31%)0-10(86%)

26+(5%)

26+(4%)

21-25(6%)

21-25(6%)

16-20(16%)

16-20(13%)

11-15(20%)

11-15(12%)

0-10(53%)

0-10(65%)
450

Satoko Fukahori

500

550

600

650

2013.12.10 NIER Symposium0-10(80%)
26+(16%)21-25(11%)
16-20(18%)
11-15(24%)
0-10(31%)

500

550

600

650

11-15(22%)

26+(1%)
21-25(3%)
16-20(5%)
11-15(5%)
0-10(86%)

26+(5%)
21-25(6%)
16-20(16%)
11-15(20%)
0-10(53%)450

16-20(12%)
0-10(53%)

HE Sample
in Country A

0-10(80%)

16-20(6%)
11-15(10%)

HE Sample
in Country B

11-15(22%)

HE Sample
in Country A

16-20(12%)

HE Sample
in Country B

16-20(6%)
11-15(10%)

HE Sample
in Japan26+(6%)
21-25(7%)

HE Sample
in Japan26+(2%)
21-25(1%)

26+(4%)
21-25(6%)
16-20(13%)
11-15(12%)
0-10(65%)

450

Satoko Fukahori

500

550

600

650

450

500

550

600

650

2013.12.10 NIER Symposium

Informing Universities for Educational Improvement
based on Higher Education Learning Outcomes• The information on education that universities seek•– Information that will allow universities to better understand the state oflearning outcomes of their students in relation to international standards.
• International benchmarks• Competence frameworks– Information that will provide implications on ways to improve education.
• The relationship between teaching and learning environment and student
learning outcomes.

–
••
–
•

• When does learning outcomes assessment most effectively inform
universities?

•– When the competences being measured are aligned with the competencies
being pursued through university education.
– The importance of developing competence based degree programs.

–•
Satoko Fukahori

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER SymposiumReferences•••••••OECD. Testing student and university performance globally: OECD’s AHELO.
(http://www.oecd.org/edu/skills-beyondschool/testingstudentanduniversityperformancegloballyoecdsahelo.htm)
OECD (2011). A Tuning-AHELO Conceptual Framework of Expected
Desired/Learning Outcomes in Engineering, OECD Education Working Papers,
No.60, Paris: OECD Publishing
OECD (2012). AHELO Feasibility Study Report Volume 1 – Design andImplementation, Paris: OECD Publishing.
OECD (2013a). AHELO Feasibility Study Report Volume 2 – Data Analysis andNational Experiences, Paris: OECD Publishing.
OECD (2013b). AHELO Feasibility Study Report Volume 3 – Further Insights, Paris:OECD Publishing.
Tuning Association. Tuning Educational Structures in Europe.
(http://www.unideusto.org/tuningeu/)
Gonzalez, J. and Wagenaar, R. (2008). Tuning Educational Structures in Europe –Universities’ Contribution to the Bologna Process. Publicaciones de la Universidad
de Deusto.
–Satoko Fukahori

fukahori@nier.go.jp

Satoko Fukahori

2013.12.10 NIER Symposium•••••••

OECD. Testing student and university performance globally: OECD’s AHELO.
(http://www.oecd.org/edu/skills-beyondschool/testingstudentanduniversityperformancegloballyoecdsahelo.htm)
OECD (2011). A Tuning-AHELO Conceptual Framework of Expected
Desired/Learning Outcomes in Engineering, OECD Education Working Papers,
No.60, Paris: OECD Publishing
OECD (2012). AHELO Feasibility Study Report Volume 1 – Design andImplementation, Paris: OECD Publishing.
OECD (2013a). AHELO Feasibility Study Report Volume 2 – Data Analysis andNational Experiences, Paris: OECD Publishing.
OECD (2013b). AHELO Feasibility Study Report Volume 3 – Further Insights, Paris:
OECD Publishing.
Tuning Association. Tuning Educational Structures in Europe.
(http://www.unideusto.org/tuningeu/)
Gonzalez, J. and Wagenaar, R. (2008). Tuning Educational Structures in Europe –Universities’ Contribution to the Bologna Process. Publicaciones de la Universidad
de Deusto.

Satoko Fukahori

Thank you for your attention!
fukahori@nier.go.jp
tuning-ahelo@nier.go.jp

tuning-ahelo@neir.go.jp

2013.12.10 NIER Symposium

2013.12.10 NIER Symposium

Satoko Fukahori

2013.12.10 NIER Symposium