The AHELO experience – implementation,
outcomes and learning from an Australianperspective
Dr Daniel Edwards, Principal Research Fellow, Australian Council for Educational
Research and National Project Manager, Australia for AHELO.

Introduction

This paper provides background, implementation processes and outcomes relating
to Australia’s participation in the Organisation for Economic Cooperation andDevelopment’s (OECD) Assessment of Higher Education Learning Outcomes (AHELO)
Feasibility Study. The Federal Department of Innovation, Industry, Science, Research
and Tertiary Education (DIISRTE) funded the Australian participation in this projectvia direct contribution to the OECD, and through funding of a National Centre to
implement the study in Australian universities.
The AHELO Feasibility Study was a major OECD project. Its objective was todetermine a robust approach to measuring learning outcomes in ways that are valid
across cultures and languages, and across the diversity of institutional settings andmissions. Further detail including aims and final reports relating to the project can befound at http://www.oecd.org/edu/ahelo.
The AHELO Feasibility Study involved the testing of students in three strands –Generic skills, Economics and Civil Engineering. For the Feasibility Study, Australia
participated in the Civil Engineering strand and it is participation in this strand that is
the focus of this chapter. In addition to these assessments, contextual dimension
questionnaires were deployed to students, faculty and institutional staff at all
participating institutions.
The implementation of the Feasibility was split into two phases. Phase One wasundertaken in 2011 and involved engaging institutions, coordinating focus groups inparticipating universities; undertaking an academic review of draft assessment
instruments; marking of student responses; and the collation and communication of
feedback from institutions and students. Phase Two took place in 2012 and involved
re-recruiting universities, sampling of students and staff, technical assistance inpreparation for AHELO assessments, administration of the assessments andquestionnaires, scoring of test scripts and reporting of results.
This paper offers insight into the processes, activities, outcomes and disseminationof results from the AHELO Feasibility Study in Australia with particular emphasis on
the student-related data. It begins by providing a brief background about theconcept of AHELO and the practicalities relating to the project. It then examines theprocess of implementing the study in Australia, beginning with Phase One and the1collection of feedback relating to the draft instruments. The implementation of the
AHELO assessment (Phase Two) is then detailed, alongside some insight into
participation numbers and some notable but broad outcomes reported. The final
section of the paper discusses overall outcomes of the Feasibility Study from anAustralian perspective and considers some possibilities for alternative approaches todissemination and reporting.

AHELO Background

This section provides a background to the AHELO project. It was jointly preparedwith members of the AHELO Consortium and provides insight into the aims of thestudy. It is included here to provide an overall perspective for framing the work
detailed in this paper. The overall aims and ambitions of the Feasibility Study areimportant and are sometimes lost in the detailed discussions of process in this work,so this section offers the chance for reflecting on the ‘bigger picture’ of what wastrying to be achieved.
Is it possible to undertake an international assessment of final-year students’capacity to use, apply and act on the knowledge and reasoning they have gained
from their degrees? Is it possible to assess these outcomes in an efficient andinternationally comparable way? Can policymakers, institutional leaders, faculty andstudents be convinced that the assessment of higher education learning outcomes isan essential checkpoint in the educational process? These questions lie at the core of
the OECD’s AHELO Feasibility Study.
AHELO involves the development and validation of assessments in three core areas –Generic Skills, Economics and Civil Engineering, as well as the development ofcontextual instruments to aid with the interpretation of assessment data. The
assessments are targeted at students in the final year of bachelor degrees and aim toassess their capacity to apply their skills and knowledge to real-world problems.
AHELO is an ambitious project in that it aims to develop new methodologies andtechnical standards for the assessment of higher education learning outcomes. It is
taking place on a global scale, with 17 countries participating in the development
and validation of assessments, and engagement from experts, institutions,
governments, and key higher education bodies from around the world. The project isrun by a consortium of international organisations, led by the Australian Council for
Educational Research.
AHELO responds to a critical information gap. Efforts to improve the quality ofteaching and to enhance students’ learning outcomes are stymied by the absence of
reliable information which enables comparative judgments to be made about thecapabilities of students in different institutions and in different countries, or about
the quality of teaching. In the absence of such data on core higher education
activities of learning and teaching, the standing of a higher education institution isbased largely on reputation and research performance. AHELO’s objective is tocreate a richer source of information through designing and testing measures whichgive due weight to teaching practices and learning outcomes.

2

The AHELO assessments were developed through the collaboration of experts fromaround the world and subsequently translated and adapted for use in allparticipating countries. Validation of the assessments with students in participating
countries was completed in 2011. The assessments that were developed through for
AHELO were implemented across 17 countries in 2012.
The study has had an impact on the way in which discussions about higher education
and assessment are carried out, suggesting that AHELO has the potential to reshape
the higher education landscape in important ways. First, discussions among somestakeholders appear to have moved on from whether learning outcomes should bemeasured to how they can be measured. At the same time, the engagement of
national systems and institutions (both within and outside the OECD) has increased
significantly throughout the life of the Feasibility Study. Taken together, these
indicate the existence of a desire for data on the quality of teaching and learning,both to inform improvements and also to demonstrate quality.

Facilitating AHELO in Australia

Australia’s participation in AHELO was funded and supported by the Federal
Government. The Government facilitated the organisation of a National Centre
which included a National Project Manager (NPM), some basic support and stipend
for travel. The NPM in Australia was essentially responsible for implementing all
aspects of the AHELO Feasibility Study, under the direction of the AHELO Consortiumand the OECD Secretariat for the study. Many of the key activities of the NPM andNational Centre are detailed in the sections of the paper dealing with activities inPhase One and Two. One aspect of the work not covered as much in these latter
sections is the element of communication required through the duration of the
project. This aspect is discussed briefly here.
The NPM maintained close and frequent communication with the Federal
Government through the Department of Innovation, Industry, Science, Research andTertiary Education (DIISRTE) throughout the Feasibility Study, providing monthly
progress reports during the busy times in the project and having regularteleconferences. The progress reports and discussions helped keep the Department
aware of the day-to-day activities of the NPM as well as providing insight into theextent to which the project was being embraced by universities, university leaders,
academics and students.
The National Centre also played a critical role in developing relationships andcommunication with participating institutions. Contact and collaboration withinstitutions was maintained during the study through teleconferences, a symposium,site visits and email. The key institutional relationship maintained by the NPM waswith each Institution Coordinator, who was in charge of activities within their
university. Building personal connections with these people was an important andhighly valuable facet of the project.

3

Another key role of the NPM that is not explored as much in the other sections of
this paper is that this role provided a channel through which Australian national
interests are represented in the implementation of the AHELO Feasibility Study. The
Australian Government were interested in ensuring that there was important
Australian input into the development and adaptation of relevant test instruments
and in the processes and procedures used in data collection.

Implementation of AHELO

This section of the paper provides an overview of the AHELO Feasibility Study as awhole from the perspective of the in-country implementation of the study.
Implementation within countries and institutions participating in the Feasibility
Study was carried out in two phases. Phase One involved qualitative testing of items
through surveys and focus groups and providing feedback to the OECD and AHELO
Consortium relating to the instruments constructed for the study. Phase Two
involved the implementation of the AHELO assessments in universities and withstudents under test conditions as well as the completion of staff and institution-level
surveys.
During Phase One of the Feasibility Study, qualitative testing of the draft
assessments was undertaken in a number of countries. During focus groups andcognitive labs, students provided feedback on the assessments, while a number ofsenior academics in the area of civil engineering were asked to provide commentand review of the draft AHELO assessments. In summary, the key facets of Phase
One were:• University engagement and recruitment – securing the interest andparticipation of universities.
• Support for participating universities – providing detailed information to
universities about participation, including, a Manual, discussions and training
for Institutional Coordinator and ongoing contact and support.
• Implementation of focus groups – carried out at each university andfacilitated by the Institutional Coordinator. Involved students taking asample of the draft assessment and providing feedback through a survey andfocus group.
• Academic Review of AHELO instruments – by experts in civil engineering
education.
• Collation of test results and feedback from students and academics –organised by NPM and included marking of test papers, collation and codingof surveys and construction of a national database.
• Providing feedback on test instruments – involving analysis of responses andcollation of key points for feedback to the AHELO Consortium.
Phase One activities in Australia were undertaken in 2011 and involved ten
universities (further detail is contained in the ‘outcomes’ section of the paper).

4

Phase Two involved the full implementation of the AHELO assessment in universities
to cohorts of students online under test conditions. It also involved the collection ofsurvey data from academic staff and Institution Coordinators.
In summary, Phase Two involved the following activities for Australian participation:• Institutional recruitment – re-securing the interest and participation ofuniversities.
• Development of manuals for institutions – to assist in implementation of theonline testing and in recruiting of students.
• Institutional Coordinator training and support – to ensure all participating
institutions were familiar with the AHELO test system and AHELO test
administration protocol.
• Institutional sampling – collecting population data on eligible students andstaff members in participating institutions.
• Setting up test computers – providing advice and information on establishingsecure online profiles for testing centres.
• Test administration – involving the implementation of the AHELO tests tostudents in participating universities and monitoring the administrations toconfirm test sessions were run according to protocol.
• Survey administration – sending the staff and institutional survey toidentified population and following up to secure maximum response
numbers.
• Scoring of assessment responses – involving recruitment of a Lead Scorer,training in scoring and facilitating scoring among a group of experts.
• Supporting data processing and international analyses – ensuring adherence
to AHELO protocol and maximising international comparability of data.
• Attend international training and dissemination meetings – maximising thevalue of lessons drawn in the Australian context as well as benefitting frominsights in other countries.
Phase Two was conducted in Australia during 2012, with testing being undertaken
between March and May in eight universities (details in ‘outcomes’ section).

Recruiting Institutions
Based on requests from the OECD and AHELO Consortium, Australia aimed to haveten universities participate in the Feasibility Study. As such, early in 2011, the ViceChancellor and Dean of Engineering at ten Australian universities were written to
and invited to participate. All ten invited institutions agreed to be part of at least
Phase One of the study.
Selection of universities involved liaison between the NPM, the Department(DIISRTE) the Australian Council of Engineering Deans (ACED). Universities wereselected on the following basis (in order of selection priority):
1. They had actively shown prior interest in AHELO (through informal discussion
with the Government).

5

2. They had previously provided contact details for an Institutional Coordinator
to DEEWR.
3. They had an active Civil Engineering program.
4. The ‘sample’ included universities from a range of geographic locations.
5. The ‘sample’ included universities from a range of different ‘groupings’ ofinstitution (i.e. research intensive, ‘new’ universities, regional universitiesetc.)
6. The ‘sample’ included universities of various sizes.
As each university confirmed participation in the Feasibility Study, they were
contacted by the NPM and provided with an overview of the project. Institutional
Coordinators were identified and directly contacted by the NPM to provide details
on involvement in the project.
Across the two years of the project, students from 11 different Australian
universities were involved. These institutions are listed below. Institutions with one
asterisk (*) participated only in Phase One, these institutions generally citedresourcing issues as the key impediment to further participation. The one with twoasterisks (**) participated only in Phase Two following an invitation from the NPM inearly 2012. All other institutions listed were involved in both phases of the Feasibility
Study.












Charles Darwin University
Curtin University of Technology*
James Cook University
Swinburne University of Technology
The University of Adelaide
The University of Melbourne
The University of New South Wales*
The University of Newcastle*
University of Technology Sydney
University of Western Sydney
RMIT University**

Outcomes

In this section the broad outcomes from students in Australian institutions
participating in AHELO are discussed. The discussion is relatively broad and focuses
more on the participation levels than it does on the actual ‘results’ from the AHELO
assessments.

Qualitative testing – Phase One
Phase One was carried out in ten participating institutions in Australia in April and in
August 2011. In total 78 students took part in focus groups in which they were asked
to take some sections of the draft assessment and then complete a survey and6

provide feedback in a focus group setting. The draft assessments included in thissession were:• a Constructed Response task - where students are presented with a scenario
or problem and asked a number of questions in relation to this. In mostcases these required ‘open answer’ responses (i.e. they were not multiple
choice); and• a set of Multiple Choice questions.
All sessions were run within participating universities by the Institution Coordinator
or a designated member of faculty. Sessions lasted approximately one hour andwere undertaken by final year Bachelor of Civil Engineering students. Table 1
displays the demographic characteristics of the students who participated in thefocus groups for Phase One.
Table 1: Focus group demographics

Demographics
GenderAge

CitizenshipLanguage backgroundTotal students

Male
Female
20
21
22
23
24
25
26-30
31 and above
Australia
Other country
EnglishOther language

(n)
60
18
4
18
26
12
9
3
3
2
54
23
52
25
78

%
76.9
23.1
5.1
23.1
33.3
15.4
11.5
3.8
3.8
2.6
69.2
29.5
66.7
32.1
100

After taking part in the test, students were asked to focus on the following topics
during the moderated discussion:• The task challenged me to think (challenge)• The materials stimulated my interest in the task (interest)• The task made me apply knowledge and skills in real-world ways (apply)• The task assessed an appropriate range of knowledge and skills (range)• The task was relevant to my program of study (program)• The task was relevant to future professional practice (future)
The responses of students were scored and input into a national dataset for Phase
One that was provided to the AHELO Consortium. At the national level, the National
Centre undertook some analyses of the student responses.
The main outcomes derived from Phase One in Australia was that students rated thetest mostly positively on the criteria of challenge, interest, application, range,
relevance to program and future professional practice. The Constructed Response7

tasks were rated slightly higher on these criteria compared to the Multiple Choicequestions.
An overview of the Australian student responses to survey questions about the
AHELO assessment instruments is provided in Figure 1 and Figure 2 below. Theyshow that students were highly challenged by the Constructed Response tasks andwere also generally positive about the other specific areas explored through thesurvey – with relevance to program and relevance to their future work being ofparticular interest (Figure 1). Levels of challenge and relevance to the degree
program for these students was also relatively good for the Multiple Choice items,
while the detail in Figure 2 suggests interests, application and the future relevance of
these items were given lower scores by participating students.

Figure 1: Feedback on Constructed Response Tasks

8

Figure 2: Feedback on Multiple Choice Tasks

In addition to rating the test through a questionnaire, the student focus groups alsoentailed a moderated survey format. In brief, the major responses of thesemoderated discussions were:• When asked about the Constructed Response tasks, in general students
found these made sense and the instructions were adequate.
Overwhelmingly students found the task interesting, because of the diagrams,
real-world problems, and because the test was different to how we students
are tested usually.
• Student opinion about the Multiple Choice tasks was generally that thesewere more like the tasks they were familiar with at university. Students
found the tasks to be more recall oriented and while relevant to their degree
seemed less relevant in the long term for their future career.
A review of the draft assessment instruments was also undertaken by threeacademics in Civil Engineering from participating Australian universities. The
outcomes of the academic review can be summarised as follows:• In general, the feedback was similar to that of students.
• There was some suggestion that the content was focused on the early stagesof degree, rather than final years.
• There was more specific item-based feedback relating to technicalexplanations, language used etc., suggested for change in order to makesome items clearer.
Based on the results from the student focus groups and academic review both inAustralia and in other participating countries, a number of changes were made by9the AHELO Consortium to the instruments used in the Civil Engineering strand. A
major outcome at this point in the study was that more time was allocated for thetest than originally planned. One Constructed Response task was completely
removed from the test since it was too difficult. Other Constructed Response tasks
were amended.

Quantitative testing – Phase Two
Phase Two of AHELO was conducted in 8 Australian universities in 2012. Students
participating in the Feasibility Study undertook the AHELO Civil Engineering
assessment, which involved a 90 minute test including one Constructed Responsetask and a module of Multiple Choice Questions. Students also completed a context
questionnaire at the end of the test session which collected data relating todemographics and other background characteristics of students.
Sampling and student recruitment
A major task in preparing for test administration was the collection of samplingframes from each participating university in accordance with the AHELO protocol. To
ensure international comparability of the AHELO data and to facilitate processing of
the data collected, all participating institutions were required to provide lists of alleligible students and staff members, in a pre-defined table format. Internationally,
this format enabled the validation of target populations and where necessary theselection of a sample. In Australia, Civil Engineering faculty sizes were sufficiently
small to allow for a census rather than a sample. Eligibility for being included in thetest population was defined as students enrolled in civil engineering programmes intheir final year before Bachelor completion.
The provision of sampling frames to the NPM was a significant effort on the part ofthe institutional coordinators, since it required up-to-date information on theenrolment status of students and a range of demographic characteristics.
Institutions were invited to add any background indicators to their sampling frames
they deemed enriching to the data, and some provided information such as the
programme students were enrolled in, country of birth of the student, or student
aptitude measures such as grade point average. Other institutions were moreconstrained by regulations of data sharing, and were only able to provide essentials
such as gender.
Test administration took place during April and May 2012 in all institutions. For theadministration of the student assessment and questionnaire, most institutionsorganised a number of sessions, while other smaller institutions organised a single
session. In some institutions, the institutional coordinator supervised all test
administration sessions, whereas others hired test administration assistants
financially supported through the NPM budget. During test sessions, the NPM wasavailable for support regarding the AHELO test system and any other queries
regarding the AHELO protocol.
A major challenge for the Australian implementation of AHELO was motivating
students to participate in the study. This was a problem also highlighted by some10

other countries during the international NPM meeting in October 2011. Given thevariety of institutions participating in Australia, the NPM took a flexible approach toincentivising participation, allowing institution coordinators to propose localised
incentive ideas that were agreed to and funded by the NPM. Table 2 details the kindsof incentives offered by institutions to participating students. It shows a range ofdifferent options and approaches offered. In some institutions, full student
population numbers were small enough to offer a voucher to each participant, inothers it was necessary to have a draw of prizes for participants. Besides monetary
incentives, students were motivated to participate by explaining to them therelevance of the test to their future work experience, and the value of their
participation to the success of the study. One university chose not to offer any
monetary or ‘prize’ incentives for students and instead focus solely on theexperience of the assessment as being an incentive for participants.
All institutions were provided with participation certificates to provide to students
who sat the test. These certificates were developed by the AHELO Consortium andthe OECD and included the OECD logo.
Table 2: Student participation incentives

Institution
Uni A
Uni B
Uni C
Uni D
Uni E
Uni F
Uni G
Uni H

Incentive$AU100 voucher for each participant
ipad draw. Lunch and discussion of Outcomes Assessments forparticipants$AU50 voucher for participating students
ipad draw. Voucher draw in each room (8 different sessions) chance ineach room to win $AU200 or a $AU100 voucher.
2 ipads drawn, 5 $AU50 vouchers drawn, a lunch$AU1000 prize draw, and offering student society $AU1000 for a highoverall participation rate (75% - not achieved).
none
Made test a 'requirement' for a core subject.

Table 3 provides the participation numbers and participation rates (based on thenumbers of eligible participants identified through the selection of a sampling frame)for both students and staff for each university. In total 187 students from Australian
institutions participated in the AHELO assessment. Across the institutions involved,
one in five final year civil engineering students participated in the assessment.
However, this average does not adequately represent the notable variation inparticipation rates of students between institutions. Five of the eight institutions had
participation rates below 25 per cent, while one institution (Uni H) had almost fullparticipation from its cohort.

11

Table 3: Participation numbers and participation rates of students, Australian institutions for
AHLEO
Institution
Uni A

Number ofparticipants
3

Participation
rate* (%)
30.0

Uni B

27

12.3

Uni C

4

12.5

Uni D

29

19.3

Uni E

27

29.0

Uni F
Uni G

25

14.2

19

12.8

Uni H
Total

53

98.1

187

21.1

*The participation rate is based on the number of participating students as a proportion of the full student
population identified in the sample frames constructed by institutions for the study.

Numerous issues affected the participation rates of students in Australia. These
included students having limited ‘on-campus’ time in their course due to final year
internships, a large proportion of students in part-time work and therefore little orno free time, and short notice of the testing window.
However, as the table shows, the Uni H was able to secure a large participation rate,
proving that it is possible within Australian higher education to achieve wider
engagement of the student cohort in this kind of exercise. Given the success of thisinstitution in securing this level of engagement from students, further information of
their approach is detailed in the box below.
Successful engagement model
The approach taken by the Uni H offers an insight into an ideal implementation model. The approach
involved strong and dispersed leadership within the institution, insightful planning, and the merging
of the assessment into a dialogue about learning outcomes between students and staff.
The implementation of AHELO at this institution was planned in advance, beginning in November theyear prior to testing – four months before testing – following an information session facilitated by the
National Project Manager. Leadership in the project came at three levels within the institution – fromthe Chancellery, where a member of the Pro-Vice Chancellor’s office was directly involved, from theHead of the School of Engineering and from a senior faculty member who had ultimate responsibility
for implementation. During planning for the testing, the team within the institution identified a coreunit (a project/thesis unit) in its final year civil engineering program in which the theme of learning
outcomes and graduate capabilities was appropriate. The unit was planned in such a way that AHELO
became an integral part of the work during the semester. Students were asked to undertake theassessment as part of the unit of study and following the assessment, were involved in discussions toreflect on their assessment experience and the relationship between their coursework, the skills they
expected to employ in the workforce following graduation and professional responsibility related toassurance of educational and practice standards.

12Participant characteristics
The characteristics of the student participants in AHELO are displayed in Table 4.
Notable student demographics include the dominance of male students inengineering, and the high proportion of students with a main language other thanEnglish. The average age of students participating was 23.2. Most students werestudying full-time and on campus. A basic comparison of the sample frame with thesecured participation group suggests that by gender the sample is representative,
although it appears that the participant age may be slightly older on average than
the full population.
Table 4: Student participant characteristics
Characteristic
Category

GenderAge

Enrolment type
Enrolment mode
Country of birthParticipant
population(% oftotal)

Male

80.1

Female19.9

20 and below

5.3

21

20.5

22

30.4

23

19.3

24

9.4

25-30

11

Above 30

4Mainly part time

4.1

Mainly full time

95.9

Entirely on-campus

91.2

Mix of external/distance and on-campus

8.8Australia

53.8

China

14.6

India

2.9

Malaysia

5.8

Other

22.8

Assessment outcomes
Discussion relating to the data outcomes from AHELO participation by Australia islimited in this paper due to the fact that this data is relatively sensitive and thesample numbers and representativeness of students for most of the universities
involved in Australia was below the level that would be required for the collection ofreliable data. Instead, some relatively broad level findings are shown here using thefull Australian data set. Two issues are discussed – relationship between student
grades and outcome on AHELO, and the link between work ‘practice’ and outcomesin AHELO. For broad reference, in examining the scores below, the internationalmean for AHELO is 500 with a standard deviation of 100.
Figure 3 shows the relationship between student outcomes on the AHELO
assessment and their grades during university study – self-reported by students. It

13

clearly shows that the Australian students who identified themselves as being among
the top in their class achieved a much higher average score in AHELO than otherstudents. This goes some way to suggesting there is some concurrent validity
present in the AHELO instruments.

AHELO mean score

640.00620.00600.00580.00560.00540.00520.00500.00480.00Among the top
students

Above average

About average

Self-reported academic grade

Below average

Figure 3: AHELO mean score by self-reported outcome in university studies, Australian participants

An important issue in Australian Higher Education is the link between study andwork and the extent to which study is preparing young people for their future
careers. The outcomes from Phase One displayed earlier suggested that at leas interms of the content of the tests, students saw substantial relevance with future
work in the Constructed Response tasks. Interestingly, the AHELO data for Australia
shows that those students who are engaged in paid work relating to their degree in
their final year of university, and those who are planning to work in the industry
related to their study when they graduate are much more likely to have higherscores than students not currently and not planning on working in their field in the
future.
Figure 4 shows that students who worked at least six hours – but less than 31 hours– per week in a paid job that was related to their study on average had a higher
outcome on the AHELO assessment than those not working in a paid job in a related
field or those working more than 30 hours in such employment. This is an interestingfinding in the context of considering the extent to skills gathered through working in
the field might help influence the extent to which students were able to successfully
answer questions in the AHELO assessment. It also suggests that for those students
with a very heavy work burden during their final year of study (paid work of morethan 30 hours), this paid work may be having an adverse impact on their ability toperform at university.

14

AHELO mean score

680.00660.00640.00620.00600.00580.00560.00540.00520.00500.00480.00None

1 to 5hours

6 to 10 11 to 15 16 to 20 21 to 25 26 to 30 More
hours
hours
hours
hours
hours than 30
hours
Hours in paid work related to study (per week)

Figure 4: AHELO mean score by number of paid hours worked in job related to study per week, Australianparticipants

The other work-study related issue that shows some interesting data among theAustralian participants in AHELO relates to the post-graduation ambitions ofstudents. The data in Figure 5 indicate strongly that those students who had plans topursue a career in Civil Engineering straight after completing their degree wereclearly more likely to have a higher score on the AHELO assessment. It is likely thatthese are more motivated students at this time in their degree and perhaps more
focussed than those students with no plan to pursue work in the field. Interestingly,
those students intending on pursuing further study in the area had a higher meanscore than those not planning on working in the field, but lower than those intending
on entering the Civil Engineering workforce.

15

AHELO mean score580.00560.00540.00520.00500.00480.00460.00Work in an
occupation
related to yourmain field of
studyWork in an
Enrol in further
occupation notstudy
related to yourmain field of
study
Plan for year after graduation

Other

Figure 5: AHELO mean score by pathway planned by student following graduation, Australian participants

Conclusion and ideas for the future

The Australian participation in the AHELO feasibility study has resulted in a numberof valuable lessons. Firstly, it has shown that the Australian sector is equipped toparticipate in this type of international study. There was interest from stakeholders
in participating, and administrative systems in institutions allowed relatively
straightforward production of the required student and staff data. The process of a
National Centre overseeing a network of institutional coordinators functioned well,and coordinators report finding this type of cooperation valuable.
Secondly, participation in the process of developing and trialling an internationallyapplicable engineering test has proven to be insightful both to participating
institutions and students. AHELO has proven to be a truly ground-breaking test in thesense that students reported not having been exposed previously to the type of
applied, integrated problems mirroring their future work. The cooperation between
engineering experts from multiple countries both in meetings and via online
communication has provided an excellent opportunity to strengthen internationalbonds. In addition the cooperation between participating institutions opened up
opportunities for future cooperation within engineering faculties.
In practical terms, AHELO has shown that students in Australia are not easily
motivated to participate in a voluntary test or questionnaire. Future iterations of
AHELO or similar studies would do well to integrate as much as possible these typesof assessment into the curriculum, using implementation models such as that usedby Uni H. Though participation rates in the feasibility study for Australia were16

disappointing, the process of implementation has built substantial knowledge on theprocesses and systems needed for engagement among students and institutions infuture studies of this kind.
AHELO has provided some small insights into the current state of engineering
bachelor education at selected universities in Australia. While this data does notallow any strong conclusions about the skill level of Australian students as itcompares to that of students in other countries (and therefore reduces theoutcomes analysis of this paper), it allows a glimpse into what could be possible infuture iterations of the study. Importantly, involvement in the AHELO Feasibility
Study has provided Australia with valuable lessons and models for future
implementation of such assessments in the future.
For Australia, there are some worthwhile considerations for future international
participation that have become apparent through the feasibility study. One is that in
the testing window used in this phase, Australian students were technically one
semester behind those from institutions in other countries (except Japan). The test
window fell in Semester 1 of the final year of the Australian students, but in the final
semester of study for those in most other participating countries. For accurateinternational comparisons, future iterations of the study should be implemented incomparable times during academic years across all countries. A second is that a
number of the institutions involved in the Australian participation have substantiallylarge final year internships or research projects in the final year, meaning thatstudents spend significant time in this year off-campus. As such, being able to find atime in which a large cohort is able to participate in a secure assessment is difficultfor institutions. Longer term planning for the running of such assessment could help
in minimising the impact of these key events in the final year. However, theimportance of the flexibility of internships and research projects may make this achallenge to achieve. A final issue recommended for consideration in the future is
the production of student-level reports for individuals who participate. It is
recognised that this was beyond the scope of the feasibility study, but Australia
believes building such capabilities in the future would help to stimulate engagementof students in these types of studies.
Since the conclusion of the Feasibility Study, the Australian NPM has worked withcolleagues on identifying useful ways of developing reports for institutions within
the constraints of the study (in particular the fact that the assessment items are stillsecure and confidential). Some innovative work has been undertaken in this regard
and an example institution level and national level summary report are presented at
the end of this paper. These reports show outcomes at the individual item level,
order items by level of difficulty, and provide a reference to the type of capability
being tested by each item. The example here is a useful one in consideration ofdeveloping such reporting in the future. It is the opinion of the author of this paperthat greater momentum following the implementation of the Feasibility Study would
have been gained by having clearer and more detailed reporting at the student andinstitution level.

17

Acknlowledgements
Thanks to colleagues and members of the ACER Consortium for advice and content
for this paper and assistance throughout the Feasibility Study Eva van der Brugge,
Sarah Richardson, Hamish Coates, Jacob Pearce, Yan Bibby and Xiaoxun Sun.

18

EXAMPLE OF FORMAT FOR REVISED INSTITUTIONAL REPORT – PATTERNS OFSTUDENT PERFORMANCE

19

20

21

EXAMPLE OF FORMAT FOR REVISED NATIONAL SUMMARY REPORT22

23