AHELO: The experience of Ontario’s
institutions
Mary Catharine Lennon
Senior Research Analyst
Higher Education Quality Council of Ontario0Introduction
In 2011 Ontario joined the Organisation for Economic Co-operation and Development’s (OECD)
Assessment of Higher Education Learning Outcomes (AHELO) feasibility study. The Higher Education
Quality Council of Ontario (HEQCO) led the project on behalf of the Ministry of Training, Colleges andUniversities (MTCU) and in cooperation with the Council of Ministers of Education, Canada (CMEC).
Initiated in 2006, AHELO is a feasibility study to determine if standard generic and discipline-specifictests can be used in different countries to measure what university students know and are able to do.
Intending to contribute to the international conversation on establishing better indications of learning
quality, the study aimed to develop common learning outcomes and assess student performance at theend of a bachelor’s degree (first cycle) in a variety of educational cultures, languages and institutions
through standard tests. The feasibility study developed three assessments: one for generic skills, andtwo for discipline specific skills in economics and civil engineering.
Seventeen countries are represented in this global project, and Canada is one of nine jurisdictions
participating in the engineering strand. Nine out of ten Ontario universities with civil engineering
programs participated in the study. Nearly 450 Ontario students participated, representing
approximately 61% of all civil engineering graduating students.
The following report reviews the experience of Ontario’s participation in the feasibility study, focussing
primarily on the implementation and administration activities, and the value to institutions. While theinstitutions did not gain specific insight on their programming, AHELO generated considerable interest ininternational assessments and comparative understanding, and provided significant experience in theadministration of large-scale assessments.

Background and Rationale for the AHELO Feasibility Study
The late 20th century saw significant changes to higher education worldwide. 1 The massification ofhigher education produced a diverse profile of institutions, programs and students unlike the small elite
systems of previous times (OECD, 2012a). This expansion was aided by a number of elements. Higher
education is no longer contained by either bricks and mortar or national/jurisdictional boundaries. The
proliferation of technology-enhanced learning allows programs to operate whenever and wherever the
student chooses, and international student and faculty mobility have opened institutions to the world.
Furthermore, the student demand for education has supported the creation of new and alternative
providers, such as private institutions or those with specialised programming.
The now complex and global “market” for postsecondary education (PSE) demands new forms ofgovernance, accountability and signalling mechanisms. In line with traditional forms of accountability
and governance, measures of inputs (funding, library holdings, etc.) and outputs (retention rate,
graduation rate, publications) have been the yardstick of performance both within systems and1

Information contained in this section is largely based on OECD documentation. For further information, please
see Tremblay, Lalancette and Roseveare (2012) and OECD (2013a1

internationally. International rankings based on these performance indicators play a significant role inweighing the research capacities of institutions.
What these traditional measures fail to capture is the quality of education: the teaching and learningthat is the heart of most institutions. Recognition of this gap led to the inclusion of another set ofindicators that can be called “proxies”. These “proxies” of quality include student evaluations, surveys ofstudent satisfaction and engagement, labour market outcomes, etc. These indicators suggest that if thestudent is satisfied and has fared-well beyond PSE, then the institution must have provided high-quality
education.
More recently, and considered by some to be a paradigm shift (OECD, 2012a: 35), there has been afocus on learning outcomes as a means to understand, demonstrate and assess educational quality.
Defined learning outcomes – clear statements of what a learner knows and can do – along withappropriate assessment measures, provide a transparent means to measure student learning. It is thus
possible to gain an objective picture of the quality of teaching and learning, and ultimately the quality ofeducation provided.
Clear indications of teaching and learning quality are beneficial in a number of ways. They support abetter understanding of educational value to students, employers, and the public at large. They alsoenhance institutional and programmatic improvement in coordination, curriculum development andteaching practices. Furthermore, they can provide measures of what has been considered intangible
until now in the world of quality assurance and accountability – educational quality. They also provide atransparency of programming, which allows for greater international and comparative understanding for
institutions and programs. Hence, learning outcomes demystify education to the benefit of the program,institution, and wider public and international community, but ultimately they put students at thecentre of it all. This indicates a significant shift towards student-centred learning.
The purpose of the feasibility study was to provide ‘proof of concept’ (OECD, 2009a; 15) to determine
whether it is possible to measure what undergraduates know and can do at the international level, toprovide relevant information to higher education institutions (HEI’s), governments, and otherstakeholders including students and employers (OECD 2009b: 2)
The primary questions that emerged from the work included:•••

Is it possible to have international agreement of expected learning outcomes?
Is it possible to implement the same test across cultures and languages?
Are the assessments valid and reliable?

Three primary areas of work made up the feasibility study. Three strands were developed to assess
student performance. One sought to assess generic skills, such as capacities in critical thinking, analyticalreasoning, problem solving, etc. 2. The other two assessments concentrated on discipline-specific skills

2

For more information, see Tremblay et al. (2012); p 112-114.

2

in economics 3 and engineering. 4 Rather than assessing content knowledge, both discipline assessments
focused on the application of knowledge (i.e., can a student “think like an engineer”). An additional
strand, “value-add”, was intended to explore the contribution of higher education to student learning.
It is expected that there will be gains in skills and competencies from when a student enters PSE andwhen they graduate (see OECD, 2013b).
In addition to these two areas of work, a Context Dimension 5 was developed to survey students, faculty,
institutions and jurisdictions to better understand the educational environment of each jurisdiction andidentify factors that might explain differences in observed performance of students.
Concurrent to the development of the assessment frameworks, the OECD invited member countries 6
and other interested countries to join the feasibility study, share in the development of the assessments
and pilot the test to a sample of their institutions. Seventeen jurisdictions agreed to take part in the
study, representing six continents and 12 languages. Ontario participated in the Civil Engineering Strand.

AHELO in Ontario
In July 2011, the Ministry of Training, Colleges and University in Ontario decided to join the study
independent of other Canadian provinces 7. The Higher Education Quality Council of Ontario (HEQCO 8),
an arms-length agency of the Ontario government, was asked to lead the project.
As part of a broad research agenda to understand and demonstrate the value of postsecondary
education, Ontario, through HEQCO, has engaged in a number of research projects to explore learning
outcomes. HEQCO has been engaged in a multitude of activities in the area, including establishinglearning outcomes across sectors of disciplines (the Tuning project; see Lennon, Frank, Humphreys,
Lenton, Madsen, Omri & Turner, forthcoming A); measuring generic learning outcomes through piloting
the Collegiate Learning Assessment (see Lennon, forthcoming B); as well as supporting the incorporationand demonstration of leaning outcomes through a variety of institutional activities. Thus, participatingin the international study supported the research base by providing comparative information on
systems and programming.
Acting as the National Project Centre, HEQCO was responsible for liaising with the OECD and AHELO
Consortium in order to administer and implement the assessment. This entailed acting as national
experts to the OECD AHELO conversation, providing a National Project Manager (NPM), implementing
and administering the study with participating institutions, and providing analysis.
3

For more information, see OECD 2011a; Tremblay et al. (2012), p 116-118.
For more information, see OECD 2011b; Tremblay et al. (2012), p 121-123; OECD, 2012a; OECD 2012b
5
For more information, see Tremblay et al. (2012) p 128-132; OECD, 2012c
6
For a list of member countries, see http://www.oecd.org/general/listofoecdmembercountriesratificationoftheconventionontheoecd.htm
7
The Council of Ministers of Education Canada represents higher education at the OECD table and, withagreement from all of the provinces, engages in various OECD projects. Despite general interest, CMEC was unable
to gain consensus from the provinces to join the feasibility study.
8
See www.heqco.ca for more information
43

There were a number of activities involved in setting up and administering the study in Ontario (seeFigure 2). The feasibility study, operationalized by the Consortium, provided guidance and protocols for
implementation in order to ensure that comparable activities took place in all jurisdictions. The
following sections review the primary areas of field work and implementation, documenting both OECD
protocols and Ontario’s actions.
Figure 1: Basic Timeline for Ontario’s Implementation Activities

July 2011
Ontariojoined study

October
Recruited
institutions

DecemberDocuments
andassessments
vetted andadaptedJanuary 2012
IC training
and ethics

February
Recruitment,IT
preparations
and TA
training

MarchStudent
testing

April
Faculty andinstitutional
surveycollection

May

Scoring

Institution Recruitment and Participation
The OECD recommended that each jurisdiction have a convenience sample of 10 institutions torepresent a range of types of institutions providing bachelor-level (or first-cycle) civil engineering
degrees. It was desirable to have public and private universities/polytechnics/colleges, institutions ofdifferent sizes and with different mandates (teaching or research), and those in rural and urban settings,
etc. The goal was to ensure that all types of institutions and of students were adequately represented inorder to have a valid and reliable understanding of each of the systems.
In Ontario, 10 institutions provide bachelor-level civil engineering programs. Hence, each institution was
invited to participate via a letter to the vice-presidents academic and the deans of engineering. To
support participation in the project, each institution was offered a nominal sum to cover the costs ofadministration. Responses from the institutions were immediate and extremely favourable: nine out of
10 institutions agreed to participate, noting their interest in taking part in this international assessment
as a way of understanding their own students and program, as well as those internationally.
Ontario Engineering programs were already familiar with notions of learning outcomes and strategies toassess them. As part of the Washington Accord agreement, and for accreditation purposes, engineering
programs across Canada are moving to outcomes-based programming, assessment and accreditation for2016 (EGADS, 2013). Hence there was considerable interest in exploring this large-scale assessment as a4tool. Approximately 90% of all of Ontario’s civil engineering students were represented in these
programs.
The participating institutions were all public institutions offering a broad range of arts and science
programs up to the doctoral level. Located primarily in urban areas, they ranged in size from 14,595 to
75,941 full-time equivalent (FTE) students. Participating institutions included:
•
•
•
•
•

Carleton University
University of Ottawa
Ryerson University
University of Waterloo
University of Windsor

•
•
•
•

McMaster University
Queen’s University
University of Toronto
Western University

The civil engineering programs were housed in faculties of engineering and were occasionally partneredwith environmental engineering. The programs had between 17 and 40 faculty members. The full-time
equivalent (FTE) student populations in the programs ranged from 231 to 573. The number ofundergraduate degrees awarded annually ranged from 20 to 101 in the 2010 academic year.

Adapting, Vetting and Validating the Documents
A Lead Scorer was identified in each country to provide discipline/content expertise in the developmentof the assessments. The individual, a recognised expert in his or her field, was required to review all test
documentation, adapting the assessments and translations to ensure appropriateness for theirjurisdiction, and vet the assessment documents through pilot tests of students and faculty. The leadscorers from each country worked together to determined appropriate grading of test items and were
responsible for training an in-country scoring team.
Ontario’s Lead Scorer, in collaboration with the National Project Manager, organised various activities tovet the test in the Ontario context, including piloting the test to students 9 and seeking feedback on thetest from faculty members and senior graduate students. The feedback commented on the suitability of
the content and difficulty level of the questions and their appropriateness for the Ontario context(including issues of language nuances and technical jargon).
This information was presented to the OECD and incorporated into the assessment. Ontario
recommended few significant changes. Indeed there were very few items of contention within theentire international scoring team, and jurisdictional lead scorers came to agreement rather quickly.

Institutional Activities
Each institution participating in the study identified an institutional coordinator. This individual was
responsible for administering the study within their institution and liaising with the HEQCO as the
National Centre. Playing a vital role in the AHELO study, institutional coordinators were responsible for
operationalizing AHELO within their institutions.
9

Five third-year students wrote the test and provided feedback. Third-year students were selected so that the
population of fourth-year students would not be compromised.

5

Standardising field activities was critical to ensure that any resulting information would be reliable.
Regardless of how well-designed the assessment might have been, administration and implementation
issues could have impacted the validity and reliability of the data. Thus, an important part of thefeasibility study involved securing common field activities.
Operationalizing AHELO Within Institutions
Each institution was in close contact with its National Centre to ensure consistency of implementation
across a jurisdiction, and also to receive any support required. In Ontario, Institutional Coordinators (ICs),
many of whom were the chairs of civil engineering, were identified within their institutions. The nine
individuals attended a one-day training session at HEQCO to provide them with context and backgroundon AHELO, and a roadmap for activities. Following the face-to-face meeting, the ICs and the NMP had
weekly teleconferences to provide support and advice on activities.
There were a number of tasks involved in operationalizing the field work, and each institutional
coordinator developed a team to assist them in the tasks. Recommended team members included an IT
specialist and test administrators. The majority of Ontario institutions followed this recommendation,some with larger teams than others.
One of the ICs’ first tasks was to apply for institutional ethics approval to implement the test. Unlike
other jurisdictions, Ontario institutions require permission to perform research on human subjects,
including students. Based on the short timelines and the need to implement the test quickly, Ontario
institutions were required to modify the parameters of the research in order to obtain ethics approval.
For example, whereas AHELO had requested that the ICs have access to students’ administrative
information (to ensure that the sample was representative of the general population and to better
understand the assessment results based on GPA, high school average, etc.), Ontario institutions were
not given permission to link the AHELO test results to student files 10. This resulted in less reliable
information on the student sample writing the test and reduced institutions’ ability to examine the data.
For example, had institutions had access to administrative files, they would have been able to examine
individual AHELO test scores with other indicators of student ability, such as GPA. Institutional
information of this nature would have supported a better understanding of the validity of the AHELO
scores in Ontario 11.
Implementing Field Work
Student and faulty recruitment
Student recruitment was a critical aspect of the AHELO feasibility study. Each institution was requested
to identify a sample 200 students to write the test and aim for a 75% recruitment rate. For thoseinstitutions with more than 200 students in the designated area (engineering or economics, or the entire
10

Only ICs were to have access to the student administrative data. The national centre, consortium and OECD
would not.
11
Despite the challenges of gaining ethics approval for this feasibility study, any subsequent assessment of thisnature would likely be accepted without significant modification given additional time to develop ethics
applications and to communicate the goals of the study.

6institution in the case of Generic Skills), ICs were required to provide a sampling frame indicating which
students were identified as potential participants. 12 This purposeful sampling frame was intended toensure that a representative sample of the population wrote the test. In cases where institutions had
less than 200 potential test writers, the goal was to have all students participate as a census. All Ontario
civil engineering programs had less than 200 final-year students 13, so all eligible students 14 wereidentified as potential candidates for the assessment at each institution.
Jurisdictions independently determined their own student recruitment strategies. While some madestudent participation mandatory, others made it voluntary. Those with voluntary participation did notalways incentivize students, though the majority did (OECD, 2013a, p. 169). Student participation in
Ontario was entirely voluntary. ICs recruited students in any way they felt was appropriate, as long as it
was within the guidelines of their ethical protocol. A range of promotional activities and incentives were
offered to students to entice participation, such as posters and information sessions, offers of gift
certificates, prize draws and donations to civil engineering class societies. Recognising that it was
imperative to have sufficient numbers to conduct any type of analysis, student recruitment was by far
the most time-consuming and anxiety-provoking activity for ICs. As student participation was voluntary,
all Ontario institutions ended up with a non-random, voluntary sample.
Faculty participation in the short context survey was conducted in a similar way. If a faculty (orinstitution in the case of generic skills) had more than 40 members, a sampling frame was conducted toensure that a representative sample was recruited. If there were fewer than 40 faculty members, as was
the case in all participating Ontario institutions, a census of all students was attempted. Unable tomandate or entice all faculty members to participate in the survey, Ontario institutions ended up with avoluntary sample of volunteers.
Test administration
The AHELO test system was operational between February and June 2012, and institutions could test
their students at any time during this window. Each institution was required to a run test of thecomputer platform to ensure there were no technical difficulties. The times and dates of test sessions
were reported in advance to the NPM (in case of technical or emergency issues), who reported thisinformation in turn to the consortium to ensure that the online system was prepared to handle theinflux of activity. Institutional coordinators were responsible for organizing the AHELO test session andensuring that the test administrator (invigilator) was suitably trained (based on AHELO training
guidelines).
In Ontario, the timing of the test window was challenging. The academic calendars schedule a one-weekbreak in mid-February followed by mid-term exams and the close of the academic year in April,
preceded by final projects and exams. As a result, a very short period of time remained for test
administration. Most institutions ran their tests in early to mid-March. Institutions offered the test at avariety of times and days, often taking into consideration student class schedules.
12

This sampling framework was conducted under the guidance of the Consortium.
Participating programs had between 34 and 137 final year-students
14
Student were excluded if they were out of the country or had disabilities requiring special arrangements.
13

7

Despite testing the system, one Ontario institution ran into technical difficulties and students wereunable to submit their tests. While the Consortium rectified the technical issue, all but 8% of student
responses were lost at that institution.

Scoring
While the multiple choice questions on student assessments were scored automatically using acomputerized system, each jurisdiction was responsible for the manual scoring of the constructed
responses of its own students. The lead scorers attended two training sessions with fellow lead scorers
to finalize the assessment questions and determine appropriate scoring matrices. This activity ensured
the consistency of marking around the world. The lead scorer, in collaboration with the NPM, wasresponsible for recruiting a small team of scorers and training them in both the test system and thescoring matrix.
In Ontario, a team of six engineers made up the scoring team. The majority of scorers were ICsinterested in reviewing the assessments and the work of Ontario students. Scoring took place over twodays in June 2012 at the National Centre. As part of a memorandum of understanding (MOU) with
Australia 15 , Ontario scored some Australian answers and vice versa. This was conducted as anexperiment in inter-rater reliability between scoring jurisdictions and as a point of interest for thescoring teams (who would not otherwise have seen other jurisdictions’ student responses).

Successes and Challenges of Administration
A primary objective of the AHELO feasibility study was to understand if it was administratively possible
to implement a standard assessment, online, in a common way, to students around the world. Indeed it
was proven to be possible: experts and faculty members agreed on the common learning outcomes andthe assessment questions, and project management and execution of tasks followed a common protocol
around the world. There were minor administrative issues in various countries –some institutions
dropped out, others had very low response rates or technical issues – but overall the administration of
the international assessment was proven successful. The first volume of the AHELO report lays out anumber of international lessons learned from the administration phase of the study (OECD, 2012a, Ch.
6).
There were considerable differences in administrative successes and challenges between participating
jurisdictions, as well differences between the three strands. For example, one country recognised thatthe institutions did not have enough computers for the number of students completing the assessments,
and thus required a travelling van to deliver computers to the institutions. Incredibly, Egypt wasscheduled to implement the test during the Arab Spring, which created significant – though notinsurmountable – challenges. The Generic Skills strand faced challenges in student recruitment.
These implementation issues are critical to understanding the practicality of administering the test – a
primary goal of the study. Hence, a great deal of work has been done in collecting and international
experiences (see OECD, 2013a; Ch. 8 for a description of each nations’ experience).
15

More on the MOU sections below.

8Within Ontario, challenges and successes were both generic to all large-scale testing and specific to the
AHELO context. Generally, student recruitment for low-stakes testing is extremely challenging. It is time
consuming in both the advertising (posters, emails, class visits, etc.) and in the organizing of test
sessions. Furthermore it can become extremely expensive when students are provided with material
incentives, financial or otherwise. Despite the challenges of student recruitment, the institutions were
extremely creative in their strategies, and despite being the most time-consuming activity, it was quite
rewarding for the ICs to see their students participate.
As one would expect in a feasibility study, there were a few hiccups in Ontario. The need to obtain ethics
approval quickly required institutions to modify the assessment framework to reduce the institutional
capacity to link the AHELO results to individual student level data. This reduced the potential value ofthe results for the institutions; at the time of writing, no institution has yet analysed their own
institutional data but have expressed interest in examining it when they receive the comparative
jurisdictional report.
In follow up interviews and surveys student and faculty participants indicated appreciation inparticipating in an international exercise, thereby validating their decision to participate in the study.
This suggests that rather than being fearful of benchmarks or comparisons, there was true interest, fromall levels, in understanding their programmatic characteristics, strengths and weaknesses compared tothose of others in jurisdictions around the world.

Discussion and Conclusions
Analysis of Ontario’s Results
Nine jurisdictions and more than 70 institutions participated in the engineering strand of the feasibility
study. In Ontario, out of the ten institutions that currently offer a civil engineering program, nine ofthem participated in the engineering strand of AHELO. Context information was collected frominstitutions. Across the nine participating institutions in Ontario, there were 155 faculty members and
443 final-year students who participated in the engineering strand of AHELO. Ontario had a response
rate of 72% for faculty and 61% 16 for students. While the demographic characteristics, employment
status and qualifications of the faculty who participated in AHELO were similar across the institutions in
Ontario, there was a noticeable difference in the amount of time faculty reported to spend on teaching
and research. Focusing on the student population within the civil engineering program that participatedin AHELO, there were also differences in involvement characteristics, such as how students spend theirtime preparing for class, attending class, or working in paid work related to or unrelated to field of study.
Please see Lennon, forthcoming C, for more information on the institutional, faculty, and student
characteristics of the participating institutions in Ontario and how they compared to Australia and allnine participating jurisdictions.
16

For institution 9, out of the 36 students that participated in the assessment, 33 of the responses were notrecorded as a result of a technological difficulty with the online system. The response rate for Ontario wasdetermine using the number of students that participated in the assessment rather than the number of actual
recorded responses.

9What does AHELO contribute to our understanding?
Recall that this feasibility study was not intended to provide any comparative ranking data, but rather toexplore the potential for this type of work in both field work and in trialling the assessment tools. Thus,
the potential for digging into to the data was either a) not agreed to in the framework, or b) impossible
due to data limitations discovered later.
Acknowledging that the data gathered must be interpreted with extreme caution, it is impossible tocomment on the results and value they provide to stakeholders (i.e. jurisdictions, institutions, faculty
members and students), as they are unable to presently use the information.
Thus, the feasibility study revealed that the tools need to be refined in order to say what contribution
the information can make. What also became apparent, is that any reworking of the frameworks orassessments further begs the question of who is this information valuable to, in order to tailor it
appropriately.
Let us now consider the value of the feasibility results in two ways:
1. What did the various stakeholder groups hope to gain from participating in AHELO?
2. How could the assessments be tailored to suit the needs of the various groups?
Jurisdictions
At the outset of AHELO it was made clear that the purpose was not for international rankings ofcountries or institutions either internationally or within a jurisdiction. However, the potential forcomparing and contrasting across and within countries/jurisdictions was a significant draw. For
governments, the ability to know how their institutions and programs are organised and examine theimpact on student learning presents valuable information. This has the potential to lead to better
comparative policy understanding, and improve the way systems interact with their institutions.
For a variety of reasons, this was not possible in the AHELO feasibility study, but the potential forsystem-level learning exists. For example, if the information was presented in a way that did not permit
for rankings on simple data points, but instead provided comparable information to peer countries or aninternational average, the potential for international comparisons is possible.

Institutional and program level data
The primary goal of the AHELO feasibility study was to provide information to institutions on how theirstudents performed in various capacities compared to others. Similarly, information collected in thecontext surveys was intended to provide insight into the characteristics of the educational environments
in which students work.
The assessments used in AHELO were not sensitive enough to provide institution level information on
student capacities by specific competency areas 17. Thus, it was not possible to compare institutions on
17

Engineering Design, Practice, and Analysis, and Generic and Basic Engineering skills.

10anything other than the aggregate score of their students. This was disappointing to institutions
participating in the feasibility study, however, with a refined tool it would be possible it develop a test
that compares competency areas of students in institutions. This information would be very useful inproviding an understanding of their strengths and weaknesses. For example, an institution might
discover they are very strong in design and practice, but weaker in basic engineering skills. This type ofdetailed information could provide significant information to program and faculty members when
considering curriculum design 18. Understanding of how they compared to other programs in thejurisdiction or internationally was the primary reason Ontario institutions chose to participate in the
study. Despite not being able to receive this information, the potential to reshape the tests in order to
do it is possible.
The context data collected at the institutional and program level also has the potential to hold valuable
information. Ontario institutions also collect administrative information on faculty and students. Thus,
much of this information is readily available in other sources, and the information on their own learning
environments was not particularly new information to institutions or programs, nor is the withinjurisdiction comparison.
Yet, the point of interest came from seeing the institutions and programs in comparison to others andthe international averages. It could be possible to compare institutions and programs around the world
to see the commonalities and differences in learning environments and understand if they impact
student capacities. Similarly, a pooled analysis of the international trends in program design andenvironmental characteristics could be provocative in providing information on trends of successfulprograms or in providing benchmarking through common structural traits.
Thus, while there is value in the institutional information and comparisons currently provided, greater
insight would be gained from international contrasts, as well as pooled information on global trends.

Student level data
Student-level data collected in AHELO had the possibility to provide information on observable trends instudent demographics and characteristics, and assessment scores.
However, within the feasibility study framework, it was expected that institutions would be provided acomparison of their students’ situations to those in local institutions, and that with that information
they might be better able to support student success. For example, discovering that older students areless likely to be successful - perhaps due to outside employment - institutions could choose to develop
different student support strategies. Providing comparative information might also encourage
collaborations between programs grappling with similar issues. While not provided in the current AHELO
framework, it would be interesting to identify international trends in habits of student success, thus
supporting cooperation and collaboration regardless of jurisdictional boundaries.
18

It has been suggested that proper assessment of competency levels would require a re-design of the assessment,
and would likely necessitate a longer test for students.

11

The tests were not designed in a way to deliver student level feedback, as the intention was to combine
student-level data at the institutional level. However it became clear in discussions that there is value
and interest in providing student-level feedback: a way for students to demonstrate their capacities
compared to their peers. In order to provide student level feedback the test would need to besignificantly altered to ensure each student was writing a precisely equivalent (or same) test.
One aspect that was discussed at length by participating AHELO jurisdictions, the Consortium, and theOECD was the possibility of providing students with feedback on their assessment scores. The
assessment frameworks were not designed in a way to provide reliable scores at the individual level, as
it intended to provide institution level feedback. However, if students received their scores relative toothers (either in the institution, jurisdiction, or internationally), there could be increased interest fromstudents as it would provide a global and objective documentation of their abilities. This, in turn, could
improve student recruitment and student effort in writing the test (creating more reliable information).
This was not the intention of the AHELO feasibility study, but was of significant interest to many
participating jurisdictions. This is one of many possible outputs of this type of assessment, but to doing
so would necessitate a different assessment framework.

Conclusion
The AHELO feasibility study was a successful enterprise in building international relationships, aiding
comparative system-level understanding, supporting institutional/programmatic understanding andexploring the potential of international student-level assessments. A great deal was learned from thisresearch, both within jurisdictions and at the international level of the OECD. We know, for example,
that is it is possible to administer a standard test to students around the world, and it seems there isinterest from a variety of stakeholders, particularly the engineering programs, to do so.
Within Ontario, we benefited from participating in the international conversation on the broader valueof establishing and measuring learning outcomes. As Ontario is grappling with how to develop a systemthat incorporates learning outcomes, understanding how other systems are engaging with them has
been appreciated. It was particularly interesting to see the tension between how various jurisdictions
desire to use international learning outcomes assessments as either system level benchmarks or for
institutional and program level improvement.
As a feasibility study, the AHELO work produced many lessons learned but raised even more questions.
It was found that there is there is interest in international assessments from governments, institutions,programs, faculty members and students. It was also determined that it is possible to have agreementon expected learning outcomes and appropriate assessments from around the world. Furthermore, it
was found that it is possible to test students around the world in compatible ways. These were theprimary questions of the AHELO study.

12

References
EGAD(2013) Engineering Graduate Attribute
http://egad.engineering.queensu.ca/

Development

Project.

Retrievedfrom:

Lennon, M. C. (2010). Signalling Abilities and Achievement: Measuring and Reporting on Skill andCompetency Development. Toronto: Higher Education Quality Council of Ontario.
Lennon, M.C. (forthcoming A). Tuning: Identifying and Measuring Sector-Based Learning Outcomes in
Postsecondary Education. Toronto: Higher Education Quality Council of Ontario.
Lennon, M.C. (forthcoming B). Piloting the Collegiate Learning Assessment in Ontario: Lessons Learned.
Toronto: Higher Education Quality Council of Ontario.
Lennon, M.C. (forthcoming C). The AHELO Feasibility Study in Ontario. Toronto: Higher Education
Quality Council of Ontario.
OECD (2009a). AHELO Terms of Reference. Paris, OECD
OECD (2009b). Roadmap for the OECD Assessment of Higher Education Learning Outcomes (AHELO)
Feasibility Study. Paris, OECD
OECD (2011a). Tuning-AHELO Conceptual Framework of Expected and Desired Learning Outcomes in
Economics. Paris, OECD Publishing.
OECD (2011b). A Tuning-AHELO Conceptual Framework of Expected/Desired Learning Outcomes inEngineering. Paris, OECD Publishing
OECD (2012a). Engineering Assessment Framework. Paris, OECD.
OECD (2012b) Final Engineering Assessment Development Report. Paris, OECD.
OECD (2012c) Contextual Dimension Framework. Paris, OECD.
OECD (2013a) AHELO: Feasibility Study Report: Volume 2: Data Analysis and National Experiences. Paris,
OECD.
OECD (2013b) AHELO: Feasibility Study Report: Volume 3: Further Insights. Paris, OECD.
Tremblay, K. Lalancette, D., and D. Roseveare (2012). AHELO: Feasibility Study Report: Volume 1: Design
and Implementation, Paris, OECD
Tuning Association (2009). A Tuning-AHELO Conceptual Framework of Expected/Desired Learning
Outcomes in Engineering. Netherlands, Tuning Association.

13