Japan-Nordic Symposium

On the Frontier of University Evaluation-Making the Most of Nordic Successes-

Keynote Speech
How university evaluations are being innovated in Europe

Christian Thune

Executive Director, EVA, Denmark＊Slides〔English〕： P. 139～152

21

Kato: Now we would like to move on to the keynote address. The speaker is Dr.
Christian Thune, Executive Director, EVA. He is going to talk on the topic of “How
University Evaluations Are Being Innovated in Europe.”
I would like briefly to introduce Dr. Thune’s personal background. Dr. Thune completed
a doctoral course at University of Copenhagen, and his area of academic specialization
is history. He has held important positions in quality assurance and academic
evaluation in Denmark, and has also served as a member of the Board of INQAAHE,
the International Network for Quality Assurance Agencies in Higher Education. In
addition, from 2000 through 2005, he served as the President of the European
Association for Quality Assurance in Higher Education, ENQA and played a leadingrole in European higher education, which has seen dramatic changes including theSorbonne Declaration and Bologna Process. He has also served as the Chair of theNordic Quality Assurance Network in Higher Education, NOQA. His current position as
Executive Director of EVA allows him to continue his leading contribution to issues ofquality assurance in higher education across Europe.
Dr. Thune, please.
(Slide1)
Christian Thune: Thank you for the presentation, and Mr. President, ambassadors,
representatives of the ministries, universities and other stakeholders, I am of course
honored to have been invited to speak at this highly-relevant occasion of the symposiumon Nordic-Japanese experiences in evaluation of universities.
(Slide2)
I have been asked to cover in my presentation developments and innovations at theEuropean, Nordic and Danish level, which certainly makes for a very comprehensive
effort on my part, and part of the journey I will be taking you through during the next
half an hour or so will be, more than anything, a helicopter tour a certain distance above
the many highly interesting details at the ground level. But I hope I will be able toprovide you with an idea of where the relevant developments and innovations aretaking place these years.
So I’ll start out by presenting the European and go on to the Nordic in the formal shapeof NOQA and end up with a few words on my own national’s quality assurance system
in Denmark.

23

(Slide3)
As it is, and that will be the conclusion I’ll be moving you forward through in terms ofwhat have been the developments in Europe, is that they have to a remarkable degree
been successful, but as we all started out in ’92, we weren’t that many in ’92, actually,
the British, the French, the Dutch and the Danes were first in place. I’m the only
survivor from ’92; there is an attrition rate in my business which is impressive. But as
we began already at that time to discuss frameworks for cooperation, exchange of bestpractices, there were distinct positive indicators for the future of European cooperation.
We were as a rule all government initiated and owned. Actually I know of not one single
European case where the universities were the initiators of external quality assurance;
it was a government process. But we were independents as agencies in terms of our
choice of methods and processes. We all applied the four-stage model, as we called it,
based on independent agencies, self-evaluations on the part of the higher education
institutions, site visits by experts, and finally, a public report. And that was more thananything what kept us together, this shared approach, in terms of basic methodology.
We also were all in the process of setting up regular, systematic cyclical review
activities in our national systems. We were positive towards the idea of getting the mostout of sharing best practices with our European colleagues. We had very strong back-up
from the European Commission, not least in terms of funding of traveling andconferences, much the same function which the World Bank has taken upon itself in
relation to the Asia-Pacific Quality Network. And this is important because one of the
major developments was that is the mid-nineties we had the European pilot projects
covering all the European Union countries, introducing the basic strategies andmethodologies also in those countries where national systems were not yet in place.
(Slide4)
There were also negative indicators. National approaches were still fragmented,
basically because in Western Europe we were improvement oriented, basically applying
evaluation strategies, whereas Central and Eastern Europe, after the democratization
process, oriented themselves towards more accountability accreditation processes.
There was what I call a considerable variance in professional capacity-building, andbehind that is that some of the agencies, especially in Western Europe, from the start
became sufficiently professional in terms of having the staff necessary to develop thecredibility and consistency of processes, whereas others, especially in Central andEastern Europe, did not possess from the start that professional capacity.

24

We had language and terminology problems. No wonder. Even basic terms such as“evaluation” had their various associations in the different countries.
And, not to forget, basically, we all had to face in the early nineties a general skepticism
and mistrust from universities towards these new strange ideas of external quality
assurance.
(Slide5)
But as we move ten years forward to 2002, we had national agencies in place in almost
every European country. My agency did a survey for ENQA and could identify at thattime that there were five top approaches in terms of methodologies. Accreditation ofprograms, somewhat to our surprise, was the winner, but that was because the Central
and Eastern European agencies all adapted accreditation methodologies. So evaluationof programs came in as a strong second, which was at that time basically the Western
European and Northwestern European especially approach. And audit of institutions
were beginning to come in strongly. I’ll return in a while to the concept of audit and let
further definition be for the moment.
(Slide6)
Then we had two formal processes which became extremely important in achieving theresults we have today in 2006. One was the Bologna Process, already mentioned by the
Finnish ambassador. This became an extremely important catalyst for European
developments. It started out with another governmental meeting in Bologna in ’99. And
I can give you the little anecdote that I was invited by the Danish government to go
along because there was something about quality assurance on the agenda, but I calledback and said, “Okay, just another governmental conference; you’ve seen one, you’veseen them all.” So I actually didn’t go to Bologna. That was a major misjudgment on mypart certainly, because what started in Bologna was very ambitious but became
eventually a process with fundamental implications for the participating European
countries, of which there is, I thought it was 44, but I trust the Finnish ambassador;
there are now 45.
The important themes in terms of what we are discussing here today was that from ’99
there was a distinct argument that what we needed was transparency, compatibility,
comparability and flexibility of European higher education, serving both the needs ofstakeholders, students, employers, whoever, in being able to discern what is credible
quality and to make European higher education fully adapted to the challenges of the25

globalization of higher education.
As the process moved on there became obvious and increasing focus on quality
assurance of higher education as a key issue, and along with that the key stakeholders,
university students, teachers, employers and society in general, also became fairly
acutely aware of the challenges of having a closer look on the setup of quality assurance
in the participating countries.
(Slide7)
The other formal development was the establishment of ENQA, the European
Association for Quality Assurance in Higher Education, which was formally set up
in ’99, but based surely on the informal network in sharing of best practices that had
started already in the early nineties. But by that time it became obvious that there wasa need for formalization. Membership soon began to grow rather drastically. There aretoday 42 member agencies. I was honored by my European colleagues by being elected
as the first President of ENQA until I stepped down a year ago, and part of my task wasin a sense to move the corroboration from the basic and original objective of sharing anddisseminating knowledge and best practice of quality assurance towards taking uponourselves the increasingly important political role and implications of being aprominent participant in the Bologna Process, and along that, the European Union
political process as well.
(Slide8)
And what I’m telling you here became certainly very obvious when the ministers in the
Bologna Process, who meet every other year, when they met in Berlin in 2003 they
invited ENQA in cooperation with institutions of higher education and the student
associations, to develop an agreed set of standards and guidelines for the quality
assurance of higher education as one task, and the other task, to set up an adequate
review system for external quality assurance agencies. And this to you may seem simple
enough as a task. I can tell you that the little term “agreed” made the next two years
rather heavy in terms of added workload for me and my colleagues in the ENQA board
because we had, in less than two years, till the next ministerial meeting in Bergen, to
set up these standards and guidelines. But we did succeed and it’s still to be regarded asquite an achievement that when the ministers met in Norway in Bergen in May 2005,
ENQA could present the report on “Standards and Guidelines for Quality Assurance inthe European Higher Education Area.” It had already been endorsed by ENQA
members and the European partners, and our standards and guidelines were adopted
by the European ministers. This was a highly important occasion because the26

implication of course is that we now have standards and guidelines for which there is aneed for all the European participants in the process to ensure that their individual
national quality assurance systems comply with the highest degree possible.
(Slide9)
I’ll try to give you an idea of the implications and content of the European standards
because I’m sure that we can agree that this is, as I say, a fairly crucial state in thedevelopment of European quality assurance.
We have now agreed European standards for the internal quality assurance within thehigher education institutions and for the processes adopted by the agencies. Thesestandards will now become the common reference points for higher education
institutions and their agencies. We have a system where it is obligatory that quality
assurance agencies themselves are reviewed at least every five years, and we will now
develop a so-called European register of quality assurance agencies. I’ll return to theimplications of that in a moment.
But one important point before I proceed is surely that this is not the end of a process;this is very much the beginning of a process, and to that extent not necessarily a very
easy process.
(Slide10)
Now for a few details in terms of the standards. The standards for the higher education
part I’ll cover as well, as I’ve said, internal as well as external quality assurance, andthey are applicable to all higher education institutions in the Bologna signatory states,
which, by geographical implication, is a territory going from Ireland to somewhere
pretty close to the Ural Mountains in the former Soviet Republic.
The purpose is to improve the higher education available to students, highly important
to assist higher education institutions in managing and enhancing their own quality,
and to form a background for the work of the agencies.
A few words on the standards problem, and I give you only the headings. Behind theheadings is the elaborate text and the guidelines, and I’ll give you in my final overhead
the links which will make it possible for interested Japanese to go into the proper text of
the ENQA report.
(Slide11)
The standards for internal quality assurance, as I say, these are the standards which27

universities now should apply for their internal processes, and so they must have inplace, when agencies come up through a university for a review, the university must beable to identify that they have policy and procedures for quality assurance which areevident in what you could call a quality culture in a university, and on that basis thatthey do, as a university, approve, monitor and review their own programs and rewards.
They must identify the criteria for the assessment of students that are appliedconsistently at the university and are in the public domain. They must identify theprocedures through which the university makes sure that the teachers of the students
are themselves qualified and competent for the work that they are doing.
They must provide documentation that there are adequate and proportional resources
available for the teaching and learning and the report of students. That they have, andthis is one which will cost a lot of extra work in many European universities, they must
be able to identify the information systems through which they collect, analyze and use
information for managing their programs and rewards. I can tell in this close circle thatnot many European universities have at the moment that kind of information system inplace.
And finally, and not least important, they must testify that in terms of public
information

they

provideup-to-date,

impartial

andobjective

information

tostakeholders on the quality and content of the programs and awards.
(Slide12)
The standards for the external quality assurance, these then by implication are thestandards supposed to be applied by agencies as they conduct their reviews ofuniversities. Firstly, of course, external reviews must make sure that they do in fact
look into the extent and quality of the internal quality assurance mechanisms of theuniversity. They must develop adequate external quality assurance processes. The
criteria for decisions taken on quality by the agencies must be based on publishedcriteria that are applied consistently. The processes must be fit for the specific purposesof the quality assurance. And there must be reporting on the results of quality
assurance and reported in a form which is readily accessible for the public.
There must be follow-up procedures for the quality assurance results, which arepredetermined, and again, applied consistently. The reviews must be periodic or
systematic. And finally, agencies must on a regular basis provide more system-wide28

analysis where they make reports that provide summary results of the reviews they
have been doing within the higher education sector.
(Slide13)
As I told you, external quality agencies are themselves now supposed to be reviewed atleast every fifth year. And the background is that there is a growing number ofEuropean external quality assurance agencies and, at the same time, operators fromoutside Europe, especially American professional agencies, such as ABET in theengineering field or some of the regional U.S. accrediting commissions. So we have as atarget group those quality assurance agencies primarily that operate in one or more
Bologna signatory states, and the idea of course is to make visible the professionalism,
credibility and integrity of the agencies.
(Slide14)
The standards applied for the agencies are that a review must identify what quality
assurance procedures are in place at the agency, that the agency is formally recognized
by a competent public authority, that its activities are regular, that is has resources forthe quality assurance work which are adequate and proportional with the task both inhuman and financial terms. And this again may seem easy enough but I can tell youthat quite a number of the European agencies, especially in the former Soviet Republic,
have distinct problems with that specific standard.
There must be a clear and public statement of the mission of the agency. And we comeinto the independence issue which, in terms of the necessary credibility for agencies, isa very essential one. That definition of independence applies in the standards is apragmatic one, considering the fact that most agencies are government-funded; we haveto accept that as a fact of quality assurance life. But the independence must be
established in the sense that the processes and methodologies applied by the agencies istheir autonomous domain and that no third party is able to interfere with theconclusions and recommendations in reports.
The criteria and processes must be published and in principle discussed with the higher
education institutions before they are settled.
And finally, and not least important, quality assurance agencies must have their own
accountability procedures in place, implying that they must be prepared to undertake or
to undergo these external reviews, and external reviews which, by the implication ofwhat I’m telling you, primarily will focus on the compatibility of the agencies with the29European standards. But agencies are also supposed now to have mechanisms thatprovide consistent and regular feedback from the universities and their experience ofthe quality assurance from stakeholders and other interested parties so each agency
will have a running, viable feedback system in terms of the quality of its activities.
(Slide15)
I told you that we are going to have a European register of quality assurance agencies,
and this is a difficult task to give you in a few sentences, the idea of the register, but I
can give you the short version which is that in Bergen the ministers invited ENQA andpartners to set up a European register as a proposal for their next meeting in London in2007. And the idea would be that we will have a register which will provide recognitionof the various agencies in terms of compliance with European standards and make itpossible for higher education institutions that are looking for qualified quality
assurance agencies to find the necessary information in the register. But I’ll be brief on
this because setting up the register is turning out to be a fairly complicated politicalprocess.
(Slide16)
So what are the main challenges for success in the European context: firstly, thewillingness of governments, higher education institutions and the agencies toimplement the European standards. At this time, one-and-a-half years after Bergen, thewillingness is impressive and I am very favorably impressed by the extent to whichgovernments, agencies and universities are taking this task very seriously in mostcountries.
There must be a balance between the national subsidiaries, which is the European term
for the concept, that what can be done satisfactorily at a national level we do not need tomove into a European level. And that’s a very important balance.
We must face that there are different interests and goals between governments,
agencies, institutions, students and other stakeholders, and these differences must now
to a last degree be reconciled in terms of fulfilling the ambitions of the Bologna Process.
And finally, there is, again to mention briefly, the register, the risk that this becomes apolitical rather than a professional instrument, and what I’m hinting at here is that theEuropean Union has the ambition that the possibility that the register should be a very
strictly European register only for European agencies compatible with the standards,
whereas the original idea was for a more inclusive register which also provided

30

information on non-European agencies. So much about the European dimension inquality assurance.
(Slide17)
Now a few words on the Nordic quality assurance cooperation, which today has theformal framework of NOQA. We did meet annually since ’92. Actually, I invited in theautumn of ’92 Nordic colleagues to a meeting in Copenhagen. At that time only theDanish agency was formally in place, but we began to exchange ideas of how to bringabout the best and most credible processes in the Nordic countries.
So by 2003 we decided that this informal cooperation should have a more formal
expression, not least because we were so heavily contributing to the overall ENQA
process, so we set up NOQA as a regional network under ENQA. Still the level offormalization was not very high. We have a rotating chairmanship—Sweden has thehonor of being chair of NOQA for this year—but apart from that we have not made an
over-bureaucratic and formalized network.
And I should add that in contrast to the European process where ENQA has had toaccept a very strong linkage to the European political process, we have no substantial
links to the Nordic political processes, which gives us the freedom of operation which I
personally find attractive.
(Slide18)
So what are the indications for Nordic networking? The positive ones are that allagencies are harmonious in the sense that they are national and they are government
initiated. We share the culture and tradition for Nordic cooperation. We have the added
pleasure that Nordic universities are generally positive or very positive toward theconcept of quality assurance. We share the language, especially the Scandinavian
countries, Norway, Denmark and Sweden. We share basically a language which makes
it possible to exchange experts. We have all of us a clear focus on the need to combine anaccountability and improvement-oriented approach and we share basically the same
methods, even in individual applications.
A negative indication is that we still, if you look closely, are remarkably different in our
distinct setup in terms of agencies and the processes. A primary reason here to my mind
is that when Nordic governments decide to change their quality assurance setup forhigher education, there is surprisingly little consideration of in what way the similar
processes are working in the Nordic countries. I think this is one of the problems we will31

have to tackle in the future.
(Slide19)
The main methods of quality assurance, just to give you the standards, the definitions ofevaluation, that’s the review of the quality of a program or institution with a focus on
input, output and processes. Audit is a review of the quality culture and internal quality
assurance of a university. That’s the definition of audit. You will I’m sure be perceptive
enough to realize that the European standards for internal quality assurance are very
closely linked to the audit concept.
And we have accreditation. That’s the review of the extent to which an institution or aprogram fulfills predefined standards and resulting in a formal “yes” or “no” decision oncompliance.
(Slide20)
The activities of NOQA, we have our annual conferences and at each annual conference
we decide on a joint project, which is very important. The agencies contribute one or two
staff members each to a joint project where we set up and report on a relevant theme forour mutual efforts. And as you look down the list you can see that the way in which wehave handled these joint projects has been very much sort of on the forward edge ofEuropean development. Actually, the first one was on the accreditation-like practices
which were at that time beginning to spread into Western Europe. We have had student
involvement. We have done last year a European study on the extent to which theNordic agencies are already compatible with European standards. And now this year weare moving into audit as a concept and a method. So no surprise.
(Slide21)
The third dimension of our results is that we have been able to contribute the results ofwhat we have doing into reports for ENQA as a whole. We have had a substantial
amount of staff exchanges; a Norwegian or Swedish staff member comes to Denmarkand stays at my agency for a period and discusses best practices and perhaps even gets
some new dimensions.
Very important is the element of consultation and clearing mechanism for external
experts. In all the reviews we do in Denmark, we use in the panel of experts at least one
expert from one of the other Nordic countries, sometimes two. So it’s highly importantthat we have viable consultation and clearing mechanisms.
And finally, I can say in all modesty that the Nordic countries have been able to work32

together in terms of being a very strong element in ENQA developments and policies.
(Slide22)
So what are the challenges? One challenge is because ENQA is moving so strongly,
because the political European process is demanding for the agencies, we must strive tomaintain still the Nordic dimension, and on the other hand, finding a good balance
between increased Nordic cooperation and convergence in quality assurance on the one
hand and the fact that we do have these national differences in policies for educationand quality assurance systems on the other.
(Slide23)
This brings me finally to the external quality assurance in Denmark of higher education.
We started out in ’92 with a center only for higher education which did a full cycle ofreviews and programs which became, again in all modesty, quite a success because theuniversities became in general acceptable of the concept we were offering.
So by ’99 we were almost overwhelmed by our own success because the parliament
decided that we were going to become a new institution the Danish Evaluation Institute,(EVA) with a mandate covering the whole educational system from pre-schooling touniversities. So EVA has been faced with a very comprehensive and demanding task.
We are independent, and independence more than anything is significant because wehave the right of initiative. No one, for example neither the ministry nor universities,
can tell us what reviews we are going to do and in which order.
(Slide24)
Our tasks, as you can see from the overheads if you read fast enough because I won’t
give you much time, is to make sure we are actually conducting external quality
assurance and having a dimension which has a research implication, development ofmethodologies, collecting on national and international experience. This has been ahighly relevant part of our mandate, which again is based in the law, and has given us
the possibilities for also taking upon us a fairly extensive international role.
(Slide25)
In the first phase from 2000, it was in a sense the experimental phase. We had follow-up
on our program reviews from the nineties. We did thematic review, drop-out rates,
career counseling in universities. We used our mandate to follow specific subjects
through the whole educational system, for instance, teaching and training in English
from primary school to universities. We have been doing cross-national reviews. The
last one finished this week is cooperation with the British Quality Assurance Agency for

33

Higher Education (QAA) on chemistry where the Danish departments have beenreviewed together with a number of British departments and the results from the twonational contexts combined in and compared in one report.
We do international bench markings, audits, accreditation, and we have a substantial
amount of revenue-generating activities.
(Slide26)
So this adds up to a second phase where we have now a system which is a combinationof audits at the university level of the 12 Danish university-level institutions. We aremoving there from national/Nordic panels into international panels. The implication ofthis is interesting: the audits are being conducted in English and the universities haveaccepted to provide the documentation in English, which adds substantially to theirwork effort. And we have moved from a fairly open format of audit into basing the audit,
no surprise, on the European standards.
We have selective criteria-based reviews of university programs where we take on these
programs where we perceive there may be a quality problem. And finally, 50 percent ofhigher education, which is the medium cycle on non-university higher education, theprofessional training programs, there we systematically certificate all programs.
(Slide27)
So what are the key challenges for Danish success? Firstly, I would mention that weshould be very careful in how we manage our very comprehensive mandate. And we dohave a coverage problem because I can tell you when we got the daycare institutions,
there are 6,000 even in a small country like Denmark, so we are going to look very
carefully into the methodologies which give credible coverage.
We have to continue to find a good balance between the original improvement focus of
our activities and the strong element of accountability coming into the life of most
European agencies, also the Nordic agencies, and certainly the Danes, because
governments have begun to become very fond of quality assurance procedures that havea distinct accountability focus. By implication, accreditation is moving in stronger.
We must find a good balance between the institutional and the program level. You
cannot concentrate your activities, neither on looking at the whole of a university,
ignoring the quality of the partial elements, that’s the programs, nor can you on theother hand only look at the programs and ignore the quality of the university as a34

whole.
We have, as I’ve said, a distinct acceleration in our profit-generating activities, that is,quality assurance reviews which are paid for by ministries, universities, local
authorities and which we take on alongside the activities funded by the taxpayers. Thisputs a strain on the organization in the sense that I have to expand my staff accordingly,
but basically on the fragile basis of our position in a market for this kind of activity.
I put in the information strategy toward the media, the authorities and the educational
institutions because it is so important. We give a distinct priority to make the results of
our activities known to the public, who in their shape of taxpayers basically fund theDanish institute, but we have the problem of course that the way to the public isthrough the media. What is relevant for the media in terms of quality assurance
processes? The sad bad stories of no quality, which gives us a problem because whatleads to follow-up in the universities? Reports which are fair and do not create a trenchmentality in those universities. So that is a continuing effort to achieve the rightbalance with the media.
And of course finally the compliance with European standards. This is an extremely
important issue. We had our external review in 2005 by the Swedish agency and it has
now been recognized by the ENQA board as making the Danish agency compatible with
European standards, so we have been one of the first agencies to take the path which all
the other 40-plus European agencies must now take, and it has been for us a very
rewarding and developing experience.
(Slide28)
So this is what I would like to give you today as an idea of development and innovation
at these three European levels. You’ll have the overheads and my abstract which will
give a little more detail. Apart from that, I invite you to go into the links for the threelevels: the ENQA link, the NOQA link and the EVA link for the Danish case. And as I’ll
use some of the time given me by the president and the Icelandic ambassador, I’m surethere is time available if any of you wish to pose a question to me in relation to what I
have been telling you, then you are very welcome. Thank you for your attention.
Kato: Dr. Thune, thank you very much. There will probably be some questions for you
during the course of subsequent proceedings, and we may need to call on you again then.
Thank you.

35