Georgia Institute of Technology
From the SelectedWorks of Diana Hicks

2012

Performance-based university research funding
systems
Diana Hicks, Georgia Institute of Technology - Main Campus

Available at: https://works.bepress.com/diana_hicks/27/Performance-Based University Research Funding Systems1

Please cite this article in press as: Hicks, D., Performance-based university research funding systems. Res. Policy (2011),
doi:10.1016/j.respol.2011.09.007
Diana Hicks
School of Public Policy
Georgia Institute of Technology
Atlanta, GA, USA, 30332-0345
dhicks@gatech.edu
Abstract
The university research environment has been undergoing profound change in recent decades and performance-based
research funding systems (PRFSs) are one of the many novelties introduced. This paper seeks to find general lessons
in the accumulated experience with PRFSs that can serve to enrich our understanding of how research policy andinnovation systems are evolving. The paper also links the PRFS experience with the public management literature,
particularly new public management, and understanding of public sector performance evaluation systems. PRFSs
were found to be complex, dynamic systems, balancing peer review and metrics, accommodating differences
between fields, and involving lengthy consultation with the academic community and transparency in data andresults. Although the importance of PRFSs seems based on their distribution of universities’ research funding, this issomething of an illusion, and the literature agrees that it is the competition for prestige created by a PRSF that creates
powerful incentives within university systems. The literature suggests that under the right circumstances a PRFS will
enhance control by professional elites. PRFSs since they aim for excellence, may compromise other importantvalues such as equity or diversity. They will not serve the goal of enhancing the economic relevance of research.
Keywords
RAE, ERA, REF, university, research, funding

1

This article is based on "Overview of models of performance-based research funding systems" originally published by OECD in
OECD (2010), Performance-Based Funding of Public Research in Tertiary Education Institutions, OECD Publishing,
http://dx.doi.org/10.1787/9789264094611-en.

11 Introduction
The Research Assessment Exercise (RAE) in the UK was launched in 1986, since then many countries have followed
suit and introduced performance-based research funding systems (PRFSs). At least fourteen such systems were found in
2010. Such widespread adoption represents a shift in national research policies of some significance since universities are
so central in national innovation systems. Going forward, to understand the evolution of national innovation systems one
must address the design and execution of PRFSs. To advance our understanding of research policy and its effect onnational innovation systems, we must build an understanding of PRFSs in general. This necessitates stepping back fromthe intricacies of individual systems and taking a broad perspective on PRFS characteristics and the challenges they face
as instruments of research policy.
PRFSs are highly intricate, dynamic and embedded in national research systems; therefore any general understanding
will only be possible through the collective effort of scholars who understand each system in depth. As experience with
PRFSs has grown, along with the salience of the exercises for funding and success, a rich tradition of critique anddiscussion has emerged, particularly in the UK and Australia. This literature is nation-specific and usually disciplinespecific. Building on the national literature, Geuna and Martin (2003), Dolan (2007) and the European Commission(2010) took an international perspective offering comparative descriptions of various systems. This analytical review
attempts to further this collective effort by identifying common themes emerging from in-depth analyses of individual
PRFSs.
Note that understanding the literature on PRFSs requires acknowledging the dual identity of university research. On
the one hand, university research is part of the larger enterprise of the university and is shaped by university governanceand university-related policy making. On the other hand, university research is a substantial element of every nationalinnovation system, and so is of concern to scholars of innovation and to governments seeking to enhance theinnovativeness of their economies. These two perspectives on PRFSs differ somewhat. The higher education literature
treats the research mission of universities in a somewhat sketchy fashion towards the end of documents that are mainly
concerned with accreditation, completion rates, harmonisation, etc. The innovation literature tends to ignore theeducational mission of universities and the changes under way in the allocation of the teaching component of university
funding. Although this paper takes the innovation perspective for the most part, the analysis draws from the higher
education literature a framework for understanding the introduction of PRFS as part of new public management, amovement to reshape government.
This overview of PRFSs begins by defining and identifying existing systems, even systems that have not yet been
discussed in the English language literature. The rationales underpinning PRFSs will be explored from the scholarly
perspectives of the innovation and higher education literatures as well as from the perspective of policy makers. A
typology of PRFSs is developed and then common challenges are identified. Issues of autonomy and control reoccur
regularly in PRFS discussion and these are discussed before final thoughts on the future of these systems are offered.2 I
am guided throughout by the question: Are there common lessons for innovation policy to be drawn by stepping backfrom the details of individual systems to frame university performance based research funding systems more generally?
The basis of the analysis is a literature review conducted in early 2010 that sought to identify all known PRFSs and drawout as much detail on their operation as possible in order to compare the rationale and design of PRFSs and their relation
to funding systems.
2 Performance Based Research Funding Systems defined and listed
Research is evaluated in many places, by many types of organizations, for many different purposes. This heterogeneity
can confound attempts to produce a coherent review. To avoid this, I begin by narrowly defining the PRFS and listing
systems that meet or will meet the narrow definition. Systems included in this review met the following criteria:
 Research must be evaluated. Evaluations of the quality of degree programmes and teaching are excluded.

2This review does not attempt to comprehensively assess the effects of PRFSs on universities. For recent reviews of effects see
Whitley and Martin (2010) and L. Butler (2010).

2

 Research evaluation must be ex post. Evaluations of research proposals for project or programme funding
are ex ante evaluations and are excluded.
 Research output must be evaluated. Systems that allocate funding based only on PhD student numbers andexternal research funding are excluded.
 Government distribution of research funding must depend, or will soon depend, on the results of theevaluation. Ex post evaluations of university research performance used only to provide feedback touniversities or to the government are excluded.
 It must be a national system. University evaluations of their own research standing, even if used to inform
internal funding distribution, are excluded.
The first PRFS was introduced in 1986 in the United Kingdom - the Research Assessment Exercise (RAE). Use of
PRFSs has expanded since, and by early 2010, 14 countries were found to have launched PRFSs that conform to theabove definition or to be planning to launch a PRFS (systems in the design phase are included). These systems are listed
in Table 1 which reports the country, the name of the system, year the system was introduced and the ministry or agency
responsible. In general PRFSs are run by Ministries of Education or Ministries of Education and Research. In a few cases- Flanders in Belgium, Italy and Spain - dedicated agencies were established to run the evaluation, and in Australia the
Research Council runs the system. The systems are not static. Most have undergone major redesign. In fact new versionsof seven systems are under discussion or in the first stages of rollout. This means that it is important to be aware of theversion of a system when reading about a PRFS.

3

Table 1. National performance-based research funding systems for universities
Country

SystemUnited Kingdom

RAE moving to REF – research
excellence frameworkSpainsexenio

Slovak RepublicHong Kong, China
Australia

PolandPortugal
Italy

New Zealand
Belgium (Flemish Community)
NorwaySwedenDenmark
Finland

RAE
Composite Index, Research
Quality Framework (RQF),
Excellence in Research forAustralia (ERA)parametric evaluation

Year implemented/majorrevision
1986/current

1989
1992/2002
1993
CI - 1995/ERA - 2010 (nextscheduled for 2012)

1991/1998-99

research unit evaluation
Valutazione Triennale della
Ricerca (VTR)/ Valutazione
Quinquennale della Ricerca(VQR)
Performance-based researchfunding (PBRF)
BOF-key
Norwegian model (new model
for result-based university
research funding)
New model for allocation of
resources

1996
Evaluated 2001-03 in 2006
(VTR),
Evaluation of 2004-2008 (VQR)
to be conducted
2003/current

Implementation of theNorwegian model
Funding formula for allocation ofuniversity resources

Current

Agencyformerly Higher Education Funding
Council for England (HEFCE), now
Department of Business, Innovationand Skills
National Commission for Evaluationof Research Activity (CNEAI)
Ministry of Education
University Grants Committee
Australian Research Council (ARC)

Science Council – advisory body to
the Minister of Science and Higher
Education
Science and Technology Foundation
Agency for the Evaluation of
University System and Research(ANVUR) / CIVR
Tertiary Education Commission

2003/2008
2006

Steunpunt O&O Statistieken (SOOS)
Ministry for Research and Education

2009

Ministry of Education with somemethodological support from theSwedish Research Council

1998/2010

4

Ministry of Education

The universality of evaluation combined with the variety in its implementation means that a number of countries
have related systems or have discussed such a system but did not meet the criteria for inclusion. In France, thegovernment established AERES, the Evaluation Agency for Research and Higher Education, to conduct comprehensive
evaluations of university degree programs and research in response to the Bologna process. The peer review evaluations
involve site visits and dialogue with the aim of improving the quality of the university system. In Germany the federal
government seems to be pursuing some goals of performance-based funding, such as international excellence, byawarding large centres of excellence to universities based on assessment of proposals (CHEPS Germany, 2007, 37-38).
The Netherlands implements an evaluation system based on peer review which is not used to inform distribution offunding (Geuna and Martin, 2003). In South Africa the Foundation for Research and Development conducts evaluationsof individuals' research record. Peer-reviewed grants of quite lengthy duration are awarded based both on this ex post
evaluation and on peer review of proposals.
3 Rationales
The rationale of performance funding is that funds should flow to institutions where performance is manifest:‘performing’ institutions should receive more income than lesser performing institutions, which would provide
performers with a competitive edge and would stimulate less performing institutions to perform. Output should
be rewarded, not input. (Herbst, 2007, p. 90)
Although seemingly straightforward, the motivations behind PRFSs do contain a few nuances. Broadly speaking,different parties evoke seemingly different types of rationales when explaining the introduction of national researchevaluation systems. Those primarily concerned with understanding research and innovation tend to speak of globalisation,competitiveness, and the knowledge economy. Those who study higher education and who are mainly concerned witheducation, accreditation and the like tend to trace the introduction of these systems to the spread of the tenets of new
public management. Discussions of the introduction and evolution of individual systems tend to take a historical turn,
focusing on changes in government, ministerial actions and stated rationales of particular governments (JiménezContreras et al., 2003; Sanz-Menendez, 1995; Tapper and Salter, 2003). National governments themselves seem touniformly aspire to achieve research “excellence” through their PRSF. Of course, each approach addresses the same
phenomenon from a different angle. For example, Kettl traces the widespread introduction of new public management
reforms to a shared set of political, social, economic and institutional challenges faced by many governments. These
include the shift from the industrial economy to the information economy and globalisation (Kettl, 2000).
An example of the global competitiveness argument can be found in the introduction to the European Commission
report, Assessing Europe's University Based Research:…research performance is widely regarded as being a major factor in economic performance. Because of theirinterlinked roles in education, research, and innovation, universities are considered key to the success of the
Lisbon Strategy with its move towards a global and knowledge-based economy. Improving the capacity andquality of university-based research is thought to be vitally important for innovation, including social
innovation…
The economic dimension of (university-based) research in terms of expected economic and societal benefit andincreased expenditure goes a long way to explain the heightened concern for quality and excellence in research,
for transparency, accountability, comparability and competition, and for performance indicators andassessment. (European Commission, 2010, p. 9)
A very similar statement can be found on the UK government website introducing the new REF (Research
Excellence Framework):
Through the REF, the UK funding bodies aim to develop and sustain a dynamic and internationally competitive
research sector that makes a major contribution to economic prosperity, national wellbeing and the expansion
and dissemination of knowledge. (HEFCE, 2011)5

In contrast, higher education authors often view the changing relationship between universities and their government
in the context of new public management reforms (for example, Herbst, 2007). Donald Kettl is a leading analyst of new
public management, and in a nutshell his view is the following:Over the past quarter century, governments around the world have launched ambitious efforts to reform the way
they manage. Citizens have demanded smaller, cheaper, more effective governments while asking for moreprograms and better services. To resolve this paradox, governments are experimenting with many ideas toimprove performance and production and to reduce costs… Reviewing the standard strategies and tactics
behind these reforms, [Kettl's The Global Public Management Revolution] identifies six common core ideas(Kettl, 2005):







greater productivity
more public reliance on private markets
a stronger service orientation
devolution to sub-national government
increased capacity to formulate and evaluate policy
enhanced accountability

Performance-based funding systems for public research in universities can be interpreted in light of these six coreideas (Kettl, 2000):

Increasing productivity: Output-based evaluation aims to increase research output without adding researchresources to the system, thus increasing productivity.



Replacing traditional command-and-control systems with market-like incentives: In many nations, universities
were not autonomous and were often little more than extensions of their ministry. In Whitley's terms they were“administrative shells” (Whitley, 2008, p. 12), or as Herbst describes it “government agencies were practically
distributing funds down to each individual faculty member… Until recently, we may claim in jest, a typical
Rector's office just had a yearly budget to pay for various banquets” (Herbst, 2007, p. 87). Universities had nodiscretionary budget and did not control hiring, tuition, student numbers, etc. The shift to performance-based
funding is part of a broader movement to make universities more autonomous and introduce more strategic
university management. This also involves competition for funding - the market-like side of the reform.

Stronger service orientation: This dimension refers to increased attention to the needs of citizens. In the researchworld this would be analogous to giving more weight to serving the needs of the high-technology economy andless to the self-governed programmes of work of the community of scholars subject only to peer validation(Marginson, 1997, p. 69).



Devolution: The idea here is that programmes are more responsive and effective when managed closer to theprovision of services. In the university world this means making universities autonomous, strategically managed
Entities rather having all key decisions taken by ministries.



Formulating policy: This refers to a shift from formulating policy and delivering the service to formulating
policy and contracting for the service. The government as purchaser of “education services” was explicitly
articulated in the Australian context (Marginson, 1997, p. 71).

Enhanced accountability: This means focusing on outputs and outcomes rather than processes and structures.
Measuring research output and distributing funding on the basis of results is clearly meant to enhanceaccountability compared to the prior method of basing funding on numbers of faculty.

Note that although implementation of a PRFS is related to new public management, it is not an exemplar. Importantnew public management characteristics such as efficiency gains and increased use of contracting play little or no role in
PRFSs.
6

The rationales provided by individual governments for their PRFSs are worth examining briefly because they reveal
national differences in priorities due to the pre-existing state of university research in each country. In the United
Kingdom, greater selectivity in funding allocation was initially an explicit goal of the RAE. Selectivity was a government
response to limited resources and the increasing costs of research. The goal was to maintain research excellence but at
fewer places (Tapper and Salter, 2003). Sweden also looks to concentrate resources, believing that international
competition requires concentration and priorities to maintain high scientific quality (European Commission, 2010). The
goal of the Spanish sexenio as stated in law is to foster university professors' research productivity and improve thediffusion of this research both nationally and internationally. Fostering international publication was an explicit goal of
Spanish science and technology policy more generally, and overall the policies have been successful (Jiménez-Contreras
et al., 2003).
Excellence appears frequently as a goal. In Australia the stated goal of the new ERA system is to identify andpromote excellence across the full spectrum of research activity, including discovery and applied research, in Australia's
higher education institutions (ARC, 2009, p. 11). In New Zealand, the primary purpose of the PBRF is to ensure thatexcellent research in the tertiary education sector is encouraged and rewarded (New Zealand, Tertiary Education
Commission, 2010). In Norway the goal is to increase research activities and allocate resources to centres performing
excellent research (European Commission, 2010, p. 120). Sivertsen reports the goal of the Norwegian publication
indicator to be “to measure and stimulate the research activity at the level of institutions and to enhance the focus andpriority they give to research as organizations” (Sivertsen, 2009, p. 6). Government statements of PRFS rationale thus
reveal several independent themes: resource concentration, encouraging international (i.e. English-language) publicationand the general pursuit of excellence. While the resource concentration theme bears some similarity to the search forincreased efficiency of new public management reforms, research excellence is more reminiscent of the newer “public
values” movement (Stoker, 2006).
4 Typology and timelines
Performance-based research funding systems vary in their unit of analysis, methods of measurement, frequency andcensus period. Individuals, research groups, departments and universities are all possible targets for evaluation. Researchgroups are the unit of evaluation with the best theoretical support because research is conducted by groups, not by
individuals or departments. Departmental level PRFSs are routinely criticised because of this (see for example Herbst,
2007, p. 91). The recent assessment of university research evaluation in Europe by an expert working group of theEuropean Commission makes a best practice recommendation that groups, or “knowledge clusters”, be the preferred unitof analysis (European Commission, 2010, pp. 38-39). However, research group evaluation is probably impossible toimplement on a national scale because there are so many groups and their boundaries are fluid and indistinct. The
Australian RQF proposed evaluation at the research group level. This elaborate exercise was the result of an extensive
consultation and design exercise, but was abandoned by a new government before implementation in favour of the ERA,
which emphasises simplicity. Because the ideal research group level evaluation is impractical, actual PRFSs will always
be subject to criticism. However, departmental and university level PRFSs have proved their worth in practice and while
imperfect, can be considered quite sound and usable.
The more practical and thus widely implemented unit of analysis is the department or the analogous field-inuniversity. The UK RAE evaluates university departments; the Portugese evaluate research units; the Australian ERA and
Italian VTR/VQR evaluate fields in universities. In the RAE the results by department are published but are aggregated tothe university level in order to award a single block grant to each university. In Poland faculties within universities areevaluated and money is awarded to faculties directly by the Ministry of Science and Higher Education. The Slovak
Republic also appears to conduct its evaluation at the field-in-university level.
A minority of systems evaluate individuals or universities. Two evaluate individuals - the Spanish sexenio and the
New Zealand PBRF. The sexenio results are used to increase the salaries of individuals achieving a high grade. The
PBRF results are aggregated into a rating for universities used to allocate block funding. Evaluations at the university
level were produced by the old Australian Composite Index and have been adopted by the Norwegian and Danish PRFSs.
The methods used in assessment correlate with the unit of analysis. Peer review is used for individual evaluations in
Spain and New Zealand. Peer review is also used for departmental and field evaluations such as the RAE, VTR/VQR and
Portugese evaluation. The RAE reviews are informed by narratives submitted by departments. In Portugal, evaluation
7

committees visit all units, in addition to studying their activity reports. In ERA, fields have the option of using peer
review or metrics. University-level evaluations rely on quantitative formulas. Such formulas use bibliometric output
information but differ as to whether papers alone are counted or whether citation information is also included. The
Australian Composite Index and the Norwegian and Danish systems employ paper counts only. However, Butler
identified a weakness in simply counting papers, in that it encourages a move to lower quality journals to increase
publication frequency (Butler, 2003). Therefore the Norwegian and Danish systems use an enhanced method in which
10% or more of journals are assigned to a higher quality category and given extra weight in the formula. Spain
incorporates the Thomson-Reuters impact factor in its deliberations. Systems using citation information include ERA, theSlovak Republic, Sweden and Belgium (Flemish Community). Spain’s sexenio includes self-reported impact informationsuch as citations or journal impact factor. The formulas include a range of other measures in addition to publication output
including: employment of graduates; external research funding; faculty characteristics and qualifications; faculty size;
graduate students graduated; implementation/application of research; international memberships; student load.
PRFS timelines vary, both in the frequency of evaluation and the census period over which data is gathered.
University level, indicator based evaluations tend to occur annually, with peer review evaluations less frequent. The
Australian Composite Index and the Norwegian model require annual data submissions to calculate the following year's
budget allocation. The second Australian ERA will be conducted in 2012, two years after the first. The Slovak Republic
evaluates every three years (Geuna and Martin, 2003), Poland no less frequently than once every five years (in practice
every four years), and Spain every six years (hence sexenio). Italy conducted an evaluation in 2006 using 2001-03 data
and used the results to allocate funding in 2009 (European Commission, 2010, p. 116). Evaluations are not necessarily
conducted on a regular schedule. The first two rounds of New Zealand's PBRF and Hong Kong’s RAE were conducted atthree-year intervals. The next are scheduled at intervals of six and seven years, respectively. The interval between RAEs
has increased from three to four, five and then seven years.
Census periods reveal no pattern. The Australian Composite Index and the Norwegian model base calculations on
one year of data. The VTR was based on three years of data, the VQR will be based on five years, Poland and Portugal
four years, Hong Kong basically four years, the PBRF five years, the sexenio six years. ERA uses six years forbibliometric measures and three years for other indicators. RAE 2008 used publications from a seven-year period. The
BOF-key is based on ten years for bibliometric measures and four years for non-bibliometric measures (see Poland,
Ministry of Science and Higher Education, 2010; Hong Kong, China, UGC, 2006, p. 11; Hodder and Hodder, 2010; ARC,
2009, p. 13; Debackere and Glänzel, 2004, p. 268; CIVR, 2011).
Table 2 summarises this information in a broad typology of systems based on the unit evaluated & method used:
individual/peer review, research group (theoretical, not realised), department or field in university/peer review,
university/indicator based. A further subdivision based on type of bibliometrics used is included. The typology issimplified because non-bibliometric metrics are not considered, nor is the unit to which funding is allocated.

8

Table 2 – Bibliometrically oriented classification of PRFSs
Unit of evaluation

MethodCountry & system

University

Indicators – paper counts

Australia Composite Index*
Denmark
Finland
NorwayFlanders, Belgium
PolandSlovak Republic
SwedenUK RAE
Italy VTR
Portugal
Italy VQR

Indicators – paper & citation
metrics

Department or field

peer review

Peer review informed by
metrics
Expert review informed by
metrics or peer review
Group (never implemented)
Individual

Peer review

Frequency
1
1
1
1
1
5
3
1
3,4,5,73

Censusperiod
2

1
10
4

8
3
4
5

Australia ERA

2

6

Poland5

4Spain - sexenio
New Zealand PBRF

6
3, 6

5
5

* Countries/systems in italics represent older versions of the PRFS.
5 Challenges
PRFSs exhibit similarities in design because they face similar challenges. Peer review is held in high esteem as it iswell accepted by the academic community in every country. However, being expensive and time consuming, it is usedirregularly. As a result, funding decisions may be based on out-of-date information. Arguably departmental or field level
PRFS using peer judgment based on indicators represents the state of the art. ERA is implementing this method.
University-level evaluation is metric-based and conducted annually using bibliometric methods. As these are notmuch liked by the academic community, governments that propose them may face heavy criticism. In smaller nations, theability to handle very large datasets has made possible departmental-level bibliometrics that are national in scope.
However, the departmental affiliation of authors, as indicated on papers, is often ambiguous, and cleaning the data isonerous (Debackere and Glänzel, 2004). When funding is allocated on the basis of such data, they must be absolutely
clean, because universities would invest much time in challenging their allocations if errors were found. A key advance in
the PRFS bibliometric method is the introduction of weighted categories of journals. PRFS frameworks now assign higher
weights to the top 10-20% of journals.
PRFSs are challenged by differences in the patterns of fields’ output. Counting journal articles indexed in databases
such as Web of Science or Scopus works for scientific fields but is inadequate for social sciences and humanities, bothbecause the indexing of social science and humanities journals is inadequate and because scholars in these fields produce
more than English-language journal articles (Hicks, 2004; Hicks and Wang, 2009). For these reasons the Norwegianmodel, though metrics-based, does not simply count indexed publications but is based on a national research information
system of university submissions of outputs. Peer review systems convene field-based committees that generally havelatitude for developing appropriate standards of judgment for their field (Poland, ERA, RAE, VTR). The ERA suite ofindicators varies by field, with scholars in each field choosing the most appropriate metrics for their field. The Swedish
formula contains an intricate field-weighting system. Spain’s sexenio permits submissions to include a broader range of
publication types in social sciences and humanities, though there are indications that this does not work very well in thatthe applications of social scientists are less likely to be approved than those of scientists, and success rates have notincreased over time (European Commission, 2010, p. 122; Jiménez-Contreras et al., 2003, p. 138).
9

The state-of-the-art in PRFS design incorporates highly consultative processes in which the government gathers input
from universities using comment periods, and evaluation processes are designed by expert panels made up ofrepresentatives of university or field-based associations. This was not the case initially; the first UK RAE and theAustralian Composite Index were imposed. However, in the United Kingdom an extensive public discussion of the RAE
ensued. Studies of effects were commissioned and the current design process for the RAE's replacement includes
extensive formal consultation. Similarly in Australia the design of both the RQF and the ERA involved “exhaustive
consultation with researchers and the 39 universities in the Australian system. There is a strong requirement of procedural
fairness and transparency and acceptance by key stakeholders” (European Commission, 2010, p. 86). In Denmark years of
discussion did not lead to a consensus, thereby preventing the introduction of a PRFS. Lately the discussion has moved
forward and the introduction of the Norwegian system is being designed in collaboration with the research community(Fosse Hansen, 2009). In Poland the regulations governing the PRFS are being revised and comments have been solicitedfrom the heads of academic units. The Norwegian model was designed by the ministry in consultation with the university
sector, represented by the Norwegian Association of Higher Education Institutions (i.e. Rectors' Conference), and thisbody has on going responsibility for the publication indicator (Sivertsen, 2010, p. 2). The increased consultation in PRFS
design may signal a shift from the new public management origins of the RAE and the Composite Index to the newer
public values/networked governance model in which extensive consultation with stakeholders is preferred (Stoker, 2006).
Most systems emphasise transparency of methods and data. Instructions to universities concerning their submissions
are easily available over the Internet as are the formulas used to convert measures into final rankings, grades or weights.
The final grades are publicly available as well, again often posted on government websites. In Norway and the RAE 2008,
all university submissions are public. The public nature of the results means that they are often used by others, picked upby the media and used in other funding decisions such as internal university allocation of funds or as one factor
considered in awarding research grants. Individual grades are used in tenure decisions in Spain. In this respect, Australia
lags the state of the art as all requests for access to ERA data have been refused and the weightings given to various
components of information available to committees are not known.
Notable in PRSFs is a tension between complexity and practicality. The RQF goal of assessing at the level of theresearch group was too complex to be practical. The complexity of submissions required by the RAE increased over theyears, and departments elaborated their submissions over time in an effort to become more competitive. This raisedquestions about the cost/benefit ratio of the exercise, and the UK government initially proposed a metrics-only future in
the REF (Whitley and Martin, 2010). Complexity emerges in these systems as a response to consultation which produces
pressures for fairness across heterogeneous academic disciplines. In Hong Kong extensive consultation prior to the 1999
RAE resulted in a broadening of the activities assessed from research only to include research-related scholarly activities:
discovery, integration, application and teaching (French et al., 2001, p. 37). Presumably, complexity increases easily inthe absence of any accounting of the full cost (see below).
To recognize that complexity increases over time is to acknowledge that PRFSs are dynamic systems. Heinrich andMarschke (2010) argue that public sector performance evaluation systems are inevitably dynamic, though not often
viewed as such. They use a principal agent theoretical framework to interpret studies of performance evaluation in otherareas of public policy arguing that:
Performance incentive system designers also need to appreciate and confront the evolutionary dynamic thatappears to be an important feature of many performance measurement schemes. As illustrated, an incentive
designer’s understanding of the nature of a performance measure’s distortions and employees’ means for
influencing performance is typically imperfect prior to implementation. It is only as performance measures aretried, evaluated, modified, and/or discarded that agents’ responses become known. . . .
Of course, a complicating factor is that performance measures can be gamed. Agents (or employees), throughtheir day-to-day experience with the technology of production, come to know the distinct weaknesses or
distortions of performance measures and how they can exploit them. If it takes the agent time to learn how togame a new performance measure, an equilibrium solution may be a periodic revision of performance measures
or a reassignment of agents. However, if both principals and agents are learning over time, the dynamic is likely
to become more complex. And depending on the relative speeds of principal and agent learning and the extent towhich performance measures degrade over time, this dynamic to measurement systems will not necessarily end. . .
(Heinrich and Marschke, 2010, p. 203)
10

The PRFS experience supports Heinrich and Marschke’s argument that performance incentive systems are dynamic:
the RAE changes with every iteration, most recently becoming the REF; the Composite Index is discarded in favour of
ERA; the VTR becomes the VQR. Butler’s analysis of changes in Australia’s publication profile after the introduction of
the Composite Index provided particularly compelling evidence of agent learning (Butler, 2003). The adoption of tieredjournal classifications in subsequent PRFS designs illustrates the response of principals. The discarding of ERAs tieredjournal classification in May 2011 suggests that PRFSs are characterized by perpetual modification in response touniversity learning (Carr, 2011).
PRFS designers face a common set of challenges. A balance must be struck between peer review and metrics that will
depend on the unit chosen for analysis, and although peer review is better accepted; metrics enable broader scale and morecurrent results to be used in funding decisions. One size does not fit all: PRFSs must accommodate differences acrossfields, particularly differences between the sciences, social sciences and humanities. Lengthy consultation with theacademic community and transparency of data and results are highly valued. Systems will be dynamic, and complexity
will increase over time. As complex as these challenges are, every PRFS resolves them. This stands in stark contrast to
the assessment of costs versus benefits.
6 The impossibility of assessing costs and benefits
An analytical comparison of PRFSs would assess costs and benefits of these systems. At present, this is impossible
for two reasons. First, costs are not discussed. Second, there are no standards for reporting the amount of university
funding dependent on the PRFS.
The costliness of the PRFS should be a concern for any government. PRFSs incur indirect costs for universities thatcompile submissions and direct costs for the evaluation of those submissions. The RAE and VTR incur heavy costs infaculty time because they are peer-review exercises. The RAE in 2001 involved 70 panels of 10 or more members
convened to work on assessing 180 000 publications, which made the exercise expensive. Panels were expected to read
papers, though given the impossibility of comprehensive reading, thoroughness varied (Harman, 2000, p. 115). One
author noted that the exercise was conducted as if it was to appraise 50 000 individual researchers and their
180 000 pieces of work in order to make 160 funding decisions (Sastry and Bekhradnia, 2006), which seemed
disproportionate. There were also indirect costs borne by departments whose efforts for preparing submissions increased
with each round. A particularly clear cost statement was found for the Italian exercise which took place in 2006:“The evaluation involved 20 disciplinary areas, 102 research structures, 18 500 research products and 6 661 peer
reviewers (1 465 from abroad); it had a direct cost of 3.55 million Euros and a time length spanning over 18
months.” (Franceschet and Costantini, 2009, p. 1)
The VTR panels were expected to assess the quality of each of the 17 300 unique written works submitted, 2 900 ofwhich were books. Two reviewers were assigned to assess each work independently. If every reviewer fully fulfilled theirmandate, 5 800 reviewers read a book in addition to the four articles they were assigned. Is it any wonder that the intervals
between the RAEs increased over time or that a second VTR is only now being mounted?
Indicator-based systems also incur substantial direct costs. The direct costs of indicator-based exercises include
establishing and maintaining a national research documentation system, buying citation information from databaseproviders, data cleaning and validation, and indicator calculation. As these data systems are intricate and large, the costs
are considerable. Auditing of submissions is also necessary because fraudulent submissions could be used to increase
funding allocations. Audits of Composite Index submissions were conducted by KPMG and found a high error rate (34%in the second audit in 1997); 97% of errors affected final scores and thus funding allocations (Harman, 2000, pp. 118119). If costs were articulated, some consideration could be given to cost/benefit ratios.
PRFSs are used in decision making concerning the distribution of block grants to universities. They replace methodsof distributing block grants including using a historical basis, i.e. the previous year’s allocation, or formulas based on
faculty or student numbers. Knowing the amount of funding allocated using a PRFS is crucial to understanding the effectof PRFSs on universities and on the innovation system. Somewhat surprisingly it is difficult to obtain consistent11

information on this from the literature. The amount of funding distributed based on a PRFS can be defined in many ways,
and there is no consistency among authors in their reporting. Reported figures include shares of:

total university resources (government plus other funding);
government funding for universities (block grant plus research grants and contracts);
block grant or “general university funds” (GUF);
research resources (total or government).

Introducing research performance into a funding formula likely entails splitting the traditional block grant into
teaching and research components. The research component might include research output assessment as well as graduate
student numbers and amount of outside funding raised. So despite the high profile of the PRFS, its role in allocating
funding may be deeply buried in a complex formula.
It is not entirely obvious which figure is most relevant to assessing the impact of the PRFS on a university. Would it
be the average across all universities, the share only in the most research-intensive universities, or the highest and lowest
shares in the system? Are static snapshots less interesting than trends over time? Finally, the amount that moves between
universities in any two years might be most illuminating. Ideally, each of these figures would be available for every
university system for current and past allocations. Table 3 reports the limited information that was found.
Table 3 – Share of university funding dependent on PRFSCountry

Share

Of What3

In year

SourceAustralia

6%

Total revenue

2008

Australian Government Department of Education,
Employment and Workplace Relations, 2009, Table 1

Italy

2%4

Block grant2009

European Commission, 2010, p. 115

New Zealand

10%5

Block grant2010

Tertiary Education Commission, 2010

Norway2%

Total funding

2008

Sivertsen, 2009, p. 6

Slovak Republic

15%

Total funding

2006

Strehl, 2007, p. 41

UK

25%6

Researchsupport2006

Various3

Total revenue from all sources including national and state governments, student fees, royalties, donations, industry etc.

4This is calculated as follows: 7% of the block funding to universities (FFO, Fondo di Finanziamento Ordinario) is allocatedaccording to a performance formula of which research comprises two-thirds of the total (and teaching one-third). Of this,
50% is in proportion to the grade received by the university from VTR in 2006

5

In New Zealand the research output evaluation comprises 60% of the PBRF, which is one of six Tertiary Education Organisation
Component (TEOC) funding elements

6

The RAE informs the distribution of a block grant which comprised 36% of support for science and engineering R&D in UK
universities in 2004-05 (Department of Trade and Industry [DTI], and Office of Science and Innovation, 2007). General
support for university research is allocated largely, but not entirely, on the basis of RAE ratings. Other criteria such assupporting new subjects or making allowance for the higher cost of living in London also play a role. In 2006-07, theHigher Education Funding Council for England (HEFCE) allocated 70% of its general support using RAE-linked criteria(HEFCE, 2006). Just prior to 2008 approximately 25% of all research support in UK universities appears to have been
allocated to universities based on the RAE ratings of their departments. RAE-based allocations are quite stable. By 2001 themarginal impact of the RAE on university finances was small. Sastry and Bekhradnia (2006) calculated that as a result of
the 2001 RAE, only one institution saw its total revenues affected by more than 3.7% and the median impact was less than
0.6%.

12

Some information on time trends is also available:


In Belgium (Flemish Community), the block grant for universities is divided into three parts: general, basic
research (BOF) and applied research (IOF), and 45% of general and BOF are allocated according to a formula
called the BOF-key. The weighting given to the evaluation of research outputs in the calculation of the BOF-keyrose from 0.10 in 2003 to 0.30 in 2006 (Debackere and Glänzel, 2004, p. 268; Luwel, 2010).



In Poland the proportion of research funding distributed in block grants is likely to decline as the government
promises to increase the share dedicated to competitive awards (Pain, 2010).



In the Slovak Republic, government funding for research in universities more than doubled between 2002 and2005. At the same time, the share of block funding allocated according to evaluated research performance aswell as the “quality of development projects” increased from 9.7% to 23.1% at the expense of historicalallocation and allocation according to evaluated teaching performance (Strehl, 2007, p. 113).

In the United Kingdom, the share of university resources from RAE-governed processes declined from 58% in
1984 to 35% in 1997 because the share of research funding from all other sources increased, including the sharefrom competitive grants.

This information is difficult to summarise. However, the precise amounts of money moved may not in fact beterribly relevant. Analysts consistently emphasise the small amounts of money involved or the small amount that moves
in any one year as a result of evaluation (Jiménez-Contreras et al., 2003; Sanz-Menendez, 1995; Sivertsen, 2010; Sastry
and Bekhradnia, 2006; Rodríguez-Navarro, 2009). Universities have high fixed costs and require stable funding streams,
and no government would benefit from a university becoming bankrupt. Therefore, governments would be unwise to
create a highly unstable funding system that would swiftly reallocate large amounts of funding and might bankrupt someinstitutions, though the Polish government has threatened to remove research funding from underperforming universities(Pain, 2010). The short-term financial consequences of a PRFS are likely to be less than is often feared, although small,consistent movements can accumulate over time. Only the UK system has been operating long enough to assess thepossible financial effects of PRFS over the long term. Since the UK government had an explicit goal of selectivity, orconcentrating resources in fewer universities, the appearance of this effect would not be unexpected.
PRFSs can create pressure to increase funding. Grade increases in the RAE between rounds were used to argue formore government money for research to reward improved performance (Tapper and Salter, 2003). Hong Kong faces thesame phenomenon (French et al., 2001). If the success rate in Spain’s sexenio increases, the government is automaticallyobliged to increase faculty salaries. It seems likely that pressures to increase university funding will spread with the spreadof PRFSs. Universities that can demonstrate increased excellence, particularly in highly publicised, measurable,internationally comparable ways will have strong arguments when budgets are discussed.
It is possible that a PRFS will entrain other parts of the research funding system, with the result that the effect of the
PRFS will go beyond the amount of money directly allocated. This will happen if grant review is not double-blind and theprobability of a successful grant application is increased if the applicant is located in a higher-ranking department. Thisappears to be the case in the U.K. where there is a very close correlation between research council funding and RAE
results (Whitley and Martin, 2010). In Spain a minimum number of sexenios are required for securing tenure or for
becoming a member of the commission that grants tenure, thereby aligning the formation of an elite with the PRFS. Also,
sexenios help with applications for competitive research funding thus aligning the two major research funding
mechanisms (Jiménez-Contreras et al., 2003, p. 135).
Analysts who point to the small direct financial consequences attached to PRFSs do not therefore believe that theylack influence. On the contrary, they argue that the systems have strong effects on universities, though less through the
incentives funding provides than through public judgments about relative prestige. Comprehensive assessment of
universities and their departments creates intense interest among universities. Experience has shown that universities are
extremely responsive to hierarchical ranking. One effect of the RAE was to create what McNay termed assured, aspiring
and anxious universities (HEFCE, 1997, p. 47). Attention devoted to RAE submissions did not decrease, even though, asmentioned above, Sastry and Bekhradnia calculated that the median impact on total university revenue of the 2001
exercise was 0.6%. In preparations for the 2008 RAE, “research-intensive institutions indicated that they would seek the13

best ratings rather than the financial rewards” when choosing how to structure their submissions (Lipsett, 2007).
Marginson noted in relation to the introduction of a university assessment in Australia in 1993: “Nothing less than thepositional status of every institution was at stake; the process of competitive ranking had a compelling effect, leading to
the rapid spread of a reflective culture of continuous improvement” (Marginson, 1997, p. 74). Harman related that inAustralia allocation of funding based on the Composite Index had become “an important vehicle for developing status
hierarchies” as data are published in newspapers and widely used (Harman, 2000, p. 116).
The attention paid to research output assessments that are not linked directly to funding confirms this. In the UnitedStates, the annual ranking of university departments by US News & World Report is extremely influential. Similarly inAustralia, ERA was conducted before it was used to determine funding allocation, yet it was the focus of intense interest.
The Shanghai Jiao Tong University Academic Ranking of World Universities and the THES-QS World University
Rankings have been so influential that President Sarkozy has ordered France's science and higher education ministry to set“the objective of having two French establishments in the top 20, and 10 in the top 100” (Anonymous, 2010). That
universities pay close attention to rankings and their attendant prestige is entirely rational since prospective students use
rankings to decide on their destinations, especially at the graduate level, and since money follows students. In addition,
the more productive, grant-raising faculty seek to work at more highly ranked institutions.
The emphasis in the literature on prestige rather than financial incentives as the main mechanism through which
PRFSs work is consistent with conclusions of the new public management literature. Kettl (2000, p. 38), summarising areport from the OECD's Public Management Committee, explains that “public managers around the world have indeed
been strongly motivated by incentives, but the incentives have had more to do with their jobs than with the often sporadic
performance driven pay systems”. Although PRFSs may not be sporadic in that the formulas are applied to every funding
allocation, authors seem to agree that the amount of money moved as a result of research rankings is not all that great andprestige is a more important motivating factor, and the influence of independent rankings supports this. In publicmanagement, contestability, that is, the prospect of competition (Kettl, 2000, p. 40) is seen as central to the benefits
derived from new public management reforms. Universities compete for prestige, and PRFSs harness this.
The dynamic nature of PRFSs further complicates the assessment of costs and benefits. Heinrich and Marschke
propose that the assessment of costs versus benefits “will likely depend on where in [the] dynamic cycle the program isbeing evaluated” (Heinrich and Marschke, 2010, p. 203). For example, while the introduction of a PRFS might bringperformance gains initially, after a few iterations, improvements without funding increases may no longer be possible andthe costly exercises then return little to no benefit (Geuna and Martin, 2003; Hicks, 2008). The idea is that the PRFS
initially harnesses latent capacity in the system without adding more research resources, but eventually that capacity is
exhausted and further gains require more research resources. It would be rash to use this reasoning to remove a PRFS,
however, because this analysis neglects the demonstrable benefits of introducing contestability and incentives into asystem. The gains in performance are clear when contestability is introduced for the first time. But eventually
improvements slow at a higher performance level. If contestability, which produced the higher-level state, were removed,
one suspects performance might degrade. Strictly speaking, a government should maintain a PRFS if the difference inresearch excellence between the before and after states is achieved more cost effectively with a PRFS than by adding
research resources. Although possible in theory, in practice it is impossible to make this calculation since the costs of
PRFSs are not articulated, the benefits have not been quantified, and the result will change over time as principals andagents evolve their strategies in the never ending battles of wits that is a PRFS.
7 Autonomy versus control
Autonomy, self-governance and competition are sensitive issues for the academic community and the introduction ofa PRFS creates tensions between autonomy and control. British and Australian commentators on PRFSs are sensitive tothe subtleties of rhetoric and reality in relation to university autonomy so this theme is prominent in English-language
commentary (Marginson, 1997). Tapper and Salter argue that: “Ironically, such a model of governance may constrain
higher education institutions more severely whilst giving the impression, or at least creating the illusion, that university
autonomy has been retained.” (Tapper and Salter, 2003, p. 11) For example, the Australian Composite Index is used toallocate a block grant; it would seem universities have autonomy in deciding how to spend the block of money theyreceive. However, this block grant is divided into seven parts, each of which is subject to a separate set of “broad”
guidelines by the funding agency which thereby retains a large degree of government control over expenditure. Similarly,
though scholars were consulted widely on ERA and had some say in the choice of indicators, it was within limits defined14

by the government. Herbst, an American, observes: “The move of European higher education systems toward managerial
autonomy has strings attached: institutional autonomy appears to be offered with one hand - and taken back with theother.” (Herbst, 2007, p. 79) In addition, the influence of a PRFS will depend on how universities allocate funding
internally; conceivably university management could negate or enhance PRFS incentives (Frölich, 2008; L. Butler, 2010).
Although the autonomy of academic disciplines is not generally seen as threatened (rather criticism tends to centre on
who controls the discipline because those who control the discipline, control the implementation of the PRFS),
governments can limit the autonomy of disciplines in peer review evaluation processes by dictating the criteria reviewers
must use. In Spain the criteria are set down in the regulations and are quite specific, reducing the degrees of freedom forreviewers and enhancing government control. In other countries less specific criteria provide scholars with moreautonomy in their judgments (Jiménez-Contreras et al., 2003). When the RAE began, disciplinary committees hadcomplete freedom to determine criteria; later the government introduced a standardised statement of criteria to ensure
fairness across fields (Tapper and Salter, 2003).
The effect of PRFSs on the perceived autonomy of individual scholars is a source of perennial dissatisfaction and of
the accusation that PRFSs harm scholarship and the research enterprise. The mechanism through which PRFSs restrictautonomy is by enhancing control of disciplinary elites. A great deal of this commentary implicitly assumes a prior statein which collegial relations governed the academic community through informal peer review. Others argue that this is an
idealised view which neglects professional hierarchies and intellectual authority relations (Harley and Lee, 1997). Whitleydevelops this argument in expressing his concern about the possible deleterious effects of PRFSs on the vibrancy of thescholarly community and the knowledge it creates. In essence, he argues that strong evaluation systems will reinforce theinfluence of conservative scientific elders, thereby suppressing novelty, new fields, diversity and pluralism. This problem
will be exacerbated if a country's scientific elite is cohesive and if they also control project-based funding through peer
review. University autonomy will reduce this effect (Whitley, 2008, pp. 14-16). The problems generated by cohesive
intellectual elites in control of a PRFS may be particularly visible in economics, with its striking division between
neoclassical scholars and everybody else (Corsi et al., 2010; European Commission, 2010, p. 116; Harley and Lee, 1997;
Lee, 2007). Rafols et al. have produced empirical evidence that “the rather narrow and idiosyncratic view of excellence”
of disciplinary elites serves to suppress the rankings of interdisciplinary research efforts in a way not supported by citation
measures (Rafols et al., 2011) This point is connected to the magnified effect of PRFSs through their effects on other partsof the funding system mentioned above.
Autonomy is a sensitive issue for scholars at both the institutional and individual levels, and one that is increasingly
complex to navigate for a number of reasons including but not limited to PRSFs (Henkel, 2005). At least in the literature
in English, critics often focus on the shifts of control over academic work that accompany the introduction of a PRFS.
Ultimately, the effects of a PRFS on institutional autonomy seem contradictory and ambiguous, and may become more so
as extensive consultation with the academic community over PRFS design becomes routine. But under the rightcircumstances a PRFS will certainly enhance control by professional elites.
8 Thoughts on the future
The long-term future of a PRFS will depend on how well it meets the government's goals. Many governments havearticulated a clear goal for their PRFS: the enhancement of research excellence. A competition for prestige among
universities based on research performance will likely achieve that goal. Problems may arise however if governments
realise that their goals, or values, are broader. First, a PRFS will not be a good way to encourage interaction with industry
and application of research, activities with demonstrable economic benefits. Enhancing universities' contributions to theeconomy is a common policy goal that is not well addressed in current PRFSs. The Australian RQF included measures ofbroader impact, but this was dropped due to complexity. ERA does not include broader impact measures. The sexenio has
been criticised for excluding consideration of broader impact (Sanz-Menendez, 1995). The UK REF is then leading theway by piloting broader impact assessment in the form of case studies.7
Equity and diversity are also important public values in relation to universities. Excellence and equity have always
been in tension in research policy. PRFSs encourage excellence at the expense of equity. A more subtle conflict may arise
7

See special issue Research Evaluation on the state of the art in evaluation of broader impacts, September 2011.

15

in a PRFS because of the strong reliance on the academic elite in its design and implementation and the possibly enhanced
effect if the rest of the funding system is entrained by the PRFS. Whitley suggests that novelty, innovation and intellectual
diversity may be suppressed because elites tend to judge academic quality in part on how well work advances paradigms
they themselves have established. In addition, the contribution of universities to national and cultural identity may lessen
because these are devalued in systems that focus on research excellence at the international level (i.e. published in
English). There may be circumstances in which any of these value conflicts becomes politically unacceptable. Twochoices will present themselves, either further increasing the complexity of the PRFS to broaden the definition of
performance (and so increase cost), or reducing the influence of the PRFS and adding another programme to distributeresearch support based on consideration of other values. The vision of an ever more burdensome PRFS, forced to servegoals for which it is less than ideal seems to hold less appeal than a suite of programmes, each optimally designed to servea different goal. Many systems already have programs in place that ameliorate the effects of PRFSs such as granting
schemes focused on interdisciplinary groups, or university-industry collaborations. Such diversity in the funding systemshould facilitate differentiation among autonomous universities managed strategically and enhance the ability of thesystem as a whole to serve a complex suite of public values.
Any consideration of the future of PRFSs should take into account relevant non-PRFS mechanisms. Sincecontestability seems to be at the heart of the benefits of a PRFS, it may be possible to encourage research excellence
simply by relying on independent rankings. International rankings are proliferating. Times Higher Education (or THE),
Shanghai Jiao Tong, and emerging corporate products are discussed internationally, are analysed and compared, and arepolitically influential (D. Butler, 2010; European Commission, 2010; Anonymous, 2010). Their influence looks likely toincrease greatly over the next few years. In providing contestability at the level of universities, such rankings duplicate
one of the most important functions of PRFSs. (They do not however involve peer review or assess departments or fields.)
A government that chose to substitute international rankings for a PRFS might not even need to link funding to theranking. A government could simply rely on media attention and student and faculty pressure to create incentives fromrankings. This is in fact a notable element of the highly successful US university system, for which US News & World
Report annual departmental rankings and National Academy decadal departmental rankings are highly influential.
Substituting an international ranking for a PRFS would have the benefit of eliminating costs and shifting divisive andresource-consuming methodological arguments to the international scholarly community. However, universities can only
compete effectively for prestige if they have institutional autonomy and discretionary resources (again this is the case in
the US system). This recalls the new public management origins of PRFSs. The introduction of a PRFS tends to be justone part of larger changes and although independent rankings may substitute for the research evaluation component,
without the larger changes universities cannot respond to incentives to increase their prestige. This suggests that the focus
should not be the PRFS per se, but rather increasing contestability and institutional autonomy in a university system.
The future of PRFSs also depends on how successful they prove to be in comparison to the alternative centre-ofexcellence approach in which governments award a limited number of very large, long-term block grants to universities
based on competitive proposals. Germany, Japan and Poland use this approach. China has also taken this approach with its
Project 985, though without the competitive proposal aspect. This is clearly a mechanism meant to concentrate funding
and encourage international levels of excellence - the same goals that motivate PRFSs. A long-term comparison of therelative merits of centre-of-excellence and PRFS approaches might help governments understand and explore theiroptions for increasing university research excellence.
9 Conclusions
Performance-Based Research Funding Systems (PRFS) are national systems of research output evaluation used to
distribute research funding to universities. Fourteen such systems have been implemented as of 2010 in response to theknowledge economy, new public management and a universal desire for research excellence on the part of governments.
This paper sought to step back from the intricacies of particular systems to draw out general lessons that can serve toenrich our understanding of how research policy and innovation systems are evolving. PRFSs were found to be complex,
dynamic systems, balancing peer review and metrics, accommodating differences between fields, and involving lengthy
consultation with the academic community and transparency in data and results. Although the importance of PRFSs
seems based on their distribution of universities’ research funding, this is something of an illusion, and the literatureagrees that it is the competition for prestige created by a PRSF that creates powerful incentives within university systems.
Institutional, professional and individual autonomy are important values in the academic community and the effects of16

PRFSs on autonomy are much discussed. The literature suggests that effects on institutional autonomy are ambiguous,
but under the right circumstances a PRFS will enhance control by professional elites. PRFSs, since they aim forexcellence, may compromise other important values such as equity or diversity. They will not serve the goal of enhancing
the economic relevance of research. Finally, there may be other methods to achieve research excellence and future
research should seek to compare the effectiveness of the different methods in achieving the primary, stated goal of PRFSs- increasing the excellence of a nation’s research.

Acknowledgements
The OECD Directorate for Science, Technology and Industry funded the research underpinning this paper. The
comments, feedback and editing contributed by Sarah Box at OECD and an anonymous reviewer were most helpful inimproving the paper. The first version of this paper was presented at the June 2010 OECD-Norway Workshop on
Performance-based Funding for Public Research in Tertiary Education Institutions, Paris, France.
References
Anonymous, 2010. The ratings game. Nature, 464(7285), p. 7, 4 March.
Australian Research Council (ARC), 2009. ERA 2010 Submission Guidelines, (downloaded on June 9, 2011 fromwww.arc.gov.au/era/key_docs10.htm).
Australian Government Department of Education, Employment and Workplace Relations, 2009. Finance 2008, Financial
Reports of Higher Education Providers, Canberra.
Butler, D., 2010. University rankings smarten up, Nature, 464(7285), 16-17, 4 March.
Butler, L., 2003. Explaining Australia's increased share of ISI publications – the effects of a funding formula based on
publication counts, Research Policy, 32, 143-155.
Butler, L., 2010. Impacts of performance-based research funding systems: A review of the concerns and the evidence.
Chapter 4 in OECD, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop
Proceedings, OECD Publishing.
Carr, K., 2011. Improvements to Excellence in Research for Australia. Press release, Senator the Hon Kim Carr, Minister
for Innovation, Industry, Science and Research, 30 May 2011 (accessed on June 9, 2011 at
http://minister.innovation.gov.au/Carr/MediaReleases/Pages/IMPROVEMENTSTOEXCELLENCEINRESEARCHFORAUSTRALIA.aspx ).
CHEPS, 2007. Higher Education in Germany: Country Report. Center for Higher Education Policy Studies (CHEPS), The
Netherlands.
CIVR, 2011. 2004-2008 VQR Guidelines, accessed on June 9, 2011 at http://www.civr.miur.it/en/vqr_decree.html .
Corsi, M, C. D’Ippoliti, and F. Lucidi. Pluralism at Risk? Heterodox Economic Approaches and the Evaluation of
Economic Research in Italy. American Journal of Economics and Sociology, 69, 1495-1527.
Debackere, K. and W. Glänzel, 2004. Using a bibliometric approach to support research policy making: The case of theFlemish BOF-key. Scientometrics, 59, 253–276.
Department of Trade and Industry (DTI) and Office of Science and Innovation, 2007. PSA Target Metrics 2006. HMSO,
London.
Dolan, C., 2007. Feasibility Study: the Evaluation and Benchmarking of Humanities Research in Europe. HERA D4.2.1,
Arts and Humanities Research Council, United Kingdom.
17

European Commission, 2010. Assessing Europe's University-Based Research. EUR 24187 EN, Science in Society 2008
Capacities, 1.4.1, European Commission, Brussels.
Fosse Hansen, H., 2009. Research Evaluation: Methods, Practice, and Experience. Danish Agency for Science,
Technology and Innovation, (downloaded on June 9, 2011 from http://en.fi.dk/research/rearch-evaluation-methodspractice-and-experience/Research%20Evaluation%20Methods-%20Practice-%20and%20Experience.pdf/).
Franceschet, M. and A. Costantini, 2009. The first Italian research assessment exercise: a bibliometric perspective. Paper
submitted to Research Policy, (downloaded on June 9, 2011 from http://eprintweb.org/S/article/cs/0912.2601).
French, N.J., W.F. Massy and K. Young, 2001. Research assessment in Hong Kong. Higher Education, 42, 35-46.
Frölich, N., 2008. The politics of steering by numbers: Debating performance-based funding in Europe. Studier av
innovasjon, forskning og utdanning, No. 3/2008, NIFU STEP, Oslo.
Geuna, A. and B.R. Martin, 2003. University research evaluation and funding: An international comparison. Minerva, 41,
277-304.
Harley, S. and F.S. Lee, 1997. Research selectivity, managerialism, and the academic labor process: The future ofNonmainstream Economics in U.K. Universities. Human Relations, 50, 1427.
Harman, G., 2000. Allocating Research Infrastructure Grants in Post-binary Higher Education Systems: British andAustralian Approaches. Journal of Higher Education Policy and Management, 22, 111-126.
Heinrich, Carolyn J. and Gerald Marschke, 2010. Incentives and their Dynamics in Public Sector Performance
Management Systems. Journal of Policy Analysis and Management, 29, 183-208.
Herbst, M., 2007. Financing Public Universities. Higher Education Dynamics, Vol. 18, Springer.
Hicks, D., 2004. The Four Literatures of Social Science, in Moed, H. (ed.) Handbook of Quantitative Science andTechnology Studies. Kluwer Academic Press, Dordrecht, 473-496.
Hicks, D., 2008. Evolving regimes of multi-university research evaluation. Higher Education, 57, 393-404.
Hicks, D. and J. Wang, 2009. Towards a Bibliometric Database for the Social Sciences and Humanities – A European
Scoping Project. Final report on project for the European Science Foundation.
Higher Education Funding Council for England (HEFCE), 1997. The impact of the 1992 Research Assessment Exercise
on higher education institutions in England. No. M6/97, Higher Education Funding Council for England, Bristol,(downloaded on June 9, 2011 from www.hefce.ac.uk/pubs/hefce/1997/m6_97.htm).
HEFCE, 2006. Funding Higher Education in England: How HEFCE Allocates its Funds, Higher Education Funding
Council for England, Bristol.
HEFCE, 2011. REF2014: Research Excellence Framework, (accessed on June 9, 2011 fromwww.hefce.ac.uk/Research/ref/).
Henkel, M., 2005. Academic Identity and Autonomy in a Changing Policy Environment. Higher Education, 49, 155-176.
Hodder, A.P.W. and C. Hodder, 2010. Research culture and New Zealand’s performance-based research fund: someinsights from bibliographic compilations of research outputs. Scientometrics, 84, 887-901.
Hong Kong, University Grants Committee (UGC), 2006. Research Assessment Exercise 2006 Guidance Notes (accessedon June 9, 2011, www.ugc.edu.hk/eng/ugc/publication/prog/rae/rae.htm).
18

Jiménez-Contreras, E., F. Anegon and E. Lopez-Cozar, 2003. The evolution of research activity in Spain – The impact of
the National Commission for the Evaluation of Research Activity (CNEAI). Research Policy, 32, 123-142.
Kettl, D.F., 2000. The Global Public Management Revolution: A Report on the Transference of Governance (2nd ed.).
Brookings Institution Press, Washington, DC.
Kettl, D.F., 2005. The Global Public Management Revolution. Brookings Institution Press.
Lee, F.S., 2007. The Research Assessment Exercise, the state and the dominance of mainstream economics in British
universities. Cambridge Journal of Economics, 31, 309-325.
Lipsett, A., 2007. RAE Selection Gets Brutal. Times Higher Education Supplement 1779, 1.
Luwel, M., 2010. Performance-based R&D institutional funding in Flemish Universities. Presented at the OECDworkshop, 21 June, Paris.
Marginson, S., 1997. Steering from a distance: Power relations in Australian higher education. Higher Education, 34, 63–
80.
Martin, B.R. and R. Whitley, 2010. The UK Research Assessment Exercise: A Case of Regulatory Capture? Chapter 2 in
R. Whitley, J. Gläser and L. Engwall (eds.), Reconfiguring Knowledge Production: Changing Authority
Relationships in the Sciences and their Consequences for Intellectual Innovation, Oxford University Press, Oxford,
UK, 51-80.
New Zealand, Tertiary Education Commission, 2010. Performance-Based Research Fund – Tertiary Education
Commission, (accessed on June 9, 2011, www.tec.govt.nz/Funding/Fund-finder/Performance-Based-ResearchFund-PBRF-/).
OECD, 2010. Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings,
OECD Publishing. (accessed on June 9, 2011, http://dx.doi.org/10.1787/9789264094611-en).
Pain, E., 2010. Polish Science Reforms Bring Fear and Hope. Science, 327, 1442.
Poland, Ministry of Science and Higher Education, 2010. General principles of parametric evaluation of scientificinstitutions, (access on June 9, 2011 at:
http://translate.google.com/translate?hl=en&sl=pl&u=http://www.nauka.gov.pl/finansowanie/finansowanienauki/dzialalnosc-statutowa/&ei=lDG7SKLEJGu9gSdlZCFCA&sa=X&oi=translate&ct=result&resnum=7&ved=0CCQQ7gEwBg&prev=/search%3Fq%3
Dfinansowania%2Bbada%25C5%2584%2Bstatutowych%26hl%3Den%26sa%3DG%26rlz%3D1G1GGLQ_ENUS375.).
Rafols, I. L. Leydesdorff, A. O’Hare, P. Nightingale and A. Stirling, 2011. How journal rankings can suppress
interdisciplinarity. The case of innovation studies in business and management, May.
Rodríguez-Navarro, A., 2009. Sound research, unimportant discoveries: Research, universities, and formal evaluation ofresearch in Spain. Journal of the American Society for Information Science and Technology, 60, 1845-1858.
Sanz-Menendez, L., 1995. Research actors and the state: research evaluation and evaluation of science and technology
policies in Spain. Research Evaluation, 5, 79-88.
Sastry, T. and B. Bekhradnia, 2006. Using Metrics to Allocate Research Funds: A short evaluation of alternatives to the
Research Assessment Exercise. Higher Education Policy Institute, Oxford.

19Sivertsen, G., 2009. A bibliometric funding model based on a national research information system. Presented at the ISSI
2009, Rio de Janeiro, Brazil, (downloaded on June 9, 2011 fromwww.issi2009.org/agendas/issiprogram/activity.php?lang=en&id=108).
Sivertsen, G., 2010. A Performance Indicator Based on Complete Data. ISSI Newsletter, Vol. 6(1), 22-28.
Stoker, G., 2006. Public Value Management: A New Narrative for Networked Governance?, The American Review of
Public Administration, 36, 41-57.
Strehl, F., 2007. Funding systems and their effects on higher education systems - international report. Education Working
Paper, No. 6, OECD, Paris.
Tapper, T. and B. Salter, 2003. Interpreting the Process of Change in Higher Education: The Case of the Research
Assessment Exercises. Higher Education Quarterly, 57, 4-23.
Whitley, R., 2008. Changing Governance of the Public Sciences. in R. Whitley and J. Gläser (eds.), The Changing
Governance of the Sciences, Sociology of the Sciences Yearbook, Springer, Dordrecht.

20