247

The ‘Preferred Model’ for Australia’s Research Quality FrameworkPeter Hall

On the 9th of September the AustralianGovernment released its ‘Preferred Model’for the Research Quality Framework, orRQF. The discussion paper can be downloaded from the web page containing theMinister’s announcement:
http://www.dest.gov.au/sectors/
research_sector/policies_issues_reviews/key_issues/research_quality_framework/rqf_preferred_model.htm
In a nutshell, the Preferred Model would
lead to a methodology for research assessment that has many similarities to the UK
Research Assessment Exercise, or RAE. The
latter has been in operation since 1986; thesixth RAE will take place during 2008. The
development of an RQF was signalled by
the Prime Minister back in May 2004, inconnection with the second tier of the government’s ‘Backing Australia’s Ability’ program. In principle the RQF applies to all
federally funded research agencies, and intheory it could be used to assess the workof CSIRO or ANSTO, for example. However, no attempt is made in the discussion
paper to connect the Preferred Model to either of these bodies. However, the linkage
to universities is emphasised throughout thepaper. The minister has even implied that
the RQF might be play a role in the review
process for regular ARC grants; we shall return to this topic towards the end of thisarticle.
The RQF would use discipline panels,
each with 12 to 15 members, to assess research performance. Neither the number ofpanels nor their discipline areas has been
given at this stage, so we can only guess at
the level of specialisation. For the 2008 UK
RAE there will be a single ‘mathematics’panel (Panel F), broken up into sub-panels

representing, respectively, Pure Mathematics, Applied Mathematics, Statistics & Operational Research, and Computer Science& Informatics. It is unclear whether the theRQF panel for mathematics and statistics
would reflect this level of generality, or bestill more diverse. In the smaller scientificenvironment of Australia, the panel is unlikely to be more narrow.
Of course, the RAE sub-panels have specialist representation in their respective discipline areas. The sub-panels do much of
their work autonomously, and UK mathematical scientists tend to feel that their research is being assessed by people who areexpert in fields that are not especially distant from their own. It is not clear that
this would be the case in Australia; no subpanel structure is foreshadowed in the government’s Preferred Model for the RQF.
Similarities to the UK RAE would include the requirement that each person considered by the RQF have produced at least
four research outputs (usually research papers and books, in the mathematical sciences) over the assessment period, and submit no more than four research outputs fromthat time-frame. The assessment period
would generally be six or seven years long.
Some flexibility in the number of researchoutputs would be allowed, for example inthe case of early-career researchers.
One major difference from the UK model
would be in terms of the involvement of ‘atleast 50 per cent international experts, primarily international experts who are resident overseas as well as some international
experts resident in Australia.’ In anotherdeparture, apparently reflecting the government’s desire to ensure specific practical outcomes from research funding, eachpanel would include ‘at least two experts,

248

The ‘Preferred Model’ for Australia’s Research Quality Framework

as deemed appropriate to the panel structure, who could represent the views of usersof the research under assessment.’
Thus, at least one quarter, and probably close to one half, of all panel members
would be from abroad, and two panel members would vet the practical credentials ofresearch. The latter aspect is troubling, particularly since a great deal of the research
conducted in Australian universities is of abasic or strategic nature.
The likely backgrounds of the remaining panel members are not clear from thePreferred Model paper. However, panelmembers would be chosen in a way which,in principle, is similar to that currently
used to select members of the ARC Collegeof Experts – through ‘targeted invitations
to relevant organisations and representative
groups, including universities, the Learned
Academies, PFRAs, business and community groups, etc.’ It is quite unclear how thisprocess could be extended to international
panelists; only with difficulty, one suspects.
The mathematical sciences community has
been quite unhappy with its representationon the ARC College of Experts, and so weshould pay particular attention to this aspect of the process.
The matter of ‘metrics,’ including bibliometric information such as citation data, isstill on the table. However, an attempt has
been made to relate metrics to subject areas. For example, it is stated in the Preferred Model paper that the RQF ‘will include metrics where appropriate to discipline areas, such as bibliometrics, industry
research grants and short supporting statements of research impact by relevant endusers.’
In the mathematical sciences, any meaningful use of bibliometric data would haveto involve a time-scale much longer than thesix or seven years used for RQF reporting.
Therefore, it would be particularly difficult
to combine bibliometric data with reporting
data in a meaningful way. It is not clear thatthese issues are appreciated by the committee that wrote the Preferred Model paper.
One of the major criticisms of the UK
RAE has been the amount of academic-staff
time it absorbs. The RQF would be very
similar in this respect. Australian universities would be required to ‘nominate research groupings, provide researchers’ evidence portfolios for inclusion in researchgroupings packages, [and] provide context
statements which will include supportingevidence, e.g. bibliometrics, measures ofpeer esteem, etc, for research groupings.’
The British International Studies Association, in a document submitted to the joint
UK funding bodies, noted that ‘The cost of
the RAE is not negligible, but the bulk of
the cost is borne by the academics that carryout the exercise.’In addition to the expenses incurred byuniversities in operating the RQF, there
would be costs that would have to be borneby the government. In an environment
where the Treasury’s economic models cause
it to question the need to spend money onpeer review of grant proposals, would theRQF be paid for by new funds, or would
the expense be taken out of other projects,
for example the money already set aside tosupport research? Will universities be compensated for the real cost of their work for
the RQF?
The RQF would also inherit other problems directly from its UK ancestor. Amongthese would be problems assessing multidisciplinary research. In fact, the Preferred
Model paper acknowledges that this is a potential weakness of the proposal: ‘Assessing cross-disciplinary work may be difficult.’
In a press release issued on 9th September, the Council for the Humanities, Artsand Social Sciences (CHASS) took the government to task on this issue, noting thatthe committee that produced the Preferred
Model ‘has not found an adequate way tomeasure the increasingly-significant area ofcross-disciplinary research.’ These remarks
echo sentiments expressed five years ago

Peter Hallby the UK Arts and Humanities Research
Board, which observed that the RAE ‘militates against interdisciplinarity.’
CHASS is also concerned that the RQF
would place ‘undue emphasis on the ‘quality’ of research rather than the ‘impact’,’and comments that ‘the model proposed by
the Advisory Group does need further work.
The approach it takes is too conservative,on nearly every substantive issue.’ Nevertheless, CHASS pronounced itself pleased
that the government has made ‘a genuine
attempt to work across the [university] sector and to include different kinds of measures for different research fields.’ At thetime of writing, the Australian Academy ofScience had not produced a response to thePreferred Model.
The International Review of Mathematics (IRM), which reported on mathematics research in the UK to the UK Physical Sciences Research Council and to theCouncil for the Mathematical Sciences, discussed the UK RAE at some length. Although the IRM found that the RAE had‘enhanced the importance given to researchin departmental and university policies,’ italso pointed to a number of causes for concern. These included the heavy administrative burden placed on staff preparing for an
RAE (see above), discouragement of risktaking in research, and the encouragementof ‘incremental research’ (that is, the slicingof research results into relatively small, publishable parts so that the researcher would
have enough papers to submit to the RAEpanel). The IRM observed that that RAE
led to personnel decisions that created tensions within departments, and which ran
counter to collegiality. In some instancesthese effects of the RAE reduced research
productivity.
Most importantly, the IRM found that
the RAE did not adequately support isolated pockets of excellence, that it produced
a concentration of mathematics research resources into a small number of departments,
and that it led those strong departments to249

concentrate on a relatively small number offields, where their academic staff had specialised expertise. In particular, the IRM
lamented the decline in UK expertise in areas such as modern analysis, related areasof PDEs, and statistics. Some of the pockets of strength noted by the IRM were quitefragile, since they existed only in one or two
universities in the country, and were basedon the activities of only a few people. Commented the IRM, ‘The UK cannot afford
to have its high quality research concentrated in too few leading departments thatare competitive at an international level.’
The tendency of the RAE to result in narrow and fragile concentrations of researchresources is felt right across the spectrumof fields in which research is conducted inUK universities. For example, in a very
different research field, the British Societyof Criminology reported on the impact of
the RAE on research criminologists, and expressed three major concerns: ‘The first is
the cost of the RAE; the second is its centralising tendency; the third is its failure to
support small groups of excellent researchers
working in larger departments.’
These problems are likely to be significantly more serious in the Australian context than they are in the UK. In a country
which has only a modest scientific and scholarly culture, and only a small number of universities that can field mathematics departments of very high calibre, any move which
reduces diversity and concentrates strength
can create potentially serious problems. It
could lead to an unstable national research
environment, where the loss of one or two
key people could have a calamitous effect.
In terms of hiring mathematical scientists, Australia has an almost closed cycleof employment. Our special circumstances,including the distance from here to Europe
or North America, mean that we seldom attract strong mathematicians from abroad tomake their careers in Australia. The UKcan, and does, hire mathematicians from theContinent (e.g., Germany, Scandinavia and250

The ‘Preferred Model’ for Australia’s Research Quality Framework

Eastern Europe), but the difficulties experienced in the UK, arising from a reduction indiversity attributable at least in part to theRAE, are nevertheless pronounced.
The intense competition engendered inAustralian universities, by government policies over the last decade, has already greatly
reduced inter-university cooperation in research and teaching. The new contests thatwould be produced by an RQF, when magnified by the small size of the Australian research enterprise, might carry competition
to a still more detrimental level. In the UK
the Royal Society of Edinburgh has noted
that, ‘in a small country like Scotland,’ thefact that ‘there has not been sufficient stimulus to inter-institution collaboration’ resulting from the RAE ‘has been counter productive.’On the same theme, the UK’s National
Academies Policy Advisory Group has remarked that, while the RAE encourages
inter-university competition, it fails to promote the benefits of collaboration and networking. NAPAG has also noted that thesubstantial funding and performance pressures flowing from the RAE make it relatively difficult for early-career researchers toestablish themselves. Attracting, and finding career paths for, young Australian mathematical scientists is one of the most serious
challenges we face.
The Australian government appears tohave a hidden, although somewhat distant,agenda for the RQF. In an interview reported in the Financial Review on 10th September, the Minister remarked: ‘It is myclear intention that we will also be building
a model to apply the RQF, or a variationof it, to the ARC competitive funding.’ Anearlier comment by Dr Nelson, in a press
release, had been less specific: ‘My Department will conduct a process . . . to develop
the way in which the outcomes of the RQF
will impact on the funding distributed by
the research councils.’

Understandably, the Minister’s remarks
drew immediate fire from university researchers. A linkage between the RQF andspecific research grants would lead universities to tie RQF results, and both contested and block-grant research funding,very closely to individual scientists andscholars. The linkage would also greatly reduce the ‘diversity of perception’ in judging research, by narrowing the range of outlooks of the people who assess Australianresearch. Diversity is crucial to making
wise decisions about research funding. Moreinsidiously, Dr Nelson might be considering using the RQF to determine those universities or departments that are suitable
for research-grant funding, and might refuse
access to the grant competition to thosegroups that do not score sufficiently highly
in RQF assessments.
To be fair to the Minister, it should be
noted that he qualified his remarks with theobservation that his plans ‘probably won’t
eventuate in my time in the portfolio.’ So,
we probably have a while in which to develop our arguments against his proposal.
The Minister’s comments pull into sharp
focus the intrinsic redundancy of a researchassessment program which both assesses research proposals by peer review (for example, reviews of ARC and NH&MRC grants)
and subsequently, at great cost in time andmoney, dissects the entrails of the research
to determine, in effect, whether the original decisions were correct. Bob May, in
his President’s Anniversary Address to theRoyal Society in 2003, commented that hewas ‘unaware of any other country [besidesthe UK] which parallels the usual peerreviewed competitive process for specific research projects with a second, and hugely
elaborate, process of frequent review andgrading of essentially every department inevery university in order separately to allocate infrastructure costs.’ The Preferred
Model paper argues that Australia should
join the UK in this category.

Peter Hall

Under the Preferred Model the first RQF
would be implemented in 2007, from which
point it would occur on a cycle of approximately six years, except that there would bea gap of only three years between the firstand second RQFs. The first RQF would assess work in calendar years 2001 to 2006,
and would judge institutions on the basisof those researchers in their employ on 1stJanuary 2007.
In the meantime, comments on the ‘Preferred Model’ paper should be submitted
to the government by 4th October 2005.
They can be sent to the email address
rqf@dest.gov.au. I suggest that the Mathematical and Statistical Societies quickly develop responses to the paper. Recommendations and suggestions might include one ormore of the following:• Rules and guidelines for operation of theRQF must safeguard career paths forearly-career researchers, and in particular must ensure that these people areencouraged, and not penalised, by the
research assessment process;• The RQF must have the ability (whichit would not enjoy under the Preferred Model) to assess effectively crossdisciplinary work, for example mathematical work in finance, biology andbioinformatics;• The representation of fields by panels
and sub-panels in the RQF should assure scientists and scholars that theirwork is being carefully and knowledgeably assessed by their peers;• Measures should be put in place toensure that the RQF does not makeAustralian National University, Canberra ACT 0200
E-mail: Peter.Hall@anu.edu.au251

the Australian research enterprise evenmore fragile than it already is, throughconcentrating excellence in two few institutions and too few fields, and failing to support pockets of excellence inlarger groups;• In fields where quality and impact are
not closely linked to short-term practical outcomes, this fact should be fullyrecognised by the RQF assessment process;• When metrics and other sources of information are used as part of the assessment process, care must be taken
to incorporate them in an appropriate manner (note, for example, thatappropriate time-scales for bibliometric data in mathematics are usuallymuch longer than the formal ‘assessment period’ foreshadowed in the Preferred Model);• The potential advantages of cooperation by institutions in research should
be recognised by the RQF, and cooperation should not be sacrificed in favourof competition; and• The work-load on universities for implementing the RQF should not be
any greater than that for current research assessment procedures, without
adequate financial support through newfunding.
Comments and other assistance fromTony Guttmann, Alex Reisner and Jan
Thomas were very helpful in the preparation of this article.