|

1|

2

Evaluating researchcompetitiveness using metrics
Dr Lisa Colledge, Director of Research Metrics
26 October 2016

The importance of research metrics
Overview of types of research metrics
Recommendations for measuring research competitiveness
Trends in the use of research metrics

|

3|

4

Research metrics help us to make better decisions
More informed- Reduce chance of error-Metrics are responsive to changes in performance
Metrics enable benchmarking
Metrics can save time and money, if used wisely to indicate wherepeer review should be used for validation
Metrics are an objective complement to expert opinion
Both approaches have strengths and weaknesses- Valuable information is available when these approaches differ inmessage- Always use both approaches in combination-Rankings

Peer review

Qualitative

Expert opinion

with

Quantitative

Narratives

Benchmarking
Narrative support

Always use more than 1 research metric
A strong research ecosystem produces and recognizes diverse types ofimpact- Reduces the chance of gaming- Reduces the chance of driving undesirable changes in behaviour-Select metrics that represent behaviour you want to encourage
Data source coverage, disciplinary focus, and timeline should
affect the metrics selected
Select a set of metrics whose
weaknesses

|

Field-Weighted Citation
Impact= 2.53Citations per Publication= 27.8withCompensates for differences infield, type and age

Large number
Simple, easy to validate
Communicates magnitude ofactivity

253% of expected
× People may not like small numbers
× Complicated; difficult to validate
× No idea of magnitude: how manycitations does it represent?

5× Affected by differences in field,
type and age
× Meaningless without additionalbenchmarking
5|Facet

Theme

Metrics in areas ofFundingAwards

Number, value and duration of awards

Can I support my research?

Outputs

Productivity

Number, types and growth of outputs

How productive am I?

Visibility

Impact of publication outlets

How prominent is my output in top outlets?

Research
ImpactInfluence
How is my output used in academia?

Enterprise
How is my output used in industry?

Engagement

Network

Commercial use (patents, licenses, spin outs,consultancy)

How well linked am I within academia?

Collaboration: geographical, crossdisciplinary
Network: number of collaborators, centrality,connectedness, geographical extent

Connections

Collaboration: cross-sector

How well linked am I outside academia?

Mentoring
How do I transmit knowledge?

SocialImpact

Views, citations
Reputation: awards, prizes, editorships

Social Impact
What is my wider impact?

Crowd-sourcing: collect and analyze data,raise funding
Who supervised me, and who have Isupervised?
well being, and understanding of research

6

|FacetTheme

FundingAwards

7

Can I support my research?

Outputs

Productivity
How productive am I?

Visibility
How prominent is my output in top outlets?

ResearchImpact

InfluenceHow is my output used in academia?

Enterprise
How is my output used in industry?

Engagement

Network

A selection of metrics, eachanswering a different need:
Based on different typesof data
Includes both basic
counts and sophisticatedfield-normalization

How well linked am I within academia?

Connections
How well linked am I outside academia?

Mentoring
How do I transmit knowledge?

SocialImpactSocial Impact
What is my wider impact?

|Article

Researcher

JournalBook

Research data
Facet
Funding
Outputs

ResearchImpact
Engagement

SocialImpactInstitutionCountry

Funder

Library

Subject area

Publisher8|CharacteristicRanking

Benchmarking

ExampleTimes Higher institutional
ranking

SciValUse cases

Ranking

Evaluating, showcasing,scenario modelling

Metric selection

Fixed, independent ofparticular situation

Flexible, responsive to particularsituationPeer selection

All institutions meeting
inclusion criteria

Flexible, responsive to particularsituation

Nature ofcomparison

Absolute performance

Relative performance

Currency

Annual snapshot

Real-time

Manner of use ofmetrics

Aggregated metrics withweighted contributions

Individual, separate metrics
custom selected for the question

Intended use

PublicPrivate within personal network|Article

Researcher

JournalBook

Research dataInstitution

Country

Funder

Library

Subject area

Publisher

Facet
Funding
Outputs

ResearchImpact
Engagement

SocialImpactBenchmarking works well when:
1. The question is clearly defined, so that
2. Appropriate metrics can be selected, and
3. Calculated for the entities whose
performance you are investigating, and also
4. For suitable peers

910

|

Facet

Theme

FundingAwards
Can I support my research?

Outputs

Productivity
How productive am I?

Visibility
How prominent is my output in top outlets?

ResearchImpact

Influence

11A selection ofmetrics from thebasket to cover
multiple facets andthemes ofimportance to theexercise

How is my output used in academia?

Enterprise
How is my output used in industry?

Engagement

NetworkHow well linked am I within academia?

Connections
How well linked am I outside academia?

Mentoring
How do I transmit knowledge?

SocialImpact

Social Impact
What is my wider impact?

Number of metrics shown is indicative only|A core set ofmetrics to be usedby all institutions+12

Complementary sets ofmetrics for disciplines andinstitution types

Consider also different weightings per facet and theme perdiscipline and institution type
Number of metrics shown is indicative only

|13

Many existing metrics are suitable but the coverage of large data
sources is not sufficient
Any such metrics will need to draw on Japanese data sources
There may be multiple sources e.g. at each institution
Recommendations
Benchmarking is useful when it compares apples with apples, so it isimportant to define Japanese language metrics unambiguously
Test the metric definition before finalising
Work with a group of Japanese experts with local data sources
Consider validating the method with other countries with local
language considerations e.g. China, Germany, France|

Practical expertise is growing from shared experiences
Increasing use of research metrics- Increasing forums, expert groups, and discussions-All stakeholder groups are using metrics, not only
bibliometricians
Also concern when use of metrics is not declared openly- Resistance when evaluators are not transparent about metrics they use-Growing demand for a more diverse range of metrics, especially
Social impact metrics- Alternative metrics- Metrics for articles and researchers-14

Practical expertise is growing
from shared experiences
Increasing use of research metrics- Increasing forums, expert groups,
and discussions
-|

15|

16

Learn from the researchcommunity e.g. Snowball Metrics,
user testing, surveys

WorldUsage from Benchmarking module from 11 March 2014 to 28 June 2015)
Scholarly Output it excluded since this is the default

|17

All (n=31)
Relative importance
now

65
4

Relative importanceif available inputs
were not a limitation

32
10
Funding

Outputs

ResearchImpact

Current weighting

Engagement Social Impact Qualitative
Ideal weighting

UK (n=19)

New Zealand (n=12)66554433221100
Funding

Outputs

ResearchImpact

Current weighting

Engagement Social Impact Qualitative

Funding

Outputs

Ideal weighting

ResearchImpact

Engagement Social Impact Qualitative

Current weighting

Ideal weighting|18

All (n=31)
Relative importancenow

65

Relative importanceif available inputs
were not a limitation

432
10
Funding

Outputs

ResearchImpact

Current weighting

Engagement Social Impact Qualitative
Ideal weighting

Librarians (n=5)

Institutional data analysts (n=24)66554433221

100
FundingOutputs

ResearchImpact

Current weighting

Engagement Social Impact Qualitative

Ideal weighting

Funding

Outputs

ResearchImpact

Current weighting

Engagement Social Impact Qualitative
Ideal weighting

|Practical expertise is growing
from shared experiences

Learn from the researchcommunity e.g. Snowball Metrics,
user testing, surveys

All stakeholder groups are using
metrics, not only bibliometricians

We broadcast what we have learnt
as simply as possible

Also concern when use of metrics isnot declared openly- Resistance when evaluators are nottransparent about metrics they use

We provide transparency tounderlying data and about metrics

19-|

Always use both qualitativeand quantitative input into
your decisions

Always use more than one
research metric as thequantitative inputThis is about benefitting from thestrengths of both approaches, notabout replacing one with the othercomplement the weaknesses ofothers

Combining both approaches will get
you closer to the whole story

There are lots of different ways of
being excellent

Valuable intelligence is available
from the points where theseapproaches differ in their message

Using multiple metrics drivesdesirable changes in behaviour

20|

https://www.elsevier.com/researchintelligence/resource-library/scivalmetrics-guidebook

https://www.elsevier.com/researchintelligence/resource-library/usageguidebook|

Practical expertise is growing
from shared experiences

Learn from the researchcommunity e.g. Snowball Metrics,
user testing, surveys

All stakeholder groups are using
metrics, not only bibliometricians

We broadcast what we have learnt
as simply as possible

Also concern when use of metrics is
not declared openly- Resistance when evaluators are nottransparent about metrics they use

We provide transparency tounderlying data and about metrics-Growing demand for a morediverse range of metrics, especially
Social impact metrics- Alternative metrics- Metrics for articles and researchers-21

Investing in new data sources tosupport a broader range ofresearch metrics
Continue to develop and champion
traditional, familiar metrics
e.g. journal, citation

22

CiteScore 2015 value =|

23|

24

AA
B
B

Partial screenshot from Scopus prototype; not yet final|Research metrics help us to make better decisions, if used
responsibly
Golden Rule 1: Always use both qualitative and quantitative input into
your decisions- Golden Rule 2: Always use more than one research metric as thequantitative input-Successful benchmarking relies on a clearly defined question, aselection of appropriate metrics benchmarked against suitable peers
Suggest a common core of metrics, with complementary sets
suitable for particular disciplines and institution types
Test Japanese language metrics with a group of Japanese experts
with their own data sources, and consider validating with e.g. China
Use new metrics alongside traditional metrics, not instead ofthem25