|

Current status of quantitativeevaluation of humanities andsocial science in global
M’hamed el Aisati,
Vice President of Product Management Funding & Content Analytics
ElsevierTokyo, 21 June 20191

|Agenda1

Should metrics be considered for evaluating
research performance of Social Sciences (SS) andArt & Humanities (A&H) output?

2

What arrays of metrics should be considered?

3

What are global performance evaluators doing
evaluate SS and A&H?

2

|Two Golden Rules of using research metrics
to give a balanced, multi-dimensional view3|

Agenda1Should metrics be considered for evaluating
research performance of Social Sciences (SS) andArt & Humanities (A&H) output?

2

What arrays of metrics should be considered?

3

What are global performance evaluators doing
evaluate SS and A&H?

4

|

Global Representation means global discovery

Across all subjects and content types
Scopus includes content from more than 5,000 publishers and 105
different countries•••

40 different languages covered
Updated daily
Multiple regional content types covered (journals, conferences, books, book series)
Number ofJournals bysubject area

PhysicalSciences
12,263
HealthSciences
13,819SocialSciences
10,905
Life
Sciences
6,809

Journals

ConferenceBooks

23,507

106K

613

Peer-reviewed journals

Conference events

Book series

301

8.3M

38K

Trade journals

Conference papers

Volumes

3,784

Mainly Engineering andComputer Sciences

1.5M

Active Gold Open Accessjournals>8,000Articles in Press

Full metadata, abstracts
and cited references

Source: Scopus.com, April 2018

Items

219,967
Stand-alone books

1.34M
Items
Focus on Social Sci and A&H

5

|Global Representation means global discovery

Across all subjects and content types
NorthAmerica
6,000+Middle East& Africa
750+

WesternEurope
11,000+

East Europeincl. Russia
1,400+

50% morethan nearest
competitor

212% morethan nearest
competitor

69% morethan nearest
competitor

168% morethan nearest
competitor

Latin
America
700+

Asia
Pacific
2,000+

Australia/
New Zealand300+

168% morethan nearest
competitor

230% morethan nearest
competitor

206% morethan nearest
competitor

GlobalRepresentation(number of titles)

6|

More titles across all subjects compared to nearest
competitor

60.5% moreunique titles

49% more unique titles

7

|

SS&AH publication coverage evolution
CAGR= 7.8%

# of docs published
600,000

500,000

400,000

Books expansion
program
300,000

200,000

Scopus launch
100,000

0
2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

8

|Titles reviewed per subject area: SS&AH have thehighest acceptance rate

>72%
~50%

9

|

How differently are SS and A&H papers read and cited?

% of Scopus papers in Mendeley with readership stats

Plum Capture Readership
100.0%

90.0%

80.0%

70.0%

60.0%

50.0%

40.0%

Citations10

|Correlation between Plum Capture Readership andCitations
100.0%% of Scopus papers with Plum Capture Readership

R(SS and H&A) = 0.9243516

90.0%

Business,Managementand Accounting

Psychology

Economics,Econometrics
and FinanceSocial Sciences

80.0%

Decision Sciences
Arts and Humanities
70.0%

60.0%

50.0%

40.0%
40.0%

50.0%

60.0%

70.0%

80.0%

% of Scopus papers with citation stats

90.0%

100.0%

11

|Agenda

1

Should metrics be considered for evaluating
research performance of Social Sciences (SS) andArt & Humanities (A&H) output?

2

What arrays of metrics should be considered?

3

What are global performance evaluators doingevaluate SS and A&H?

12

|

Arrays of metrics for eachentity in the researchecosystem

13

|

Example: Top 10 most used metrics in SciVal

14|Evaluating research performance through citations: theFWCI
Field-weighted citation impact is an indicator of mean citation
impact, and compares the actual number of citations received byan article with the expected number of citations for articles of thesame document type (article, review or conference proceedingpaper), publication year and subject field*.

The Field-Weighted Citation Impact (FWCI) for a set of N publications isdefined as:
𝑁FWCI ≡1𝑐𝑖෍𝑁𝑒𝑖
𝑖=1𝑐𝑖 = citations received by publication i
𝑒𝑖 = expected number of citations received by all similar publications in the publication year plus
following 3 years*: Where the article is classified in two or more subject fields, the harmonic mean of the actual and expected citation rates is used.

15

|

16

Capturing other metrics to better tell the story ofResearch

USAGE(clicks, downloads, views,
library holdings, video plays)

MENTIONSSOCIAL MEDIA(blog posts, comments,reviews, Wikipedia links)(+1s, likes, shares, tweets)

CAPTURES(bookmarks, code forks, favorites,readers, watchers)

CITATIONS(citation indexes, patentcitations, clinical &policy citations)

16

17

|

17

Innovation in Research Metrics : Clinical and Policy “citations”

17|Societal Impact – Clinical Citations

18

|

Societal Impact – Policy Citations

19

|

Visualizing Impact: Plum Print• Embeddable widget- Article Pages- Search Results- Institutional Repository Pages- CRIS/RIMS Pages• Includes the 5 categories ofmetrics• Circles dynamically change size
based on metrics in each category20

|Agenda

1

Should metrics be considered for evaluating
research performance of Social Sciences (SS) andArt & Humanities (A&H) output?

2

What arrays of metrics should be considered?

3

What are global performance evaluators doingevaluate SS and A&H?

21

|

The Excellence of Research for Australia (ERA) Clusters
Cluster 1

Physical, Chemical & Earth Sciences

Cluster 2

Humanities and Creative Arts

Cluster 3

Engineering and Environmental Sciences

Cluster 4

Social, Behavioural and Economic Sciences

Cluster 5

Mathematics, Information and Communication Sciences

Cluster 6

Biological Sciences and Biotechnology

Cluster 7

Biomedical and Clinical Research

Cluster 8

Public and Allied Health, and Health Sciences

22

|23

ERA - Different measurement criteria
Volume andActivity- Staffing Profile- Research Outputs- Journal articles andlisted conferences
are apportioned
against the FoR
codes of the ERA
Journal andConference Lists.
- All other outputs
and eligible
researchers can be
assigned andapportioned to up to
3 four-digit FoR
codes of theinstitution’s choice.
- Staff must be
employed on thecensus date to counttowards ‘FTE’.

RankedOutlets- Ranked Journals- Ranked
Conferences- All disciplines use
Ranked Journals as
an indicator; only
some FoRs use
RankedConferences.
- Where Ranked
Conferences isapplicable,
institutions must
apportion against theFoR codes of theERA Conference list.
- Where an output ofa listed conference
does not belong to
the listed FoR,
institutions cansubmit the output asa non-listed
conference andassign up to three
FoRs of their choice.

Citation
Analysis- Relative Citation
Impact (RCI) against
world and Australianbenchmarks- RCI Classes- Centile Profile- Only applies tojournal articles- Low volume
threshold is 50
apportioned indexed
journal articles.
- The citation supplierfor 2010 is Scopus.
- The citation census
date is 1 March 2010.

Peer Review- Peer review- Applies to a rangeof outputs including
journal articles,
books, bookchapters, creative
outputs, etc.
- Low volume
threshold is 30
apportioned outputs(any type).
- Institutions
nominate 20% of
total output for a FoR
for peer review.

EsteemMeasures- Editor of prestigious
works of reference- Recipient of, Category 1
Fellowship or Australia
Council Grant or Fellowship- Membership of, statutory
committee and Learned
Academy- Institutions canselect up to 3 fourdigit FoRs
(apportioned) for
each individual
esteem measure.
- Esteem must be
linked to a listed staffof the institution.
- Individuals
researcher cannot be
identified through theesteem measures.

Research
Income- Category 1-4- Institutions canselect as many fourdigit FoRs
(apportioned) for
each income item
submitted.
- Number of grants iscollected forCategory 1 income
only.
- FTE is used as adenominator for all
Categories.
- Category 3 income
is disaggregated into
the 3 subcategories(Australian,International A andB).

Appliedmeasures- Patents- Commercialisation
Income- Plant Breeder’sRights- NHMRC Endorsed
Guidelines- Registered Designs- Institutions canselect up to 3 fourdigit FoRs
(apportioned) for
each applied
measure submitted,
except forcommercialisation
income where there
are no limit on thenumber of four-digit
FoR codessubmitted.
- Applied measures
are linked to theinstitution, not theindividual.

|24

11 Unit of Assessments (UoAs) opted to use bibliometricsfor the Research Excellence Framework (REF) 2014

|REF 2014 - Contextual dataFor Panels
Contextual Data (11 sub panels using citations)
••••
•
•

Number of citations received by papers in the top 1%, 5%, 10% and 25% of cited papers (percentiles)
Average (mean) number of times that papers have been cited
Period 2008-2012
Provided at level of each Unit of Assessment (UoA)
And provided at level of each ASJC included in each UoA
Provided right after closure of submission period Dec. 2013

Comparative Data (Main panels A & B)•••••

Provide a set of indicators to help inform these main panels’ deliberations about the overall output
sub-profiles at UOA level
For: UK, USA, Germany, France, Japan, Canada
At UoA level
Period 2008-2012
Indicators will be:• The average field normalised citation score for papers from each country• The proportion of each country’s outputs that are in the top 1%, 5%, 10% and 25% of the world’s
outputs in each of these categories• The number of outputs produced by each country25

|SS and A&H in university rankings (1/3)
Example (THE WUR):
Which ranking indicators are driving Japan current ranking
performance?

Ranking
indicators
TeachingResearch
Citations
IndustryInternationalOutlook

All subjects (89 universities)
Percentage Percentage
ofofuniversities universities
with score with score
above
below
Medianmedianmedian24.9
31.50%
68.50%
18.9
20.20%
79.80%
47
5.60%
94.40%
38.4
47.20%
43.8

43.7

1.10%

98.60%

Clinical and Health research (26universities)
Percentage Percentage
ofofuniversities universities
with score with score
above
below
Medianmedian
median24.5
57.70%
42.30%
21.9
50%
50%
62.7
7.70%
92.30%
40.6
69.20%
30.80%

49.3

0%

100%

Social Sciences (6 universities)
Percentage Percentage
ofofuniversities universities
with score with score
above
below
Medianmedian
median27.1
66.70%
33.30%
26.6
50%
50%
68.4
33.30%
66.70%
39.8
16.70%
83.30%

51.5

66.70%

33.30%

26

|

Importance of books in assessment and rankings
Article, Review,
Conference Papers,
Books and Book
Chapters

September 12, 201127

|Contribution of Research to the UN Sustainable Development
Goals (SDGs) also evaluated through metrics

28

|Questions?
m.aisati@elsevier.com

www.elsevier.com/research-intelligence29