Interpreting thecitation performanceof individual researcherswith beamplots
By Martin Szomszor and David A. Pendlebury

Executive summary
This report presents a new visualization tool that showcases the range of a
researcher’s publication and citation impact in a single data exhibit – the Webof Science™ Author Impact Beamplots. Rather than drawing attention to asingle point metric, such as the h-index, beamplots draw the user into the dataand surface details that are usually obscured by summary indicators.

This presentation significantly
enhances the researcher evaluation
toolkit by providing deeper insights
into an individual’s publication portfolio
and is well aligned with community
efforts to reform research assessmentand encourage the responsible useof metrics. Beamplots make use of
field-normalized citation metrics anddo not unduly penalize those with gaps
in the publication record or those whowork in fields with distinctly different
publication activity.
We provide guidance on how
to interpret beamplots using ahypothetical hiring scenario and real(anonymized) data. The guidance
highlights that although publicationand citation data are useful indicatorsof research activity and impact,
they must be considered alongsidea variety of contributions made byacademics and suitably contextualized
for the individual’s particular history.

2

Beamplots make useof field-normalized
citation metrics
and do not unduly
penalize thosewith gaps in the
publication recordor those who workin fields with
distinctly different
publication activity.

Context
Many researchers are rightly
concerned about management
approaches that reduce their workto performance scores (Hammarfelt& Rushforth, 2017). Publication andcitation metrics have become morecommon and more prominent inacademic appointments, promotions
and funding despite the historic
preeminence of peer review. When
individual evaluation involves such
metrics, the task is both delicate anddangerous (Edwards & Roy, 2017).
The h-index (Hirsch, 2005) is one of
the most frequently used metricsof achievement because it appears
to summarize both research output
and influence in a single number. For
example, a researcher with an h-index
of 17 has 17 publications each cited
at least 17 times. But it is important
to ask, "What’s behind the score?"
The h-index of two individuals may
be the same despite very different
contributions: one may have somemoderately cited and many uncited
papers while another has fewer but
much more highly cited papers.
The h-index has other defects(Waltman & van Eck, 2012). It isbiased in favor of senior researcherswith many publications and to thedetriment of early career researcherswith briefer bibliographies. It cannot
compare disciplines that differ in theiraverage citation rates.

3

Nor does it account for the time papers
take to accrue citations. It remains steadyand does not fall even if a researcher
produces nothing new. Dependence onsuch simple and inadequate metrics isa sign of indicator impoverishment andlack of awareness of best practices.
Researchers have proposed reforms
and practices to address misguided
and improper uses of metrics. The San
Francisco Declaration on Research
Assessment (DORA, 2012) condemns
the use of the Journal Impact Factor™
to judge individual papers or people.
Scientometricians themselves state
explicitly: "Don’t reduce individual
performance to a single number"(Wouters et al, 2013). The Leiden
Manifesto (Hicks et al, 2015) offers 10
principles for responsible researchevaluation and Clarivate also concurs
that "Indicators must not substitute for
informed judgement."
Dubious evaluation practices also
undermine research integrity(Szomszor & Quaderi, 2020) because
they incentivize practices that canbe gamed through manipulation andmisrepresentation of the publicationand citation record. Consequently,
Responsible Research Assessment(RRA) – "…an umbrella term forapproaches to assessment which
incentivize, reflect and reward theplural characteristics of high-quality
research, in support of diverse andinclusive research cultures" (Curry et
al, 2020) – is a growing topic of interest
and participation from institutions andfunders alike.

Qualitative considerations, which
require peer expertise, are essential toinformed judgment, and deploymentof more informed metrics that address
the multidimensional nature ofresearch activity is needed (Moed &Halevi, 2015). For example, percentile
scores should be preferred in dealing
with skewed citation distributions,and these should be normalized
for field and year of each paper(Bornmann & Marx, 2014a). It should
also be mandatory to drill down intoa collection of papers to understandwhat went into a summary statistic.
Clarivate recommends that individual
researcher evaluation should always
include a variety of appropriatequantitative indicators and qualitative
assessment, not least the carefulreading of the researcher’s best
representative publications. For
assessors, the goal should be torecognize and appreciate the gestaltof individuals. To this end, Clarivateemphasizes profiles over metrics(Adams et al, 2019).
A new researcher evaluation tool,
the beamplot visualization, is an
excellent alternative to a single point
metric, such as the h-index, because
it reveals the volume and citation
impact of an individual’s publication
portfolio through time. Author
Impact Beamplots are now available
through the Web of Science™
Author Records. In this report,
we show how to read a beamplot
and provide guidelines on theirinterpretation and limitations.

Beamplots
It was Lutz Bornmann andWerner Marx of the Max Planck
Society who first introduced anddeveloped the use of beamplots
for scientometric data in 2014
(Bornmann & Marx 2014a, 2014b;
also, Bornmann & Haunschild,
2018, Haunschild et al, 2019).
A beamplot puts a researcher’s
publications into a context suitablefor comparison and unpacks
the citation performance of apublication portfolio. Each paper’s
citation count is normalized (forexample, it is benchmarked against
other similar publications) andmeasured as a percentile (seesidebar ‘What are percentiles?’).
An example beamplot is annotatedin Figure 1. The x-axis plots citation
performance as a percentile (0 onthe left is lowest and means zerocitations - 100 on the right is the
maximum and corresponds to thehighest citation count in the field),with a purple circle representing an
individual paper that is placed on
the beam according to its citation
percentile. Multiple papers withthe same percentile are shown witha larger marker. The y-axis shows
publication years – with papers
for each successive year plottedon a beam, most recent at the top
and oldest at the bottom. In eachyear, the green circle is the medianpercentile value for publications inthat year, and the overall average isshown using a vertical dotted line.
This plot was generated in 2021,
so the current year and prior yearof publications are not shown.

4

We suppress recent years fromthe plot for two reasons:• it takes years (approximately three,longer for social sciences andhumanities) for papers toaccumulate meaningful citation
impact (Wang,2013); and• in recent years, where mostpublications have zero citations,
there is a skew in percentiles towards
100. For example, the receipt ofjust one citation in the current
year can elevate the percentile
score to over 50 immediately. If
the current and prior years were
included in the beamplot, there
would be a tendency to observea skew towards 100 in the recent
years that would unduly influenceinterpretation of the data.
These limitations mean that new
researchers will not be able to see their
publication record in a beamplot.
But as Bornmann and Marx note in
the original publication, "…young
scientists should not pay too muchattention to bibliometric indicatorsand should just concentrate on theirresearch. As a rule, good research
results in attention and, later on,
in a substantial citation impact"(Bornmann & Marx 2014b).
Beamplots provide an excellent
way to unpack a single point metric.
Because they are based on percentile
measurements of citation impact, they
are more amenable to comparison
across fields than the h-index. The
method of normalization used alsocaters for article type (i.e., an article
or a review) so there is no implicit
advantage associated with certain
types of documents that on average
are more cited, such as reviews.

Like the h-index, the use of beamplots
is limited to individuals who haveaccumulated a significant publication
portfolio over a number of years.
Unlike the h-index, beamplots are notnecessarily biased against individuals
with career breaks or changes in volume
of publication outputs – gaps in therecord will be obvious in a beamplot,
but the citation performance on eitherside will be accurately reflected.
The crucial feature of beamplotsthat positions them favorably in thecontext of responsible researchassessment is that they steer us awayfrom reduction to a single point metricand force us to consider why thecitation performance is the way it is. As
we outline further in the next section,
the beamplot is a useful narrative
tool that can refute or corroborate
other evaluation criteria, and should
be considered alongside contextual
information about an individual, such aswhere they were working at the time,
the nature of any collaborative projects
and the type of research involved.

A beamplot puts
a researcher’s
publications into acontext suitable forcomparison andunpacks the citation
performance of apublication portfolio.

Recent papers

Figure 1: Example Beamplot

2019
2018
2017

Older papers

2016
2015
2014
2013

Earliest papers

2012
2011
2010
2009
0

20

40

60

80

100

Citation percentile
Most citations

Zero citations

Individual papers are plottedon the x-axis according tocitation percentile

Multiple papers with thesame percentile are shownwith a larger marker

Yearly medians show theaverage percentile foreach year in the series

Global median shows
the average percentile
for all papers

The crucial feature of beamplots that positions themfavorably in the context of responsible researchassessment is that they steer us away from reduction
to a single point metric and force us to consider why
the citation performance is the way it is.

5

90%+
score means that apaper is among the 10%
most cited in the field

When comparing citation counts, it
is important to account for variation
in how citations accumulate acrossdifferent fields. Differences in thevolume of papers published, theaverage number of references made(citation density) and the tendency
to cite older or more recent work
all affect the rate at which citations
accrue. Hence, it is common for
bibliometric indicators to accountfor these differences using
normalization – the comparison ofcitation counts to a reference set(or benchmark) that is based on:• Papers publishedin the same year
• Papers of the same document type(to distinguish articles from reviews
or proceedings, for example)• Papers published in the same
field or subject category

6

A simple way to address this problem
is to use percentiles instead of theraw citation counts. We illustrate thisin Figure 2 where the distribution ofcitation counts is shown for organic

chemistry articles published in
the Web of Science in 2015. In purple(left axis), the number of citations
received is plotted from 462 on theleft (the highest citation count), all theway to 0 on the right and shows thecharacteristic long-tail distribution. In
green (right axis), the corresponding
percentile is shown, starting on
the left at 100 (for example, 100%of other articles in the distribution
have a lower citation count) anddropping regularly down to zero. The
figure highlights three points (30, 10
and 5 citations) to show percentile
values. The steps that become more
noticeable toward the right convey
the discrete drops in citation count,with the final step denoting thearticles that received just one citation.

Figure 2: Citation counts (purple, left axis) for 19,264
articles published in organic chemistry in 2015 and theircorresponding percentile values (crimson, right axis)

30 citations = 89th percentile

100

400
10 citations = 52nd percentile

75

300
50
200

25

5 citations = 28th percentile

100

0

0
0

10,000Articles in organic chemistry 2015

19,264

Percentile

Because citation distributions
are strongly skewed, percentiles
provide a good measure ofcentral tendency.

Citation Count

Percentiles

Using a reference set enables
comparisons of citation counts acrossdisciplines. However, the underlying
skewness in the distribution ofcitations is also a problem. Many
papers are never cited, most will
only collect a few and a very small
number will gather hundreds andperhaps thousands of them. This skew
means traditional measurements,
such as the mean or median, do
not accurately convey the average(central tendency) because thedata are not normally distributed.

Interpretation
To describe the utility of the beamplot
and to position its use within a wider
evaluation context, we propose ahypothetical hiring scenario andreview citation profiles for four
potential candidates. We suppose
the vacancy is for a junior teaching
and research position (for example,
junior lecturer or assistant professor)
in the field of organic chemistry.

When hiring for such a role, manyfactors would be considered that might
include aspects relating to professional
skills, such as experience and successin applying for grants and knowledgeof the grant award system; teachingexperience and possible quality
metrics that may have been collected,
peer-review research contributions
and editorial positions; engagement
activities (such as speaking
events, invited talks and panels,
consultations, industry links); and,socio-economic impact of research

Table 1: Bibliometric indicators for four sample researchers in organic chemistry
A

B

C

D

Paper count

28

33

23

21

Times cited

698

354

384

345

H-index

15

13

12

11

Mean CNCI

1.17

0.52

0.86

1.09

Mean percentile

63

41

56

50

% inter. collab

50%

24%

22%

14%

Q1 JIF papers

13

13

13

11

% papers in top 10%

21%

3%

4%

19%

7

(spin-out companies, patenting, policy
influence), among others. Such factors
are usually assessed in relation to theoverall strategy of the organization andthe potential for the candidate to fit wellwithin the local research environment.
To find sample candidates, we used
ResearcherID data to search forindividuals whose discipline was‘organic chemistry’, who began
publishing in 2008 or later and who had
at least 20 papers indexed in the Webof Science Core Collection™. This set ofcandidates was then refined until four
representative profiles from the samegeographic region were selected,
referred to herein as researcher A, B,
C and D. Basic summary metrics foreach are listed in Table 1 – Bibliometricindicators for four sample researchers
in organic chemistry. Each researcher
has produced a similar number of
papers, published in high quality
journals (Q1 papers in Journal Citation
Reports™), but there is variation in
their citation metrics – Times Cited,
h-index, Mean CNCI (the average
Category Normalized Citation Impact)
and Mean Percentile. Researcher
A has the largest number of papers
with international collaborators (50%).
Based on this information alone, one
might rank Researcher A highest.

In Figure 3, we provide beamplots for thesame set of researchers. With this view,new details on the trends in publicationand citation performance for each areapparent. For example, Researcher A(who has the highest overall citation
performance) has an excellent record
between 2012 and 2015, but exhibits amarked drop-off in average percentile
for the years 2016 to 2018. Researcher
B also has a similar overall trend (forexample, better citation performancein earlier publication years). Researcher
D has a different profile, with a year-onyear increase in citation performance.
Of course, there are a variety ofreasons for changes in citation
performance, many of which are
expected as careers progress. To
use the beamplot in a responsible
evaluation setting, it would benecessary to bear in mind the following:
Location – Where were they based?
Locality will influence publication
opportunity, both at a national level andat a specific institution. As researchers
move around, there is likely to be an
impact on their publication profile.
Role – What job were they doing?
Different researchers are able to devote
different amounts of time to researchowing to other responsibilities (forexample, teaching, commercializationor public engagement). Those
who supervise Ph.D. students haveadditional opportunities to publish,
although the quality of that research
may vary. Doctoral and post doctoral
students may have different prospects
depending on where they study
and who their supervisor is.

8

Collaboration – Who were theyworking with? Participation in a large
international consortium provides
additional routes to publication that
may be hard to achieve otherwise.
When internationally collaborative
papers are published, they generally
attract more citations than domestic
research and earn higher percentile
scores (Adams et al, 2019; Potter et al,2020). It may be important to consider
industry collaboration as that canbe used as a proxy for engagement
in some applied disciplines.
Nature of research – What kindof research was conducted?
Whether basic or applied, the typeof research may influence citation
performance. Interdisciplinary
research may lead to publications
across a broad set of fields that could
make interpretation of metrics more
complicated. Many academics
change disciplines over the courseof their career and that could
explain changes in performance.
In fact, the beamplot provides abasis to clarify these questions andan opportunity to investigate whenchanges may have had a positiveor negative impact. It is possible
to imagine a beamplot serving as
the focus for discussion between
evaluators and evaluatees, either ina formative or summative review.
It would be wrong to form an
opinion based solely on the data
presented in a beamplot, just as it
is to depend on a single metric. As
outlined above, context is essential tounderstanding trends in performance.

In our hypothetical hiring scenario,
finding out that Researcher D had
in fact trained in a less well known
and poorly networked institution(2010 to 2013) before moving on
to a more prestigious appointment
would likely affect the evaluation.
Therefore, it should be apparent thatneither single point metrics nor abeamplot should be used as a crude
evaluation instrument. However,
beamplots do offer advantages
because they surface variation in thedata that should be investigated andcompared against other quantitativeand qualitative indicators, leading toa more responsible use of metrics.
Importantly, they reveal the data
behind composite scores such as the
h-index, show the underlying data ona paper-by-paper basis and provide
a picture of performance over time.

It is possible toimagine a beamplot
serving as thefocus for discussion
between evaluatorsand evaluatees, either
in a formative orsummative review.

Figure 3: Beamplots for four sample researchers in organic chemistry

Figure 3a

Figure 3b

2019

2019

2018

2018

2017

2017

2016

2016

2015

2015

2014

2014

2013

2013

2012

2012

2011

2011

2010

2010

2009

2009
0

20

40

60

80

0

100

20

Citation percentile

Figure 3d

2019

2019

2018

2018

2017

2017

2016

2016

2015

2015

2014

2014

2013

2013

2012

2012

2011

2011

2010

2010

2009

2009
20

40

60

Citation percentile

9

60

80

100

80

100

Citation percentile

Figure 3c

0

40

80

100

0

20

40

60

Citation percentile

Conclusions and future work
We believe that the addition of beamplots to the Web of Science Author Records
will provide researchers and evaluators with valuable insights relating to theperformance of a publication portfolio to help remove the current dependence
on existing single-point metrics, eliminate indicator impoverishment and raiseawareness of responsible research evaluation practices.

The introduction of field-normalized
percentiles to measure citation impact
is a crucial aspect of beamplots that
addresses many criticisms of the
h-index relating to comparability acrossdisciplines and biases to portfolio sizeand continuity.
But perhaps more importantly,
we believe that providing such a tool willencourage users of researcher metrics
to consider what actually makes a metricand to engage more actively with thedata. Assuming that users consider
the interpretive guidance given in thisreport, beamplots will provide newopportunities to conduct researchassessment in a responsible manner.
We foresee a number of future
developments that would enhance
the capability of beamplots. In thisfirst iteration, only a single person’s
beamplot is displayed. Of course,
it is possible to compare themside-by-side, but a feature thatprovides this functionality explicitly
may be desirable. This would be achallenging design task due to thecomplex nature of the visualization.
Currently, all publications are displayed,
but filters could be applied to segregate
the portfolio according to certain facets
such as funder acknowledgements,
open access status, collaboration type(for example domestic, international
or industrial), author position (first,
corresponding, or last) or named coauthors and collaborating institutions.

10This kind of improvement would allow
researchers to learn more about the
publication performance of certain
projects and collaborations.
See for yourself – Go to the new Web
of Science Author Search to look up a
researcher and check out their author
record to discover their beamplot.

The introductionof field-normalized
percentiles tomeasure citation
impact is a crucial
aspect of beamplotsthat addresses many
criticisms of the
h-index relating tocomparability acrossdisciplines and biases
to portfolio sizeand continuity.

References
Adams, J., Pendlebury, D., Potter, R., &Szomszor, M. (2019). Global research report:
Multi-authorship and research analytics.
Philadelphia and London: Institute for Scientific
Information, Clarivate. ISBN 978-1-9160868-6-9

Haunschild, R., Bornmann, L., & Adams, J. (2019).
R package for producing beamplots as a preferredalternative to the h index when assessing single
researchers (based on downloads from Web of
Science). Scientometrics, 120(2), 925-927.

Adams, J., McVeigh, M., Pendlebury, D., & Szomszor,
M. (2019). Profiles, not metrics. Philadelphia
and London: Institute for Scientific Information,
Clarivate Analytics, 6-7. ISBN 978-1-9160868-2-1

Hicks, D., Wouters, P., Waltman, L., de Rijcke,S., & Rafols, I. (2015). The Leiden Manifesto forresearch metrics. Nature, 520(7548), 429-431.

Bornmann, L., & Marx, W. (2014a). How toevaluate individual researchers working in
the natural and life sciences meaningfully? A
proposal of methods based on percentiles ofcitations. Scientometrics, 98(1), 487-509.
Bornmann, L., & Marx, W. (2014b). Distributions
instead of single numbers: Percentiles and beamplots for the assessment of single researchers.
Journal of the Association for Information
Science and Technology, 65(1), 206-208.
Bornmann, L., & Haunschild, R. (2018). Plotsfor visualizing paper impact and journal
impact of single researchers in a single
graph. Scientometrics, 115(1), 385-394.
Curry, S., de Rijcke, S., Hatch, A., Pillay, D.,van der Weijden, I., & Wilsdon, J. (2020). Thechanging role of funders in responsible researchassessment: Progress, obstacles and the way ahead.
Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.13227914.v1

Hirsch, J. E. (2005). An index to quantify anindividual's scientific research output. Proceedingsof the National Academy of Sciences of the UnitedStates of America, 102(46), 16569-16572.
Moed, H. F., & Halevi, G. (2015). Multidimensional
assessment of scholarly research impact.
Journal of the Association for Information
Science and Technology, 66(10), 1988-2002.
Potter, R. W. K., Szomszor, M., & Adams, J. (2020).
Interpreting CNCIs on a country-scale: The effectof domestic and international collaboration
type. Journal of Informetrics, 14(4), 101075.
San Francisco Declaration on Research Assessment(DORA). (http://am.ascb.org/dora/ )Szomszor, M., & Quaderi, N. (2020). Research Integrity:
Understanding our shared responsibility for a sustainablescholarly ecosystem. London and Philadelphia: Institute for
Scientific Information, Clarivate. ISBN 978-1-9160868-9-0
Wang, J. (2013). Citation time window choice for research
impact evaluation. Scientometrics, 94(3), 851-872.

Edwards, M. A., & Roy, S. (2017). Academic researchin the 21st century: Maintaining scientific integrity in a
climate of perverse incentives and hypercompetition.
Environmental Engineering Science, 34(8), 51-61.

Waltman, L., & van Eck, N. J. (2012). The inconsistencyof the h-index. Journal of the American Society for
Information Science and Technology, 63(2), 406–415.

Hammarfelt, B., & Rushforth, A. D. (2017).
Indicators as judgment devices: An empirical
study of citizen bibliometrics in researchevaluation. Research Evaluation, 26(3), 169-180.

Wouters, P., Glänzel, W., Gläser, J., & Rafols, I.
(2013). The dilemmas of performance indicatorsfor individual researchers: An urgent debate inbibliometrics. ISSI Newsletter [no. 35], 9(3), 48-53.

11WS622456320 / 04

About Clarivate
Clarivate™ is a global leader in providing
solutions to accelerate the lifecycle ofinnovation. Our bold mission is to help
customers solve some of the world’s mostcomplex problems by providing actionable
information and insights that reduce thetime from new ideas to life-changing
inventions in the areas of science andintellectual property. We help customers
discover, protect and commercialize theirinventions using our trusted subscription
and technology-based solutions coupledwith deep domain expertise. For more
information, please visit clarivate.com.

www.clarivate.com/webofsciencegroup© 2021 Clarivate. Clarivate and its logo, as well as all other trademarks used herein
are trademarks of their respective owners and used under license.