Bibliometrics for
Research Management andResearch Evaluation
A Brief Introduction

Bibliometrics for
Research Management andResearch Evaluation
A Brief Introduction

Ludo Waltman
Ed Noyons
March 2018
CWTS BV
P.O. Box 905
2300 AX Leiden, The Netherlands
Tel.

+31 71 527 3909

Fax

+31 71 527 3911

E-mail info@cwts.leidenuniv.nl

0Table of contents
Preface ..................................................................................................................................................... 2
1 Introduction ....................................................................................................................................... 3
2 Bibliometric data sources ............................................................................................................ 6
3 Bibliometric analyses .................................................................................................................... 9
4 Bibliometric indicators of scientific impact ..................................................................... 13
5 Responsible use of bibliometrics ......................................................................................... 221

Preface
The use of bibliometrics in research management and research evaluation is
widespread. Consequently, a strong need has emerged for information about theapplication of bibliometrics in this context. Our center, the Centre for Science andTechnology Studies (CWTS) at Leiden University, has a long tradition in providing
bibliometric training and education. By bringing together the most essential
information about the use of bibliometrics in research management and researchevaluation, the present report serves to anchor the knowledge our center has
developed over the years.
This report is intended for anyone with a professional interest in the application of
bibliometrics in the context of research management and research evaluation. Target
audiences include policy makers, research managers, and their support staff at
research institutions and research funders, as well as university librarians andindividual researchers. The report offers a brief introduction into the use of
bibliometrics to support research management and research evaluation. It covers themost important topics, but does not aim to be comprehensive. References to othersources are provided for more detailed information.
We hope that this report will contribute to an improved understanding of the use of
bibliometrics in research management and research evaluation. Feedback on thereport will be greatly appreciated. If you have any comments or suggestions for
improvements, please do not hesitate to contact us.
We gratefully acknowledge the feedback provided by our colleagues at CWTS on an
earlier version of this report. We also acknowledge funding provided by the Austrian
Science Board to support the work on this report.

Ludo Waltman

Ed Noyons

Deputy director

Operational director

CWTS

CWTS BV

2

1 Introduction
Bibliometrics is the quantitative analysis of written documents. It is frequently used
to analyze scientific and scholarly publications. Researchers may for instance use
bibliometrics in order to get an overview of their research field and of theconnections with neighboring research areas. Bibliometrics is also often used inresearch management and research evaluation. This application of bibliometrics is
our concern in the present report. The report is based on the longstanding
experience of our center, the Centre for Science and Technology Studies (CWTS) at
Leiden University, with the use of bibliometrics in research management andresearch evaluation (see Box 1).
Box 1. Centre for Science and Technology Studies (CWTS)

The Centre for Science and Technology Studies (CWTS) is an interdisciplinary research institute atLeiden University that studies scientific research and its connections to technology, innovation, andsociety. CWTS is a world-leading center of expertise on bibliometrics and scientometrics. It extensively
studies the use of bibliometrics and scientometrics in research management and research evaluation.
CWTS BV is a company owned by Leiden University that offers bibliometric and scientometric analyses
to support research management and research evaluation. Analyses performed by CWTS BV are for
instance used by universities and other research organizations, research funders, governments, andscientific publishers, mostly in Europe but also elsewhere in the world. CWTS BV also provides
consultancy and training in the areas of bibliometrics, scientometrics, and research evaluation.
For more information, visit www.cwts.nl and www.cwtsbv.nl.

3

Research management involves complex decision making, for instance about setting
research priorities and allocating resources. Bibliometrics offers quantitative
information that can be used to support decision making in a research management
context. This information is often provided by quantitative indicators (sometimes
also referred to as ‘metrics’, although in this report we will not use this term). The
journal impact factor and the h-index are probably the best-known examples of such
indicators. However, bibliometric information can also be made available in otherways, for instance by presenting lists of influential publications or by presenting
information in graphs and other types of visualizations.
Bibliometric indicators
In a bibliometric analysis, we are often interested in obtaining quantitative
measurements of concepts such as scientific output, scientific impact, or scientificcollaboration. Exact measurements usually cannot be obtained. Instead, we typicallywork with indicators that provide a proxy of the concept of interest. For instance,
indicators based on citation counts are usually interpreted as proxies of scientificimpact. They do not provide exact measurements of scientific impact, but they doofferapproximate

information

about

the

scientificimpactofpublications,researchers, or research institutions. Indicators of scientific output and scientificimpact are the best-known bibliometric indicators, but bibliometric indicators canalso be used to provide other types of information, for instance related to scientificcollaboration, mobility, interdisciplinarity, gender, and open access publishing.
When using bibliometric indicators, it is essential to keep in mind that indicators do
not usually provide an exact measurement of the concept of interest, but can only
provide approximate information. This is one of the reasons why important research
management decisions should not be made based solely on bibliometric indicators.
Bibliometric indicators provide relevant information that is often helpful insupporting decision making. However, for most decisions the information provided
by bibliometric indicators is too limited to rely on this information exclusively.
Beyond bibliometrics
Bibliometric information can be seen as one element within a broader range of
information sources available to support decision making in a research management
context. Relevant information can for instance also be obtained from peer review aswell as from other quantitative sources, such as data on research funding and4research staff and altmetric data. However, a discussion of these information sources
is beyond the scope of this report.
Organization of this report
This report provides a brief introduction into the use of bibliometrics in research
management and research evaluation. Chapter 2 discusses the most important data
sources available for bibliometric analyses. Chapters 3 and 4 offer an overview ofdifferent types of bibliometric analyses, focusing in particular on analyses ofscientific impact. Finally, Chapter 5 addresses the responsible use of bibliometrics.
Summary•

Bibliometric information can be made available in different ways, but most often quantitative
indicators are used.

•

Bibliometric indicators provide approximate information. They do not offer an exact
measurement of a concept of interest.

•

Bibliometric information can be seen as one element within a broader range of information
sources available to support research management and research evaluation.

Further reading
Moed, H.F. (2017). Applied evaluative informetrics. Springer.
Sugimoto, C.R., & Larivière, V. (2018). Measuring research: What everyone needs to know. Oxford
University Press.

5

2 Bibliometric data sources
Bibliometric analyses require a bibliometric data source. Simple analyses cansometimes be carried out using a research institution’s internal data sources, forinstance using data extracted from a university’s internal information system. In
most cases, however, an external data source is needed. Web of Science, producedby Clarivate Analytics, and Scopus, produced by Elsevier, are the two most commonlyused bibliometric data sources. Google Scholar is also used quite frequently. In
addition, bibliometric analyses that are restricted to a particular scientific fieldsometimes rely on a field-specific data source, such as INSPIRE (high energy physics),
MathSciNet (mathematics), PsycINFO (psychology), or PubMed (biomedical researchand life sciences). Similarly, bibliometric analyses that focus on a particular region
may use a data source oriented specifically on that region. Examples of such data
sources include the Chinese Science Citation Database, the Russian Science Citation
Index, and the SciELO Citation Index. Interesting new data sources that canpotentially be of value for bibliometric analyses are Microsoft Academic andCrossref. However, at the moment the use of these new data sources is still in an
early stage and therefore experience with them is still quite limited.
Web of Science and Scopus
Web of Science consists of a number of citation indices. The most commonly usedcitation indices are part of the Web of Science Core Collection. This includes the
Science Citation Index Expanded, the Social Sciences Citation Index, the Arts &Humanities Citation Index, and the Emerging Sources Citation Index. These indicescover scientific journals in the sciences, social sciences, and arts and humanities. In
addition, the Conference Proceedings Citation Index and the Book Citation Index,
which cover conference proceedings and books, are also included in the Web of
Science Core Collection. Like Web of Science, Scopus covers journals, conference
proceedings, and books in the sciences, social sciences, and arts and humanities.
Although Web of Science and Scopus provide a broad coverage of scientific andscholarly literature, it is important to be aware that especially in the social sciences
and humanities their coverage is limited. Coverage of conference proceedings andbooks also is far from complete.
Web of Science and Scopus both require a subscription. Subscribers have access tothe data sources through a web interface. These web interfaces can be used to6search for scientific and scholarly literature and also to perform simple small-scale
bibliometric analyses. However, they are not suitable for carrying out large-scale
bibliometric analyses. Such analyses can be performed using InCites and SciVal,
commercial bibliometric analysis tools that make use of, respectively, Web of Scienceand Scopus data. There is also the possibility of performing large-scale bibliometricanalyses based directly on Web of Science or Scopus data, but this requires a special
license that provides direct access to the data. CWTS for instance has a license thatprovides direct access to Web of Science data for carrying out large-scale bibliometricanalyses.
Google Scholar
Google Scholar is an online search engine for scientific and scholarly literature. It canalso serve as a data source for bibliometric analyses. Compared to Web of Scienceand Scopus, Google Scholar has two key advantages. One advantage is that Google
Scholar is freely available. No subscription is required. The other advantage is that
Google Scholar offers a more comprehensive coverage of scientific and scholarly
literature. This is important especially for bibliometric analyses in computer science,the social sciences, and the humanities, and also for bibliometric analyses oriented
not only on international scientific and scholarly literature but also on national andregional literature. For such analyses, Web of Science and Scopus provide only
limited coverage, while the coverage of Google Scholar is more comprehensive.
However, Google Scholar also has some major limitations. First of all, it lacks
transparency. Very little information is available on what is and is not covered by
Google Scholar. Second, Google Scholar data is of lower quality than Web of Scienceand Scopus data. Duplicate records for instance represent an important problem in
Google Scholar. Web of Science and Scopus also suffer from data quality problems,
but their data is of significantly higher quality than data in Google Scholar.
Researchers can also manipulate Google Scholar data more easily than Web of
Science and Scopus data, in particular by making fake publications available online.
Finally, although Google Scholar has a more comprehensive coverage of scientificand scholarly literature than Web of Science and Scopus, it is very difficult to use
Google Scholar for large-scale bibliometric analyses. Google Scholar does not provide
large-scale access to its data.

7

Summary•

Web of Science, Scopus, and Google Scholar are the most commonly used bibliometric data
sources.

•

Large-scale analyses based on Web of Science or Scopus can be performed using InCites or
SciVal. Alternatively, direct access to Web of Science or Scopus data is required.

•

Google Scholar is freely available and provides a more comprehensive coverage of thescientific and scholarly literature. However, Google Scholar lacks transparency, suffers fromdata quality problems, and is very difficult to use for large-scale analyses.

Further reading
Halevi, G., Moed, H., & Bar-Ilan, J. (2017). Suitability of Google Scholar as a source of scientificinformation and as a source of data for scientific evaluation—Review of the literature. Journal of
Informetrics, 11(3), 823–834.
Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: Acomparative analysis. Scientometrics, 106(1), 213–228.

8

3 Bibliometric analyses
Bibliometrics can be used for a broad range of problems in research management
and research evaluation. It can for instance be applied to support strategic decision
making by the management of a research institution, to support the evaluation ofresearch institutes and research groups, or to provide information for assessing
candidates for a scientific position. In many cases, bibliometric information is madeavailable to peer reviewers, resulting in so-called informed peer review. Bibliometric
information can also be used to define indicators for allocating research funding.
Types of information
A bibliometric analysis can yield different types of information. Important types of
information include:•

Scientific output. Information about the number of publications produced bya research unit.

•

Scientific impact. Information about the number of citations that publications
have received.

•

Scientific collaboration. Information about co-authored publications, focusing
for instance on national and international collaboration or on universityindustry collaboration.

•

Mobility. Information about researchers that change their affiliation.

•

Interdisciplinarity. Information about the interdisciplinarity of publications,
usually based on the fields that are cited by a publication.

•

Gender. Information about the gender of the authors of publications.

•

Open access publishing. Information about the open access status of
publications, distinguishing for instance between gold open access, green
open access, and no open access.

Because bibliometric indicators of scientific impact play a prominent role in many
research evaluations, we discuss these indicators in more detail in the next chapter.
Level of detail
The level of detail at which information is presented in a bibliometric analysis can be
adjusted to the objective of the analysis. Strategic decision making at the level of an
entire research institution requires a high-level overview that may be best served bya limited set of quantitative indicators, perhaps complemented with time trends and9breakdowns by scientific field. Such a high-level overview is for instance provided in
the CWTS Leiden Ranking (see Box 2). Decision making at lower levels within an
organization may benefit from richer bibliometric analyses in which quantitative
indicators are complemented with other more in-depth types of bibliometric
information, for instance by providing bibliometric visualizations that offer moredetailed insights. Likewise, when comparing candidates for a scientific position, it
may be helpful to have a comprehensive bibliometric profile of each candidate, which
presents not only quantitative indicators but also links these indicators to theunderlying bibliometric information. For instance, the profile does not just report the
number of highly cited publications of a candidate, but also presents a list of thesepublications, thereby enabling a more substantive evaluation of the publications.
Box 2. CWTS Leiden Ranking

The CWTS Leiden Ranking is a bibliometric ranking of about 900 major universities worldwide. The
Leiden Ranking is produced annually by CWTS based on data from Web of Science. It offers
bibliometric indicators of scientific output, impact, and collaboration. A breakdown of the bibliometricstatistics is provided at the level of five broad fields of science, and the statistics are presented inthree different ways, referred to as the list view, the chart view, and the map view. The Leiden Ranking
is available at www.leidenranking.com.

10

Bibliometric visualizations
Bibliometric visualizations offer a powerful way to present more detailed information
in a bibliometric analysis. Visualizations may for instance provide a geographicalperspective (e.g., bibliometric indicators at the level of countries or institutions
presented in a geographical map) or a network perspective (e.g., co-authorshipnetworks or citation networks, visualized for instance using the VOSviewer softwaredeveloped at CWTS; see Box 3). When dealing with large numbers of publications, an
overview of the content of the publications can be obtained by presenting avisualization of the most important terms occurring in the titles and abstracts of the
publications. Moreover, using so-called overlay visualization techniques, it ispossible to include additional information in such a visualization, for instance toshow developments over time (see Box 3).
Summary•

Examples of applications of bibliometrics in research management and research evaluation
include supporting strategic decision making, supporting the evaluation of research
institutes and research groups, informing the assessment of candidates for a scientificposition, and allocating research funding.

•A bibliometric analysis may provide information about scientific output, scientific impact,
and scientific collaboration as well as a number of other types of information.

•

Depending on the objective of a bibliometric analysis, the information that is provided may
range from a limited set of quantitative indicators to a comprehensive bibliometric profile.

•

Bibliometric visualizations offer a powerful way to present more detailed information in abibliometric analysis.

Further reading
Van Eck, N.J., & Waltman, L. (2014). Visualizing bibliometric networks. In Y. Ding, R. Rousseau, & D.
Wolfram (Eds.), Measuring scholarly impact: Methods and practice (pp. 285–320). Springer.
Van Raan, A.F.J. (2018). Measuring science: Basic principles and application of advanced bibliometrics.
In W. Glänzel, H.F. Moed, U. Schmoch, & M. Thelwall (Eds.), Handbook of science and technology
indicators. Springer.
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E.C.M., Tijssen, R.J.W., Van Eck, N.J., ... Wouters, P.
(2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of
the American Society for Information Science and Technology, 63(12), 2419–2432.

11

Box 3. Bibliometric visualization using VOSviewer
VOSviewer is a popular software tool for producing bibliometric visualizations. It is developed by CWTS
and is freely available at www.vosviewer.com.
The figure below presents a VOSviewer visualization of a co-authorship network. Each circle represents
a researcher. The size of a circle indicates the number of publications of a researcher. Lines indicate
co-authorship links between researchers. Colors designate clusters of researchers that are strongly
connected to each other by co-authorship links.

A second VOSviewer visualization is presented below. This visualization shows the most important
terms occurring in the titles and abstracts of publications on Alzheimer’s disease. Terms that arelocated close to each other in the visualization tend to occur together in a large number ofpublications. These terms can therefore be considered to be strongly related. Colors show thedevelopment of research on Alzheimer’s disease over time. Terms colored blue occur mainly in older
publications, while terms colored yellow, orange, or red occur mainly in more recent publications.

124 Bibliometric indicators of scientificimpactScientific impact is typically analyzed by counting the number of citations thatpublications have received. There are many different impact indicators, with thejournal impact factor and the h-index being the best-known examples. Citations
occur for a variety of reasons. Some citations indicate that the citing publicationbuilds on the cited publication. These citations may be seen as an acknowledgmentof the impact of the cited publication on the citing one. Negative citations are of an
opposite nature. They reflect the citing publication’s critical perspective on the cited
publication. However, many citations are neither positive nor negative. These
citations often reflect a more superficial connection between the citing and the cited
publication. They are sometimes referred to as perfunctory citations. Given thediversity of citations, citation counts provide only an approximate indication ofscientific impact.
Citation counts are also sometimes interpreted as indicators of scientific quality
rather than scientific impact. However, this interpretation is of an even more
approximate nature. The quality of a publication can be expected to influence thenumber of citations the publication will receive, but a high-quality publication on an
obscure topic is likely to receive fewer citations than an average-quality publication
on a popular topic.
Types of impact indicators
Impact indicators can be classified in many different ways. An essential distinction is
between size-dependent and size-independent impact indicators. Size-dependent
impact indicators reflect the total scientific impact of a research unit’s publications,while size-independent impact indicators reflect the average scientific impact per
publication. A further distinction is between impact indicators that count citations
directly and impact indicators that first identify highly cited publications and thencount these publications.
Table 1 summarizes the above classification of impact indicators. The different typesof impact indicators are labeled according to the terminology used at CWTS. The
simplest impact indicators are based on direct counts of citations. These are thesize-dependent total citation score and the size-independent mean citation score.

13

The total citation score equals the total number of citations received by a research
unit’s publications, while the mean citation score equals the average number ofcitations received per publication. Impact indicators based on counting highly cited
publications first require the choice of the threshold n that determines whether apublication is classified as highly cited or not. A publication is classified as highly
cited if it has received at least n citations. The value of n is variable and could for
instance be set at 10, 20, 50, or 100, depending on how strict one would like to be
in classifying publications as highly cited. After choosing the threshold n, thenumber and the proportion of a research unit’s highly cited publications can becalculated. These indicators provide respectively a size-dependent and a sizeindependent perspective on the scientific impact of the research unit’s publications.

Table 1. Classification of impact indicators.
Size-dependent
Citations

Highly cited publications

Size-independent

Total citation score

Mean citation score

TCS

MCS

Number of highly cited pub.

Prop. of highly cited pub.

P(≥ n cit.)PP(≥ n cit.)

Compared to impact indicators based on a direct count of citations, impact
indicators based on counting highly cited publications are less sensitive to
publications that have received a very large number of citations. Impact indicators
based on counting highly cited publications are therefore more robust, which isoften seen as an advantage of these indicators.
The classification of impact indicators presented in Table 1 includes some commonlyused indicators, but many indicators are not included. An example is the h-index.
The h-index of a research unit equals the largest number h such that the research
unit has h publications that have received at least h citations each. The h-index is asize-dependent impact indicator. Like the number of highly cited publications, it isrelatively insensitive to publications that have received a very large number ofcitations. CWTS normally does not use the h-index. This is because of its
inconsistency. When two research units make the same improvement in terms of
publications and citations, their ranking relative to each other according to the hindex may reverse. The inconsistency of the h-index is illustrated in Box 4.

14

Box 4. Inconsistency of the h-index

The above figures illustrate the inconsistency of the h-index. The solid and the dashed line indicate thenumber of citations received by the publications of, respectively, research unit A and research unit B.
Publications are presented in decreasing order of their number of citations. The left figure shows theinitial situation, in which the h-index of research units A and B equals, respectively, 5 and 4. The rightfigure shows the situation after research units A and B have both published 2 new publications, eachwith 8 citations. In the new situation, the h-index of research units A and B equals, respectively, 5 and6. Hence, compared to the initial situation, the two research units have made the same improvement interms of publications and citations, but their ranking relative to each other according to the h-index
has reversed. This shows why the h-index is inconsistent.

Normalization for scientific field and publication age
Different scientific fields have different citation practices. Because of this, there arelarge differences between fields in citation density, that is, in the average number ofcitations received per publication. For instance, the average number of citations
received by publications in mathematics is about an order of magnitude smaller than
the average number of citations received by publications in some fields in the life
sciences. When a bibliometric analysis of scientific impact covers multiple fields, it isoften desirable to correct for differences between fields in citation density.
Performing such a correction is called field normalization. Field normalization isusually carried out by comparing the number of citations of a publication to thenumber of citations of other publications in the same field.
Older publications have had more time to be cited than more recent publications. On
average, older publications therefore tend to have received more citations than more
recent publications. Again, a normalization can be performed to correct for this.
Such a normalization is carried out by comparing the number of citations received by15

a publication to the number of citations received by other publications from thesame year.
Table 2 presents the normalized counterparts of the non-normalized impact
indicators listed in Table 1. The indicators are again labeled according to theterminology used at CWTS. The total and the mean normalized citation scores equal,
respectively, the total and the average normalized number of citations received by aresearch unit’s publications. A publication’s normalized number of citations is
calculated by dividing the number of citations of the publication by the average
number of citations of all publications in the same field and from the same year. In
the case of impact indicators based on counting highly cited publications, apublication is classified as highly cited if it belongs to the top x% most highly cited
publications of its field and its year. The focus is often on the top 10% most highly
cited publications, but it is also possible to consider for instance the top 1%, top 5%,
or top 50% most highly cited publications.

Table 2. Classification of normalized impact indicators.

Citations

Highly cited publications

Size-dependentSize-independent

Total normalized citation score

Mean normalized citation score

TNCS

MNCS

Number of highly cited pub.

Prop. of highly cited pub.

P(top x%)PP(top x%)

The use of normalized impact indicators involves some choices. Normalization for
scientific field requires the choice of a field classification system. In practice, thejournal categories in Web of Science and Scopus are often used as a fieldclassification system. An alternative is to define fields at the level of individual
publications instead of journals. This can for instance be done by algorithmicallygrouping publications into fields based on citation relations. A publication-levelclassification system will typically be more fine-grained than a journal-levelclassification system. Normalization based on a publication-level classification
system can therefore be expected to yield more accurate results. However,normalization based on a journal-level classification system may be more transparent
and easier to understand.

16

Another choice that needs to be made relates to the minimum age of publications
that are included in the calculation of normalized impact indicators. Very recent
publications usually have received no or almost no citations. Normalization for
publication age does not give meaningful results for these publications. Very recent
publications (e.g., less than one year old) are therefore often excluded from thecalculation of normalized impact indicators.
Credit allocation
In most scientific fields, a large majority of publications are co-authored by multiple
researchers, often also affiliated to multiple research institutions and residing in
multiple countries. This leads to the problem of credit allocation. When a publicationis co-authored by multiple research units, how should the credits of the publication
be allocated to the different research units?
The two most commonly used approaches for addressing the issue of credit
allocation are referred to as the full and the fractional counting approach. In the fullcounting approach, the credits of a publication are fully allocated to each of the coauthoring research units. In the fractional counting approach, the credits of apublication are fractionally allocated to each of the co-authoring research unit. For
instance, in the case of a publication co-authored by three research units, each unit
receives one-third of the credits of the publication.
Table 3 illustrates the full and the fractional counting approach in the calculation of
the total and the mean citation score of a research unit. We are interested in research
unit A. This research unit has authored three publications. It is the only author of
publication 1, while it has co-authored publications 2 and 3 with other research
units. For each publication, Table 1 reports the number of citations received by the
publication. In the full counting approach, the three publications and their citations
are fully assigned to research unit A. This results in a total and a mean citation scoreof, respectively, 17 and 17 / 3 = 5.67 for research unit A. In the fractional counting
approach, publications and citations are allocated fractionally to research unit A.
Consider for instance publication 2. As can be seen in Table 3, this publication is coauthored by two research units, A and B, and therefore the publication is allocated toresearch unit A with a weight of 1 / 2 = 0.50. The publication has received 3
citations, which are allocated to research unit A with a weight of 0.50, yielding 0.50
× 3 = 1.50 citations for research unit A. By performing these calculations for all threepublications, the fractional counting approach results in a total citation score of
10.17 and a mean citation score of 10.17 / 1.83 = 5.55.

17

Table 3. Example illustrating the full and the fractional counting approach.
Co-authoring

No. ofFractional

Fractional

research units

citationspublication

citationallocation

allocation

Publication 1

A

6

1.00

6.00

Publication 2

A, B

3

0.50

1.50

Publication 3

A, B, C

8

0.33

2.67

17

1.83

10.17

Total

Opting for either the full or the fractional counting approach should be informed by
the requirements of a particular analysis. However, when working with normalized
impact indicators, the fractional counting approach has an important advantage overthe full counting approach. Using the fractional counting approach, normalized
impact indicators correct not only for differences between fields in citation density,
but also for differences between fields in collaboration practices. The use of thefractional counting approach is recommended especially for analyses in which
normalized impact indicators are applied at high levels of aggregation, for instanceat the level of countries or institutions.
In addition to the full and the fractional counting approach, there are also otherapproaches that can be taken to address the issue of credit allocation. Most of theseapproaches rely on the order of the authors in the author list of a publication. This
may involve allocating the credits of a publication mostly, or even exclusively, to thefirst or the last author of a publication, or to the corresponding author. However,there are no universal norms that determine the order of the authors in the author
list of a publication, and relying on this order therefore always involves someuncertainty.

Most

importantly,

different

fieldshave

different

practicesfordetermining the order of authors. In some fields, in particular in economics, highenergy physics, and mathematics, it is common practice to order authorsalphabetically. Clearly, credit allocation in these fields should not be based on theorder of authors.
Author self-citations
Author self-citations are often excluded from the calculation of impact indicators.
Many self-citations are given for perfectly valid reasons. Nevertheless, if self-citations

18

are not excluded from the calculation of impact indicators, they can be used tomanipulate the indicators in a relatively easy way. Such manipulation can beprevented by excluding self-citations.
Impact indicators for journals
Indicators of scientific impact can also be calculated for journals. The best-known
example of an impact indicator for journals is the journal impact factor. The impact
factor of a journal is essentially determined by calculating the average number ofcitations received in a certain year by publications that have appeared in the journalin the two preceding years. Journal impact factors, calculated by Clarivate Analytics
based on Web of Science data, are published in the Journal Citation Reports. Other
examples of impact indicators for journals are the 5-year impact factor, theeigenfactor, and the article influence score, which are also published in the Journal
Citation Reports, and CiteScore, SJR, and SNIP, which are made available by Elsevier
based on Scopus data.
Journal impact indicators are often used not only for evaluating journals, but also forevaluating individual publications in a journal or the research units by which
individual publications have been authored. This is a controversial way of using
journal impact indicators. Within a journal, there are typically large differences
between publications in the number of citations received, and therefore it is often
considered inappropriate to use a journal impact indicator for evaluating individual
publications in a journal. The San Francisco Declaration on Research Assessment(www.ascb.org/dora/), an influential statement that has been signed by a large
number of individuals and organizations, for instance rejects the use of the journal
impact factor and other journal impact indicators at the level of individual
publications.
CWTS recognizes that the journal impact factor often plays a too dominant role inresearch management and research evaluation. An obsessive focus on the journal
impact factor, or on any other bibliometric indicator, is harmful. However, CWTS
does not reject the use of journal impact indicators for evaluating individual
publications and their authors. Journals have different standards for selecting the
research they publish. As a result, some journals on average publish higher-impact
research than other journals. Journal impact indicators provide information about the
average impact of the publications in a journal. The average impact of the
publications in a journal can serve as a proxy of the impact of an individual19

publication. Because of this, journal impact indicators can be helpful tools forevaluating individual publications in a journal.
In addition to the journal impact indicators mentioned above, other journal impact
indicators can be used as well. For instance, the mean normalized citation score,discussed earlier in this chapter, can also be calculated for journals. The publicationsof a research unit can then be evaluated based on the mean normalized citation
scores of the journals in which they have appeared. This results in an indicator thatis referred to as the mean normalized journal score at CWTS.

20

Summary•

Citation counts provide approximate information about the scientific impact of publications.

•

Impact indicators can be classified as size-dependent or size-independent. In addition, they
can be classified based on whether they count citations or highly cited publications.

•

The h-index is inconsistent. The indicator behaves counterintuitively in certain situations.

•

Normalized impact indicators correct for differences between fields in citation density andfor differences in the age of publications.

•

Co-authorship of publications leads to the problem of credit allocation. Full and fractionalcounting are two approaches for dealing with this problem. Fractional counting corrects fordifferences between fields in collaboration practices.

•

To prevent manipulation of impact indicators, author self-citations are often excluded fromthe calculation of the indicators.

•

Journal impact indicators may provide helpful information to support research management
and research evaluation, but these indicators should not play a too dominant role.

Further reading
Bornmann, L., & Daniel, H.D. (2008). What do citation counts measure? A review of studies on citing
behavior. Journal of Documentation, 64(1), 45–80.
DORA (2013). San Francisco declaration on research assessment. Retrieved November 23, 2017, fromwww.ascb.org/dora/.
Nicolaisen, J. (2007). Citation analysis. Annual Review of Information Science and Technology, 41,
609–641.
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,
10(2), 365–391.
Waltman, L., & Traag, V.A. (2017). Use of the journal impact factor for assessing individual articles
need not be wrong. arXiv:1703.02334.
Waltman, L., & Van Eck, N.J. (2012). The inconsistency of the h-index. Journal of the American Society
for Information Science and Technology, 63(2), 406–415.
Waltman, L., & Van Eck, N.J. (2015). Field-normalized citation impact indicators and the choice of an
appropriate counting method. Journal of Informetrics, 9(4), 872–894.
Waltman, L., & Van Eck, N.J. (2018). Field normalization of scientometric indicators. In W. Glänzel, H.F.
Moed, U. Schmoch, & M. Thelwall (Eds.), Handbook of science and technology indicators. Springer.

21

5 Responsible use of bibliometrics
Bibliometrics offers valuable information to support research management andresearch evaluation. However, this information should be used responsibly. The
following eight principles can serve as guidelines for the responsible use of
bibliometric information.
1 Be aware of the limited coverage of bibliometric data sources
Bibliometric data sources such as Web of Science and Scopus offer only a limited
coverage of the scientific and scholarly literature. International journals in thesciences are typically well covered, but coverage of national journals, journals in thesocial sciences and humanities, and conference proceedings and books is much
more limited. One should be aware of these limitations when performing bibliometricanalyses.
2 Acknowledge the importance of accurate data collection
Accurate data collection is essential for high-quality bibliometric analyses. Poor datacollection may result in bibliometric analyses that provide incorrect or misleading
conclusions. The efforts needed for accurate data collection should not be
underestimated. In a research evaluation context, the units under evaluation shouldhave the opportunity to verify the data collection.
3 Recognize that bibliometric analyses capture research performance only in apartial manner
It is essential to recognize that bibliometric analyses reflect only specific aspects ofthe performance of research units. For instance, citation statistics provide insight
into the scientific impact of research units, but they do not capture the broader
societal impact. Even scientific impact is captured only partly by citation statistics,
since these statistics consider only scientific impact that results from publications.
Scientific impact resulting from other activities, such as curation of data sets anddevelopment of software tools, is not taken into account. Furthermore, bibliometricanalyses focus on the outputs of the research process and typically do not consider
the inputs. This for instance means that bibliometric analyses provide no insight into
the productivity of research units.

22

4 Account for differences between scientific fields in publication, authorship,
and citation practices
Different scientific fields have different publication, authorship, and citation
practices. Publications in the life sciences for instance tend to have more authors andtend to receive more citations than publications in the social sciences. In bibliometricanalyses that cover multiple fields, differences between fields should be carefully
accounted for. This can be done by normalizing bibliometric indicators for field
differences or by comparing research units to relevant benchmarks active in thesame field.
5

Find

an

appropriatebalancebetweentransparency

andanalyticalsophistication
Transparency of a bibliometric analysis helps to ensure that the analysis isinterpreted correctly and facilitates a well-informed discussion about the outcomesof the analysis. Analytical sophistication, for instance the use of advanced fieldnormalized bibliometric indicators, has the potential to provide insights that are
hard to obtain using more straightforward bibliometric approaches. However, an
increase in analytical sophistication often causes a decrease in transparency.
Bibliometric analyses therefore require a careful trade-off between transparency andanalytical sophistication.
6 Embrace the value of multidimensional and contextualized bibliometrics
Bibliometrics is sometimes used as a tool for making one-dimensional performancemeasurements. When bibliometrics is used in this way, its value is limited. To takefull advantage of bibliometric information, a multidimensional and contextualized
approach to bibliometrics needs to be adopted. Such an approach recognizes that
research management and research evaluation benefit from being supported bydiverse types of bibliometric information. It also recognizes the necessity ofcontextualizing bibliometric information (e.g., by explicitly linking the information to
the underlying data) in order to enable in-depth interpretation of the information.
7 Use bibliometrics as part of a broader range of information sources
Bibliometrics offers just one source of information to support research management
and research evaluation. There are other sources of information as well. In addition
to quantitative sources, such as data on research funding and research staff andaltmetric data, this also includes peer review by scientific experts. The use of
bibliometrics should be considered within this broader framework. The best way to23

support research management and research evaluation typically is to combine
bibliometric information with other information sources.
8 Anticipate the effects of bibliometric analyses on the science system
The use of bibliometrics in research management and research evaluation is likely to
influence the behavior of researchers and other actors in the science system. These
actors may change their behavior both in intended and in unintended ways. It is
important to anticipate these effects of the use of bibliometrics and to be aware thata strong reliance on bibliometrics may have undesirable consequences (e.g.,
researchers trying to improve their citation statistics in questionable ways).
Leiden Manifesto
The above principles for the responsible use of bibliometrics have partly been
derived from the Leiden Manifesto, an influential statement presenting best practiceguidelines for the proper use of quantitative indicators in research evaluations. The
Leiden Manifesto is available at www.leidenmanifesto.org.
Summary•

Eight principles have been presented that can serve as guidelines for the responsible use of
bibliometrics.

Further reading
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research
metrics. Nature, 520, 429–431.

24