BIBLIOMETRIC REPORT

Bibliometric study of the ETH Domain (2009-2020/2021)
August 22, 2022

Bibliometric study of the ETH Domain (2009-2020/2021)

Report for the ETH Board
Nicolas Leclaire
Häldeliweg 158092 Zurich, Switzerland
E-mailnicolas.leclaire@ethrat.ch
Webpage https://www.ethrat.ch/en

CWTS
Ed Noyons, Clara Calero, Rodrigo Costas, Jeroen van Honk
CWTS B.V.
P.O. Box 905
2300 AX Leiden, The Netherlands
Tel+31 71 527 5806
E-mailinfo@cwts.nl
Webpage http://www.cwtsbv.nl/

General parameters of the bibliometric reportParameters
Database:

VersionClassification system::

Publication window
Citation window
Counting Method:::

Self-citations
Top indicators::

Web of Science (Articles, Reviews and Proceedings
papers in the SCIE, SSCI, AHCI, and CPCI)
2152 (CWTS)
Publication-level classification system (about 4000
fields, referred to as research areas)
2009–2020
Maximum 4 years (and until 2021)
Fractional counting at the level of organisation forcitation impact measurement
ExcludedTop 10%Contents
List of Indicators

5

Definitions, abbreviations and acronyms

7

1 Introduction

8

2 Data collection and methodology
2.1 Summary of method . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Additional indicators . . . . . . . . . . . . . . . . . . . . . . . .

10
10
10
13

3 Results
3.1 Overall output and impact . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 General statistics . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Open Access publishing . . . . . . . . . . . . . . . . . . . . . .
3.2.3 Author gender diversity . . . . . . . . . . . . . . . . . . . . . .
3.2.4 Output in the context of developments worldwide . . . . . . .
3.3 Collaboration and partners . . . . . . . . . . . . . . . . . . . . . . . . .

18
18
23
23
25
27
28
30

Annexes37

A Citation data and analysis
A.1 Database coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Database Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Citation Window, Counting Method and Field Normalisation . . . .

37
37
37
38

B Publication level classification
B.1 Journal scope (including multi-disciplinary journals) . . . . . . . . . .
B.2 Granularity of the WoS subject categories . . . . . . . . . . . . . . . .
B.3 Multiple assignment of journals to categories . . . . . . . . . . . . . .
B.4 The CWTS publication-based classification scheme . . . . . . . . . .

40
40
40
40
41

C Interdisciplinary research

43

4

www.cwtsbv.nl

List of indicators
Avg Reads Average number of reads per DOI. A read is defined by saving a publication in a Mendeley user account.
IntCov Internal coverage. Estimated Web of Science coverage of a set of publications. A description of the calculation is provided in Annex A.1.
IntDisc Measure of interdisciplinary research, defined by the proportion of references in a publication assigned to other fields. Fields are defined by journal
categories. In addition, the cognitive distance of fields to each other is also
considered (more info at Section 2.1 (p. 15) and Annex C).
MCS Mean citation score. The average number of citations received by a publication (TCS/P[full]).
MNCS The mean normalised citation score. This represents average citation scoreper publication, normalised by research area and publication year. Researchareas are defined by a detailed publication classification system of CWTS,
consisting of about 4000 areas. The average MNCS in the entire databaseis 1. Scores higher than 1 reflect a citation-based impact that is higher than
the world average.
MNJS The mean normalised journal score. This represents the normalised average
citation impact of journals. The MNJS is an average score for all publicationsin the same journals in which an institution published. The normalisation isbased on the same principles as the MNCS. The average MNJS in the entire
database is 1. Scores higher than 1 reflect a journal citation impact that ishigher than the world average.
P[full] The number of publications, full counting. Each publication is counted infull (i.e. as 1).
P[fract] The number of publications, fractionally counted. The fraction is determined based on the number of co-authoring organisations.
P[OA] Number of publications, full counting, in Open Access(OA). In addition, we
provide the number for the different kinds of OA: Gold, Hybrid, and Green.
A publication is tagged by one type only. Gold and Hybrid overrule Green.
Information is based on Unpaywall data (July 2021).
PP[OA] The proportion of publications in Gold, Hybrid or Green OA, while publications without a DOI are discarded (OA unknown).
PP[collab] Proportion of publications, full counting, involving collaboration (atleast two institutions co-authoring).
www.cwtsbv.nl

5

PP[int collab] Proportion of publications, full counting, involving international collaboration (co-authorship of organisations from more than one country).
PP[industry] Proportion of publications, full counting, involving industry (co-authorshipwith companies).
PP[uncited] Proportion of publications, full counting, that are not cited.
PP[self cits] The average number of author-self citations per publication. A selfcitation is defined as any of the authors of a cited publication is the same as
any of the authors of the citing publication.
P[top10%] The number of publications, counted in full belonging to the top 10%of their research area. The area is determined on the basis of a detailed
publication classification system of CWTS, consisting of about 4000 areas(See Annex B).
PP[top10%] The proportion of publications (P[fract]) belonging to the top 10% mostcited of their area and in the same year. The areas are determined using
a detailed publication-level classification system , consisting of about 4000
areas. The PP[top10%] in the entire database is 10%. A score above 10%
represents impact that is higher than the world average.
PA[F inst] Share of female authors of an institution within a publication.
PA[F pubs] Share of female authors within a publication (institution plus coauthors).
A[M inst] Number of male authors of an institution.
A[FM inst] Number of authors of an institution for which we could define gendermale or female.
RPA[F] Proportion of female authors compared to the total of authors for which
gender (male or female) was defined (more info at Section 2.1).
TCS The total citation score. This represents the total number of citations accumulated within the citation window, excluding author self-citations.
For more details about the normalised citation indicators, please refer to Waltman
et al. (2012). More information about the mentioned publication-level classification
is in Annex B.

6www.cwtsbv.nl

Definitions, abbreviations and acronyms
CWTS Centre for Science and Technology Studies, Leiden University
A&HCI Arts & Humanities Science Citation Index
SCIE Science Citation Index Expanded
SSCI Social Science Citation Index
CPCI Conference Proceedings Citation Index
DOI Digital Object Identifier (a permanent ID for publications)
JSC Journal Subject Category
OA Open AccessResearch area A set of publications on a certain topic, identified by the Leiden
Algorithm (Annex B)
Subject A set of publications in journals belonging to a (subject) category
WoS Web of Science

www.cwtsbv.nl

7

Introduction

Introduction

1

The ETH Domain consists of two Federal Institutes of Technology, ETH Zurich and
EPFL, and four research institutes PSI, WSL, Empa and Eawag. Together, theyplay a vital role in the Swiss science system for education, research and transfer
of knowledge and technology.
The ETH Board commissions an intermediate evaluation every four years. The mostrecent one took place in 2019. The bibliometric study was executed in 2018. The
evaluation is a moment for the Swiss Federal Council, the ETH Board, as well as
staff and management of ETH Domain to find out where ETH Domain stands visa-vis the ambitions and measures formulated in the strategic planning document.
Moreover, the intermediate evaluation should lead to recommendations relating tothese ambitions and measures.
Bibliometric studies can provide evidence related to ambitions and measures as partof a self-assessment report. Although we consider that meeting the standards ofobjectivity for determining the impact of scientific research is important, we believethat decision-making towards the goal of evaluating the quality of institution’s research ought to be multi-dimensional rather than overwhelmingly quantitative. Bibliometric measures provide objective evidence about production, collaboration andimpact but only for the research that has been published in (international) journals
and proceedings. Therefore, we strongly recommend that quantitative evaluations
are complemented with qualitative information (for example the mission and theresearch goals of a department) and expert assessments.
This report includes the results of a concise bibliometric analysis of the scientificoutput of the ETH Domain, covering the period 2009-2020, with citations up to
2021. The studies are based on a quantitative analysis of scientific publications injournals and proceedings processed for the Web of Science (WoS) versions of the
Science Citation Index and associated citation indices: the Science Citation Index(SCI), the Social Science Citation Index (SSCI), the Arts & Humanities Citation
Index (A&HCI) and the Conference Proceedings Citation Index (CPCI). For the firsttime conference proceedings (as far covered by the WoS database) are included in
the study.
Although most of the methodology is similar to the study performed four years ago
for ETH Domain, the results may sometimes differ substantially, due to the factthat in the current report conference proceedings papers are included and fully
integrated, but that depends on the role conferences play for an institution if thisis actually the case. Moreover, new indicators were introduced: RPA[F], IntDisc,P[OA], PP[OA], and Avg Reads.
We introduce each result in brief, while more detailed information about data andmethod is provided in Section 2 and Annex A) of this report. In Section 3 the resultsof our analysis and interpretations are reported.
8

www.cwtsbv.nl

Introduction

In Section 3 the results of our analysis and interpretations are reported. These
results are discussed in 3 parts:
1. Section 3.1: Overall output and impact
2. Section 3.2: Trends
3. Section 3.3: Collaboration and partners
In the annexes, we provide more detailed scores for some indicators, more detailed information about specific approaches, as well as information about CWTS
infrastructural elements involved in the analyses.

www.cwtsbv.nl

9

Data collection and methodology

2

Data collection and methodology
For this report, we used the publication data provided by the individual ETH Domain institutions. Publications were deduplicated at the level of the ETH Domain.
ETH Domain institutions provided us with a list of publications from their repositories. The publications were matched with the data in the CWTS version of the
Web of Science (WoS) database.

2.1 Summary of method

In this section, we discuss the methods underlying the bibliometric analysis developed. We discuss the basic principles of our indicators and the context in which
they can (or should not) be used. Additional and more detailed information about
methods and data can be found in the annexes.

2.1.1

Indicators

In bibliometric analyses regarding research performance, we usually discern two
types of indicators: size-dependent and size-independent, taking into account thedifferent size of institutions under investigation. Larger institutions, for instance, will
be involved in more publications than smaller ones. Subsequently, this will affectthe absolute number of top 10% publications, as well as all other size-dependent
indicators. In Figure 1 we visualise the correlation between the two indicators forthe 6 ETH institutions.
10

www.cwtsbv.nl

Data collection and methodology

16,
000
15,
000
14,
000
13,
000
12,
000
11,
000

P[
t
op10%]

10,
000
9,
000
8,
000
7,
000
6,
000
5,
000
4,
000
3,
000
2,
000
1,
000
0
0

5,
000

10,
000

15,
000

20,
000

25,
000

30,
000

35,
000

40,
000

45,
000

50,
000

55,
000

60,
000

65,
000

70,
000

75,
000

P[
f
ul
l
]

Figure 1: P[full]vs.P[top10%]for 6 ETH institutions

Proportion indicators (e.g., PP[collab], PP[int collab], PP[industry], PP[OA], PP[top10%])and average indicators (MNCS, MNJS) are size-independent, while others used inthis study (e.g., P[full], P[fract], TCS) are size-dependent. In the report we will
primarily discuss the results using the size-independent indicators to account forthe size differences of the organisations. Moreover, the results for size-independent
indicators can, in most cases, be related to the world average.
Output indicators
Size-dependent
The total number of publications in which researchers from an institution wereinvolved (P[full]) is the basic output measure. In addition, we provide the indicator P[fract] which assesses an institution’s contribution to the output P[full]. Eachindividual publication is divided by the number of organisations co-authoring, regardless of the number of organisations involved. If authors have two affiliations
and mention both, both affiliations are counted as fractions. P[fract] is the sum ofthese fractions of publications in which an institution was involved.
Size-independent
Proportion indicators characterise sets of publications regardless of the number andare therefore size-independent. They are often used to characterise output. For
www.cwtsbv.nl

11

Data collection and methodology
instance, PP[collab] indicates the proportion of output with at least two different
organisations involved. PP[int collab] indicates the proportion of output involving
international collaboration. In this report, a publication is tagged as an international
collaboration if at least one of the co-authoring organisations is based outside of
Switzerland. Furthermore, two other proportion indicators are used: PP[industry],
representing the proportion of P[full] co-authored with a company and PP[OA], theproportion of P[full] published in Open Access (OA).
For OA publications, we discern different types: OA Gold, OA Hybrid and OA
Green. The definition of the types used in this report are:• Gold: The publisher makes all articles and related content available for free
immediately on the journal’s website.
• Hybrid: Publication freely available under an open license in a paid-accessjournal.
• Green: Published in toll-access journals, self-archived by authors (in repositories or researchers’ websites), independently from publication by a publisher.
OA publications are counted only as one type at the same time. If a paper is bothGreen and Gold, it is counted as Gold. Bronze OA publications are free to read
only on the publisher page without a license. As such, they were disregarded as
OA. These were identified as Closed Access publications.
Impact indicators
Size-dependent
The scientific impact of an institution’s output is measured by citations. We provide
the total number of citations received (TCS) in the period of maximum 4 years after
publication, up to 2021. For more recent years the citation window is shorter than
4 years. We exclude author self-citations. Another size-dependent indicator of
impact is P[top10%], i.e. the absolute number of publications belonging to the top
10% most cited publications (in their area and from the same year).
It should be noted that all citation-based indicators (including TCS) are calculatedusing a limited and fixed time-window. The total amount of citations for early
publications may therefore be higher than processed for this report.
Size-independent
The MNCS is the indicator to measure citation impact after normalising by researcharea and publication year. The research area to which a publication belongs isdefined by a publication-level classification (for details, see Annex B). In this classification each publication is uniquely assigned to a research area. Areas are defined12

www.cwtsbv.nl

Data collection and methodology

by their citation environment (cited and citing publications). This classification ismore fine-grained and is considered more accurate than a journal classification(Ruiz-Castillo and Waltman, 2015). In a journal classification all publications fromone journal are in the same class. Similar journals are in the same class andjournals may belong to more than one class. We use this journal classification
to characterise an institution’s output in its research profiles but not to normalise
impact. The journal classification is less fine-grained and as such easier to relate
to the main subjects addressed.
In addition, we provide the proportion of publications in the top 10% most cited publications (within their research area, i.e. class, and in the same year, PP[top10%]).
This indicator correlates strongly with the MNCS but is not sensitive to outliers.
The MNCS can sometimes be biased by one paper being cited many times. The
PP[top10%] is not influenced by this one paper, as it is ‘just’ one of the top 10%
or not. An MNCS that is relatively much higher than the PP[top10%] points toa highly skewed distribution of impact across publications. In other words, a few
publications receive a huge number of citations, compared to the other publications.
Finally, we also use an indicator measuring the impact of journals, the Mean
Normalised Journal Score (MNJS). This indicator assesses the impact in term ofcitations of the journals (aggregated), in which the institution has published, using
the same normalisation as we use for measuring the impact (MNCS). As such, the
MNJS does not measure the (average) impact of an institution’s publications, butrather the impact of the journals in which its researchers publish.
2.1.2

Additional indicators

In this study we introduce indicators that relate to the context of the publishedresearch. We will discuss them in brief in the next subsections.
Worldwide growth of research fields
An indicator to position an institution’s research activities in the context of what
happens at a larger scale is the [Field growth]. We use the science landscape (seeAnnex B) to reflect what happens worldwide, by calculating a growth indicator foreach area (the [Area Growth]).
The [Field growth] relates the output of an institution to these area growth values([Area Growth]) as follows. First, we calculate for each of the 4000 research areasin the science landscape, the share output of the most recent two years (20192020) as compared to the total in 2009-2020 (the period under study). This shareof output in the most recent years is normalised by a reference value, which is theresult of the number of recent years (2) and the number of years of the total period
considered (12): 0.17. Areas in which the share of output in the recent years iswww.cwtsbv.nl

13

Data collection and methodology

higher than 0.17, have a [Area Growth] above 1, a positive growth.
Any value above 1 means a positive growth, while values below 1 indicate a negative
growth. In Figure 2, we plotted the [Area Growth] in the landscape of all science,by color-coding. Green areas show a positive growth (>1) in the most recent
two years, while red areas show a negative growth (<1). The size of a circle
proportionally reflects the number of ETH Domain publications published in 20092020 worldwide, ranging from 1 up to 1,400.

Rel
at
i
veAr
ea.
.
0.
00

2.
00

Figure 2: Landscape of all science, color-coded by [Area Growth][Field growth]
We use the [Area Growth] to characterise the fields in which ETH Domain researchers are active. Thus we contribute to the answer to the question: is ETH
Domain’s research positioned in fields with an increasing interest worldwide ornot?
The [Field growth] is the average of [Area Growth] values of the areas in which
an institution’s publications can be found. Consider the output of an institution
X, with 100 publications. These 100 publications may be in 20 different areas.
Depending on the [Area Growth] values of these areas, these 100 publications
relate to 20 different [Area Growth] scores. The average [Area Growth] values of
the 100 publications, then indicates the estimated growth of fields in which X is
active: the [Field growth] of institution X.
14

www.cwtsbv.nl

Data collection and methodology
Interdisciplinary research
We introduce a measure related to the interdisciplinary character of the publishedresearch. Being more or less interdisciplinary is defined by the knowledge base(the prior art that is being cited) of the published research. The content of cited
publications is defined by the journal subject categories.
If a publication cites research from one (and most likely its own) subject category
only, it is defined as mono-disciplinary (measure close to 0). If a publication cites
research from different subjects, we consider it as interdisciplinary. If the subjects
are cognitively at a long distance from each other, the measure of interdisciplinarity
is even higher, with a maximum of 1.
The cognitive distance between subject categories is determined by the density of
the citation traffic between them. If a publication (A) cites output in subject X and Y,
while X and Y are remote from each other (little citation traffic between them), it isconsidered more interdisciplinary than publication B, which cites publications fromY and Z, which are cognitively closely related (i.e., in subject categories frequently
citing each other).
For each publication we calculate an interdisciplinary value and for sets of publications we then calculate their average (IntDisc), which is a value between 0 and
1, where 0 indicates mono-disciplinary and 1 means maximum interdisciplinarity.
In summary, interdisciplinarity is:
1. Defined by cited references in a publication;
2. On the basis of the variety of journal categories of cited publications;
3. Considering cognitive distance between these categories;
4. While this distance between categories is based on mutual citation traffic.
The above leads to the definition of interdisciplinarity we use in this report:
The interdisciplinarity indicator (IntDisc) relates to the diversity of
research supporting the current research.
In order to be able to interpret the IntDisc measure in a broader context, we
calculated a reference value (Ref Intdisc), which is the IntDisc for the journal
category at large in 2020. In this way interdisciplinarity can be assessed withineach journal subject category by relating it to the world average. We integrated
both scores (IntDisc and Ref Intdisc) in profiles, where interdisciplinarity is included.
More info can be found in Annex C.
www.cwtsbv.nl

15

Data collection and methodology
Share of female authors
We also introduce an indicator related to gender diversity of research staff. We
calculated the probability of an author name to be male or female, by looking at
the first name. If first names (or nicknames) point to a gender within a specificcountry, the gender is set using the following four-step procedure (also described
at CWTS Leiden Ranking):
1. Author disambiguation. Using an author disambiguation algorithm developed
by CWTS (Caron and van Eck, 2014), authorships are linked to authors. If
there is sufficient evidence to assume that different publications have beenauthored by the same individual, the algorithm links the corresponding authorships to the same author.
2. Author-country linking. Each author is linked to one or more countries.
If the country of the author’s first publication is the same as the country
occurring most often in the author’s publications, the author is linked to thiscountry. Otherwise, the author is linked to all countries occurring in his or
her publications.
3. Retrieval of gender statistics. For each author, gender statistics are collectedfrom three sources: Gender API, Genderize.io , and Gender Guesser. Gender
statistics are obtained based on the first name of an author and the countries
to which the author is linked.
4. Gender assignment. For each author, a gender (male or female) is assigned
if Gender API is able to determine the gender with a reported accuracy ofat least 90%. If Gender API does not recognize the first name of an author,
Gender Guesser and Genderize.io are used. If none of these sources are able
to determine the gender of an author with sufficient accuracy, the genderis considered unknown. For authors from Russia and a number of othercountries, the last name is also used to determine the gender of the author.
Using the above procedure, the gender can be determined for about 70% of all
authorships of major universities. For the remaining authorships, the genderis unknown.
For each publication, we counted the number of female authors at the level of theinstitution (A[F inst]) as well as at the level of the entire publication (A[F pubs]).
In addition we counted those for male authors. We disregarded authors for which
the gender cannot be defined or is ambiguous. The total amount of authors which
we defined female or male is indicated by A[FM inst] and A[FM pubs].
Hence, for each publication in which ETH Domain authors were involved, there isa share of female ETH Domain authors (PA[F inst]), and a share of female authors
for the publication at large (PA[F pubs]). The latter is used as a benchmark for16

www.cwtsbv.nl

Data collection and methodology
the former. RPA[F] indicates the ETH Domain share, normalised by the shareof the benchmark. A value higher than 1 for an institution X, indicates a higher
proportion of female authors at X than for its community at large (X plus coauthoring partners).

www.cwtsbv.nl

17

Results

3

Results
In this chapter we discuss the performance of the ETH Domain over the entire
period 2009-2020 and in a trend analysis of overlapping 4 year blocks. We discuss
the output and impact and collaboration, as well as some indicators relating to
the context in which the research is executed, such as gender diversity and Open
Access publishing.

3.1 Overall output and impact
Main findings

ETH Domain researchers were involved in 136,535 WoS publications,which is estimated at 79% of the total scientific output. Almost 60%
is published in Open Access. Almost 80% of the publications is coauthored with other organisations, while 67% involves international
collaboration. 9% is co-authored with the private sector.
The impact of ETH Domain output is well above the world average,
64% by MNCS and almost twice the world average by PP[top10%].
The share of female authors at the ETH Domain is 9% higher than thebenchmark (the co-authoring partners).
The ETH Domain research has a broad variety and covers basicallythe entire landscape of science. The research foci of the six individual
institutions show some mutual overlap but merely points to complementarity.
In this section, we discuss the overall performance of the ETH Domain in the period
2009 up to 2020. It should be noted that these results are often heavily biased
towards the larger institutions (ETH Zurich and EPFL), especially for the size
dependent indicators.
Nevertheless, the results should provide a proper general overview of the bibliometric performance of the ETH Domain at large. This section contains the overall
statistics as well as a positioning of ETH Domain research in the landscape of allscience. By providing such positioning of all six institutions next to each other, we
visualise their overlap and complementarity.
It should be noted that the provided overview (covering the entire period 2009-2020)
allows only little opportunity for interpretation or contextualisation. We discuss ananalysis over time of the indicators reported in this section, in the next section (seesection 3.2). Thus, we provide better insight in the (intended or not) developments.
In Table 1, we list the ETH Domain scores for 5 types of indicators.
18

www.cwtsbv.nl

Results

Table 1: Overall bibliometric performance statistics ETH Domain
Indicator

Score
OutputP[full]P[fract]
Int Cov
InterDiscP OA [Gold, Hybrid, Green]
PP [OA]

136,535
64,0520.79
0.35
72,00759%

Collaboration
PP[collab]
PP[industry]
PP[int collab]

79%
9%
67%
Citedness

TCS
MCS
P[top10%]
PP[top10%]
MNCS
MNJS
PP[self cits]
PP[uncited]

1,633,206
11.96
26,973
19%
1.64
1.48
26%
16%
Author genderA[F inst]
A[FM inst]
PA[F inst]
PA[F pubs]
RPA[F]*

50,324
251,339
0.20
0.18
1.09
Readership

N reads
N pubs read
Avg Reads*

350,985
68,966
5.09

RPA[F] may differ from the ratio PA[F inst] to PA[F pubs] due to rounding.

OutputResearchers of the ETH Domain were involved in 136,535 WoS publications from2009 up to 2020 (P[full]). Normalised by the number of co-authoring institutions,the output adds up to 64,052 (P[fract]). We estimate that almost 80% of the output
is covered by WoS (IntCov: 0.79%). The interdisciplinarity (IntDisc) of research atthe ETH Domain is estimated at 0.35. 60% of the output (P[OA]: 72,007) in the
www.cwtsbv.nl

19

Results

entire period of 12 years was published in OA.
Collaboration
We found for the ETH Domain that almost 80% of ETH Domain’s publications
involved collaboration (i.e. co-authored by more than one institutions). 67% involved
international collaboration (PP[int collab]). Furthermore, 9% of the output involveda private partner (PP[industry]).
Citedness
Publications by ETH Domain researchers were cited more than 1,6 million times(TCS), which means almost 12 times on average (MCS). The impact of ETH Domain
publications, when normalised by research area and year, reaches an MNCS of 1.64,
which means 64% above the world average of 1.
An important contribution to this high impact is due to the large number of publications belonging to the 10% most cited worldwide (P[top10%]: 26,973). The
proportion of ETH Domain output belonging to the top 10% most cited publications(PP[top10%]) is 19%, which means almost twice the worldwide average value of
10%. We calculated 16% of the output not being cited. Finally, we found 26% of
the citations to be author self-citations (PP[self cits]), which were not considered
for the impact measurement.
Author gender
We found that 50,324 of the 251,339 of the ETH Domain author names (A[FMinst]) are female, which represents 20% (PA[F inst]: 0.20). This share is 18% (PA[Fpubs]: 0.18) for all co-authors of the publications in which ETH Domain researchers
were involved (the benchmark). The ratio (RPA[F]: 1.09) indicates a slightly higher
involvement of female authors at the ETH Domain as compared to the benchmark.
Readership
We counted 68,966 ETH Domain publications to be saved by 350,985 Mendeleyusers, which is 5.09 on average per publication, stored in Mendeley (Avg Reads).

20

www.cwtsbv.nl

Results
ETH Domain research focus
To provide more (general) insight in the foci of ETH Domain’s research, we plotted
the output across the landscape of science (Figure 3).

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSciFigure 3: Distribution of ETH Domain’s output across landscape of science (interactive version via this link)

The landscape in Figure 3 is a two-dimensional representation of all science (covered by WoS) with an overlay of the output of the six institution of the ETH Domain
together in the different research areas. In Annex B we provide a more detailed
description of the landscape and the way it is created. The size of a circle reflects the relative number of publications in which ETH Domain researchers wereinvolved. The colors in the landscape point to 5 main disciplines we use to support
the interpretation of the landscape. The map of the landscape shows a broad
distribution of ETH Domain’s research.
In addition to that, we plotted the output across the landscape of the 6 ETH Domain
institutions (Table 2). These landscapes underlay the ETH Domain landscape andshow partly the complementarity and overlap of research foci of the six institutes.
ETH Zurich and EPFL cover the entire map, being the most general universities.
WSL and Eawag show primary interest in life and earth sciences, while Empa and
PSI focus primarily on physical sciences and engineering. These landscapes canbe explored interactively via this interface. Open the menu on the left to change
the perspective to an institution.
www.cwtsbv.nl

21

Results

Table 2: Landscapes of all ETH Domain institutions (open in browser)
ETH Zurich

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSc
i

EPFL

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSc
i

PSI

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSc
i

WSL

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSc
i

Empa

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSc
i

22

Eawag

Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSciwww.cwtsbv.nl

Results

3.2 Trends
Main findings

ETH Domain researchers were involved in an increasing number ofpublications, although it seems to saturate somewhat in the most recent period, which relates to the delayed processing and uptake ofproceedings in WoS. We found an increasing number and share of
OA publications, publications, involving international collaboration andpublications co-authored with industry. The impact if the output remains at a high level throughout. Another striking increase is the shareof female authors in ETH Domain publications. Finally, we measured
an increasing involvement of ETH Domain researchers in growing aeas.

3.2.1

General statistics

In this section we discuss the trend for key indicators related to the performanceof the ETH Domain at large. By looking at trends we provide a sense of how theETH Domain research has developed between 2009 and 2020.
Note that many of the indicators and trends depicted in Table 1 are biased towards
the bigger institutions ETH Zurich and EPFL. In other words, these institutions
disproportionately influence the trends observed for the ETH Domain at large,
especially for size-dependent indicators.
Table 3 shows a steady increase of output (P[full]) in which ETH Domain researchers were involved (from 37,017 up to 52,000, more than 40% increase). Thisis also the case for the contribution of the ETH Domain (P[fract]), but the increase isrelatively less (from 19,630 up to 21,909, 15% increase). This means that researchersof the ETH Domain were involved in larger teams.
Another important trend we can see in these results is the increase of the numberof Open Access publications (P[OA]). Relative to the total output in which ETH
Domain researchers were involved (PP[OA]), we see a significant positive trend as
well, from 49% up to 68%.
www.cwtsbv.nl

23

Results

2011-20142012-20152013-20162014-20172015-20182016-20192017-2020

P[full]P[fract]
InterDiscP [OA]PP [OA]
PP[collab]
PP[int collab]
PP[industry]P[top10%]
PP[top10%]
MNCS
PA[F inst]
RPA[F]

2010-2013

Indicator

2009-2012

Table 3: Trends of ETH Domain’s bibliometric performance

37,017
19,6300.33
15,671
49%
72%
60%
8%
7,295
19%
1.67
0.18
1.04

39,352
20,4080.33
17,363
51%
73%
61%
8%
7,868
19%
1.68
0.19
1.10

42,021
21,2790.34
19,176
52%
75%
62%
8%
8,456
19%
1.67
0.19
1.12

44,875
22,1270.34
21,344
54%
76%
64%
8%
9,129
19%
1.69
0.20
1.12

47,51822,5140.35
23,810
57%
78%
66%
9%
9,528
19%
1.66
0.20
1.11

49,535
22,6380.35
26,230
59%
80%
68%
10%
9,785
19%
1.64
0.20
1.11

51,092
22,6640.36
28,677
62%
81%
70%
10%
10,142
19%
1.64
0.21
1.11

52,37722,6050.37
31,101
65%
83%
71%
10%
10,224
18%
1.61
0.21
1.11

52,00021,9090.37
32,526
68%
83%
72%
10%
10,150
18%
1.59
0.22
1.12

The share of output in which ETH Domain researchers collaborated with researchers from other institutions (PP[collab]) increased with 11 percent points (from72% up to 83%), while the share of co-authored publications with foreign partners(PP[int collab]) increased from 60% up to 72%. The share of output in which industry
was involved (PP[industry]) increased as well (8% up to 10%). These figures point
to an increased international integration together with an increased involvement ofindustry with ETH Domain’s research.
The measure of interdisciplinarity (IntDisc) increased somewhat over time from 0.33
up to 0.37. It is difficult to say whether this is significant, as research is becoming
more interdisciplinary in general. We interpret this trend as an indication that
research performed at the ETH Domain is increasingly on a broader, more diverse
basis.
Regarding the impact we found that on the one hand, the number of publications
that are in the Top 10% cited worldwide (P[top10%]) increased along with the total
output. Hence, the proportion of publications in the top 10% (PP[top10%]) remains
stable over the time period observed. On the other hand the MNCS score slightly
decreases.
The fact MNCS and PP[top10%] don’t show the same trend may be explained by a
few very highly cited publications in the first years of our analysis. These scores
influence the MNCS score but not so much the PP[top10%]. The latter indicatesthat research at the ETH Domain has a high impact throughout (above 18%).
24

www.cwtsbv.nl

Results
3.2.2

Open Access publishing

Open Access publishing is a major element in the context of executing open science.
As shown in Table 3, the number and share of Open Access (OA) publications
increases significantly during the period we studied. In this section we will lookat this in more detail. First of all, we look at the impact indicators of OA and nonOA publications. In Table 4, we present four indicators by type (Open or Closed
Access): P[full], P[top10%], PP[top10%] and PP[int collab].
P[full] for Closed Access publications drops from 2017 onwards, while the number of
OA publications doubles during the period we studied. In Figure 4 the increase of
all three OA types is visualised. Particularly, Gold and Hybrid publications show
a strong increase. Green OA publications may be at a saturation point.

2009-20122010-20132011-20142012-20152013-20162014-20172015-20182016-20192017-2020

Table 4: ETH Domain’s performance statistics trend, Closed vs. Open Accesspublications

ClosedP[full]P[top10%]
PP[top10%]
PP[int collab]

16,615
3,074
18%
57%

16,986
3,155
18%
58%

17,454
3,214
18%
60%

17,919
3,235
18%
60%

18,021
3,151
17%
62%

17,943
3,018
17%
64%

17,471
2,832
16%
65%

16,562
2,633
16%
67%

15,623
2,489
16%
68%

P[full]P[top10%]
Open
PP[top10%]
PP[int collab]

15,671
3,442
20%
68%

17,363
3,837
20%
69%

19,176
4,290
21%
70%

21,344
4,866
21%
71%

23,810
5,382
21%
73%

26,230
5,854
21%
74%

28,677
6,444
21%
75%

31,101
6,801
21%
76%

32,526
6,992
20%
76%

Indicator

We see a higher impact for OA publications (PP[top10%]) during the entire period.
The impact of Closed Access publications (PP[top10%]) drops somewhat, together
with the absolute number of top 10% publications in Closed Access (P[top10%]). We
also included the PP[int collab] in this table, because international co-authored papers contribute a lot to scientific (citation-based) impact. For both open and Closed
Access, the share of publications involving international collaboration (PP[int collab]) increases. The share is always higher for OA publications.
www.cwtsbv.nl

25

Results
Per
i
od
18,
000

16,
000

14,
000

P[
f
ul
l
]

12,
000

10,
000

8,
000

6,
000

4,
000

2,
000
0
20092012

20102013

20112014

20122015

20132016

20142017

20152018

20162019

20172020

OAt
ype
Cl
osed
OAGol
d
OAGr
een
OAHybrid
OAunknown

Figure 4: ETH Domain’s output trend by Open Access (OA) type26

www.cwtsbv.nl

Results
3.2.3

Author gender diversity

In this section we look at gender diversity in publications by calculating the shareof female authors.
We estimated that 20% of the ETH Domain authors are female (PA[F inst]: 0.20).
In comparison, women amount to 18% of all authors listed on the output publishedby the ETH Domain (PA[F pubs]: 0.18, i.e. authors from the ETH Domain andco-authors from other institutions altogether). This means that the share of female
authors participating to publications is higher within the ETH Domain as compared
to co-authoring institutions (RPA[F]: 1.09).
In Figure 5, we depict the trend of female authors (PA[F inst], blue line) and theshare of female authors at ETH Domain compared to all co-authoring institutions(RPA[F]) (red line) over time. Looking at these results, we see a steady increaseof the share of female ETH Domain authors and a stable 9% above the benchmark
from 2011 onwards.
Per
i
od
0.
22
1.
10
0.
20

1.
00

0.
18

0.
90

0.
16

0.
80

0.
14

0.
70

0.
12

0.
60

0.
10

RPA[
F]

PA[
Fi
nst
]

1.
00

0.
50

0.
08

0.
40

0.
06

0.
30

0.
04

0.
20

0.
02

0.
10

0.
00

0.
00
20092012

20102013

20112014

20122015

20132016

20142017

20152018

20162019

20172020

PA[
Fi
nst
]
RPA[
F]

Figure 5: Share of female ETH Domain authors (PA[F inst]), and share of female
authors compared to benchmark (RPA[F])www.cwtsbv.nl

27

Results
3.2.4

Output in the context of developments worldwide

The final part of results in these trend sections relate to the growth of fields inwhich researchers at the ETH Domain are active. For this, we combine the activityof the ETH Domain as distributed on the landscape of science, and the growth of
the areas in that landscape worldwide.

Fi
el
dgr
owt
h
0.
00

2.
00

Figure 6: Positioning of ETH Domain research in landscape of all science, colorcoded by [Area Growth]

In Figure 6, we depict the distribution of ETH Domain research output (similar to
Figure 3) and color coded-each areas by the estimated growth worldwide ([AreaGrowth]). This map positions ETH Domain’s activity and relates it to the developments worldwide. We can see that the vast majority of areas in which ETH
Domain researchers publish, is growing (green). Besides that, there are regionsin the landscape with substantial ETH Domain output, and stable (grey) activity
worldwide or somewhat negative growth in volume (red). In these areas knowledge
production has saturated. Worldwide the attention has shifted towards other areas.
Subsequently, we processed this information in a trend analysis, in which we linked
publications per 4 years period to the recent growth factor of the area to which
they belong ([Area Growth]). The results are plotted in Figure 7. The blue line
plots the number of publications per 4-years period (P[full]), while the red linereflects the estimated volume growth of the areas to which the publications in eachperiod belong ([Field growth]). The latter point out that ETH Domain researchers
28

www.cwtsbv.nl

Results

published in growing research areas. Moreover, we found that the number of ETH
Domain publications not only increases over the years (P[full], blue line), but alsothat the research is increasingly published in growing areas (red line).
Per
i
od

55,
000

1.
30
1.
20

50,
000

1.
10

45,
000
1.
00

1.
00

40,
000
0.
90
0.
80

30,
000

0.
70

25,
000

0.
60

Fi
el
dgr
owt
h

P[
f
ul
l
]

35,
000

0.
50

20,
000

0.
40
15,
000
0.
30
10,
000

0.
20

5,
000

0.
10

0

0.
00
20092012 20102013 20112014 20122015 20132016 20142017 20152018 20162019 20172020

MeasureNames
Fi
el
dgr
owt
h
P[
f
ul
l
]

Figure 7: ETH Domain number of publications over time and estimated volume
growth of subjects in which ETH Domain researchers are active

www.cwtsbv.nl

29

Results

3.3 Collaboration and partners
Main findings

To assess collaborative work of ETH Domain researchers, we look atcollaborative output within the ETH Domain and output outside theETH Domain. Within the ETH Domain, we found large differences in
number of co-publications, due to the different sizes. The impact is thehighest for publications involving ETH Zurich and EPFL. Co-authorshipanalysis outside the ETH Domain shows a prominent position of other
Swiss and German institutions. In terms of impact, co-authorship with
US institutions stand out, besides a set of European institutions.

Regarding collaboration, we consider co-authorship and impact within and outside
the ETH Domain. Output numbers (co-publications) are size-dependent, while
impact measures (MNCS) are size independent. We discuss them as such.

Collaboration within the ETH Domain

The scores in Table 5 show the co-authorship (output and impact) between the institutions of the ETH Domain. The absolute numbers of output are clearly dominatedby ETH Zurich and EPFL, but that does not mean that the number of co-authoredpublications between these two are the highest. In Table 2, we visualised a similar
research focus of ETH Zurich and EPFL. In general, we see there is collaboration among all members, with only a few pairs with less than 100 co-publications:
Eawag with PSI and WSL with Empa, which can be explained by the different profiles. Another less productive co-authorship between is found (in absolute number)
between WSL and Eawag, which can be explained by the lower output overall by
these institutions.
It should also be noted that the highest impact is found for the publications in which
ETH Zurich and EPFL collaborate (MNCS: 2.02). We also found high impact forco-authored output by ETH Zurich and WSL (MNCS: 1.76).
30

www.cwtsbv.nl

Results

Table 5: Co-authorship and impact within the ETH Domain
Unit

Indicator

ETH Zurich

EPFL

PSI

WSL

Empa

Eawag

ETH ZurichP[full]
MNCS

74,190
1.71

1,894
2.02

4,294
1.54

1,107
1.76

2,264
1.57

1,832
1.54

EPFL

P[full]
MNCS

1,894
2.02

45,073
1.63

1,279
1.45

390
1.46

591
1.42

528
1.58

PSI

P[full]
MNCS

4,294
1.54

1,279
1.45

14,191
1.34

125
1.47

512
1.64

27
1.50

WSL

P[full]
MNCS

1,107
1.76

390
1.46

125
1.47

4,936
1.42

20
1.06

65
1.60

Empa

P[full]
MNCS

2,264
1.57

591
1.42

512
1.64

20
1.06

7,575
1.44

121
1.54

Eawag

P[full]
MNCS

1,832
1.54

528
1.58

27
1.50

65
1.60

121
1.54

4,497
1.62

Eawag

Empa

ETH Zurich

EPFL
WSL

VOSviewer

PSI

Figure 8: Co-authorship network of ETH Domain’s institutions (line width reflects
number of co-publications, node size reflects total output)

In Figure 8 we visualise the co-author network of the six ETH Domain institutions,taking all connections into account. This network shows the central position of ETH
Zurich and EPFL, with the other institutions taking their own position around them.
As all connections are considered, the graph positions ETH Zurich and EPFL close
to each other in the center.
www.cwtsbv.nl

31

Results
Collaboration outside the ETH Domain
The results in Table 6 and 7 show the 40 most prominent partners of the ETH
Domain at large and distributed by institution, in terms of number of co-publications.
In these results ETH Domain internal collaborations were not considered.
Table 6: Top 40 collaborating ETH Domain’s institutions, outside the ETH Domain
only (fractional output and impact)
Inst
Univ Zurich
Max Planck Soc Advance Sci
Univ Lausanne
Univ Bern
Univ Geneva
Ctr Natl Rechr Sci
Univ Basel
Chinese Academy of Sciences
Massachusetts Inst Technol
Univ California - Berkeley
Tech Univ Munich
Karlsruhe Inst Technol
Harvard Univ
Univ Oxford
Univ Cambridge
Russian Academy of Science
Ist Nazl Fis Nuclr
Stanford Univ
Katholieke Univ Leuven
Politec Milano
Univ Bologna
California Inst Technol
Spanish Natl Res Cncl (CSIC)
CERN Europe Org Nuclr Res
Cons Nazl Ricrc
Delft Univ Technol
Univ Fribourg
Tech Univ Denmark
Univ Freiburg
Agroscope
Princeton Univ
Ludwig-Maximilians Univ München
Univ Padova
Universidade Lisboa
Tech Univ Dresden
Heidelberg Univ
Univ Tokyo
Univ Neuchâtel
RWTH Aachen Univ
Imperial Coll London

32

CountryCo-pubs

MNCS

CH
DE
CH
CH
CH
FR
CH
CN
US
US
DE
DE
US
GB
GB
RU
IT
US
BE
IT
IT
US
ES
CH
IT
NL
CH
DK
DE
CH
US
DE
IT
PT
DE
DE
JP
CH
DE
GB

3,129
1,138
992
898
825
685
637
446
441
379
373
372
341
323
317
311
299
298
296
291
283
280
264
256
254
251
237
231
226
218
208
207
196
194
187
185
182
182
182
182

1.61
2.08
1.57
1.63
1.48
1.82
1.69
1.96
2.29
2.44
1.81
1.86
2.17
2.32
2.02
1.39
1.54
2.48
2.43
1.47
1.62
2.18
1.79
1.53
1.34
2.15
1.52
1.94
1.93
1.36
2.76
1.92
1.66
1.53
1.92
1.77
1.80
1.55
1.87
2.25

www.cwtsbv.nl

Results

The list shows the prominent position of the University of Zurich and the Max
Planck Society as well as three other Swiss institutions with more than 800 copublications. Publications with almost all top 40 partners achieve a high impact
by MNCS between 1.50 and almost 2.50.
Looking at the distribution across ETH institutions (Table 7), we see that the mostproductive partnerships are primarily by ETH Zurich, EPFL and PSI. This relates
obviously to the size of these institutions. With this list we provide an overview of
key partners with the ETH Domain, and refer to the institutions’ reports for moredetail.

www.cwtsbv.nl

33

Results

Table 7: Number of co-authored publications (fractional counting) with top 40
collaborators by ETH Domain institution
Institution

CountryETH Zurich

EPFL

PSI

WSL

Empa

Eawag

Total

Univ Zurich
Max Planck Soc Advance
Sci
Univ Lausanne
Univ Bern
Univ Geneva
Ctr Natl Rechr Sci
Univ Basel
Chinese Academy of Sciences
Massachusetts Inst Technol
Univ California - Berkeley
Tech Univ Munich
Karlsruhe Inst Technol
Harvard Univ
Univ Oxford
Univ Cambridge
Russian Academy of Science
Ist Nazl Fis Nuclr
Stanford Univ
Katholieke Univ Leuven
Politec Milano
Univ Bologna
California Inst Technol
Spanish Natl Res Cncl
(CSIC)CERN Europe Org Nuclr
Res
Cons Nazl Ricrc
Delft Univ Technol
Univ Fribourg
Tech Univ Denmark
Univ Freiburg
Agroscope
Princeton UnivLM Univ München
Univ Padova
Universidade Lisboa
Tech Univ Dresden
Heidelberg Univ
Univ Tokyo
Univ Neuchâtel
RWTH Aachen Univ
Imperial Coll London

CH
DE

2,750
572

137
420

177
135

93
11

101
56

75
20

3,129
1,138

CH
CH
CH
FR
CH
CN

201
470
233
308
408
143

757
130
532
282
92
120

19
166
56
85
82
123

45
76
9
25
25
56

3
57
16
28
53
14

21
122
31
14
31
19

992
898
825
685
637
446

US
US
DE
DE
US
GB
GB
RU

273
189
183
173
191
205
170
119

155
150
96
96
139
81
104
142

17
35
70
94
14
39
24
61

4
12
28
10
3
4
16
10

10
7
15
18
4
6
14
4

4
10
8
8
4
5
3
12

441
379
373
372
341
323
317
311

IT
US
BE
IT
IT
US
ES

137
179
220
134
201
213
138

116
109
56
131
78
58
54

128
13
15
28
9
13
39

0
1
2
1
2
3
15

1
2
37
13
5
3
23

0
6
9
2
0
3
16

299
298
296
291
283
280
264

CH

57

163

52

1

1

0

256

IT
NL
CH
DK
DE
CH
US
DE
IT
PT
DE
DE
JP
CH
DE
GB

84
105
100
119
123
183
121
155
75
57
108
131
102
69
116
108

119
105
78
56
41
13
72
34
95
117
25
43
63
89
45
54

38
18
48
42
19
1
20
15
13
6
44
13
28
2
25
14

10
1
14
4
40
23
3
5
14
8
7
4
0
31
0
1

17
18
30
17
20
10
2
5
4
2
13
2
3
2
7
11

3
17
2
14
1
13
2
4
9
8
6
3
3
20
13
4

254
251
237
231
226
218
208
207
196
194
187
185
182
182
182
182

In Table 8, we list the most prominent countries collaborating with the ETH Domain.
In 20% of the publications, researchers from the united States. In 19% of the output
researchers from Switzerland (outside the ETH Domain) are co-authors. Together
with Germany (18%) these are the most important co-authoring countries for the
ETH Domain.
34

www.cwtsbv.nl

Results

Table 8: Top 12 countries co-authoring with ETH Domain researchers, excluding
ETH Domain internal co-authorship. P[full] and % to ETH Domain’s total
Country
United States
Switzerland
Germany
United Kingdom
France
Italy
Spain
China
Netherlands
Austria
BelgiumJapanwww.cwtsbv.nl

Co-pubs% to total

27,01926,576
24,706
16,113
15,403
12,392
8,344
8,139
6,992
5,502
5,043
4,929

20%
19%
18%
12%
11%
9%
6%
6%
5%
4%
4%
4%

35

REFERENCES

References
Emile Caron and Nees Jan van Eck. Large scale author name disambiguation using
rule-based scoring and clustering. In 19th International Conference on Scienceand Technology Indicators, volume 19, pages 79–86, Leiden, September 2014.
Javier Ruiz-Castillo and Ludo Waltman. Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journalof Informetrics, 9(1):102–117, January 2015. ISSN 17511577. doi: 10.1016/
j.joi.2014.11.010. URL https://linkinghub.elsevier.com/retrieve/pii/
S1751157714001126.
Ludo Waltman and Nees Jan van Eck. A new methodology for constructing apublication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12):2378–2392, December 2012.
ISSN 15322882. doi: 10.1002/asi.22748. URL http://doi.wiley.com/10.
1002/asi.22748.
Ludo Waltman, Clara Calero-Medina, Joost Kosten, Ed C. M. Noyons, Robert J. W.
Tijssen, Nees Jan van Eck, Thed N. van Leeuwen, Anthony F. J. van Raan, Martijn S. Visser, and Paul Wouters. The Leiden ranking 2011/2012: Data collection,
indicators, and interpretation. Journal of the American Society for Information
Science and Technology, 63(12):2419–2432, December 2012. ISSN 1532-2890.
doi: 10.1002/asi.22708. URL https://onlinelibrary.wiley.com/doi/abs/
10.1002/asi.22708.

36

www.cwtsbv.nl

Citation data and analysis

A Citation data and analysis
In this annex we provide more detail about the methodology developed at CWTS
and applied in this study.

A.1 Database coverage
In a bibliometric study, we base the analyses on publication data. To relate counting
and measuring to standards, we depend on international bibliographic databases,
such as Web of Science, Scopus, Dimensions. We realise that by using suchdatabases, we may be missing relevant scientific outputs and achievements. In order
to assess how much the database does cover we calculate the Internal Coverage(IntCov) indicator. This indicator is the ratio of cited references covered by the
database, to the total number of cited references. If a publications contains 10
references, five of which are also in the database, the IntCov of this publication
is 0.5. For a set of publications the IntCov is defined by the average IntCov per
publication. If the IntCov of an institution’s output in WoS is 0.8, we estimate thecoverage of WoS for this institution at 0.8 (80%).

A.2 Database Structure
At CWTS, we calculate bibliometric indicators based on an in-house version of the
Web of Science (WoS) online database, which will be referred to as the CI-system.
The WoS is a bibliographic database that covers publications of about 12,000
journals and each of these journals is assigned to one or more Journal Subject
Categories (JSC). Each publication in the CI-system has a document type. The mostfrequently occurring document types are ‘articles’, ‘reviews’, ‘proceeding papers’,‘corrections’, ‘editorial material’, ‘letters’, ‘meeting abstracts’ and ‘news items’. In
this report, we only consider document types ‘articles’, ‘reviews’ and ’proceedings
papers’. In limiting the analysis to these three types of publications, we consider
that these documents reflect most of the original scientific output in a field.
The CI-system is an improved and enhanced version of the WoS database versionsof the Science Citation Index (SCI), Social Science Citation Index (SSCI), and Arts
& Humanities Citation Index (A&HCI). The CI-system implements a publicationbased field classification which clusters publications into research areas based
solely on citation relations (Waltman and van Eck, 2012) (more detail in Annex
B). One important advantage of this publication-level classification system is thatit allows for a taxonomy of science that is more detailed and better matches thecurrent structure of scientific research. This not only reduces classification bias
but is also essential for calculating field-normalised indicators (Ruiz-Castillo andWaltman, 2015).
Moreover, in this study we include citation data up to 2021. Please note thatpublications require at least one full year to receive citations in order to makewww.cwtsbv.nl

37

Citation data and analysis

robust calculations of citation impact indicators. For this reason, we will workwith publications up to and including 2020, counting citations up to and including
2021. For each publication (and its benchmark publications), we consider 4 yearsof citations since the year of publication. For a publication from 2010, we countcitations in the years 2010-2014.

A.3 Citation Window, Counting Method and Field Normalisation
Citation window
Several indicators are available for measuring the average scientific impact of the
publications of a research unit, e,g. and institution. These indicators are all basedon the idea of counting the number of times the publications of a unit have been
cited. Citations can be counted using either a fixed-length citation window ora variable-length citation window. In the case of a fixed-length citation window,
only citations received within a fixed time period (e.g. four years fixed window)
are counted. The main advantage of a fixed-length citation window is that it ispossible to meaningfully analyse the trend patterns of the non-normalised impact
indicators, setting the same criteria for all publications included. A variable-length
window, on the other hand, uses all the citations that are available in the database
until a fixed point in time, which not only yields higher citation counts (dependingon the window length), but also more robust impact measurements. When using
a variable-length citation window, impact indicators such as the average impact(MCS) and the total impact score (TCS) may systematically present a decreasing
pattern.
In this study, we use a fixed-length window of 4 year (if available) for the overall
period of the analysis (2009-2020). The most recent year for receiving citations is
2021.
Self-citations
In the calculation of advanced citation impact indicators, we disregard self-citations.
A citation is considered a self-citation if the cited publication and the citing publication have at least one author (i.e. last name and initials) in common. The main
reason for excluding self-citations is that they often have a different purpose fromordinary citations. Specifically, self-citations may indicate how different publications of a researcher build on one another, or they may serve as a mechanism forself-promotion rather than for indicating relevant related work. Self-promotion canin turn be used to manipulate the impact of a publication in terms of the numberof citations received. Excluding self-citations from the analysis effectively reduces
the sensitivity of impact indicators to potential manipulation. In doing so, impact
indicators can be interpreted as the impact of researchers’ work on other membersof the scientific community rather than on his or her own work.
38

www.cwtsbv.nl

Citation data and analysis
Field Normalisation
There can be quite large differences in citation practices in different scientific fields.
Field normalisation is about correcting for differences in citation practices between
different scientific fields. The goal of field normalisation is to develop citation-based
indicators that allow for valid between-field comparisons.
In this report, we will use our in-house publication-based classification system ofscience to define the scientific fields that are used in this normalisation process.
This system has three major advantages compared to the conventional journal-basedclassification systems of science: Web of Science Journal Subject Categories:• Proper granularity in terms of fields.
• Fields are defined at the level of publications citing each other, not on allocating complete journals to field(s) where inaccuracies are introduced.
• Publications from journals like Nature, Science, PLoS ONE (multidisciplinary
journals) are allocated to the field they actually belong to and not to theartificial journal field ‘Multidisciplinary Sciences’.
The reasons to use this publication-based classification are furthered explained in
Annex B.
Counting method
Counting methods are about the way in which co-authored publications are handled.
For instance, if a publication is co-authored by researchers from two countries,
should the publication be counted as a full publication for each country or should
it be counted as half a publication for each of them? In this study, we use both fulland fractional counting. Full counting means that if a publication is co-authoredby multiple organisations, that publication counts multiple times, once for everyorganisation, regardless of the weight of their contribution. In this report, we use
mainly the full counted publications for output and fractionalised (by number of
institutions involved) for impact measures.

www.cwtsbv.nl

39

Publication level classification

B Publication level classification
The CWTS citation database is a bibliometric version of Web of Science (WoS). Oneof the special features of this database is the publication-based classification. This
classification is an alternative to the WoS journal classification, the WoS subject
categories. The reason to have this publication-based classification is the problems
we encounter using the journal classification for particular purposes. We discern
the following as the most prominent ones.

B.1 Journal scope (including multi-disciplinary journals)
A journal classification introduces sets of journals to represents a class, in thiscase a subject category. This implies that journals have a similar scope. They
do not need to be comparable with regard to volume (number of articles per year)
but they should represent a similar specialisation. This is not the case, of course.
Journals represent a very broad spectrum. There are very specialist journals (e.g.,
Scientometrics) and very general ones (e.g., Nature or Science but also British
Medical Journal). The classification scheme can therefore not be very specialised.
In WoS, a subject category Multi-disciplinary hosts the very general ones so that abibliometric analysis of, for instance, the Social Sciences or Nanotechnology, using
this classification, will not take papers in Nature into consideration.

B.2 Granularity of the WoS subject categories
The WoS journal classification scheme contains 255 elements. As such it is a stablesystem. In many cases however, it appears that these 255 subject categories are
insufficient to be used for proper field analyses. The problem is that the granularityof the system looks somewhat arbitrary. ‘Biochemistry & Molecular Biology’ on the
one hand and ‘Ornithology’ on the other, for instance, represent rather different
aggregates of research. This is illustrated by the number of journals in each ofthem. Where the ‘Biochemistry & Molecular Biology’ category contains almost
500 journals, ‘Ornithology’ has only 27. We acknowledge that there is no perfectgranularity, but we argue that in the WoS subject categories the differences arereally too big. A classification based on more objective grounds does not solve thisproblem but is at least transparent.

B.3 Multiple assignment of journals to categories
In journal classifications from multi-disciplinary databases, journals are assigned
to more than one category. Journals often have broader scopes than the categories
allow. Also here there are large differences between categories. In the example weused before, ‘Biochemistry & Molecular Biology,’ journals are on average assigned
to almost 2 categories. This means that (on average) each journal in this category is
also assigned to one other category. For the more specialist category of ‘Ornithol40

www.cwtsbv.nl

Publication level classificationogy’, the average is 1. This means that in this category all journals are assigned
to this category only. If publications in journals with a multiple assignment would
always cover the categories at stake, this should not necessarily be a problem.
However, it mostly means that such journals structurally contain publications fromthe different categories. Therefore, publications may be assigned to two categories
although they belong to just one of them.

B.4 The CWTS publication-based classification scheme
CWTS has developed an advanced alternative for the Web of Science journal classification. It counters three major issues:
1. Journal scope (including multi-disciplinary journals)
2. Granularity of the WoS subject categories
3. Multiple assignment of journals to categories
The CWTS publication-based classification is developed as described in Waltman
and van Eck (2012). Since the first version there have been yearly updates of thesystem. The main characteristics of the classification are as follows.
Publication to publication citation clustering
Clusters of publications are created on the basis of citations from one publication toanother. Tens of millions of publications have been processed. The clusters containpublications from multiple years (2000–2020). Each publication is assigned to one
cluster only at each level. A cluster is considered, and in many cases validated as,
representative for disciplines, research areas, fields or sub-fields. For each cluster,
we can calculate growth indices pointing at changing research focus over time.
Multi-level clustering
The classification scheme has at present three different levels. The clusters arehierarchically organised. Currently we discern the following levels.
1. A top level of 25 clusters (fields)
2. A second level of around 800 clusters (sub-fields)
3. A third level of more than 4,000 clusters (research areas or micro-fields)
A common way of visualising the landscape of science by the publication clusters is
a 2-dimensional map. In such a landscape (see Figure 9), we position publication
clusters in relation to each other on the basis of citation traffic. The denser the traffic
between two clusters, the closer they are. The two dimensions do not represent
anything. The only thing that matters is the distance. Furthermore, the size of awww.cwtsbv.nl

41

Publication level classification

cluster represents the relative volume (number of publications included), while thecolor coding adds a main clustering labeled by main disciplines.

Mai
ndi
sc
i
pl
i
ne
Soc
i
alSc
i&HumanBiomed&Heal
t
hSc
i
Physi
c
alSc
i&Engi
n
Li
f
e&Ear
t
hSc
i
Mat
hs&ComputSciFigure 9: Landscape of all science (around 30 million WoS publications). Circles
represent (over 4,000) publication clusters. Position is defined by citation traffic
between clusters. Size indicates relative volume. Color reflects 5 main disciplines

42

www.cwtsbv.nl

Interdisciplinary research

C Interdisciplinary research
While there are different understandings of interdisciplinarity, the definition thathas gained more consensus is the one provided by the US National Academy of
Sciences (2005) that states:“Interdisciplinary research (IDR) is a mode of research by teams or individuals that integrates information, data, techniques, tools, perspectives,
concepts, and/or theories from two or more disciplines or bodies of specialised knowledge to advance fundamental understanding or to solve
problems whose solutions are beyond the scope of a single disciplineor field of research practice."https://www.nap.edu/read/11153/chapter/4
There are two key elements in this definition we consider as basic notions to articulate our proposal: the concept of integration and the idea of combining knowledge
from two or more disciplines.
We characterise interdisciplinarity at the level of each individual publication, by
analysing the disciplines cited by the publication. This approach will allow us
to consider the citations to distinct disciplines by the same citing publication as aproxy of the integration of knowledge from different disciplines. For this analysis we
consider the Web of Science Journal Subject Categories as disciplines. We analyse
the degree or extent of integration through the concept of diversity. Diversity
is based on three concepts: variety, balance and disparity. We operationaliseinterdisciplinarity using Rao-Stirling diversity, an indicator which captures thethree inter-related concepts of diversity, and is computed as follows:X∆=pi pj dijij

(i ̸= j)Where pi is the proportion of cited references in the subject categoryi, pj is the proportion of cited references in the subject category j, anddij is the cognitive distance between the subject categories i and j
In this formula, disparity refers to the cognitive distance existing between two scientific disciplines (or subject categories, in our case). In order to compute thedisparity measure, we will create a similarity matrix Sij for the WoS subject categories based on the of citation flows between them. This will be then transformed
into a Salton’s cosine similarity matrix in the citing dimension. In this transformed
matrix, the Sij represents the similarity between each pair of WoS categories, thus
the cognitive distance (d) between two subject categories can be computed as d =1- Sij.
www.cwtsbv.nl

43

Interdisciplinary research

The indicators of interdisciplinarity will allow us to identify an institution’s subject
categories of a prepresenting the most interdisciplinary research.
We apply the state of the art in analysing interdisciplinarity using bibliometrictechniques. However, current approaches to characterise interdisciplinary researchfrom a bibliometric perspective remain contentious. Like any other methodology
suggested so far to measure and characterise interdisciplinarity based on scientificpublications, our approach is not free of limitations and therefore results of theseanalyses need to be interpreted with caution.

44

www.cwtsbv.nl