September 5, 2019
To: Times Higher Education

Proposal concerning Times Higher Education World University Rankings
Requests aimed at more appropriate application ofranking indicators relating to research capabilities
The decline in research capabilities at Japanese universities is becoming more
severe, and Japan’s relative presence as a producer of academic papers is falling in both
qualitative and quantitative terms. In addition to ongoing reform efforts led byindividual universities, appropriate responses at the national policy level are now moreurgent than ever. These include utilizing specialist personnel such as university researchadministrators to improve the fundamental research environment and enhancing
government subsidies for scientific research and other forms of research funding.
In implementing such policies, it is important to have evidence offering an
objective and dispassionate view of universities’ research capabilities. Needless to say,assessment of universities’ research capabilities requires a combination of indicators
rather than just one single indicator.
Against this backdrop, the placings of Japanese universities in the Times
Higher Education (THE) World University Rankings continue to fall year after year. Ofcourse, pinning all one’s hopes on university rankings would be mistaking the means
for the end, but rankings should not be regarded simply as short-term, temporary results.
We believe that universities need to proactively utilize such rankings in their own efforts
to enhance medium- to long-term research capabilities and reform themselves.
At the same time, as many Japanese universities and government agencies havealready grasped, indicators evaluating universities’ research capabilities, especially thecitation indicator, have a decisive impact on university rankings. As you know, however,
one-sided views of university research capabilities based on limited indicators causesome unease and attract a certain amount of criticism. There are fears that erroneous
application of limited factors such as the citation indicator could lead to erroneous
evaluations that do not reflect the actual state of each university’s reinforcement ofresearch capabilities.

1

In order to further develop academic research at research universities, theResearch University Consortium of Japan has disseminated information including
various proposals. Four years ago our precursor, the Research University Network ofJapan, submitted requests to THE World University Rankings regarding improvements
to ranking indicators. Although this prompted improvements to aspects including
counting methods for “kilo-author” papers, unfortunately other requests, including thoseconcerning the vital citation indicator, were not addressed. Given this, we have decided
to renew our requests to THE World University Rankings in the aim of achieving more
appropriate application of ranking indicators relating to research capabilities. Ourrequests focus on developing a common awareness of the dangers inherent in one-sidedviews of university research capabilities and ensuring dispassionate and appropriate
application of such indicators. From a global perspective, we sincerely request that you
share with us diverse ideas regarding research universities and make an appropriate
response to the points raised.

Research Capability Analysis Task Force
Research University Consortium of Japan
Contact for Inquiries:
Amane Koizumi
Project Professor
National Institute of Natural Sciences
Tel: +81-3-5425-1301
nins-ura-jimu@nins.jp

2

Details of Proposal(1) Ongoing communication with Japanese research universities
Information exchange with Japanese research universities has ceased since ourlast request was submitted four years ago. We believe that it is important to maintainongoing two-way communication with THE and hope that this request will provide
impetus to create forums for dialogue and build trust.
(2) The research and citation indicators(a) The citation indicator
At present, the citation indicator overemphasizes the standardized Field
Weighted Citation Impact (FWCI). FWCI is used as the citation indicator, and fieldadjusted and country-adjusted average citations are used in evaluation indicators.
We agree that FWCI is an excellent indicator for this purpose that should becontinuously used in the future. However, with this methodology a university’s
overall FWCI can be skewed by a very small number of papers that temporarily
attract an extreme flurry of citations. FWCI alone cannot be said to enable anaccurate grasp of the research capabilities of a university with diverse researchstrengths. We request that THE should use a combination of several indicators withFWCI rather than relying solely on FWCI.
For example, it would be possible to use the types of indicators outlined below
as citation indicators beyond FWCI.
(a)-1

Field-adjusted citations divided by researcher numbers could be used
to evaluate citations, rather than just the average FWCI. We believe it
would be more appropriate to use FWCI combined with the total numberof field-adjusted citations (e.g., the accumulated FWCI of individualpapers) divided by researcher numbers (converted into FTEs). This
“citation/researcher numbers” methodology could be expected to enable
scale-based adjustments for universities with diverse characteristics.

(a)-2Rather than evaluating only standardized citation indicators like
FWCI, methodologies should also evaluate research capabilities in termsof substantiality—or, Atsumi (in Japanese)—i.e., a certain level ofquality at or above a certain volume. Using substantiality as an3

evaluation indicator, a university’s capability to produce a given level ofoutstanding papers at or above a certain volume—for example, numberof top 10% papers —would also become an indicator of researchcapabilities at global universities. An indicator that counts number of top
10% papers divided by the number of researchers should also beintroduced.
(b) Fractional counting of papers
We welcome THE’s review of the handling of “kilo-author” papers carrying thenames of a thousand or more authors. However, there is also a need for appropriate
evaluation of papers with hundreds of authors presented under the auspices ofinternationally important research projects such as those promoting joint use andjoint research, especially relating to large devices. In order to appropriately evaluate
multi-author papers, we request that for all papers, number of papers and citationsshould be fractionally counted by number of authors or number of institutions towhich authors are affiliated.
(3) Further considerations(c) Faculty count in indicators
Universities’ own declarations are used for the faculty count, which is thedenominator for indicators, etc., but there is no international consistency incounting methods, and this disparity could cause problematic variations in score.
Faculty count should be ascertained objectively, based on common global
standards. Japanese universities will reaffirm calculation methods and request
rigorous application of such methods by THE. In the future, faculty counts willneed to be ascertained using an objective third-party database.
(d) Handling of papers published in non-English-language journals
Papers published in non-English-language journals, such as those written in thenative languages of their respective countries, target a limited audience and tend not
to be cited globally. Moreover, the characteristics of each language make it difficult
to adjust for such papers. Given these realities, it is not necessarily appropriate tocompare research capabilities on equal terms with English-language papers. For
these reasons, we request that when analyzing citations, papers published in nonEnglish-language journals should be excluded from totals, even if they are listed indatabases.
4(e) Ranking disclosure methods
Since World University Ranking results help universities to enhance theirresearch capabilities and improve their research environments, we request that THE
expands the scope of information available free of charge to include not only thepillars, but also each university’s scores for the 13 indicators and real values prior
to their conversion into scores for one’s own university. Japanese research
universities collaborate in compilation of THE World University Rankings withoutcompensation, and we would like to receive feedback from THE that will help us tostrengthen our research capabilities.
We would also like results to include not only rankings, but also an indicationof the percentile achieved among all universities ranked. Doing this would enable
universities to ascertain their relative position even if the number of universities
ranked increases.

5