Statement by the RU11 University Association
Request for a revision of the assessment methods used by Thomson Reuters
for the Times Higher Education (THE) World University Rankings
July 31, 2011
The purpose of analyzing academic research data, particularly publication and citation trends is to provide
diverse objective information on universities and other academic institutions that can be used byresearchers and institutions for various evaluations and the setting of objectives. The 2010 Thomson
Reuters / THE World University Rankings, however, do not give sufficient consideration to the uniquecharacteristics of universities in different countries or the differing research needs and demands fromsociety based on country, culture and academic field. As a result, those rankings are likely to lead to an
unbalanced misleading and misuse of the citation index.
RU11 strongly requests, therefore, that Thomson Reuters / THE endeavors to contribute to academic society
by providing objective and impartial data, rather than imposing a simplistic and trivialized form ofuniversity assessment.
Specifically, we request that Thomson Reuters addresses the following three issues:
1.

The ranking of citations based on either citations per author (or faculty) or citations per paper represent
two fundamentally different ways of thinking with regards to academic institutions: are the institutions
to be viewed as an aggregation of their researchers, or as an aggregation of the papers they haveproduced? We believe that the correct approach is to base the citations ranking on citations per faculty,
as has been the practice in the past.

2.

We request a revision of the method used for regional modification.

3.

We request the disclosure of the raw numerical data used to calculate the citation impact score for thevarious research fields at each university.

The RU11 University Association
Hokkaido University

Tokyo Institute of Technology

Tohoku University

Nagoya University

Tsukuba University

Kyoto University

University of Tokyo

Osaka University

Waseda University

Kyushu University

Keio University

Attached Reference Materials
In support of the above statement, we have provided an analysis of discrepancies in rankings based on thefollowing publicly available data:

1.

The rankings based solely on citations per paper for the 27 universities in the Top Asian Universities
2010, published in the Time HE World University Rankings 2010–2011 (published in September 2010,
covering a 5-year period, with regional modification).
http://www.timeshighereducation.co.uk/world-university-rankings/2010–2011/asia.html#score_CI%7Cs
ort_ind%7Creverse_false

2.

The total figures for the 11-year period from January 1, 2000 to December 31, 2010, as included inThomson Reuters’ Essential Science Indicators (published in April 2011, covering an 11-year period, noregional modification).
http://isiknowledge.com/esi

3.

Quacquarelli Symonds The 2010 QS Asian University Rankings (citations per paper, covering a 5-yearperiod, no regional modification)
http://www.topuniversities.com/university-rankings/asian-university-rankings/2010/indicator-rankings/citations-per-faculty4.

The 2010 QS World University Rankings (citations per faculty, covering a 5-year period, no regionalmodification)

Explanation of Reference Materials

1.

Reference 1: Comparison with the citations ranking by Quacquarelli Symonds (no regionalmodification).
This chart compares the citations per paper rankings produced by Thomson Reuters with those
produced by Quacquarelli Symonds. The former shows the results after regional modification, and thelatter shows the results with no regional modification. The huge impact that the regional modification
method used by Thomson Reuters has on the ranking of Japanese universities is very apparent. While
there may be differences in the academic databases which are being used as the raw data, it seems
likely that the regional modification is the cause of such large discrepancies.

2.

Reference 2: Comparison with the citations ranking for an 11-year period (no regional modification).
This chart compares the 5-year period citations per paper rankings (with regional modification) withthe 10-year period citations per paper rankings (no regional modification). Both rankings are based on
Thomson Reuters’ data. This comparison shows that when the period of assessment is increased, thepositions of historically respected institutions, including those in Japan, consistently improve. This isdue to the fact that basic research requires a relatively long time for its value to be understood. This
would seem to indicate that the 2010 Times Higher Education World University Rankings areexcessively weighted in favor of research which delivers quickly attainable benefits.

3.

Reference 3: Comparison of the rankings of the RU11 universities when ranked by citations per paperand by citations per faculty.
This chart compares Thomson Reuters’ citations per paper ranking with Quacquarelli Symonds’ citations

per faculty ranking. The two rankings cover the same 5-year period; however, the Thomson Reuters
ranking includes regional modification. The rank of many institutions, including the RU11 universities
is higher in the citations per faculty ranking. The 2010 Times Higher Education World University
Rankings give great recognition to so-called “home run papers,” while the smaller, accumulative
contributions to academia made by scholars producing “multi hit” papers are undervalued. It isquestionable whether this is an appropriate way to evaluate academic research.

Attachments:

Reference 1: Comparison with the citations ranking by Quacquarelli Symonds (no regional modification)
Reference 2: Comparison with the citations ranking for an 11-year period (no regional modification)
Reference 3: Comparison of the ranking of the RU11 universities when ranked by citations per paper and bycitations per facultyReference 1: Discrepancies in the ranking order of the RU11
universities in the citations ranking by Quacquarelli Symonds (noregional modification). 	
The ranking of 27 universities in the Top Asian Universities 2010
ranking, as published in the Times HE WUR 2010–2011	
Criterion: Citations (5-year period, with regional modification,citations per paper) Fundamental Database: 1The ranking of 24 universities in the 2010 QS Asian UniversityRankings	
Criterion: Citations (5-year period, no regional modification,citations per paper) Fundamental Database: 3 	

Top Asian Universities 2010 by Citations (5-year period, published in September 2010)
INSTITUTION

Asia Ranking

COUNTRY / REGION

2010 QS Asian University Rankings (citations per paper, 5-year period)

1 Hong Kong University of Science and Technology

Hong Kong

University of Hong Kong

2

2 Hong Kong Baptist University

Hong Kong

University of Tokyo

33 Pohang University of Science and Technology

Republic of Korea

Kyoto University

44 University of Hong Kong

Hong Kong

Osaka University

55 Bilkent University

TurkeyNational University of Singapore

66 University of Science and Technology of China

China

Pohang University of Science and Technology

14

7 National University of Singapore

Singapore

Tohoku University

32

8 Peking University

China

Hong Kong University of Science and Technology

33

9 Sun Yat-sen University

China

Seoul National University

35

10 National Tsing Hua University

TaiwanHong Kong Baptist University

39

11 Middle East Technical University

TurkeyTokyo Institute of Technology

47

12 Nanjing University

China

Yonsei University

52

13 Hong Kong Polytechnic UniversityHong Kong

University of Science and Technology of China

60

14 National Taiwan University

TaiwanNational Tsing Hua University

71

15 National Sun Yat-Sen University

TaiwanPeking University

74

16 University of Tokyo

Japan

Korea Advanced Institute of Science and Technology

81

17 Seoul National University

Republic of Korea

Nanyang Technological University

84

18 Tsinghua University

China

Nanjing University

105

19 Yonsei University

Republic of Korea

Sun Yat-sen University

109

20 Kyoto University

Japan

Hong Kong Polytechnic University

130

21 Korea Advanced Institute of Science and Technology

Republic of Korea

National Sun Yat-Sen University

unknown

22 Tokyo Institute of Technology

Japan

National Chiao Tung University

154

23 Nanyang Technological University

Singapore

Zhejiang University

186

24 Zhejiang University

China

Tsinghua University

214

25 Tohoku University

Japan

26 Osaka UniversityJapan

27 National Chiao Tung University

TaiwanThree other universities not included

1	Reference 2: Discrepancies in the ranking order of the RU11universities in the citations ranking over an 11-year period (noregional modification). 	
The ranking of 27 universities in the Top Asian Universities 2010
ranking, as published in the Times HE WUR 2010–2011	
Criterion: Citations (5-year period, with regional modification, citations
per paper) Fundamental Database: 1

Ranking based on the average values for the 11-year period included inThomson Reuters’ Essential Science Indicators (published in April 2011)	
Criterion: Citations (11-year period, no regional modification, citationsper paper) Fundamental Database: 2Ranking of research institutions based on Thomson Reuters’
Essential Science Indicators

Asia Ranking World Ranking(citations per paper, 11-year period from 2000–2010)

Individual Universities

1 Hong Kong University of Science and Technology Hong Kong

University of Tokyo1133

2 Hong Kong Baptist University

Hong Kong

Osaka University2144

3 Pohang University of Science and Technology

Republic of Korea

Kyoto University

3147

4 University of Hong Kong

Hong Kong

University of Hong Kong41775 Bilkent University

TurkeyPohang University of Science and Technology

5unknown

6 University of Science and Technology of China

China

Tohoku University

6189

7 National University of Singapore

Singapore

Hong Kong University of Science and Technology

7unknown

8 Peking University

China

Tokyo Institute of Technology

8193

9 Sun Yat-sen University

China

National University of Singapore

9194

10 National Tsing Hua University

TaiwanSeoul National University

10196

11 Middle East Technical University

TurkeyHong Kong Baptist University

11unknown

12 Nanjing University

China

Yonsei University

12unknown

13 Hong Kong Polytechnic University

Hong Kong

National Taiwan University

13197

14 National Taiwan University

TaiwanUniversity of Science and Technology of China

14198*

15 National Sun Yat-Sen University

TaiwanPeking University

15198*

16 University of TokyoJapanKorea Advanced Institute of Science and Technology

16unknown

17 Seoul National University

Republic of Korea

National Tsing Hua University

17unknown

18 Tsinghua University

China

Sun Yat-sen University

18198*

19 Yonsei University

Republic of Korea

Nanjing University

19198*

20 Kyoto University

JapanBilkent University

20

unknown

21 Korea Advanced Institute of Science and Technology
Republic of Korea

Hong Kong Polytechnic University

21

unknown

22 Tokyo Institute of Technology

JapanNanyang Technological University

22

unknown

23 Nanyang Technological University

Singapore

Tsinghua University

23198*

24 Zhejiang University

China

National Sun Yat-Sen University24unknown

25 Tohoku University

JapanZhejiang University

25

198*

26 Osaka University

JapanMiddle East Technical University

26

unknown

27 National Chiao Tung UniversityTaiwan

National Chiao Tung University

27

unknown

Top Asian Universities 2010 by Citations (5-year period, published in September 2010)

INSTITUTION

COUNTRY / REGION* World ranking information gathered by the Chinese Academy of Sciences
Note: The Asia rankings above were calculated by individuallyreferencing each institution from Essential Science Indicators.
Government research institutions, such as JST and RIKEN are
not included.

2	Reference 3: Comparison of the rankings of the RU11universities when ranked by citations per paper and bycitations per faculty 	
The ranking of 27 universities in the Top Asian Universities 2010
ranking, as published in the Times HE WUR 2010–2011
Criterion: Citations (5-year period, with regional modification,citations per paper) Fundamental Database: 1

The ranking of 27 universities in the 2010 QS World University
Rankings
Criterion: Citations (5-year period, no regional modification,citations per paper / citations per faculty) Fundamental
Database: 4
citations/faculty

citations/facultyWorld Ranking

Asia Ranking

Korea, South56

1Japan802Kyoto UniversityJapan86

3Hong Kong

Osaka University

Japan

88

4Bilkent University

TurkeyHokkaido UniversityJapan110

56University of Science and Technology of China

China

Tokyo Institute of Technology

Japan

111

67National University of Singapore

Singapore

National Tsing Hua University

Taiwan123

78Peking University

China

Nagoya University

Japan

127

8ChinaTohoku UniversityJapan

140

9Taiwan

University of Hong Kong

Hong Kong

156

10Hong Kong

Times HE Top Asian Universities 2010 by Citations (5-year period, published in September 2010)

QS Asian University Rankings 2010 - Top 200

INSTITUTION

COUNTRY / REGION

1Hong Kong University of Science and Technology

Hong Kong

2Hong Kong Baptist University

Hong Kong

Pohang University of Science And Technology(POSTECH)
The University of Tokyo

3Pohang University of Science and Technology

Republic of Korea

4University of Hong Kong

5910Sun Yat-sen University
National Tsing Hua University

11Middle East Technical University

TurkeyThe Chinese University of Hong Kong

161

1112Nanjing University

China

University of Science and Technology of China

China

178=

12Hong Kong

204

13Korea, South216

14Taiwan223

1513Hong Kong Polytechnic University

Hong Kong

The Hong Kong University of Science and Technology

14National Taiwan University

Taiwan15National Sun Yat-Sen University

TaiwanKAIST - Korea Advanced Institute of Science &Technology
National Chiao Tung University16University of TokyoJapan

National University of Singapore (NUS)

Singapore

224

1617Seoul National University

Republic of Korea

Seoul National University

Korea, South

233

1718

Tsinghua University

China

Kyushu University

Japan

235=

1819

Yonsei University

Republic of Korea

University of Tsukuba

Japan

266

1920

Kyoto University

Japan

National Sun Yat-sen University

Taiwan

268=

20

21

Korea Advanced Institute of Science and TechnologyRepublic of Korea

The Hong Kong Polytechnic University

Hong Kong

283

2122

Tokyo Institute of TechnologyJapan

Zhejiang University

China

284

2223

Nanyang Technological University

Singapore

Peking University

China

unknown

unknown

24

Zhejiang University

China

Tsinghua University

China

unknown

unknown

25Tohoku University

Japan

Nanyang Technological University (NTU)

Singapore

unknown

unknown

26Osaka University

Japan

Yonsei University

Korea, Southunknown

unknown

National Chiao Tung University
Below 27 The Chinese University of Hong Kong

TaiwanNational Taiwan University (NTU)

Taiwanunknown

unknownHong Kong

Keio UniversityJapan

unknown

unknown

Below 27 Nagoya UniversityJapan

Nanjing University

China

unknown

unknown

Below 27 Kyushu University

Japan

Waseda UniversityJapan

unknown

unknown

Below 27 University of Tsukuba

Japan

Hong Kong Baptist University

Hong Kong

unknown

unknown

Below 27 Hokkaido University

Japan

Turkeyunknown

unknown

Below 27 Keio UniversityJapan

Bilkent UniversitySun Yat-sen University

China

unknown

unknown

Below 27 Waseda UniversityJapan

Middle East Technical University

Turkeyunknown

unknown

273