U-Multirank : implementation of a user-driven, multi-dimensional and international ranking of higher education institutions : final report on project phase 1.
Date: 2016-02-03
Institutions: Directorate-General for Education, Youth, Sport and Culture
Subjects: Internet site, comparative education, education policy, educational system, higher education, teaching curriculum, teaching quality, university

U-Multirank: Implementation of a user-driven,
multi-dimensional and international ranking of higher
education institutions

Final Report on Project Phase 1

Europe Direct is a service to help you find answers
to your questions about the European Union.

Freephone number (*):
00 800 6 7 8 9 10 11

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2015

ISBN 978-92-79-52310-6
doi: 10.2766/833086

© European Union, 2015
Reproduction is authorised provided the source is acknowledged.

Printed in

PRINTED ON ELEMENTAL CHLORINE-FREE BLEACHED PAPER (ECF)

PRINTED ON TOTALLY CHLORINE-FREE BLEACHED PAPER (TCF)

PRINTED ON RECYCLED PAPER

PRINTED ON PROCESS CHLORINE-FREE RECYCLED PAPER (PCF)

U-Multirank
Implementation of a user-driven, multi-
dimensional and international ranking of higher
education institutions

EC Contract Number EAC-2012-0516

Final Report on Project Phase 1

“The information and views set out in this report are those of the authors and do not
necessarily reflect the official opinion of the Commission. The Commission does not
guarantee the accuracy of the data included in this study. Neither the Commission nor
any person acting on the Commission behalf may be held responsible for the use”.”.

Table of Contents

Table of Contents

| Page 4

1

2

2.1

2.1.1

2.1.2

2.1.3

2.1.4

2.2

2.2.1

2.2.2

2.2.3

2.3

2.3.1

2.3.2

2.3.3

2.4

2.4.1

2.4.2

2.4.3

2.4.4

3

3.1

3.2

3.3

3.3.1

3.3.2

3.4

3.5

3.6

Executive Summary .....................................................................................................5

An overview of progress made in the first project phase in the four project work
packages ..................................................................................................................11

Work package A: Data collection.................................................................................11

Participation in U-Multirank 2014 and 2015 .................................................................11

The data collection process ........................................................................................12

Data quality and quality assurance ..............................................................................13

U-Multirank results and achievements ........................................................................13

Work package B: Web-tool implementation.................................................................14

The process of developing the web tool.......................................................................14

U-Multirank mapping indicators: Comparing like-with-like.............................................15

The U-Multirank ranking tool: Measuring performance .................................................15

Work package C: Communication strategy and plan......................................................16

Communication strategy ............................................................................................16

Communication actions .............................................................................................16

Readymade rankings .................................................................................................17

Work package D: Further refinement ..........................................................................18

Continuous refinement..............................................................................................18

Incorporation of new fields ........................................................................................18

Funding and governance model ..................................................................................18

Pilot study on Research Performing Organisations (RPOs)..............................................19

Conclusions, challenges and key recommendations for the second phase of the
implementation project .............................................................................................19

Participation and further extension.............................................................................20

Data and data quality ................................................................................................22

Further development of indicators..............................................................................22

Difficult indicators .....................................................................................................22

New dimensions and indicators ..................................................................................23

Web tool..................................................................................................................24

Communication ........................................................................................................25

Governance and Funding Model .................................................................................26

Executive Summary

| Page 5

1

Executive Summary

In the European Commission funded U-Multirank feasibility study (2009 – 2011) a completely new and
innovative model for a multi-dimensional global ranking was developed. The model was tested with
150 pilot institutions with positive results, and, on this basis the European Commission decided to
implement U-Multirank on a global scale through an implementation project. This report covers the
progress made in the first phase of this implementation project (December 2012 – March 2015) and
identifies challenges and recommendations for the second phase (April 2015 – March 2017).
U-Multirank is being implemented by a consortium headed by Professor Dr. Frans van Vught of the
Center for Higher Education Policy Studies (CHEPS) in the Netherlands and Professor Dr. Frank Ziegele
of the Centre for Higher Education (CHE) in Germany. Other partner organisations include the Centre
for Science and Technology Studies from Leiden University (CWTS), Catholic University Leuven,
academic publishers Elsevier, the Bertelsmann Foundation, Push and software firm Folge 3. The
consortium also works closely with a range of national ranking partners and stakeholder organisations,
including student organisations.

U-Multirank has changed the world of rankings



The development and publication of U-Multirank has changed the world of rankings by
introducing a radically different approach: multi-dimensional and user-driven. It had an impact
on other global rankings which started to introduce some first interactive elements to their
web tools and to widen the scope of their indicators by adding indicators first presented by U-
Multirank. From the first publication, U-Multirank results have proven the necessity of a multi-
dimensional approach: U-Multirank puts the spotlight on higher education institutions which
have never been visible before in any other global ranking, showing their strong performance
in particular dimensions or indicators, including research indicators. U-Multirank results make
clear that there is no the best university. First, no institutions perform top on all indicators.
Second, results vary according to the selection of indicators by users. Instead U-Multirank
shows the performance profiles of higher education institutions with their particular strengths
and weaknesses.

 In addition U-Multirank increased the stock of data on global higher education and provides
general insights into the global higher education system. For example: although the growing
importance of interdisciplinary research is highlighted by many stakeholders U-Multirank data
show that still most universities are not very strong in interdisciplinary research. Only a few
institutions are strong in publishing interdisciplinary publications.

 The release of the first two U-Multirank rankings - with the number of participating institutions
exceeding the targets agreed with the European Commission - is the major achievement of the
first phase of the implementation project. This achievement required the U-Multirank
consortium to turn a ranking design into an actual ranking that stayed true to its design
principles. These principles and the key features of U-Multirank are outlined in the sections that
follow.

Executive Summary

| Page 6

Key recommendations for the second phase of the implementation project
Although successful, U-Multirank is not without challenges, and the following key recommendations
are suggested for the second phase of the implementation project.





Recruitment. While continuing efforts to recruit universities from Europe, the focus of
targeted recruitment activities should be on Universities of Applied Sciences (UAS) and on
institutions from large higher education systems outside Europe (particularly US, Japan, China,
Australia and Canada). Within Europe, active participation of more universities from the UK
would be valuable. The communication with these institutions and countries should address
their interests in a tailor-made way.
Services to participating institutions. The provision of services to the participants should be
led by considerations of the cost-benefit-analysis for institutions; U-Multirank has to
continuously provide benefits. To get a clear insight into the interests of the participants,
especially in terms of using U-Multirank for institutional management, there should be a
continuous exchange with institutions. A network of institutional data coordinators, providing
feedback to U-Multirank about institutional needs and stimulating exchange among
institutions, could be an important instrument to further increase the benefits for institutions.
 Data collection. U-Multirank should continue trying to reduce the burden on institutions in
terms of data delivery. Instruments should be reviewed annually; the necessity of all data
points in the questionnaires should be reconsidered every year, and the continuous
refinement of data definitions based on institutional feedback is crucial. In this context,
options to use data from international and national data sets and from national student
surveys (either as a tool to verify data or to prefill questionnaires) should be explored further.
 Dimensions and indicators. U-Multirank has to stay responsive to stakeholders’ needs. This
may require the development of new indicators, which may be in tension with the objective
to decrease the burden of data collection. U-Multirank should test the feasibility of new
indicators, but only introduce these if the reliability and availability of data are guaranteed.
Two new indicators should be explored: the development of a valid and reliable indicator on
applied research for 2016 and one or more on students' social conditions for 2017.

 Web tool. U-Multirank should continue to explore ways to optimise the user-friendliness of
the web tool. The visibility and usability of the U-Multirank web tool would be substantially
enhanced by the development of a mobile version. Usage data indicate that a substantial
proportion of current usage is via mobile devices. While the web tool works well on tablets, a
separate version optimised for smartphone and small tablet use and a mobile application
should be developed.
Communication. The efforts to improve search engine optimisation should be intensified as
this is crucial for U-Multirank to be easily found by potential users. U-Multirank needs more
communication throughout the year to attract users and to demonstrate the potential of the
ranking. National media partnerships should be strengthened, for example by providing
“national stories” particularly in countries where U-Multirank has a high coverage of
institutions.



 Governance and funding model. In the second phase of the project the key challenge will be
to explore with a range of different potential partners the feasibility of taking forward the
recommendations on a future governance and funding model.

Executive Summary

Introduction

| Page 7

 U-Multirank is a new multi-dimensional, user-driven approach to international ranking of
higher education institutions. The dimensions it includes are teaching and learning, research,
knowledge transfer, international orientation and regional engagement. Based on empirical
data U-Multirank compares institutions with similar institutional profiles and allows users to
develop personalised rankings by selecting performance measures/indicators in terms of their
own preferences.

 U-Multirank is based on a methodology that reflects both the diversity of higher education
institutions and the variety of dimensions of university excellence in an international context.
The data included in U-Multirank are drawn from a number of sources, providing users with a
from
comprehensive set of information: information
international bibliometric and patent data bases; and a survey of more than 85,000 students
at participating universities – one of the largest student samples in the world and offering
student users of U-Multirank a unique peer perspective.

institutions;

supplied

data

by



The first ranking – in May 2014 – included more than 850 higher education institutions, 1,000
faculties and 5,000 study programmes from 74 countries around the world. It provided an
institutional ranking of whole institutions as well as field-based rankings for electrical and
mechanical engineering, business studies and physics. In the second release – in March 2015
– the coverage increased to 1,200 higher education institutions, more than 1,800 faculties and
7,500 study programmes in seven fields of study from 83 countries. With 21,000 data scores
on the institutional level and 37,000 on the field-based level, U-Multirank is the largest global
university ranking - and the most comprehensive information system on universities - in the
world.

The U-Multirank approach to ranking

 U-Multirank is multi-dimensional. U-Multirank takes a different approach to the existing
global rankings of universities. It is multi-dimensional and compares university performances
in the different activities that they are engaged in. It is not confined to research but takes into
account different aspects and dimensions of the performance of universities.

 U-Multirank does not produce a composite, weighted score across these different areas of
performance and then use these scores to produce a numbered league table of the world’s
“top” 100 universities. The underlying principle is that there is no theoretical or empirical
justification for such composite scores. Therefore, the U-Multirank methodology looks at the
scores of universities on individual indicators and places these in five performance groups
(“very good” through to “weak”).

 U-Multirank is user-driven. Closely related to the multidimensional approach is the focus on
users’ needs. U-Multirank provides information relevant for decision making by different
parties: students, university administrators, policy-makers, academics, business leaders, etc.
Composite indicators, as used in other rankings, define the relevance of each indicator
uniformly. U-Multirank, on the other hand, recognises that different users of rankings have
quite different preferences and priorities with regard to what they see as important areas of
university activity (“Quality lies in the eye of the beholder”). Hence U-Multirank leaves the
decision about the relevance of different performance indicators to its users. This is facilitated
by U-Multirank’s interactive web tool.

Executive Summary

| Page 8

 U-Multirank is stakeholder-oriented. Many stakeholders have been involved in the
development of U-Multirank – its indicators, the underlying data collection instruments, as
well as the design of the interactive web tool through which the results are presented. The
information needs of the users of U-Multirank and their suggestions about new and existing
features and functions are monitored constantly and taken into account.

 U-Multirank is a multi-level ranking. U-Multirank combines institutional ranking (of whole
institutions) with a set of field-based rankings that focus on particular academic disciplines or
groups of programmes. Both are equally important. Whereas field-based information is most
important to many users of rankings (for example, to students who are looking for a university
in the field they want to study, to academic researchers interested in comparisons with
colleagues in their field), other users (including university presidents and policy makers) are
also interested in information about the performance of institutions as a whole.

 U-Multirank shows the diversity of institutions. In contrast to existing global rankings, which
are rankings of only one university type (internationally-oriented research universities), U-
Multirank is broader and provides information on institutions with different missions. It covers
institutions with a range of profiles: universities of applied sciences, regionally-oriented
colleges, and specialised institutions such as Arts Academies and teacher training colleges. U-
Multirank highlights the diversity in the various institutional profiles. U-Multirank includes
more than 300 institutions which have never been visible in global rankings before, many of
them showing very good performances on particular indicators.

 U-Multirank compares institutions with similar activity profiles. If a ranking includes a wide
range of institutional profiles it has to ensure that it provides meaningful comparisons. U-
Multirank enables users to compare particular sorts of universities (“like-with-like”). It does
not make much sense to compare a small regional undergraduate teaching institution with an
internationally oriented research university, nor to compare an Arts Academy with a Technical
University. U-Multirank invites the user to first choose a number of empirical “mapping
indicators” to enable the comparison of institutions with similar profiles.

 U-Multirank does not provide over-simplified league tables. Institutions are ranked into five
different performance groups (A through E, with A indicating “very good” and E “weak”
performance) for each of the over 30 indicators in the ranking. (These groups are based on the
distance of the indicator score of an individual institution to the median performance of all
institutions that U-Multirank has data for.) While the league tables produced in other global
rankings may satisfy media needs for headlines, they tend to exaggerate differences in
performance between universities and provide a false impression of exactness. U-Multirank
therefore leaves it to the user to produce their own list of universities (or particular fields),
showing the performance on a selection of indicators made by the user.

 U-Multirank allows users to decide how to sort their results. The interactive web tool can sort
universities alphabetically or sort them based on the scores of a particular indicator. As an
extra service, users can follow an approach similar to the Olympic medal table, where
universities with the highest number of “very good” scores (“A” scores) would be shown first
in the table. If there are more than one with the same number of A scores, then the number
of B scores come into play, etc.

Executive Summary

| Page 9

Data Sources
The U-Multirank indicators are calculated from data drawn from three sources:







Self-reported Data. Universities that decided to participate in U-Multirank have provided data
for the institution as a whole, as well as for the departments offering degree programmes (if
any) related to the selected academic fields covered in the 2014 and 2015 editions of U-
Multirank. Both kinds of data are provided through online questionnaires. To ensure
comparability of data across institutions, the questionnaires include guidelines and definitions
of all data items requested. In a first phase, participating institutions provide their data using
the questionnaires. Data are then intensively checked by the U-Multirank team, applying both
automated and manual checks for consistency, plausibility (including checks of outliers) and
missing data. Questions and comments related to the data are communicated to the
institutions. These are followed up in a second phase of data provision where universities are
invited to clarify, correct and add data to the original questionnaire. After the final submission
of the questionnaires, data are checked again and any remaining questions are communicated
directly (by email) to the universities. Once all data submissions are finalised and the data is
regarded as valid and complete, the indicator scores are calculated.

Student Survey. One of the purposes of U-Multirank is to help prospective and increasingly
mobile students to make an informed choice about a university. For them the assessment of
the learning experience by current students of institutions provides a unique peer perspective.
The 2015 edition of U-Multirank includes data drawn from an online survey of more than
85,000 students from the seven fields included, asking students’ opinions about various
aspects of their degree programme. The online questionnaire is available in a number of
languages (English, French, German, Spanish, Italian, Polish and Russian). Invitations to
participate in the survey are sent out by the universities (either by email or ordinary mail) to
some 500 students per field. The indicators taken from the student survey reflect different
aspects of the students’ learning experience. They refer to a particular study programme and
hence are used only in field-based rankings.

Bibliometric and Patent Data. A number of U-Multirank institutional and field level indicators
are based on bibliometric and patent data included in international databases. All indicator
scores derived from bibliometric analysis are based on information extracted from
publications that are indexed in the Web of Science (WoS) database. In order to be able to
meaningfully calculate some of the bibliometric indicators, there is a threshold on the number
of publications per university (50 WoS publications over a four year period for the institution
as a whole; 20 WoS publications for individual fields.) The data underlying the patent-related
indicators is collected from the EPO Worldwide Patent Statistical Database (PATSTAT).

Indicators
U-Multirank makes use of three different types of indicators: Ranking indicators (institutional-level and
field-based), mapping indicators and descriptive indicators.



Institutional and Field-based Ranking Indicators. U-Multirank provides a multi-dimensional
ranking both on the institutional and the field level. The dimensions are teaching and learning,
international orientation and regional engagement. The
research, knowledge transfer,
performance of a university is measured across over 30 indicators. The choice of indicators
was based on the earlier U-Multirank feasibility study and adapted to include suggestions
made by various stakeholders and U-Multirank’s Advisory Board. The definitions of each

Executive Summary

| Page 10

indicator, along with the rationale for including them in U-Multirank, can be found in the
“Indicator book” on the U-Multirank website.

 On an institution’s profile page, an easy-to-view sunburst chart is available for users to see the
performance scores of the respective institution across the different indicators. The sunburst
is downloadable and can be used to give an at-a-glance picture of an institution’s performance
on the institutional level. An example can be found here: Sunburst with Indicator Names
 Mapping Indicators. As indicated earlier, U-Multirank is designed to enable users to compare
“like with like”. In rankings universities should only be compared when their purposes and
activity profiles are sufficiently similar. Universities with broadly similar profiles have to be
identified by the user, based on indicators of their particular characteristics and activities. This
“like-with-like” selection is based on “mapping indicators” such as size, scope, age, or activity.

 Descriptive Indicators. U-Multirank also provides some additional

information on the
university, such as its location, the number of academic publications it has produced (per full-
time equivalent academic staff) according to its own counts, the rate of unemployment of its
graduates (at Bachelor and/or Master level; 18 months after graduation), and the profiles of
degree programmes.

The U-Multirank web tool
The public face of U-Multirank is its interactive, user-driven web tool that has been developed in close
consultation with the European Commission (DG EAC), the U-Multirank Advisory Board and other
stakeholders, particularly students. See www.umultirank.org





An interactive, multi-dimensional and user-driven ranking system cannot be adequately
presented in a print publication. It was clear from the beginning that U-Multirank would need
to develop an innovative user-friendly web tool to present its results to a variety of
stakeholders with different information needs and with different levels of expertise in higher
education. A major challenge was to find a balance between presenting complex data in a way
that institutions are satisfied with how their performances are represented while at the same
time making the web tool easy-to-use for “lay” users such as prospective students (and their
parents) who are one of the major target groups of U-Multirank. Designing and developing this
web tool for the launch in May 2014 was one of the major challenges of the first project period.

The web tool provides four different ways to compare university performances. A general track
to compare institutions with similar profiles (“Compare”) can be entered either by specifying
a particular institution and comparing it to similar institutions or by selecting a number of
mapping indicators to identify a group of similar institutions to compare. Second, an easy-to-
use version for comparing institutions is offered particularly for student users (“For students”).
This track places special emphasis on teaching and learning indicators (the selection was
informed by student organisations) and, with regard to the target group of prospective
students who have to make a decision, offers a simpler mapping and quicker access to the
results. Third, the web tool offers a direct route to the detailed results of a particular institution
(“At a glance”). Fourth, a set of 17 “readymade rankings” are included on the web tool (3 in
the 2014 ranking and a further 14 in the 2015 ranking). These are rankings developed by the
U-Multirank consortium as examples of the web tool’s options and logic and the possibilities
it offers to users.

An overview of progress made in the first project phase in the four project work packages

| Page 11



A major feature of the web tool, reflecting U-Multirank’s basic philosophy, is the possibility for
users to produce personalised rankings through the “Compare” and “For Students” tracks. This
refers to the user-driven selection of institutional profiles and the option to freely select (out
of the full set of indicators) those indicators that are most relevant and interesting for an
individual user. This is a unique feature of U-Multirank compared to other international
rankings.

Other important activities





The recruitment phases and launch activities in 2014 and 2015 were based on a
communication strategy, which defines priority target groups and objectives as well as a set of
specific communication actions.

Two further major deliverables of the first project phase were the outline of a funding and
governance model for the sustainable continuation of U-Multirank after the implementation
project ends in March 2017 and a pilot study to test the feasibility of including non-university
research performing organisations in U-Multirank.

2

An overview of progress made in the first project phase in the four
project work packages

The implementation of U-Multirank as a very new type of ranking was structured in the
implementation project into four major work packages: the collection and analysis of data to produce
multi-dimensional university performance profiles (see section 2.1), the development of an interactive
web tool to make this comprehensive information accessible to a diverse set of users (2.2), the
development of a communication strategy to reach out to universities and potential users (2.3) and
the development of processes to continuously improve and refine the methods and instruments (2.4).
(This last work package also included the development of the Governance and Funding Model for U-
Multirank post-March 2017, and the pilot study on the inclusion of research performing organisations
other than universities.) The following sections provide an overview of the main achievements within
the four work packages. The focus is on the achievements of the first project period (December 2012
to March 2015) as a whole. From the beginning U-Multirank was responsive to input from
stakeholders. Throughout the full project period the development of U-Multirank has been
accompanied by an Advisory Board representing major stakeholder groups (among them HEIs,
students, employers). The ideas and feedback from the Advisory Board provided important input which
substantially helped to improve U-Multirank.

2.1 Work package A: Data collection
Work package A was primarily concerned with the development and operational production of two
rankings within the first project period. It drew on the outcomes of the U-Multirank feasibility study in
which the initial set of indicators, first versions of the data collections instruments and the processes
of data collection and analysis were developed and tested empirically in a pilot study.

Participation in U-Multirank 2014 and 2015

2.1.1
The Consortium successfully recruited the target numbers of institutions for active participation in U-
Multirank: In 2015 data on 1,210 institutions could be presented of whom 683 institutions (target

An overview of progress made in the first project phase in the four project work packages

| Page 12

number 650) provided institutional data across the range of U-Multirank indicators and 529
institutions were presented with bibliometric and patent data only. (The first edition in 2014 presented
529 actively participating institutions.) From 2014 to 2015 the percentage of institutions from outside
Europe has grown from 38% to 44% (and is expected to grow further in the 2016 edition).

To increase the scope of U-Multirank the Consortium with the support of the U-Multirank Advisory
Board and DG EAC decided to include a number of additional research-intensive universities on the
basis of available bibliometric (World of Science) and patent (PATSTAT) data – which cover 12 of the
U-Multirank indicators. These institutions were selected from the CWTS Leiden Ranking. In 2014 all
institutions in the top 500 of the Leiden Ranking were included; in 2015 all institutions in the top 750
of the Leiden Ranking were included. (In both years a significant number of these institutions were
already U-Multirank participants.) This created a more differentiated and comprehensive picture of
research performance than any other global ranking. With these additional institutions, the total
number of institutions included in U-Multirank was 863 in 2014 and 1,210 in 2015.

The data collection process

2.1.2
The set of U-Multirank indicators was developed in close consultation with stakeholders and tested
during the feasibility project. The “Indicator Book” published on the website makes all definitions,
rationales and operationalisations of the indicators transparent in line with the IREG “Berlin Principles”
on rankings. The methodology section of the web tool also offers a detailed description of the process
of data collection (including all questionnaires) and of the calculation of the five performance groups.

Currently U-Multirank provides a comprehensive set of 31 indicators on the institutional level; in field-
based rankings the number of indicators varies between fields from 36 to 38. U-Multirank provides its
multi-dimensional perspective on performance by using several data sources: institutional and field-
based data provided directly by participating institutions through online questionnaires, bibliometric
data from the Web of Science data base, patent data drawn from PATSTAT, and data based on U-
Multirank’s online student satisfaction surveys (for the field-based rankings only).

The processes and procedures of data collection through online questionnaires have been further
improved each year to enhance data quality as well as to reduce the data delivery burden on
institutions. Based on feedback provided by institutions during the 2014 ranking, some definitions
were adjusted and more detailed explanations were added where institutions from different countries
had different understandings of particular concepts. Furthermore, most of the questions in the original
2014 institutional questionnaire asked for answers relating to a single year, which led in some cases to
atypical results when the answers for that particular year were particularly high or low. In the 2015
version, most questions were adapted to a three-year period; three-year averages were then
calculated in order to have more stable and reliable data. In addition, questions regarding institutional
revenues were simplified and restructured in response to feedback from institutional contact persons.
This reduced the administrative burden for the institutions and enhanced the quality of the data
provided. Some first attempts to coordinate U-Multirank data collection with national data sources
were also realised: in Norway 2015 data were included from the national student survey; while
elements of the institutional and field-based data for Spanish institutions were taken from the national
Ministry of Education data base.

After two rounds of data collection there is a clear conclusion: U-Multirank is able to produce a set of
reliable and valid indicators in all five dimensions and on both levels (institutional and field-based),
based on optimised data collection instruments and processes. In some cases, the data sets of an

An overview of progress made in the first project phase in the four project work packages

| Page 13

institution are not fully complete, but for most indicators this is an effect of a specific data situation at
the institution (gaps in institutional data/accounting systems) or reflects a situation where the
indicator is not applicable to an institution (for example, the number of PhDs for institutions which do
not award PhDs). In doing so U-Multirank provides a transparent and honest picture of data; it does
not rely, as most other international rankings do to calculate their composite score, on estimates of
missing data. Only for two indicators (graduate employment rates; degree theses in co-operation with
regional enterprises) data were not available at a substantial number of institutions and hence omitted
from the ranking tables.1 There will be more fine-tuning of data collection instruments and indicators
in the second phase of the implementation project, but the overall approach will remain stable.

2.1.3 Data quality and quality assurance
Data verification and checks have a top priority in U-Multirank. The Consortium developed a set of
procedures to ensure data quality to the highest level possible. The process of data collection includes
two cycles of feedback and interaction with institutions on their data. The issues addressed in these
two cycles are identified by automated and manual checks of individual questionnaires as well as by
statistical data verification procedures. The latter include checks of plausibility and consistency of data
within individual questionnaires as well as analyses of outliers in the whole sample and in country
related subsets of institutions. The checks are based both on the raw data and on preliminary indicator
scores. In 2015, automated consistency and plausibility checks were built directly into the online
questionnaires which now give immediate comments to institutions that enter implausible or unlikely
data. Those checks both facilitated the data provision process for institutions and increased the quality
of data and the efficiency of data verification for the U-Multirank team.

2.1.4

U-Multirank results and achievements

U-Multirank succeeded in producing a comprehensive international data set on five core dimensions
of performance in higher education. With the aim of showing diversity in performance profiles and
excellence U-Multirank developed a methodological approach which is radically different from other
global rankings. The results of the first two publications have proven the necessity of a multi-
dimensional approach to rankings: Results make clear that there is no the best university. No
institutions perform top on all indicators. This means that results vary according to the selection of
indicators by users reflecting their personal ideas about the relevance of indicators. Hence U-Multirank
shows the performance profiles of higher education institutions with their particular strengths and
weaknesses. This cannot be adequately measured by composite scores.

While confirming many views on research excellence produced by other rankings U-Multirank as well
puts the spotlight on higher education institutions which have never been visible before in any other
global ranking, showing their strong performance in particular dimensions or indicators, including
research indicators. U-Multirank presents more than 30 institutions having more than ten A group
positions which have never been visible in other global rankings.

U-Multirank increased the stock of data on global higher education, provides general insights into the
global higher education system and provides meaningful data for institutional benchmarking and

1 Graduate employment is regarded as highly relevant by major stakeholders so has been retained as a descriptive indicator
in the “at a glance” profile. Only a few institutions were able to provide data on degree theses in cooperation with regional
enterprises (in field-based rankings) so this question has been omitted from the 2016 questionnaire.

An overview of progress made in the first project phase in the four project work packages

| Page 14

internal strategic management. An EUA study on the impact of rankings on higher education
institutions confirmed the set of indicators applied by U-Multirank: U-Multirank covers exactly those
indicators which are regarded as highly relevant for internal management by institutions and does not
take into account a number of indicators (such as reputation and Nobel Prizes) used in other rankings
which are regarded as less relevant for institutional management.2 On an international scope many of
them were collected and published for the first time by U-Multirank. In a second study by EUA after
the first publication of U-Multirank 60% of their member institutions participating in U-Multirank
reported that they are already using U-Multirank data for internal analysis.3

In addition to the comparison among higher education institutions U-Multirank provides general
insights into the global higher education system: For example although the growing importance of
interdisciplinary research is highlighted by many stakeholders U-Multirank data show that still most
universities are not very strong in interdisciplinary research. Only a few institutions are strong in
publishing interdisciplinary publications.

By offering a new approach to rankings U-Multirank has changed the world of rankings. Other global
rankings have begun to introduce elements of U-Multirank into their systems. Although keeping their
composite score approach in general they started to introduce interactive elements allowing users to
personalise the rankings (particularly in their mobile versions). Second, they included indicators
introduced by U-Multirank into their rankings (i.e. international co-publications). And, third, U-
Multirank has promoted a balance of institutional and field-based rankings.

2.2 Work package B: Web-tool implementation
An interactive, multi-dimensional and user-driven ranking system cannot be adequately presented in
a print publication. It was clear from the beginning that U-Multirank would need to develop an
innovative user-friendly web tool to present its results to a variety of stakeholders with different
information needs and with different levels of expertise in higher education. A major challenge was to
find a balance between presenting complex data in a way that institutions are satisfied with how their
performances are represented while at the same time making the web tool easy-to-use for “lay” users
such as prospective students (and their parents) who are one of the major target groups of U-
Multirank. Designing and developing this web tool in time for the launch in May 2014 was one of the
major challenges of the first project period.

The process of developing the web tool

2.2.1
The web tool was developed in close consultation with stakeholders, mainly higher education
institutions and students. A first “draft” version was tested by stakeholders and experts; based on their
feedback the final version was developed for the 2014 launch. The plan for 2015 was to consolidate
the first version of the web tool and to make only minor adjustments to optimise user-friendliness.

2 Ellen Hazelkorn, Tia Loukkola, Thérèse Zhang: Rankings in Institutional Strategies and Processes: Impact or Illusion. EUA,
Brussels, 2014
3 Tia Loukkola, Rita Morais: EUA Members’ Participation in U-Multirank: Experiences from the First Round. EUA, Brussels,
2015.

An overview of progress made in the first project phase in the four project work packages

| Page 15

Accordingly, some changes were introduced, such as reducing the number of mapping indicators and
making the regional filter an optional step in order to allow users to reach their result tables faster.

Evidence now shows that a substantial number of users try to access U-Multirank via mobile devices.
A mobile version will be needed to expand the use of the web tool, in particular by student users.

2.2.2 U-Multirank mapping indicators: Comparing like-with-like
U-Multirank follows a like-with-like approach to compare institutions with similar and hence
comparable institutional profiles. The web tool offers a facility for users to define these profiles based
on their own needs and interest. U-Multirank (“Compare like with like”) presents eight criteria to
identify subsets of comparable institutions which work as filters and determine the sample of
institutions to be included in the personalised ranking. A unique feature (“Compare a university”) was
introduced which allows institutions first to identify institutions with a similar profile and then to
compare their own performance with these institutions.

The U-Multirank ranking tool: Measuring performance

2.2.3
The web tool provides four different ways to compare university performances. A general track to
compare institutions with similar profiles (“Compare”) can be entered either by specifying a particular
institution (by name) and comparing it to similar institutions or by selecting a number of mapping
indicators to identify a group of similar institutions to compare. Second, an easy-to-use version for
comparing institutions is offered particularly for student users (“For students”). This track places
special emphasis on teaching and learning indicators (the selection of indicators presented in the
default table was informed by student organisations) and allows a quick access to results. Third, the
web tool offers a direct route to the detailed results of a particular institution (“At a glance”). Fourth,
a set of 17 “readymade rankings” are included on the web tool (3 in the 2014 ranking and a further 14
in the 2015 ranking). These are rankings developed by the U-Multirank consortium as examples of the
web tool’s options and logic and the possibilities it offers to users.

A major feature of the web tool, reflecting U-Multirank’s basic philosophy, is the possibility for users
to produce personalised rankings through the “Compare” and “For Students” tracks. This refers to the
user-driven selection of both institutional profiles and the option to freely select (out of the full set of
indicators) those indicators that are most relevant and interesting for an individual user. All result
tables can be sorted by the users in different ways: Either by alphabet, by individual indicators or by
an approach similar to the Olympic medal table, where universities with the highest number of “very
good” scores (“A” scores) would be shown first in the table. If there are more than one with the same
number of A scores, then the number of B scores come into play, etc. This fully interactive tool is a
unique feature of U-Multirank compared to other international rankings.

U-Multirank presents the richness of information on each institution (on the “at a glance” page) by
providing detailed information on individual indicators (score of the institution, performance group,
as well as the minimum and maximum scores and the score distribution across the full sample of
institutions). “At a glance” also contains some qualitative information on study programmes included
in the field-based data. In order to provide an integrated graphical view of the performance of an
institution a graphical illustration was developed: the sunburst chart. This is an example of U-Multirank
developing innovative and intuitive ways of presenting performance outcomes and profiles.

An overview of progress made in the first project phase in the four project work packages

| Page 16

2.3 Work package C: Communication strategy and plan
Successful communication is crucial to creating public interest in U-Multirank. Communication has to
both motivate potential users to access the web tool to support their decisions and to ensure high and
continuous levels of participation by institutions. This means that different target groups need to be
addressed by the communication strategy with specific activities targeted at them to translate the
strategy into action.

Communication strategy

2.3.1
The Consortium developed a comprehensive communication strategy, including three main objectives:
enhancing institutional participation; the propagation of the U-Multirank approach to ranking; and
increasing web traffic. To increase and maintain interest from the target groups identified in the
communication strategy, activities have been created that best align with the most appropriate
channels for reaching the different target groups.

The strategy uses a conceptual approach of focusing on key target groups, identifying key messages
for each and prioritising these messages, as well as an operational approach to the activities to be
undertaken. The key message for higher education institutions is “U-Multirank shows what a university
is best at and where it could improve” and the key message for students is “U-Multirank allows you to
compare universities on your own terms”. Each of these key messages is supported by sub-messages.
The messages determine the branding of U-Multirank, for example the design of the logo with the
tagline “Universities compared. Your way”.

The focus of communication activities in the first phase of the implementation project was primarily
on higher education institutions; the focus is being extended to students in the second phase. Beyond
the two main target groups there are secondary target groups such as governments, ranking experts
and commentators that will not be neglected but will be addressed less intensively.

A crucial milestone for communication every year is the launch of the new ranking. This is the moment
to reach several target groups at the same time. Many communication activities are centred on the
launch: the information packages supplied to participating institutions for national public relations
purposes; information given to national media partners about the performance of institutions in their
countries; (in 2015) publication of the launch video explaining the U-Multirank approach; intensified
social media traffic; etc. Aside from these intensive launch activities, communication takes place
continuously throughout the year to generate attention for the web tool.

Communication actions

2.3.2
The target group of higher education institutions was addressed through direct channels, based on a
comprehensive address data-base: a U-Multirank electronic newsletter appearing at least twice a year
has been introduced. Social media communication (Facebook: more than 1,500 likes, Twitter: more
than 2,000 followers, LinkedIn) and appearances at various conferences (such as the IREG-7
Conference in London (2014), the IREG Forum on field-based rankings in Denmark (2015), the AIR
Forum and CHEA Conference in the US (2015), the Impact of Science Conference in Amsterdam (2015))
have also brought the project team into direct communication with institutions. U-Multirank has
received many positive responses from the higher education community; from these responses a
collection of testimonial citations representing the key messages has been assembled and is used for
communication purposes.

An overview of progress made in the first project phase in the four project work packages

| Page 17

The target group of students is more difficult to reach and is unstable as the group of people seeking
a place to study changes every year. There are few options to reach them directly, but it is crucial that
they discover U-Multirank and that advisers to (prospective) students and previous users spread the
message of U-Multirank. Various activities to this end have been initiated and will be intensified in the
second project period, for example: search engine optimisation (web links with key terminology that
is search engine friendly; writing and posting search engine friendly content, for example through the
newsletter and press sections; encouraging the use of the U-Multirank sunburst by institutions on their
websites with backlinks to their profile page, etc.); addressing student advisers; and mobilising the
ERASMUS networks. Reaching (potential) students is the only way to secure a large number of users
of the web tool.

Some activities have already demonstrated their potential to address multiple target groups, especially
working with media organisations from all over the world for the annual ranking launch. The vast
majority of articles published at the launch helped to spread the word about an alternative new ranking
tool typically by reporting on the results from a local or national perspective and highlighting the
performance of selected universities. Many articles were explicitly positive about the new approach to
international ranking and encouraged participation from more universities and the further
development of this new approach. Only a very few articles included negative comments about U-
Multirank. The packages provided to participating institutions (mentioned above) were particularly
helpful in stimulating this media coverage.

Readymade rankings

2.3.3
The Consortium introduced the idea of “authoritative” rankings in its tender document. Alongside the
capacity in the web tool for users to develop their own rankings, the U-Multirank team would develop
a set of its own pre-defined rankings focused on carefully selected institutional profiles and using a
relevant set of indicators. These rankings are now titled “readymade rankings”. They illustrate the logic
of U-Multirank by linking a particular set of ranking indicators to a particular profile of institutions. At
the same time they are an important element of the U-Multirank communication strategy as they
provide particular stories and draw media attention.

At the launch of U-Multirank in May 2014 three readymade rankings were included in the web-tool:
rankings on research and research linkages and on economic involvement both on the institutional
level, and a field-based ranking of institutions that offer Masters programmes in Business Studies
addressing teaching and learning.

In February 2015 five new readymade rankings were added: an institutional and four field-based
rankings of PhD awarding universities that have medium to high percentages of international degree
seeking students focusing on their international orientation.

When the second edition of U-Multirank was released in March 2015, it included the following 17
readymade rankings: updates of the 8 readymade rankings listed above, the extension of the teaching
and learning ranking from Business Studies to all seven fields included in the second edition, and the
extension of the international orientation ranking to the three new fields introduced in 2015.

An overview of progress made in the first project phase in the four project work packages

| Page 18

2.4 Work package D: Further refinement

Continuous refinement

2.4.1
Like any ranking system U-Multirank has to be further enhanced and improved over time while keeping
the basic approach and methodology as stable as possible. A major input into developing the web tool
further is feedback from participating institutions as well as from users of the web tool. Both inputs
led to adaptations from the first to the second release of U-Multirank which were described earlier
(see section 2.1.3).

In line with the introduction of new fields, field-specific indicators have been introduced based on an
assessment of their feasibility. Examples can be found in the 2015 publication of the medicine ranking,
in which specific indicators based both on institutional questionnaires and on the student survey were
developed (in medicine for example, on innovative forms of assessment and on the quality of 'bedside
teaching' from the student perspective).

Incorporation of new fields

2.4.2
Field-based rankings are highly relevant for many users of U-Multirank, especially for students who
wish to compare universities in the field they intend to study or for academics interested in
comparisons with other departments or programmes in their field. The inclusion of a large number of
fields is an important criterion for the success of U-Multirank. U-Multirank started with four fields in
2014; three new fields (psychology, computer science and medicine) were added in 2015. The initial
plan was to include four new fields in 2016 but in order to increase the benefit of U-Multirank to users,
six additional fields will be added, extending the scope in the sciences (by including mathematics,
chemistry and biology) and including for the first time fields from the social sciences and humanities
(sociology, history and social work). In 2017, the four 2014 fields will be updated for the first time and
more new fields will be added.

Funding and governance model

2.4.3
The Consortium produced its final report on a U-Multirank Governance and Funding Model in February
2015. This report was endorsed by the U-Multirank Advisory Board and accepted by DG EAC.

The report outlines a number of cost scenarios for U-Multirank post-2017 and then discusses potential
funding sources in terms of their advantages/disadvantages and possible risks. It draws conclusions on
the sustainability of the different potential sources of funding both individually and in combination.
The three cost scenarios depend primarily on the number of participating institutions in the period
2017 - 2021.

Four major potential funding sources are considered: commercial funding (selling data, information,
advertising and publication rights); public funding (governments, supranational organisations);
participant funding (from all participants and/or for premium benefits); and philanthropic funding
(foundations, companies, non-profit institutions). The report concludes that commercial and
participant funding would only make a small contribution; while public funding could play a supporting
role to philanthropic funding which is identified as the most promising funding source. A mixed funding
model is advocated based on these conclusions, leading to a non-profit, open access approach for U-
Multirank instead of commercialisation of the tool.

The U-Multirank Funding and Governance report envisages certain governance features such as a U-
Multirank Board (key funders and independent public figures/experts); a Stakeholder Advisory Board

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 19

and an Expert Advisory Board; and a U-Multirank Office to run the ranking as a small office with data
analysts as well as ranking experts. To avoid a hiatus in the transitional period the report recommends
establishing basic elements of the new structure before the end of the current project funding.

Pilot study on Research Performing Organisations (RPOs)

2.4.4
Major international rankings (as well as most national rankings) include higher education institutions
only and do not cover non-university research performing organisations (RPOs). A pilot study was
undertaken to explore the feasibility of including RPOs into U-Multirank.

The study was guided by two major questions: First, could RPOs be included in U-Multirank, either
being compared only amongst themselves or also being compared with higher education institutions
(within their specific range of activities which would exclude teaching in most cases)? Second, could
U-Multirank measure the contribution that RPOs make to the research capacity of higher education
institutions through various forms of cooperation?

A feasibility survey among RPOs explored their views on these questions and on the availability of data.
The low response rate in the survey (ten out of 40 organisations answered) indicates a low interest in
U-Multirank and in rankings in general on the part of RPOs. Based on the limited number of responses
the main conclusions are:



There is a strong preference for comparisons among RPOs only (and not with higher education
institutions),

Any inclusion of RPOs would need additional mapping indicators to identify comparable RPOs,
 U-Multirank indicators in the dimensions applicable to RPOs – primarily research but also
aspects of knowledge transfer and international orientation - are seen as relevant and feasible,
 New indicators to measure the cooperation of RPOs with higher education institutions are also

feasible (for example, joint appointments and publications).

Based on these conclusions, in particular the low level of interest and the strong preference for
comparisons among RPOs only, it was decided (with endorsement by the Advisory Board) not to
include RPOs in U-Multirank as part of the implementation project.

3

Conclusions, challenges and key recommendations for the second
phase of the implementation project

The activities and achievements of the first two years of U-Multirank implementation support the
following broad conclusions:



A new global, multi-dimensional university ranking has proved to be feasible and has been
successfully implemented.
The necessary operational instruments and processes are in place and work.


 U-Multirank has demonstrated that it can show what it was expected to show: the diversity of
missions and profiles in higher education and the wide variety of strong performances across
the different dimensions and indicators of performance.
Rigorous quality assurance and data verification procedures have produced reliable and valid
information.



Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 20







Institutional participation has developed in line with the implementation plan and associated
targets although not uniformly in terms of countries and regions (see section 3.1 below).
Acceptance by participating institutions and users has largely been achieved and the feedback
received is dominantly positive.
The U-Multirank web tool is innovative, user friendly and technically sophisticated. In the
second project phase it will be further enhanced in response to user suggestions and
preferences.

The experiences from the first two releases of U-Multirank and feedback from participating
institutions, users of the web tool and stakeholders also, however, indicate a number of necessary
developments for the next phase of U-Multirank. In its key recommendations for the future (in the
blue boxes that follow) the Consortium indicates how challenges will be addressed.

Participation and further extension

3.1
The U-Multirank consortium succeeded in recruiting more than the target numbers of actively
participating institutions in the first two years and included additional research universities (based on
the CWTS Leiden Ranking) and U-Multirank is now the most comprehensive international university
ranking in terms of the numbers of institutions included in the web tool. However participation can be
improved in a number of respects.
The illustration below shows that in 2015 coverage among public universities (not including art schools)
is already very high in many European countries.

Illustration 1: U-Multirank coverage among European (public) universities (% of HEIs participating per country)

Cyprus
Luxemburg
Malta
Norway
Slovenia
Portugal
Latvia
Switzerland
The Netherlands
Austria
Finland
Hungary
Estonia
Spain
Belgium
Czech Republic
Denmark
France
Poland
Lithuania
Slovakia
Ireland
Germany
Romania
Greece
Italy
Sweden
Croatia
Serbia
Turkey
Bulgaria
United Kingdom

0,0

20,0

40,0

60,0

80,0

100,0

 Within Europe recruitment can be improved particularly in the UK; recruitment activities in



Italy, Sweden, Croatia, Serbia, Turkey and Bulgaria should also be intensified.
In 2015, about 12% of participating institutions were Universities of Applied Sciences (UAS).
Although U-Multirank results show that U-Multirank is an effective tool to make visible the

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 21





particular performance profile and strengths of UAS some of them still have to be convinced
that engaging in data collection pays off (as UAS often have less elaborate data systems).
In a number of large higher education systems outside Europe participation should be
extended in order to increase global coverage. In particular, participation of US universities is
still low despite a number of initiatives taken by the consortium in the first project phase. The
consortium hopes to overcome low participation in China through cooperation with the
Chinese education authorities. The plan is to agree on a pilot sample of Chinese higher
education institutions to participate in U-Multirank (and to harmonise national data bases and
U-Multirank at the same time). During the first project phase the consortium has also taken
steps to increase participation in Japan, Latin America and Australia.
Participation in one year does not necessarily lead to continuous participation of institutions
in the long run. In their decision making about continuous participation institutions weigh up
the costs of participation (the work entailed in data collection) against the benefits they
receive from participation. Over the first two editions of the ranking the number of institutions
withdrawing from U-Multirank has however been very low (16 institutions). Discussions with
participating institutions revealed the following crucial factors determining institutional
benefits:

o Visibility of their institution in U-Multirank as an instrument for international student

recruitment.

o The benefits of being visible in U-Multirank will rise as user (particularly student)

numbers of the web tool increase.

o The general visibility, reputation and transparency of their university profile.
o The potential to use U-Multirank for internal management (strategy development,

quality management, etc.).

In addition intense communication with institutions being responsive to their needs
throughout the whole process of data collection will continue to be an important factor for
ongoing participation.

Key recommendations for the second phase of the implementation project:

 While continuing efforts to recruit universities from Europe, the focus of targeted
recruitment activities should be on Universities of Applied Science (UAS) and on
institutions from large higher education systems outside Europe (particularly US, Japan,
China, Australia, and Canada). Within Europe, participation of more universities from the
UK would be valuable. The communication with these institutions and countries should
address their interests in a tailor-made way.
The provision of services to the participants should be led by considerations of the cost-
benefit-analysis institutions make; U-Multirank has to continuously provide benefits. To
get a clear insight into the interests of the participants, especially in terms of using U-
Multirank for institutional management, there should be a continuous exchange with
institutions. A network of institutional data coordinators, providing feedback to U-
Multirank about institutional needs, could be an important instrument to increase the
benefits for institutions.



Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 22

3.2 Data and data quality
The continual enhancement of processes is a challenge for all large-scale data collections. The major
aims are to optimise data quality and, at the same time, to reduce the burden on institutions to provide
the data.
Two reports by the European University Association (EUA)4 showed that U-Multirank covers exactly
the indicators which are seen as highly relevant by institutions for their strategic management, yet, at
the same time, revealed problems in providing data on those indicators. The EUA survey among their
member universities about their experience with U-Multirank confirmed that the provision of data for
U-Multirank can be rather time consuming for some institutions while other institutions reported no
problems in providing data as they collect most of the data anyway, either for their internal steering
and quality management or for other external requests (e.g., evaluations, accreditation).

Based on the experiences of 2014 and 2015 the institutional and field-based questionnaires for the
2016 edition have both been shortened. As already indicated one indicator which turned out to be not
feasible (degree theses in cooperation with regional enterprises) has been excluded. In addition, all
data points which are not absolutely necessary to calculate indicators have been dropped from the
questionnaires.

The consortium will continue to explore options to use data from other (verified) data sources – both
from international and national data sources as well as from national student surveys. Inter alia, the
option of using publicly available data, such as the ETER database, will be explored.

Key recommendations for the second phase of the implementation project:

 U-Multirank should continue trying to reduce the burden on institutions in terms of data
delivery. Instruments should be reviewed annually; the necessity of all data points in the
questionnaires should be reconsidered every year, and the further development of data
definitions based on institutional questions is crucial. The implementation of a network of
institutional coordinators would help to stimulate exchanges of experience in data
provision in order to make the process more efficient for institutions.
In this context, options to use data from international and national data sets and from
national student surveys (either as a tool to verify data or to prefill questionnaires) should
be explored further.



3.3

Further development of indicators

Difficult indicators

3.3.1
In general, the U-Multirank indicators, which were tested in the feasibility study, proved to be feasible.
Yet there are a few indicators where there are challenges regarding availability and reliability of data.
Out of the five dimensions “regional engagement” is the most difficult – which is not surprising as U-
Multirank is the pioneer in global data collection in terms of higher education information at the
level. Some institutions are not able to disaggregate data by region (for example, on
regional

4 Ellen Hazelkorn, Tia Loukkola, Thérèse Zhang: Rankings in Institutional Strategies and Processes: Impact or Illusion. EUA,
Brussels, 2014. Tia Loukkola, Rita Morais: EUA Members’ Participation in U-Multirank: Experiences from the First Round. EUA,
Brussels, 2015.

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 23

graduates, internships and degree theses in cooperation with industry). Within Europe, the NUTS
(Nomenclature of Territorial Units for Statistics) classification of regions has been applied in data
collection, but outside Europe there is no equivalent coherent definition of regions. In some cases the
NUTS approach is also less than perfect, for example for a university lying on the border of a NUTS
region with regional cooperation partners just across the border.

As indicated earlier, the most problematic indicator – which is regarded as highly relevant by
stakeholders – is graduate employment. The difficulty of obtaining reliable and internationally
comparable data was already evident in the feasibility study, but given its high relevance for
stakeholders the indicator was retained. The available data does not allow a ranking indicator to be
calculated; instead this data for a limited number of institutions is presented as a descriptive indicator.
Many institutions and countries still do not have data on the labour market entry and employment of
their graduates. Some institutions have data based on their own surveys and in a number of countries
(including UK, Switzerland, and Australia) there are data from national surveys on graduate
employment. But in most cases these data are not comparable in terms of the definition of
employment, the disaggregation by fields and degree levels, and, most relevantly, in terms of the time
of measurement (months after graduation). These problems cannot be solved by U-Multirank – an
international/European initiative would be necessary to agree on standard definitions on graduate
employment, and this would benefit higher education in general in addition to U-Multirank.

New dimensions and indicators

3.3.2
One criterion for good ranking quality is the stability of the methodology. Yet rankings have to adapt
to changes in the higher education sector and should be prepared to widen their set of indicators. It is
also clear that different indicators are more or less significant for specific institutional profiles. The
scope of indicators should take into account the range of institutional profiles and the variety of users’
needs. U-Multirank has gone further down this road than any other international comparative higher
education data system, but there are still more possibilities (which need to be prioritised in order to
limit the increase in further data collection work).

There are two possibilities that seem to be worthwhile to explore in the second project phase: first,
one reason for the low participation of Universities of Applied Science is the lack of an indicator on
applied research, which would be able to measure and present their particular research activities and
profile within the research dimension (currently this is only possible in research-related indicators in
other dimensions, such as co-publications with industry in the knowledge transfer dimension). Second,
some stakeholders (including student organisations) have proposed the inclusion of indicators on
students' social conditions s in U-Multirank. This may be an element to promote the attractiveness of
U-Multirank to student users.

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 24

The inclusion of new fields will also require the introduction of new field-specific indicators, in
particular with regard to research and research equivalent outputs in the arts and humanities.

Key recommendations for the second phase of the implementation project:



 U-Multirank has to stay responsive to stakeholders’ needs. This requires the development
of new indicators, which may be in tension with the objective to decrease the burden of
data collection. U-Multirank should test the feasibility of new indicators, but only
introduce these if the reliability and availability of data are guaranteed. A network of
institutional data coordinators would be a good mechanism to evaluate the feasibility of
potential new indicators.
The selection of new indicators should always be assessed against their relevance for the
recruitment of institutions with specific profiles and their importance for different user
groups. Two new indicators should be explored: the development of a valid and reliable
indicator on applied research for 2016 and one or more on students' social conditions for
2017.
Information and indicators on graduate employment are regarded as highly relevant by
most stakeholders. The harmonisation of available data, including issues of comparability
of existing national surveys, are beyond the scope of the U-Multirank Consortium. Here a
European/international political
initiative to harmonise data is needed. Graduate
employment will continue to be included as a descriptive indicator.



3.4 Web tool
Empirical evidence shows that mobile use of websites in general and by young people in particular has
grown rapidly. This was not foreseen when U-Multirank was designed. The development of a mobile
version of the web tool which is optimised for smartphone and tablet use is highly relevant for U-
Multirank. Compared to the current web tool a mobile version needs to include ways of presenting
results and of guiding users that are specifically designed for mobile devices and their smaller screens.
In general feedback on the U-Multirank web-tool has been positive. The web tool is unique when
compared to those of other global rankings (although they have implemented some elements similar
to U-Multirank). Yet finding a balance between providing comprehensive, detailed information in an
interactive way and keeping it simple and user-friendly is a continuous task. In terms of user-
friendliness, the web tool has to be stable in terms of its basic functionality and design yet it still needs
to introduce innovations and new features to maintain its attractiveness to users.

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 25

Key recommendations for the second phase of the implementation project:



The visibility and usability of the U-Multirank web tool would be further enhanced by the
development of a mobile version. Usage data show that a substantial proportion of current
usage is via mobile devices. While the web tool works well on tablets, a separate version
optimised for smartphone and small tablet use and a mobile application should be
developed.

 U-Multirank should continue to explore ways to optimise the user-friendliness of the web
tool. Ways to make the student track easier to use and more tailor-made (e.g. by linking
the selection of default indicators to users’ mapping choices) should be explored.
Continuous dialogue with users about their needs and user tracking should determine
future enhancements to the web tool.

Communication

3.5
U-Multirank entered the world of international university rankings where three different rankings
already had an established presence. It is regarded as a newcomer, both in terms of its first publication
and its unique, multi-dimensional methodology. From the beginning, the Consortium was aware of the
fact that the U-Multirank approach, which does not present a composite overall score nor a league
table, is probably more complex and less headline-grabbing for the media. For this reason, U-Multirank
developed a communication strategy aimed at its major target groups (students and institutions) and
stressing its particular approach to the comparison of university performance which is regarded as
superior by many higher education stakeholders.

In the first project period after consultation with the Advisory Board the focus of communication
activities was on higher education institutions in order to achieve acceptance within higher education
and to encourage institutions to participate in U-Multirank. This has been largely successful in most
European countries and some countries outside Europe. There are indications that decision-makers
within institutions have started to make intensive use of the web tool. The higher education
community increasingly accepts the approach and tools of U-Multirank.

A major objective for the second project period is to further increase web traffic, especially throughout
the year with more frequent peaks. Being able to demonstrate to institutions that U-Multirank has
many users, particularly students, is also an important criterion for institutions in deciding whether or
not to participate. People looking for support in higher education decision-making should find U-
Multirank easily, hence one focus of future activities will be on search engine optimisation.

A major next step is to shift communication activities to the second phase of the communication
strategy: targeting students and the broader public, without neglecting higher education institutions
and the higher education sector. (Prospective) students are a continually changing community that
requires a different approach compared to the higher education sector.

Conclusions, challenges and key recommendations for the second phase of the implementation project| Page 26

Key recommendations for the second phase of the implementation project:



The efforts to improve search engine optimisation should be intensified as this is crucial
for U-Multirank to be easily found by potential users.

 U-Multirank needs communication throughout

the year to attract users and to
demonstrate the potential of the ranking. U-Multirank offers a wealth of data with the
potential to generate public interest. This potential should harnessed to generate news
stories.

 National media partnerships should be strengthened, for example by providing “national
stories” particularly in countries where U-Multirank has a high coverage of institutions.
The second phase of the communication strategy with an increased focus on students as
a major target group should now be implemented.



3.6 Governance and Funding Model
As indicated earlier in this report the consortium completed the development of the governance and
funding model in the first phase of the project and this was endorsed by the Advisory Board and
accepted by DG EAC. In the second phase of the project the key challenges will be to reach basic
agreement on the implementation of this model between DG EAC and organisations with philanthropic
motives that could be the potential funders of U-Multirank post-March 2017. The first step in this
process is a formal commitment of the first funding organisations as a basis for recruiting other
funders; and a first meeting between DG EAC and the other funding organisations. Thereafter, the
features of the model to generate some market income need to be designed and the basic structures
of the new organisation need to be established in the second project period to avoid a break in 2017.

Key recommendations for the future:



To explore with a range of different partners the feasibility of taking forward the
recommendations on a future governance and funding model.

HOW TO OBTAIN EU PUBLICATIONS

Free publications:

• one copy:

via EU Bookshop (http://bookshop.europa.eu);

• more than one copy or posters/maps:

from the European Union’s representations (http://ec.europa.eu/represent_en.htm);
from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*)The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications:

• via EU Bookshop (http://bookshop.europa.eu).

N
C
-
0
4
-
1
5
-
7
7
2
-
E
N
N

-

ISBN 978-92-79-52310-6