University Innovation
Indicators
Final report from NTNUs pilot project

Aristidis Kaloudis, Zacharias Andreadakis, Kjell Olav Skjølsvik og Ragnhild Nisja

Norg
natur

ISBN 978-82-693180-2-9 (Print)
ISBN 978-82-693180-3-6 (E-pub)Graphic design og production: NTNU Grafisk senterContent
Preface.................................................................................................................................................4
Part I - The new set of innovation indicators for HEIs............................................................................................................................. 6
Objectives and scope of the pilot project............................................................................................................................................................... 7
Definitions of Innovation .............................................................................................................................................................................................................. 8
Innovation in other societal sectors – public and civic sectors................................................................................................ 9
The University innovation eco-system concept.......................................................................................................................................... 10
What to measure – structural versus procedural indicators................................................................................................... 11
Demarcating indicators for innovation activities in HEIs.............................................................................................................. 12
Overlapping with research indicators......................................................................................... 12
Overlapping with education indicators....................................................................................... 13
HEI-innovation is a broader phenomenon than commercialization of research results..... 14
Benefits and the potential purporting use of the indicators ................................................................................................. 15
Cost-Benefit Analysis assessments of Indicators: Criteria.......................................................................................................... 15
Selection strategies for the production of indicator data in the pilot........................................................................ 16
The pilot set of innovation indicators....................................................................................................................................................................... 17
Detailed Description of the final set of Innovation Indicators..................................................... 19
Indicators that have been considered, produced but discarded from the “canon”
of HEI innovation indicators ........................................................................................................... 23
Other possible explorative innovation indicators to explore in the future............................ 25
Final remarks – part 1 .................................................................................................................................................................................................................... 26
References – part I............................................................................................................................................................................................................................... 27
Part II – Summary of the more detailed reporting on the collected indicators............................................ 32
Recommendations............................................................................................................................................................................................................................... 33
Part III – Report on Centers for Research-Based Innovation for Innovation (SFI) – ananalysis of 14 selected annual reports on contributions to Innovation.................................................................. 35
Rationale of annual report analyses.......................................................................................................................................................................... 38
Methodology................................................................................................................................................................................................................................................ 39
Innovation Dimension 1: Economic Indicators............................................................................................................................................ 41
Innovation Dimension 2: People and Work life.......................................................................................................................................... 42
Innovation dimension 3: Networks ............................................................................................................................................................................ 44
Innovation Dimension 4: Commercialization................................................................................................................................................ 48
Summary of findings and final reflections......................................................................................................................................................... 49
References – Part 3............................................................................................................................................................................................................................. 54
SFI Annual Reports............................................................................................................................................................................................................................... 59
Annex 1 - Complete Version of Recommended Indicators for NTNU in the
NTNU-report 2019a (English and Norwegian Version: 45 Indicators).......................................................................... 60

3

Preface
NTNU’s comments on the report “University Innovation Indicators – Final reportfrom NTNUs pilot project”.
This report is a direct response to the challenge posed by the Norwegian Ministryof Education and Research to NTNU. The challenge is about the need to produce
indicators capturing a broader array of innovation channels and impacts from theNorwegian higher education institutions (HEIs) than just indicators on commercialization of R&D. To address this challenge, I commissioned a research project withthe goal to develop suitable innovation indicators for HEIs with NTNU as testing
bed. The pilot project recommends 15 indicators, 2 indicators on innovation-oriented external funding sources, five indicators on various types of people interactions, three indicators on innovation networks and five indicators on academic andstudent entrepreneurship.
The indicator challenge from the Ministry echoes the growing societal expectations
that Norwegian HEIs must improve the profiling of their innovation impacts and inthat respect, perhaps, they should develop more strategic approaches to innovation as well as more flexible career paths and incentives to better balance innovation and academic careers in the future. The report emphasizes the importance of
producing quantitative indicators as one method needed to coordinate available
funding sources, people and networks into a dynamic innovation ecosystem.
HEIs have also a key role to play in the restructuring process of the Norwegian
economy away from oil and gas and towards a more sustainable development.
There is ample scholarly evidence that these transitions can only be developed in
an open and interactive innovation ecosystem where funding sources, people andnetworks are the pivotal raw ingredients. New knowledge and skills, the ability todevelop and exploit new technologies, as well as understanding how technology
and society interact, are all critical success factors that universities contribute to in
many different modalities and through different channels. Academic entrepreneurship and university spin-offs are of course important innovation channels, but they
only constitute a smaller part of universities´ overall innovation contribution tosociety. The pilot project was challenged to assess the feasibility of applying a morediversified set of indicators showing a wider university contribution to innovation.
An important insight from the pilot project is that specific initiatives and activities
for promoting innovation and value creation adopted by one department within
NTNU are not necessarily suitable for other departments or even other universities. Norwegian universities can certainly learn from each other, but the important
point is that each institution, faculty and faculty department ought to develop theirown innovation strategies on the basis of the strengths and opportunities in theirinnovation profiles. The analysis of the selected innovation indicators clearly showsa large variety of indicator profiles between departments within NTNU, even between departments within the same faculty at NTNU.
The report also points to considerable methodological challenges in measuring
innovation in the humanities and social sciences. There are several examples inwhich contributions from the humanities and social sciences led to clearly improved
public sector products and services. Yet, such contributions are difficult to capture4

through quantitative indicators. This is an argument for more research on this areaand for documentation by the means of narratives, i.e., innovation impact cases.
Student-led entrepreneurship is also not well-captured in the indicators of this pilot
project and yet we have an ambition to better measure and document impacts ofstudent-entrepreneurship activities in the future, such as NTNU ideas, patents, andspin-offs stemming from NTNU Discovery and SPARK*.
NTNU is experiencing an increasing demand for R&D collaboration from both theprivate and public sectors. In particular, university-industry research centers, suchas the Center for Research-based Innovation (SFI) and the Centers for Environmentfriendly Energy Research (FME) develop long-term relationships and deliver new
knowledge, prototypes, new methods, new technologies and innovation solutions
to their partners and to the broader society. In these centers, students, academics,
firms and end-users all collaborate to solve specific applied research problems.
And yet, one of the important findings in this report is that this rich array of innovation activities and impacts is not always documented in its full scale and scope in
the annual reports from the SFIs.
In conclusion, we must learn to better document and communicate how universities create value in our societies. This is already a prerequisite for obtaining research funding, both within the EU’s Research and Innovation programs and within
national research and innovation policy instruments. The ability to systematicallythink and coordinate actors, activities and resources for enhanced innovation impact is a key skill for the future HEIs and their departments.
Trondheim, Desember 2022

Toril Nagelhus Hernes, Pro-Rector for Innovation, NTNU

5Part IThe new setof innovation
indicators for HEIs

6

Objectives and scope of the pilot project
In recent decades, the scope of innovation in contemporary higher education has
undergone a surfeit of unforeseen changes. These changes, occasionally subsumed
under the label of “open innovation”,1 have garnered significant scholarly criticism
regarding both the conceptual clarity and the practical shapes of the agenda ofinnovation. Challenges to the innovation measures proliferate: theoretical imprecision, implementation deficiencies, and lack of comparable, replicable, and reliabledata on the field of university-driven innovation2. And, to further compound the discontents surrounding innovation, the problem is occasionally argued to run much
deeper, to the very way we understand and measure the effort towards innovation.
The tenets and practical challenges that surround particularly the measurement of
university-driven innovation have not escaped scholarly scrutiny across disciplinary
bounds. In fact, much scholarship now concedes that perhaps the thorniest, yetprevailing challenges is the absence of indicators that systematically capture impact of innovation originating from the Higher Education Institutions (HEI), beyond
standard scientific output (i.e., numbers of publications and citations) and specificcommercialization data. Scholarly inquiry, spanning from traditional research organizations such as the OECD (2019) to newest scholarly attempts from European
University Association (EUA 2018), has repeatedly drawn attention to the difficulties
surrounding both the quantification and comparability of innovation efforts in theuniversity sector, as well as the ability of existing innovation metrics to factor in theless directly observable, qualitative cases of innovation and their longer-term societal ramifications in their impact.
In response to this set of challenges a recent study (NTNU 2019) addressed this issue
and provided a set of 45 academia related innovation indicators (see Annex 2).
This work constitutes a starting point for this pilot project to select and produce amore contained number of indicators that - individually and taken together - portraya fair and broader picture of HEIs contributions to innovation.
The rationale of innovation indicator efforts is divided between two logics. The firstcategory of indicators dubbed as indirect contributions (activities and processes),
casts light on the skills, cultures, and interactions that occur in the areas of innovation. These ideas span from the integration of innovation in teaching and thegeneration of new ideas to the collaboration, networking and mobility outside thewalls of academia. Conversely, the second category of indicators entitled as directcontributions (impacts) seeks to capture the more standardized aspects of innovation, often understood as the commercialization outputs and the tangible improvements of industrial activities and performance.
The pilot project adopted the classification of indicators in NTNU 2019, refined it
and on this basis, suggests 15 indicators proposed as a permanent set of innovation indicator reporting in the future.
The theoretical perspective behind these 15 indicators is based on the idea of theuniversity innovation ecosystem where many types of resources (people, funding,
networking) interact in an open and dynamic environment that may or may not be
consciously governed and shaped by the HEIs themselves.

1

Chesbrough, 2003; Dahlander & Gann 2010; Markman 2016.

2

Gault 2012.

7

Definitions of Innovation
In order for a new idea, model, method or prototype to be considered as “innovation”, it needs to be economically and/or societally useful. This mandatory utility
aspect requires organisations to make systematic efforts to ensure that the innovation is accessible to potential users, either for the organisation’s own processes
and procedures, or to external users for its products.
At a minimum, innovations must contain characteristics that were not previously
made available by the relevant organisation to its users. These features may or
may not be new to the economy, society, or a particular market. An innovation canbe based on products and processes that were already in use in other contexts,
for instance in other geographical or product markets. In this case, the innovation
represents an example of incremental adjustments and diffusion to new markets.
Higher Education Institutions (HEIs) contribute to a large extend to the extensionof our cultural and knowledge frontiers, to new ideas, to the development of newtechnologies, inventions and prototypes by themselves or in collaboration with theindustry. The key question is, however, how exactly the HEIs contribute to the innovation processes in the business sector, in the public sector and in society in general.
In general, the term “innovation activity” refers to the process leading to innovations,
i.e., innovation processes, while the term “innovation” is limited to outcomes. Thisis an important distinction throughout this report and in the next paragraph we
present how the Oslo manual (OECD 2018) defines key concepts, such as “innovation activity”. The Oslo manual is the internationally established methodology
for collecting and using innovation statistics in the business sector worldwide. It
includes definitions of basic concepts, data collection guidelines, and classifications
for compiling innovation statistics:
Innovation activity includes all developmental, financial and commercial activities, undertaken by a firm, that are intended to or result in an innovation for thefirm.
A business innovation is a new or improved product or business process, or combination thereof, that differs significantly from the firm’s previous products or business processes and that has been introduced on the market or brought into useby the firm.
A product innovation is a new or improved good or service that differs significantly from the firm’s previous goods or services and that has been introduced on
the market.
A business process innovation is a new or improved business process for one ormore business functions that differs significantly from the firm’s previous businessprocesses and that has been brought into use in the firm.
There is also a considerable interest in business model innovations. The Oslo
Manual 2018, though avoiding metrics for this type of innovation, distinguishes between three types of comprehensive business model innovations in existing firms:• A firm extends its business to include completely new types of products andmarkets that require new business processes to deliver.
• A firm ceases its previous activities and enters into new types of products andmarkets that require new business processes.

8• A firm changes the business model for its existing products, for example itswitches to a digital model with new business processes for production and delivery and the product changes from a tangible good to an information good.
Comprehensive business model innovations are in the last years of great interest
because they can have substantial effects on supply chains and economic production, transforming markets and potentially creating new ones. They can influence
how a firm creates utility for users (product innovation) and how products are produced, brought to market, or priced (business process innovations).
Measuring the economic value creation from innovations
The realisation of the value of an innovation is uncertain and can only be fully assessed sometime after implementation, often decades after its conception. The
value of an innovation can also evolve over time and provide different types ofbenefits to different stakeholders. Complementary measures and analytical strategies must often be employed to trace innovation outcomes after a suitable lengthof time.
Therefore, it is challenging to assess the importance and the extent to which HEIs
contribute to the various impact phases of the complex interaction processes of an
innovation process. Sometimes their inputs may be pivotal or even foundational
for large impact economic and social innovations, but this contribution may be very
difficult to trace and document.
There may be some truth in the assertion that HEIs shape into a greater extent thecontent and direction of radical innovations than incremental innovations, but tothis claim there is not yet a consensus in the academic literature.

Innovation in other societal sectors – public and civic sectors
There is not yet an established set of definitions of what innovation is in the publicand civic sectors, comparable to that of the Oslo Manual for the business sector.
There is however an increasing interest in understanding the breadth and the typesof innovation activities and innovation impacts in these sectors.
Innovations in the general government sector
Many process innovations in the government sector draw on or bear similarities toinnovations in the business enterprise sector, but service innovations often meet redistributive or consumption-related goals that are unique to government. Common
characteristics of innovation in the government sector include the frequent use ofcollaboration, including with organisations in other sectors, and the co-productionof innovations.
The absence of a well-defined market is frequently cited as the major difference
between the business and government sectors. The absence of a market altersboth the incentives for innovation, the incentives for how intensively one should innovate and the methods for measuring innovation outcomes. Without data on thecost or price paid for government services, outcome measurement has relied on
subjective, self-reported measures, such as an increase in efficiency or improved
user satisfaction (Bloch and Bugge, 2013).
The study of innovation within government and the public sector more broadly
has attracted a growing body of empirical research, motivated in part by the increasing demand for benchmarking the efficiency and quality of public services aswell as identifying the factors that contribute to desirable innovation outputs andoutcomes. De Vries, Bekkers and Tummers (2016) authored a highly cited literature9

review paper, attempting to systematize research on general government innovations with respect to the following five issues: (1) the definitions of innovation, (2)innovation types, (3) goals of innovation, (4) antecedents of innovation and (5) outcomes of innovation.
Social innovations
Non-profit institutions (NPIs) produce or distribute goods or services, but do not
generate income or profit for the units that control or finance them, and they areoften non-governmental social institutions. Many NPIs seek to implement “social
innovations”, defined by social objectives to improve the welfare of individuals orcommunities.
Kleevey and Zaring (2018) defines a social innovation as a novel activity or organisational mode that is not, or at least not primarily, motivated by private gain or
business logic. There is a large number of other definitions of social innovation inliterature, a fact that indicates the growing interest in and the policy importance ofthe social innovations. Still, little literature conceptualises the role of the university
in delivering social innovation. Several researchers have argued that the third mission role of HEIs has been largely construed in the commercial sense of knowledge
and technology transfer, in what has been labelled the “entrepreneurial university”
while their contributions to social innovations has been largely neglected. Lately,
however, a growing number of research has been conducted on higher education
living labs and science shops. Living labs and science shops are aimed at creating
an open and user-centered platform where the resources of the university are leveraged to co-produce knowledge and innovations that are open and available forthe use and benefit of the entire society. They are examples of how HEIs contributeto social innovation processes.

The University innovation eco-system concept
The discussion above and additional recent scholarship in the field of innovation(Erdil et al. 2018; Fasnacht 2018; Grandstrand and Holgersson 2020; Leceta &Könnölä 2019; Oh et al. 2016; Reichert 2016; Schiuma & Carlucci 2018; Viitanen2016), seeked to move beyond the division between direct and indirect innovation
activities in an attempt to illustrate the deeper and fundamental connection between the structures and the processes of innovation.
An important premise for the future of knowledge economies and societies is thatthe next decades will be clearly driven by all the aspects of sustainability and digitalization, including AI. To embrace these changes the economic and policy actors
need to move from traditional business models to more flexible and agile ones.
Universities contribute substantially to these economic and societal transformation through their open innovation ecosystems. These ecosystems are complex,
iterative, and barely controllable. We currently observe a number of industries
experiencing rapid structural changes, new market entrants, and intermediaries
with disruptive business models that are increasingly embracing open innovation.
Speed, flexibility, reliance, and efficiency have all become equally important factors
for success or failure and universities need not only to adapt but also to proactively
provide society with the necessarily skills, knowledge, cultural understanding andpeople that will contribute to reap the benefits and to mitigate the adverse effectsof the new age coming.
On that basis, we suggest directing the attention of this pilot project to the threebasic elements of any innovation ecosystem: the finances, the people, and the networks. In that respect, it seeks to escape the confines of the distinction and eventual hierarchical thinking between direct and indirect contributions and, rather,10

draw attention to the creative interplay between the three elements of innovation.
By means of this approach, our study seeks to examine the connections in the waythat structures, which are more stable, path-dependent, and deeply weaved into
the fabric of innovation (such as e.g. a TTO office) can interact with processes which
are more dynamic, elusive, and transient (such as the disclosure of an idea).
As a feasibility study, the main objective of this project is to deliver a more crystalized, condensed, and systematic set of indicators for HEIs which meet the international benchmarks for solid indicators (cf. OECD 2019; World Bank 2016) and follow
the SMART criteria for internal and external validity: specific, measurable, attainable, relevant, and time-based.
Ultimately, the pilot seeks to establish a new set of innovation indicators which will
surface a new set of innovation indicators which will capture the landscape of HEIs
innovation efforts and allow for international visibility and, ultimately, inform ourunderstanding of both NTNU’s and other Norwegian HEIs socio-economic and societal impact and its role in its broader innovation landscape in a most efficient andcost-effective manner possible.

What to measure – structural versus procedural indicators
The rationale and exact empirical practices for innovation reporting in universities has been by and large fragmentary and undertheorized. Recently, in a recent
report from the US on Innovation Impact (Cullum Clark et al. 2020:8), the innovation impact was conceptualized and quantified in terms of four categories and ninevariables: commercialization impact (New patents issued, new licenses, license income), entrepreneurship impact (spinout companies & licenses to spinout companies), research impact (paper citations & patent citations) and teaching impact (new
STEM doctoral students and new STEM bachelor´s and master´s students).
The OECD approach (2010) presents a more nuanced and sophisticated approach,including not only innovation indicators such as economic growth, intangible assets, and patents, but also softer and less readily measurable aspects of innovation, such as (inter-)national cooperation, the convergence of scientific fields, interdisciplinary research, and education and training.
Nonetheless, aspects of social innovation and public value creation escape thespan of such metrics, disallowing fields with lower technology readiness level andnot directly commercializing direction (especially humanities and social sciences) todemonstrate the positive innovation externalities that they offer to their regional
ecosystems, from both university and industrial directions. Qualitative and quantitative studies from a social science perspective (Benneworth et.al. 2016, Benneworth
and Jongbloed 2010; Gullbradsen and Aanstad 2015) have repeatedly pointed tothis gap of research with regards to the impact of innovation from “softer” disciplines and less established- yet consequential- innovation channels. What is more,this line of inquiry has also pointed to the fact that innovation impact spans beyond
economic growth or academic credentials, but encapsulates also social innovationand public value creation, the impact of which is less directly measurable and reported in innovation outcomes and results (Ansel and Torfing 2014; Arundel et al.
2019; Bloch 2011; Brix 2017; Kelly et al. 2002).
For the purposes of this pilot project, we shall therefore pursue a slightly different
and broader analytical perspective. Based on a growing body of scientific work on
behavioral and cognitive science regarding the issue of priority-setting in decision
sciences and governance of organizations (Brunsson 2007; Gutierez et al. 2008;
Koechin 2020; Riedl, Brandstätter, & Roithmayr 2008; Patil et al. 2014; Marchau

11et al. 2019), we shall distinguish between two types of innovation indicators andakin innovation decisions: structural and procedural. On the one hand, structural
approaches represent the lion’s share in the research of innovation indicator anddescribe the relation between information stimuli (input) and decision responses(output) to infer the decision strategy used in order to achieve an envisioned innovation outcome (Abelson and Levi 1985; Einhorn, Kleinmuntz, & Kleinmuntz 1979;
Ford et al. 1989; Reidi et al. 2008). In the structural paradigm, innovation indicators
are used in order to capture quantifiable metrics and data with concrete outputs,
such as the standardized commercialization data that despite their preponderance,
do occasionally invite significant challenges regarding their depth, innovation scope
and their practical import (see Gault 2014: 441-464; World Bank 2011).
The rival approach to innovation indicators, identified as the procedural approach(Birchall et al. 2011; Dziallas and Blind 2019; Eling and Herstatt 2017; Klenner et
al. 2013) seeks to uncover some of the dynamics between inputs and impacts.
Procedural approaches seek therefore to unravel network dynamics, collaborations, and emergent activities which defy easy quantification and/or estimation of
their impact contributions. Ultimately, by surfacing the procedural dynamics in theprocess of innovation and within the university eco-system theoretical perspective,
we aim to compile data which allow for new innovation indicators to emerge which
pay attention to innovation eco-system human dynamics.

Demarcating indicators for innovation activities in HEIs
There are three important demarcation zones that ought to be addressed in our
endeavour to justify the validity of any set of innovation indicators for HEI, most
preponderant the demarcations against the domain of research, education andthe complementary relations with indicators on TTO commercialisation activities.

Overlapping with research indicators

Indicators for innovation and impact of research are routinely classified under two
scholarly categories. The first category is standardly referred to as science andtechnology indicators (STI). This line of statistical measurement and econometricwork, borne out of the need for rules towards the reliable and comparable measurement of innovation in research and development (henceforth R&D), gave riseto the first manual for the collection and interpretation of R&D data, the Frascati
Manual of 1963 (OECD 2002:151, Gault 2014:41) and a large host of subsequent
manuals following and refining these standards of innovation measurement, mostprominent of which remains the Oslo Manual (currently in its fourth edition- see
OECD: 2018). These types of indicators are not interesting in the setting of this pilot
project, although several of the innovation indicators we suggest for inclusion in
the pilot constitute subsets of the STI-indicators for the Higher education sector.
Our idea here is to exploit the finer grained information that the HEIs do possess onfunding and people that directly relates to innovation, and not mainly to researchor education.
Secondly, an established set of indicators for measuring impact of research arescientometric indicators. Scientometrics, a rapidly developed field within science andpublic policy, has been defined as the “quantitative study of science, communicationin science, and science policy” (Hess, 1997), with an emphasis on analysis of the development and mechanisms of sciences through statistical mathematical methods.
The field of scientometrics has a recent history. It has evolved from the study of indices for improving information retrieval from peer-reviewed scientific publications(commonly described as the “bibliometric” analysis of science) to comprise othertypes of documents and information sources relating to science and technology.
These sources may include statistical and curated data sets, web pages and social

12media metrics (cf. Hicks et al. 2015; Shibayama and Wang 2020). Recent evolutions
and minable data points in the field of research innovation and impact include thedevelopment citation metrics (e.g. Altmetric scores, Social Science Citation Index,
Category Normalized Citation Impact (CNCI) etc.), readability measures and scoring for papers (e.g. Flesch Reading Ease (FRES), Flesch–Kincaid (FKS), Gunning Fog(FOG), Simple Measure of Gobbledegook (SMOG) and Dale–Chall (DCS)), as well as
the identification and analysis of citation networks (cf. Hicks et al. 2015; Wang 2013,2016; Wang et al. 2015).
Scientometric indicators, as combined with STI indicators, contribute much to recent universal university efforts to standardize, collect, collate, report and analyze
a wide gamut of science, technology and innovation activities by providing evidence
on a selected set of science and technology (S&T) outcomes and ultimately the innovation and impact of research efforts. Despite their relative relevance for thefield of innovation studies, however, and their importance for understanding some
aspects of the innovation and impact of research in society, this pilot study will notharness such data. This is because we seek to produce novel indicators that are
relevant for capturing innovation dynamics not revealed by scientometric indicators. Having said that, this artificial distinction between research and innovation
indicators is not sustainable from an organizational perspective and in the futureboth research and innovation indicators must be compiled and analyzed together
for strategic decision-making purposes.

Overlapping with education indicators

As mentioned, the scholarship concedes that the development of reliable indicators for innovation performance within HEI education is still at its infancy (Loukkola
et al. 2020, OECD 2019, NTNU 2019b). Yet significant efforts have been made tomeasure the quality and innovativeness of the educational process. These are divided into two segments: learning indicators and teaching indicators.
Efforts to create indicators with regards to learning include the development oflearning outcomes for each course or an educational programme at the university
sector. Such efforts have been initially shouldered from OECD’s “Assessment ofHigher Education Learning Outcomes” feasibility study (AHELO 2013) and, more recently, from the European Commission’s “Measuring and Comparing Achievementsof Learning Outcomes in Higher Education in Europe” (CALOHEE 2020), both ofwhich were designed to allow for testing the performance of students -transnationally- on subject level. The reliability and replicability of these efforts for measuring
and indexing learning remains still inchoate.
On the front of teaching innovation metrics, the lack of indicators is also noteworthy. Standardly, rankings and quality assurance agencies reflect on teaching quality
using “student satisfaction” as an indicator in their work, typically captured throughstudent surveys. These surveys, conducted either by ranking providers (e.g.
U-Multirank and THE Europe Teaching Rankings) or through the assigned national
statistics agency (e.g. NIFU) ask specifically both on the opportunities for interaction with teachers and on the overall learning experience. Recently, online teaching
platforms and learning databases (e.g. Canvas, Blackboard etc.) are also harnessed
for yielding data on what has been identified as “learning analytics”, namely, thedegree of student engagement and interaction within a course’s trajectory (c.f. Kie
Daniel 2016; Lester et al. 2018). This recent development, harnessing on the powerof big data and AI, is also considered as a promising avenue for understanding the
dynamics of educational innovation and learning practices within institutions.
In a nutshell, indicators and proxies for capturing and assessing the quality andinnovativeness of education span across factors such as the following: student and13staff numbers, student progression (number of degrees awarded, funding formulae, graduation rate and time to graduation, drop-out and retention rates), as wellas indicators for graduate employment (e.g. U-Multirank’s indicator of contact withthe work environment, a composite indicator reflecting on number and student
participation in internships) and internationalization (e.g. language of instruction).
The overall degree of overlap of innovation processes with the core of the educational sphere is significant, particularly with regards to indicators for graduate
employment and entrepreneurial preparedness and mind-setting of students.
Cognizant of the multitude of data on this topic and its substantial promise, thispilot project suggests to only focus on developing education/student innovation
indicators, namely, harnessing data on student-driven innovation practices acrossfields and particularly within underexamined fields of innovation practices. Focus
on student-driven innovation data will yield riveting insight into the process and the
dynamics of innovation eco-systems in academia, as well as its prospects for surfacing the potential of these innovation indicators to link more closely the learning environments with student-driven commercialization. In addition to these indicators,
this pilot project will only focus on indicators measuring people mobility among
faculty members as well as PhD employability and direct collaboration between
students and the industry. All other indicators measuring student and educational
activities, irrespective to how important they actually are for the governance of all
modern academic organizations, they do belong to the realm of education indicators and as such are not viewed as novel innovation indicators to be included into
this pilot project.

HEI-innovation is a broader phenomenon than commercialization ofresearch results

In the wake of a substantial mass of scholarly efforts on innovation in universities(for an overview, see Kaloudis et al. 2019, Xu et. al. 2020), researchers and academic
practitioners have been confronted with a persistent dilemma in the monitoringand evaluation of innovation and its most representative indicators. On the one
hand, several studies point to the understanding of innovation as a broader, holistic phenomenon which carries the epiphenomenal attributes of an ecosystem(for recent overviews of this approach, see Morris et al. 2017; Talmar et al. 2020).
In this line of reasoning, innovation has been conceptualized and operationalized
as a non-linear system of interactive actors and activities, which in turn cluster into
specific patterns and features within given networks. These patterns, broadly subsumed under different interim metrics, are rendered meaningful through a longterm perspective which endorses iteration and process-based thinking as its main
attributes.
On the other hand, the standard and most predominant models for measuring
innovation in the higher education sector are inventoried under the commercialization indicators of a university. These are a) patenting (applications and filing); b)
licensing (as a measure of the extent to which patent production is economicallyuseful and marketed); c) HEI academic entrepreneurship through spin offs, incubation hubs and start-ups can be spearheaded by either students or faculty members, usually in collaboration with societal and industry stakeholders. Due to theirwide scope and straightforward measurability potential, these indicators on commercialization outputs have repeatedly attracted significant scholarly and science
policy interest for their impact and implications to both academic employees andstudents (see Fabrizio 2007; Mathisen and Rasmussen 2019). This scientific inquiry,
driving political decision-making on national priorities and economic competitiveness in Europe and the OECD zone, remains a topic of recurrent interest.

14

In Norway, HEIs also measure the number of new innovation ideas (i.e. DOFI-filings).
This is an interesting indicator to analyse in this pilot project as a proxy of the creativity generated within a specific HEI innovation ecosystem.
Collecting extant information on commercialization metrics are relatively easily accessible and we shall therefore include these indicators in this pilot project in orderto provide a more complete picture of HEI innovation foot-prints. Having said that,
there is a need for a more encompassing and systematic developmental work in order to produce more precise, more robust, and comparable R&D commercialisation
indicators in Norway and abroad (Kaloudis et al. 2019, Chapter 5).

Benefits and the potential purporting use of the indicators
Due to the velocity, high cognitive load, and complexity of innovation environments,
the ability to construct and frame decision-making priorities for what constitutes an“innovation activity” and, subsequently, “innovation impact” is fraught with indeterminacy. On the one hand, standard innovation metrics do capture several tangible
and quantifiable aspects of academic innovation activities, such as publication impact, commercial or pecuniary activities, and educational accreditation or student
progression. However, the extent to which the chain of cause and effect between
an innovation activity and innovation impact is yet to be unambiguously clarified
and evidently is not addressed in this pilot project.
Having said that, the levels of technology readiness and the familiarity of an academic environment with innovation spans across a large continuum, from the initial inception of an idea to its full maturity and dissemination (Engel 2012; Jacobsen
et al. 2016). In light of this, the indicators we propose in this pilot project seek to reflect and expand upon the innovation levels of each faculty setting, tailored aroundthe Nordic setting and the particular path dependencies of NTNU.

Cost-Benefit Analysis assessments of Indicators: Criteria
Our assessment of the degree of usefulness of individual indicators builds upon
three main factors/criteria. The first twin set of factors is the relative information
value (benefit) on the one hand and ease of retrievability (cost) on the other. By
information value we understand the reach and interconnectivity an innovation
activity has in the innovation eco-system. By ease of retrievability, we understandthe extent to which a dataset can be collected and aggregated. The third factor isthe degree of disciplinarity or domain-knowledge specificity of an indicator. Thistranslates into the extent to which an indicator captures information only within a
specific discipline or whether it carries significance across disciplines. That factor
of disciplinarity can cast light on the comparability between faculties and allow fora deeper understanding of the intrinsic and extrinsic aspects of innovation in theknowledge economy.

15

Cost/Benefit

Low

High

Low

They will be included if
relevant

Important indicators to
identify in the pilotHigh

Not interesting type ofindicators

Careful selection of somefew of those

Figure 1. Visualization of Cost-Benefit Assessment Model

By virtue of this model, our analysis aims to identify indicators that are low on
production costs, yet high on information content. Ultimately, our analysis seeks to
strike an equilibrium on the rates of return between the two antinomical forces ofcost and benefits, but always with a view to elevated quality in the collection, interpretation, and social influence of the concentrated innovation efforts.

Selection strategies for the production of indicator data inthe pilot
Standardly, the most salient metrics of innovation and their akin indicators measure on metrics such as quantity of spin-offs, the volume and quality of intellectual
property generation and research income generation. However, in response to thegrowing consensus on the importance of diversifying the data points and overall
quality of innovation data, our approach includes 14 selected departments from 8
faculties of NTNU.
The benefits from such an experimental approach (i.e. controlling for diversity) tothe collection of innovation indicators are manifold. First, such an approach canelicit and subsequently prioritize the issue of visibility in the creation of innovation indicators. As much empirical scholarship in the field of innovation studies has
showcased (Leonardi 2014; Wooten and Ulrich 2015), the ability to create visibility around knowledge transfer activities and their innovation has a unique impact
across competitive innovation arenas, and is evidenced to carry significant consequences in the formation and continuous establishment of knowledge transfer.
The second- and ensuing- benefit from this approach, which could become manifested after the establishment of visibility around innovation would be the establishment of protocols and guidelines for internal governance, which would be designed
to boost innovation performance in the future. Certainly, the establishment of reliable metrics for innovation performance is still at an embryonic stage, and more
research is required as to establish scholarly consensus as to what counts as highinnovation performance, especially beyond commercialization outputs (Apa et al.
2020; Birchall et al. 2011; Dettenhofer et al. 2019; Huang and Chen 2017; Kaloudis
et al. 2019; Ye et al. 2020). However, the consensual establishment of guidelines
for boosting innovation performance can yield high dividends in the procurementof academic innovation and the establishment of a more sophisticated innovation
ecosystem.

16

The pilot set of innovation indicators
On the basis of the demarcation criteria listed above and the cost-benefit considerations that shape and limit the scope of this pilot project we suggest investigating more closely a list of 15 procedural innovation indicators seeking to measurestrengths and weaknesses of four fundamental dimensions of academic innovation ecosystems, namely the:a) Direct commercialization results (patents, IPR-licensing, spin-offs) andincluding DOFI-filings and non-pecuniary agreements for licensing of IPRs.
b) Funding: the ability to attract innovation funding.
c) People: the complex interactions of people engaged in HEI innovation activitiesd) Networking: the number and the reach of the networking activities between theacademic environment, economic actors, and the broader society.

IMPACT

Networking and knowledge spillovers

Sharing people, collaborations

Other financial transaction
flows for innovation

Commercialization
activities

Figure 2. Depiction of the measurable layers of innovation impact, knowledge co-creation and networking
and knowledge spill-overs. The larger the layer the more difficult the measuring of the related activities andpresumably the opaquer is the impact.

17

The idea behind the Figure 2 is to show that the innovation impact of the HEIs is inmany studies assumed to be much larger than that that is possible to capture withtraditional commercialization indicators and other financial data. The more dense,
fuzzy and multi-modal the interaction patterns between HEIs and the society, thelarger the plausible impacts of knowledge spill-overs seems to become and, concurrently, the harder it is to measure this impact. This is one of the reasons why
procedural type of indicators of flows and people interactions and networking may provide a richer description of how ecosystems work and differ.
The indicators we suggest in the next chapter just represent a first attempt to illuminate some of these questions. They are included because they do not clearly
belong to the domains of research or education indicators, which are more dominant in the peer-reviewed literature on the topic. Second, as per our assessment
model, these indicators are selected from the point of view of a balancing costs for
producing them against the overall informational gains. As per Gault (2013),3 given
the high costs of producing innovation indicators and the subsequent collection of
innovation data, there is a common trade-off between the utility of an indicator andits cost of production. In that respect, our study aimed to strike a balance between
these two dimensions. In general, there is a limited or no previous academic literature for many of those indicators, but a number of them are also suggested on the
belief that they capture other correlated innovation activities, as well as the cultural
spirit encapsulated in the activities we are measuring. Yet, despite the fact that we
indeed we find some evidence of correlation between indicators in our data, there
is still no extensive evidence for this claim and in fact the issue of the informationalreach of the suggested indicators is exactly one of the important questions that we
ultimately must investigate in the future.

3

18

Cf. particularly Gault 2013 pp. 441-464 on the challenges and unresolved issues of establishing innovation indicators.

Detailed Description of the final
set of Innovation Indicators

19

1. and 2. BOA-Income in total and BOA directly from funding that has notresearch as its main goal (innovation-driven external funding)
The rationale for this indicator postulates that a high proportion of externalfunding income (BOA) involving diverse activities and funding sources indicate
a more intense and deeper relationship between the university and thenational industry, health authorities and public and regional sectors whichspans beyond traditional research funding streams. The data source is
NTNU data from the BEVISST database and an institutional questionnaire. Its
computation will be double, divided by total BOA and then by total faculty staff
in the respective department.
3. Job placement of NTNU PhDs
This indicator captures the job placement of NTNU PhDs across sectors. The
rationale for this indicator is that the employability and exact positioning of
NTNU PhDs in the labor market may reflect the attractiveness and relevanceof such research-based innovation for non-academic milieus, and provide
insight into the procedural aspects of innovation, as evinced by the patterns
underpinning the career trajectories of PhDs. In particular, The placement ofdoctorate degree holders in various types of employment is a reflection of theactive knowledge transfer links of the academic community to the broader
labour market. The data source for this indicator is NTNU’s register for student
progress, as well as institutional surveys from NTNU’s akin faculties. The
computation for this indicator will occur as a survey of Position and number of
NTNU PhDs divided by each sector’s aggregate employment numbers.
4. Industrial PhDs (Nærings-PhD and offentlig-PhD)
This indicator captures the absolute number of Industrial PhDs (funded byeither industry or other non-academic stakeholders) by absolute number andby faculty staff. The rationale for this indicator is that the attractiveness of NTNU
as a study location for these PhD candidates at the forefront of competitive
innovation may reflect the attractiveness and relevance of such researchbased innovation. The placement of Industrial PhDs is a reflection of therelevance, active collaboration and knowledge transfer links of the academic
community to the involved organisations. The literature points clearly to thefact that this measure is the preferred one amongst companies which do notwant to cooperate with others in key competence areas. The data source
for this indicator is NTNU’s potential registers for alumni information, withthe possibility of gathering data also from the Research Council of Norway.
The computation for this indicator will occur as a survey of the number ofIndustrial PhDs within NTNU divided by the absolute NTNU PhD number andfaculty staff numbers.
5. Adjunct Research Positions (II-er stillinger - sidegjøremål) from NTNU’s academic roster to private sector (including creative/cultural industries), public
sector (excluding health), research institutes, health sector, abroad
This indicator captures the adjunct research positions of NTNU academics tothe entire spectrum of non-academic activities. The rationale for this indicatoris that the more research activity is carried out in tandem between external
partners from industry and the academic sector, the more likely it is thatknowledge transfer and meaningful innovation activity takes place between
academia and business. The data source for this indicator emanate from an
institutional questionnaire and its level are to be aggregated to the institutional

20

and potentially also on the faculty level. The computation for this indicator will
occur as a survey of the number of Adjunct research Positions divided by the
total number of full-time faculty.
6. Adjunct Positions (Professor II-er stillinger) from industry, public sector(including cultural industries), research institutes, health sector, othernational universities, abroad.
This indicator captures the reverse relationship of the prior indicator, namely,
the number of researchers from the non-academic sector to hold positions
into NTNU’s academic roster. The rationale for this indicator is that themore research activity is carried out in tandem between external partners
from industry and the academic sector, the more likely it is that knowledgetransfer and meaningful innovation activity takes place between academia
and business. The data source for this indicator emanate from an institutionalquestionnaire and potential NIFU data, its level are to be aggregated to theinstitutional and potentially also on the faculty level. The computation will
occur on the basis of surveying the number of Adjunct research Positions from
non-academic sector divided by the total number of full-time faculty.
7. Work-Life Collaboration in BA and MA theses with private sector(including creative/cultural industries), public sector (excluding health),
health sector
This indicator captures the number of student projects (thesis projects or
freelance projects) that are co-developed with industrial partners and are
aimed towards capturing externally- driven activity on the subject fromstudents. The rationale underpinning this indicator is that projects that are
co-developed with industry may have higher probability of exerting regional
impact and improving the chances of student employability, financial
consequence, and measurable innovation. The data source for this indicatoremanate from an institutional questionnaire and its level are to be aggregatedto the institutional and potentially also on the faculty level. The computation
will be based on tracking the number of work-life collaboration in BA and MA
thesis divided by the total number of thesis.
8. Approved agreements/Memoranda of Understanding with social actors(e.g. DNB, municipalities, etc.).
This indicator captures another underexplored dimension of innovation,
namely the number of Approved agreements/Memoranda of Understanding
with social actors (e.g. DNB, municipalities. Its main definition is based on
identifying the five most important approved agreements with social actors (interms of size or income potential). The rationale for this indicator postulates
that the number of approved agreements and MoUs is a reflection of the
dynamic synergy between funding and people towards realizing knowledgetransfer links of the academic community to the broader society. The data
source for this indicator emanate from an institutional questionnaire and its
level are to be aggregated to the institutional and potentially also on the facultylevel. The computation will be based on tracking the number of approved
agreements and/or MoUs.

21

9. Co-participation in R&D projects per sector. Number of companies and public sector organizations which are involved in these activities.
This indicator captures a well-explored of applied innovation, namely the coparticipation in R&D projects per sector in juxtaposition with the number ofcompanies which are involved in these activities. The rationale underpinning
this indicator postulates that the number R&D co-participation is a reflectionof the dynamic synergy between funding and people towards realizing
knowledge transfer links of the academic community to the industrial sector.
The data for this indicator will emanate from an institutional questionnaireand clustered on the the institutional and potentially also on the faculty level.
The computation will be a survey of the co-participation in R&D projects per
sector as well as a descriptive survey of the number of companies which are
involved in these activities.
10. Open knowledge transfer - number of “open source” licenses
This indicator also captures an emergent aspect of innovation on the crossroads
between networking and commercialization, namely, the open knowledgetransfer and the correlated number of licenses. The rationale for this indicatorhinges on the assumption that open knowledge sharing here involves open
research and knowledge sharing of results created by employees at NTNU,
and constitutes an important contribution to open innovation from the UoH
sector. As an indicator, only sharing of results / intellectual property explicitly
subject to a license is included. NTNU’s policy for intellectual property rights
provides guidance and definitions. Examples of typical licensing mechanisms
are open- source code sharing or Creative Commons licenses. And, as a further
specification, publications and articles are not included as these are included
as an indicator of research results. The data source for this indicator emanatesfrom an institutional questionnaire and potential Cristin Data. The data levelis to be aggregated to the institutional and potentially also on the faculty level.
This indicator will be computed based on the number of licenses to industrybased actors as open sources.
11. Indicators for commercialization and student-based entrepreneurship
In addition, we suggest five additional commercialization indicators:a) Number of university-student start-upsb) Number of academic start-ups (based on members of faculty)c) Number of innovation ideas submitted to the local Technology Transfer
Office (DOFIs)d) Number of commercial licensese) Number of filed priority patent applications

22

Indicators that have been
considered, produced but
discarded from the “canon”
of HEI innovation indicators23

Indicators that have been considered, produced but discarded from the “canon” of HEI innovation indicators
The following indicators have been produced in the pilot project but due to comments from the departments´ faculty and from the plenum discussions on the
cost-benefit trade-offs we decided that either they were too costly or not importantenough or not well-defined to be included in the set of future innovation indicatorsin their present form.
Number of Stud-ENT Applications (As proxy of innovation indicator for
student-based innovation and entrepreneurship)
This indicator captures an underexplored dimension of innovation, namely the
degree of student entrepreneurship as reflected to the total amount of Stud-ENT
Applications from students. The rationale underpinning this indicator is that thestudent-led applications projects that are submitted can provide meaningful insight into the competitiveness and innovation extroversion necessary for establishing new entrepreneurial links. The 14 departments have reported a very low number of such applications (one in 2020). This renders this indicator as less reliable
and valid than what we thought in the beginning of the pilot project.
Innovation-oriented
hackathlons etc.)

conferences(Businessdays,

technology

events,This indicator captures the dimension of networking, namely, the number of conferences and their akin number of participants. The rationale that grounds this indicator is that the levels of creative interaction between people and funding resources
towards a common innovation goal can yield meaningful benchmarks regarding
the innovation activity of an academic ecosystem and their probability of forming
broader collaborations for non-purely scientific events. Specifically, The number of
non-purely scientific conferences (number of events and number of participants).
List of the five most important events in which the unit is involved. Here we are talking about non-purely scientific conferences / events (and, if possible, with the number of participants) involving non-academic actors. It is interesting to also estimate
the unit’s costs associated with organizing / co-financing these events. The data
source for this indicator emanate from an institutional questionnaire and its level
are to be aggregated to the institutional and potentially also on the faculty level.
Number of events and number of participants in university events – Concerts,
exhibitions, book-opening events etc.
Like its akin prior, this indicator also captures the dimension of networking, andparticularly the number of events and their akin number of participants in these
university events which are do not qualify as conferences (i.e. concerts, exhibitions,career days, etc). The rationale that grounds this indicator is that the levels of creative interaction between people and funding resources in events outside the scientific sphere strictly defined can yield meaningful benchmarks regarding the innovation activity of an academic ecosystem and their probability of forming broader
collaborations for non-purely scientific events. The data source for this indicator
could potentially be the Cristin Data in the future. But for now, this indicator seems
not to be considered important enough.

24

Number of health innovations
This indicator also captures an emergent aspect of innovation on the crossroads
between networking and commercialization, namely, the number of health service
innovations. The rationale for this indicator postulates that the number of healthinnovations presents a reflection of the broader synergies of academia, industry, and society. The data source for this indicator we thought emanated from
Helseforetakenes database on health innovation in Norway. The database in its
currently format does not however report collaboration with HEIs and hence it is
not possible to report these data for NTNU. This indicator will be possible to measure properly as NTNU-contributions to registered numbers of health innovations in
the future, given that this information is recorded in the future.

Other possible explorative innovation indicators to explore in the future

Below there is also a list of alternative, novel, and even more explorative innovation
indicators which, upon deliberation, could be rendered useful for data collection
and data analyses for the purposes of this project, but – taken together - with aconsiderable cost.
a) Lectures, Courses, and Presentations in Industrial Settingsb) Guest Presentations from Industry on coursesc) Several measures on Joint Labs and jointly-used spaces, co-funded by universities and companiesd) Tailor-Made teaching programs for companies or government agencies (and certifications) in addition to ordinary life-long learning activities (EVU).

25

Final remarks – part 1
In the wake of the last 30 years, the relationship between universities and businesses has been irrevocably transformed and now it is increasingly common toconceive HEI campuses as innovation eco-systems. Some of them with a strong presence and extensive regional, national and even global reach, others less systematically architectured and with more limited outreach. Facing an accelerated paceand complexity of innovation, companies and universities can no longer rely on
their internal R&D processes alone. This enhanced complexity has given rise to theneed to peruse, identify, and absorb externally sourced relevant knowledge in awide variety of disciplinary areas, sectors and institutions. By virtue of their nature,
universities are gatekeepers to the scanning of new knowledge frontiers and theearly identification of the next generation of technologies, fostering networks witha complex and constantly changing give-and-take of ideas, cultural meaning, knowledge, IP, and market opportunities. Not only are values, interactions, and roles ofuniversities, firms, and government revisited, but these changes are accompanied
by a common focus on the new forms of connectivity that can mobilize innovative
potential. Henceforth, NTNU’s role in this growing innovation ecosystem may span
beyond the traditional commercialisation indicators and capture new and promising dimensions of its broader innovation contributions. We hope and we believethat with this report presenting a set of 15 indicators that attempt to capture different aspects of innovation contributions and activities, we contributed to bring thediscussion of measuring innovation from HEIs a step further, acknowledging that
there is a more work to do in the future in four specific domains:• Evaluate the importance, quality and cost of the suggested indicators• Standardize the collection of information for these indicators in a manner thatis cost effective for the departments and if possible relying only on the extant
administrative databases• Evaluate the added value of these indicators compared to extant indicators on
research and education for the innovation governance of HEIs.
• Conduct more research (quantitative and qualitative) in order to understandinner dynamics and interconnections between the suggested indicators at the
department and HEI levels.

26

References – part I
Abelson, R. P., & Levi, A. (1985). Decision making and decision theory. In G. Lindzey &E. Aronson (Eds.), The handbook of social psychology (3rd ed., pp. 231-309). New
York: Random House
Apa, R., De Marchi, V., Grandinetti, R., & Sedita, S. R. (2020). University-SME collaboration and innovation performance: the role of informal relationships andabsorptive capacity. The Journal of Technology Transfer. https://doi.org/10.1007/
s10961-020-09802-9Bedford, T. et al. (2018), The Role of Universities of Science of Technology inInnovation Ecosytems: Towards mission 3.1, CESAER. Birchall, D., Chanaron, J.J., Tovstiga, G., & Hillenbrand, C. (2011). Innovation performance measurement:
current practices, issues and management challenges. International Journal ofTechnology Management, 56(1), 1–20.
Birchall, D., Chanaron, J.-J., Tovstiga, G., & Hillenbrand, C. (2011). Innovation performance measurement: current practices, issues and management challenges.
International Journal of Technology Management, 56(1), 1–20.
Bloch, C., & Bugge, M. M. (2013). Public sector innovation—From theory to measurement. Structural Change and Economic Dynamics, 27, 133–145.
Brunsson, N. (2007). The consequences of decision-making. New York: Oxford
University Press.
Carter-Johnson, J. (2020). University technology transfer structure and intellectual
property policies. In Research Handbook on Intellectual Property and Technology
Transfer (pp. 4–40). https://doi.org/10.4337/9781788116633.00011
Chesbrough, H. W. (2003). Open Innovation: The New Imperative for Creating andProfiting from Technology. Harvard Business Press.
Cullum Clark, J. H., Murri, K., Nijhawan, V., Blackwell, C. T., & Ingram, S. (2020).
Innovation Impact of U.S. Universities. Available at: https://gwbcenter.imgix.net/
Publications/Resources/gwbi-university-impact-report-ranks-exec-summary-full-report.pdf
Dahlander, L., & Gann, D. (2010). How Open is Innovation? In Creating Wealth from
Knowledge. https://doi.org/10.4337/9781848441248.00009
Dettenhofer, M., Doussineau, M., & Arregui-Pabollet, E. (2019). An Analysis ofUniversity Governance Dimensions in Academic Research and S3 Innovation
Performance. In Smart Specialization Strategies and the Role of Entrepreneurial
Universities (pp. 153–185). IGI Global.
De Vries, H., Bekkers, V., Tummers, L. (2016), “Innovation in the Public Sector: A systematic Review and Future Research Agenda”, Public Administration, Vol. 94, issue
1, pp. 146-166
Fabrizio, K. R. (2007). University patenting and the pace of industrial innovation.
Industrial and Corporate Change, 16(4), 505–534.
Edwards, J., Marinelli, E., Arregui-Pabollet, E., and Kempton, L. (2017) Higher
Education for Smart Specialisation. Towards Strategic Partnerships for Innovation. S3

27

Policy Brief Series No.23/2017. Technical Report of the Joint Research Centre (JRC).
Luxembourg: Publications of the European Union.
Einhorn, H. J., Kleinmuntz, D. N., & Kleinmuntz, B. (1979). Linear regression and process-tracing models of judgment. Psychological Review, 86(5), 465–485.
Eling, K., & Herstatt, C. (2017). Managing the front end of innovation-less fuzzy,yet still not fully understood: Managing the front end of innovation. The Journal of
Product Innovation Management, 34(6), 864–874.
Engel, D., Dalton, A., Anderson, K., Sivaramakrishnan, C., & Lansing, C. (2012).
Development of Technology Readiness Level (TRL) Metrics and Risk Measures.
Richland, Washington: Pacific Northwest National Laboratory, Prepared for US.
Department of Energy.
Erdil, E., Meissner, D., & Chataway, J. (2018). Innovation Ecosystems and Universities.
In D. Meissner, E. Erdil, & J. Chataway (Eds.), Innovation and the Entrepreneurial
University (pp. 3–14). Springer International Publishing.
EUA (2018) Coherent policies for Europe beyond 2020: maximizing the effectiveness of
smart specialisation strategies for regional development. EUA.
Fasnacht, D. (2018). Open Innovation Ecosystems. In D. Fasnacht (Ed.), Open
Innovation Ecosystems: Creating New Value Constellations in the Financial Services(pp. 131–172). Springer International Publishing.
Ford, J. K., Schmitt, N., Schechtman, S. L., Hults, B. M., & Doherty, M. L. (1989).
Process tracing methods: Contributions, problems, and neglected research questions. Organizational Behavior and Human Decision Processes, 43(1), 75–117.
Gault, F. (2013). Handbook of Innovation Indicators and Measurement. Edward Elgar
Publishing.
Graham, R. (2014) Creating university-based entrepreneurial ecosystems: Evidence
from emerging world leaders. Cambridge, MA: MIT Skoltech Initiative.
Granstrand, O., & Holgersson, M. (2020). Innovation ecosystems: A conceptual review and a new definition. Technovation, 90-91, 102098.
Gutierrez, E., Olundh Sandstrom, G., Janhager, J., & Ritzen, S. (2008). Innovation anddecision making: Understanding selection and prioritization of development projects. 2008 4th IEEE International Conference on Management of Innovation andTechnology, 333–338.
Hess, D. J. (1997). Science Studies: An advanced introduction. New York: New York
University Press.
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). The Leiden manifesto for research metrics. Nature,520, 429–431.
Huang, M.-H., & Chen, D.-Z. (2017). How can academic innovation performancein university–industry collaboration be improved? Technological Forecasting andSocial Change, 123, 210–215.
Jacobsen, S. Tor Andre Johansen og Kim Hamli. (2016.) Rapport: Pilotering av nasjonal indikator for innovasjonsaktivitet. https://www.regjeringen.no/contentas-

28

sets/c9d11b3ac014476f98c1176efae743a4/rapport_p ilotering-av-nasjonal-indikator-for-innovasjonsaktivitet.pdf
Kei Daniel, B. (Ed.). (2017). Big Data and Learning Analytics in Higher Education: Current
Theory and Practice. Springer, Cham.
Klenner, P., Hüsig, S., & Dowling, M. (2013). Ex-ante evaluation of disruptive susceptibility in established value networks—When are markets ready for disruptive
innovations? Research Policy, 42(4), 914–927.
Koechlin, E. (2020). Human Decision-Making beyond the Rational Decision Theory.
Trends in Cognitive Sciences, 24(1), 4–6. Elsevier.
Leceta, J. M., & Könnölä, T. (2019). Fostering entrepreneurial innovation ecosystems: lessons learned from the European Institute of Innovation and Technology.
Innovation: The European Journal of Social Science Research, 1–20.
Leonardi, P. M. (2014). Social Media, Knowledge Sharing, and Innovation: Toward a
Theory of Communication Visibility. Information Systems Research, 25(4), 796–816.
Lester, J., Klein, C., Johri, A., & Rangwala, H. (2018). Learning Analytics in Higher
Education: Current Innovations, Future Potential, and Practical Applications. Routledge.
Loukkola, T., Peterbauer, H., & Gover, A. (2020). Exploring higher education indicators. European University Association. https://dipae.ac.cy/archeia/publications/indicators_report.pdf
Lundvall, B. (Ed.). (2010). National Systems of Innovation: Toward a Theory ofInnovation and Interactive Learning. LONDON; NEW YORK; DELHI: Anthem Press.
McKelvey M and Zaring O. (2018), “Co-delivery of social innovations: exploring theuniversity’s role in academic engagement with society” Industry and Innovation, Vol.
25, issue 6 pp. 594-611.
Marchau, V. A., Walker, W. E., Bloemen, P. J., & Popper, S. W. (2019). Decision making
under deep uncertainty: from theory to practice. Springer Nature.
Markman, A. B. (2016). Open Innovation: Academic and Practical Perspectives on the
Journey from Idea to Market. Oxford University Press.
Mathisen, M. T., & Rasmussen, E. (2019). The development, growth, and performance of university spin-offs: a critical review. In The Journal of Technology Transfer(Vol. 44, Issue 6, pp. 1891–1938). https://doi.org/10.1007/s10961-018-09714-9
Morais, R. (2016) Universities Promoting Regional Innovation across Europe. EUA.
Morris, M. H., Shirokova, G., & Tsukanova, T. (2017). Student entrepreneurship andthe university ecosystem: a multi-country empirical exploration. European Journal ofInternational Management, 11(1), 65–85.
NTNU (2019). Mot et bredere målesystem for UoH-sektorens bidrag til innovasjon:
Forslag til målesystem basert på erfaringer fra NTNU. Oslo and Trondheim.
OECD. (2007). Frascati Manual 2002. https://doi.org/10.1787/9789955682684-lt

29

OECD. (2018). Oslo Manual 2018: Guidelines for collecting, reporting and using dataon innovation, 4th edition. Retrieved March 17, 2021, from http://www.oecd.org/sti/oslo-manual-2018-9789264304604-en.htm
OECD. (2019). Benchmarking Higher Education System Performance. Paris. Available
at: https://doi.org/10.1787/be5514d7-en
Oh, D.-S., Phillips, F., Park, S., & Lee, E. (2016). Innovation ecosystems: A critical examination. Technovation, 54, 1–6.
Okamuro, H., & Nishimura, J. (2013). Impact of university intellectual property policyon the performance of university-industry research collaboration. The Journal ofTechnology Transfer, 38(3), 273–301.
Power, D. and Malmberg, A. (2008) The contribution of universities to innovationand economic development: in what sense a regional problem? Cambridge Journal of
Regions, Economy and Society, 1, 233-245.
Riedl, R., Brandstätter, E., & Roithmayr, F. (2008). Identifying decision strategies: aprocess- and outcome-based classification method. Behavior Research Methods,
40(3), 795–807.
Schiuma, G., & Carlucci, D. (2018). Managing Strategic Partnerships with Universities
in Innovation Ecosystems: A Research Agenda. Journal of Open Innovation:
Technology, Market, and Complexity, 4(3), 25.
Shibayama, S., Wang, J. (2020). Measuring originality in science. Scientometrics 122,
409–427.
Siegel, D. S., Veugelers, R., & Wright, M. (2007). Technology transfer offices andcommercialization of university intellectual property: performance and policy implications. In Oxford Review of Economic Policy (Vol. 23, Issue 4, pp. 640–660). https://doi.org/10.1093/oxrep/grm036
Talmar, M., Walrave, B., Podoynitsyna, K. S., Holmström, J., & Romme, A. G. L.
(2020). Mapping, analyzing and designing innovation ecosystems: The Ecosystem
Pie Model. Long Range Planning, 53(4), 101850.
Tjong Tjin Tai, S.-Y., van den Broek, J., Maas, T., and Deuten, J. (2018) Industry seeking
university – The emergence of strategic public-private research partnerships. Den Haag:
Rathenau Institut.
Viitanen, J. (2016). Profiling regional innovation ecosystems as functional collaborative systems: The case of Cambridge. Technology Innovation Management Review,
6(12). https://timreview.ca/article/1038
Wang, J. (2013). Citation time window choice for research impact evaluation.
Scientometrics, 94, 851–872.
Wang, J. (2016). Knowledge creation in collaboration networks: Effects of tie configuration. Research Policy, 45, 68–80.
Wang, J., Thijs, B., & Glänzel, W. (2015). Interdisciplinarity and impact: Distinct effects of variety, balance, and disparity. PLoS ONE,10, e0127298.
Wooten, J. O., & Ulrich, K. T. (2015). The Impact of Visibility in Innovation Tournaments:
Evidence from Field Experiments. https://doi.org/10.2139/ssrn.2214952

30

World Bank. (2011). The changing wealth of nations: measuring sustainable development in the new millennium. Washington.
Xu, H., Hsu, W.-L., Meen, T.-H., & Zhu, J. H. (2020). Can Higher Education, Economic
Growth and Innovation Ability Improve Each Other? Sustainability: Science Practiceand Policy, 12(6), 2515.
Ye, L., Zeng, G., & Cao, X. (2020). Open innovation and innovative performance ofuniversities: Evidence from China. Growth and Change, 51(3), 1142–1157.

31

Part II
Summary of themore detailed
reporting on thecollected indicators



32

Summary of the more detailed reporting on the collectedindicators
This chapter presents a brief summary of a lengthier report written in Norwegian
for internal purposes within NTNU and for circulation in Norwegian decision-making circles. We redirect the interested reader to this unabridged Norwegian reportfor details on findings and analysis of the specific indicators.

Recommendations
One of the first recommendations we would like to make regarding measuring thefuture indicator set of innovation contributions at NTNU (and possibly for national
HEIs) is a question of method for measuring innovation performance, namely,
the process of identifying key performance indicators and normalization factors
which may benchmark the innovation work at specific departments (institutter)and across the institution. To be sure, this effort to standardization is not to be
confused with static oversimplification and reductionism of the innovation process
as linear. Rather, it should be an attempt to enter and understand the iterative
process of interplay between factors of quantification and, conversely, nuancing
and triggering for deeper quantitative and qualitative exploration of emerging performance indicators.
In a similar train of thought with this question of method for broader innovation
performance, a second recommendation for further work on innovation indicators
would be the establishment of methodological protocols for each specific indicator,
with a renewed and commonly agreed description of the data definition for eachindicator, its significance, potential for automatic generation of data, its reliability,
data quality standards, and its normalization standards. In that way, each indicator
could be more critically reexamined and gain further depth and replicability acrossinstitutes and, eventually, institutions.
Third, one of the prospects for the amelioration of this set of innovation indicators,
would be the further development and tailoring of existing data sources (from central administration/economy/HR and NTNU´s TTO). In that way, each data source
would gradually gain lower costs, greater transparency, and ease of processing, forcurrent and future data collection. In parallel, a recommendation for the development of data collection would be the assignment of the curation for the local datacollection by a newly appointed advisor who would be precisely mandated to curate this kind of data, both with regards to the range and the detail of resource-intensive datasets (such as e.g., human capital monitoring and networking activities).
This kind of advisor/ data curator would report centrally to an institution and allow
also potentially for the surfacing of further connections between datasets.
As a further corollary of this data set curation, a potential role for ameliorating
the innovation indicators would be the strengthening and consolidation of existingdata sources through information campaigning and explicit information requirements from all institutional members. A focus on information collection such as
the registering additional professional activities (sidegjøremål), the establishmentof collaboration agreements of master students with industry, the establishment ofregistered collaborations for BA thesis writing, as well as the data collection about
the labor market placement of PhD students are all potential candidates for consolidating data collection and vital strategic information flow for NTNU. Moreover, thedevelopment of a newly-minted central data collection solutions could significantly
expedite the processing and communication of such innovation data.

33

Last, but not least, the eventual identification of emergent indicators for notions
that progressively gain prominence in the landscape of innovation in higher education, such as interdisciplinarity or sustainability, could be promising avenues for theestablishment of further pioneering work with innovation indicators.

34

Part III
Report on Centers
for Research-Based
Innovation for
Innovation (SFI)– an analysis of 14 selected annual
reports on contributions to Innovation
35

Report on Centers for Research-Based Innovation for
Innovation (SFI) – an analysis of 14 selected annual reports
on contributions to Innovation
In recent decades, a wide host of peer-reviewed scholarly inquiry has put the topic
of University-Industry Collaborations (henceforth UICs) under the microscope(Ankrah and Al-Tabbaa, 2015; Link et al., 2015; Meng et al., 2019, Roncancio-Marin
et al. 2022). UICs´ history has been long and spanning across several decades,
while their dynamics have been examined from a multitude of perspectives, withdivergent units of analysis, and research methods (for a useful review, see Kaloudis
et al. 2019 and Roncancio-Marin et al. 2022). Recently, however, the collaborations
between university and industry have been intensified worldwide, and particularlyin the US, Singapore, and Europe (Gertner et al. 2011; Kaloudis et al. 2019). The
reasons for this intense collaboration have been mutual across universities and industry alike. On the one hand, industry has been subject to several business model
disruptions, shorter product life cycles, and faster-paced technological change that
requires constant vigilance for staying at the forefront of competition. On the other
hand, universities have been subject to significant changes in their funding sources
and in societal expectations for more relevance of their research and education
activities. Moreover, universities have been progressively seen more as engines of
economic growth for regions, creating additional pressure to prioritize their promotion of human capital and successful knowledge exchange. Consequently, the urgency of collaboration between sectors (and ideally the proposition of moving fromknowledge exchange to knowledge co-creation) has never been greater. Centrally,
governments progressively encourage further the creation of such collaborations,as a way of enhancing wealth creation and general prosperity for the population.
However, it is not only necessity that fuels these collaborations. Rather, due to factors such as reciprocal benefits. Industry gains access to elite human capital and,conversely, universities provide to their graduates and faculty staff members a better understanding of the current market, while they learn and test ideas in practice.
Universities also gain public legitimacy, since they remain relevant and gain accessto different sectoral innovation eco-systems.
Within this broader UIC collaboration background, one of the problems which has
recently garnered a lot of attention is how to measure and report innovation results in university-industry collaborations (Alexander et al. 2020; Arvanitis et al.
2008; Atta-Owusu et al. 2021; Cheng et al. 2020; Tseng et al. 2020; Garcia et al.
2020; OECD 215; OECD 2021; Perkman et al. 2013). Reporting these innovations
run the entire gamut from intuitive visions of what innovation results looked like tocarefully scrutinized analyses. In fact, higher education reporting has so far beenpracticed though multiple approaches, such as social reporting (Del Sordo et al.
2016; Moggi 2019), intellectual capital reporting (Leitner 2004; Nicolo et al. 2020;
Paloma Sanchez et al. 2009), sustainability reporting (Gamage and Sciulli 2017;
Talebzadehhosseini et el. 2021, and more recently integrated reporting (Adams
2018; Brusca et al. 2018).
Recognizing the importance of UICs networks and channels, the Norwegian government has since at least the 90-ies established a number of policy measures toforge and boost UIC in the Norwegian research and innovation system (Kaloudis etal. 2019, Chapter 2). The Norwegian Centers for Research-based innovation (SFIs)
is arguably the most important measure targeting the development and maintenance of UICs in Norway.
More specifically, the Centres for Research-based Innovation are to develop expertise in fields of importance for innovation and value creation in the Norwegian in-36

novation system. The purpose is to fund long-term research in close collaboration
between research-performing companies and research groups. The purpose of the
SFI centres are in particular designed to enhance technology transfer, internationalization and researcher training while maintaining and building on research mustof high international standards. The SFI centres may receive support for a totalof eight years (an initial five-year period with the possibility of a three-year extension). For the NTNU, participation in SFIs is of major significance and volume – SFI
represent a considerable share of innovation-oriented external funding - and isconsidered as one of the most crucial channels of NTNUs innovation outreach. Thisis the reason why there is a considerable attention from the NTNU leadership on
the results and innovation outcomes stemming from SFIs where the NTNU plays aleading or a significant role.
This part of the report has therefore as a goal to gain insight in how the NorwegianCenters for Research-based innovation (SFIs) themselves report on different aspects of innovation activities.
Table 3.1: The objectives of the SFI: Fourth generation. Source: RCN, 2021.
PurposeA research centre is a dedicated, long-term initiative designed to strengthen andfurther develop elite, creative research and innovation groups or to build up
research groups in strategically important areas.
The overall objective of the Centres for Research-based Innovation (SFI) scheme
is to enhance the ability of the business sector to innovate and create value
through a greater focus on long-term research.
The SFI scheme seeks to:• Facilitate active, long-term cooperation between innovation-oriented, R&Dperforming companies and prominent research groups.
• Promote the development of outstanding industry-oriented research clusters
that are an integral part of dynamic international networks and that enhance
the internationalization of the Norwegian business sector.
• Encourage and enhance researcher training and the transfer of knowledge
and technology in areas with major potential for future value creation.
The SFI scheme is characterised by broader objectives, a longer-term perspective
and a more targeted focus than other innovation-related instruments administered by the Research Council. The scheme provides the R&D-performing
component of the Norwegian business sector with the opportunity to take alonger-term perspective, enhance continuity and reduce risk in their research
initiatives.
For the research groups, the scheme makes it possible to achieve long-term development of expertise through research of high international calibre conductedin close cooperation with companies. The scheme is also intended to enhancequality and efficiency in the public sector.
When selecting centres for SFI status and funding, importance will be attachedto their potential to generate innovation, business development and sustainablevalue creation within the centre’s thematic priority areas. The scientific merit of
the research must be of high international calibre.
_______________________
Table 1 “The Objectives of the SFI: Fourth Generation”: RCN 2021

37

Rationale of annual report analyses
Over the span of several decades (Thomas 1973), annual reports have been repeatedly studied as a source of information for the management and disclosure of human capital (Brennan 2001), strategy disclosure (Santema and van de Rijt 2001), corporate responsibility (Waller and Lanis 2009), stakeholder analysis (Kent and Zucker2017), but also broader organizational identities (Ditlevsen 2012) and mythmaking(David 2001). At the core of such analyses of annual reports often inheres the deep
predicament of what researchers often term “voluntary disclosure” (Abeywardana
and Panditharathna 2016): how much can an organization share openly regarding
its strategy, its prospects, but also its challenges? Despite the hefty attention thatannual reports have garnered in the industrial sector, their import, impact, andcitation in the tertiary education sector has been minimal and fragmentary (Coyet al. 1993; Guan and Noronha 2013), whereas systematic synthesis or analysisof annual report data from such data sources has been completely absent. This
study seeks to study more deeply this dilemma of “voluntary disclosure” in a particular epiphenomenon of academic innovation which has never been previously
documented, namely, the annual reports of SFIs. Our primary hypothesis and early
empirical findings indicated that the annual reports offer a plethora of insights into
the potential innovation practices that occur in SFIs, and at the crossroads between
academia and industry. On the one hand, they do require some hard facts, such as
personnel, accounts, and publications, which veer to just pure accounting, without
any need for further explanation. On the other hand, the annual reports allow fora lot of room for observing how SFIs translate these research activities into innovations, since the amount of detail in disclosure is discretionary on length (no specificpage requirements) and bound by the following eight NFR guidelines: summary,vision/objectives, Research plan/strategy, organisation (organisational structure,
partners, cooperation between the centre’s partners), scientific activities and results, international cooperation, recruitment, and communication and dissemination activities.4 Consequently, the key objective of this section was to document andevaluate -from an empirical standpoint- how SFIs disclose their innovation results
and general innovation strategy within these empirical bounds. Ultimately, suchan inquiry will shed more light into the aspects of innovation awareness and thedifferent innovation priorities that undergird these centers.

438https://www.forskningsradet.no/nar-du-har-fatt-finansiering/prosjektrapportering/rapportering-for-sentre/Table 3.2: Reporting Template of SFI: Fourth Generation. Source: RCN 2021.
Reporting Template
The annual report is to describe the status of the centre’s activities, and should
contain the following items:••••

Summary
Vision/objectives
Research plan/strategy
Organisation- Organisational structure- Partners

•
•
•
•

- Cooperation between the centre’s partners
Scientific activities and results
International cooperation
Recruitment
Communication and dissemination activities

Attachment in the report (see attached form):- Personnel- Accounts- Publications

Methodology
We follow the methodological approach of a qualitative design and the research
protocol of content analysis method (Bowen 2009; Bryman 2016; Gray et al. 2007;
Gioia et al. 2013). Similar to other forms of qualitative design, the protocol of content analysis sought to systematize the empirical material, interpret it, and yield
meaningful findings to the research question being posed, as well as elicit meaningand depth to the question, namely, that of reporting SFI innovation results in theirannual reports (Creswell and Poth 2018).
Based on the analysis and parameters of the general pilot study for innovation indicators (see part 1 and 2 in this report), in this part of the report we set out to examine
how SFIs present their contributions on the four analytical dimensions for parsing
innovation: additionality, people and worklife, networks, and commercialization.
Sample Choice
The selected centers for this study have all been from the research of NTNU. The
choice has been made based upon three factors. First, NTNU has been involved in
the majority of these initiatives, making it a very representative university for the
purposes of this analysis. Second, NTNU has some schemes of central governanceregarding innovation (Vice-Rector for Innovation, Innovation Leader faculty positions-Innovation Managers), which could be arguably evincing a broader strategy
and coherence regarding the use of innovation. Finally, NTNU has been, to the bestof our knowledge, the only Norwegian HEI which has the largest inventory of innovation-specific resources centrally (NTNU Discovery, NTNU Student Innovation,
NTNU TTO), making it the ideal case study for understanding the broader ecosystem of innovation within the university, which spans across centers.

39

This study included 14 Annual Reports for the year 2020 from 14 SFIs (Centers for
Research-Based Innovation). In further detail, our data comes from the annual reports (2020) of 14 SFIs (SFI “Centre for Geophysical Forecasting”, SFI CIRFA – Centre
for Integrated Remote Sensing and Forecasting for Arctic Operations, CIUS Center
for Innovation Ultrasound Solutions,, SFI NorwAI, SFI iCSI (industrial Catalysis
Science and Innovation), SFI Klima 2050, SFI Metal Production, SFI NORCICS, SFI
Blues, SFI Harvest, SFI Move, SFI Sirius, SFI Smart Maritime, SFI Subsea Production
and Processing) where NTNU is the main partner or a key collaborator.
These documents were collected in order to capture the “best practices” of innovation reporting and akin correspondence with our four analytic dimensions forinnovation. A set of terms was used when mining these documents (*innovat* (301
counts); *value (107 counts); *student (133 counts); *system (617 counts); partners* (581 counts); *process* (491 counts); *impact (126 counts); *dynamic (223
counts); spin* (38 counts); *pilot (42 count); *intellectual property (5 counts); IPR (9counts); *gender (4 counts) *creation (16 counts); * knowledge transfer (4 counts),* organization or organization (57 counts), *collaboration (159 counts), *dynamic(223 times), *synerg* (13 counts); *network* (69 counts), *conference* (226counts) , * event* (26 counts), *open access (1 count), *open source (26 counts);*license (12 counts)*; *funding (56 counts); Horizon Europe (4 counts); LinkedIn(16 counts); Twitter (6 counts)). This list of keywords was informed by latest published research on UIC, the terms within the SFI call itself, SFI´s own midterm evaluation of the scheme, and the independent audit conducted by Damvad analytics(Damvad Analytics 2018).
Data Analysis
The document analysis approach was undertaken in accordance with establishedmethodological protocols, and it combines elements of content analysis and thematic analysis (Bowen 2009; Bryman 2016). Content analysis in the context of document analysis refers to the research process of identifying and collating meaningful sections of the document text, such as innovation results checklists, innovation
outcomes figures. Conversely, thematic analysis was used in order to examine howpatterns within and between the documents as key themes emerge. Such qualitative analysis combines both these analytical approaches, in order to reap the
benefits of the rich breadth of content contained within these documents, as wellas employing a structured approach to managing and organizing the data aroundkey topics (Nowell et al. 2017).
The keyword results akin to innovation were entered into a Microsoft Excel
Spreadsheet, to enable a systematic examination of the data according to emergent categories. The annual reports were examined in their original format (separate online PDF) and were subsequently merged into a comprehensive large-file
PDF (spanning 656 pages, excluding annexes), in order to allow for the systematic examination of data across SFIs. Throughout the analysis, a log containing raw
notes and emergent data was frequently updated. Color-specific codes were highlighted throughout the documents to illustrate pertinent points. The codes were
iteratively examined, and emergent patterns were labelled as potential themes
for further rumination. Data saturation was achieved when the scrutiny of all the
SFI annual reports and the examination of in-text references ceased to yield newthemes or content (Morse 1995). The themes and patterns that supported theiremergence were inspected by a wider study team, and any divergent perspectives
were resolved through group discussion.

40

Innovation Dimension 1: Economic Indicators
One of the most persistent and well-established elements of innovation is the economic performance (understood here as the ability to attract additional externalfunding, i.e., additionality) of a research endeavor, and particularly that of an SFI,
which is an established variant of a research center of excellence. After scrutiny of
the annual report data (2020), we have come across two principal findings. The firstis about the economic attractiveness of these innovation centers, as establishedthrough their ability to track and report on research funding.
From our sample, only one third of the centers tracks and reports on external financing from international sources (such as EU or other global opportunities). Thisis a finding which aligns also with the independent evaluation from the Damvad report on the lacking efforts of several SFIs to attract or align themselves for European
funding. Within this minority of centers, moreover, an additional finding has beenthat only 60% of these sources seeks for funding opportunities from the Horizon
mechanism. This finding is rather understandable given the formal agreement with
the RCN and its parameters of formal contractual obligations, yet telling with regards to the general positioning of these centers within the European landscape,which should generally be conducive to more European-targeted resources.

TRACKING OF EXTERNAL FUNDING (EF)_EUROPEAN OR INTERNATIONAL
SFIs

Tracking of
EF
35.7%

EF N/A
64.3%

Across SFIs

Other
internationalsources
40%

HorizonEurope/Horizon
2020
60%

Within EF mechanisms

Despite the seemingly low interest in engaging with external funding, the pictureof the future and the ability of these centers to kick-start new projects and attract
other types of funding is rather flourishing. Described with terms such as “pilot projects”, “associated projects from 2021”, “new projects/spin-off projects”, 5 of thesecenters have reported either completely new or recently started projects that emanate directly from the collaboration with industry or other research and industrial
partners. This is a rather encouraging finding about the ability to networks to cometogether, collaborate, and generate attractive research. And, ultimately, this attractiveness constitutes an encouraging positive trend in the workings of SFIs and theirscheme, which can further stimulate innovation collaboration between university
and industry.

41

NEWLY FINANCED SPIN-OFF PROJECTS
SFIs

Newly financed spin-off
projects
35.7%

No dataavailable
64.3%

Across SFIs

Innovation Dimension 2: People and Work life
In the second aspect of our innovation analysis, we have concentrated on theabilities of these research centers to track their human capital, and its success infinding their way to the academic or non-academic job market. In light of growing discontents about the role and function of PhDs in academia (Cyranoski et al.
2011; Germain-Alamartine et al. 2020; Germain-Alamartine and Moghadam-Saman
2020; Iversen et al. 2021; Janger et al. 2019; Sauermann and Roach 2016;), this is
an important indicator for establishing trust in the sector and the transferability ofknowledge to society (know-how, know-what and know-whom). However, the findings from these reports are rather alarming: only a small minority (20%) of thesecenters actively tracks the PhD employability of their candidates, which in itself is a
low figure in light of the developments for accountability in the sector. Moreover,this finding is compounded by the fact that two thirds of these tracked people doactually attain non-academic employment, which shows the importance of keeping
track of this employability metrics.
Further in line with the tracking of human capital, one dimension that has received
even scantier attention is the role and activation of MA students in the knowledgetransfer in society, as well as the students´ potential job prospects. Only the minority of SFIs still make explicit mention of their MA students’ contributions, andfrom this minority, only half make specific mentions of jobs or internships for MA
students. In light of growing emphasis on student entrepreneurship and attractiveness to the job market, this is a rather striking finding which merits attention andleaves room for growth with regards to integrating the prospects of research-based
innovation not only in the academic, but also in the industrial sector.

42

JOB PLACEMENT OF PHD GRADUATES
SFIs
PhD Employability
Tracking Data
21.4%

NonSpecified
33.3%

Industry
66.7%

PhD Employability
Data NA
78.6%

Across SFIs

Employability in
the Non-Academic
SectorINVOLVEMENT OF MASTERS STUDENTS
SFIs

Explicit
Mention of MA
students
42.9%
MA student
data NA
57.1%

Across SFIs

Employability
Data NA
50%

Jobs orInternships
50%

Employment Prospects ofMasters Students

43

Innovation dimension 3: Networks
Regarding now the third innovation dimension, namely, the importance of formal
networks and dissemination for the SFIs, the data reveal interesting patterns. On
the one hand, all SFIs display lengthy descriptions of their internationalization andnetworking activities, particularly also with regards to their formal agreementswith industry. However, only a fourth of the SFIs in our sample move beyond mere
mention of partnerships and actually describe the dynamics of their networking
activities (how partners enter or exit, how partnerships grow) as well as their deliberation and decision-making practices. This is an important finding, since such apattern would reveal not only formal agreements, but the very dynamics of thesecollaboration, and how these dynamics beget new knowledge (Costa et al. 2021;
Drejer and Jørgensen 2005; Guerrero and Urbano 2021; Hohberger et al. 2015;
Kaloudis et al. 2019; Köhler and Sönnichsen 2022). Moreover, this lack of dynamic
descriptions is further revealed in the lacking orientation of knowledge creation in
the ecosystem way of thinking, which allows for more spontaneous or serendipitous connections between partners, rather than the a priori assignment of work
packages and strict divisions of labor.
A second finding in the data which calls for our attention is the modes of dissemination and communication through social media channels, which go beyond an
SFI´s individual website (such as Twitter and LinkedIn). The data reveal a lacking
interest in engaging with external, well-established platforms, which could raise, inturn, questions about the scope and international orientation of both recruitment
and dissemination of research results. In addition, the detailed - and quantitatively-minded- tracking of impression data is rather rare in the SFIs under scrutiny,
which potentially impedes the broader understanding of networking dynamics anddissemination in these centers.

TRACKING DYNAMIC NETWORKING
SFIs

Ecosystem

Active Collaboration
Tracking
21.4%

PartnershipStatements
78.6%

Orientation
28.6%

Work Package
Orientation
71.4%

Across SFIs

44

DISSEMINATION THROUGH LINKEDIN
OR TWITTER
SFIs
DetailedTrackingof Impression
Data
20%

Use of
Twitter or
LinkedIN
28.6%
Home
Webpage
Only
71.4%

Brief
Mention80%

Across SFIs

Within LinkedIn
or Twitter Users

Finally, as a measure of their abilities to form networks, all SFIs present a very large
amount of total bulk of conferences and scientific arrangements they have participatedin (amounting to a total of ca.150), depending on the ways of counting these participations (presentations from the centers, presentations tangentially related tothe centers, posters and posters tangentially related), as well as more informal/
semi-formal arrangements (such as “virtual tech lunches”, “common trips”, “networking conferences”, “working groups”, “expert groups”, “workshops”, and “thematic meetings”).
From this large amount of reported arrangements, and despite the ubiquity of theterms “partner” in every SFI report (a total of 430 mentions), only 5 describe indetail their modus operandi for how they track in practice their collaboration with
their partners ( “invitations to annual meetings”, “adjunct professors for the industry partners”, “reference groups”, “innovation projects_ researchers working withindustry partners organizations”) and only one reported perhaps more inconvenient results, such as the departure/withdrawal of some partners (SFI Move). This
comparison between arrangements for the public and, conversely, for their moreprivate audience (between partners) can be rather telling with regards to disclosureof new innovation approaches and outcomes. However, the general state of the
SFIs ability to report their networking abilities is high and presents on average a lotof detail.

45

NETWORKING EVENTS
SFIs

Detailed tracking of all events(public and partners-only)
33.3%
Reported data on only public
networking events
66.7%

Across SFIs

Ideally, we should capture whether these events are purely scientific or including industry and, even more open lay audiences and public, with the purpose toachieve dissemination and develop the innovation ecosystem of the SFI. Given thevariability of quality of the annual reports and the time limitations of this pilot project, in-depth granular analysis of this specific aspect of the networking activities/
events was not possible to fully identify.
Further on the front of open access, the general issue of open access and open
license IP has been a cornerstone in the pursuit of innovation, particularly for research institutes and projects which seek to create connections across societal actors. However, the actual explicit mention or presence of the word open access islimited in the SFIs, with the vast majority (9 out of 14 centers) making no explicit
mention of open access in their annual reports. The centers which do use theopen access declarations do so either in the form of open databases (“e.g. open
source software toolbox”, “open drift trajectory model”, “open simulation platform
initiative”, “vessel response tool”), or with strategy declarations (“Open innovation
model”). In addition, the more specific pursuit of innovation deliverable, with open
license IP has been even strikingly low in the reports, with only one explicit mentionof this innovation outcome. This creates some dissonance between the general
growing consensus on the importance of pursuing open access in the university
sector on all possible fronts and the actual existing practices on the matter. To be
sure, our analysis here does not focus on the practice of open access publication,
which is only one of the ways that centers pursue openness (and the SFIs do publish on open access with growing rates). Rather, we are talking about other open
access tools, platforms, and databases which have direct transferability and operational value for work-life problems and practical challenges. In that respect, the SFI
practices still leave much room for growth and open access to all innovation fronts,
and particularly the free distribution of software or data.

46

OPEN ACCESS
SFIs

Explicit mention ofopen access
35.7%
No mention ofopen access
64.3%

Across SFIs

OPEN LICENSE IP
SFIs
Open license IP
7.1%

Open license data NA
92.9%

Within IPR-Specific SFI Deliverables

47

Innovation Dimension 4: Commercialization
Finally, with regards to Innovation Dimension 4 and the topic of commercialization,
the data reveal a very low interest (14.3 %) in the tracking of IPR data (patents, licenses, and DOFIs). Moreover, only half of this small minority group makes explicit
mention of spin-off ambitions or actions, raising also concerns about the ability ofthese centers to scale up their efforts and yield revenues for their stakeholders.

IPR: PATENTS, DOFI, AND SPIN-OFFS
SFIs
Tracking of IPR
14.3%

Holistic IPR

Solely Spin

Data

Offs

50%

50%

IPR Data N/A
85.7%

Across SFIs

Within IPR-Specific SFI
Deliverables

Unexplored Innovation Dimensions
The final set of data comes from more explorative avenues of innovation, namely,
public value innovation data. Our focus, commanded from the data available atthe annual reports, is on gender tracking and the ways in which these centers engage in making explicit gender dynamics in their research efforts. Alarmingly, thevast majority (64%) of these centers makes no mention of gender in their reporting,raising concerns about the implicit, yet persistent, biases that may inhere in the
dynamics of these centers. Moreover, some of this gender reporting is also making brief, rather than detailed, mentions of gender dynamics, showing that there
is room for improvement even at SFIs where there already exists awareness ofgender imbalance, and its ensuing problems for the public value generation thatthese SFIs propone. In line with a lot of growing research on implicit bias and its
role on gender balance and academic welfare in European academia and transnationally (Gvozdanović and Maes 2018; Menter 2020; Pritlove et al. 2019; Timmers
et al. 2010), gender is an aspect that should be scrutinized and accounted for in
the fostering of an innovation culture amongst SFIs. In that respect, this aspect ofinnovation leaves much room for growth and future development.

48

GENDER CONSIDERATIONS
SFIsBrief Mentions
20%
Explicit Gender
Mention
35.7%

No Gender
Mention64.3%
Detailed Data
80%

Across SFIs

Within Gender-Aware SFIs

Summary of findings and final reflections
One of the key conclusions that we can draw from analysis of the SFIs is that, as a
general trend, the establishment age of an SFI clearly plays a role in the volume andscope of reportability, the quality of reportability, and the readiness of disclosure.
For example, established and mature centers such as CIUS have reached a very
high level of reportability, presenting their readers with precise and granular innovation statistics (see figure below). On the other hand, the majority of innovation
centers present very little by means of innovation statistics, and in particular withregards to details of commercialization and economics of innovation that occur ineach center. Moreover, the dimensions of human capital management, particularly with regards to the labor market placement and market attractiveness of PhD
and MA students is not particularly salient in the reportability of these findings.
The same finding goes with regards to networking, where the attention to public events, as well as direct participation of partners in meetings is not met withvery meticulous reporting, making it hard to give a very clear picture as to whether
several of these networking activities are just scientific conferences or events, andwhether these events have bearing to a business partner audience or the generalpublic. This is an elemental dimension for the communication potential of an SFI,
and the general transferability and translatability of a center´s innovation findings
to its end users and, by extension, to the general public as clear and identifiable
innovation statistics.

49

Centre forInnovative
UltrasoundSolutions

CONFERENCECONTRIBUTIONS: 34

INNOVATION STATISTICS

CIUS 2020

1 NEW INDUSTRY PARTNER

1 NEW PRODUCT
27 NEW METHODS/PROCESSES
6 NEW SERVICES
0 PATENT APPLICATIONS
6 DISCLOSURE OF INVENTIONS
74 JOURNAL AND
PROCEEDINGS ATRICLES

PhDCIUS results (so far)

Master

‘17

‘18

‘19

‘20

DOFIs

4

7

7

6

Patent pending

0

1

1

0

Licencing considered or discussed

3

5

3

3

Licence agreements

0

0

0

0

Open Source licences

0

0

0

1

CIUS results (so far). (Row 2) Number of Disclosure of Innovation/inventions (DOFI); (Row 3) Numberof patent pending or patent applications; (Row 4) Number of licence considerations or discussions;(Row 5) Number of closed licence agreements; (Row 6) Number of Open Source Licences.

50

The general impression of innovation at SFIs are positive and definitely merit ourattention. The publication rates, the collaboration with industrial partners, as wellas the overall ability of SFIs to communicate across sectors and audiences, as wellas to generate measurable value, are solid and should not be taken for granted.
However, the reporting of innovation at SFIs leaves occasionally much room for
growth. First and foremost, as engines of growth, the SFIs could definitely allow for
more reporting and general public information on the additionalities they create,
namely, on the new EU/RCN or pilot projects that emanate directly from the establishment of an SFI. This kind of reporting of the positive innovation externalities ofan SFI would create a much better sense of accountability and trust in the innovation work that occurs, and the innovation spillovers that such work can generate.
To further corroborate this point, the aspect of commercialization and its measurement or effects on the innovation performance of an SFI are not well-reported.
Consequently, the establishment of a baseline of key performance indicators (KPIs)
for commercialization, particularly with regards to declaration of new ideas (DOFIs),
pending or approved patents, as well as spin-off companies is a key prospect for
future reporting of SFIs, which would create healthy benchmarks to strive towards
and reflect upon innovation efforts. A third dimension to allow for future improvement of SFI efforts is the more detailed reporting of activities that yield open access (data, codes, digital platforms, software programs, methods, open-source licenses). Open access is a critical dimension, both in terms of adding value to theend users of an innovation, but also for the broader public. This kind of open accessknowledge creates major benefits for translating scientific knowledge into practice,making research and international environments more globally competitive, andelevating the reach of knowledge beyond the academic walls.
A fourth dimension for the general improvement of SFIs reporting is the management of human capital, and the way centers engage with the labor market placement of their masters and PhD graduates, to secure the strategic placement offresh talent and new skills in both industry and academia.
Given the growing dissatisfaction with the market placement of PhDs and thefrequent misalignment of skills and work tasks (Cyranoski et al. 2011; GermainAlamartine et al. 2020; Germain-Alamartine and Moghadam-Saman 2020; Iversen
et al. 2021; Janger et al. 2019; Sauermann and Roach 2016;), the careful monitoringof the labor market placement of qualified human capital could mean a great deal
both for the credibility of the university as a site of growth, but also for the abilityof industry to absorb human capital and put forefront knowledge into practical,
occasionally life-altering applications. This is particularly true of a more neglected
aspect of the graduate population, namely, the master student graduates, which isa body of the university´s human capital which has not been carefully monitored
in terms of career development.5
Finally, an aspect of the SFIs which leaves room for development is that of thereportability of values such as gender balance, equal opportunity, and processtransparency in the recruitment and practical execution of innovation efforts. The
reportability of such considerations and less directly tangible aspects of value creation carry a great deal of weight not only to the results of innovation per se, butalso to the general reputational cascades of a university´s ecosystem of values.

5 In the analysis of the 14 NTNU departments, we do not include an indicator on employability of masters graduates since this is anindicator more relevant to measure as a direct educational output and not as a direct innovation output. However, there is no doubt that
this is an important indicator to capture at the level of SFI, department within a higher education institution (HEI), and at the level of of
HEIs.

51

Underscoring the prominence of this dimension can allow for greater trust and authority in the university´s role in society, and its ability to put forth examples worth
emulating and aspiring towards.
In general, the SFIs do display variability in their reporting of the innovation, andtheir ability to indicate or signal their innovation efforts, both in terms of innovation outcomes, but also in terms of innovation processes and the underpinning
dynamics that innovation is constructed upon. If we were to present our findings bymeans of an innovation spectrum, at the lower end we may put a more traditionalview of academic reporting with emphasis on scientific reporting and research outcomes. This kind of reporting, captured by publication metrics and the emphasis on
publication impact, is placing prime value in the ability of a research group to reportpathbreaking research and achieve visibility within a field or across research fields.
To be sure, this is a key function of an SFI, which is clearly mandated from the NFR
to “further develop elite research and innovation groups”.
On the opposite end of the spectrum, we find the prospect of innovation reporting,which places emphasis primarily on the innovation outcomes per se and considers
research as the springboard and means of innovation, not the end in itself. In thiskind of reporting, the prime emphasis is put on how innovation stems out of acenter´s research effort, and how these research efforts have direct applicability
in the end user´s, industrial or business partners´, and general public´s everyday
practices and working routines. In this fashion, innovation reporting turns the traditional dynamic of academic work on its head and prioritizes real-life applications
and hands-on transferability as the cornerstone of innovation, seeking to combine
the drive of curiosity with that of practical utility and measurable value creation.
And more strikingly, this kind of value creation does not simply encapsulate monetary value. Rather, it captures also more intangible value creation, such as that ofsocial and public values of more normative texture, such as internationalization
values, transparency, and open access to information, as well as gender balance
considerations.
This kind of reporting across the spectrum of both monetary/commercial and normative values lends innovation reporting with a greater sense of depth which spans
beyond commercial trends, fads, or financial gains, to incorporate public values
with a long-term and direct social purpose. This reporting can allow for innovation
to play a key role in the transformation of a university to an engine of growth, agrowth that is both value-centered, long term, and -in the final analysis- sustainable. Ultimately, this outlook of innovation outreach is in direct alignment with theobjectives and the findings of the overall pilot project, which targets broader innovation contributions in a university´s ecosystem.
Certainly, no dataset is without trade-off choices and limitations. The innovation
indicators offer various insights in terms of cost-benefit analysis for the SFIs, aswell as some significant methodological limitations for the kind of evidence that annual reports may represent (since 4 out of the 14 SFIs are very recently establishedand cannot be expected to provide tantamount levels of detail and granularity of
results).
In conclusion, the indicators which have been conceptualized for the evaluation of
the main innovation functions across an entire university ecosystem in part 1 ofthis report are not directly applicable to the structure and functions of an SFI, which
follows an organization logic premised on the call administered by the ResearchCouncil of Norway. Nonetheless, the four dimensions which underpin the suggested
indicators, namely, economic (measured as additionality performance, defined as
the ability to attract external funding in addition to the funding from the Research

52

Council of Norway and the SFIs industrial partners), people and work life, networks,
and commercialization remain highly relevant and salient for the structure and innovation functions of an SFI, as well as for the furthering of an innovation culturearound innovation reporting from these centers. Consequently, in order to further
develop and nuance the reporting on innovation, several ideas are in place. Firstand foremost, a common discontent also from the published research is the lack ofgeneral key performance indicators which can clearly orient researchers or centersabout their innovation presence. To be sure, we do not advocate for a linear modelof technology readiness level or catch-all terms. However, some preliminary consensus on what constitutes reliable performance indicators are still a desideratum
across centers, especially at larger scales of collaboration which entail more interaction volume and- therefore- more complexity. Secondly, the annual and generalreporting of each center does not seem to entail any insight into their specific innovation models. The reporting so far allows a great sense of outcome transparency, but little on process transparency, therefore hindering the understanding of
the dynamics that underpin deliberation and decisions. Consequently, allowing forsome reporting of inner constructive disagreements or fruitful arguments could
greatly enhance the process of innovation itself, which is well-established to be theresult of the contention with different perspectives. In that area, some input fromalumni or from graduate networks could be useful. Finally, the establishment ofactive dialogue with best practices with other centers could greatly advance thereporting practices for the mutual benefits of the individual SFI and for the funding
policy scheme as a whole.

53

References – Part 3
Abeywardana, N. L. E., & Panditharathna, K. M. (2016). The extent and determinants of voluntary disclosures in annual reports: evidence from banking and finance companies in Sri Lanka. Accounting and Finance Research, 5(4), 147–162.
Adams, C. A. (2018). Debate: Integrated reporting and accounting for sustainable
development across generations by universities. Public Money & Management,
38(5), 332–334. https://doi.org/10.1080/09540962.2018.1477580
Alexander, A., Martin, D. P., Manolchev, C., & Miller, K. (2018). University–industry
collaboration: using meta-rules to overcome barriers to knowledge transfer. The
Journal of Technology Transfer. https://doi.org/10.1007/s10961-018-9685-1
Ankrah, S., & AL-Tabbaa, O. (2015). Universities–industry collaboration: A systematic review. Scandinavian Journal of Management, 31(3), 387–408. https://doi.
org/10.1016/j.scaman.2015.02.003
Ansell, C., & Torfing, J. (2014). Collaboration and design: new tools for public
innovation. In Public Innovation through Collaboration and Design (pp. 19–36).
Routledge. https://doi.org/10.4324/9780203795958-9
Arvanitis, S., Kubli, U., & Woerter, M. (2008). University-industry knowledge andtechnology transfer in Switzerland: What university scientists think about co-operation with private enterprises. Research Policy, 37(10), 1865–1883. https://doi.
org/10.1016/j.respol.2008.07.005
Arundel, A., Bloch, C., & Ferguson, B. (2019). Advancing innovation in the public sector: Aligning innovation measurement with policy goals. Research Policy.
https://www.sciencedirect.com/science/article/pii/S0048733318302956
Atta-Owusu, K., Fitjar, R. D., & Rodríguez-Pose, A. (2021). What drives university-industry collaboration? Research excellence or firm collaboration strategy?
Technological Forecasting and Social Change. https://www.sciencedirect.com/
science/article/pii/S0040162521005175
Benneworth, P., Gulbrandsen, M., & Hazelkorn, E. (2016). Promoting Innovation,
and Assessing Impact and Value. In The Impact and Future of Arts and Humanities
Research (pp. 149–184). https://doi.org/10.1057/978-1-137-40899-0_6
Benneworth, P., & Jongbloed, B. W. (2010). Who matters to universities? A stakeholder perspective on humanities, arts and social sciences valorisation. Higher
Education, 59(5), 567–588. https://doi.org/10.1007/s10734-009-9265-2
Berkun, S. (2010). The Myths of Innovation. O’Reilly Media, Inc.
Bloch, C. (2011). Measuring Public Innovation in the Nordic Countries (MEPIN). https://www.diva-portal.org/smash/record.jsf?pid=diva2:707193
Bowen GA. (2009). Document Analysis as a Qualitative Research Method.
Qualitative Research Journal, 9(2), 27–40. https://doi.org/10.3316/QRJ0902027
Brennan, N. (2001). Reporting intellectual capital in annual reports: evidence from
Ireland. Accounting, Auditing & Accountability Journal, 14(4), 423–436. https://doi.
org/10.1108/09513570110403443

54

Brix, J. (2017). Exploring knowledge creation processes as a source of organizational learning: A longitudinal case study of a public innovation project.
Scandinavian Journal of Management, 33(2), 113–127. https://doi.org/10.1016/j.
scaman.2017.05.001
Brusca, I., Labrador, M., & Larran, M. (2018). The challenge of sustainability and integrated reporting at universities: A case study. Journal of Cleaner Production, 188,
347–354. https://doi.org/10.1016/j.jclepro.2018.03.292
Bryman, A. (2016). Social Research Methods. Oxford University Press.
Cheng, H., Zhang, Z., Huang, Q., & Liao, Z. (2020). The effect of university–industry collaboration policy on universities’ knowledge innovation and achievements
transformation: Based on innovation chain. The Journal of Technology Transfer.
https://link.springer.com/article/10.1007/s10961-018-9653-9
Costa, J., Neves, A. R., & Reis, J. (2021). Two Sides of the Same Coin. UniversityIndustry Collaboration and Open Innovation as Enhancers of Firm Performance.
Sustainability: Science Practice and Policy, 13(7), 3866. https://doi.org/10.3390/
su13073866
Coy, D., Tower, G., & Dixon, K. (1993). Quantifying the quality of tertiary education
annual reports. Accounting & Finance. https://doi.org/10.1111/j.1467-629X.1993.
tb00323.x
Creswell, J. & Poth, C. (2018). Qualitative inquiry and research design: Choosing
among five approaches (4th ed). Sage.
Cyranoski, D., Gilbert, N., Ledford, H., Nayar, A., & Yahia, M. (2011). Education: The
PhD factory. Nature News, 472(7343), 276–279. https://doi.org/10.1038/472276a
David, C. (2001). Mythmaking in Annual Reports. Journal of Business and TechnicalCommunication, 15(2), 195–222. https://doi.org/10.1177/105065190101500203
Del Sordo, C., Farneti, F., Guthrie, J., Pazzi, S., & Siboni, B. (2016). Social reports in
Italian universities: disclosures and preparers’ perspective. Meditari Accountancy
Research, 24(1), 91–110. https://doi.org/10.1108/MEDAR-09-2014-0054
Ditlevsen, M. (2012). Revealing corporate identities in annual reports. Corporate
Communications: An International Journal, 17(3), 379–403. https://doi.
org/10.1108/13563281211253593
Drejer, I., & Jørgensen, B. H. (2005). The dynamic creation of knowledge: Analysing
public–private collaborations. Technovation, 25(2), 83–94. https://doi.org/10.1016/
S0166-4972(03)00075-0
Forsberg, E.-M. (2019). Responsible research and innovation in the broader innovation system: reflections on responsibility in standardisation, assessment andpatenting practices. In International Handbook on Responsible Innovation. Edward
Elgar Publishing. https://doi.org/10.4337/9781784718862.00017
Gamage, P., & Sciulli, N. (2017). Sustainability reporting by Australian universities. Australian Journal of Public Administration, 76(2), 187–203. https://doi.
org/10.1111/1467-8500.12215

55

Garcia, R., Araújo, V., Mascarini, S., Santos, E. G., & Costa, A. R. (2020). How longterm university-industry collaboration shapes the academic productivity of research groups. Innovations. https://www.tandfonline.com/doi/abs/10.1080/144793
38.2019.1632711
Germain-Alamartine, E., Ahoba-Sam, R., Moghadam-Saman, S., & Evers, G. (2020).
Doctoral graduates’ transition to industry: networks as a mechanism? Cases from
Norway, Sweden and the UK. Studies in Higher Education, 1–16. https://www.tandfonline.com/doi/abs/10.1080/03075079.2020.1754783
Germain-Alamartine, E., & Moghadam-Saman, S. (2020). Aligning doctoral education with local industrial employers’ needs: a comparative case study. European
Planning Studies, 28(2), 234–254. https://doi.org/10.1080/09654313.2019.1637401
Gertner, D., Roberts, J., & Charles, D. (2011). University‐industry collaboration: a
CoPs approach to KTPs. Journal of Knowledge Management, 15(4), 625–647. https://doi.org/10.1108/13673271111151992
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013). Seeking Qualitative Rigor in
Inductive Research: Notes on the Gioia Methodology. Organizational Research
Methods, 16(1), 15–31. https://doi.org/10.1177/1094428112452151
Gray, PS. J.B. Williamson, D.A. Karp, J.R. Dalphin. (2007) Content analysis. P.S.
Gray, J.B. Williamson, D.A. Karp, J.R. Dalphin (Eds.), The Research Imagination: An
Introduction to Qualitative and Quantitative Methods, Cambridge University Press,
Cambridge (2007), pp. 203-304.
Guan, J., & Noronha, C. (2013). Corporate social responsibility reporting researchin the Chinese academia: a critical review. Social Responsibility, Journalism, Law,
Medicine, 9(1), 33–55. https://doi.org/10.1108/17471111311307804
Guerrero, M., & Urbano, D. (2021). Looking inside the determinants and the effects of entrepreneurial innovation projects in an emerging economy. Industry andInnovation, 28(3), 365–393. https://doi.org/10.1080/13662716.2020.1753021
Gulbrandsen, M., & Aanstad, S. (2015). Is innovation a useful concept for arts andhumanities research? Arts and Humanities in Higher Education, 14(1), 9–24. https://doi.org/10.1177/1474022214533890
Gvozdanović, J., & Maes, K. (2018). Implicit Bias in Academia: A Challenge to the
Meritocratic Principle and to Women‘s Careers--and What to Do About It. Leagueof European Research Universities (LERU) Advice Paper No, 23. https://www.leru.
org/files/LERU-PPT_Bias-paper_Jadranka_Gvozdanovic_January_19_18.pdf
Hohberger, J., Almeida, P., & Parada, P. (2015). The direction of firm innovation:
The contrasting roles of strategic alliances and individual scientific collaborations.
Research Policy, 44(8), 1473–1487. https://doi.org/10.1016/j.respol.2015.04.009
Iversen E., Andreadakis, Z. & Capasso, M. (2021). Labor-market placement of doctorate degree holders in Norway. Work in Progress.
Presentation Link available at: https://www.risis2.eu/2021/03/23/
the-labor-market-placement-of-trained-phds-outside-academia/
Janger, J., Campbell, D. F. J., & Strauss, A. (2019). Attractiveness of jobs in academia: a cross-country perspective. Higher Education, 78(6), 991–1010. https://doi.
org/10.1007/s10734-019-00383-7

56

Kaloudis, A., Aspelund, A., Koch, P. M., Lauvås, T. A., Mathisen, M. T., Strand, Ø.,Sørheim, R., & Aadland, T. (2019). How Universities Contribute to Innovation: A
Literature Review-based Analysis. Oslo: NTNU.
Kelly, G., Mulgan, G., & Muers, S. (2002). Creating public value: An analytical framework for public service reform. London: Strategy Unit, Cabinet Office.
Kent, P., & Zunker, T. (2017). A stakeholder analysis of employee disclosures inannual reports. Accounting and Finance, 57(2), 533–563. https://doi.org/10.1111/
acfi.12153
Köhler, J., Sönnichsen, S. D., & Beske-Jansen, P. (2022). Towards a collaboration
framework for circular economy: The role of dynamic capabilities and open innovation. Business Strategy and the Environment. https://doi.org/10.1002/bse.3000
Leitner, K.-H. (2004). Intellectual capital reporting for universities: Conceptual
background and application for Austrian universities. Research Evaluation, 13(2),129–140. https://doi.org/10.3152/147154404781776464
Link, A. N., Siegel, D. S., & Wright, M. (2015). The Chicago Handbook of University
Technology Transfer and Academic Entrepreneurship. University of Chicago Press.
Marianne, G. D. (2012). Revealing corporate identities in annual reports. Corporate
Communications: An International Journal, 17(3), 379–403. https://doi.
org/10.1108/13563281211253593
Meng, D., Li, X., Rong, K., (2019.) Industry-to-university knowledge transfer in ecosystem-based academic entrepreneurship: case study of automotive dynamics &control group in
Tsinghua University. Technol. Forecast. Soc. Change 141, 249–262. https:// doi.
org/10.1016/j.techfore.2018.10.005
Menter, M. (2020). Entrepreneurial universities and innovative behavior: the impact of gender diversity. Economics of Innovation and New Technology, 1–15. https://doi.org/10.1080/10438599.2020.1843988
Moggi, S. (2019). Social and environmental reports at universities: a Habermasianview on their evolution. Accounting Forum, 43(3), 283–326. https://doi.org/10.1080/
01559982.2019.1579293
Nicolo, G., Manes-Rossi, F., Christiaens, J., & Aversano, N. (2020). Accountability
through intellectual capital disclosure in Italian Universities. Journal of Management
and Governance, 24(4), 1055–1087. https://doi.org/10.1007/s10997-019-09497-7
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014).
Standards for reporting qualitative research: a synthesis of recommendations. Academic Medicine: Journal of the Association of American Medical
Colleges, 89(9), 1245–1251. https://doi.org/10.1097/ACM.0000000000000388
OECD (2010). Measuring innovation: A new perspective. Paris.
OECD 2015. The Innovation Imperative: Contributing to Productivity, Growth and WellBeing. (n.d.). Retrieved November 6, 2021, from https://www.oecd.org/innovation/the-innovation-imperative-9789264239814-en.html

57

OECD (2021). Science, Technology and Innovation Outlook 2021: Times of Crisis andOpportunity. (n.d.). Retrieved November 6, 2021, from https://www.oecd.org/sti/
oecd-science-technology-and-innovation-outlook-25186167.html
Paloma Sánchez, M., Elena, S., & Castrillo, R. (2009). Intellectual capital dynamics
in universities: a reporting model. Journal of Intellectual Capital, 10(2), 307–324.
https://doi.org/10.1108/14691930910952687
Perkmann, M., Tartari, V., McKelvey, M., Autio, E., Broström, A., D’Este, P., Fini, R.,
Geuna, A., Grimaldi, R., Hughes, A., Krabel, S., Kitson, M., Llerena, P., Lissoni, F.,
Salter, A., & Sobrero, M. (2013). Academic engagement and commercialisation: A
review of the literature on university–industry relations. Research Policy (Vol. 42,
Issue 2, pp. 423–442). https://doi.org/10.1016/j.respol.2012.09.007
Pritlove, C., Juando-Prats, C., Ala-Leppilampi, K., & Parsons, J. A. (2019). The good,the bad, and the ugly of implicit bias. The Lancet, 393(10171), 502–504. https://doi.
org/10.1016/S0140-6736(18)32267-0
Roncancio-Marin, J., Dentchev, N., Guerrero, M., Díaz-González, A., & Crispeels,
T. (2022). University-Industry joint undertakings with high societal impact: A micro-processes approach. Technological Forecasting and Social Change, 174, 121223.
https://doi.org/10.1016/j.techfore.2021.121223
Santema, S., & van de Rijt, J. (2001). Strategy disclosure in Dutch annual reports. European Management Journal, 19(1), 101–108. https://doi.org/10.1016/
S0263-2373(00)00075-X
Sauermann, H., & Roach, M. (2016). Why pursue the postdoc path? Science,
352(6286), 663–664. https://doi.org/10.1126/science.aaf2061
Sorescu, A., & Schreier, M. (2021). Innovation in the digital economy: a broader
view of its scope, antecedents, and consequences. Journal of the Academy ofMarketing Science, 49(4), 627–631. https://doi.org/10.1007/s11747-021-00793-z
Talebzadehhosseini, S., Garibay, I., Keathley-Herring, H., Al-Rawahi, Z. R. S.,
Garibay, O. O., & Woodell, J. K. (2021). Strategies to enhance university economic
engagement: evidence from US universities. Studies in Higher Education, 46(6),1112–1131. https://doi.org/10.1080/03075079.2019.1672645
Thomas, J. (1973). Discourse in the Marketplace: The Making of Meaning in Annual
Reports. The Journal of Business Communication (1973), 34(1), 47–66. https://doi.
org/10.1177/002194369703400103
Timmers, T. M., Willemsen, T. M., & Tijdens, K. G. (2010). Gender diversity policies in universities: a multi-perspective framework of policy measures. Higher
Education, 59(6), 719–735. https://doi.org/10.1007/s10734-009-9276-z
Waller, D. S., & Lanis, R. (2009). Corporate Social Responsibility (CSR) Disclosureof Advertising Agencies: An Exploratory Analysis of Six Holding Companies’
Annual Reports. Journal of Advertising, 38(1), 109–122. https://doi.org/10.2753/
JOA0091-3367380107

58

SFI Annual Reports
1. NTNU SFI Centre for Geophysical Forecasting (2020). Annual Report.
Available at: https://www.ntnu.edu/documents/1294850894/1298578697/
AnnualReport2020_ver2.8.pdf/809c6cb0-8718-1a1f-3ef2-55fc39d52bc7?t=1619024740285
2. CIRFA – Centre for Integrated Remote Sensing and Forecasting for Arctic
Operations (2020). Annual Report. Available at: https://cirfa.uit.no/wp-content/
uploads/2021/06/CIRFA_arsrapport_2020.pdf
3. CIUS: Centre for Innovative Ultrasound Solutions (2020).
Annual Report. Available at: https://www.ntnu.edu/documents/1262947227/1268580566/CIUS_Annual_Report_2020_web_small.
pdf/0ef21000-36ba-45a5-dc21-afd0b9f2c1be?t=1616786836933
4. Industrial Catalysis Science and Innovation (2020).
Annual Report. Available at: https://www.ntnu.edu/documents/1263635073/0/ICSI+Annual+Report+2020+%281%29.
pdf/5fbf695c-721c-5c88-1437-25eaac184cbd?t=1616961541660
5. SFI Klima 2050 (2020). Annual Report. Available at: https://static1.
squarespace.com/static/54ff1c6be4b0331c79072679/t/607565bc22cbb67d51081dd6/1618306502761/klima2050_annualreport2020_mobileversion.
pdf
6. SFI Metal Production (2020). Annual Report. Available at: https://indd.adobe.
com/view/d4c387fc-cddc-465f-accb-14952642b285
7. SFI NorwAI (2020). Annual Report. Available at: https://www.ntnu.edu/documents/1294735055/1298789471/NorwAi+annual+report+2020_Final_06-04-21.
pdf/e0d72759-84fa-e5d5-4e35-a4c0da706891?t=1617696048907
8. SFI NORCICS Norwegian Center for Cybersecurity in Critical
Sectors. Annual Report. Available at: https://www.ntnu.edu/documents/1294734959/1300988649/SFI+NORCICS+Annual+Report+2020.pdf/
c2189003-5079-9646-4d3b-8dbc1c4a063c?t=1624015907570
9. SFI Blues (2020). Annual Report. Available at: https://www.sintef.no/globalassets/sintef-ocean/sfi-blues/sfi_blues_annual_report_2020_print.pdf
10.SFI Harvest (2020). Annual Report. Available at: https://sfiharvest.no/
wp-content/uploads/2021/07/SFI_Harvest_annual_report_2020_LR.pdf
11.SFI Move (2020). Annual Report. Available at: https://www.ntnu.edu/
documents/1266972546/0/SFI_MOVE_%C3%85rsrapport_2020-final.
pdf/20f0689d-47a4-4bf7-da8c-3e5976c67766?t=1618225746079
12.SFI Sirius (2020). Annual Report. Available at: https://sirius-labs.no/wp-content/
uploads/2021/07/Sirius-Annual-Report-2020-lowres.pdf
13.SFI Smart Maritime (2020). Annual Report. Available at: http://www.
smartmaritime.no/documentation/activity-plans-reports/annual-reports/
smart-maritime-annual-report-2020/744/
14.SFI SUBPRO Submarine Production and Processing. (2020-2021). Annual
Report. Available at: https://folk.ntnu.no/skoge/subpro-annual-reports/
SUBPRO%20Annual%20Report%202020-2021.pdf

59

Annex 1
Complete Version
of Recommended
Indicators for NTNU
in the NTNU-report2019a(English and Norwegian Version:
45 Indicators)

60

A. Indicators for Indirect Contribution
A1. Innovation competence and innovation culture
1. Master students’ satisfaction with the educational trajectory’s emphasis forinnovation.
2. Master students’ satisfaction with the educational trajectory’s contribution toentrepreneurial skills.
3. Employers’ satisfaction with the Master students’ ability for innovative
thought.
4. Applications/Filings for patents from employees and students at universities.
5. Submitted innovation ideas from employees and students at universities.
6. STUD-ENT applications from the university
7. Applications for internal incentive funds from the university (f.i. NTNU
Discovery for NTNU)
8. Number of participations in architectural competitions, business ideas competitions, and other disciplinary competitions.
9. Trademark applications from employees and students at universities.
10. Design applications from employees and students at universities
11. Number of applications to RCN
12. Number of applications to Horizon 2020
13. Number of internal positions with a focus on innovation
14. Study programs and topics/teaching modules with innovation focus (EVU) as
well as number of study point credits and students
A2. Collaboration Indicators (Universities’ academic core areas)
Project Collaboration
15. Income from innovation-oriented NFR projects (BIA, FORNY etc)
16. Income from participation in Innovasjon-Norge projects (IFU/OFU etc.)
Collaboration with Industry
17. Number of articles with co-authorship from the business sector
18. Part of Research and Development (FoU) considered as directly relevant fromthe creation of value in the industrial sector.
19. Research and Development financed from the industry sector as a part ofcollaborative Research and Development
20. Number of participants as project partners in SKATTEFUNN (Tax Deduction
Scheme for Companies with Research and Development Projects).
Other Collaboration
21. Share of articles with co-authorship with the public sector (hereafter healthtrusts)22. Share of articles with co-authorship with research institutes
23. Share of BOA-income as part of total revenues
24. Number of leadership of excellence clusters and excellence centers (SFF, SFI,
FME, GCE, Arena, NCE)
25. Number of Participation in excellence clusters and excellence centers (SFF, SFI,
FME, GCE, Arena, NCE)

61

26. Bachelor and master theses in collaboration with external and/or as part of a
research project
27. Number of shared positions as a share of faculty positions
28. Number of cooperation agreements (memoranda of understanding) withcompanies and public enterprises
Mobility and shared Positions
29. University-employed and student participation in public committees
30. Number of Professor II positions financed from external funding sources
31. Number of former PhD students with employment in innovation-participating
businesses/enterprises
32. Number of Business-financed PhDs with a sole university as an awardgranting institution
B. Indicators for Direct Contribution (results/changes)
B1. Commercialization and business/industry development
33. Initiated licencing contracts from students and employees at the university
34. Registered business establishment from students and employees at theuniversity
35. Registered patents from students and employees at the university
36. Registered trademarks from students and employees at the university
37. Registered designs from students and employees at the university
38. RCN’s STUD-ENT awards in a university
39. Reported results regrading business development from SFF, SFI, FME andexcellence clusters
40. Reported results regarding business development from other RCN-funded
projects
41. Reported results regarding business development from EU-funded projects
B2. Innovation in the public sector and society in general
42. Number of innovations from the health sector in collaboration with theuniversity
43. Prizes and distinctions (for example for good design)
44. Qualitative examples regarding innovation in society initiated by the university
45. Results of the university’s artistic development work

62

Norwegian Version (NTNU-report 2019: 61-63)
A. Indikatorer for indirekte bidrag
1. Innovasjonskompetanse og
innovasjonskultur

KilderNasjonalt
sammenlignbar

Internasjonalt
sammenlignbar

Master-studenters tilfredshet
med utdanningens vektlegging
av evne til nyskaping

NIFU/Kandidatundersøkelsene

X

Master-studenters tilfredshet
med utdanningens bidrag til
entreprenørskapsevne

NIFU/Kandidatundersøkelsene

X

Arbeidsgiveres tilfredshet med
masterkandidaters evne tilnytenkning

NIFU/Arbeidsgiverundersøkelsen

X

Patentsøknader fra ansatte og
studenter ved universitetet

DBH/TTO/FORNY/SPARK/ES

X

X

Mottatte innovasjonsideer
fra studenter og ansatte ved
universitetet

DBH/TTO/FORNY/SPARK/ESX

X

STUD-ENT-søknader fra
universitetet

FORNY

X

Søknader til interne incentivmidler ved universitetet (eks NTNU
Discovery for NTNU)

Det enkelte universitet (ForNTNU: NTNU Discovery/
Innovasjonsstipend etc)

(X)

Antall deltakelser i arkitektkonkurranser, forretningsidekonkurranser og andre faglige
konkurranser

Norske arkitekters landsf./andre
kilder(X)

Varemerkesøknader fra ansatteog studenter ved universitetet

Patentstyret

X

X

Designsøknader fra ansatte og
studenter ved universitetet

Patentstyret

X

X

Antall søknader til innovasjonsrettede NFR-prosjekter

NFR

X

Antall søknader til Horisont 2020

EcordaX

Antall interne stillinger med
innovasjonsfokus

Det enkelte universitet(X)

Studieprogrammer og -emner
med innovasjonsfokus (inkl EVU)
antall studiepoeng og kandidater

Det enkelte universitet

(X)

63

2. Samspillsindikatorer(universiteters faglige
kjerneområder)Kilde

Nasjonalt
sammenliknbar

Internasjonalt
sammenliknbar

Inntekter fra innovasjonsrettede NFRprosjekter (BIA, FORNY mv.)

Det enkelte universitet /NFR

XInntekter fra deltakelse i Innovasjon
Norge-prosjekter (IFU/OFU/ mv.)

Det enkelte universitet /IN

X

Inntekter fra Horisont 2020

ECORDAX

X

Andel artikler med samforfatterskap
med næringsliv

Christin/NIFU

X

X

Andel FoU vurdert som direkte relevant
for verdiskaping i næringslivet

NIFU/FoU-statistikk

X

FoU finansiert av næringslivet som
andel av samlet FoU

NIFU/FoU-statistikk

X

Antall deltakelser som prosjektpartner i
Skattefunn-prosjekter

Skattefunn

X

Andel artikler med samforfatterskap
med offentlig sektor (herunderhelseforetak)

Christin/NIFU

X

Andel artikler med samforfatterskap
med forskningsinstitutter

Christin/NIFU

X

Andel BOA-inntekter som andel av
samlede inntekter

KD/DBHX

Antall ledelser av klynger og sentra(SFF, SFI, FME, GCE, Arena, NCE)

Det enkelte universitet/
NFR

X

Antall deltakelser i klynger og sentra(SFF, SFI, FME, GCE, Arena, NCE)

Det enkelte universitet/
NFR

X

Bachelor- og masteroppgaver i samarbeid med eksterne og/eller som en delav forskningsprosjekter

Det enkelte universitet

Antall delte stillinger som andel av
faglige stillinger

NTNU

Antall samarbeidsavtaler med bedrifter
og offentlige virksomheter

Det enkelte universitet

Prosjektsamarbeid

Samarbeid med næringslivØvrig samarbeid

X

Mobilitet og delte stillinger

64

Universitets-ansatte og studentersdeltakelse i offentlige utvalg

Organbasen

X

Antall professor II finansiert av
eksterne kilder

NIFU/
ForskerpersonalregisteretX

Antall tidligere PhD. studenter
med arbeid i innovasjonsaktive
virksomheter

NIFU/
Doktorgradsmonitoren

X

Kandidater utdannet fra universitet
med arbeid i innovasjonsaktive
bedrifter/virksomheter

NIFU/SSB-Registerdata

X

Antall Nærings-PhD med det
enkelte universitet som gradsgivende
institusjon

NFR

X

X

(X)

B. Indikatorer for direkte bidrag(resultater/endringer)

3. Kommersialisering ognæringsutvikling

Kilde

Nasjonalt
sammenlignbar

Internasjonalt
sammenlignbarInngåtte lisensieringskontrakter fra
studenter og ansatte ved universitetet

DBH/TTO/FORNY

X

XRegistrerte bedriftsetableringer fra
studenter og ansatte ved universitetet

TTO/FORNY/
Brønnøysund

X

X

Registrerte patenter fra ansatte og
studenter ved universitetet

DBH/TTO/FORNY

X

XRegistrerte varemerkebeskyttelser fra
ansatte og studenter ved universitetet

SSB/Patentstyret

X

X

Registrerte design fra ansatte og
studenter ved universitetet

SSB/Patentstyret

X

X

NFRs STUD-ENT-tildelinger tiluniversitetet

NFR/FORNY

X

Rapporterte resultater om næringsutvikling fra SFF, SFI og FME og klynger

Universitetet/sentrene(se tabell 4.3)

X

Rapporterte resultater om næringsutvikling fra andre NFR-prosjekter

NFR

Rapporterte resultater om næringsutvikling fra EU-prosjekterEU

4. Innovasjon i offentlig sektor og
samfunnet for øvrig
Antall innovasjoner fra helsesektoren isamarbeid med universitetet

Helseforetakenes system
for innovasjonsutvikling

Priser og utmerkelser (bl.a. for godt
design)DOGA m.fl.

Kvalitative eksempler på innovasjon isamfunnet med utspring i universitetet

Universitetet, basert på
ulike metoder

Resultater av universitetets kunstneriske utviklingsarbeid

Research Catalogue foraristic researchX65