# Quadruple-helix in university innovation programs: toward a digital tool for socio-economic impact evaluation

Alex Gabriel, Ferney Osorio, Laure Morel

► To cite this version:

Alex Gabriel, Ferney Osorio, Laure Morel. Quadruple-helix in university innovation programs: toward a digital tool for socio-economic impact evaluation. 2023 IEEE 29th International Conference on Engineering, Technology and Innovation (ICE/ITMC), University of Edinburgh, Jun 2023, Edinburgh, Scotland, United Kingdom. 10.1109/ICE/ITMC58018.2023.10332436. hal-04189022

HAL Id: hal-04189022

https://hal.science/hal-04189022v1

Submitted on 27 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Quadruple-helix in university innovation programs: toward a digital tool for socio-economic impact evaluation

Alex Gabriel

Université de Lorraine
ERPI

F-54000 Nancy, France

https://orcid.org/0000-0002-3676-6417

Ferney Osorio

Université de Lorraine
ERPI

F-54000 Nancy, France

https://orcid.org/0000-0001-6199-2304

Laure Morel

Université de Lorraine
ERPI

F-54000 Nancy, France

https://orcid.org/0000-0002-6666-8375

**Abstract**— This research work presents the development of a digital tool designed to assist in the impact evaluation of quadruple helix strategic innovation programs in university settings. The motivation for this tool stemmed from a 9-year university project that sought to generate socio-economic impacts through a range of activities involving stakeholders such as economic actors, policy makers, and citizens. The project proposal primarily emphasized output indicators for evaluating the overall project and its individual actions. However, due to the diverse nature of the project, which comprised multiple work packages with distinct objectives and involving different stakeholders, formulating an impact evaluation strategy based on the concept of a results chain within the micro-meso-macro environment framework proved challenging. Furthermore, effectively communicating this strategy with collaborators proved even more difficult. Consequently, there emerged a need for a tool that could facilitate the structuring of the assessment strategy and provide a comprehensive list of commonly used indicators for evaluating the impact of innovation programs. The proposed tool aims to integrate multiple perspectives within the project to offer indicators aligned with specific objectives and the results chain. Moreover, leveraging these indicators, the tool enables the creation of data collection forms and generates a multifactorial dashboard of outcome indicators.

**Keywords**—quadruple helix, impact evaluation, project management, methodology, digital tool

## I. INTRODUCTION

Helix innovation models are characterized by their multi-structural, multi-functional, and nonlinear nature [1]. Over the past few decades, these models have evolved from the triple helix to the quadruple helix and quintuple helix innovation models. The triple helix model assumes that knowledge-based societies and economies rely on the institutional spheres of the university, industry, and government [2]. The quadruple helix model extends this triad by including civil society as a central actor in the innovation process, emphasizing the involvement of users and citizens [2]. The quintuple helix model further incorporates the natural environment as the fifth helix, acknowledging its dynamic influence [2]. Cross-sectoral collaborations, inherent in the triple/quadruple/quintuple helix models, are essential for successful innovation, as they allow for a comprehensive understanding of diverse perspectives, facilitate knowledge exchange, provide additional opportunities (e.g., funding, projects, products), reduce redundant knowledge, and stimulate innovation and economic performance [3].

The helix innovation model particularly emphasizes the role of the university as a crucial player within this local synergy [4]. However, challenges persist regarding the optimal management of interactions among the quadruple helix actors to enhance the effectiveness and efficiency of an innovation ecosystem [3], [5]. Successful management of this collaboration necessitates the development and monitoring of performance measures [4].

This research work is conducted within the context of implementing a strategic innovation program at a university. The program aims to establish mechanisms for promoting the adoption of the Mode 3 "non-linear innovation mode" [2], [6] on a university-wide scale. The primary objective of the program is to enhance the socio-economic impact of the university and research institutions within the region. To achieve this goal, the program is structured into four synergistic work packages. The first work package focuses on fostering collaboration between the university and companies through the establishment of thematic communities. These communities are complemented by public events that involve public authorities as well. The second work package aims to facilitate interactions between all innovation stakeholders (universities, companies, public authorities, and citizens) by creating innovation spaces. These spaces serve as testing grounds for innovations, host ideation workshops, and facilitate conferences involving citizens. The third work package addresses the issue of fostering innovation culture within the university and research institutions by transforming organizational structures into more flexible and adaptable ones, with a global approach to end-user needs. This is achieved through actions such as personal training plans and streamlining internal and external interfaces. Lastly, the fourth work package focuses on improving the visibility of the university and research institutions, along with their innovation strategy. However, when practitioners involved in the program's assessment attempted to monitor and evaluate its impact, they encountered a lack of clear and accessible methodologies.

This study aims to outline the role and characteristics of a tool designed to support the management of multi-stakeholder programs within the framework of the quadruple helix innovation model. In other words, we address the question of how operational managers and practitioners can be supported in monitoring and evaluating the impact of a program based on the quadruple helix innovation model. To answer this question, this paper is structured as follows. Firstly, we provide an overview of different impact evaluation
approaches found in the literature, along with the digital tools and indicators associated with them. Next, we establish the foundations for this experimental work through a research action approach that guided the co-design process. We then present the first version of the tool, along with the initial results from user testing. Finally, we discuss the upcoming features and development of the tool before concluding on this work. Finally, the upcoming features and development will be presented in the discussion before concluding about this work.

## II. EXISTING THEORIES & PREVIOUS WORK

### A. Impact Evaluation Approaches and Tools

According to the discussions presented by [7], impact can be defined as “the positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”. Impact evaluation, in turn, refers to the systematic process of assessing the value or significance of the outcomes of an intervention, such as an activity, program, or policy. This process involves analyzing the long-term effects on the beneficiaries of the intervention, both positive and negative, primary and secondary, and understanding how these interventions contribute to producing an effect, in order to derive valuable lessons [8]. One of the main objectives of impact evaluations is to facilitate dialogue among stakeholders at different decision-making levels within a specific initiative. Evaluations serve as a feedback loop, enabling stakeholders to learn, adapt their actions, and develop common strategies [9]. In the field of innovation, this has led to the development of various frameworks and indices that offer guidance on where to focus and what to monitor when assessing the impact of innovation programs.

Within the context of the helix innovation model assessment, several indexes are available. Reference [10] introduces composite innovation metrics, which utilize multi-criteria decision analysis techniques, notably AHP (Analytic Hierarchy Process) and TOPSIS (Technique for Order Preference by Similarity to Ideal Solution), to determine the weight of indicators and evaluate alternative options [10]. Additionally, Reference [3] presents an efficiency index for innovation systems, based on a two-phase Data Envelopment Analysis (DEA) calculation method. DEA is a commonly used approach for evaluating the efficiency of innovation systems, involving the measurement of performance for decision-making units that convert multiple inputs into multiple outputs [12]. DEA is also utilized in other fields, such as urbanism, to evaluate urban smartness and enhance the effectiveness of policymakers' actions [13].

Beyond innovation assessment, there is an emerging field known as Social Impact Assessment (SIA). SIA is defined as “the process of identifying the future consequences of a current or proposed action which are related to individuals, organizations and social macro-systems” [14]. The objective of SIA is to ensure that developments or planned interventions maximize the benefits and minimize the costs, particularly those borne by the community [14]. SIA is a social science discipline/practice that is widely supported and promoted by the International Association for Impact Assessment (IAIA) [15], which has identified the core values and principles of SIA. Social impact encompasses various aspects, such as changes in people's way of life, culture, community, political system, environment, health and well-being, personal and property rights, as well as fears and aspirations [14]. In

essence, impact refers to a change in an outcome caused by an organization, which can be positive or negative, intended or unintended [16]. An outcome refers to the level of well-being experienced by a group of people or the condition of the natural environment resulting from an event or action [16].

In practice, there are multiple evaluation approaches available, including cost-benefit analysis (CBA), cost-effectiveness analysis (CEA), and multi-criteria decision analysis (MCDA) [17]. CBA estimates the total expected benefits of a program compared to its total expected costs [18]. It involves assigning monetary values to all costs and benefits associated with a project or policy strategy and examining the ratio of total benefits to total costs [17]. CEA is commonly used to assess the efficiency of specific technologies, programs, or policies by comparing multiple alternatives [17]. It estimates the initial capital and annualized operating costs of a policy and compares total costs against a specific criterion or alternative scenario [17]. However, the limitation of these two approaches lies in the translation of non-monetary aspects into costs. MCDA serves as a solution to this limitation [17] and is suggested as an approach for evaluating the impact of territorial programs with multi-scale and multi-level perspectives [19].

Nevertheless, impact evaluations are distinct from other types of evaluations in that they specifically aim to address cause-and-effect questions. In the context of impact evaluation, impact refers to the changes that can be directly attributed to a program, program modality, or design innovation [18]. While CBA and CEA provide cost-related information, they complement impact evaluation [18]. The primary role of impact evaluation is to generate evidence regarding the performance of a program, preferably through effectiveness studies conducted under normal conditions. These studies allow for broader generalizations compared to efficacy studies, which assess a program under ideal conditions [18]. To observe the changes induced by a program or policy, impact assessment is based on the theory of change and the result chain. A theory of change provides a description of how an intervention is expected to deliver the desired results [18]. Theory of change evaluations examine whether the expected outcomes actually occur and to what extent they can be attributed to the interventions [20]. They explore the conditions and assumptions necessary for the desired change to take place, make the causal logic behind the program explicit, and map the interventions along logical causal pathways [18]. Logic models and result chains are commonly used representations of theories of change. Logic models present the relationships among the inputs, activities, and outcomes of a program, while result chains depict the assumed causal linkages between interventions and desired impacts through a series of intermediate results [20]. The “logic model” covers both implementation and results [18]. Implementation concerns the work delivered by the project, including inputs, activities, and outputs [18]. A good results chain will help surface assumptions and risks implicit in the theory of change [18]. Results chains are often equated to logic models, but in reality, they are much more specific and show direct assumed relationships among discrete actions, intermediate outcomes, and the desired final impact [20]. It is notably the case of reference [18] that presents the logical model as a result chain.

An effective impact evaluation that utilizes theory of change and result chains is framed by a well-structured
evaluation question, which should be testable and quantifiable [18]. The evaluation question needs to be accompanied by the desired outcomes and relevant indicators to track the causal logic [18]. Defining indicators along the result chain is crucial for monitoring and assessing the impact of interventions [18].

## B. Indicators

The measurement and monitoring of helix innovation models serve as a control mechanism for policy implementation, enabling the estimation of their efficiency and effectiveness [3]. Performance evaluation plays a crucial role in improving the interactions within multi-helix systems, as it helps identify weak links and good practices within the monitored systems [3]. Furthermore, assessing the effectiveness of multi-helix systems can contribute to the development of tools for ranking global innovation competitiveness [3]. Indicators are necessary for conducting evaluations, and internationally comparable indicators have been developed and utilized for cross-country comparisons [10]. The use of limited single indicators has been replaced by multiple and composite approaches, which facilitate evaluation, benchmarking, and policy formulation [10].

At the European level, there are three innovation scoreboards: the European Innovation Scoreboard (EIS), previously known as the Innovation Union Scoreboard (IUS), and the Regional Innovation Scoreboard (RIS). In the triple helix efficiency index developed by [3], 19 indicators have been selected and classified based on their functions, namely evaluating wealth generation, normative control, or novelty production.

## C. Digital tools for evaluation

Based on a comprehensive literature review across various research fields, a limited number of digital tools have been identified to support the impact evaluation of programs, projects, or policies. Many of these tools are methodological aids and templates for workshops, such as those available in the Social Impact Toolbox [21] and the World Bank resources. Most of the digital tools related to impact evaluation are primarily focused on conservation actions and sustainable development. One well-known tool in this regard is the SDG Impact Assessment tool [22], a web application that enables organizations to self-assess their impact in relation to the 17 sustainable development goals. Users are prompted to determine the relevance of each development goal to their situation and then evaluate and justify the impact (Figure 1-A and 1-B).

Another tool in this domain is Miradi, “a project management software designed specifically for conservation practitioners” [23]. One notable feature of Miradi is its diagraming capabilities, which facilitate the description of the situation and the formalization of the theory of change. The graphical formalism employed in Miradi is based on the results chains presented in [20]. Both Miradi [23] and the graphical formalism presented in [20] were developed by the Foundation of Success, an organization focused on improving conservation practices. The foundation also provides guidance on applying results chains in the context of conservation [24].

Outside the conservation field, Sopact [25] offers a tool for measuring organizational impact with a significant emphasis on technology and data. Sopact is based on the impact management norms defined by the Impact Management Project [16].

From a technological perspective, there is an underexplored domain concerning the knowledge modeling of impact assessment. Some exploration has been conducted in the field of Environmental Impact Assessment (EIA) to structure and develop EIA methodologies. In this regard, the authors created an ontology reference for EIA methodology developers to access structured information, including natural language definitions [26]. However, this ontology is no longer available, and only partial graphical representations of the article remain. The use of semantic technology has also been considered to manage data heterogeneity and reduce uncertainty in flood modeling and flood risk management [27].

Interestingly, the few examples of ontology use in the field of impact assessment are predominantly focused on sustainability and the environment, and they tend to lack a generic representation of the theory of change.

In conclusion, the state of the art in impact assessment and supporting tools reveals several approaches and resources

<figure><img src="image_2.png" /><figcaption>B – Evaluation of the impact on the goal</figcaption></figure>

<caption>Figure 1. Screenshots of the SDG Impact Assessment tool</caption>
available. The theory of change and the result chain stand out as the primary frameworks utilized in many guides and materials for implementing impact assessment, such as those provided by AVISE. However, there is a limited availability of technological tools to assist in the practical implementation of these approaches, particularly in the context of triple/quadruple helix innovation models.

The development of a technological tool specifically tailored to impact assessment in the context of helix innovation models would have significant value. Such a tool could facilitate the structured collection of impact evaluation data from various projects, utilizing a shared vocabulary to describe different concepts. This would enable meta-analysis and synthesis of the collected data. Additionally, the use of a semantic model would contribute to clear conceptualization of impact, allowing for a better understanding of the positive and negative interactions between different impacts and their distribution across diverse socio-demographic groups [28].

### III. METHODS

#### A. Research question

Given the inherent complexity of impact assessment and the intricate nature of programs and projects requiring such evaluations, it is important to acknowledge that individuals involved in project or program evaluation may not possess expertise in impact assessment. Furthermore, the abundance of methodological guides available, coupled with the multitude of decisions to be made during the impact assessment process, and the limited availability of IT tools for data collection, visualization, and monitoring, have led us to contemplate the following question:

How to digitally support operational managers and practitioners in monitoring and evaluating the impact of a program based on helix innovation models?

The ambition is to formalize a process and create a digital tool that would help innovation program managers. The perspective of this tool would be to be open and accessible to various structures involved in quadruple helix innovation approaches. One of the inspirations is the eNAP tool of the German federal government [29]. This SIA tool has had an indirect effect on the way SIA works and in particular on the access to relevant information for impact assessment [30].

#### B. Research methods

This work adopts an action research approach, combining empirical observations and inputs from a conceptual background to inform the different phases of the approach. Action research methodology [31], [32] is commonly employed to study the innovation process. It involves a continuous learning process where researchers actively participate and collaborate with the participants [32]. The typical process of action research comprises planning, acting, observing, and reflecting steps [33]. In this study, the researchers collaborated with the operational program manager within the strategic program to carry out the work.

The development of the digital tool follows a two-folded process. Firstly, a tool was developed to validate the usefulness of a digital tool for impact evaluation with practitioners. Secondly, the initial version was refined to enhance the user experience and transition towards a more

generic architecture utilizing linked data principles [34], [35]. The second step involves transferring or mirroring data from a database to a triple store. This article focuses on the first phase of design and development, which also includes a preliminary evaluation.

The development process involved collaboration between researchers and the operational program manager. The evaluation phase encompassed the participation of the operational program manager, work package leaders, and other actors involved in impact evaluation in a separate project. The evaluation employed the standardized UX questionnaire "Attrakdiff" [36] and post-test interviews to gain insights into participants' experiences and identify potential areas for improvement. This evaluation and the interviews took place following the completion of a predefined scenario.

### IV. INTERMEDIATE RESULTS

#### 1) Tool architecture

In order to systematize impact evaluation at the university scale, an initial version of a web application has been developed. The architecture of the tool is based on a microservices architecture. Drawing from a systematic mapping study on microservice architecture [37], the adopted architecture, as depicted in Figure 2, is inspired by the API-gateway pattern. However, certain technological choices made regarding data storage have facilitated the prototyping process but have limited control over the APIs that provide data access.

The front-end of the application was developed using VueJS<fnref n="1" />, while the data was stored in Appwrite<fnref n="2" />, an open-source backend server that offers APIs for data access. Considering the intention to work with linked data, another API was designed to interact with the triple store, utilizing OwlReady2 [38].

#### 2) Functionalities

Three distinct user roles were identified for the tool:

* Program managers who formalize the project/program to evaluate. It means their responsibility is to define the scope, the goals, the potential work packages, the result chain, the indicators associated with elements of the results chain, and questions associated with the indicators. They also manage access right to the tool.

<figure><img src="image_3.png" /><figcaption>Figure 2. Schema of the architecture of the tool (dashed rectangle not yet implemented)</figcaption></figure>

<footnote n="1">¹ Vue.JS page: https://vuejs.org/ (21/02/2023)</footnote>

<footnote n="2">² Appwrite page: https://appwrite.io/ (21/02/2023)</footnote>
* Evaluators who are responsible for collecting data concerning the program. They have access mainly to the evaluation forms and dashboard.

* Viewers who are stakeholders of the program/project but not directly involved in the evaluation. They might be interested in the result of the evaluation and so access to dashboards.

Based on the identified user roles, the tool encompasses eight main functionalities:

* Define the program's goal and, if necessary, work packages along with their respective goals.

* Formalize the result chain of the program and its work packages based on their goals. Each activity within the program and work packages can have its own result chain, consisting of inputs, outputs, outcomes, and final outcomes (Figure 3).

* Add indicators to the database and list/filter them according to the step they evaluate. Indicators related to university-industry collaboration have been included for program managers' reference [39]. These indicators are categorized based on the step they correspond in the result chain (inputs, outputs, activities, outcomes, or final outcomes). However, program managers have the flexibility to define new indicators using a text field.

<figure><img src="image_4.png" /><figcaption>Figure 3. Result chain formalization on the program</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4. Indicators formalization with objectives for a given result chain item</figcaption></figure>
* Assign indicators to the elements of interest in the theory of change and set yearly objectives for these indicators (Figure 4). Cumulative indicators can be defined, where data collected over the measurement period can be aggregated. For example, the number of visits can be considered cumulative as it can be aggregated to provide an annual indicator.

* Formalize evaluation questions for the assigned indicators. These questions will be presented to the evaluators when assessing the program's progress.

* Access to a form for evaluating the progress of the program and each work package. Evaluators can input their assessments and provide feedback on each element. A dashboard is available to help evaluators track their evaluations, ensuring they have not missed any and allowing them to revisit previous evaluations for corrections if needed.

* Visualize the data for each indicator associated with the program and its work packages. The data can be viewed in a comparative manner across different years and can be filtered for each specific indicator. Additionally, a second level of visualization is provided to assess the achievement of objectives.

* Manage access to the program evaluation based on user roles. The creator of a program evaluation has administrative access and can invite individuals to participate in the evaluation, assigning them roles as administrators, evaluators, or viewers. Access can be restricted accordingly.

The usage of the tool can be divided into these three main phases, each involving specific user roles (Table 1): program evaluation setup, data collection and program monitoring. Table 1. Involvement of roles according to the phases

<table><thead><tr><td>Phase</td><td>Role</td></tr></thead><tbody><tr><td>Program evaluation setup</td><td>Program Manager</td></tr><tr><td>Data collection</td><td>Evaluator</td></tr><tr><td>Program monitoring</td><td>Program Manager<br>Viewer</td></tr></tbody></table>

In the first phase of program evaluation setup, the program manager is responsible for performing the following tasks:

* **Formalizing the Program:** The program manager elicits the goals of the program and defines the goals of the work packages associated with it. The manager also establishes the result chain for each activity within the program. During this process, the manager can refer to guiding questions to assist in formalizing these elements effectively.

* **Associating Indicators:** The program manager associates relevant indicators to each element of the result chain. The tool provides preloaded indicators that can be selected, or the manager can define new indicators as needed. Indicators serve as measurable metrics to assess the progress and impact of the program.

* **Defining Yearly Objectives or Baseline:** For each indicator, the program manager sets yearly objectives or establishes a baseline. These objectives or baselines are used to calculate the completion rate of indicators during the evaluation process.

* **Defining Evaluation Questions:** The program manager formulates the questions that will be presented to the evaluators during the data collection phase.

In the second phase of data collection, the following tasks are involved:

* **Completing Evaluation Form:** One or more evaluators fill out the evaluation form corresponding to the specific work package they are assigned to.

* **Monitoring Evaluation:** Evaluators have access to an evaluation dashboard, which provides them with an overview of their assigned evaluations. The dashboard allows evaluators to review and correct any previously filled-out evaluations and fill out any evaluations that may have been missed.

The third phase, program monitoring, is not strictly sequential to the second phase. It can be carried out concurrently with the data collection phase. The tasks involved in program monitoring are as follows:

* **Exploring Indicators:** Program managers, evaluators, or viewers can access bar charts representing the indicator performance through bar charts for each work package. It also provides a bar chart for objective fulfillment based on the ratio between result and objectives.

* **Exploring Objective Contribution:** Users can also access bar charts that illustrate the contribution of each objective toward the overall fulfilment of the work package.

The tasks supported by the tools can be represented as an activity diagram as illustrated by Figure 5.

### 3) Preliminary feedback

During the design meetings focusing on impact evaluation methodologies and indicators, it was observed that program members lacked awareness of the theory of change and result chain approach. The initial step involved establishing a shared vocabulary to foster mutual understanding of the various concepts related to impact evaluation. This phase culminated in the development of an ontology that defined and clarified these concepts [40]. However, as the main objective of this article is to present the initial outcomes of the digital tool and its associated process, detailed discussion of the ontology is beyond its scope.

Subsequently, attention turned to data visualization, which was swiftly addressed after achieving a common understanding of impact evaluation. The operational program manager and those responsible for managing indicators expressed enthusiasm for visualizing indicator data through yearly histograms and histograms representing the measure-objective ratio over time. The challenge entailed creating an application that would be accessible and comprehensible to individuals unfamiliar with the result chain concept.
<figure><img src="image_6.png" /><figcaption>Figure 5. Activity diagram of the task supported by the program evaluation tool</figcaption></figure>

To assess the usability and user experience of the initial version of the tool and its impact evaluation implementation, a standardized UX questionnaire called AttrakDiff was administered to four individuals involved in impact evaluation using AttrakDiff Shiny [41]. Although these individuals were familiar with project impact evaluation, they had no prior knowledge of the result chain concept. The usability tests were conducted with the participants assuming the role of operational program managers, following a scripted scenario that provided them with program and program evaluation information but offered no guidance on where to input the information in the application.

Qualitative feedback collected during the post-test interviews complete the evaluation. The experience was described as qualitative but with a downside in the ability to use it without the instructions. Some users clearly stated the need for additional explanations of the meaning of the terms used in the interface and questions to guide the formulation of result chain elements. Another comment concerned the homogenization of interface behaviors, especially when entering data. An improvement on this aspect would impact the unpredictable-predictable item.

The questionnaire results, as shown in Figure 6, indicate that the early testers generally perceived the application positively based on the portfolio of results, which combines pragmatic and hedonic dimensions. While Figure 7 lacks confidence intervals, each dimension's score is very close to 1, making it difficult to determine if it is neutral or positive.

However, the overall evaluation leans towards the positive side of the graph. Figure 8 provides further insights to address the research question. The application appears to be satisfying in various aspects. From an attractiveness perspective, it is seen as pleasant, attractive, inviting, and likeable, but not highly motivating. The identification items indicate that the application is considered presentable and professional,

<figure><img src="image_7.png" /><figcaption>Figure 6. Portfolio of results</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 5. Mean value diagram for each dimension</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6. Graph of word pairs</figcaption></figure>

confirming the aesthetic choices made. In terms of social aspects, such as socialization and integration, the evaluation is neutral, despite the presence of team management and access control features. Further study using Kano evaluation could provide additional insights to interpret this evaluation. Collaboration features may be considered a "must-be" aspect, meaning that while the feature is present, users do not expect anything more. The stimulation items show a neutral response, which is not surprising as the goal is not to be creative, innovative, or bold. In terms of pragmatic items, the application is seen as manageable, efficient, and practical, although additional effort is needed to be evaluated as simple.

Qualitative feedback obtained from post-test interviews provided supplementary insights for the evaluation process. Participants described their experience as qualitative but identified a limitation in their ability to use the tool without instructions. Several users explicitly expressed the need for further explanations regarding the terminology used in the interface, as well as guidance in formulating result chain elements.

Additionally, participants commented on the need for more consistent interface behaviors, particularly during data

entry. They highlighted the importance of homogenizing these behaviors to improve user experience, which would have an impact on the "unpredictable-predictable" aspect of the tool.

### 4) Discussion & perspective

This work represents an intermediate stage in developing a digital tool for program impact evaluation. The tool was evaluated with four individuals directly involved in impact evaluation activities. The results from the AttrakDiff questionnaire indicate a generally positive perception, considering the limitations of the laboratory setting and artificial testing scenarios [42].

This initial evaluation highlights priority areas for improvement and validates the design choices. Despite a generally positive evaluation, there is a need to enhance the tool's simplicity, particularly in supporting the operational program manager in formalizing the result chain. Adding hints and tips to the interface is an upcoming improvement. Furthermore, new indicators were created to better suit the program context, and further work is needed for indicator capture and classification. The authors anticipate further development of the ontology.

### 5) Match & Contribution

The tool presented in this article aims to support program evaluation and assist in their design. According to the authors, this article contributes to the technology community in terms of capability and productivity. It is particularly relevant to fields such as program and project management, strategy, and the socioeconomic impact of engineering and technology management.

## V. CONCLUSIONS

Evaluating the impact of a program, especially in the context of a quadruple helix innovation model, presents significant challenges. This work aims to address these challenges by providing a digital tool designed to support operational managers and practitioners. This article presents the intermediary results of a prototype digital tool developed and utilized within an ongoing 9-year strategic innovation program at a French university. Although the tool offers a pleasant, practical, manageable, and efficient user experience, it is not yet considered simple. From an innovation management perspective, the tool aims to gather indicators for monitoring quadruple helix innovation and formalizing the result chain.

One major limitation of this study is the limited number of participants involved in the evaluation. While the tool has been tested on a larger user base, assessing its true impact can only be done over time. At this stage, there is no conclusive evidence of the tool's real benefits for impact studies. To further investigate this, the list of indicators should be expanded through a systematic literature review, and additional features should be developed to facilitate users' interpretation of the collected data, such as analyzing relationships, cause and effect, alignment, and more. Collecting and structuring data are merely the initial steps in conducting a comprehensive study of programs based on the quadruple helix innovation model.

## ACKNOWLEDGMENT

This work was supported partly by the French PIA project « LUE-SIRIUS », reference ANR-20-IDES-0008.

## REFERENCES
[1] M. Meyer, K. Grant, P. Morlacchi, et D. Weckowska, « Triple Helix indicators as an emergent area of enquiry: a bibliometric perspective », Scientometrics, vol. 99, no 1, p. 151-174, avr. 2014, doi: 10.1007/s11192-013-1103-8.

[2] E. G. Carayannis et R. Rakhmatullin, « The Quadruple/Quintuple Innovation Helixes and Smart Specialisation Strategies for Sustainable and Inclusive Growth in Europe and Beyond », J. Knowl. Econ., vol. 5, no 2, p. 212-239, 2014, doi: 10.1007/s13132-014-0185-8.

[3] M. Jovanović, G. Savić, Y. Cai, et M. Levi-Jakšić, « Towards a Triple Helix based efficiency index of innovation systems », Scientometrics, vol. 127, no 5, p. 2577-2609, mai 2022, doi: 10.1007/s11192-022-04304-x.

[4] M. M. Jovanović, J. Đ. Rakićević, V. M. Jeremić, et L. Jakšić, « How to measure Triple Helix performance? A fresh approach », 2018, p. 16.

[5] E. G. Carayannis, E. Grigoroudis, D. F. J. Campbell, D. Meissner, et D. Stamati, « The ecosystem as helix: an exploratory theory-building study of regional co-opetitive entrepreneurial ecosystems as Quadruple/Quintuple Helix Innovation Models », RD Manag., vol. 48, no 1, p. 148-162, 2018, doi: 10.1111/radm.12300.

[6] H. Etzkowitz et L. Leydesdorff, « The dynamics of innovation: from National Systems and “Mode 2” to a Triple Helix of university-industry-government relations », Res. Policy, vol. 29, no 2, p. 109-123, févr. 2000, doi: 10.1016/S0048-7333(99)00055-4.

[7] H. White, « A Contribution to Current Debates in Impact Evaluation », Evaluation, vol. 16, no 2, p. 153-164, avr. 2010, doi: 10.1177/1356389010361562.

[8] E. Stern, N. Stame, J. Mayne, K. Forss, R. Davies, et B. Befani, « Broadening the range of designs and methods for impact evaluations », Institute for Development Studies, avr. 2012. doi: 10.22163/fteval.2012.100.

[9] E. Saari et K. Kallio, « Developmental Impact Evaluation for Facilitating Learning in Innovation Networks », Am. J. Eval., vol. 32, no 2, p. 227-245, juin 2011, doi: 10.1177/1098214010387658.

[10] E. G. Carayannis, Y. Goletsis, et E. Grigoroudis, « Composite innovation metrics: MCDA and the Quadruple Innovation Helix framework », Technol. Forecast. Soc. Change, vol. 131, p. 4-17, juin 2018, doi: 10.1016/j.techfore.2017.03.008.

[11] E. G. Carayannis, E. Grigoroudis, et Y. Goletsis, « A multilevel and multistage efficiency evaluation of innovation systems: A multiobjective DEA approach », Expert Syst. Appl., vol. 62, p. 63-80, nov. 2016, doi: 10.1016/j.eswa.2016.06.017.

[12] A. Ishizaka et P. Nemery, Multi-criteria decision analysis: methods and software. Chichester, West Sussex, United Kingdom: Wiley, 2013.

[13] F. Ramirez, P. Palominos, M. Camargo, et D. Grimaldi, « A new methodology to support smartness at the district level of metropolitan areas in emerging economies: The case of Santiago de Chile », Sustain. Cities Soc., vol. 67, no January, p. 102713, 2021, doi: 10.1016/j.scs.2021.102713.

[14] H. A. Becker et F. Vanclay, Éd., The International handbook of social impact assessment: conceptual and methodological advances. Cheltenham, UK ; Northampton, MA, USA: Edward Elgar, 2003.

[15] IAIA, « The leading global network on impact assessment ».
https://www.iaia.org/ (consulté le 22 février 2023).

[16] Impact Frontiers, « Impact Management Norms | Impact Frontiers », 24 février 2022. https://impactfrontiers.org/norms/ (consulté le 21 février 2023).

[17] D. Browne et L. Ryan, « Comparative analysis of evaluation techniques for transport policies », Environ. Impact Assess. Rev., vol. 31, no 3, p. 226-233, avr. 2011, doi: 10.1016/j.eiar.2010.11.001.

[18] P. J. Gertler, S. Martinez, P. Premand, L. B. Rawlings, et C. M. J. Vermeersch, Impact Evaluation in Practice. 2016. doi: 10.1596/978-1-4648-0779-4.

[19] M. Enjolras, L. Morel, F. Kasmi, et L. Dupont, « Toward a systemic decision-making tool of territorial resilience based on multicriteria methodologies: The case of the DHDA Project », 2021 IEEE Int. Conf. Eng. Technol. Innov. ICEITMC 2021 - Proc., 2021, doi: 10.1109/ICE/ITMC52061.2021.9570118.

[20] R. Margoluis et al., « Results Chains: a Tool for Conservation Action Design, Management, and Evaluation », Ecol. Soc., vol. 18, no 3, p. art22, 2013, doi: 10.5751/ES-05610-180322.

[21] B. Dalton et al., « The Frameworks - Social Impact Toolbox », Social Impact Toolbox, 2019. https://www.socialimpacttoolbox.com/the-frameworks/ (consulté le 21 février 2023).

[22] Gothenburg Centre for Sustainable Development, « SDG Impact Assessment Tool ». https://sdgimpactassessmenttool.org/en-gb (consulté le 21 février 2023).

[23] Foundations of Success, « About Miradi - Miradi Share 2.0 ». https://www.miradishare.org/ux/about (consulté le 21 février 2023).

[24] Foundations of Success, « Using Results Chains to Improve Strategy Effectiveness: An FOS How-To Guide », Foundations of Success, Bethesda, Maryland, USA, 2007.

[25] Sopact, « Impact Measurement and Management Software ». https://www.sopact.com (consulté le 21 février 2023).

[26] J. Garrido et I. Requena, « Proposal of ontology for environmental impact assessment: An application with knowledge mobilization », Expert Syst. Appl., vol. 38, no 3, p. 2462-2472, 2011, doi: 10.1016/j.eswa.2010.08.035.

[27] V. Nundloll, R. Lamb, B. Hankin, et G. Blair, « A semantic approach to enable data integration for the domain of flood risk management », Environ. Chall., vol. 3, p. 100064, avr. 2021, doi: 10.1016/j.envc.2021.100064.

[28] A. Dumitru, N. Frantzeskaki, et M. Collier, « Identifying principles for the design of robust impact evaluation frameworks for nature-based solutions in cities », Environ. Sci. Policy, vol. 112, p. 107-116, oct. 2020, doi: 10.1016/j.envsci.2020.05.024.

[29] C. Wanckel, « eNAP: An Electronic Sustainability Assessment Tool for the Ministerial Bureaucracy in Germany », Tropico Project, 28 mars 2019. https://tropico-project.eu/cases/administration-costs-for-bureaucracy/enap-an-electronic-sustainability-assessment-tool-for-the-ministerial-bureaucracy-in-germany/ (consulté le 23 février 2023).

[30] C. Wanckel, « Introducing a digital tool for sustainability impact assessments within the German Federal Government: A neo-institutional perspective », Int. Rev. Adm. Sci., p. 00208523211047093, oct. 2021, doi: 10.1177/00208523211047093.

[31] B. Somekh, Action Research: A Methodology for Change and Development. Open University Press, 2005.

[32] J. Clark, S. Porath, J. Thiele, et M. Jobe, « Action Research », NPP EBooks, janv. 2020, [En ligne]. Disponible sur: https://newprairiepress.org/ebooks/34

[33] A. Burns, « Action Research », in Qualitative Research in Applied Linguistics: A Practical Introduction, J. Heigham et R. A. Croker, Éd., London: Palgrave Macmillan UK, 2009, p. 112-134. doi: 10.1057/9780230239517_6.

[34] A. Versteden, E. Pauwels, et A. Papantoniou, « An Ecosystem of User-facing Microservices supported by Semantic Models », in USEWOD-PROFILES@ESWC, 2015.

[35] A. Versteden et E. Pauwels, « State-of-the-art Web Applications using Microservices and Linked Data », in CEUR Workshop Proceedings, 2016. doi: 10.5281/zenodo.1233427.

[36] C. Lallemand, V. Koenig, G. Gronier, et R. Martin, « Création et validation d'une version française du questionnaire AttrakDiff pour l'évaluation de l'expérience utilisateur des systèmes interactifs », Eur. Rev. Appl. Psychol., vol. 65, no 5, p. 239-252, sept. 2015, doi: 10.1016/j.erap.2015.08.002.

[37] D. Taibi, V. Lenarduzzi, et C. Pahl, « Architectural Patterns for Microservices: A Systematic Mapping Study », in Proceedings of the 8th International Conference on Cloud Computing and Services Science, Funchal, Madeira, Portugal: SCITEPRESS - Science and Technology Publications, 2018, p. 221-232. doi: 10.5220/0006798302210232.

[38] J.-B. Lamy, « Owlready: Ontology-oriented programming in Python with automatic classification and high level constructs for biomedical ontologies », Artif. Intell. Med., no 80C, p. 11-18, 2017, doi: 10.1016/j.artmed.2017.07.002.

[39] E. Albats, I. Fiegenbaum, et J. A. Cunningham, « A micro level study of university industry collaborative lifecycle key performance indicators », J. Technol. Transf., vol. 43, no 2, p. 389-431, avr. 2018, doi: 10.1007/s10961-017-9555-2.

[40] A. Gabriel, « Ontology for Program Impact Evaluation », Ontology specification draft, 2023. https://purl.org/pieo (consulté le 24 mars 2023).

[41] A. Gabriel, « AttrakDiff Shiny », déc. 2022, doi: 10.17605/OSF.IO/8RTBH.

[42] C. Lallemand et V. Koenig, « Lab Testing Beyond Usability: Challenges and Recommendations for Assessing User Experiences », vol. 12, no 3, 2017.
<figure><img src="image_10.png" /></figure>